Classification method and device of steady-state visual evoked potentials, and electronic equipment

Abstract

The application provides a steady-state visual evoked potential classification method and device and electronic equipment. The method comprises the following steps: obtaining a steady-state visual evoked potential signal; based on a plurality of different frequency band bandpass filters, extracting short time domain window data corresponding to each frequency band in the plurality of different frequency bands from the steady-state visual evoked potential signal; inputting the short time domain window data corresponding to the plurality of frequency bands into a pre-trained hybrid network model to obtain a classification result of the steady-state visual evoked potential signal; wherein the hybrid network model comprises a convolutional neural network and a bidirectional gated recurrent unit, the convolutional neural network is used to obtain a fusion feature map based on the short time domain window data corresponding to the plurality of frequency bands, and compress the fusion feature map into one-dimensional data; the bidirectional gated recurrent unit is used to obtain the classification result of the steady-state visual evoked potential signal based on the one-dimensional data. The application can accurately classify the short time domain window potential signal data and improve the classification performance.

Description

Technical Field

[0001] This invention relates to the field of electroencephalogram (EEG) signal classification technology, and in particular to a method, apparatus, and electronic device for classifying steady-state visual evoked potentials. Background Technology

[0002] Brain-computer interface (BCI) technology can convert electroencephalogram (EEG) signals expressing human intentions into control commands. It was initially used to assist patients with motor disorders or partial motor impairments in interacting with their external environment. Among EEG signals, steady-state visual evoked potentials (VEVPs) are widely studied in the field of BCI technology due to their advantages such as high classification accuracy and high signal-to-noise ratio.

[0003] Currently, filter bank convolutional neural networks are mainly used to classify SSVEP (Steady-State Visual Evoked Potential). However, when using sliding windows of different sizes to extract data from the filtered data, the classification performance for EEG signals in short time-domain windows is poor because the features of EEG signals are not obvious in the short time-domain window, and the phase information of each electrode channel is also ignored. Summary of the Invention

[0004] This invention provides a method, apparatus, and electronic device for classifying steady-state visual evoked potentials, in order to solve the problem of poor classification performance of EEG signals in short time-domain windows in related technologies.

[0005] In a first aspect, embodiments of the present invention provide a method for classifying steady-state visual evoked potentials, including:

[0006] Acquire steady-state visual evoked potential signals;

[0007] Based on bandpass filters of multiple different frequency bands, short time-domain window data corresponding to each frequency band in multiple different frequency bands are extracted from steady-state visual evoked potential signals;

[0008] Short-time-domain window data corresponding to multiple frequency bands are input into a pre-trained hybrid network model to obtain classification results of steady-state visual evoked potential signals. The hybrid network model includes a convolutional neural network and a bidirectional gated recursive unit. The convolutional neural network is used to obtain a fused feature map based on the short-time-domain window data corresponding to multiple frequency bands and compress the fused feature map into one-dimensional data. The bidirectional gated recursive unit is used to obtain classification results of steady-state visual evoked potential signals based on one-dimensional data.

[0009] Secondly, embodiments of the present invention provide a classification device for steady-state visual evoked potentials, comprising:

[0010] The acquisition module is used to acquire steady-state visual evoked potential signals;

[0011] The extraction module is used to extract short time-domain window data corresponding to each frequency band from the steady-state visual evoked potential signal based on bandpass filters of multiple different frequency bands.

[0012] The computation module is used to input short-time-domain window data corresponding to multiple frequency bands into a pre-trained hybrid network model to obtain the classification results of steady-state visual evoked potential signals. The hybrid network model includes a convolutional neural network and a bidirectional gated recursive unit. The convolutional neural network is used to obtain a fused feature map based on the short-time-domain window data corresponding to multiple frequency bands and compress the fused feature map into one-dimensional data. The bidirectional gated recursive unit is used to obtain the classification results of steady-state visual evoked potential signals based on the one-dimensional data.

[0013] Thirdly, embodiments of the present invention provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method as described in the first aspect or any possible implementation of the first aspect.

[0014] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method as described in the first aspect or any possible implementation thereof.

[0015] This invention provides a method, apparatus, and electronic device for classifying steady-state visual evoked potentials (SVPs). The method involves acquiring a steady-state visual evoked potential (SVP) signal; extracting short-time-domain window data corresponding to each frequency band from the SVP signal using multiple bandpass filters of different frequency bands; inputting the short-time-domain window data corresponding to multiple frequency bands into a pre-trained hybrid network model to obtain the classification result of the SVP signal; wherein the hybrid network model includes a convolutional neural network (CNN) and a bidirectional gated recursive unit (BRU). The CNN is used to obtain a fused feature map based on the short-time-domain window data corresponding to multiple frequency bands and compress the fused feature map into one-dimensional data; the BRU is used to obtain the classification result of the SVP signal based on the one-dimensional data. This invention is applied to short-time-domain window SVP signals. By extracting and fusing features from the signals corresponding to the short-time-domain windows in the SVP signal, a fused feature map is obtained. Finally, the fused feature map is compressed into one-dimensional data, and the classification result is obtained through the BRU. The category to which the current SVP signal belongs can be determined based on the evaluation result. This invention achieves the technical effect of accurately classifying short-time-domain window potential signal data and improving classification performance. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is an application scenario diagram of the classification method for steady-state visual evoked potentials provided in the embodiments of the present invention;

[0018] Figure 2 This is a flowchart illustrating the implementation of the classification method for steady-state visual evoked potentials provided in this embodiment of the invention.

[0019] Figure 3 This is a schematic diagram of the hybrid network model of the classification method for steady-state visual evoked potentials provided in an embodiment of the present invention;

[0020] Figure 4 This is a structural diagram of the spatial convolution submodule of the classification method for steady-state visual evoked potentials provided in this embodiment of the invention;

[0021] Figure 5 This is a structural diagram of the temporal convolution submodule of the classification method for steady-state visual evoked potentials provided in this embodiment of the invention;

[0022] Figure 6 This is a structural diagram of the bidirectional gated recursive unit submodule of the classification method for steady-state visual evoked potentials provided in this embodiment of the invention;

[0023] Figure 7 This is a structural diagram of the hybrid network model of the classification method for steady-state visual evoked potentials provided in this embodiment of the invention;

[0024] Figure 8 This is a flowchart illustrating the implementation of a classification method for steady-state visual evoked potentials provided in another embodiment of the present invention.

[0025] Figure 9 This is a schematic diagram of the structure of the classification device for steady-state visual evoked potentials provided in an embodiment of the present invention;

[0026] Figure 10 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0027] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0028] To make the objectives, technical solutions, and advantages of the present invention clearer, specific embodiments will be described below in conjunction with the accompanying drawings.

[0029] Figure 1 This diagram illustrates an application scenario of the classification method for steady-state visual evoked potentials provided in this embodiment of the invention. Figure 1 As shown, embodiments of the present invention can be applied to Brain-Computer Interfaces (BCIs). A brain-computer interface, sometimes called a "brain port" or "brain-machine interface," is a direct connection pathway established between the human or animal brain (or a culture of brain cells) and an external device. In the case of a unidirectional brain-computer interface, the computer either receives commands from the brain or sends signals to the brain (e.g., video reconstruction), but cannot send and receive signals simultaneously. A bidirectional brain-computer interface, however, allows for bidirectional information exchange between the brain and external devices.

[0030] Specifically, this embodiment can be applied to smart environments. For example, it can perform brainprint recognition, identifying the specific identity information of a person or animal through their brainprint information. For instance, recognizing an animal's brainprint can confirm that the animal is a white kitten; or recognizing a person's brainprint can confirm their gender, name, age, and other relevant identity information. It can also be used for smart home control, where developers can create more intelligent and user-friendly applications. For example, brain-computer interface technology can be used to control and adjust the brightness, temperature, and sound of the home environment. Furthermore, it can be used for autonomous driving, using brainwave signals to control vehicles for autonomous driving without manual steering wheel control.

[0031] This embodiment can also be applied to gaming and entertainment. In traditional game human-computer interaction, users typically control a gamepad or mouse and keyboard using their limbs. However, many patients with neurological dysfunctions are unable to freely control their limbs to complete interactions due to blocked neural pathways, severely impacting their user experience. Game BCI, on the other hand, combines rich traditional game interaction methods with enhanced BCI control, interacting directly with the game simply by decoding brain signals.

[0032] This embodiment can also be applied to education, military, or medical fields, such as conducting education and training, developing military equipment, providing rehabilitation training for patients, or facilitating communication between patients.

[0033] See Figure 2 The flowchart illustrating the implementation of the classification method for steady-state visual evoked potentials provided in this embodiment of the invention is described in detail below:

[0034] Step 210: Obtain steady-state visual evoked potential signals.

[0035] In this embodiment, when a human body is subjected to a visual stimulus that flashes at a fixed frequency or changes in pattern, the electrical activity of the cerebral cortex is modulated, thereby generating a continuous response that is frequency-dependent (at the fundamental or harmonic frequency of the stimulus). This response has a periodic rhythm similar to that of the visual stimulus, which is called steady-state visual evoked potential (SSVEP). The SSVEP EEG signal is represented in the EEG (Electroencephalogram) signal by spectral peaks at the stimulus frequency or harmonics in the power spectrum. By analyzing the frequencies corresponding to the detected spectral peaks, the stimulus source of the subject's visual fixation can be detected, thereby enabling the identification of the subject's intention.

[0036] Considering that the visual delay typically occurs approximately 130-140 ms after the stimulus begins, preprocessing is necessary to remove interference from the visual delay before acquiring SSVEP EEG signals. Subsequently, since the SSVEP EEG signal response usually occurs in the occipital region of the brain, the occipital region and nearby pathways can be selected for analysis and research when selecting SSVEP EEG signals.

[0037] Step 220: Based on multiple bandpass filters of different frequency bands, extract short time-domain window data corresponding to each frequency band from the steady-state visual evoked potential signal.

[0038] In this embodiment, due to the characteristic that the harmonics of the SSVEP EEG signal contain important information, the SSVEP EEG signal can be intercepted within a preset time period when the subject gazes at different frequency stimuli. The intercepted SSVEP EEG signal can be filtered by three Butterworth bandpass filters of different frequency bands to filter out the harmonic information of the three different frequency bands, so that the sub-filters completely contain the harmonic information of all specific stimulus targets.

[0039] Subsequently, since the research focus of this embodiment is on SSVEP EEG signals under a short time-domain window, sliding windows of different sizes can be used to extract data from the filtered SSVEP EEG signals of three different frequency bands, obtaining short time-domain window data for three different frequency bands. This ensures that the time-domain EEG data within the same window have non-overlapping segments. Short time-domain window data refers to data shorter than 1 second, specifically 0.5s, 0.7s, 0.8s, etc., without limitation. Sliding windows of different sizes refer to using the same window for data extraction within the same batch of classifications, while using different sliding window sizes between different batches. This is because different types of signals, when classified in different time domains, will have varying classification results. Therefore, to make the classification results more accurate, it is necessary to compare the classification results under different sliding windows and select the sliding window corresponding to the classification result with the highest accuracy as the short time-domain window for that type of SSVEP EEG signal.

[0040] Step 230: Input short-time-domain window data corresponding to multiple frequency bands into a pre-trained hybrid network model to obtain the classification results of steady-state visual evoked potential signals; wherein, the hybrid network model includes a convolutional neural network and a bidirectional gated recursive unit. The convolutional neural network is used to obtain a fused feature map based on the short-time-domain window data corresponding to multiple frequency bands and compress the fused feature map into one-dimensional data; the bidirectional gated recursive unit is used to obtain the classification results of steady-state visual evoked potential signals based on one-dimensional data.

[0041] In this embodiment, the pre-trained hybrid network model includes a convolutional neural network and a bidirectional gated recursive unit. Multiple frequency bands can be three bands; that is, after the short-time-domain window data corresponding to the three frequency bands are input into the pre-trained hybrid network model, the convolutional neural network convolves the short-time-domain window data corresponding to the three frequency bands to obtain a fused feature map. Since the bidirectional gated recursive unit intelligently inputs one-dimensional data, the convolutional neural network also compresses the feature fusion map into one-dimensional data. The bidirectional gated recursive unit obtains the classification result of the steady-state visual evoked potential signal based on the input one-dimensional data.

[0042] In summary, this invention acquires steady-state visual evoked potential (VEP) signals; extracts short-time-domain window data corresponding to each frequency band from the VEP signals using bandpass filters of multiple different frequency bands; inputs the short-time-domain window data corresponding to multiple frequency bands into a pre-trained hybrid network model to obtain the classification result of the VEP signals; wherein, the hybrid network model includes a convolutional neural network (CNN) and a bidirectional gated recursive unit (BRU). The CNN is used to obtain a fused feature map based on the short-time-domain window data corresponding to multiple frequency bands and compress the fused feature map into one-dimensional data; the BRU is used to obtain the classification result of the VEP signals based on the one-dimensional data. This invention is applied to short-time-domain window VEP signals. By extracting and fusing features from the signals corresponding to the short-time-domain windows in the VEP signals, a fused feature map is obtained. Finally, the fused feature map is compressed into one-dimensional data, and the classification result is obtained through the BRU. The category to which the current VEP signal belongs can be determined based on the evaluation result. This invention achieves the technical effect of accurately classifying short-time-domain window potential signal data and improving classification performance.

[0043] Figure 3 This is a schematic diagram of a hybrid network model for the classification method of steady-state visual evoked potentials provided in an embodiment of the present invention. Figure 3 As shown:

[0044] The hybrid network model includes convolutional neural networks and bidirectional gated recurrent units. The convolutional neural network includes spatial convolution modules, temporal convolution modules, and feature fusion modules. Specific applications of each module can be found in the following examples.

[0045] In some embodiments, the convolutional neural network in step 230 may include a spatial convolution module, a temporal convolution module, and a feature fusion module;

[0046] The spatial convolution module is used to obtain the spatial feature map corresponding to each frequency band based on the short time-domain window data corresponding to each frequency band;

[0047] The temporal convolution module is used to obtain the spatiotemporal feature map corresponding to each frequency band based on the spatial feature map corresponding to each frequency band;

[0048] The feature fusion module is used to fuse spatiotemporal feature maps corresponding to multiple frequency bands to obtain a fused feature map, and then compress the fused feature map into one-dimensional data.

[0049] In this embodiment, the convolutional neural network includes a spatial convolution module, a temporal convolution module, and a feature fusion module. Specifically, when short-time-domain window data corresponding to multiple frequency bands are input into the pre-trained hybrid network model, the spatial convolution module convolves the short-time-domain window data corresponding to each frequency band to obtain a spatial feature map containing spatial features for each frequency band. The temporal convolution module obtains a spatiotemporal feature map containing both temporal and spatial features for each frequency band through convolution based on the spatial feature map corresponding to each frequency band. Finally, the fusion module fuses the spatiotemporal feature maps corresponding to multiple frequency bands to obtain a fused feature map. That is, it fuses the spatiotemporal feature maps corresponding to multiple frequency bands into a fused feature map of a single frequency band. Since the feature fusion map at this point is a multi-dimensional feature map, the fusion module also needs to compress the feature fusion map into one-dimensional data.

[0050] This embodiment directly uses the time-domain signal as the input to the convolutional neural network, and uses three filters to preserve the harmonic information in the time-domain signal to the maximum extent, reducing the amount of network parameter calculation required by related technical solutions to convert it to the frequency domain through Fourier transform.

[0051] In some embodiments, the steady-state visual evoked potential signal has multiple channels, and the short time-domain window data corresponding to each frequency band includes the short time-domain window data of each channel corresponding to that frequency band;

[0052] The spatial convolution module can be specifically used for:

[0053] For each frequency band, perform the following steps:

[0054] Extract the spatial feature map of the short time-domain window data of each channel corresponding to the frequency band, and use the attention expression mechanism to compress the spatial feature map of each channel into a single value corresponding to each channel.

[0055] Calculate the weight of each channel in the frequency band based on the individual value of each channel in the frequency band;

[0056] The spatial feature map of each channel in the frequency band is calculated based on the weight of each channel and the spatial feature map of each channel in the frequency band.

[0057] In this embodiment, when acquiring SSVEP EEG signals, signals from the occipital region and nearby channels can be filtered. Since multiple channels are acquired, the acquired SSVEP EEG signals also have multiple channels. The number of channels corresponding to each frequency band is the same as the number of channels in the SSVEP EEG signal. The short-time domain window data corresponding to each frequency band includes the short-time domain window data of each channel within that frequency band. That is, assuming the SSVEP EEG signal has eight channels, then each frequency band also includes eight channels, and the short-time domain window data corresponding to each frequency band is the short-time domain window data of the eight channels within that frequency band.

[0058] Figure 4 This is a structural diagram of the spatial convolution submodule of the classification method for steady-state visual evoked potentials provided in this embodiment of the invention, as shown below. Figure 4 As shown:

[0059] When data from each channel in any frequency band is input into the spatial convolution module, the first layer (FB) in the spatial convolution module defines various variables, i.e., the data corresponding to each channel in that frequency band. Then, convolution is used to extract the spatial feature map corresponding to each channel in that frequency band, which is the spatial feature map (H*W*C) containing Nc electrodes in the figure. The obtained spatial feature map corresponding to each channel in that frequency band is passed through two fully connected layers to obtain the weights of each channel in that frequency band. Finally, the spatial feature map corresponding to each channel in that frequency band is associated with its corresponding weights to obtain the spatial feature map corresponding to each channel in that frequency band.

[0060] Specifically, for each frequency band, the spatial convolution module can be used for:

[0061] Extract the spatial feature map of each channel in the frequency band; based on the spatial feature map of each channel in the frequency band, obtain the feature map corresponding to each channel in the frequency band; based on the feature map corresponding to each channel in the frequency band, perform convolution through the spatial convolution module to obtain the spatial domain feature corresponding to each channel in the frequency band.

[0062] The spatial domain features corresponding to each channel in the frequency band are compressed into a single value, and the weight of each value is calculated. Then, based on the spatial domain feature map corresponding to each channel in the frequency band and the weight of each channel, the spatial feature map corresponding to each channel in the frequency band is obtained.

[0063] In this embodiment, the spatial convolution module can be an adaptive spatial convolution module that includes an attention mechanism. The adaptive spatial convolution module may include a batch normalization (BN) unit, an activation function Elu, a random deactivation dropout layer, a global average pooling (GlobalAveragePooling2D) unit, a convolution operation Conv2D, and a spectral normalization operation.

[0064] Because different subjects exhibit individual differences in the spatial domain, a spatial feature map can be extracted using a convolutional kernel of size (C,1), where C is the number of channels in the input short temporal window data (in this embodiment, the number of channels can be 8). An attention mechanism is applied to the extracted spatial feature map, performing global average pooling on each channel to compress each feature map into a single value. The feature map can be represented as (1×1×C), representing the set of local features, thus giving each channel a global receptive field.

[0065] The expression for the attention mechanism is:

[0066]

[0067] F sq The Squeeze operation represents a compression operation, where is a k-dimensional feature embedding. The average of the k-dimensional numbers contained in the embedding is calculated to obtain a numerical value that represents the summary information of this feature. t represents a value from 1 to k. Then, the information in the embedding of the i-th feature is compressed into a single numerical value. For each feature V... i All of them were compressed into a single value Z. i .

[0068] After obtaining the individual values ​​for each channel in the frequency band, the weight of each channel in the frequency band is calculated using the following formula based on the individual values ​​of each channel:

[0069] S = F ex (Z,W)=σ(W2σ(W1Z))

[0070] F ex This represents the excitation activation operation. In the formula, σ represents the sigmoid activation function, and δ represents the ReLU activation function. Here, W1 is first multiplied by Z. Where C is the number of channels, r is the dimensionality reduction rate, and R represents a real number. This is a fully connected layer operation. Then it goes through a ReLU layer, keeping the output dimension unchanged; then it is multiplied by W2, which is also a fully connected layer process, resulting in an output dimension of 1*1*C; finally, it passes through a sigmoid function to obtain S, where S represents the weights of the C feature maps. In other words, the weights of each channel in this frequency band are passed through two fully connected layers, fusing the feature map information of each channel in the band. Then, the two features output by the fully connected layers are added element-wise and activated by a sigmoid function to obtain the channel attention map for each channel, i.e., the weights of each channel.

[0071] Finally, the spatial feature map calculation method for each channel in this frequency band is obtained through the adaptive spatial convolution module as follows:

[0072]

[0073] In the formula, M c The output spatial feature map is represented by M, where Conv2D represents the convolution operation, M represents the spatial feature map of each channel, and S represents the spatial feature map of the output channel. c This represents the weight of each channel after the attention mechanism. Represents element-wise multiplication.

[0074] The derived weights are multiplied channel-by-channel by the original input to obtain the spatial feature map corresponding to each channel in the frequency band. A random dropout layer is used during the calculation to prevent overfitting.

[0075] In this embodiment, the use of an attention mechanism enables each channel in each frequency band to have a global receptive field. Furthermore, when calculating the weights of each channel in each frequency band, two fully connected layers are used to fuse the EEG signals of all channels, reducing information loss caused by channel selection, so that the final spatial feature map is an enhanced spatial feature map.

[0076] Figure 5 This is a structural diagram of the temporal convolution submodule of the classification method for steady-state visual evoked potentials provided in this embodiment of the invention. Figure 5 As shown:

[0077] Each temporal convolutional submodule is a spatial feature map of each channel in any frequency band. For each frequency band, after the spatial feature map of each channel is input into the temporal convolutional submodule, it is convolved twice by a convolutional kernel of size (1, Ws*Fs) and a convolutional kernel of size (1, 5), and finally the spatiotemporal feature map of each channel in that frequency band is obtained.

[0078] Specifically, in some embodiments, the temporal convolution module is used for:

[0079] For each frequency band, perform the following steps:

[0080] Extract the temporal features from the spatial feature map of each channel corresponding to this frequency band;

[0081] Based on the temporal features in the spatial feature map of each channel corresponding to the frequency band, the spatiotemporal feature map of each channel corresponding to the frequency band is obtained.

[0082] In this embodiment, for each frequency band, the spatial feature map of each channel in that band includes both spatial and temporal information. Therefore, the temporal features of the spatial feature map of each channel in that frequency band are extracted using a temporal convolution module. Then, based on the temporal features of the spatial feature map of each channel corresponding to that frequency band, the spatiotemporal feature map of each channel corresponding to that frequency band is obtained. Here, the spatiotemporal feature map includes both temporal and spatial features.

[0083] In some embodiments, the temporal convolution module may include a first convolution unit and a second convolution unit;

[0084] The first convolutional unit is used to extract the first temporal information from the spatial feature map corresponding to each channel in each frequency band;

[0085] The second convolutional unit is used to extract the temporal features corresponding to each channel in each frequency band based on the first time information corresponding to each channel in each frequency band.

[0086] In this embodiment, the temporal convolution module may include a batch normalization (BN) unit, an activation function Elu, a random dropout layer, a convolution operation Conv2D, and a spectral normalization operation. It includes two convolution operations to extract temporal information: a first convolution unit and a second convolution unit. The first convolution unit may include a large convolution kernel of size (1, H), where H is the pre-set number of input sampling points. The second convolution unit may include a small convolution kernel of size (1, 5).

[0087] Because the same person's sensitivity to different paradigms varies, their reaction time to stimuli may be earlier or later. Extracting temporal information can reveal characteristics within a time range. Therefore, using CNNs to extract temporal information ensures that each frame of output contains sufficient temporal information. The first convolutional unit extracts initial temporal information, ensuring that each frame of output contains adequate temporal information. Then, the second convolutional unit continues to extract temporal information, deepening the network structure and extracting deeper layers of information, ultimately yielding a spatiotemporal feature map.

[0088] In this embodiment, a large convolutional kernel with a sampling point size is used in the temporal convolution module to extract temporal information, ensuring that each output frame contains sufficient temporal information. A small convolutional kernel of size (1,5) is also used to deepen the network structure before further local extraction of the extracted temporal information, strengthening the focus on temporal features. The two convolutional operations are implemented sequentially, resulting in a feature map that contains both global and local information.

[0089] In some embodiments, the feature fusion module is specifically used for:

[0090] For each channel, the spatiotemporal feature maps corresponding to each frequency band of that channel are stitched together to obtain the fused feature map corresponding to that channel;

[0091] The fused feature maps of each channel are compressed into one-dimensional data.

[0092] In this embodiment, the feature fusion module includes the ADD() operation, the batch normalization (BN) unit, the activation function Elu, and the convolution operation Conv2D.

[0093] Each frequency band can have eight channels, and each channel corresponds to three frequency bands. That is, there are eight different spatiotemporal feature maps on the same frequency band, and three different frequency bands on the same channel. That is, there are a total of 24 spatiotemporal feature maps on the three frequency bands. The feature fusion module can add the contents of the tensor elements of the spatiotemporal feature maps corresponding to the three different frequency bands on each channel through the ADD() operation and splice them into a fusion feature map. That is, after splicing, eight fusion feature maps corresponding to the eight channels are obtained. Among them, the eight channels correspond to eight dimensions. Since the input of the bidirectional gated recursive unit is one-dimensional data, the eight fusion feature maps can be convolved through a convolution kernel of size (1, shape[2]) using Conv2D to compress the multidimensional spatiotemporal information into one dimension and obtain one-dimensional data as the input of the bidirectional gated recursive unit.

[0094] In some embodiments, the training process of a hybrid network model includes the following steps:

[0095] Acquire sample data of steady-state visual evoked potential signals of multiple categories and generate a sample dataset of steady-state visual evoked potential signals;

[0096] Randomly select the first proportion of sample data in the sample dataset as the training set, and use the remaining sample data in the sample dataset as the test set;

[0097] Obtain the average value of the training results for a preset number of consecutive times, and use the average value as the training result of the hybrid network model.

[0098] In this embodiment, the network for training steady-state evoked potential signals within a short time window is trained as a hybrid network based on filter bank convolutional neural networks and bidirectional gated recursive units.

[0099] In this embodiment, Categorical Crossentropy in TensorFlow is defined as the loss function:

[0100]

[0101] The Adam optimization algorithm was used, and the batch size of the 12JFPM_SSVEP dataset was set to 32, where JFPM_SSVEP represents the dataset of phase-modulated SSVEP EEG signals. Since the 12_JFPM_SSVEP dataset does not explicitly divide the training and test sets, a 10-fold cross-validation method was employed. 90% of the SSVEP-BCI EEG signals were randomly selected as the training set, and 10% as the test set. The final experimental result was obtained by averaging the optimal accuracy from 10 consecutive trials; that is, the first ratio could be 90%, and the preset number of consecutive trials could be 10.

[0102] A pre-trained hybrid network model is used to classify steady-state visual evoked potential (VEVP) EEG signals. Performance comparisons with existing mainstream models within a 0.5-second time window reveal that the method provided in this embodiment achieves an average classification accuracy of 81.5% and an information transfer rate of 142.3 bpm on the 12_JFPM_SSVEP dataset. Compared to the existing filter bank convolutional neural network described in the background, the average classification accuracy and information transfer rate are improved by 4% and 12.8 bpm, respectively. This demonstrates that the present invention exhibits good classification performance and robustness for SSVEP EEG signals within a short time window.

[0103] In this embodiment, the average classification accuracy is calculated using the following formula, where ACC is the average classification accuracy, represents the number of classes correctly predicted by the model, and represents the number of classes incorrectly predicted by the model:

[0104]

[0105] Information transmission rate (ITR) is used to represent the speed at which brain-controlled external devices perform operations, i.e., the amount of information transmitted per unit time. It is influenced by three factors: time, classification accuracy, and the number of stimulus targets. The formula for calculating ITR is as follows: ITR represents the amount of information transmitted in each output; ITR represents the number of stimulus targets; ITR represents the classification accuracy; and ITR represents the single target selection time, which includes both the target selection time and the pause time between adjacent selections.

[0106] ITR = B(60 / T)

[0107] T = t s +t b

[0108]

[0109] Figure 6 This is a structural diagram of the bidirectional gated recursive unit submodule of the classification method for steady-state visual evoked potentials provided in this embodiment of the invention. Figure 6 As shown:

[0110] The bidirectional gated recurrent unit (BRU) is used to construct a bidirectional RNN-type neural network, including parameters `layer`, `merge_mode`, `weights`, and `backward_layer`. The `layer` parameter is used to select the model; here, GRU is chosen instead of LSTM. This is because LSTM can address the vanishing and exploding gradient problems caused by long-term dependencies in recurrent neural networks, but it has three different gates, resulting in more parameters and making training more difficult. GRU contains only two gate structures, and with all hyperparameters tuned, the performance of GRU and LSTM is comparable. However, GRU is simpler, requires fewer training samples, and is easier to implement. The BRU is added to learn the spatiotemporal dependencies of the data flow. When administering SSVEP stimulation to subjects, attention may become sluggish over time. By compressing the multidimensional spatiotemporal feature maps corresponding to each channel into one dimension, the BRU allows for sequential forward and backward input sequences to learn the spatiotemporal dependencies.

[0111] Figure 7 This is a structural diagram of the hybrid network model of the classification method for steady-state visual evoked potentials provided in this embodiment of the invention. For example... Figure 7 As shown:

[0112] SSVEP EEG signals can be filtered through filters of different frequency bands to obtain three different frequency bands of SSVEP, each corresponding to eight channels. The SSVEP EEG signals are then input into a hybrid network model according to the three frequency bands. After variable definition through each FB layer, channel convolution is used to obtain the spatial feature map of each channel in each frequency band. Then, global average pooling is performed on the spatial feature maps of each channel in each frequency band, followed by two temporal convolutions to obtain the spatiotemporal feature maps of each channel in each frequency band. Next, feature fusion is performed to merge the three frequency bands into one. Since this new frequency band includes spatiotemporal feature maps of eight channels, convolution is used to compress the eight channels into one-dimensional data, which is then input into a bidirectional gated recurrent unit (BRU). The data output from the BRU is then processed through a fully connected layer to obtain the classification result of the SSVEP EEG signal.

[0113] Figure 8 This is a flowchart illustrating the implementation of a classification method for steady-state visual evoked potentials according to another embodiment of the present invention. The specific process structure and module components involved can be found in [reference needed]. Figure 7 .

[0114] Example 1: This example can be used to classify steady-state visual evoked potential EEG signals within a short time window of less than 1 second in the 12_JFPM_SSVEP dataset. This example uses a 0.5-second time window as an example:

[0115] Raw EEG signals were acquired: In this embodiment, the 12_JFPM_SSVEP dataset recorded a 12-classification task performed by 10 subjects across 8 lead electrode channels. Stimuli were arranged in a 4×3 matrix, and a frequency-phase joint encoding method was used, causing two adjacent targets to flash at different frequencies (f0 = 9.25 Hz, Δf = 0.5 Hz) and phases (φ0 = 0, Δφ = 0.5π). The sampling frequency was 1024 Hz, and the paradigm was designed as 0.5 s cue, 4 s flash, and 0.5 s rest. Each subject's experiment consisted of 15 blocks, each block containing 12 trails, randomly corresponding to the 12 stimuli. Considering visual delay, the first 135 ms of data were removed.

[0116] The raw EEG signals underwent data preprocessing: Considering that visual delay typically occurs approximately 130-140 ms after stimulus onset, the interference of visual delay on the EEG signals was first removed during preprocessing. Secondly, since the SSVEP EEG signal response usually occurs in the occipital region of the brain, the occipital region and nearby channels were selected for study. Thirdly, taking advantage of the important information contained in the harmonics of SSVEP EEG signals, the truncated EEG signals were filtered through three Butterworth bandpass filters of different frequency bands to extract different harmonic information, ensuring that the sub-filters fully contained the harmonic information of all specific frequencies of the stimulus target. The parameters were set as follows: the bandpass ranges of the three sub-filters in the 12_JFPM_SSVEP public dataset were 6-16 Hz, 16-32 Hz, and 32-64 Hz. Finally, the filtered data was truncated using sliding windows of different sizes to ensure that the time-domain EEG data within the same time window had non-overlapping segments. The input after data preprocessing was (1440, 10, 128, 1). The number of input channels is 10 because, in this embodiment, the preprocessed data of the first two EEG channels is reused to achieve full convolution.

[0117] Reference Figure 7The three preprocessed data sets are input in parallel into the adaptive spatial convolution module to obtain the enhanced spatial feature map for each channel. In this embodiment, because different subjects exhibit individual differences in the spatial domain, a convolutional kernel of size (10,1) is used to extract the spatial feature map, where 10 represents the number of channels in the input sequence data and out_channel is set to 18. An attention mechanism is applied to the extracted spatial feature map, performing global average pooling on each channel to compress each spatial feature map. The output after average pooling is (None,18), and the feature representation is a (1×1×18) spatial feature map, representing the set of various local features, thus giving it a global receptive field. Then, two fully connected layers fuse the spatial feature maps of each channel. The first fully connected layer uses the ReLU activation function and outputs 2 nodes for compression to remove redundant information and reduce computation. The second fully connected layer then uses the sigmoid activation function and outputs 18 nodes to restore the size and fully capture channel dependencies. Finally, this channel attention result map is multiplied with the original image, and a random dropout layer is used to prevent overfitting, resulting in an enhanced spatial feature map. In this embodiment, the dropout parameter can be set to 0.4.

[0118] The feature maps corresponding to the three frequency bands are input in parallel into the temporal convolution module to obtain a spatiotemporal feature map containing global and local temporal information for each sampling point, i.e., a spatiotemporal feature map. In this embodiment, a CNN can be used to extract temporal information so that each frame output contains sufficient temporal information. Therefore, a large convolutional kernel of size (1, 128) is used to extract temporal information, where 128 is the number of sampling points of the input sequence data, which ensures that each frame output contains sufficient temporal information. The stride is 5, and padding="same" is used to pad the edges of the input image matrix with 0. This yields both global and local feature maps, resulting in 26 feature maps that contain both global and local information, with an output of (None, 1, 26, 18). Then, a small convolutional kernel of size (1, 5) is used to continue extracting temporal information, with a stride of 1 and padding="valid" to deepen the network structure and extract deeper information. Finally, the spatiotemporal feature maps of each channel in each frequency band are obtained, with an output of (None, 1, 22, 18).

[0119] The spatiotemporal feature maps corresponding to the three frequency bands are fused: Three parallel convolution operations are performed using ADD(), summing the contents of the tensors of the spatiotemporal feature maps corresponding to the three frequency bands. This yields a fused map containing both spatial and temporal features.

[0120] The fused multidimensional spatiotemporal information is compressed to one dimension and input into a bidirectional gated recursive unit to learn the spatiotemporal dependency information of the data stream. In this embodiment, the feature fusion map is then convolved using a (1, 22) kernel using Conv2D to compress the multidimensional spatiotemporal information to one dimension, with out_channel set to 72. The output at this point is (None, 1, 1, 72), which is then reshaped to (None, 1, 72), resulting in one-dimensional data that serves as the input to the bidirectional gated recursive unit.

[0121] The number of neurons in the bidirectional gated recursive unit can be set to 72; return_sequences should be set to True; the activation function should be tanh; and merge_mode should be the default value. The output of the bidirectional gated recursive unit is (None, 1, 144).

[0122] In this embodiment, the multidimensional input is first flattened into one dimension (None, 144) using a flattening layer to transition from the convolutional layer to the fully connected layer; then, a dropout layer is used to prevent overfitting. Finally, the input is fed into the fully connected layer, where the softmax activation function is used to classify the SSVEP EEG signals.

[0123]

[0124] Where xi is the output value of the i-th node, and N is the number of output nodes, i.e., the number of categories. The Softmax function can be used to convert the multi-class output values ​​into a probability distribution ranging from [0,1] to 1. The output is (None, 12).

[0125] Example 2: This embodiment of the invention can also be applied to the classification of steady-state visual evoked potential EEG signals within a short time window of less than 1 second in the Benchmark dataset. This example uses a 0.5-second time window as an example.

[0126] In this embodiment, the Benchmark public dataset recorded 40 classification tasks performed by 35 participants across 64 lead electrode channels. Stimulus targets were arranged in a 5×8 matrix, and a frequency-phase joint encoding method was used, causing two adjacent targets to flash at different frequencies (f0 = 8.0 Hz, Δf = 0.2 Hz) and phases (φ0 = 0, Δφ = 0.5π). The sampling frequency was 1000 Hz, and the paradigm was designed as a 0.5 s cue, 5 s flash, and 0.5 s rest. Each participant's experiment consisted of 6 blocks, each containing 40 trails, randomly corresponding to 40 stimulus targets. O1, Oz, O2, PO3, POz, PO4, Pz, PO5, and PO6 were selected as input channels; considering visual delay, the first 140 ms of data were removed.

[0127] Considering that the visual delay typically occurs approximately 130-140 ms after the stimulus begins, the interference of the visual delay on the EEG signal is removed first during preprocessing. Secondly, since the SSVEP response usually occurs in the occipital region of the brain, channels in and around the occipital region are selected for study. Thirdly, taking advantage of the important information contained in SSVEP harmonics, the truncated EEG signal is filtered through three Butterworth bandpass filters of different frequency bands to extract different harmonic information, ensuring that the sub-filters fully contain the harmonic information of all specific frequencies of the stimulus target. The parameters are set as follows: the bandpass ranges of the three sub-filters in the publicly available benchmark dataset are 6-18 Hz, 14-36 Hz, and 28-64 Hz. Finally, the filtered data is truncated using sliding windows of different sizes to ensure that the time-domain EEG data within the same time window have non-overlapping segments. The input after data preprocessing is (2400, 11, 125, 1). The number of input channels is 11 because, in this embodiment, the preprocessed data of the first two EEG channels is reused to achieve full convolution.

[0128] In this embodiment, because different subjects exhibit individual differences in the spatial domain, a convolutional kernel of size (11,1) is used to extract spatial feature maps, where 11 represents the number of channels in the input sequence data, and out_channel is set to 20. An attention mechanism is applied to the extracted spatial feature maps, performing global average pooling on each channel to compress each spatial feature map. The output after average pooling is (None,20), and the feature map is represented as a (1×1×20) spatial feature map, which can represent the sets of various localities, giving it a global receptive field. Next, two fully connected layers fuse the feature map information from each channel. The first fully connected layer uses the ReLU activation function with 2 output nodes for compression to remove redundant information and reduce computation. The second fully connected layer then uses the sigmoid activation function with 20 output nodes to restore the size and fully capture channel dependencies. Finally, this channel attention result map is multiplied by the original image, and a random dropout layer is used to prevent overfitting, resulting in an enhanced spatial feature map. In this embodiment, the dropout parameter is set to 0.4.

[0129] Because the same person has different sensitivities to different paradigms, their reaction time to stimuli may be earlier or later. Extracting temporal information can reveal characteristics within a time range. In this embodiment, a CNN is used to extract temporal information, ensuring that each frame of output contains sufficient temporal information. Therefore, a large convolutional kernel of size (1, 125) is used to extract temporal information, where 125 is the number of sampling points in the input sequence data. This ensures that each frame of output contains sufficient temporal information. The stride is 5, and padding is set to "same". Zeros are padded to the edges of the input image matrix, thus obtaining both global and local feature maps. Finally, 25 feature maps containing both global and local information are obtained, with the output being (None, 1, 25, 20). Then, a small convolutional kernel of size (1, 5) is used to further extract temporal information, with a stride of 1 and padding set to "valid". This deepens the network structure and extracts deeper information, finally yielding a spatiotemporal feature map, with the output being (None, 1, 21, 20).

[0130] The feature fusion module uses ADD() to perform three parallel convolution operations to sum the contents of the tensor elements in the spatiotemporal feature map corresponding to each channel, resulting in the feature fusion map corresponding to each channel.

[0131] In this embodiment, the feature fusion map corresponding to each channel is then convolved using a Conv2D operation with a kernel of size (1, 21) to compress the multidimensional spatiotemporal information into one dimension, with out_channel set to 120. The output at this point is (None, 1, 1, 120), which is then transformed into (None, 1, 120) using reshape, resulting in one-dimensional data that serves as the input to the bidirectional gated recursive unit.

[0132] The output, obtained through a bidirectional gated recursive unit, is (None, 1, 240). The fully connected module within the bidirectional gated recursive unit includes a flattening layer, a dropout operation, and a dense fully connected layer. In this embodiment, a flattening layer is first used to simplify the multidimensional input to (None, 240), facilitating the transition from the convolutional layer to the fully connected layer. A dropout layer is then used to prevent overfitting. The input is then fed into the fully connected layer, where the softmax activation function is used for SSVEP EEG signal classification. The softmax function converts the multi-class output values ​​into a probability distribution ranging from [0, 1] to 1, resulting in an output of (None, 40).

[0133] In this embodiment, the hybrid network model includes a spectral normalization operation. The central idea of ​​spectral normalization in convolution is to constrain the updatable weight matrix in the neural network to the Lipschitz condition. Intuitively, the Lipschitz condition limits the severity of changes in the function's gradient. In one-dimensional space, it's easy to see that y = sin(x) is 1-Lipschitz, with a maximum slope of 1. To ensure that the weight matrix W satisfies K-Lipschitz continuity, the minimum value of K will be... Here, λ1 represents the maximum singular value of the matrix WTW. Therefore, to ensure that the weight matrix W satisfies 1-Lipschitz continuity and to make the network training process more stable, all elements of the weight matrix W need to be transformed:

[0134]

[0135] In this embodiment, a hybrid network model is used to classify steady-state visual evoked potential (SSVEP) EEG signals. Performance comparisons with existing mainstream models within a 0.5s time window reveal that the proposed method achieves an average classification accuracy of 54.7% and an information transfer rate of 118.95 bpm on the Benchmark dataset. Compared to the existing filter bank convolutional neural network described in the background of the invention, the average classification accuracy and information transfer rate are improved by 2% and 5.82 bpm, respectively. This demonstrates that the present invention exhibits good classification performance and robustness for SSVEP EEG signals within a short time window.

[0136] The technical effects achieved by using the above-described technical means in the embodiments of the present invention are as follows:

[0137] 1. This invention combines a filter bank convolutional neural network and a bidirectional gated recursive unit for the classification of steady-state visual evoked potentials, providing a solution to the problem of poor classification performance caused by insufficient latent feature extraction within a short time window. This method directly uses the time-domain signal as network input, reducing the computational cost of converting it to the frequency domain via Fourier transform in existing technologies. By extracting local information through a convolutional neural network and using a bidirectional gated unit to extract global information, it solves the problem of subject inattention caused by time migration.

[0138] 2. This invention employs an improved, highly efficient attention module, which is incorporated into the adaptive spatial convolution module. This addresses the issue of differences in the spatial characteristics of EEG signals from different subjects at different times for the same stimulus target; it also resolves the problem that different subjects produce different stimuli of varying locations and amplitudes for different paradigms. Furthermore, it fuses EEG signals from all channels, reducing information loss during channel selection.

[0139] 3. In this invention, a large convolutional kernel with a sampling point size is used in the temporal convolution module to extract temporal information, ensuring that each output frame contains sufficient temporal information. A small convolutional kernel of size (1,5) is also used to deepen the network structure before further local extraction of the extracted temporal information. These two convolutional operations are implemented sequentially, resulting in a feature map that contains both global and local information.

[0140] 4. This invention uses a bidirectional gated recursive unit to learn the spatiotemporal dependencies of data. This is because during SSVEP EEG stimulation of subjects, attention deficit issues can arise over time, reaction times vary between subjects, and the stimulation intensity for each channel also differs. By compressing multidimensional spatiotemporal information into one dimension, the bidirectional gated recursive unit allows for sequential input of forward and backward sequences, learning the spatiotemporal dependencies.

[0141] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0142] The following are device embodiments of the present invention. For details not described in detail, please refer to the corresponding method embodiments described above.

[0143] Figure 9 A schematic diagram of the structure of the classification device for steady-state visual evoked potentials provided in an embodiment of the present invention is shown. For ease of explanation, only the parts related to the embodiment of the present invention are shown, and are described in detail below:

[0144] like Figure 9 As shown, the classification device 8 for steady-state visual evoked potentials includes:

[0145] Acquisition module 91 is used to acquire steady-state visual evoked potential signals;

[0146] Extraction module 92 is used to extract short time-domain window data corresponding to each frequency band from the steady-state visual evoked potential signal based on multiple bandpass filters of different frequency bands;

[0147] The computation module 93 is used to input short-time-domain window data corresponding to multiple frequency bands into a pre-trained hybrid network model to obtain the classification results of steady-state visual evoked potential signals. The hybrid network model includes a convolutional neural network and a bidirectional gated recursive unit. The convolutional neural network is used to obtain a fused feature map based on the short-time-domain window data corresponding to multiple frequency bands and compress the fused feature map into one-dimensional data. The bidirectional gated recursive unit is used to obtain the classification results of steady-state visual evoked potential signals based on one-dimensional data.

[0148] In some embodiments, the convolutional neural network in the computing module 93 includes a spatial convolution module, a temporal convolution module, and a feature fusion module;

[0149] The spatial convolution module is used to obtain the spatial feature map corresponding to each frequency band based on the short time-domain window data corresponding to each frequency band;

[0150] The temporal convolution module is used to obtain the spatiotemporal feature map corresponding to each frequency band based on the spatial feature map corresponding to each frequency band;

[0151] The feature fusion module is used to fuse spatiotemporal feature maps corresponding to multiple frequency bands to obtain a fused feature map, and then compress the fused feature map into one-dimensional data.

[0152] In some embodiments, the steady-state visual evoked potential signal has multiple channels, and the short time-domain window data corresponding to each frequency band includes the short time-domain window data of each channel corresponding to that frequency band;

[0153] The spatial convolution module in computation module 93 is specifically used for:

[0154] For each frequency band, perform the following steps:

[0155] Extract the spatial feature map of the short time-domain window data of each channel corresponding to the frequency band, and use the attention expression mechanism to compress the spatial feature map of each channel into a single value corresponding to each channel.

[0156] Calculate the weight of each channel in the frequency band based on the individual value of each channel in the frequency band;

[0157] The spatial feature map of each channel in the frequency band is calculated based on the weight of each channel and the spatial feature map of each channel in the frequency band.

[0158] In some embodiments, the temporal convolution module in the computation module 93 is specifically used for:

[0159] For each frequency band, perform the following steps:

[0160] Extract the temporal features from the spatial feature map of each channel corresponding to this frequency band;

[0161] Based on the temporal features in the spatial feature map of each channel corresponding to the frequency band, the spatiotemporal feature map of each channel corresponding to the frequency band is obtained.

[0162] In some embodiments, the temporal convolution module in the computation module 93 includes a first convolution unit and a second convolution unit;

[0163] The first convolutional unit is used to extract the first temporal information from the spatial feature map corresponding to each frequency band;

[0164] The second convolutional unit is used to extract the temporal features corresponding to each frequency band based on the first time information corresponding to each frequency band.

[0165] In some embodiments, the feature fusion module in the calculation module 93 is specifically used for:

[0166] For each channel, the spatiotemporal feature maps corresponding to each frequency band of that channel are stitched together to obtain the fused feature map of that channel;

[0167] The fused feature map of each channel is compressed into one-dimensional data.

[0168] In some embodiments, the training process of the hybrid network model in the computing module 93 includes the following steps:

[0169] Acquire sample data of steady-state visual evoked potential signals of multiple categories and generate a sample dataset of steady-state visual evoked potential signals;

[0170] Randomly select the first proportion of sample data in the sample dataset as the training set, and use the remaining sample data in the sample dataset as the test set;

[0171] Obtain the average value of the training results for a preset number of consecutive times, and use the average value as the training result of the hybrid network model.

[0172] Figure 10 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. For example... Figure 10 As shown, the electronic device 10 of this embodiment includes: a processor 100, a memory 101, and a computer program 102 stored in the memory 101 and executable on the processor 100. When the processor 100 executes the computer program 102, it implements the steps in the above-described embodiments of the classification methods for steady-state visual evoked potentials, for example... Figure 2 Steps 210 to 230 are shown. Alternatively, when the processor 100 executes the computer program 102, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 9 The functions of modules / units 91 to 93 shown.

[0173] For example, the computer program 102 can be divided into one or more modules / units, which are stored in the memory 101 and executed by the processor 100 to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program 102 in the electronic device 10. For example, the computer program 102 can be divided into... Figure 9 Modules / units 91 to 93 are shown.

[0174] The electronic device 10 can be a desktop computer, laptop, handheld computer, cloud server, or other computing device. The electronic device 10 may include, but is not limited to, a processor 100 and a memory 101. Those skilled in the art will understand that... Figure 10 This is merely an example of electronic device 10 and does not constitute a limitation on electronic device 10. It may include more or fewer components than shown, or combine certain components, or different components. For example, the electronic device may also include input / output devices, network access devices, buses, etc.

[0175] The processor 100 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0176] The memory 101 can be an internal storage unit of the electronic device 10, such as a hard disk or memory of the electronic device 10. The memory 101 can also be an external storage device of the electronic device 10, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 10. Furthermore, the memory 101 can include both internal and external storage units of the electronic device 10. The memory 101 is used to store the computer program and other programs and data required by the electronic device. The memory 101 can also be used to temporarily store data that has been output or will be output.

[0177] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0178] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0179] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0180] In the embodiments provided by this invention, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0181] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0182] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0183] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the above embodiments of the classification methods for steady-state visual evoked potentials. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0184] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method of classifying steady-state visual evoked potentials, characterized by, include: Acquire multi-channel steady-state visual evoked potential signals; Based on multiple bandpass filters of different frequency bands, multi-channel short time-domain window data corresponding to each frequency band is extracted from the steady-state visual evoked potential signals corresponding to multiple channels; wherein, the short time-domain window data is time-domain data of less than 1 second; the short time-domain window data corresponding to each frequency band includes the short time-domain window data of each channel corresponding to that frequency band; The multi-channel short-time-domain window data corresponding to the multiple frequency bands are all input into a pre-trained hybrid network model to obtain the classification result of the steady-state visual evoked potential signal. The hybrid network model includes a convolutional neural network and a bidirectional gated recursive unit. The convolutional neural network is used to obtain a fused feature map based on the multi-channel short-time-domain window data corresponding to the multiple frequency bands, and compress the fused feature map into one-dimensional data. The bidirectional gated recursive unit is used to obtain output data based on the one-dimensional data, and the fully connected layer obtains the classification result of the steady-state visual evoked potential signal based on the output data of the bidirectional gated recursive unit. The convolutional neural network includes a spatial convolution module, a temporal convolution module, and a feature fusion module. The spatial convolution module is used to obtain the multi-channel spatial feature map corresponding to each frequency band based on the multi-channel short time-domain window data corresponding to each frequency band. The temporal convolution module is used to obtain a multi-channel spatiotemporal feature map corresponding to each frequency band based on the multi-channel spatial feature map corresponding to each frequency band. The feature fusion module is used to fuse the multi-channel spatiotemporal feature maps corresponding to the multiple frequency bands to obtain a fused feature map, and compress the fused feature map into one-dimensional data.

2. The method of claim 1, wherein, The spatial convolution module is specifically used for: For each frequency band, perform the following steps: Extract the spatial feature map of the short time-domain window data of each channel corresponding to the frequency band, and use the attention expression mechanism to compress the spatial feature map of each channel into a single value corresponding to each channel. Calculate the weight of each channel in the frequency band based on the individual value of each channel in the frequency band; The spatial feature map of each channel in the frequency band is calculated based on the weight of each channel and the spatial feature map of each channel in the frequency band.

3. The method of claim 2, wherein, The temporal convolution module is specifically used for: For each frequency band, perform the following steps: Extract the temporal features from the spatial feature map of each channel corresponding to this frequency band; Based on the temporal features in the spatial feature map of each channel corresponding to the frequency band, the spatiotemporal feature map of each channel corresponding to the frequency band is obtained.

4. The method of claim 3, wherein, The temporal convolution module includes a first convolution unit and a second convolution unit; The first convolutional unit is used to extract the first temporal information from the spatial feature map corresponding to each frequency band; The second convolutional unit is used to extract the temporal features corresponding to each frequency band based on the first time information corresponding to each frequency band.

5. The method according to claim 3, characterized in that, The feature fusion module is specifically used for: For each channel, the spatiotemporal feature maps corresponding to each frequency band of that channel are stitched together to obtain the fused feature map of that channel; The fused feature map of each channel is compressed into one-dimensional data.

6. The method of claim 1, wherein, The training process of the hybrid network model includes the following steps: Acquire sample data of steady-state visual evoked potential signals of multiple categories and generate a sample dataset of steady-state visual evoked potential signals; A first proportion of sample data is randomly selected from the sample dataset as the training set, and the remaining sample data in the sample dataset is used as the test set. Obtain the average value of the training results for a preset number of consecutive times, and use the average value as the training result of the hybrid network model.

7. A device for classifying steady-state visual evoked potentials, characterized by include: The acquisition module is used to acquire multi-channel steady-state visual evoked potential signals; An extraction module is used to extract multi-channel short-time-domain window data corresponding to each frequency band from the steady-state visual evoked potential signals corresponding to multiple channels based on bandpass filters of multiple different frequency bands; wherein, the short-time-domain window data is time-domain data of less than 1 second; the short-time-domain window data corresponding to each frequency band includes the short-time-domain window data of each channel corresponding to that frequency band; The computation module is used to input multi-channel short-time-domain window data corresponding to the multiple frequency bands into a pre-trained hybrid network model to obtain the classification result of the steady-state visual evoked potential signal. The hybrid network model includes a convolutional neural network and a bidirectional gated recursive unit. The convolutional neural network is used to obtain a fused feature map based on the multi-channel short-time-domain window data corresponding to the multiple frequency bands and compress the fused feature map into one-dimensional data. The bidirectional gated recursive unit is used to obtain output data based on the one-dimensional data, and the fully connected layer obtains the classification result of the steady-state visual evoked potential signal based on the output data of the bidirectional gated recursive unit. The convolutional neural network includes a spatial convolution module, a temporal convolution module, and a feature fusion module. The spatial convolution module is used to obtain the multi-channel spatial feature map corresponding to each frequency band based on the multi-channel short time-domain window data corresponding to each frequency band. The temporal convolution module is used to obtain a multi-channel spatiotemporal feature map corresponding to each frequency band based on the multi-channel spatial feature map corresponding to each frequency band. The feature fusion module is used to fuse the multi-channel spatiotemporal feature maps corresponding to the multiple frequency bands to obtain a fused feature map, and compress the fused feature map into one-dimensional data.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 6 above.

9. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 6 above.

Images