Electroencephalography classification method, system, device and medium combining language and visual imagination

Abstract

The application discloses a brain electrical classification method and system combining language and visual imagination, a device and a medium. The method inputs time-domain electroencephalogram signals combining language imagination and visual imagination to a first branch, and performs feature enhancement on the first branch according to a channel attention sub-module to obtain an output result of the first branch; inputs transformed frequency-domain electroencephalogram signals to a second branch, and performs feature enhancement on the second branch according to the channel attention sub-module to obtain an output result of the second branch; inputs the output result of the first branch and the output result of the second branch to a cross-attention sub-module to obtain an output result of the cross-attention sub-module; and trains a double-branch electroencephalogram classification model according to the output result of the cross-attention sub-module, the output result of the first branch and the output result of the second branch, so as to classify target electroencephalogram signals by using the trained double-branch electroencephalogram classification model. The application can improve the classification accuracy of electroencephalogram.

Description

Technical Field

[0001] This application relates to the field of electroencephalogram (EEG) signal classification technology, and in particular to an EEG classification method, system, device, and medium that combines language and visual imagination. Background Technology

[0002] Brain-computer interface (BCI) technology allows direct communication between the brain and a machine. Electroencephalography (EEG) is widely used to monitor brain activity due to its low cost, convenience, and high temporal resolution. Commonly used methods in EEG-based BCI research include P300 evoked potentials, steady-state visual evoked potentials (SSVEP), and motor imagery. Each method has its advantages. For example, P300 and SSVEP provide explicit signals but require external stimulation; while motor imagery does not require external stimulation, but training time is longer and the number of repetitions is limited. When performing certain tasks, subjects need to map imagined movements to required control commands according to pre-agreed mapping relationships, resulting in a significant cognitive load.

[0003] Therefore, research on BCI related to cognitive function has become a new trend; however, the classification accuracy of related EEG decoding-based classification techniques is still relatively low. Summary of the Invention

[0004] This application aims to propose a method, system, device, and medium for classifying electroencephalograms (EEGs) by combining language and visual imagination, which can improve the classification accuracy of EEGs.

[0005] In a first aspect, embodiments of this application provide an EEG classification method that combines language and visual imagination, the method comprising:

[0006] Obtain EEG signals combining linguistic and visual imagination;

[0007] A two-branch EEG classification model is constructed, comprising a first branch, a second branch, and an attention module; wherein the attention module includes a channel attention submodule and a cross attention submodule.

[0008] The time-domain EEG signal combining the language imagination and visual imagination is input into the first branch, and the first branch is enhanced with features according to the channel attention submodule to obtain the output result of the first branch;

[0009] The time-domain EEG signal combining language imagination and visual imagination is subjected to Fast Fourier Transform to obtain the transformed frequency-domain EEG signal.

[0010] The transformed frequency domain EEG signal is input into the second branch, and feature enhancement is performed on the second branch according to the channel attention submodule to obtain the output result of the second branch;

[0011] The output results of the first branch and the second branch are input into the cross-attention submodule to obtain the output result of the cross-attention submodule;

[0012] Based on the output results of the cross-attention submodule, the first branch, and the second branch, a two-branch EEG classification model is trained so that the trained two-branch EEG classification model can be used to classify the target EEG signal.

[0013] Compared with the prior art, the first aspect of this application has the following beneficial effects:

[0014] This method inputs the time-domain EEG signal combining language and visual imagination into the first branch, and enhances the features of the first branch using the channel attention submodule. Then, the frequency-domain EEG signal after Fast Fourier Transform is input into the second branch, and the second branch is also enhanced using the channel attention submodule. By learning the EEG signal combining language and visual imagination through the two branches, and using the channel attention submodule for feature enhancement, the accuracy of the model classification can be improved. The output results of the first branch and the second branch are input into the cross-attention submodule to obtain the output result of the cross-attention submodule. Based on the output results of the cross-attention submodule, the first branch, and the second branch, a two-branch EEG classification model is trained. This trained two-branch EEG classification model can then be used to classify the target EEG signal. By capturing the relationship between the output results of the first and second branches through the cross-attention submodule, and combining the output results of the cross-attention submodule, the first branch, and the second branch together to train the two-branch EEG classification model, the classification accuracy of EEG can be further improved.

[0015] In some implementations, the first branch includes a first convolutional module, a first depthwise convolutional module, a first depthwise separable convolutional module, and a GRU module. The time-domain EEG signal combining the language imagination and visual imagination is input to the first branch, and feature enhancement is performed on the first branch according to the channel attention submodule to obtain the output of the first branch, including:

[0016] The time-domain EEG signal combining the language imagination and visual imagination is processed by the first convolution module to extract the time feature map of the EEG signal;

[0017] The temporal feature map is enhanced by the channel attention submodule to obtain the enhanced first feature map.

[0018] The enhanced first feature map is input into the first depthwise convolutional module to obtain the output result of the first depthwise convolutional module.

[0019] The output of the first depthwise convolution module is input into the first depthwise separable convolution module to obtain the output of the first depthwise separable convolution module;

[0020] The output of the first depthwise separable convolution module is input into the GRU module to obtain the output of the first branch.

[0021] In some implementations, the second branch includes a second convolutional module, a second depthwise convolutional module, a second depthwise separable convolutional module, a Flatten layer, and a Linear layer. The transformed frequency-domain EEG signal is input to the second branch, and feature enhancement is performed on the second branch according to the channel attention submodule to obtain the output of the second branch, including:

[0022] The frequency feature map of the transformed frequency domain EEG signal is extracted by the second convolution module.

[0023] The frequency feature map is enhanced by the channel attention submodule to obtain an enhanced second feature map.

[0024] The enhanced second feature map is input into the second deep convolution module to obtain the output result of the second deep convolution module;

[0025] The output of the second depthwise convolution module is input into the second depthwise separable convolution module to obtain the output of the second depthwise separable convolution module;

[0026] The output of the second depthwise separable convolutional module is processed through the Flatten layer and the Linear layer to obtain the output of the second branch.

[0027] In some implementations, the output results of the first branch and the second branch are input to the cross-attention submodule to obtain the output result of the cross-attention submodule, including:

[0028] Q f =X f W q ,K t =X t W k V t =X t W v

[0029]

[0030] Among them, Q f Indicates a query, X f W represents the output of the second branch. q Let K represent the second projection matrix. t Indicates key, X t W represents the output of the first branch. k V represents the first projection matrix. t Represents the value, W v Represents the third projection matrix, CrossAttention(Q) f ,K t V t ) represents the output of the cross-attention submodule, Softmax(·) represents the activation function, and d represents the dimension of the time feature map or frequency feature map.

[0031] In some implementations, training a two-branch EEG classification model based on the output of the cross-attention submodule, the output of the first branch, and the output of the second branch includes:

[0032] The output of the cross-attention submodule is passed through a Linear layer and a Softmax activation function to obtain the first prediction probability result;

[0033] The output of the first branch is passed through a Linear layer and a Softmax activation function to obtain the second prediction probability result;

[0034] The output of the second branch is passed through a Linear layer and a Softmax activation function to obtain the third prediction probability result;

[0035] The total loss function is determined based on the first prediction probability result, the second prediction probability result, and the third prediction probability result;

[0036] A two-branch EEG classification model is trained based on the total loss function.

[0037] In some implementations, determining the total loss function based on the first prediction probability result, the second prediction probability result, and the third prediction probability result includes:

[0038] Based on the first predicted probability result, the first loss value is calculated using the cross-entropy loss function;

[0039] Based on the second predicted probability result, the second loss value is calculated using the cross-entropy loss function;

[0040] Based on the third prediction probability result, the third loss value is calculated using the cross-entropy loss function;

[0041] The total loss function is determined based on the first loss value, the second loss value, and the third loss value.

[0042] In some implementations, determining the total loss function based on the first loss value, the second loss value, and the third loss value includes:

[0043] The first loss value, the second loss value, and the third loss value are added together to construct the total loss function.

[0044] Secondly, embodiments of this application also provide an EEG classification system that combines language and visual imagination, the system comprising:

[0045] The data acquisition unit is used to acquire electroencephalogram (EEG) signals combining linguistic and visual imagination.

[0046] The first building unit is used to build a two-branch EEG classification model including a first branch, a second branch, and an attention module; wherein the attention module includes a channel attention submodule and a cross attention submodule.

[0047] The first output unit is used to input the time-domain EEG signal combining the language imagination and visual imagination into the first branch, and to perform feature enhancement on the first branch according to the channel attention submodule to obtain the output result of the first branch;

[0048] The signal transformation unit is used to perform a fast Fourier transform on the time-domain EEG signal combining language imagination and visual imagination to obtain the transformed frequency-domain EEG signal.

[0049] The second output unit is used to input the transformed frequency domain EEG signal into the second branch, and to perform feature enhancement on the second branch according to the channel attention submodule to obtain the output result of the second branch;

[0050] The third output unit is used to input the output results of the first branch and the output results of the second branch into the cross-attention submodule to obtain the output result of the cross-attention submodule.

[0051] The EEG classification unit is used to train a two-branch EEG classification model based on the output results of the cross-attention submodule, the output results of the first branch, and the output results of the second branch, so as to classify the target EEG signal using the trained two-branch EEG classification model.

[0052] Thirdly, embodiments of this application also provide an electronic device, including at least one control processor and a memory for communicatively connecting to the at least one control processor; the memory stores instructions executable by the at least one control processor, the instructions being executed by the at least one control processor to enable the at least one control processor to perform an EEG classification method combining language and visual imagination as described above.

[0053] Fourthly, embodiments of this application also provide a computer-readable storage medium storing computer-executable instructions for causing a computer to perform an EEG classification method combining language and visual imagination as described above.

[0054] It is understood that the beneficial effects of the second to fourth aspects compared with the related technologies are the same as the beneficial effects of the first aspect compared with the related technologies. Please refer to the relevant description in the first aspect above, which will not be repeated here. Attached Figure Description

[0055] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0056] Figure 1 This is a flowchart illustrating an embodiment of the EEG classification method combining language and visual imagination provided in this application;

[0057] Figure 2 This is a schematic diagram of an imagination experiment in an embodiment of the EEG classification method combining language and visual imagination provided in this application;

[0058] Figure 3 This is a schematic diagram of electrode placement in an embodiment of the EEG classification method combining language and visual imagination provided in this application;

[0059] Figure 4 This is a schematic diagram of the overall model structure of an embodiment of the EEG classification method combining language and visual imagination provided in this application;

[0060] Figure 5 These are schematic diagrams of experimental results under different experimental paradigms in the embodiments of the EEG classification method combining language and visual imagination provided in this application.

[0061] Figure 6 This is a schematic diagram of an embodiment of the EEG classification system combining language and visual imagination provided in this application;

[0062] Figure 7 This is a schematic diagram of an embodiment of an electronic device provided in this application. Detailed Implementation

[0063] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0064] In the description of this application, the use of terms such as "first," "second," etc., is for the purpose of distinguishing technical features only and should not be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated or the order of the technical features indicated.

[0065] In the description of this application, it should be understood that the orientation descriptions, such as up, down, etc., are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application.

[0066] In the description of this application, it should be noted that, unless otherwise explicitly defined, terms such as "setup," "installation," and "connection" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this application in conjunction with the specific content of the technical solution.

[0067] Brain-computer interface (BCI) technology allows direct communication between the brain and machines. Electroencephalography (EEG) is widely used to monitor brain activity due to its low cost, convenience, and high temporal resolution. Common methods in EEG-based BCI research include P300 evoked potentials, steady-state visual evoked potentials (SSVEP), and motor imagery. Each method has its advantages. For example, P300 evoked potentials and SSVEP provide clear signals but require external stimulation; while motor imagery does not require external stimulation, but training time is long and the number of repetitions is limited. When performing certain tasks, subjects need to map imagined motor patterns to required control commands according to pre-agreed mapping relationships, resulting in a high cognitive load. Currently, research on BCI related to cognitive function is a new trend. Language imagery and visual imagery, as commonly used human thinking methods, can greatly improve the naturalness and user-friendliness of brain-computer interaction systems. However, current research on language imagery and visual imagery is still in its early stages, and the classification accuracy based on EEG decoding needs further improvement.

[0068] To address the issue that the classification accuracy based on EEG decoding still needs improvement, this application proposes an EEG classification method, system, device, and medium that combines language and visual imagination.

[0069] Reference Figure 1 This application provides an EEG classification method that combines language and visual imagination, the method comprising the following steps:

[0070] Step S100: Obtain the EEG signal combining language imagination and visual imagination;

[0071] Step S200: Construct a bi-branch EEG classification model including a first branch, a second branch, and an attention module; wherein, the attention module includes a channel attention sub-module and a cross attention sub-module;

[0072] Step S300: Input the time-domain EEG signal combining language imagination and visual imagination into the first branch, and perform feature enhancement on the first branch according to the channel attention submodule to obtain the output result of the first branch;

[0073] Step S400: Perform a fast Fourier transform on the time-domain EEG signal combining language imagination and visual imagination to obtain the transformed frequency-domain EEG signal.

[0074] Step S500: Input the transformed frequency domain EEG signal into the second branch, and perform feature enhancement on the second branch according to the channel attention submodule to obtain the output result of the second branch;

[0075] Step S600: Input the output results of the first branch and the second branch into the cross-attention submodule to obtain the output results of the cross-attention submodule;

[0076] Step S700: Based on the output results of the cross-attention submodule, the first branch, and the second branch, train the bi-branch EEG classification model so that the trained bi-branch EEG classification model can be used to classify the target EEG signal.

[0077] In this embodiment, a two-branch EEG classification model is constructed, comprising a first branch, a second branch, and an attention module. The time-domain EEG signal combining language and visual imagination is input to the first branch, and feature enhancement is performed on the first branch using the channel attention submodule. Then, the EEG signal after Fast Fourier Transform (FFT) is input to the second branch, and feature enhancement is performed on the second branch using the channel attention submodule. By learning the EEG signal combining language and visual imagination through both branches, and using the channel attention submodule for feature enhancement, the model's classification accuracy can be improved. The outputs of the first and second branches are input to the cross-attention submodule to obtain its output. Based on the outputs of the cross-attention submodule, the first branch, and the second branch, the two-branch EEG classification model is trained. This trained model is then used to classify the target EEG signal. The cross-attention submodule captures the relationship between the outputs of the first and second branches, and the two-branch EEG classification model is trained by combining these outputs, further improving the classification accuracy of the EEG.

[0078] In some implementations, the first branch includes a first convolutional module, a first depthwise convolutional module, a first depthwise separable convolutional module, and a GRU module. The time-domain EEG signal combining language imagination and visual imagination is input to the first branch, and feature enhancement is performed on the first branch according to the channel attention submodule to obtain the output of the first branch, including:

[0079] The time-domain EEG signal combining language imagination and visual imagination is processed by the first convolution module to extract the time feature map of the EEG signal;

[0080] The temporal feature map is enhanced by the channel attention submodule to obtain the enhanced first feature map.

[0081] The enhanced first feature map is input into the first depthwise convolutional module to obtain the output of the first depthwise convolutional module;

[0082] The output of the first depthwise convolutional module is input into the first depthwise separable convolutional module to obtain the output of the first depthwise separable convolutional module;

[0083] The output of the first depthwise separable convolution module is input into the GRU module to obtain the output of the first branch.

[0084] In this embodiment, by extracting the temporal feature map of the EEG signal combining language imagination and visual imagination through the first convolutional module, the nonlinear characteristics of the two-branch EEG classification model can be enhanced and the convergence of the training process can be accelerated. Then, the temporal feature map of the EEG signal combining language imagination and visual imagination is extracted through the first convolutional module. The channel attention submodule enhances the temporal feature map, enabling the first branch of the two-branch EEG classification model to emphasize important channel features and suppress unimportant channel features. The enhanced first feature map is then input into the first depthwise convolutional module to obtain its output. The output of the first depthwise convolutional module is then input into the first depthwise separable convolutional module to obtain its output, and finally into the GRU module to obtain the output of the first branch. This process better captures the temporal dependencies of the feature sequences, thereby improving the prediction accuracy of the two-branch EEG classification model.

[0085] In some implementations, the second branch includes a second convolutional module, a second depthwise convolutional module, a second depthwise separable convolutional module, a Flatten layer, and a Linear layer. The transformed frequency-domain EEG signal is input to the second branch, and feature enhancement is performed on the second branch according to the channel attention submodule to obtain the output of the second branch, including:

[0086] The frequency feature map of the transformed frequency domain EEG signal is extracted by the second convolution module.

[0087] The frequency feature map is enhanced by the channel attention submodule to obtain the enhanced second feature map.

[0088] The enhanced second feature map is input into the second deep convolutional module to obtain the output of the second deep convolutional module;

[0089] The output of the second depthwise convolutional module is input into the second depthwise separable convolutional module to obtain the output of the second depthwise separable convolutional module;

[0090] The output of the second depthwise separable convolutional module is processed through the Flatten layer and the Linear layer to obtain the output of the second branch.

[0091] In this embodiment, by extracting the frequency feature map of the transformed frequency domain EEG signal through a second convolutional module, the nonlinear characteristics of the two-branch EEG classification model can be enhanced and the convergence of the training process can be accelerated. Then, the frequency feature map is enhanced through a channel attention submodule. This enhancement allows the second branch of the two-branch EEG classification model to emphasize important channel features and suppress unimportant ones. The enhanced second feature map is then input into a second depthwise convolutional module to obtain its output. This output is then input into a second depthwise separable convolutional module to obtain its output, which better captures the temporal dependencies of the feature sequences, thereby improving the prediction accuracy of the two-branch EEG classification model. The output of the second depthwise separable convolutional module is processed through a Flatten layer and a Linear layer to obtain the output of the second branch. The output of the second branch is made to have the same length as the output of the first branch for input into the two-branch EEG classification model for subsequent processing.

[0092] In some implementations, the outputs of the first branch and the second branch are input to the cross-attention submodule in the following manner to obtain the output of the cross-attention submodule:

[0093] Q f =X f W q ,K t =X t W k V t =X t W v

[0094]

[0095] Among them, Q f Indicates a query, X f W represents the output of the second branch. q Let K represent the second projection matrix. t Indicates key, X t W represents the output of the first branch. k V represents the first projection matrix. t Represents the value, W v Represents the third projection matrix, CrossAttention(Q) f ,K t V t ) represents the output of the cross-attention submodule, Softmax(·) represents the activation function, and d represents the dimension of the time feature map or frequency feature map.

[0096] In this embodiment, the cross-attention submodule is used to capture the relationship between the features output by the two branches, which can further improve the accuracy of the bi-branch EEG classification model.

[0097] In some implementations, the trained two-branch EEG classification model is determined based on the output of the cross-attention submodule, the output of the first branch, and the output of the second branch, including:

[0098] The output of the cross-attention submodule is passed through a Linear layer and a Softmax activation function to obtain the first prediction probability result;

[0099] The output of the first branch is passed through a Linear layer and a Softmax activation function to obtain the second prediction probability result;

[0100] The output of the second branch is passed through a Linear layer and a Softmax activation function to obtain the third prediction probability result;

[0101] The total loss function is determined based on the first prediction probability result, the second prediction probability result, and the third prediction probability result;

[0102] The trained bibranch EEG classification model is determined based on the total loss function.

[0103] In this embodiment, by combining the output results of the cross-attention submodule, the output results of the first branch, and the output results of the second branch to train the two-branch EEG classification model, the classification accuracy of EEG can be further improved.

[0104] In some implementations, the total loss function is determined based on the first prediction probability result, the second prediction probability result, and the third prediction probability result, including:

[0105] Based on the first predicted probability result, the first loss value is calculated using the cross-entropy loss function;

[0106] Based on the second prediction probability result, the second loss value is calculated using the cross-entropy loss function;

[0107] Based on the third prediction probability result, the third loss value is calculated using the cross-entropy loss function;

[0108] The total loss function is determined based on the first loss value, the second loss value, and the third loss value.

[0109] In this embodiment, the total loss function is constructed by combining multiple loss values, which can better train the bibranch EEG classification model and improve the prediction accuracy of the bibranch EEG classification model.

[0110] In some implementations, the total loss function is determined based on the first loss value, the second loss value, and the third loss value, including:

[0111] The first loss value, the second loss value, and the third loss value are added together to construct the total loss function.

[0112] In this embodiment, by adding the first loss value, the second loss value, and the third loss value to construct the total loss function, the bibranch EEG classification model can be trained better, thereby changing the prediction accuracy of the bibranch EEG classification model.

[0113] To facilitate understanding by those skilled in the art, a set of preferred embodiments is provided below:

[0114] In EEG-based brain-computer interface (BCI) research, commonly used methods include P300 evoked potentials, steady-state visual evoked potentials (SSVEP), and motor imagery. Each method has its advantages. For example, P300 evoked potentials and SSVEP can provide clear signals but require external stimulation; while motor imagery does not require external stimulation, but training time is long and the number of repetitions is limited. When performing certain tasks, subjects need to map imagined motor patterns to required control commands according to pre-agreed mapping relationships, resulting in a high cognitive load for the subjects (in this example, the subjects).

[0115] Currently, research on brain-computer interfaces (BCIs) related to cognitive functions is becoming a new trend. Language imagination and visual imagination, as commonly used human thinking methods, can greatly improve the naturalness and user-friendliness of brain-computer interface systems. However, current research on language imagination and visual imagination is still in its early stages, and the classification accuracy based on EEG decoding needs further improvement.

[0116] To address the aforementioned issues, this embodiment designs a novel network framework by combining linguistic and visual imagination paradigms, verifying the feasibility of the fusion paradigm and improving decoding accuracy.

[0117] Specifically, the steps include the following:

[0118] 1. Prepare an EEG signal dataset.

[0119] (1) In this embodiment, seven healthy subjects were selected to participate in the experiment (age range: 22-29 years, mean age: 24.57±2.82 years, 6 males and 1 female). All subjects had normal vision or corrected vision and no history of mental illness, neurological disease or speech disorder. Before the experiment, all subjects were informed of the purpose and procedure of the experiment, and all subjects signed informed consent forms in accordance with the Declaration of Helsinki. All subjects were required to ensure sufficient sleep before the experiment and abstain from alcohol.

[0120] (2) In order to explore whether the experimental paradigm of integrating language imagery and visual imagery can enhance brain activity, all subjects were required to perform three types of imagery tasks: language imagery (SI), visual imagery (VI), and language imagery combined with visual imagery (SI+VI).

[0121] In each imagination task, each participant was required to perform 15 rounds of imagination experiments, with each round consisting of 4 sets of different imagination materials presented in a random order. Figure 2 This demonstrates the detailed process of three imagination tasks within a single round of an imagination experiment. The experiment in this embodiment comprises 15 rounds, each consisting of four sets (set1, set2, set3, and set4) of different imagination materials presented in a random order. Each set includes a 2-second cue presentation, four trials totaling 18 seconds of imagination, and a 3-second rest. The cue materials for the three imagination tasks are: a) the Chinese characters “up,” “down,” “left,” and “right”; b) a video of a drone flying up, down, left, and right; and c) a video of a drone flying up, down, left, and right, along with the corresponding Chinese characters.

[0122] At the start of each group, the computer beeps to signal the participant to begin the experiment immediately. Simultaneously, a 2-second cue is displayed on the screen, followed by a 1.5-second cross in the center of the screen to encourage the participant to focus and immediately enter the imagination phase. After the cross disappears, the screen goes black for 3 seconds (imagine), during which the participant performs the imagination task related to the given cue. This cross and imagination phase is repeated four times after each cue. After four imagining sessions, there is a 3-second rest period before moving to the next group. There is a 15-second rest between each round to allow participants to adjust and clear their minds. Therefore, each paradigm will have 240 trials of experimental data, with 60 trials per category.

[0123] The three imaginative tasks share the same stage division; the difference lies in the use of different prompts. Participants perform different imaginative actions based on these prompts. In the verbal imaginative task, the prompts are the Chinese characters for "up," "down," "left," and "right." Participants must imagine themselves saying these characters while keeping their facial muscles still and without making a sound. In the visual imaginative task, the prompts present a video of a drone ascending, descending, turning left, and turning right. Participants must imagine a drone performing these flight maneuvers while keeping their eyes as still as possible. In the combined verbal and visual imaginative task, the prompts simultaneously present the drone's flight maneuvers (ascending, descending, turning left, and turning right) and the corresponding Chinese characters ("up," "down," "left," and "right"). Participants silently recite the words corresponding to the drone's flight maneuvers while imagining them.

[0124] (3) During data acquisition, each subject was approximately 0.5m away from the monitor, with their hands resting naturally on the armrests of the chair, maintaining a comfortable position. The experimental paradigm was implemented using a psychopharmaceutical design. EEG data were acquired using the NeuSen W series wireless digital EEG acquisition system from Boruikang, with a sampling frequency of 1000Hz. A 64-channel EEG cap was used. Data was not collected from five electrodes: ECG, HEOR, HEOL, VEOU, and VEOL. Therefore, 59 channels of electrodes were actually used for data acquisition. The electrode positions were placed according to the 10-20 international system, such as... Figure 3 As shown, the AFz electrode is the ground electrode, the CPz electrode is the reference electrode, and there are a total of 59 channels for actual effective acquisition of EEG signals.

[0125] The acquired EEG signals were bandpass filtered using a 1-80Hz FIR bandpass filter, and then a 50Hz FIR notch filter was used to remove power line interference. The signals were downsampled to 256Hz, and ICA (Inductively Coupled Angiography) was used to remove electrooculography (EOG) and electromyography (EMG) artifacts. This yielded EEG data of shape C×T for each trial, where C represents the electrode channel and T represents the time sample point. A 5-fold cross-validation method was then used to divide the data into five equal parts according to categories, with four parts serving as the training set and the remaining part as the test set. In each fold, Z-score normalization was applied to all data.

[0126]

[0127] Where x represents EEG data, Represents the data after Z-score standardization, μ and σ 2 This represents the mean and variance calculated from the training set, which are directly applied to the Z-score standardization of the training and test sets.

[0128] Finally, a sliding window was applied to both the training and test sets to increase the number of samples. The sliding window was set to 2 seconds and the step size was set to 0.1 seconds.

[0129] 2. Design the two-branch EEG classification model in this embodiment.

[0130] This embodiment designs an end-to-end dual-branch network (TFCA-DBNet) that uses cross-attention to fuse time and frequency features (i.e., the dual-branch EEG classification model in this embodiment). Figure 4 As shown, Figure 4 (a) is a schematic diagram of the framework of the two-branch EEG classification model in this embodiment. Figure 4 (b) is a schematic diagram of the cross-attention module; Figure 4 (c) is a schematic diagram of the SE channel attention module. This framework mainly consists of three parts: a time-domain branch (i.e., the first branch), a frequency-domain branch (i.e., the second branch), and an attention module. The input to the two-branch EEG classification model is a preprocessed EEG signal (which can be an EEG signal combining language and visual imagery). The features of the EEG signal can be described primarily from the time, frequency, and spatial domains. The time-domain and frequency-domain branches of TFCA-DBNet (i.e., the two-branch EEG classification model in this embodiment) can learn the time-domain and frequency-domain features of the EEG signal, respectively. The SE attention mechanism of the attention module can extract spatial features, making the model pay more attention to important electrode channels. The model output includes the prediction probability of the time-domain branch, the prediction probability of the frequency-domain branch, and the prediction probability after fusing time and frequency features.

[0131] The outputs of the three models mentioned above are used to train a bibranch EEG classification model. After training, only the predicted probabilities after fusing time-frequency features are used. Then, based on the predicted probabilities after fusing time-frequency features, the category of the EEG combining language imagination and visual imagination is determined. That is, the category with the higher probability of the EEG is determined. The EEG combining language imagination and visual imagination includes four different contents of language imagination and visual imagination, namely, simultaneously presenting the flight movements of a drone (i.e., ascending, descending, turning left, and turning right) and the corresponding Chinese characters (i.e., "up", "down", "left", and "right").

[0132] The design of the bibranch EEG classification model specifically includes:

[0133] (1) Temporal Branch: The temporal branch mainly includes three convolutional modules and one recurrent neural network (GRU) module. Specifically:

[0134] In this embodiment, a 1D CNN module (i.e.) is first used. Figure 4 The 1D CNN Block in the model performs convolutions along the time dimension to extract temporal features of the EEG signal. This module uses 25 convolutional kernels of size (1,15) with a stride of (1,1), followed by ReLU and batch normalization layers to enhance the model's non-linearity and accelerate training convergence. Max pooling layers with kernel size (1,3) are then used to reduce the number of parameters, and a dropout layer with a dropout rate of 0.3 is used to prevent overfitting.

[0135] The second convolutional module is a depthwise convolutional module (i.e. Figure 4 The Depthwise Conv Block in the module contains a depthwise convolutional layer of size 2 with a kernel size of (C,1) to convolve each feature map along the electrode channel dimension. Therefore, the electrode channel dimension of the feature tensor is compressed to 1 after passing through this layer, thus fusing spatial features.

[0136] The last convolutional module is a depthwise separable convolutional module (i.e. Figure 4 The Separable Conv Block (SCB) consists of a depthwise convolutional layer with a kernel size of (1, 15) and a pointwise convolutional layer with 45 kernels. This module further extracts temporal features and combines different feature maps. In addition, both the depthwise convolutional module and the depthwise separable convolutional module use batch normalization layers, ReLU, average pooling layers with a kernel size of (1, 3), and dropout layers with a dropout rate of 0.3 after their respective convolutional layers.

[0137] After the three convolutional modules is a GRU module (i.e., ... Figure 4 The GRU Block in the module is used to capture the temporal dependencies of feature sequences. This module consists of two stacked GRU units with a hidden state size of 16, and the output is the last hidden state generated by the second GRU unit, which is a feature vector of length 16.

[0138] (2) Frequency domain branching: The frequency domain branching first uses the Fast Fourier Transform (FFT) to transform the time-domain EEG signal into a frequency-domain signal. Since the EEG signal is a real number signal, its FFT has the property of conjugate symmetry. Therefore, only the positive frequency part needs to be taken. The shape of the frequency domain signal after FFT is C×F, where C is the electrode channel and F is the number of frequency points.

[0139] The frequency domain branch uses the same three convolutional modules as the time domain branch (i.e., a 1D CNN module, a depthwise convolutional module, and a depthwise separable convolutional module). After passing through the depthwise separable convolutional module, a Flatten layer is used to flatten the feature tensor output by the depthwise separable convolutional module into one dimension. A Linear layer is then used to adjust the length of the one-dimensional feature vector to 16, making it the same as the length of the time-domain feature output by the time-domain branch, for use as input to subsequent modules. The main layer parameters of the time-domain branch and the frequency domain branch are shown in Table 1.

[0140] Table 1. Main layer parameters for time-domain and frequency-domain branches.

[0141] Time-domain / frequency-domain branching 1D CNN 1 25 (1,15) Time-domain / frequency-domain branch Depthwise Conv 25 25×2 (C,1) Time-domain / frequency-domain branch Separable Conv 25×2 45 (1,15) Temporal branch GRU_1 45 16 - Temporal branch GRU_2 16 16 -

[0142] (3) Attention Modules: TFCA-DBNet uses two attention modules: the SE attention module (i.e., ... Figure 4 SEBlock in the middle) and cross-attention module (i.e. Figure 4 Cross-Attention (in the context of cross-attention).

[0143] The temporal and frequency domain branches share a single SE attention module, located between the 1D CNN module and the deep convolutional module. This module enables the model to focus on important electrode channel features, and its structure is as follows: Figure 4 As shown in (c), first, global average pooling (i.e., ...) is used. Figure 4 The pooling process transforms the feature tensor of shape (D,C,S) into (1,C,1), where D is the number of feature maps (feature dimension), C is the electrode channel, and S is the number of sample points (T for the time domain branch and F for the frequency domain branch). Then, two linear layers are used for squeezing and excitation to learn a weight vector of shape (1,C,1) to measure the importance of each electrode channel. Finally, by multiplying the weight vector (1,C,1) with the original feature tensor (D,C,S) channel by channel, the model emphasizes important channel features and suppresses unimportant channel features.

[0144] The cross-attention module is used to capture the relationship between the features output by the time-domain branch and the frequency-domain branch. Its structure is as follows: Figure 4 As shown in (b). Let the characteristics of the time-domain branch output be... The frequency domain branch output is Where d is the dimension of the time or frequency feature. X f via projection matrix Transform into queries Q, X t After passing through the projection matrix respectively and The transformation is into key K and value V, i.e.:

[0145] Q f =X f W q ,K t =X t W k V t =X t W v

[0146] First, through the dot product Q f and K t To capture the correlation between time-domain and frequency-domain features, a scaling factor is used. To prevent gradient vanishing, the cross-attention score is obtained through the Softmax function (i.e., ...). Figure 4 (AttentionScore in V) and compare it with V t Dot product. This process can be represented as:

[0147]

[0148] (4) Loss Function: To facilitate the learning of feature representations by the temporal and frequency branches, the temporal branch, frequency branch, and cross-attention module all use a Linear layer and Softmax activation function to obtain the prediction output, which is then included in the loss function calculation. The prediction outputs of the temporal branch, frequency branch, and cross-attention module are respectively represented as the first prediction probability result y. c Second prediction probability result y t And the third predicted probability result y f This embodiment uses cross-entropy as the loss function:

[0149]

[0150] Where N represents the number of samples in a batch, C represents the number of classes, and y represents the true label of the sample. This represents the probability distribution predicted by the model. The model's total loss function can be expressed as:

[0151] L total =L t +L f +L c

[0152] Among them, L total L represents the total loss function. c L represents the loss of the cross-attention module (i.e., the first loss value). t L represents the loss of the time-domain branch (i.e., the second loss value). f This represents the loss of the frequency domain branch (i.e., the third loss value).

[0153] 3. The experiment in this embodiment.

[0154] (1) Training settings.

[0155] This embodiment implements the two-branch EEG classification model using Python 3.8 and PyTorch 2.0 on a Geforce RTX 2080 Ti GPU with CUDA version 11.8. The Adam optimizer is used, with the learning rate set to 0.001, for 200 epochs of training. Every 30 epochs, the learning rate is reduced to 90% of its original value.

[0156] (2) Accuracy results for different paradigms.

[0157] Figure 5 This presents the average accuracy results for each participant after 5-fold cross-validation under different experimental paradigms, and the average accuracy results for all participants. Figure 5 The x-axis represents the subjects, and the y-axis represents the accuracy. It can be seen that for the SI task, subject 4 achieved the best classification performance at 64.76%. For both the VI and SI+VI tasks, subject 3 achieved the best classification performance, at 72.21% and 82.33%, respectively. Subjects 2 and 3 showed the most significant improvement in the SI+VI task, with improvements of 27.05% and 29.12% respectively compared to the SI task, and 11.40% and 10.12% respectively compared to the VI task. Subject 7 showed a smaller improvement in the SI+VI task, with improvements of 7.01% and 2.99% compared to the SI and VI tasks, respectively. Overall, however, all subjects achieved improved accuracy in the SI+VI task compared to performing SI or VI alone, and most subjects achieved higher accuracy in the VI task than in the SI task. The average accuracy rate for all participants on the SI task was 51.53%, the average accuracy rate for the VI task was 60.43%, and the average accuracy rate for the SI+VI task was 67.33%, representing improvements of 15.80% and 6.90% respectively compared to the SI and VI tasks. This indicates that the SI+VI imagery task can enhance brain activity and improve the distinguishability of brain activity patterns.

[0158] (3) Accuracy results of different methods.

[0159] To evaluate the impact of temporal and frequency domain branches on the performance of TFCA-DBNet (i.e., the two-branch EEG classification model in this embodiment), this embodiment compares it with other methods that have shown significant performance in previous studies, including CSP+SVM, DWT+EMD+SVM, EEGNet, DeepConvNet, ShallowConvNet, EEGInception, and EEGConformer. This embodiment validates the classification performance of different experimental paradigms using single-branch networks, namely a temporal branch network with an SE module (TB-SENet) and a frequency branch network with an SE module (FB-SENet). The results are shown in Table 2, using accuracy and standard deviation as evaluation criteria. CSP+SVM, DWT+EMD+SVM, EEGNet, DeepConvNet, ShallowConvNet, EEGInception, and EEGConformer in Table 2 are all existing technologies and are not specifically described in this embodiment. As mentioned above, the average accuracy results for different paradigms show that the SI+VI task achieved the highest classification performance under different methods. Furthermore, compared to single-branch networks, TFCA-DBNet, which fuses temporal and frequency domain features through cross-attention, can further improve classification performance, demonstrating the superiority of this architecture in three experimental paradigms.

[0160] Table 2 Comparison of classification performance of different methods (Acc±std)

[0161]

[0162] Reference Figure 6 This application also provides an EEG classification system that combines language and visual imagination. The system includes a data acquisition unit 100, a first construction unit 200, a first output unit 300, a signal transformation unit 400, a second output unit 500, a third output unit 600, and an EEG classification unit 700, wherein:

[0163] The data acquisition unit 100 is used to acquire electroencephalogram (EEG) signals combining language imagination and visual imagination.

[0164] The first building unit 200 is used to build a two-branch EEG classification model that includes a first branch, a second branch, and an attention module; wherein, the attention module includes a channel attention submodule and a cross attention submodule;

[0165] The first output unit 300 is used to input the time-domain EEG signal combining language imagination and visual imagination into the first branch, and to perform feature enhancement on the first branch according to the channel attention submodule to obtain the output result of the first branch;

[0166] The signal transformation unit 400 is used to perform a fast Fourier transform on the EEG signal combining language imagination and visual imagination to obtain the transformed frequency domain EEG signal.

[0167] The second output unit 500 is used to input the transformed frequency domain EEG signal into the second branch, and to perform feature enhancement on the second branch according to the channel attention submodule to obtain the output result of the second branch;

[0168] The third output unit 600 is used to input the output results of the first branch and the second branch into the cross-attention submodule to obtain the output results of the cross-attention submodule.

[0169] The EEG classification unit 700 is used to train a two-branch EEG classification model based on the output results of the cross-attention submodule, the first branch, and the second branch, so as to classify the target EEG signal using the trained two-branch EEG classification model.

[0170] It should be noted that since the EEG classification system combining language and visual imagination in this embodiment is based on the same inventive concept as the EEG classification method combining language and visual imagination described above, the corresponding content in the method embodiment is also applicable to this system embodiment, and will not be described in detail here.

[0171] Reference Figure 7 This application also provides an electronic device, which includes:

[0172] At least one memory;

[0173] At least one processor;

[0174] At least one program;

[0175] The program is stored in memory, and the processor executes at least one program to implement the above-described EEG classification method combining language and visual imagination.

[0176] This electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), and in-vehicle computers.

[0177] The electronic devices according to embodiments of this application will now be described in detail.

[0178] The processor 1600 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this disclosure.

[0179] The memory 1700 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1700 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1700 and is called and executed by the processor 1600 to execute the EEG classification method combining language and visual imagination of the embodiments of this disclosure.

[0180] The input / output interface 1800 is used to implement information input and output.

[0181] The communication interface 1900 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0182] Bus 2000 transmits information between various components of the device (e.g., processor 1600, memory 1700, input / output interface 1800, and communication interface 1900);

[0183] The processor 1600, memory 1700, input / output interface 1800 and communication interface 1900 are connected to each other within the device via bus 2000.

[0184] This disclosure also provides a storage medium, which is a computer-readable storage medium storing computer-executable instructions for causing a computer to perform the above-described EEG classification method combining language and visual imagination.

[0185] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0186] The embodiments described in this disclosure are for the purpose of more clearly illustrating the technical solutions of this disclosure and do not constitute a limitation on the technical solutions provided by this disclosure. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by this disclosure are also applicable to similar technical problems.

[0187] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this disclosure, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0188] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0189] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0190] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0191] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0192] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0193] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0194] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0195] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks. The embodiments of this application have been described in detail above with reference to the accompanying drawings, but this application is not limited to the above embodiments. Various changes can be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of this application.

Claims

1. A brainwave classification method combining language and visual imagination, characterized in that, The method includes: Obtain EEG signals combining linguistic and visual imagery; A two-branch EEG classification model is constructed, comprising a first branch, a second branch, and an attention module; wherein the attention module includes a channel attention submodule and a cross attention submodule. The time-domain EEG signal combining the language imagination and visual imagination is input to the first branch, and feature enhancement is performed on the first branch according to the channel attention submodule to obtain the output result of the first branch; the first branch includes a first convolution module, a first depthwise convolution module, a first depthwise separable convolution module, and a GRU module, including: The time-domain EEG signal combining the language imagination and visual imagination is processed by the first convolution module to extract the time feature map of the EEG signal; The temporal feature map is enhanced by the channel attention submodule to obtain the enhanced first feature map. The enhanced first feature map is input into the first depthwise convolutional module to obtain the output result of the first depthwise convolutional module. The output of the first depthwise convolution module is input into the first depthwise separable convolution module to obtain the output of the first depthwise separable convolution module; The output of the first depthwise separable convolution module is input into the GRU module to obtain the output of the first branch; The time-domain EEG signal combining language imagination and visual imagination is subjected to Fast Fourier Transform to obtain the transformed frequency-domain EEG signal. The transformed frequency-domain EEG signal is input to the second branch, and feature enhancement is performed on the second branch according to the channel attention submodule to obtain the output result of the second branch; the second branch includes a second convolution module, a second depthwise convolution module, a second depthwise separable convolution module, a Flatten layer, and a Linear layer, including: The frequency feature map of the transformed frequency domain EEG signal is extracted by the second convolution module. The frequency feature map is enhanced by the channel attention submodule to obtain an enhanced second feature map. The enhanced second feature map is input into the second deep convolution module to obtain the output result of the second deep convolution module; The output of the second depthwise convolution module is input into the second depthwise separable convolution module to obtain the output of the second depthwise separable convolution module; The output of the second depthwise separable convolutional module is processed through the Flatten layer and the Linear layer to obtain the output of the second branch; The output results of the first branch and the second branch are input into the cross-attention submodule to obtain the output result of the cross-attention submodule; Based on the output results of the cross-attention submodule, the first branch, and the second branch, a two-branch EEG classification model is trained so that the trained two-branch EEG classification model can be used to classify the target EEG signal.

2. The EEG classification method combining language and visual imagination according to claim 1, characterized in that, The output results of the first branch and the second branch are input into the cross-attention submodule in the following manner to obtain the output results of the cross-attention submodule, including: in, Indicates a query. This indicates the output of the second branch. Represents the second projection matrix. Indicates key, This indicates the output of the first branch. Denotes the first projection matrix. Represents value, Represents the third projection matrix. This represents the output of the cross-attention submodule. This represents the activation function. This indicates the dimension of the time feature map or frequency feature map.

3. The EEG classification method combining language and visual imagination according to claim 1, characterized in that, The step of training a two-branch EEG classification model based on the output of the cross-attention submodule, the output of the first branch, and the output of the second branch includes: The output of the cross-attention submodule is passed through a Linear layer and a Softmax activation function to obtain the first prediction probability result; The output of the first branch is passed through a Linear layer and a Softmax activation function to obtain the second prediction probability result; The output of the second branch is passed through a Linear layer and a Softmax activation function to obtain the third prediction probability result; The total loss function is determined based on the first prediction probability result, the second prediction probability result, and the third prediction probability result; A two-branch EEG classification model is trained based on the total loss function.

4. The EEG classification method combining language and visual imagination according to claim 3, characterized in that, The step of determining the total loss function based on the first prediction probability result, the second prediction probability result, and the third prediction probability result includes: Based on the first predicted probability result, the first loss value is calculated using the cross-entropy loss function; Based on the second predicted probability result, the second loss value is calculated using the cross-entropy loss function; Based on the third prediction probability result, the third loss value is calculated using the cross-entropy loss function; The total loss function is determined based on the first loss value, the second loss value, and the third loss value.

5. The EEG classification method combining language and visual imagination according to claim 4, characterized in that, The step of determining the total loss function based on the first loss value, the second loss value, and the third loss value includes: The first loss value, the second loss value, and the third loss value are added together to construct the total loss function.

6. A brainwave classification system combining language and visual imagination, characterized in that, The system employing the EEG classification method combining language and visual imagery as described in any one of claims 1 to 5, comprises: The data acquisition unit is used to acquire electroencephalogram (EEG) signals combining linguistic and visual imagination. The first building unit is used to build a two-branch EEG classification model including a first branch, a second branch, and an attention module; wherein the attention module includes a channel attention submodule and a cross attention submodule. The first output unit is used to input the time-domain EEG signal combining the language imagination and visual imagination into the first branch, and to perform feature enhancement on the first branch according to the channel attention submodule to obtain the output result of the first branch; The signal transformation unit is used to perform a fast Fourier transform on the time-domain EEG signal combining language imagination and visual imagination to obtain the transformed frequency-domain EEG signal. The second output unit is used to input the transformed frequency domain EEG signal into the second branch, and to perform feature enhancement on the second branch according to the channel attention submodule to obtain the output result of the second branch; The third output unit is used to input the output results of the first branch and the output results of the second branch into the cross-attention submodule to obtain the output result of the cross-attention submodule. The EEG classification unit is used to train a two-branch EEG classification model based on the output results of the cross-attention submodule, the output results of the first branch, and the output results of the second branch, so as to classify the target EEG signal using the trained two-branch EEG classification model.

7. An electronic device, characterized in that, It includes at least one control processor and a memory for communicatively connecting to the at least one control processor; the memory stores instructions executable by the at least one control processor to enable the at least one control processor to perform the EEG classification method combining language and visual imagery as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the EEG classification method combining language and visual imagination as described in any one of claims 1 to 5.

Images