Dual-branch EEG signal decoding method and device based on multi-scale feature fusion

Abstract

The present invention relates to a dual-branch EEG signal decoding method and device based on multi-scale feature fusion. The method comprises: obtaining an EEG time series signal and a corresponding frequency spectrum as a pair of training data; constructing a dual-branch EEG signal decoding model, the dual-branch EEG signal decoding model comprising a multi-level time domain feature extractor, a frequency domain feature extractor, a multi-scale feature fusion module and a classifier; inputting the training data into the dual-branch EEG signal decoding model to obtain a prediction result; calculating a loss function based on the label in the EEG time series signal and the prediction result, and training the dual-branch EEG signal decoding model based on minimizing the loss function to obtain a trained dual-branch EEG signal decoding model; and performing category prediction on the EEG signal to be classified using the trained dual-branch EEG signal decoding model to generate a classification result. The present invention can better utilize the time domain signal and frequency domain information in the EEG signal, thereby improving the decoding accuracy of the EEG signal.

Description

Technical Field

[0001] The present invention relates to the technical field of electroencephalogram (EEG) signal decoding, and in particular to a dual-branch EEG signal decoding method and device based on multi-scale feature fusion. Background Art

[0002] Brain Computer Interface (BCI) technology can quickly and directly convert brain signals into commands to control external devices without going through peripheral nerves and muscle tissue, thereby realizing information transmission between the brain and the outside world. Electroencephalogram (EEG) has become the most important mode for implementing BCI systems due to its non-invasiveness, high temporal resolution and convenience. Steady State Visual Evoked Potential (SSVEP) is an EEG signal generated by continuous visual stimulation of a specific frequency. Brain-computer interfaces designed based on SSVEP have been widely used in application fields with high requirements for information transmission due to their advantages such as obvious frequency domain characteristics, a large number of selectable targets, a fast information transmission rate and the fact that subjects do not need to be trained. Therefore, improving the recognition accuracy of SSVEP signals and the information transmission rate remains the research focus of BCI systems based on SSVEP signals.

[0003] In recent years, deep learning-based SSVEP signal decoding methods have been widely studied due to its powerful feature representation capabilities and flexibility. In existing research, since SSVEP signals have more pronounced signal characteristics in the frequency domain, in addition to directly extracting features from SSVEP time series signals, the SSVEP time domain signals are often converted to the frequency domain for feature extraction. This is then combined with the powerful automatic feature extraction capabilities of neural networks to further improve the performance of SSVEP signal classification. Furthermore, to obtain more comprehensive representation information, time-frequency analysis methods such as short-time Fourier transform, continuous wavelet transform, and discrete wavelet transform are also widely used to extract time-frequency features from SSVEP signals. However, most existing methods focus solely on sequentially extracting different features from time or frequency domain signals, or simply concatenate time and frequency domain features, ignoring the dependencies between different types of features. This can reduce the comprehensive representation capability of the fused features. Summary of the Invention

[0004] Based on this, it is necessary to provide a dual-branch EEG signal decoding method and device based on multi-scale feature fusion that can solve the problem of low SSVEP signal decoding accuracy in response to the above technical problems.

[0005] A dual-branch EEG signal decoding method based on multi-scale feature fusion, the method comprising:

[0006] Acquire a data set, the data set including an EEG time series signal and a frequency spectrum corresponding to the EEG time series signal;

[0007] Constructing a dual-branch EEG signal decoding model, wherein the dual-branch EEG signal decoding model includes a multi-level time domain feature extractor, a frequency domain feature extractor, a multi-scale feature fusion module and a classifier;

[0008] The EEG time series signal and the corresponding spectrum are input into the dual-branch EEG signal decoding model as a pair of training data. The multi-level time domain feature extractor performs multi-level feature extraction on the EEG time series signal to obtain spatiotemporal features. The frequency domain feature extractor performs feature extraction on the spectrum to obtain frequency domain features. The spatiotemporal features and the frequency domain features are input into the multi-scale feature fusion module for fusion at different scales to obtain multi-scale fusion features. The multi-scale fusion features are input into the classifier for category prediction to obtain a prediction result.

[0009] Calculating a loss function based on the label in the EEG time series signal and the prediction result, and training the dual-branch EEG signal decoding model in a manner of minimizing the loss function to obtain a trained dual-branch EEG signal decoding model;

[0010] The trained dual-branch EEG signal decoding model is used to perform category prediction on the EEG signal to be classified and generate a classification result.

[0011] A dual-branch EEG signal decoding device based on multi-scale feature fusion, the device comprising:

[0012] A data acquisition module, configured to acquire a data set, wherein the data set includes an EEG time series signal and a frequency spectrum corresponding to the EEG time series signal;

[0013] A model construction module is used to construct a dual-branch EEG signal decoding model, which includes a multi-level time domain feature extractor, a frequency domain feature extractor, a multi-scale feature fusion module and a classifier;

[0014] A model training module is used to input the EEG time series signal and the corresponding spectrum as a pair of training data into the dual-branch EEG signal decoding model, the multi-level time domain feature extractor performs multi-level feature extraction on the EEG time series signal to obtain spatiotemporal features; the frequency domain feature extractor performs feature extraction on the spectrum to obtain frequency domain features; the spatiotemporal features and the frequency domain features are input into the multi-scale feature fusion module for fusion at different scales to obtain multi-scale fusion features; the multi-scale fusion features are input into the classifier for category prediction to obtain a prediction result; a loss function is calculated based on the label in the EEG time series signal and the prediction result, and the dual-branch EEG signal decoding model is trained based on minimizing the loss function to obtain a trained dual-branch EEG signal decoding model;

[0015] The classification module is used to predict the category of the EEG signal to be classified using the trained dual-branch EEG signal decoding model to generate a classification result.

[0016] The above-mentioned dual-branch EEG signal decoding method and device based on multi-scale feature fusion first obtains a data set, which includes an EEG time series signal and a spectrum corresponding to the EEG time series signal; constructs a dual-branch EEG signal decoding model, which includes a multi-level time domain feature extractor, a frequency domain feature extractor, a multi-scale feature fusion module and a classifier; the EEG time series signal and the corresponding spectrum are input into the dual-branch EEG signal decoding model as a pair of training data, and the multi-level time domain feature extractor extracts features from the EEG time series signal to obtain spatiotemporal features; the frequency domain feature extractor extracts The detector extracts features from the spectrum to obtain frequency domain features; the spatiotemporal features and frequency domain features are input into the multi-scale feature fusion module for fusion at different scales to obtain multi-scale fusion features; the multi-scale fusion features are input into the classifier for category prediction to obtain the prediction results; the loss function is calculated according to the labels in the EEG time series signal and the prediction results, and the dual-branch EEG signal decoding model is trained based on the method of minimizing the loss function to obtain the trained dual-branch EEG signal decoding model; the category of the EEG signal to be classified is predicted by the trained dual-branch EEG signal decoding model to generate the classification results.

[0017] The dual-branch EEG signal decoding model constructed by the present invention can significantly improve the decoding performance of EEG signals by simultaneously extracting time and frequency spectrum features; wherein, the multi-level time domain feature extractor can perform multi-level feature extraction to adapt to the dynamic fluctuations of EEG signals at different time scales and frequencies, thereby improving the time feature extraction capability; the multi-scale feature fusion module can fuse spatiotemporal features with frequency domain features at different scales, thereby enhancing the interaction between time features and frequency spectrum features, and improving robustness and classification accuracy. In this way, the time domain signal and frequency domain information in the EEG signal can be better utilized, the decoding accuracy of the EEG signal can be improved, and the information transmission rate of the brain-computer interface system based on the EEG signal can be improved, which has important significance and practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0019] Figure 1 This is a diagram illustrating an application environment of a dual-branch EEG signal decoding method based on multi-scale feature fusion in one embodiment;

[0020] Figure 2 A schematic diagram of the structural framework of a dual-branch EEG signal decoding model in one embodiment;

[0021] Figure 3 Schematic diagram of the structural framework of a multi-level Convformer unit in one embodiment;

[0022] Figure 4 Schematic diagram of the structural framework of a multi-scale Convformer unit in one embodiment;

[0023] Figure 5 Schematic diagram of experimental results comparing average information transmission rates on dataset 1 in one embodiment;

[0024] Figure 6 Schematic diagram of experimental results comparing average information transmission rates on dataset 2 in one embodiment;

[0025] Figure 7 2 is a structural block diagram of a dual-branch EEG signal decoding device based on multi-scale feature fusion in one embodiment.

[0026] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0027] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0028] It can be understood that the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in this field can implement it. When the combination of technical solutions is mutually contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0029] The following describes the embodiments of the present invention in detail with reference to the accompanying drawings in which the embodiments of the present invention are shown.

[0030] Example 1

[0031] This embodiment discloses a dual-branch EEG signal decoding method based on multi-scale feature fusion, and designs a dual-branch EEG signal decoding model. By simultaneously extracting time and spectrum features, the decoding performance of EEG signals can be significantly improved. Among them, the multi-level time domain feature extractor can perform multi-level feature extraction to adapt to the dynamic fluctuations of EEG signals at different time scales and frequencies, and improve the time feature extraction capability. The multi-scale feature fusion module can fuse spatiotemporal features with frequency domain features at different scales, thereby enhancing the interaction between time features and spectrum features, and improving robustness and classification accuracy. In this way, the time domain signal and frequency domain information in the EEG signal can be better utilized, the decoding accuracy of the EEG signal can be improved, and the information transmission rate of the brain-computer interface system based on EEG signals can be improved, which has important significance and practical value.

[0032] It is worth noting that this embodiment is mainly described using SSVEP signals, but the method proposed in the present invention is also applicable to other EEG signals and is not limited to the SSVEP signals proposed in this application.

[0033] like Figure 1 As shown in FIG, the dual-branch EEG signal decoding method based on multi-scale feature fusion provided in this embodiment includes the following steps:

[0034] Step 201: Acquire a data set, where the data set includes an EEG time series signal and a frequency spectrum corresponding to the EEG time series signal.

[0035] Step 202: construct a dual-branch EEG signal decoding model, which includes a multi-level time domain feature extractor, a frequency domain feature extractor, a multi-scale feature fusion module and a classifier.

[0036] In step 203, the EEG time series signal and the corresponding spectrum are input into a dual-branch EEG signal decoding model as a pair of training data. The multi-level time domain feature extractor performs multi-level feature extraction on the EEG time series signal to obtain spatiotemporal features; the frequency domain feature extractor performs feature extraction on the spectrum to obtain frequency domain features; the spatiotemporal features and the frequency domain features are input into a multi-scale feature fusion module for fusion at different scales to obtain multi-scale fusion features; the multi-scale fusion features are input into a classifier for category prediction to obtain a prediction result.

[0037] Step 204 , calculating a loss function based on the labels and prediction results in the EEG time series signal, and training the dual-branch EEG signal decoding model based on a method of minimizing the loss function to obtain a trained dual-branch EEG signal decoding model.

[0038] Step 205 : Use the trained dual-branch EEG signal decoding model to perform category prediction on the EEG signal to be classified and generate a classification result.

[0039] In the specific real-time process of step 201, EEG signal data is acquired and preprocessed to obtain an EEG time series signal containing a label; then, a zero-filled fast Fourier transform is performed on the EEG time series signal to obtain a spectrum corresponding to the EEG time series signal; the EEG time series signal and the corresponding spectrum are used as a pair of training data; and multiple pairs of training data constitute a data set.

[0040] Specifically, two public SSVEP datasets containing 35 and 70 subjects respectively were obtained.

[0041] The original SSVEP dataset is preprocessed by filtering, segmenting, and other operations to obtain two sets of SSVEP time series signal samples with 40 categories of labels. ,in Indicates the time series signal samples, For the The labels corresponding to the samples, is the total number of EEG signal samples in the data set. For different groups of data, The values ​​are different, among which, , Indicates the number of channels of the signal, It is worth noting that in order to verify the performance improvement effect of the dual-branch EEG signal decoding model on SSVEP signals of different lengths, data of 0.5s, 0.6s, 0.7s, 0.8s, 0.9s, and 1.0s were used as the input SSVEP time series signals.

[0042] The SSVEP time series signal obtained after preprocessing is subjected to a zero-filled Fast Fourier Transform (FFT) to obtain the corresponding spectrum data. ,in Indicates the number of frequency components.

[0043] Construct a training data pair containing time series signal and spectrum for each group of SSVEP time series signal samples. , and use leave-one-out cross-validation to divide the sample pairs into training and test sets. Specifically, the data of one subject in the dataset is selected as the test set, and the data of the remaining subjects are selected as the training set. The model is trained on the training set, and the test set is used to test the trained model. In addition, during the training process, all training data will be shuffled to prevent the randomly generated training parameters of the model from affecting the experimental results.

[0044] In the specific implementation process of step 202, a dual-branch EEG signal decoding model is constructed, such as Figure 2 As shown in the figure, the dual-branch EEG signal decoding model includes a multi-level time domain feature extractor, a frequency domain feature extractor, a multi-scale feature fusion module and a classifier.

[0045] The multi-level time-domain feature extractor consists of a time-domain embedding network and a multi-level Convformer network. The time-domain embedding network performs a weighted combination of data from different channels of the EEG time-series signal to obtain weighted combined time-series data. The multi-level Convformer network then performs multi-level feature extraction on the weighted combined time-series data to obtain multi-level spatiotemporal features. The time-domain embedding network is primarily a one-dimensional convolutional neural network structure. It learns the importance weights of different channels by sliding across the time-domain data and performing a weighted combination of data from different channels, thereby enhancing the time-domain features that are crucial for classification.

[0046] The multi-level Convformer network is composed of multiple multi-level Convformer units with the same structure and stacked continuously, such as Figure 3 As shown in Figure 1, the multi-level Convformer unit includes a normalization layer, a multi-level convolution layer, a normalization layer, and a feedforward neural network connected in sequence. The multi-level convolution layer includes Layer and layer, where The layer is used to extract short-term features. Layers are used to learn long-term features based on short-term features. Multi-layer Convformer networks enable more complex feature learning and representation, gradually extracting and refining features. As you can see, in multi-layer Convformer units, both multi-layer convolutional layers and feedforward neural networks utilize residual connections and layer normalization techniques, resulting in better training results and higher performance.

[0047] The frequency domain feature extractor consists of a frequency domain embedding network and a convnet. The frequency domain embedding network performs a weighted combination of data from different spectral channels to generate weighted spectral data. The convnet then extracts features from this weighted spectral data to generate frequency domain features. The frequency domain embedding network is also a convolutional neural network. The convolution kernel slides across the frequency data to learn the importance weights of the frequency data from different channels and then performs a weighted combination to enhance the frequency features that are critical for classification.

[0048] The Convformer network is composed of multiple Convformer units with the same structure and stacked continuously. The Convformer unit includes a normalization layer, a convolution layer, a normalization layer, and a feedforward neural network connected in sequence. Through the Convformer network, the local features of the frequency domain data can be effectively captured, and the global features of the frequency domain data can be integrated to improve the efficiency and accuracy of feature extraction. It can be understood that in the Convformer network, the convolution layer and the feedforward neural network also use residual connections and layer normalization technology, which can achieve better training results and higher performance.

[0049] The multi-scale feature fusion module includes an initial fusion network and a multi-scale Convformer network. The initial fusion network concatenates and fuses spatiotemporal features with frequency-domain features along the feature dimension, effectively integrating the information advantages of spatiotemporal and frequency domains. The model can learn the relationship between spatiotemporal and frequency domain information, enhancing its understanding and representation of data.

[0050] The multi-scale Convformer network is composed of multiple multi-scale Convformer units with the same structure and stacked continuously. The multi-scale Convformer units are as follows: Figure 4 As shown in Figure 1, it includes a normalization layer, a multi-scale convolution layer, a normalization layer, and a feedforward neural network connected in sequence. The multi-scale convolution layer includes layer, Layer and Layers are used to extract fused features at different scales. The multi-scale Convformer network captures fused features at different scales, enriching the feature representation and thus enhancing the model's robustness and generalization. As can be seen, in the multi-scale Convformer network, both the multi-scale convolutional layers and the feedforward neural network utilize residual connections and layer normalization techniques, resulting in better training results and higher performance.

[0051] In the specific implementation process of step 203, in the multi-level time domain feature extractor, the time domain embedding network performs weighted combination on the input EEG time series signals to obtain the weighted combined time series data, which is expressed as:

[0052] ;

[0053] Where, Represents the time series data after weighted combination; represents a one-dimensional convolutional layer; Representation layer normalization function; represents the Gaussian error linear unit activation function.

[0054] The multi-level Convformer network includes multiple multi-level Convformer units with the same structure and stacked continuously. The weighted combined time series data are input into each multi-level Convformer unit for data processing.

[0055] In each multi-level Convformer unit, the weighted combined time series data After the input is processed by the normalization layer, it is then processed by the multi-level convolution layer. The short-term feature extraction is performed at the layer to obtain the short-term feature, which is expressed as:

[0056] ;

[0057] Will Short-term features of layer output enter , to extract long-term features, the expression is:

[0058] ;

[0059] In order to integrate the features extracted from different levels, the short-term features and long-term characteristics Perform addition merging to obtain time series features with multi-level information, expressed as:

[0060] ;

[0061] Then, the weighted combined time series data is connected by residual connection It is superimposed with the time series features with multi-level information to obtain the time series features that integrate multiple information.

[0062] After the multi-level convolutional layer, a feed-forward neural network (FFN) consisting of two fully connected layers and a GELU activation function is connected to enhance the model's perception and nonlinear learning capabilities.

[0063] After normalizing the time series features that integrate multiple information, the features are input into the feedforward neural network for enhancement, and the enhanced time series features that integrate multiple information are obtained. The expression is:

[0064] ;

[0065] Where, Indicates short-term characteristics; represents a one-dimensional convolutional layer with a medium convolution kernel; express The convolution kernel size of the layer; Indicates long-term characteristics; Represents a one-dimensional convolutional layer with a large convolution kernel; express The convolution kernel size of the layer; Represents temporal features with multi-level information; , and denote the weight and bias of the fully connected layer respectively.

[0066] By means of residual connection, the temporal features of fusion of multiple information are superimposed with the enhanced temporal features of fusion of multiple information to obtain the enhanced temporal features of each multi-level Convformer unit. Multiple multi-level Convformer units are used to enhance temporal features. The final spatiotemporal features are obtained by integration .

[0067] Extracting spatiotemporal features At the same time, the frequency domain feature extractor extracts the spectrum of SSVEP As input, the frequency domain embedding network is used to perform weighted combination of the spectrum data of each channel to obtain the weighted combined spectrum data. , and then get the weighted combined spectrum data Input the Convformer network. The data processing process of the Convformer network is similar to that of the multi-level Convformer network, so we will not go into details here.

[0068] In summary, the Convformer network extracts features from the weighted combined spectrum data to obtain frequency domain features, which are expressed as:

[0069] ;

[0070] Where, represents the extracted frequency domain features; Represents the Convformer unit; Represents the spectrum data after weighted combination.

[0071] The obtained spatiotemporal characteristics and frequency domain characteristics Input the multi-scale feature fusion module, and the initial fusion network splices the spatiotemporal features and frequency domain features along the feature dimension to obtain the initial fusion features, which are expressed as:

[0072] ;

[0073] In each multi-scale Convformer unit, the multi-scale convolution layer has three branches with small, medium and large convolution kernel sizes. 、 and , for the initial fusion features After normalization, it is input into the multi-scale Convformer unit for processing. In each multi-scale Convformer unit, 、 and Initial fusion features Perform multi-scale feature extraction to obtain fusion features of different scales. The expression is:

[0074] ;

[0075] ;

[0076] ;

[0077] The fusion features of different scales in the three branches are added and merged to obtain the time-frequency fusion features with multi-scale information, which is expressed as:

[0078] ;

[0079] Where, represents the initial fusion features; Represents spatiotemporal characteristics; represents the extracted frequency domain features; represents the splicing function; represents the Gaussian error linear unit activation function; represents a one-dimensional convolutional layer with a small convolution kernel; represents a one-dimensional convolutional layer with a medium convolution kernel; Represents a one-dimensional convolutional layer with a large convolution kernel; 、 and Respectively 、 and The convolution kernel of 、 and Represents fusion features of different scales.

[0080] By using residual connection, the initial fusion features Time-frequency fusion features with multi-scale information The superposition processing is performed to obtain the comprehensive time-frequency fusion features that integrate features at different levels.

[0081] The comprehensive time-frequency fusion features of features at different levels are normalized and then input into the feedforward neural network for enhancement processing to obtain enhanced comprehensive time-frequency fusion features.

[0082] Then, the comprehensive time-frequency fusion features and the enhanced comprehensive time-frequency fusion features are superimposed through residual connection to obtain the enhanced time-frequency fusion features of each multi-scale Convformer unit.

[0083] Finally, multiple multi-scale Convformer units perform enhanced time-frequency fusion feature integration to obtain the final multi-scale fusion features.

[0084] The final multi-scale fusion features are input into a classifier composed of a flatten layer and a multilayer perceptron (MLP) for category prediction. The category with the highest prediction probability is the prediction result of the sample.

[0085] In the specific implementation process of step 204, the loss function is calculated based on the label in the EEG time series signal and the obtained prediction result, and the expression is:

[0086] ;

[0087] Where, Indicates the number of training data samples in a batch; Indicates the number of categories; and Represent the label and predicted probability values ​​respectively.

[0088] The stochastic gradient descent (SGD) optimizer was then used to minimize the loss function of the two-branch EEG decoding model. The optimizer updates and adjusts model parameters during training to minimize the loss function, optimizing the parameters of the two-branch EEG decoding model. The batch size, epoch number, and learning rate were set to 64, 100, and 0.001, respectively. The model parameters were updated until convergence, resulting in a trained two-branch EEG decoding model.

[0089] During the specific implementation of step 205, the SSVEP signal to be classified in the test set is input into the trained dual-branch EEG signal decoding model, and the average classification accuracy (Accuracy) and information translation rate (ITR) of all subjects are calculated as performance evaluation indicators of the SSVEP signal recognition method. ITR refers to the amount of information transmitted by a single target selection. It is not only related to the classification accuracy, but also takes into account the number of categories and the speed of single target recognition. It is a comprehensive performance indicator for measuring the performance of the brain-computer interface system. The calculation formula of ITR is as follows:

[0090] ;

[0091] Where, Indicates the number of classification targets; Indicates the classification accuracy; Indicates the single target selection time.

[0092] In one embodiment, in order to verify the effectiveness of the method proposed in the present invention, the classification results of the method proposed in the present invention and deep learning methods such as EEGNet, SSVEPNet, and CCNN were statistically analyzed when the SSVEP signal lengths were 0.5s, 0.6s, 0.7s, 0.8s, 0.9s, and 1.0s, respectively. Figure 5 and Figure 6 The following table shows the average classification accuracy of all the compared methods on the two datasets.

[0093] Table 1 Average classification accuracy of different methods in comparative experiments

[0094]

[0095] The comparative test results show that the average classification accuracy and information transmission rate of the method proposed in this embodiment are superior to those of other comparative methods. Therefore, the dual-branch EEG signal decoding method based on multi-scale feature fusion proposed in this embodiment can better extract the time domain and frequency domain features in the EEG signal, and fully complement and fuse the time domain and frequency domain features, thereby improving the comprehensive representation ability of the fused features, thereby improving the model's decoding accuracy for EEG signals and increasing the system's information transmission rate. This has important significance and practical value for promoting the practical application of brain-computer interfaces.

[0096] Although this embodiment Figure 1 The steps in the diagram are shown in the order indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0097] Example 2

[0098] Based on the dual-branch EEG signal decoding method based on multi-scale feature fusion in Example 1, this embodiment discloses a dual-branch EEG signal decoding device based on multi-scale feature fusion, such as Figure 7 As shown, the dual-branch EEG signal decoding device based on multi-scale feature fusion includes: a data acquisition module 401, a model construction module 402, a data processing module 403, a model training module 404 and a classification module 405, wherein:

[0099] The data acquisition module 401 is used to acquire a data set, which includes an EEG time series signal and a frequency spectrum corresponding to the EEG time series signal.

[0100] The model construction module 402 is used to construct a dual-branch EEG signal decoding model, which includes a multi-level time domain feature extractor, a frequency domain feature extractor, a multi-scale feature fusion module and a classifier.

[0101] The data processing module 403 is used to input the EEG time series signal and the corresponding spectrum as a pair of training data into the dual-branch EEG signal decoding model, and the multi-level time domain feature extractor performs multi-level feature extraction on the EEG time series signal to obtain spatiotemporal features; the frequency domain feature extractor performs feature extraction on the spectrum to obtain frequency domain features; the spatiotemporal features and the frequency domain features are input into the multi-scale feature fusion module for fusion at different scales to obtain multi-scale fusion features; the multi-scale fusion features are input into the classifier for category prediction to obtain the prediction results.

[0102] The model training module 404 calculates a loss function based on the labels and prediction results in the EEG time series signal, and trains the dual-branch EEG signal decoding model based on a method of minimizing the loss function to obtain a trained dual-branch EEG signal decoding model.

[0103] The classification module 405 is used to perform category prediction on the EEG signal to be classified using the trained dual-branch EEG signal decoding model to generate a classification result.

[0104] In this embodiment, the specific working process and working principle of the data acquisition module 401, the model construction module 402, the data processing module 403, the model training module 404 and the classification module 405 are the same as those in the method of Example 1, and therefore are not described in detail in this embodiment. Each unit module can be implemented in whole or in part by software, hardware or a combination thereof. Each unit module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above unit modules.

[0105] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0106] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0107] The above-described embodiments merely illustrate several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that a person skilled in the art would be able to make numerous modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A dual-branch EEG signal decoding method based on multi-scale feature fusion, characterized in that: The method comprises: Acquire a data set, the data set including an EEG time series signal and a frequency spectrum corresponding to the EEG time series signal; Constructing a dual-branch EEG signal decoding model, wherein the dual-branch EEG signal decoding model includes a multi-level time domain feature extractor, a frequency domain feature extractor, a multi-scale feature fusion module and a classifier; The EEG time series signal and the corresponding spectrum are input into the dual-branch EEG signal decoding model as a pair of training data. The multi-level time domain feature extractor performs multi-level feature extraction on the EEG time series signal to obtain spatiotemporal features. The frequency domain feature extractor performs feature extraction on the spectrum to obtain frequency domain features. The spatiotemporal features and the frequency domain features are input into the multi-scale feature fusion module for fusion at different scales to obtain multi-scale fusion features. The multi-scale fusion features are input into the classifier for category prediction to obtain a prediction result. Calculating a loss function based on the label in the EEG time series signal and the prediction result, and training the dual-branch EEG signal decoding model in a manner of minimizing the loss function to obtain a trained dual-branch EEG signal decoding model; Performing category prediction on the EEG signal to be classified using the trained dual-branch EEG signal decoding model to generate a classification result; The multi-scale feature fusion module includes an initial fusion network and a multi-scale Convformer network; the multi-scale Convformer network includes a plurality of multi-scale Convformer units with the same structure and stacked continuously; The spatiotemporal features and the frequency domain features are spliced ​​along the feature dimension through the initial fusion network to obtain the initial fusion features, which are expressed as follows: ； In each multi-scale Convformer unit, the initial fusion features are normalized and then multi-scale feature extraction is performed to obtain fusion features of different scales, which are expressed as: ； ； ； The fusion features of different scales are added and merged to obtain the time-frequency fusion features with multi-scale information, which is expressed as: ； The initial fusion feature is superimposed on the time-frequency fusion feature with multi-scale information by residual connection to obtain a comprehensive time-frequency fusion feature that integrates features at different levels; After normalizing the comprehensive time-frequency fusion features of features at different levels, they are input into the feedforward neural network for enhancement processing to obtain enhanced comprehensive time-frequency fusion features; Through the residual connection method, the comprehensive time-frequency fusion features and the enhanced comprehensive time-frequency fusion features are superimposed to obtain the enhanced time-frequency fusion features of each multi-scale Convformer unit; Multiple multi-scale Convformer units perform enhanced time-frequency fusion feature integration to obtain the final multi-scale fusion feature; Where, represents the initial fusion features; Represents spatiotemporal characteristics; represents the extracted frequency domain features; represents the splicing function; represents the Gaussian error linear unit activation function; represents a one-dimensional convolutional layer with a small convolution kernel; represents a one-dimensional convolutional layer with a medium convolution kernel; Represents a one-dimensional convolutional layer with a large convolution kernel; 、 and Respectively 、 and The convolution kernel of 、 and Represents fusion features of different scales.

2. The dual-branch EEG signal decoding method based on multi-scale feature fusion according to claim 1 is characterized in that: Get the dataset, including: Acquire EEG signal data, preprocess the EEG signal data, and obtain an EEG time series signal containing a label; Performing a zero-filled fast Fourier transform on the EEG time series signal to obtain a frequency spectrum corresponding to the EEG time series signal; The EEG time series signal and the corresponding spectrum are used as a pair of training data; multiple pairs of training data constitute the data set.

3. The dual-branch EEG signal decoding method based on multi-scale feature fusion according to claim 1 is characterized in that: The multi-level time domain feature extractor includes a time domain embedding network and a multi-level Convformer network; Performing weighted combination on the data of different channels of the EEG time series signal through the time domain embedding network to obtain weighted combined time series data; The multi-level Convformer network is used to perform multi-level feature extraction on the time series data after weighted combination to obtain multi-level spatiotemporal features.

4. The dual-branch EEG signal decoding method based on multi-scale feature fusion according to claim 3 is characterized in that: The time domain embedding network is used to perform weighted combination on the data of different channels of the EEG time series signal to obtain the weighted combined time series data, which is expressed as: ； Where, Represents the time series data after weighted combination; represents a one-dimensional convolutional layer; Representation layer normalization function; represents the Gaussian error linear unit activation function.

5. The dual-branch EEG signal decoding method based on multi-scale feature fusion according to claim 4 is characterized in that: The multi-level Convformer network is used to extract multi-level features from the weighted combined time series data to obtain multi-level spatiotemporal features, including: The multi-level Convformer network includes a plurality of multi-level Convformer units with the same structure and stacked continuously; In each multi-level Convformer unit, the time series data after weighted combination is normalized and then short-term feature extraction is performed to obtain short-term features, which are expressed as: ； The long-term features are extracted based on the short-term features, and the expression is: ； The short-term feature and the long-term feature are added and merged to obtain a time series feature with multi-level information, which is expressed as: ； The weighted combined time series data is superimposed on the time series features with multi-level information by using a residual connection method to obtain a time series feature that integrates multiple types of information; After normalizing the time series features that fuse multiple information, the features are input into a feedforward neural network for enhancement, thereby obtaining enhanced time series features that fuse multiple information, which can be expressed as follows: ； By means of residual connection, the temporal features of fusion of multiple information are superimposed on the enhanced temporal features of fusion of multiple information to obtain the enhanced temporal features of each multi-level Convformer unit. ; Multiple multi-level Convformer units to enhance temporal features The final spatiotemporal features are obtained by integration ; Where, Indicates short-term characteristics; represents a one-dimensional convolutional layer with a medium convolution kernel; express The convolution kernel size of the layer; Indicates long-term characteristics; Represents a one-dimensional convolutional layer with a large convolution kernel; express The convolution kernel size of the layer; Represents temporal features with multi-level information; , and denote the weight and bias of the fully connected layer respectively.

6. The dual-branch EEG signal decoding method based on multi-scale feature fusion according to claim 1, characterized in that: The frequency domain feature extractor includes a frequency domain embedding network and a Convformer network; Performing weighted combination on the data of different channels of the spectrum through the frequency domain embedding network to obtain weighted combined spectrum data; The Convformer network is used to extract features from the weighted combined spectrum data to obtain frequency domain features.

7. The dual-branch EEG signal decoding method based on multi-scale feature fusion according to claim 6, characterized in that: The Convformer network is used to extract features from the weighted combined spectrum data to obtain frequency domain features, which are expressed as follows: ； Where, represents the extracted frequency domain features; Represents the Convformer unit; Represents the spectrum data after weighted combination.

8. The dual-branch EEG signal decoding method based on multi-scale feature fusion according to any one of claims 1 to 7, characterized in that: The loss function is calculated based on the label in the EEG time series signal and the prediction result, and the expression is: ； Where, is the number of training data samples in a batch, is the number of categories, and are the labels and predicted probability values ​​respectively.

9. A dual-branch EEG signal decoding device based on multi-scale feature fusion, characterized in that: The dual-branch EEG signal decoding method based on multi-scale feature fusion according to any one of claims 1 to 8 is adopted, wherein the device comprises: A data acquisition module, configured to acquire a data set, wherein the data set includes an EEG time series signal and a frequency spectrum corresponding to the EEG time series signal; A model construction module is used to construct a dual-branch EEG signal decoding model, which includes a multi-level time domain feature extractor, a frequency domain feature extractor, a multi-scale feature fusion module and a classifier; A model training module is used to input the EEG time series signal and the corresponding spectrum as a pair of training data into the dual-branch EEG signal decoding model, the multi-level time domain feature extractor performs multi-level feature extraction on the EEG time series signal to obtain spatiotemporal features; the frequency domain feature extractor performs feature extraction on the spectrum to obtain frequency domain features; the spatiotemporal features and the frequency domain features are input into the multi-scale feature fusion module for fusion at different scales to obtain multi-scale fusion features; the multi-scale fusion features are input into the classifier for category prediction to obtain a prediction result; a loss function is calculated based on the label in the EEG time series signal and the prediction result, and the dual-branch EEG signal decoding model is trained based on minimizing the loss function to obtain a trained dual-branch EEG signal decoding model; The classification module is used to predict the category of the EEG signal to be classified using the trained dual-branch EEG signal decoding model to generate a classification result.

Images