Double-branch electroencephalogram signal decoding method and device based on multi-scale feature fusion

Abstract

The invention relates to a double-branch electroencephalogram signal decoding method and device based on multi-scale feature fusion. The method comprises the following steps: acquiring an electroencephalogram time sequence signal and a corresponding frequency spectrum as a pair of training data; constructing a double-branch electroencephalogram signal decoding model, wherein the double-branch electroencephalogram signal decoding model comprises a multi-level time domain feature extractor, a frequency domain feature extractor, a multi-scale feature fusion module and a classifier; inputting the training data into the double-branch electroencephalogram signal decoding model to obtain a prediction result; calculating a loss function according to a label in the electroencephalogram time sequence signal and a prediction result, and training a double-branch electroencephalogram signal decoding model based on a mode of minimizing the loss function to obtain a trained double-branch electroencephalogram signal decoding model; and performing category prediction on the to-be-classified electroencephalogram signals through the trained double-branch electroencephalogram signal decoding model to generate a classification result. According to the method, time domain signals and frequency domain information in the electroencephalogram signals can be better utilized, and the decoding precision of the electroencephalogram signals is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of electroencephalogram (EEG) signal decoding, and in particular to a dual-branch EEG signal decoding method and device based on multi-scale feature fusion. Background Art

[0002] Brain Computer Interface (BCI) technology can quickly and directly convert brain signals into commands to control external devices without going through peripheral nerves and muscle tissue, thus realizing information transmission between the brain and the outside world. Electroencephalogram (EEG) has become the main mode for realizing BCI systems due to its non-invasiveness, high temporal resolution and convenience. Steady State Visual Evoked Potential (SSVEP) is an electroencephalogram signal generated by continuous visual stimulation of a specific frequency. The brain-computer interface designed based on SSVEP has been widely used in application fields with high requirements for information transmission due to its advantages such as obvious frequency domain characteristics, a large number of selectable targets, fast information transmission rate and no need for training of subjects. Therefore, improving the recognition accuracy of SSVEP signals and the information transmission rate remains the research focus of BCI systems based on SSVEP signals.

[0003] In recent years, due to the powerful feature representation ability and flexibility of deep learning, SSVEP signal decoding methods based on deep learning have been widely studied. In existing studies, since the signal characteristics of SSVEP signals in the frequency domain are more significant, in addition to directly extracting features from the time series signals of SSVEP, the SSVEP time domain signals are usually converted to the frequency domain for feature extraction, and then combined with the powerful automatic feature extraction ability of neural networks to further improve the performance of SSVEP signal classification. In addition, in order to obtain more comprehensive representation information, time-frequency analysis methods, such as short-time Fourier transform, continuous wavelet transform, and discrete wavelet transform, are also widely used to extract time-frequency features from SSVEP signals. However, most of the existing methods only focus on extracting different features from time domain or frequency domain signals in sequence, or simply splicing time domain and frequency domain features, ignoring the dependency between different types of features, which may reduce the comprehensive representation ability of fused features. Summary of the invention

[0004] Based on this, it is necessary to provide a dual-branch EEG signal decoding method and device based on multi-scale feature fusion that can solve the problem of low SSVEP signal decoding accuracy in response to the above technical problems.

[0005] A dual-branch electroencephalogram signal decoding method based on multi-scale feature fusion, the method comprising: Acquire a data set, the data set including an EEG time series signal and a frequency spectrum corresponding to the EEG time series signal; Constructing a dual-branch EEG signal decoding model, wherein the dual-branch EEG signal decoding model includes a multi-level time domain feature extractor, a frequency domain feature extractor, a multi-scale feature fusion module and a classifier; The EEG time series signal and the corresponding spectrum are input into the dual-branch EEG signal decoding model as a pair of training data; the multi-level time domain feature extractor performs multi-level feature extraction on the EEG time series signal to obtain spatiotemporal features; the frequency domain feature extractor performs feature extraction on the spectrum to obtain frequency domain features; the spatiotemporal features and the frequency domain features are input into the multi-scale feature fusion module to perform fusion at different scales to obtain multi-scale fusion features; the multi-scale fusion features are input into the classifier to perform category prediction to obtain prediction results; Calculating a loss function according to the label in the EEG time series signal and the prediction result, and training the dual-branch EEG signal decoding model in a manner of minimizing the loss function to obtain a trained dual-branch EEG signal decoding model; The trained dual-branch EEG signal decoding model is used to perform category prediction on the EEG signal to be classified to generate a classification result.

[0006] A dual-branch electroencephalogram signal decoding device based on multi-scale feature fusion, the device comprising: A data acquisition module, used to acquire a data set, wherein the data set includes an EEG time series signal and a frequency spectrum corresponding to the EEG time series signal; A model building module, used to build a dual-branch EEG signal decoding model, wherein the dual-branch EEG signal decoding model includes a multi-level time domain feature extractor, a frequency domain feature extractor, a multi-scale feature fusion module and a classifier; A model training module is used to input the EEG time series signal and the corresponding spectrum as a pair of training data into the dual-branch EEG signal decoding model, the multi-level time domain feature extractor performs multi-level feature extraction on the EEG time series signal to obtain spatiotemporal features; the frequency domain feature extractor performs feature extraction on the spectrum to obtain frequency domain features; the spatiotemporal features and the frequency domain features are input into the multi-scale feature fusion module for fusion at different scales to obtain multi-scale fusion features; the multi-scale fusion features are input into the classifier for category prediction to obtain a prediction result; a loss function is calculated based on the label in the EEG time series signal and the prediction result, and the dual-branch EEG signal decoding model is trained based on minimizing the loss function to obtain a trained dual-branch EEG signal decoding model; The classification module is used to predict the category of the EEG signal to be classified by using the trained dual-branch EEG signal decoding model to generate a classification result.

[0007] The above-mentioned dual-branch EEG signal decoding method and device based on multi-scale feature fusion first obtains a data set, which includes an EEG time series signal and a spectrum corresponding to the EEG time series signal; constructs a dual-branch EEG signal decoding model, which includes a multi-level time domain feature extractor, a frequency domain feature extractor, a multi-scale feature fusion module and a classifier; the EEG time series signal and the corresponding spectrum are input into the dual-branch EEG signal decoding model as a pair of training data, and the multi-level time domain feature extractor extracts features from the EEG time series signal to obtain spatiotemporal features; the frequency domain feature extractor extracts The detector extracts features from the spectrum to obtain frequency domain features; the spatiotemporal features and frequency domain features are input into the multi-scale feature fusion module for fusion at different scales to obtain multi-scale fusion features; the multi-scale fusion features are input into the classifier for category prediction to obtain prediction results; the loss function is calculated according to the labels in the EEG time series signal and the prediction results, and the dual-branch EEG signal decoding model is trained based on the method of minimizing the loss function to obtain the trained dual-branch EEG signal decoding model; the category of the EEG signal to be classified is predicted by the trained dual-branch EEG signal decoding model to generate the classification result.

[0008] The dual-branch EEG signal decoding model constructed by the present invention can significantly improve the decoding performance of EEG signals by extracting time and frequency spectrum features simultaneously; wherein, the multi-level time domain feature extractor can perform multi-level feature extraction to adapt to the dynamic fluctuations of EEG signals at different time scales and frequencies, and improve the time feature extraction capability; the multi-scale feature fusion module can fuse spatiotemporal features with frequency domain features at different scales, thereby enhancing the interaction between time features and frequency spectrum features, and improving robustness and classification accuracy. In this way, the time domain signal and frequency domain information in the EEG signal can be better utilized, the decoding accuracy of the EEG signal can be improved, and thus the information transmission rate of the brain-computer interface system based on the EEG signal can be improved, which has important significance and practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0010] Figure 1 This is a diagram of an application environment of a dual-branch EEG signal decoding method based on multi-scale feature fusion in one embodiment; Figure 2 A schematic diagram of the structural framework of a dual-branch EEG signal decoding model in one embodiment; Figure 3 A schematic diagram of the structural framework of a multi-level Convformer unit in one embodiment; Figure 4 Schematic diagram of the structural framework of a multi-scale Convformer unit in one embodiment; Figure 5 It is a schematic diagram of experimental results of comparing average information transmission rates on data set 1 in one embodiment; Figure 6 It is a schematic diagram of experimental results of comparing average information transmission rates on data set 2 in one embodiment; Figure 7 1 is a structural block diagram of a dual-branch EEG signal decoding device based on multi-scale feature fusion in one embodiment.

[0011] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0012] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0013] It can be understood that the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0014] The following will describe the implementation of the present invention in detail with reference to the accompanying drawings in the embodiment diagram of the present invention.

[0015] Example 1 This embodiment discloses a dual-branch EEG signal decoding method based on multi-scale feature fusion, and designs a dual-branch EEG signal decoding model, which can significantly improve the decoding performance of EEG signals by extracting time and spectrum features simultaneously; wherein, the multi-level time domain feature extractor can perform multi-level feature extraction to adapt to the dynamic fluctuations of EEG signals at different time scales and frequencies, and improve the ability to extract time features; the multi-scale feature fusion module can fuse spatiotemporal features with frequency domain features at different scales, thereby enhancing the interaction between time features and spectrum features, and improving robustness and classification accuracy. In this way, the time domain signal and frequency domain information in the EEG signal can be better utilized, the decoding accuracy of the EEG signal can be improved, and the information transmission rate of the brain-computer interface system based on the EEG signal can be improved, which has important significance and practical value.

[0016] It is worth noting that this embodiment is mainly described using SSVEP signals, but the method proposed in the present invention is also applicable to other EEG signals and is not limited to the SSVEP signals proposed in this application.

[0017] like Figure 1 As shown, the dual-branch EEG signal decoding method based on multi-scale feature fusion provided in this embodiment includes the following steps: Step 201 : Acquire a data set, where the data set includes an EEG time series signal and a frequency spectrum corresponding to the EEG time series signal.

[0018] Step 202, constructing a dual-branch EEG signal decoding model, which includes a multi-level time domain feature extractor, a frequency domain feature extractor, a multi-scale feature fusion module and a classifier.

[0019] Step 203, the EEG time series signal and the corresponding spectrum are input into the dual-branch EEG signal decoding model as a pair of training data, the multi-level time domain feature extractor performs multi-level feature extraction on the EEG time series signal to obtain time-space features; the frequency domain feature extractor performs feature extraction on the spectrum to obtain frequency domain features; the time-space features and the frequency domain features are input into the multi-scale feature fusion module for fusion at different scales to obtain multi-scale fusion features; the multi-scale fusion features are input into the classifier for category prediction to obtain prediction results.

[0020] Step 204, calculating the loss function according to the labels and prediction results in the EEG time series signal, and training the dual-branch EEG signal decoding model based on a method of minimizing the loss function to obtain a trained dual-branch EEG signal decoding model.

[0021] Step 205 , using the trained dual-branch EEG signal decoding model to perform category prediction on the EEG signal to be classified, and generate a classification result.

[0022] In the specific real-time process of step 201, EEG signal data is acquired, and the EEG signal data is preprocessed to obtain an EEG time series signal containing a label; then, a zero-filled fast Fourier transform is performed on the EEG time series signal to obtain a frequency spectrum corresponding to the EEG time series signal; the EEG time series signal and the corresponding frequency spectrum are used as a pair of training data; and multiple pairs of training data constitute a data set.

[0023] Specifically, two public SSVEP datasets containing 35 and 70 subjects respectively were obtained.

[0024] The original SSVEP data set is preprocessed by filtering, segmenting, and other operations to obtain two sets of SSVEP time series signal samples with 40 categories of labels. ,in Indicates time series signal samples, For the The labels corresponding to the samples are is the total number of EEG signal samples in the data set. For different groups of data, The values ​​are different, among which, , Indicates the number of channels of the signal, It is worth noting that in order to verify the performance improvement effect of the dual-branch EEG signal decoding model on SSVEP signals of different lengths, data of 0.5s, 0.6s, 0.7s, 0.8s, 0.9s, and 1.0s were used as the input SSVEP time series signals.

[0025] The SSVEP time series signal obtained after preprocessing is subjected to a zero-filled Fast Fourier Transform (FFT) to obtain the corresponding spectrum data. ,in Indicates the number of frequency components.

[0026] For each group of SSVEP time series signal samples, a training data pair containing time series signals and spectrum is constructed. , and the leave-one-out cross-validation method is used to divide the sample pairs into training sets and test sets. Specifically, the data of one subject in the data set is selected as the test set, and the data of the remaining subjects are selected as the training set. The model is trained with the training set, and the test set is used to test the trained model. In addition, during the training process, in order to avoid the training parameters randomly generated by the model affecting the experimental results, all training data will be shuffled.

[0027] In the specific implementation process of step 202, a dual-branch EEG signal decoding model is constructed, such as Figure 2As shown, the dual-branch EEG signal decoding model includes a multi-level time domain feature extractor, a frequency domain feature extractor, a multi-scale feature fusion module and a classifier.

[0028] The multi-level time-domain feature extractor includes a time-domain embedding network and a multi-level Convformer network. The time-domain embedding network performs weighted combination of data from different channels of the EEG time series signal to obtain the weighted combined time series data; the multi-level Convformer network performs multi-level feature extraction on the weighted combined time series data to obtain multi-level spatiotemporal features. Among them, the time-domain embedding network is mainly a one-dimensional convolutional neural network structure, which learns the importance weights of different channels by sliding on the time-domain data, and performs weighted combination of data from different channels, thereby enhancing the time-domain features that are critical to classification.

[0029] The multi-level Convformer network consists of multiple multi-level Convformer units with the same structure and stacked continuously, such as Figure 3 As shown in the figure, the multi-level Convformer unit includes a normalization layer, a multi-level convolution layer, a normalization layer and a feedforward neural network connected in sequence. The multi-level convolution layer includes Layer and Layer, where The layer is used to extract short-term features. The layer is used to learn long-term features based on short-term features. Through the multi-level Convformer network, more complex feature learning and representation can be performed, and features can be gradually extracted and refined. It can be understood that in the multi-level Convformer unit, the multi-level convolutional layer and the feedforward neural network both use residual connections and layer normalization technology, which can achieve better training results and higher performance.

[0030] The frequency domain feature extractor includes a frequency domain embedding network and a Convformer network. The frequency domain embedding network performs weighted combination of data of different channels of the spectrum to obtain the weighted combined spectrum data; the Convformer network performs feature extraction on the weighted combined spectrum data to obtain frequency domain features. The frequency domain embedding network is also a convolutional neural network. The convolution kernel slides on the frequency data to learn the importance weights of the frequency data of different channels, and performs weighted combination on them to enhance the frequency features that are critical to classification.

[0031] The Convformer network is composed of multiple Convformer units with the same structure and stacked continuously. The Convformer unit includes a normalization layer, a convolution layer, a normalization layer and a feedforward neural network connected in sequence. Through the Convformer network, the local features of the frequency domain data can be effectively captured, and the global features of the frequency domain data can be integrated to improve the efficiency and accuracy of feature extraction. It can be understood that in the Convformer network, the convolution layer and the feedforward neural network also use residual connections and layer normalization technology, which can achieve better training results and higher performance.

[0032] The multi-scale feature fusion module includes the initial fusion network and the multi-scale Convformer network. The initial fusion network combines spatiotemporal features with frequency domain features along the feature dimension, which can effectively integrate the information advantages in spatiotemporal and frequency domains. The model can learn the relationship between spatiotemporal and frequency domain information, and enhance the model's ability to understand and represent data.

[0033] The multi-scale Convformer network is composed of multiple multi-scale Convformer units with the same structure and stacked continuously. The multi-scale Convformer units are as follows: Figure 4 As shown in Figure 1, it includes a normalization layer, a multi-scale convolution layer, a normalization layer, and a feedforward neural network connected in sequence. The multi-scale convolution layer includes layer, Layer and Layers are used to extract fusion features of different scales. Through the multi-scale Convformer network, it is possible to capture fusion features at different scales and enrich feature representation, thereby enhancing the robustness and generalization ability of the model. It can be understood that in the multi-scale Convformer network, both the multi-scale convolutional layer and the feedforward neural network use residual connections and layer normalization technology, which can achieve better training results and higher performance.

[0034] In the specific implementation process of step 203, in the multi-level time domain feature extractor, the time domain embedding network performs weighted combination on the input EEG time series signals to obtain the weighted combined time series data, which is expressed as: ; In the formula, represents the time series data after weighted combination; represents a one-dimensional convolutional layer; Representation layer normalization function; represents the Gaussian error linear unit activation function.

[0035] The multi-level Convformer network includes multiple multi-level Convformer units with the same structure and stacked continuously. The weighted combined time series data is input into each multi-level Convformer unit for data processing.

[0036] In each multi-level Convformer unit, the weighted combined time series data After the input is processed by the normalization layer, it is then processed by the multi-level convolution layer. The short-term feature extraction is performed at the layer to obtain the short-term feature, which is expressed as: ; Will Short-term features of layer output enter , to extract long-term features, the expression is: ; In order to integrate the features extracted from different levels, the short-term features With long-term characteristics Additive merging is performed to obtain time series features with multi-level information, expressed as: ; Then, the weighted combined time series data is connected by residual connection. The time series features with multi-level information are superimposed to obtain the time series features that integrate multiple information.

[0037] After the multi-level convolutional layer, a feed-forward neural network (FFN) consisting of two fully connected layers and a GELU activation function is connected to enhance the perception and nonlinear learning capabilities of the model.

[0038] After normalizing the time series features that integrate multiple information, they are input into the feedforward neural network for enhancement processing to obtain enhanced time series features that integrate multiple information. The expression is: ; In the formula, Indicates short-term characteristics; represents a one-dimensional convolutional layer with a medium convolution kernel; express The convolution kernel size of the layer; Indicates long-term characteristics; represents a one-dimensional convolutional layer with a large convolution kernel; express The convolution kernel size of the layer; Represents temporal features with multi-level information; , and They represent the weight and bias of the fully connected layer respectively.

[0039] Through the residual connection method, the time series features that integrate multiple information are superimposed with the enhanced time series features that integrate multiple information to obtain the enhanced time series features of each multi-level Convformer unit. Multiple multi-level Convformer units are used to enhance temporal features. The final spatiotemporal features are obtained by integrating .

[0040] Extracting spatiotemporal features At the same time, the frequency domain feature extractor converts the spectrum of SSVEP As input, the frequency spectrum data of each channel is first weighted and combined through the frequency domain embedding network to obtain the weighted combined spectrum data , and then get the weighted combined spectrum data Input the Convformer network. The data processing process of the Convformer network is similar to that of the multi-level Convformer network, which will not be described here.

[0041] In summary, the Convformer network extracts features from the weighted combined spectrum data to obtain frequency domain features, which are expressed as: ; In the formula, represents the extracted frequency domain features; Represents the Convformer unit; Represents the spectral data after weighted combination.

[0042] The obtained spatiotemporal characteristics Frequency domain characteristics Input the multi-scale feature fusion module, the initial fusion network splices the spatiotemporal features and frequency domain features along the feature dimension to obtain the initial fusion features, which are expressed as: ; In each multi-scale Convformer unit, the multi-scale convolution layer has three branches with small, medium and large convolution kernel sizes. , and , for the initial fusion features After normalization, it is input into the multi-scale Convformer unit for processing. In each multi-scale Convformer unit, , and Initial fusion features Perform multi-scale feature extraction to obtain fusion features of different scales. The expression is: ; ; ; The fusion features of different scales in the three branches are added and merged to obtain the time-frequency fusion features with multi-scale information, which is expressed as: ; In the formula, represents the initial fusion features; Represents spatiotemporal characteristics; represents the extracted frequency domain features; represents the concatenation function; represents the Gaussian error linear unit activation function; represents a one-dimensional convolutional layer with a small convolution kernel; represents a one-dimensional convolutional layer with a medium convolution kernel; represents a one-dimensional convolutional layer with a large convolution kernel; , and Respectively , and The convolution kernel of , and Represents fusion features of different scales.

[0043] Through the residual connection method, the initial fusion features Time-frequency fusion features with multi-scale information The superposition processing is performed to obtain the comprehensive time-frequency fusion features that integrate the features of different levels.

[0044] The comprehensive time-frequency fusion features of features at different levels are normalized and then input into the feedforward neural network for enhancement processing to obtain enhanced comprehensive time-frequency fusion features.

[0045] Then, through the residual connection method, the comprehensive time-frequency fusion features and the enhanced comprehensive time-frequency fusion features are superimposed to obtain the enhanced time-frequency fusion features of each multi-scale Convformer unit.

[0046] Finally, multiple multi-scale Convformer units perform enhanced time-frequency fusion feature integration to obtain the final multi-scale fusion features.

[0047] The final multi-scale fusion features are input into a classifier composed of a flatten layer and a multilayer perceptron (MLP) for category prediction. The category with the highest prediction probability is the prediction result of the sample.

[0048] In the specific implementation process of step 204, the loss function is calculated according to the label in the EEG time series signal and the obtained prediction result, and the expression is: ; In the formula, Indicates the number of training data samples in a batch; Indicates the number of categories; and Represent the label and predicted probability value respectively.

[0049] Then, the Stochastic Gradient Descent (SGD) optimizer is used to minimize the loss function of the dual-branch EEG signal decoding model. The optimizer can update and adjust the model parameters during the training process to minimize the loss function and achieve parameter optimization of the dual-branch EEG signal decoding model. The batch size during the training process is set to 64, the epoch is 100, and the learning rate is 0.001. The model parameters are updated until the model converges to obtain a trained dual-branch EEG signal decoding model.

[0050] In the specific implementation process of step 205, the SSVEP signal to be classified in the test set is input into the trained dual-branch EEG signal decoding model, and the average classification accuracy (Accuracy) and information translation rate (ITR) of all subjects are calculated as the performance evaluation index of the SSVEP signal recognition method. ITR refers to the amount of information transmitted by a single target selection, which is not only related to the classification accuracy, but also takes into account the number of categories and the single target recognition speed. It is a comprehensive performance indicator for measuring the performance of the brain-computer interface system. The calculation formula of ITR is as follows: ; In the formula, Indicates the number of classification targets; Indicates classification accuracy; Indicates the single target selection time.

[0051] In one embodiment, in order to verify the effectiveness of the method proposed in the present invention, the classification results of the method proposed in the present invention and deep learning methods such as EEGNet, SSVEPNet, and CCNN were statistically analyzed when the SSVEP signal lengths were 0.5s, 0.6s, 0.7s, 0.8s, 0.9s, and 1.0s, respectively. Figure 5 and Figure 6 The table below shows the average classification accuracy of all the compared methods on the two datasets.

[0052] Table 1. Average classification accuracy of different methods in comparative experiments

[0053] From the results of the comparative test, it can be seen that the average classification accuracy and information transmission rate of the method proposed in this embodiment are better than those of other comparative methods. Therefore, the dual-branch EEG signal decoding method based on multi-scale feature fusion proposed in this embodiment can better extract the time domain and frequency domain features in the EEG signal, and fully complement and fuse the time domain and frequency domain features, improve the comprehensive representation ability of the fusion features, thereby improving the model's decoding accuracy for EEG signals and increasing the system's information transmission rate, which is of great significance and practical value for promoting the practical application of brain-computer interfaces.

[0054] Although this embodiment Figure 1 The steps in the process are shown in sequence as indicated by the arrows, but the steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of the steps, and the steps can be executed in other orders. Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0055] Example 2 Based on the dual-branch EEG signal decoding method based on multi-scale feature fusion in Example 1, this embodiment discloses a dual-branch EEG signal decoding device based on multi-scale feature fusion, such as Figure 7 As shown, the dual-branch EEG signal decoding device based on multi-scale feature fusion includes: a data acquisition module 401, a model construction module 402, a data processing module 403, a model training module 404 and a classification module 405, wherein: The data acquisition module 401 is used to acquire a data set, which includes an EEG time series signal and a frequency spectrum corresponding to the EEG time series signal.

[0056] The model building module 402 is used to build a dual-branch EEG signal decoding model, which includes a multi-level time domain feature extractor, a frequency domain feature extractor, a multi-scale feature fusion module and a classifier.

[0057] The data processing module 403 is used to input the EEG time series signal and the corresponding spectrum as a pair of training data into the dual-branch EEG signal decoding model; the multi-level time domain feature extractor performs multi-level feature extraction on the EEG time series signal to obtain time-space features; the frequency domain feature extractor performs feature extraction on the spectrum to obtain frequency domain features; the time-space features and the frequency domain features are input into the multi-scale feature fusion module for fusion at different scales to obtain multi-scale fusion features; the multi-scale fusion features are input into the classifier for category prediction to obtain prediction results.

[0058] The model training module 404 calculates the loss function according to the labels and prediction results in the EEG time series signal, and trains the dual-branch EEG signal decoding model based on the method of minimizing the loss function to obtain a trained dual-branch EEG signal decoding model.

[0059] The classification module 405 is used to perform category prediction on the EEG signal to be classified by using the trained dual-branch EEG signal decoding model to generate a classification result.

[0060] In this embodiment, the specific working process and working principle of the data acquisition module 401, the model building module 402, the data processing module 403, the model training module 404 and the classification module 405 are the same as those in the method of Embodiment 1, so they are not described in detail in this embodiment. Each unit module can be implemented in whole or in part by software, hardware and a combination thereof, and each unit module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above unit modules.

[0061] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0062] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0063] The above-mentioned embodiments only express several implementation modes of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the attached claims.

Claims

1. A dual-branch EEG signal decoding method based on multi-scale feature fusion, characterized in that: The method comprises: Acquire a data set, the data set including an EEG time series signal and a frequency spectrum corresponding to the EEG time series signal; Constructing a dual-branch EEG signal decoding model, wherein the dual-branch EEG signal decoding model includes a multi-level time domain feature extractor, a frequency domain feature extractor, a multi-scale feature fusion module and a classifier; The EEG time series signal and the corresponding spectrum are input into the dual-branch EEG signal decoding model as a pair of training data; the multi-level time domain feature extractor performs multi-level feature extraction on the EEG time series signal to obtain spatiotemporal features; the frequency domain feature extractor performs feature extraction on the spectrum to obtain frequency domain features; the spatiotemporal features and the frequency domain features are input into the multi-scale feature fusion module to perform fusion at different scales to obtain multi-scale fusion features; the multi-scale fusion features are input into the classifier to perform category prediction to obtain prediction results; Calculating a loss function according to the label in the EEG time series signal and the prediction result, and training the dual-branch EEG signal decoding model in a manner of minimizing the loss function to obtain a trained dual-branch EEG signal decoding model; The trained dual-branch EEG signal decoding model is used to perform category prediction on the EEG signal to be classified to generate a classification result.

2. The dual-branch EEG signal decoding method based on multi-scale feature fusion according to claim 1, characterized in that: Get the dataset, including: Acquire EEG signal data, and preprocess the EEG signal data to obtain an EEG time series signal containing a label; Performing a zero-filled fast Fourier transform on the EEG time series signal to obtain a frequency spectrum corresponding to the EEG time series signal; The EEG time series signal and the corresponding frequency spectrum are used as a pair of training data; multiple pairs of training data constitute the data set.

3. The dual-branch EEG signal decoding method based on multi-scale feature fusion according to claim 1 is characterized in that: The multi-level time-domain feature extractor includes a time-domain embedding network and a multi-level Convformer network; Performing weighted combination on the data of different channels of the EEG time series signal through the time domain embedding network to obtain weighted combined time series data; The multi-level Convformer network is used to perform multi-level feature extraction on the time series data after weighted combination to obtain multi-level spatiotemporal features.

4. The dual-branch EEG signal decoding method based on multi-scale feature fusion according to claim 3 is characterized in that: The data of different channels of the EEG time series signal are weighted combined by the time domain embedding network to obtain the weighted combined time series data, which is expressed as: ； In the formula, represents the time series data after weighted combination; represents a one-dimensional convolutional layer; Representation layer normalization function; represents the Gaussian error linear unit activation function.

5. The dual-branch EEG signal decoding method based on multi-scale feature fusion according to claim 4 is characterized in that: The multi-level Convformer network is used to extract multi-level features from the weighted combined time series data to obtain multi-level spatiotemporal features, including: The multi-level Convformer network includes a plurality of multi-level Convformer units with the same structure and stacked continuously; In each multi-level Convformer unit, the time series data after weighted combination is normalized and then short-term feature extraction is performed to obtain short-term features, which are expressed as: ； Based on the short-term features, long-term features are extracted, and the expression is: ； The short-term feature and the long-term feature are added and combined to obtain a time series feature with multi-level information, which is expressed as: ； The weighted combined time series data and the time series features with multi-level information are superimposed on each other by residual connection to obtain a time series feature integrating multiple information; After the time series features integrating multiple information are normalized, they are input into a feedforward neural network for enhancement processing to obtain enhanced time series features integrating multiple information, which are expressed as follows: ； By means of residual connection, the temporal features of fusion of multiple information are superimposed on the enhanced temporal features of fusion of multiple information to obtain the enhanced temporal features of each multi-level Convformer unit. ; Multiple multi-level Convformer units to enhance temporal features The final spatiotemporal features are obtained by integrating ; In the formula, Indicates short-term characteristics; represents a one-dimensional convolutional layer with a medium convolution kernel; express The convolution kernel size of the layer; Indicates long-term characteristics; represents a one-dimensional convolutional layer with a large convolution kernel; express The convolution kernel size of the layer; Represents temporal features with multi-level information; , and They represent the weight and bias of the fully connected layer respectively.

6. The dual-branch EEG signal decoding method based on multi-scale feature fusion according to claim 1, characterized in that: The frequency domain feature extractor includes a frequency domain embedding network and a Convformer network; Performing weighted combination on the data of different channels of the spectrum through the frequency domain embedding network to obtain weighted combined spectrum data; The Convformer network is used to extract features from the weighted combined spectrum data to obtain frequency domain features.

7. The dual-branch EEG signal decoding method based on multi-scale feature fusion according to claim 6, characterized in that: The Convformer network is used to extract the features of the weighted combined spectrum data to obtain frequency domain features, which are expressed as follows: ； In the formula, represents the extracted frequency domain features; Represents the Convformer unit; Represents the spectral data after weighted combination.

8. The dual-branch EEG signal decoding method based on multi-scale feature fusion according to any one of claims 1 to 7, characterized in that: The multi-scale feature fusion module includes an initial fusion network and a multi-scale Convformer network; the multi-scale Convformer network includes a plurality of multi-scale Convformer units with the same structure and stacked continuously; The spatiotemporal features and the frequency domain features are concatenated along the feature dimension through the initial fusion network to obtain the initial fusion features, which are expressed as follows: ； In each multi-scale Convformer unit, the initial fusion features are normalized and then multi-scale feature extraction is performed to obtain fusion features of different scales, which are expressed as: ； ； ； The fusion features of different scales are added and merged to obtain the time-frequency fusion features with multi-scale information, which is expressed as: ； The initial fusion feature is superimposed on the time-frequency fusion feature with multi-scale information by residual connection to obtain a comprehensive time-frequency fusion feature that integrates features at different levels; After the comprehensive time-frequency fusion features of features at different levels are normalized, they are input into the feedforward neural network for enhancement processing to obtain enhanced comprehensive time-frequency fusion features; Through residual connection, the comprehensive time-frequency fusion features and the enhanced comprehensive time-frequency fusion features are superimposed to obtain the enhanced time-frequency fusion features of each multi-scale Convformer unit; Multiple multi-scale Convformer units perform enhanced time-frequency fusion feature integration to obtain the final multi-scale fusion feature; In the formula, represents the initial fusion features; Represents spatiotemporal characteristics; represents the extracted frequency domain features; represents the concatenation function; represents the Gaussian error linear unit activation function; represents a one-dimensional convolutional layer with a small convolution kernel; represents a one-dimensional convolutional layer with a medium convolution kernel; represents a one-dimensional convolutional layer with a large convolution kernel; , and Respectively , and The convolution kernel of , and Represents fusion features of different scales.

9. The dual-branch EEG signal decoding method based on multi-scale feature fusion according to claim 8, characterized in that: The loss function is calculated based on the label in the EEG time series signal and the prediction result, and the expression is: ； In the formula, is the number of training data samples in a batch, is the number of categories, and are the labels and predicted probability values ​​respectively.

10. A dual-branch EEG signal decoding device based on multi-scale feature fusion, characterized in that: The device comprises: A data acquisition module, used to acquire a data set, wherein the data set includes an EEG time series signal and a frequency spectrum corresponding to the EEG time series signal; A model building module, used to build a dual-branch EEG signal decoding model, wherein the dual-branch EEG signal decoding model includes a multi-level time domain feature extractor, a frequency domain feature extractor, a multi-scale feature fusion module and a classifier; A model training module is used to input the EEG time series signal and the corresponding spectrum as a pair of training data into the dual-branch EEG signal decoding model, the multi-level time domain feature extractor performs multi-level feature extraction on the EEG time series signal to obtain spatiotemporal features; the frequency domain feature extractor performs feature extraction on the spectrum to obtain frequency domain features; the spatiotemporal features and the frequency domain features are input into the multi-scale feature fusion module for fusion at different scales to obtain multi-scale fusion features; the multi-scale fusion features are input into the classifier for category prediction to obtain a prediction result; a loss function is calculated based on the label in the EEG time series signal and the prediction result, and the dual-branch EEG signal decoding model is trained based on minimizing the loss function to obtain a trained dual-branch EEG signal decoding model; The classification module is used to predict the category of the EEG signal to be classified by using the trained dual-branch EEG signal decoding model to generate a classification result.

Images