Motor imagery electroencephalogram signal classification method based on multi-scale time sequence fusion

Abstract

The invention relates to a motor imagery electroencephalogram signal classification method based on multi-scale time sequence fusion. The method comprises the following steps: S1, standardizing electroencephalogram signals before inputting a network model; s2, inputting the processed data into a multi-scale channel attention convolution module, wherein the multi-scale channel attention convolution module focuses on capturing low-level spatial-temporal characteristics with discrimination from original motor imagery electroencephalogram signals (MI-EEG); s3, segmenting the electroencephalogram signal into a plurality of local time sequence segments by sliding the window; s4, inputting the data output by the sliding window into the time fusion residual network in parallel so as to further extract advanced time features from the time sequence; and S5, fusing the high-order time sequence characteristics of all windows through a full connection layer, and outputting probability prediction of a motor imagery task through a Softmax function. The method can effectively solve the problems that convolution scale division of the motor imagery network is limited, and features of all channels and advanced time features are ignored, and the average classification accuracy of the motor imagery electroencephalogram signals is improved.

Description

Technical Field

[0001] The present invention belongs to the field of electroencephalogram (EEG) signal processing and pattern recognition, and specifically provides a motor imagery EEG signal classification method based on multi-scale temporal fusion. Background Art

[0002] The statements in this section only provide background information related to the present disclosure, and these statements may constitute prior art. In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art.

[0003] A brain-computer interface (BCI) is an advanced system that establishes a direct communication bridge between the brain and external devices by interpreting and analyzing electroencephalogram (EEG) signals. This system captures weak electrical signals from the brain, reflecting thoughts and emotions, and, through algorithmic processing, converts these signals into commands or information, enabling control or interaction with external devices. BCI technology, due to its innovative neural encoding and decoding capabilities, is considered a key future direction in science and technology. In neurorehabilitation (such as motor rehabilitation and speech rehabilitation), BCI technology can identify patients' motor imagery intentions and guide them to actively imagine physical movements, thereby aiding their recovery. In education, cognitive and affective BCIs can provide insights into brain mechanisms, offering the potential to improve learning strategies and enhance brain-based functions. By monitoring brain activity in real time, BCI technology can provide a more solid empirical foundation for teaching methods. For example, it can dynamically adjust learning content based on students' brain workload, gauge their interest in a topic, and even help them maintain focus on specific tasks. With continued technological advancements, BCI technology also holds great promise in robotic control and military applications.

[0004] Based on the different signal detection technologies, BCI systems are mainly divided into two categories: invasive BCI and non-invasive BCI. Invasive BCI systems surgically implant electrodes directly into the cerebral cortex, which can directly record or stimulate brain neuronal activity. Non-invasive BCI technology collects signals from outside the brain through non-surgical methods. Common technologies include electroencephalography (EEG), magnetoencephalography (MEG), and functional near-infrared spectroscopy (FNIRS). With the development of BCI technology, EEG-based BCI can be divided into different research directions, such as motor imagery (MI), event-related potentials (ERP), and steady-state visual evoked potentials (SSVEP). Among them, MI, as one of the important applications of BCI technology, has attracted the attention of many scholars.

[0005] In recent years, with the rapid development of deep learning technology, many deep learning-based classification methods have been proposed and applied to EEG signal classification tasks, especially in the field of motor imagery EEG classification research. For example, the patent application number 202211336463.6, entitled "A motor imagery EEG signal classification method, system, and electronic device based on deep learning", uses a convolutional neural network model to process the original EEG data and the spatiotemporal attention features of the EEG data separately, and fuses and classifies the processing results to obtain the final classification results. However, most of these technologies use a fixed convolution scale, the extraction of MI-EEG features is limited, and the characteristics of each channel and the high-level temporal features are ignored, resulting in low recognition accuracy.

[0006] In order to solve the limitation of single-scale feature extraction, the patent application number 202411596275.6, entitled "A method for classifying motor imagery EEG signals based on multi-scale feature extraction and adaptive feature fusion", uses a multi-scale feature extraction (MFE) module to extract multi-scale time domain information and channel space information of EEG signals. However, the applicant's in-depth research found that due to the differences in individual brains, the patent's subsequent use of an adaptive feature fusion (AFF) module to fuse global and local feature information, and finally completing the classification task through a classifier module, cannot effectively solve the problems of low recognition rate and weak generalization ability of the motor imagery EEG signal recognition model, and the motor imagery recognition accuracy is still not ideal. Summary of the Invention

[0007] In view of the above problems, the present invention aims to solve some of the problems in the prior art, or at least alleviate these problems.

[0008] A motor imagery EEG signal classification method based on multi-scale temporal fusion includes the following steps:

[0009] Normalize the original EEG signal to obtain a standardized EEG signal;

[0010] The normalized EEG signal is input into the multi-scale channel attention convolution module to capture the discriminative low-level spatiotemporal features and obtain the time series Z i ;

[0011] Use sliding window to transform the time series Z i Split into multiple time series segments;

[0012] The data output by the sliding window is input in parallel into the time fusion residual module to further extract high-order temporal features from the time series;

[0013] The high-order temporal features of all windows are fused through the fully connected layer, and the probability prediction of the motor imagery task is output through the Softmax function.

[0014] The specific formula of the normalized EEG signal is as follows:

[0015]

[0016] Among them, x i represents the original input EEG signal data, mean and std represent the channel mean and standard deviation parameters calculated in all experiments, respectively, and C represents the number of EEG channels.

[0017] The multi-scale channel attention convolution module includes a multi-scale convolution module and a channel attention module;

[0018] The multi-scale convolution module contains three independent convolution branches, namely 1×15, 1×31 and 1×63 two-dimensional convolution kernels of three different scales, which process the input signal independently in parallel. The 1×15 small kernel branch focuses on local detail features, the 1×31 medium kernel branch captures relatively coarse local features, and the 1×63 large kernel branch extracts global features. The feature maps output by each branch are spliced together in the channel dimension for multi-scale fusion, ultimately forming a comprehensive feature representation with a rich hierarchical structure.

[0019] The channel attention mechanism of the channel attention module includes the following steps:

[0020] Global average pooling (GAP) is used to compress spatial information and generate global statistical features of the channel;

[0021] After the GAP layer, a new feature map is learned by using adaptive one-dimensional convolution (1D).

[0022] Furthermore, each channel and its k neighbors (i.e., the coverage of local cross-channel interactions of kernel size k in 1D convolution) are considered to perform local cross-channel exchange information, and its adaptation function is defined as follows:

[0023]

[0024] Where k is the convolution kernel size, It expresses the linear mapping between the number of channels C and k, and γ and b are used to change the ratio of channels and convolution kernels.

[0025] Furthermore, the multi-scale channel attention convolution module also includes spatial convolution and spatiotemporal convolution, and batch normalization and activation layers are added after both convolutions.

[0026] The sliding window transforms the time series Z i Divide into multiple local sequences In order to extract a single local feature; where T ω Represents the sliding window length, d is the dimension; the length of the sliding window T ω Determined by the following formula:

[0027]

[0028] Where T c represents the total length of the time series after pooling, n is the number of sliding windows, T is the time point of the original EEG signal, and P2 is the pooling size.

[0029] The temporal fusion residual module includes two temporal convolution blocks, namely a temporal convolution residual block and a temporal fusion convolution block; the RFS (receptive field) of the temporal convolution block is 19, and the time series of the input temporal convolution block is less than or equal to 19; the receptive field calculation formula includes:

[0030] RFS=1+2(K T -1)(2 L -1)

[0031] The size of the receptive field RFS is controlled by two parameters, namely the number of residual blocks L and the kernel K T size.

[0032] The high-order temporal features of all windows are fused through the fully connected layer, and the probability prediction of the motor imagery task is output through the Softmax function, which also includes evaluation using accuracy and Kappa score.

[0033] Furthermore, the formula for evaluation using accuracy is:

[0034]

[0035] Where: TP is the true positive sample, that is, the number of samples that are actually positive samples that are correctly predicted; TN is the true negative sample, that is, the number of samples that are actually negative samples that are correctly predicted; FP is the false positive sample, that is, the number of samples that are actually negative samples that are incorrectly predicted; FN is the false negative sample, that is, the number of samples that are actually positive samples that are incorrectly predicted;

[0036] The formula for evaluation using the Kappa score is:

[0037]

[0038] In the formula, p o It is the sum of the number of correctly classified samples in each category divided by the total number of samples, that is, the overall classification accuracy, p e is the accuracy of random guessing.

[0039] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the motor imagery EEG signal classification method based on multi-scale temporal fusion.

[0040] The present invention has the following beneficial effects:

[0041] The present invention proposes a multi-scale attention convolution method, which studies the in-depth expression and comprehensive extraction of relevant features of motor imagery EEG signals. Multi-scale convolution is adopted to solve the limitation problem of single-scale feature extraction, and the efficient attention mechanism is added to optimize the channel weights. The advanced time extraction method combined with time-fused residual convolution can effectively extract high-level time features in time series. Finally, the high-order time series features of all windows are fused through the fully connected layer, and the probability prediction of the motor imagery task is output through the SoftMax function. The model proposed by the present invention can effectively solve the problems of low recognition rate and weak generalization ability of the motor imagery EEG signal recognition model caused by differences in individual brains, and improve the accuracy of motor imagery recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 The present invention provides a framework diagram of a method for classifying motor imagery EEG signals based on multi-scale temporal fusion;

[0043] Figure 2 This is a multi-scale attention convolutional network framework diagram for extracting primary temporal features in the present invention;

[0044] Figure 3 This is a detailed framework diagram of the multi-scale convolution of the present invention;

[0045] Figure 4 This is the framework diagram of the efficient attention mechanism of the present invention;

[0046] Figure 5This is a diagram of the time fusion residual network framework used by the present invention to extract high-level time features. DETAILED DESCRIPTION

[0047] The present invention will be further described below in conjunction with the accompanying drawings. The embodiments of the present invention are only used to illustrate the present invention and are not intended to limit the present invention. Without departing from the technical concept of the present invention, various substitutions and modifications can be made based on common technical knowledge and customary means in the field, and all should be included in the scope of the present invention.

[0048] In order to solve the above technical problems, Figure 1 As shown, the present invention provides a motor imagery EEG signal classification method based on multi-scale time series fusion, comprising the following steps:

[0049] S1: Normalize the original EEG signal to obtain a standardized EEG signal that is easy to process;

[0050] S2: The EEG signal normalized in step S1 is input into the multi-scale channel attention convolution module, which focuses on capturing discriminative low-level spatiotemporal features from the original motor imagery EEG signal (MI-EEG) to obtain the time series Z i ;

[0051] S3: Use sliding window to transform the time series Z i Split into multiple time series segments to achieve the purpose of data enhancement;

[0052] S4: The data output by the sliding window is input in parallel into the time fusion residual module to further extract high-order temporal features from the time series;

[0053] S5: The high-order temporal features of all windows are fused through the fully connected layer, and the probability prediction of the motor imagery task is output through the Softmax function.

[0054] The specific formula for the standardization in step S1 is as follows:

[0055]

[0056] Among them, x i represents the original input EEG signal data, mean and std represent the channel mean and standard deviation parameters calculated in all experiments, respectively, and C represents the number of EEG channels.

[0057] The multi-scale channel attention convolution module of step S2 is as follows: Figure 2 As shown in the figure, it mainly includes a multi-scale convolution module and a channel attention module.

[0058] (1) The multi-scale convolution module contains three independent convolution branches, such as Figure 3As shown in the figure, three different scales of two-dimensional convolution kernels are used: 1×15, 1×31, and 1×63, forming a gradually expanding receptive field. Convolutions of different scales independently process the input signal in parallel. The small 1×15 kernel branch focuses on local details, the medium 1×31 kernel branch captures relatively coarse local features, and the large 1×63 kernel branch extracts global features. The feature maps output by each branch are concatenated across the channel dimension for multi-scale fusion, ultimately forming a comprehensive feature representation with a rich hierarchical structure. This multi-scale fusion mechanism enables the model to adaptively integrate spatiotemporal features of different granularities, effectively expanding the model's receptive range while maintaining local feature accuracy.

[0059] (2) The channel attention mechanism includes the following steps:

[0060] like Figure 4 As shown in the figure, the input feature map first compresses the spatial dimension through global average pooling (GAP) to generate a 1×1×C channel descriptor; then it passes through a one-dimensional convolution layer (the kernel size k is dynamically determined by the adaptive function) to capture local cross-channel interactions, and generates channel attention weights through the Sigmoid activation function; finally, the weights are multiplied by the original input feature map channel by channel, and the adjusted feature map is output to achieve channel-level feature enhancement. Its adaptive function is defined as follows:

[0061]

[0062] Where k is the convolution kernel size, It expresses the linear mapping between the number of channels C and k, and γ and b are used to change the ratio of channels and convolution kernels.

[0063] like Figure 2 As shown in the figure, an efficient channel attention mechanism is embedded after multi-scale convolution to improve the network's ability to learn EEG signal channel information. Batch normalization and activation layers are added after both spatial and spatiotemporal convolutions, which effectively accelerates training, improves convergence speed, and enhances the model's nonlinear fitting ability and robustness.

[0064] like Figure 1 As shown, step S3 uses a sliding window to convert the time series Z i Divide into multiple local sequences In order to extract a single local feature, T ω Indicates the window length, d is the dimension. The length of the sliding window is T ω Determined by the following formula:

[0065]

[0066] Where T crepresents the total length of the time series after pooling, n is the number of sliding windows, T is the time point of the original EEG signal, and P2 is the pooling size.

[0067] Step S4: The time fusion model residual module includes the following specific steps: Figure 5 As shown in the figure, this module consists of a temporal convolution residual block and a temporal fusion convolution block. The input signal (x) first passes through three consecutive dilated causal convolution layers to gradually expand the temporal receptive field and extract temporal features. Subsequently, the features after the third processing are connected to the intermediate features after the first dilated convolution through a residual connection, forming a cross-layer interaction of the temporal fusion module. Finally, the fusion result is combined with the original input signal (x) through a residual connection and the output is activated through the ELU. The entire module captures long-term and short-term dependencies through dilated causal convolution and uses the residual structure to retain the original information, achieving efficient fusion and stable transmission of temporal features. The receptive field calculation formulas used include:

[0068] RFS=1+2(K T -1)(2 L -1)

[0069] The size of the receptive field RFS is controlled by two parameters: the number of residual blocks L and the kernel K T size.

[0070] In step S5, the high-order temporal features of all windows are fused through a fully connected layer, and the probability prediction of the motor imagery task is output through a Softmax function. Finally, the present invention uses accuracy and Kappa coefficient as evaluation criteria.

[0071]

[0072] Where: TP is the true positive sample, that is, the number of samples that are actually positive samples that are correctly predicted; TN is the true negative sample, that is, the number of samples that are actually negative samples that are correctly predicted; FP is the false positive sample, that is, the number of samples that are actually negative samples that are incorrectly predicted; FN is the false negative sample, that is, the number of samples that are actually positive samples that are incorrectly predicted.

[0073]

[0074] In the formula, p o It is the sum of the number of correctly classified samples in each category divided by the total number of samples, that is, the overall classification accuracy, p e is the accuracy of random guessing.

[0075] Experimental verification:

[0076] The proposed model achieved an average classification accuracy of 87.85% and a Kappa coefficient of 0.8380 in the BCI IV-2a simulation experiment, and an average classification accuracy of 92.85% and a Kappa coefficient of 0.8571 in the BCI IV-2b simulation experiment, which are both improvements over the existing benchmark models.

[0077] The main innovations of this invention are concentrated in the two parts S2 and S4. By rationally using the multi-scale channel attention convolution module and the time fusion residual module, it effectively solves the problem of limited scale division of the motor imagery network convolution and ignores the characteristics of each channel and high-level temporal features, thereby improving the average classification accuracy of motor imagery EEG signals.

[0078] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the motor imagery EEG signal classification method based on multi-scale temporal fusion.

[0079] The methods described in the above embodiments may be implemented by a computer chip or entity, or by a product with certain functions.

[0080] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0081] The above embodiments should be understood as merely illustrating the present invention and not as limiting the scope of protection of the present invention. After reading the contents of the present invention, technicians may make various changes or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.

Claims

1. A motor imagery EEG signal classification method based on multi-scale temporal fusion, characterized in that: The following steps are involved: Normalize the original EEG signal to obtain a standardized EEG signal; The normalized EEG signal is input into the multi-scale channel attention convolution module to capture the discriminative low-level spatiotemporal features and obtain the time series Z i ; Use sliding window to transform the time series Z i Split into multiple time series segments; The data output by the sliding window is input in parallel into the time fusion residual module to further extract high-order temporal features from the time series; The high-order temporal features of all windows are fused through the fully connected layer, and the probability prediction of the motor imagery task is output through the Softmax function.

2. The motor imagery EEG signal classification method based on multi-scale temporal fusion according to claim 1 is characterized in that: The specific formula of the normalized EEG signal is as follows: Among them, x i represents the original input EEG signal data, mean and std represent the channel mean and standard deviation parameters calculated in all experiments, respectively, and C represents the number of EEG channels.

3. The motor imagery EEG signal classification method based on multi-scale temporal fusion according to claim 1 is characterized in that: The multi-scale channel attention convolution module includes a multi-scale convolution module and a channel attention module; The multi-scale convolution module contains three independent convolution branches, namely 1×15, 1×31 and 1×63 two-dimensional convolution kernels of three different scales, which process the input signal independently in parallel. The 1×15 small kernel branch focuses on local detail features, the 1×31 medium kernel branch captures relatively coarse local features, and the 1×63 large kernel branch extracts global features. The feature maps output by each branch are spliced together in the channel dimension for multi-scale fusion, ultimately forming a comprehensive feature representation with a rich hierarchical structure; The channel attention mechanism of the channel attention module includes the following steps: Global average pooling (GAP) is used to compress spatial information and generate global statistical features of the channel; After the GAP layer, a new feature map is learned by using adaptive one-dimensional convolution (1D).

4. The method for classifying motor imagery EEG signals based on multi-scale temporal fusion according to claim 3, characterized in that: Considering each channel and its k neighbors (i.e., the coverage of local cross-channel interactions of kernel size k in 1D convolution) to perform local cross-channel exchange information, its adaptation function is defined as follows: Where k is the convolution kernel size, It expresses the linear mapping between the number of channels C and k, and γ and b are used to change the ratio of channels and convolution kernels.

5. The method for classifying motor imagery EEG signals based on multi-scale temporal fusion according to claim 3 or 4, characterized in that: The multi-scale channel attention convolution module also includes spatial convolution and spatiotemporal convolution, and batch normalization and activation layers are added after both convolutions.

6. The motor imagery EEG signal classification method based on multi-scale temporal fusion according to claim 1, characterized in that: The sliding window transforms the time series Z i Divide into multiple local sequences In order to extract a single local feature; where T ω Represents the sliding window length, d is the dimension; the length of the sliding window T ω Determined by the following formula: T ω =T c -n+1,T c >n≥1 Where T c represents the total length of the time series after pooling, n is the number of sliding windows, T is the time point of the original EEG signal, and P2 is the pooling size.

7. The method for classifying motor imagery EEG signals based on multi-scale temporal fusion according to claim 1, characterized in that: The temporal fusion residual module includes two temporal convolution blocks, namely a temporal convolution residual block and a temporal fusion convolution block; the RFS (receptive field) of the temporal convolution block is 19, and the time series of the input temporal convolution block is less than or equal to 19; the receptive field calculation formula includes: RFS=1+2(K T -1)(2 L -1) The size of the receptive field RFS is controlled by two parameters, namely the number of residual blocks L and the kernel K T size.

8. The method for classifying motor imagery EEG signals based on multi-scale temporal fusion according to claim 1, characterized in that: The high-order temporal features of all windows are fused through the fully connected layer, and the probability prediction of the motor imagery task is output through the Softmax function, which also includes evaluation using accuracy and Kappa score.

9. The method for classifying motor imagery EEG signals based on multi-scale temporal fusion according to claim 8, characterized in that: The formula for evaluation using accuracy is: Where: TP is the true positive sample, that is, the number of samples that are actually positive samples that are correctly predicted; TN is the true negative sample, that is, the number of samples that are actually negative samples that are correctly predicted; FP is the false positive sample, that is, the number of samples that are actually negative samples that are incorrectly predicted; FN is the false negative sample, that is, the number of samples that are actually positive samples that are incorrectly predicted; The formula for evaluation using the Kappa coefficient is: In the formula, p o It is the sum of the number of correctly classified samples in each category divided by the total number of samples, that is, the overall classification accuracy, p e is the accuracy of random guessing.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for classifying motor imagery EEG signals based on multi-scale temporal fusion according to any one of claims 1 to 9 are implemented.

Images