Efficient electroencephalogram signal recognition method based on time sparse convolution reservation network

Abstract

The invention discloses an efficient EEG (electroencephalogram) recognition method based on a time sparse convolutional retention network, relates to the technical field of EEG recognition, and mainly provides a convolutional retention network and the time sparse convolutional retention network. The convolution reservation network is composed of a convolution module, a cascade bidirectional reservation module and a classifier module. A time sparse convolutional retention network is further provided to accelerate the reasoning speed, an adaptive time token sparse mechanism is introduced into a bidirectional retention module, and the weight of each EEG time token is calculated by using a lightweight network. By preventing the token with lower weight from interacting with other tokens, the sparsification of the EEG time tokens is realized, and the network parallel training is supported at the same time. According to the method, the EEG explicit time priori can be reserved, the redundant time token can be removed, and the good balance between the EEG reasoning speed and accuracy is achieved.

Description

TECHNICAL FIELD

[0001] The application relates to an efficient electroencephalogram signal recognition method based on a time sparse convolution reservation network and belongs to the technical field of electroencephalogram signal recognition. BACKGROUND

[0002] Electroencephalogram (EEG) has the advantages of high time resolution and non-invasiveness and is widely used in brain-computer interface (BCI). BCI based on EEG signals can directly communicate between the brain and external devices without relying on muscle control, which makes its value in the field of neural rehabilitation engineering increasingly prominent.

[0003] At present, deep learning methods based on convolutional neural networks and Transformers have made significant progress in electroencephalogram decoding, but there are still two key challenges. First, these methods usually ignore the explicit temporal prior in EEG signals, i.e., the dependence between EEG signal time segments tends to decay as the relative distance between them increases, which is crucial for EEG temporal representation learning; second, the useful information region in EEG signal time features is usually sparse, i.e., the final prediction usually depends on the time feature region with rich information. Effectively identifying and focusing on these unstructured time feature regions is crucial for accelerating the inference of EEG signal recognition models.

[0004] To solve the above problems, the application provides an efficient electroencephalogram decoding method based on a time sparse convolution reservation network. First, the application develops a convolution reservation network, CRNet, to capture local features, global correlations and temporal prior information of EEG signals; then, a time sparse convolution reservation network, TS-CRNet, is proposed to further accelerate the inference. By introducing an adaptive time token sparsification (TTS) mechanism in the bidirectional reservation module, redundant EEG time tokens in the bidirectional reservation module are adaptively discarded. The TTS mechanism inserts a lightweight network between two consecutive bidirectional reservation modules to calculate the importance score of each EEG time token, facilitating the removal of redundant tokens. In addition, the application further proposes a reservation masking technique that prevents tokens with low importance scores from interacting with other tokens, achieving redundant token removal while maintaining hardware efficiency. During inference, a fixed number of uninformative tokens can be discarded for each input instance, significantly improving inference efficiency. SUMMARY

[0005] The application provides an efficient electroencephalogram (EEG) signal recognition method based on a time sparse convolution reservation network, develops a convolution reservation network, can capture local features, global correlation and time prior information of an EEG signal, on the basis, proposes a time sparse convolution reservation network, uses an adaptive time token sparse mechanism, calculates an importance score of each EEG time token, simultaneously combines a reservation mask technology to prevent interaction of tokens with low importance scores and other tokens, ensures that the token sparse process is friendly to hardware, supports parallel training, and realizes balance between calculation efficiency and inference accuracy.

[0006] The application adopts the following technical solutions to solve the above problems:

[0007] An efficient electroencephalogram (EEG) classification method based on a time sparse convolution reservation network comprises the following steps:

[0008] Step 1: divide the collected EEG data set into a training set and a test set Wherein y i ∈{1,2,…,C} respectively represent the i-th EEG data and the corresponding label. In addition, t and ch respectively represent the number of EEG data time points and the number of channels.

[0009] Step 2: pre-process the EEG data, including intercepting a specific time period and band-pass filtering, to obtain pre-processed training and test EEG data.

[0010] Step 3: for the training EEG data obtained in step 2, a convolution module composed of a series of time and space convolution operations is constructed to learn local EEG features.

[0011] Step 4: for the local EEG features obtained in step 3, a cascade bidirectional reservation module composed of a series of bidirectional reservation modules is constructed to learn global correlation information of the EEG features, and an attenuation matrix is introduced for each bidirectional reservation module to capture time prior information of the EEG features.

[0012] Step 5: for the EEG features obtained in step 4, a simple classification module composed of an average pooling and a full connection layer is constructed, and a cross-entropy loss function is defined to optimize the parameters of the convolution reservation network.

[0013] Step 6: to speed up the inference speed of the convolution reservation network, an adaptive time token sparse mechanism is designed to generate local-global EEG features according to the EEG tokens output by the bidirectional reservation module in step 4. At the same time, a lightweight network is constructed to calculate the sparse binary mask of the token, and Gumbel-Softmax is used to realize end-to-end optimization of the lightweight network.

[0014] Step 7: To achieve parallel training, a preservation masking technique is designed. According to the sparse binary mask in step 6, redundant tokens are removed while maintaining hardware efficiency by preventing them from interacting with other tokens.

[0015] Step 8: During inference, redundant EEG time tokens are discarded based on the probability predictions generated by the lightweight network, and predictions on the test EEG data are implemented using a temporally sparse convolutional preserving network.

[0016] Preferably, the convolution module is composed of a series of temporal and spatial convolution operations for learning local EEG features. For the pre-processed training EEG data X, the specific representation is as follows:

[0017] X t =Conv(X,T) (1)

[0018] X s =Conv(X t ,S) (2)

[0019]

[0020] Where T represents the time convolution kernel of size (1,ker), X t is the feature after time convolution. s represents the spatial convolution kernel of size (ch, 1), X s is the feature after spatial convolution. represents local EEG features, BN(·), GELU(·) and AP(·) represent batch normalization, Gaussian error linear unit activation function and average pooling, respectively.

[0021] Preferably, the cascaded bidirectional retention module is composed of a plurality of bidirectional retention modules stacked together to learn EEG global related information and time prior information. The bidirectional reservation mechanism is expressed as follows:

[0022]

[0023] Where, The nth part Θ in Θ n =e in θ , attenuation matrix D B The (n, m)th element D in B，nm =γ |n-m| ⊙ represents element-wise multiplication, and γ = 1-2 -5 , is the complex conjugate of Θ.

[0024] The bidirectional reservation module with multi-scale bidirectional reservation mechanism has multiple heads, which are specifically represented as follows:

[0025] MSBR = [head1, head2, …, head H ] (5)

[0026] In the formula, the Θ n = e inθ .

[0027] Finally, a learnable weight feature matrix W O is used to obtain the feature representation F of the output of the bidirectional reservation module:

[0028] F = MLP(GN(MSBR)·W O ) (6)

[0029] In the formula, GN(·) represents group normalization, and MLP(·) represents a multi-layer perceptron composed of two fully connected layers and a GELU activation function.

[0030] Preferably, the adaptive temporal token sparsity mechanism includes two parts: constructing local-global EEG features and calculating a binary mask using a lightweight network.

[0031] First, the local-global EEG feature is represented as follows:

[0032] Given the feature representation F of the output of the bidirectional reservation module and the initialized sparse binary mask matrix M with a value of 1, the specific representation is as follows:

[0033]

[0034] In the formula, LN(·) represents a layer normalization operation, and MLP(·) represents a perceptron composed of a fully connected layer and a GELU activation function. The output Z l contains each local feature. In order to obtain the global feature, all information-rich local features are aggregated according to the mask:

[0035]

[0036] The local-global EEG feature can be obtained by combining the above local and global features:

[0037]

[0038] In the formula, MLP(·) represents a multi-layer perceptron composed of three fully connected layers and two GELU activation functions.

[0039] Secondly, the lightweight network computing binary mask is as follows:

[0040] For the obtained local-global EEG features z t The current binary mask is calculated using Gumbel-Softmax

[0041]

[0042] Then the global binary mask is updated as follows:

[0043]

[0044] Preferably, the said reservation masking technique removes redundant tokens while maintaining hardware efficiency by preventing redundant tokens from interacting with other tokens. This can be specifically represented as follows:

[0045]

[0046] wherein, Here, G h,ij = 1 indicates that the jth EEG time token participates in the calculation of the ith EEG time token. R h represents the bidirectional attention matrix of the hth head, which only retains the interaction between non-redundant EEG time tokens while maintaining the size of T x T throughout the optimization process, and can realize efficient parallel training.

[0047] Preferably, in the inference stage, according to the given EEG time token reservation rate r, the redundant EEG time tokens are directly discarded according to the probability prediction of the lightweight network. The index of the reserved EEG time tokens can be specifically represented as follows:

[0048]

[0049] wherein, represents the time token of the test EEG data.

[0050] Advantages

[0051] 1. A convolution module is designed, which is composed of a series of time and spatial convolution operations, and is used to learn local EEG features.

[0052] 2. A cascaded bidirectional reservation module is designed, which is composed of a plurality of bidirectional reservation modules stacked, and is used to learn EEG global correlation information. An exponential decay matrix is introduced in the bidirectional reservation module to learn EEG feature time prior information.

[0053] 3. An adaptive time token sparsity mechanism is designed, including generating local-global EEG features and using them as input to calculate EEG time token binary masks using a lightweight network, which reflects the judgment of redundant EEG time tokens.

[0054] 4. A reserved masking technique is designed, which enables efficient parallel training of the model on the GPU while removing redundant tokens by preventing their interaction with other tokens, enabling a good balance between inference accuracy and efficiency for EEG recognition models. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor. Among them:

[0056] Figure 1 is the framework diagram of the EEG signal recognition method based on convolutional reserved network;

[0057] Figure 2 is the framework diagram of the adaptive time token sparsity mechanism. DETAILED DESCRIPTION

[0058] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present application, rather than all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present application.

[0059] The present application will be further explained below with reference to examples.

[0060] The main implementation process of the present application is as follows, and the related flowchart is shown in Figure 1 and Figure 2 .

[0061] Step 1: Divide the training and test EEG data sets, the training EEG data set and the test EEG data set are composed of N and EEG data, denoted as and where y i ∈{1,2,…,C} respectively represent the i-th EEG data and the corresponding label. In addition, t and ch represent the number of EEG data time points and the number of channels, respectively.

[0062] Step 2: Preprocess the EEG data, including intercepting a specific time period and band-pass filtering, to obtain preprocessed training and test EEG data.

[0063] Step 3: For the training EEG data obtained in step 2, a convolution module composed of a series of time and space convolution operations is constructed to learn local EEG features, and the specific process is as follows:

[0064] For the training EEG data X, first use the time convolution operation to learn the local time feature X t , which is as follows:

[0065] X t =Conv(X,T) (15)

[0066] Here, T represents a time convolution kernel with a size of (1, ker). Then, use the spatial convolution to extract the spatial feature representation X s , which is as follows:

[0067] X s =Conv(X t , S) (16) Here, s represents a spatial convolution kernel with a size of (ch, 1). Then, the final local EEG feature can be represented as:

[0068]

[0069] In the formula, BN(·), GELU(·) and AP(·) represent batch normalization, Gaussian error linear unit activation function and average pooling, respectively.

[0070] Step 4: For the local EEG features obtained in step 3, a cascade of bidirectional reservation modules composed of a series of bidirectional reservation modules is constructed to learn the global correlation information of EEG features, and an attenuation matrix is introduced for each bidirectional reservation module to capture the time prior information of EEG features. Given the local EEG feature representation The bidirectional reservation mechanism is represented as follows:

[0071]

[0072] In the formula, The nth part of Θ in Θ n =e in θ , the (n, m) element of the attenuation matrix D B D B，nm =γ |n-m|where denotes element-wise multiplication, and -5 , is the complex conjugate of Θ.

[0073] The MSBR module with multi-scale bidirectional reservation mechanism has multiple heads, which are denoted as follows:

[0074] MSBR = [head1, head2, …, head H ] (19)

[0075] where is the output of the h-th head. Θ n = e inθ .

[0076] Finally, a learnable weight feature matrix W O is used to obtain the feature representation F of the output of the MSBR module:

[0077] F = MLP(GN(MSBR) · W O ) (20)

[0078] where GN(·) denotes group normalization, and MLP(·) denotes a multi-layer perceptron consisting of two fully connected layers and a GELU activation function.

[0079] Step 5: Using the EEG features obtained in step 4 as input, a simple classification module consisting of average pooling and fully connected layers is constructed, and a cross-entropy loss function is defined to optimize the parameters of the convolutional reservation network. The specific formula is as follows:

[0080]

[0081] where y i and p i represent the true label and predicted probability of the i-th EEG data, respectively.

[0082] Step 6: To speed up the inference speed of the convolutional reservation network, an adaptive temporal token sparsity mechanism is designed, including constructing local-global EEG features and using a lightweight network to calculate a binary mask. Given the feature representation output by a certain MSBR module and a sparse binary mask matrix M initialized to 1, the specific representation is as follows:

[0083]

[0084] where LN(·) denotes a layer normalization operation, and MLP(·) denotes a perceptron consisting of a fully connected layer and a GELU activation function. The output Z lcontains each local feature. To obtain the global feature, all the information-rich local features are aggregated according to the mask:

[0085]

[0086] The local-global EEG features can be obtained by combining the above local and global features:

[0087]

[0088] where MLP(·) represents a multi-layer perceptron composed of three fully connected layers and two GELU activation functions.

[0089] The obtained local-global EEG features z t are used to calculate the current binary mask

[0090]

[0091] The global binary mask is updated as follows:

[0092]

[0093] Step 7: To achieve parallel training, a reservation shielding technique is designed. According to the sparse binary mask in step 6, the interaction between redundant tokens and other tokens is prevented, and redundant token removal is achieved while maintaining hardware efficiency. By designing the calculation of the bidirectional attention matrix in the bidirectional reservation module, the interaction between redundant tokens and other tokens can be prevented. The specific calculation is as follows:

[0094]

[0095] where Here, G h，ij = 1 indicates that the jth EEG time token participates in the calculation of the ith EEG time token. R h represents the bidirectional attention matrix of the hth head, which only retains the interaction between non-redundant EEG time tokens, while maintaining the size of T x T throughout the optimization process, which can achieve efficient parallel training.

[0096] Step 8: In the inference process, the redundant EEG time tokens are discarded according to the probability predicted by the lightweight network, and the time sparse convolution reservation network is used to realize the prediction of test EEG data. According to the given EEG time token reservation rate r, the index of the reserved EEG time token is obtained by selecting the rT tokens with the highest prediction probability, which can be specifically represented as:

[0097]

[0098] wherein, represents a time token of the test EEG data.

[0099] It should be noted that the above embodiments are only used to illustrate but not to limit the technical solutions of the present application. Although the present application is described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present application, and they should be covered in the scope of the claims of the present application.

Claims

1. An efficient EEG classification method based on a temporally sparse convolutional network, characterized by: Step 1: Divide the collected EEG dataset into a training set and the test set in y i ∈{1,2,…,C} represents the i-th EEG data and the corresponding label. In addition, t and ch represent the number of time points and channels of EEG data, respectively. Step 2: Preprocess the EEG data, including intercepting a specific time period and bandpass filtering to obtain preprocessed training and test EEG data. Step 3: For the training EEG data obtained in step 2, a convolution module consisting of a series of temporal and spatial convolution operations is constructed to learn local EEG features. Step 4: Based on the local EEG features obtained in step 3, a cascaded bidirectional retention module consisting of a series of bidirectional retention modules is constructed to learn the global relevant information of the EEG features. At the same time, an attenuation matrix is ​​introduced for each bidirectional retention module to capture the temporal prior information of the EEG features. Step 5: Based on the EEG features obtained in step 4, a simple classification module consisting of average pooling and fully connected layers is constructed, and a cross-entropy loss function is defined to optimize the parameters of the convolutional network. Step 6: To speed up the inference of the convolution-preserving network, an adaptive temporal token sparsification mechanism is designed to generate local-global EEG features based on the EEG tokens output by the bidirectional preservation module in step 4. Simultaneously, a lightweight network is constructed to calculate the sparsified binary masks of the tokens, and Gumbel-Softmax is used to achieve end-to-end optimization of the lightweight network. Step 7: To achieve parallel training, a preservation masking technique is designed. According to the sparse binary mask in step 6, redundant tokens are removed while maintaining hardware efficiency by preventing them from interacting with other tokens. Step 8: During inference, redundant EEG time tokens are discarded based on the probability predictions generated by the lightweight network, and predictions on the test EEG data are implemented using a temporally sparse convolutional preserving network.

2. The efficient EEG classification method based on a time-sparse convolutional network according to claim 1 is characterized in that: In step 3, the convolution module is composed of a series of temporal and spatial convolution operations to learn local EEG features. For the preprocessed training EEG data x, the specific representation is as follows: X t =Conv(X,T) (1) X s =Conv(X t ,S) (2) Where T represents the time convolution kernel of size (1, ker), X t is the feature after time convolution. S represents the spatial convolution kernel of size (ch, 1), X s is the feature after spatial convolution. represents local EEG features, BN(·), GELU(·) and AP(·) represent batch normalization, Gaussian error linear unit activation function and average pooling, respectively.

3. The efficient EEG classification method based on a time-sparse convolutional network according to claim 1 is characterized in that: In step 4, the cascaded bidirectional retention module is composed of multiple bidirectional retention modules stacked together to learn EEG global related information and time prior information. The bidirectional reservation mechanism is expressed as follows: Where, The nth part Θ in Θ n =e inθ , attenuation matrix D B The (n, m)th element D in B，nm =γ |n-m| ⊙ represents element-wise multiplication, and γ = 1-2 -5 , is the complex conjugate of Θ. The bidirectional retention module with multi-scale bidirectional retention mechanism has multiple heads, which are specifically represented as follows: MSBR=[head1,head2,…,head H ] (5) In the formula, the h-th head Θ n =e inθ . Finally, a learnable weight feature matrix W is used O , obtain the feature representation F output by the bidirectional retention module: F=MLP(GN(MSBR)·W O ) (6) Where GN(·) represents group normalization, and MLP(·) represents a multi-layer perceptron consisting of two fully connected layers and a GELU activation function.

4. The efficient EEG classification method based on a time-sparse convolutional network according to claim 1, characterized in that: In step 6, the adaptive time token sparsification mechanism includes two parts: constructing local-global EEG features and calculating binary masks using a lightweight network. First, the local-global EEG features are represented as follows: Given the feature representation of the bidirectional retention module output And the sparse binary mask matrix M with an initialization value of 1, specifically expressed as follows: Where LN(.) represents the layer normalization operation, and MLP(.) represents a perceptron consisting of a fully connected layer and a GELU activation function. Output Z l Contains each local feature. To obtain the global feature, all the information-rich local features are aggregated according to the mask: Combining the above local and global features can obtain local-global EEG features: Where MLP(·) represents a multi-layer perceptron consisting of three fully connected layers and two GELU activation functions. Second, the lightweight network computes the binary mask as follows: The local-global EEG feature Z t , use Gumbel-Softmax to calculate the current binary mask Then the global binary mask is updated as follows:

5. The efficient EEG classification method based on a time-sparse convolutional network according to claim 1 is characterized in that: In step 7, the retention shielding technology prevents redundant tokens from interacting with other tokens, thereby achieving redundant token removal while maintaining hardware efficiency. The specific representation is as follows: in, Here, G h,ij =1 means that the jth EEG time token participates in the calculation of the ith EEG time token. h represents the bidirectional attention matrix of the h-th head, which only retains the interactions between non-redundant EEG time tokens while maintaining the size of T×T throughout the optimization process, enabling efficient parallel training. Preferably, in the inference phase, redundant EEG time tokens are directly discarded according to the probability prediction of the lightweight network based on the given EEG time token retention rate r. The index of the retained EEG time tokens is obtained by selecting the rT with the highest prediction probability, which can be specifically expressed as: in, A time token representing the test EEG data.

Images