A multi-branch multi-object electroencephalogram data training method based on adaptive pruning

Abstract

The application discloses a multi-branch multi-object electroencephalogram data training method based on adaptive pruning. The method constructs a weighted multi-branch structure on a deep network of a traditional single-branch electroencephalogram deep model, each branch is used for fitting data of a target object and a source object, and an adaptive weight measures the importance of the branch to the target object. Further, in order to prevent the learning burden caused by too many branches, the application flexibly prunes the branches based on the adaptive weight during network training. The method can effectively reduce the distribution difference of multi-objects, improve the decoding ability of the model to multi-object data, and finally improve the classification accuracy of the target object.

Description

Technical Field

[0001] This invention relates to the classification of electroencephalogram (EEG) signals and the network framework for multi-object EEG training, specifically to a multi-branch, multi-object EEG data training method based on adaptive pruning. Background Technology

[0002] Currently, numerous deep learning-based EEG classification algorithms are emerging. However, the limited amount of EEG data for many tasks can lead to overfitting of these deep models. For example, in the motor imagery task, using the BCICIV-2a dataset, there are only 288 training data points per object, which is insufficient for training deep models. Therefore, many research methods continuously lightweight deep models to reduce training parameters or utilize data augmentation algorithms to increase the amount of data. However, data augmentation algorithms can introduce excessive redundant information and even noise, further reducing the signal-to-noise ratio of EEG data. Fully utilizing EEG data from other objects to enhance the training effect on the target object is undoubtedly the most promising approach. However, simply training with EEG data from all objects together can even lead to a decrease in accuracy for the target object. This anomaly is due to the significant differences in EEG signal distribution among different objects. For the target object, data from other source objects may become noise; that is, the positive impact of increasing the amount of data is less than the noise effect caused by this distributional difference. This patent's in-depth research reveals that current deep models used for EEG classification cannot map their feature spaces into a single cluster when fitting multi-object data, even if the data from different objects share the same classification category. This is also the direct reason why the target object cannot improve its classification performance by leveraging data from other objects. Summary of the Invention

[0003] Purpose of the invention: The purpose of this invention is to propose a multi-branch, multi-object EEG data training method based on adaptive pruning, which solves the problem that when training multi-object data, the large difference in distribution between the target object and the dataset makes it impossible to effectively expand the dataset to improve the classification performance of the target object.

[0004] Technical Solution: To solve the above problems, the present invention provides the following technical solution:

[0005] A method for training multi-branch, multi-object EEG data based on adaptive pruning, the method comprising the following steps:

[0006] Step 1: Construct the temporal module, channel module, and classification module of the model; the temporal module is used to extract features from the temporal dimension of EEG data, the channel module is used to extract spatial or spatiotemporal features across channels, and the classification module performs classification based on the extracted features;

[0007] Step 2: Construct a base network based on the time series module, and construct a weighted multi-branch network based on the channel module and the classification module;

[0008] Step 3: Input all source object EEG data and target object EEG data into the base network to obtain shallow features for each channel;

[0009] Step 4: Construct (source, target) sample pairs based on the shallow features of the source and target objects;

[0010] Step 5: Determine whether the adaptive weight of each branch is less than the pruning threshold. If it is less than the threshold, remove the branch completely.

[0011] Step 6: Use the channel modules of each remaining branch to perform deep feature extraction on the (source, target) sample pairs;

[0012] Step 7: Input the (source, target) sample pair features obtained from each channel module branch into the corresponding multi-branch classification module to obtain the probability prediction of the (source, target) sample pair.

[0013] Step 8: Based on the adaptive weights, combine the classification probabilities of the remaining branches to obtain the final classification of the target object.

[0014] Furthermore, in step 2, the number of branches in the channel module is the same as the number of source objects, and each branch is an identically initialized channel module responsible for feature extraction of each pair of source and target objects; the number of branches in the classification module is the same as the number of source objects, and each branch is an identically initialized classification module used for classification of each pair of source and target objects.

[0015] Furthermore, in step 3, the method for inputting all source and target objects into the base network to obtain the temporal features of each channel includes the following steps:

[0016] Step 3.1: Randomly select EEG data from n source objects. and the target's EEG number X t The number of data points for each data point is b. The total data X is obtained by summing the data points by batch channel. st ∈R (n+1)b×C×T R represents a real number, C is the number of channels, and T is the timing number.

[0017] Step 3.2, X st The input is fed into the base network to obtain shallow features f. st ∈R (n+1)b×C×l×d , where l is the length of the time series after convolution of the temporal module, and d represents the number of filters in the last layer of the base network.

[0018] Step 3.3, f st Re-divide the shallow features f of n source objects according to the batch channel.s,1 f s,2 , ..., f s,n and shallow features f of the target object t .

[0019] Furthermore, in step 4, the shallow features are constructed into (source, target) sample pairs using a 1-to-1 matching rule between the target object and the source object, resulting in n sample pairs (f s,1 f t ), (f s,2 f t ), ..., (f s,n f t ).

[0020] Furthermore, in step 5, the method for determining whether the adaptive weight of each branch is less than the pruning threshold, and if it is less than the threshold, then completely removing the branch, includes the following steps:

[0021] Step 5.1: Activate the adaptive weights using the softmax function, with the following formula:

[0022]

[0023] Where w i This represents the weight of the i-th branch, and n is the number of remaining branches.

[0024] Step 5.2, obtain the historical maximum value of each adaptive weight (from the current training epoch to the past c epochs):

[0025]

[0026] Where c is a hyperparameter, representing how much historical adaptive weight needs to be recorded. This represents the largest adaptive weight in the i-th branch from the current j-th epoch to the past c+1 epochs;

[0027] Step 5.3: Prune the branches according to a predefined threshold μ. If the value is less than μ, the i-th branch is pruned, meaning that the i-th branch will not be considered in subsequent training. If a branch is pruned, it is re-labeled according to the branch index order, and the remaining number of branches n is updated.

[0028] Furthermore, in step 6, the method for extracting spatial features from (source, target) sample pairs using the channel modules of each branch includes the following steps:

[0029] Step 6.1: Stack each pair of samples in batches to obtain f. st,1 ,f st,2 ,…,fst,n ∈R 2b×C×l×d .

[0030] Step 6.2, f st,1 ,f st,2 ,…,f st,n The deep features are obtained by inputting them into the channel modules of the branch network. Among them l * d represents the length of the time series after convolution of the channel modules. * This indicates the number of filters in the last layer of the channel module in the minute network;

[0031] Step 6.3, will Reclassified into source deep features and target deep features: as well as

[0032] Furthermore, in step 7, the method of inputting the (source, target) sample pair features obtained from each channel module branch into the corresponding multi-branch classification module to obtain the classification prediction of the (source, target) sample pair is as follows: and The inputs are fed into the corresponding multi-branch classification modules to obtain the classification probability p of the i-th source object. s,i The classification probability p of the target object in the i-th branch t,i Classification loss L for source and target objects s and L t They are respectively:

[0033]

[0034]

[0035] Where K represents the number of categories, σ represents the signal function, and y s,ij y represents the label of the j-th sample of the i-th source object. t,ij p represents the label of the j-th sample in the i-th branch of the target object. s,ij,k p represents the probability that the label of the j-th sample of the i-th source object is predicted to be of the k-th class. t,ij,k This represents the probability that the label of the j-th sample in the i-th branch of the target object is predicted to be of the k-th class.

[0036] Furthermore, in step 8, the final classification of the target object is obtained by combining the classification probabilities of the remaining branches. The classification loss of the comprehensive decision for the target object is:

[0037]

[0038] p t,i This is the output of the remaining i-th branch. The final total loss is:

[0039]

[0040] w i The update method is as follows:

[0041]

[0042] Where η is the learning rate.

[0043] The present invention also provides a computer system, including a memory, a processor, and a computer program / instructions stored in the memory and executable on the processor, wherein the computer program / instructions, when executed by the processor, implement the steps of the aforementioned multi-branch multi-object EEG data training method based on adaptive pruning.

[0044] The present invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the aforementioned multi-branch, multi-object EEG data training method based on adaptive pruning.

[0045] Beneficial Effects: This invention provides a multi-branch, multi-object EEG data training method based on adaptive pruning. It establishes a multi-branch structure using a "divide and conquer" approach, with each branch fitting a pair of source-target object data. To reduce the increased model complexity caused by the increased number of branches, this invention further utilizes adaptive weights to prune branches corresponding to adaptive weights that have been at low values ​​for a long time during network training. This effectively removes source object branches that do not significantly improve the target object's performance, reduces network complexity, and further improves the network's training speed. This method allows for the reuse of multi-object training data, thereby improving the classification performance of the target object. In various EEG classification problems, such as motor imagery, action-related potentials, and steady-state visual evoked states, this invention achieves high classification accuracy and can be applied to various classic deep model-based EEG classification methods. Compared with traditional data augmentation methods, this invention exhibits better classification performance. Attached Figure Description

[0046] Figure 1 A flowchart of a multi-branch, multi-object EEG data training method based on adaptive pruning;

[0047] Figure 2 This is a detailed flowchart of an example of classifying EEG data using EEGNet as the baseline model. Detailed Implementation

[0048] The following will provide a clear and complete description of the concept, specific structure, and technical effects of the present invention in conjunction with the embodiments and accompanying drawings, so as to fully understand the purpose, solution, and effects of the present invention. It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0049] To fully utilize the source object data and reduce its internal distributional differences, and considering the inherent characteristics of EEG signals in the task, this invention discloses a multi-branch, multi-object EEG data training method based on adaptive pruning, such as... Figure 1 As shown, it includes the following steps:

[0050] Step 1: Construct the temporal module, channel module, and classification module of the model; the temporal module is used to extract features from the temporal dimension of EEG data, the channel module is used to extract spatial or spatiotemporal features across channels, and the classification module performs classification based on the extracted features;

[0051] Step 2: Construct a base network based on the time series module, and construct a weighted multi-branch network based on the channel module and the classification module;

[0052] Step 3: Input all source object EEG data and target object EEG data into the base network to obtain shallow features for each channel;

[0053] Step 4: Construct (source, target) sample pairs based on the shallow features of the source and target objects;

[0054] Step 5: Determine whether the adaptive weight of each branch is less than the pruning threshold. If it is less than the threshold, remove the branch completely.

[0055] Step 6: Use the channel modules of each remaining branch to perform deep feature extraction on the (source, target) sample pairs;

[0056] Step 7: Input the (source, target) sample pair features obtained from each channel module branch into the corresponding multi-branch classification module to obtain the probability prediction of the (source, target) sample pair.

[0057] Step 8: Based on the adaptive weights, combine the classification probabilities of the remaining branches to obtain the final classification of the target object.

[0058] Specifically, in step 1, the model can be constructed based on classic EEG classification networks, including a temporal module, a channel module, and a classification module. Some representative deep EEG classification models, such as EEGNet, typically first extract features along the temporal dimension; in this embodiment, this part of the network is defined as the temporal module. Then, spatial or spatiotemporal features are extracted across channels; in this embodiment, this part of the network is defined as the channel module. Finally, a fully connected layer is used for classification, i.e., the classification module.

[0059] The following example, using EEGNet, illustrates the method for constructing a multi-branch structure:

[0060] like Figure 2 As shown, the first convolutional layer of EEGNet is used to construct a temporal module to extract the temporal features of EEG data; the second and third convolutional layers of EEGNet are used to construct a channel module to extract the spatiotemporal features of EEG data; and the last fully connected layer is used to construct a classification module to output the classification probability distribution of EEG data.

[0061] The temporal module extracts shallow features (temporal features) from all source objects and one target object, which are then used to construct the base network in this embodiment. The channel module is expanded to n branches (the number of which matches the number of source objects), meaning each branch is an identically initialized channel module responsible for feature extraction for each pair of source and target objects. The classification module is also expanded to n branches, meaning each branch is an identically initialized classification module used for classifying each pair of source and target objects.

[0062] Based on the network model constructed in this embodiment, an iterative optimization method for training multi-branch, multi-object EEG data based on adaptive pruning includes the following steps:

[0063] (1) Randomly select N source objects' EEG data and the target's EEG number X t The number of data points for each data point is b. The total data X is obtained by summing the data points by batch channel. st ∈R (n+1)b×C×T R represents a real number, C is the number of channels, and T is the timing number.

[0064] (2) X st The input is fed into the base network to obtain shallow features f. st ∈R (n+1)b×C×l×d , where l is the length of the time series after convolution of the temporal module, and d represents the number of filters in the last layer of the base network.

[0065] (3) f st The batch channel is re-divided into n source object shallow features F. s,1 f s,2 F s,n and shallow features F of the target object t .

[0066] (4) The method of constructing shallow features into (source, target) sample pairs adopts a one-to-one matching rule between the target object and the source object to obtain n sample pairs (f s,1 f t ), (f s,2 f t), ..., (f s,n f t ).

[0067] (5) Stack each pair of samples in batches to obtain f st,1 ,f st,2 ,…,f st,n ∈R 2b×c×l×d .

[0068] (6) Determine if the adaptive weight of each branch is less than the pruning threshold. If it is less than the threshold, remove the branch completely. First, activate the adaptive weights using the softmax function, with the following formula:

[0069]

[0070] Where w i The weight (w) of the i-th branch i Initialized as Then, the historical maximum value of each adaptive weight (from the current training epoch to the past c epochs) is obtained:

[0071]

[0072] Where c is a hyperparameter, representing how much historical adaptive weight needs to be recorded. It represents the largest adaptive weight in the i-th branch from the current j-th epoch to the past c+1 epochs.

[0073] (7) f st,1 ,f st,2 ,…,f st,n The deep features are obtained by inputting them into the channel modules of the branch network. Among them l * d represents the length of the time series after convolution of the channel modules. * This indicates the number of filters in the last layer of the channel module in the branch network.

[0074] (8) Reclassified into source deep features and target deep features: as well as

[0075] (9) and The inputs are fed into the corresponding multi-branch classification modules to obtain the classification probability p of source object 1. s,1 The classification probability p of the target object in the first branch t,1 Similarly, we can obtain p s,2 , ..., p s,nand p t,2 , ..., p t,n So, the classification loss L for the source object and the target object is... s and L t They are respectively:

[0076]

[0077] Where K represents the number of categories, σ represents the signal function, and y s,ij y represents the label of the j-th sample of the i-th source object. t,ij p represents the label of the j-th sample in the i-th branch of the target object. s,ij,k p represents the probability that the label of the j-th sample of the i-th source object is predicted to be of the k-th class. t,ij,k This represents the probability that the label of the j-th sample in the i-th branch of the target object is predicted to be of the k-th class.

[0078] (10) The method of obtaining the final classification of the target object by combining the classification probabilities of the remaining branches, the classification loss of the target object's comprehensive decision is:

[0079]

[0080] p t,i This is the output of the remaining i-th branch. The final total loss is:

[0081]

[0082] w i The update method is as follows:

[0083]

[0084] Where η is the learning rate.

[0085] (11) Calculate the gradient based on the loss and update the base network, multi-branch network (including channel module and classification module) and adaptive weights.

[0086] (12) Repeat steps (1)-(10) until the loss no longer decreases.

[0087] Table 1 shows the basic information of the BCICIV-2a EEG dataset based on motor imagery used in this embodiment.

[0088] Table 1. Relevant statistical information of the BCICIV-2a dataset.

[0089]

[0090] BCICIV-2a has 9 objects, and 9 experiments are conducted. In the first experiment, object 1 is selected as the target object, and the remaining 8 are used as source objects; in the second experiment, object 2 is selected as the target object, and the remaining 8 are used as source objects, and so on.

[0091] Table 2 Classification results of the method of this invention compared with other classification models

[0092]

[0093] Table 2 shows FBCSP+SVM, Shallow ConvNet, and EEGNet as comparison methods, and WMB_EEGNet as the method proposed in this invention. Table 2 shows that the accuracy of the four methods for Subject 1 was as follows: WMB_EEGNet > Shallow ConvNet > EEGNet > FBCSP+SVM.

[0094] The accuracy of the four methods for Subject 2 was compared and found to be: WMB_EEGNet method > EEGNet method > ShallowConvNet method > FBCSP+SVM method;

[0095] The accuracy of the four methods compared by subject 3 was as follows: WMB_EEGNet method > EEGNet method > ShallowConvNet method > FBCSP+SVM method;

[0096] The accuracy of the four methods was compared among the participants, and the results were as follows: WMB_EEGNet method > Shallow ConvNet method > EEGNet method > FBCSP+SVM method;

[0097] The accuracy of the four methods for subject 5 was compared and found to be: WMB_EEGNet method > EEGNet method > ShallowConvNet method > FBCSP+SVM method;

[0098] The accuracy of the four methods compared by subject 6 was as follows: WMB_EEGNet method > EEGNet method > ShallowConvNet method > FBCSP+SVM method;

[0099] The accuracy of the four methods compared by subject 7 was as follows: WMB_EEGNet method > Shallow ConvNet method > EEGNet method > FBCSP+SVM method;

[0100] The accuracy of the four methods compared among the 8 participants was as follows: WMB_EEGNet method > EEGNet method > ShallowConvNet method > FBCSP+SVM method;

[0101] The accuracy of the four methods compared among the 9 participants was as follows: EEGNet method > WMB_EEGNet method > ShallowConvNet method > FBCSP+SVM method;

[0102] As shown above, our method can effectively utilize multi-object data to improve the classification accuracy of target objects.

[0103] This invention also discloses a computer system, including a memory, a processor, and a computer program / instructions stored in the memory and executable on the processor. When the computer program / instructions are executed by the processor, they implement the steps of the aforementioned multi-branch, multi-object EEG data training method based on adaptive pruning.

[0104] This invention also discloses a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the aforementioned multi-branch, multi-object EEG data training method based on adaptive pruning.

[0105] Although the description of the invention has been quite detailed and particularly of several described embodiments, it is not intended to limit it to any of these details or embodiments or any particular embodiment, but should be considered as providing a broad possible interpretation of the claims by referring to the appended claims and taking into account the prior art, thereby effectively covering the intended scope of the invention. Furthermore, the invention has been described above with respect to embodiments foreseeable by the inventors in order to provide a useful description, and non-substantial modifications to the invention that have not yet been foreseen may still represent equivalent modifications.

Claims

1. A method for training multi-branch, multi-object EEG data based on adaptive pruning, characterized in that, The method includes the following steps: Step 1: Construct the temporal module, channel module, and classification module of the model; the temporal module is used to extract features from the temporal dimension of EEG data, the channel module is used to extract spatial or spatiotemporal features across channels, and the classification module performs classification based on the extracted features; Step 2: Construct a base network based on the time series module, and construct a weighted multi-branch network based on the channel module and the classification module; Step 3: Input all source object EEG data and target object EEG data into the base network to obtain shallow features for each channel; Step 4: Construct (source, target) sample pairs based on the shallow features of the source and target objects; Step 5: Determine whether the adaptive weight of each branch is less than the pruning threshold. If it is less than the threshold, remove the branch completely. Step 6: Use the channel modules of each remaining branch to perform deep feature extraction on the (source, target) sample pairs; Step 7: Input the (source, target) sample pair features obtained from each channel module branch into the corresponding multi-branch classification module to obtain the probability prediction of the (source, target) sample pair. Step 8: Based on the adaptive weights, combine the classification probabilities of the remaining branches to obtain the final classification of the target object.

2. The method for training multi-branch, multi-object EEG data based on adaptive pruning according to claim 1, characterized in that, In step 2, the number of channel module branches is the same as the number of source objects. Each branch is an identical initialized channel module, responsible for feature extraction of each pair of source and target objects. The number of branches in the classification module is the same as the number of source objects. Each branch is an identically initialized classification module used for classifying each pair of source and target objects.

3. The method for training multi-branch, multi-object EEG data based on adaptive pruning according to claim 1, characterized in that, Step 3 includes the following steps: Step 3.1: Randomly select EEG data from n source objects. and target subject's EEG data X t The number of data points for each data point is b. The total data X is obtained by summing the data points by batch channel. st ∈R (n+1)b×C×T R represents a real number, C is the number of channels, and T is the timing number; Step 3.2, X st The input is fed into the base network to obtain shallow features f. st ∈R (n+1)b×C×l×d Where l is the length of the time series after convolution of the temporal module, and d represents the number of filters in the last layer of the base network; Step 3.3, f st Re-divide the shallow features f of n source objects according to the batch channel. s,1 f s,2 , ..., f s,n and shallow features f of the target object t .

4. The method for training multi-branch, multi-object EEG data based on adaptive pruning according to claim 1, characterized in that, In step 4, a one-to-one matching rule between the target object and the source object is used to obtain n pairs of sample objects (f s,1 f t ), (f s,2 f t ), ..., (f s,n f t ), where f s,1 f s,2 , ..., f s,n For shallow features of n source objects, f t These are shallow features of the target object.

5. The method for training multi-branch, multi-object EEG data based on adaptive pruning according to claim 1, characterized in that, Step 5 includes the following steps: Step 5.1: Activate the adaptive weights using the softmax function, as shown in the following formula: Where w i This represents the weight of the i-th branch, and n is the number of remaining branches; Step 5.2, obtain the historical maximum value of each adaptive weight: Where c is a hyperparameter, representing how many historical adaptive weights need to be recorded. This represents the largest adaptive weight in the i-th branch from the current j-th epoch to the past c+1 epochs; Step 5.3: Prune the branches according to a predefined threshold μ. If the value is less than μ, the i-th branch is pruned, meaning that the i-th branch will not be considered in subsequent training. If a branch is pruned, it is re-labeled according to the branch index order, and the remaining number of branches n is updated.

6. The method for training multi-branch, multi-object EEG data based on adaptive pruning according to claim 1, characterized in that, Step 6 includes the following steps: Step 6.1: Stack each pair of samples in batches to obtain f. st,1 ,f st,2 ,…,f st,n ∈R 2b×C×l×d Where n is the number of remaining branches, b is the batch size, C is the number of channels, l is the length of the time series after convolution of the temporal module, and d is the number of filters in the last layer of the base network. Step 6.2, f st,1 ,f st,2 ,…,f st,n The deep features are obtained by inputting them into the channel modules of the branch network. Among them l * d represents the length of the time series after convolution of the channel modules. * This indicates the number of filters in the last layer of the channel module in the branch network; Step 6.3, will Reclassified into source deep features and target deep features: as well as 7. The method for training multi-branch, multi-object EEG data based on adaptive pruning according to claim 1, characterized in that, In step 7, the source deep features obtained by the channel module are... and target deep features The inputs are fed into the corresponding multi-branch classification modules to obtain the classification probability p of the i-th source object. s,i The classification probability p of the target object in the i-th branch t,i Classification loss L for source and target objects s and L t They are respectively: Where n is the number of remaining branches, b is the batch size, L represents the number of categories, σ represents the signal function, and y s,ij Let y represent the label of the k-th sample of the i-th source object. t,ij p represents the label of the j-th sample in the i-th branch of the target object. s,ij,k p represents the probability that the label of the j-th sample of the i-th source object is predicted to be of the k-th class. t,ij,k This represents the probability that the label of the j-th sample in the i-th branch of the target object is predicted to be of the k-th class.

8. The method for training multi-branch, multi-object EEG data based on adaptive pruning according to claim 1, characterized in that, In step 8, the classification loss for the comprehensive decision of the target object is: in The weight w of the i-th branch i The value p after activation using the softmax function t,i Let be the probability vector of the predicted target object in the remaining i-th branch, n be the number of remaining branches, K be the number of classes, σ represent the signal function, b be the batch size, and y be the probability vector of the remaining i-th branch. t,j Let be the label of the k-th target data sample. Let $\frac{j}{j}$ be the probability that the j-th target data sample is predicted to be of class $k$; the final total loss is: Where L s and L t The classification losses for the source object and the target object are respectively; w i The update method is as follows: Where η is the learning rate.

9. A computer system comprising a memory, a processor, and computer programs / instructions stored in the memory and executable on the processor, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps of the multi-branch, multi-object EEG data training method based on adaptive pruning according to any one of claims 1-8.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps of the multi-branch, multi-object EEG data training method based on adaptive pruning according to any one of claims 1-8.

Images