Multi-branch multi-object electroencephalogram data training method based on adaptive pruning

Abstract

The invention discloses a multi-branch multi-object electroencephalogram data training method based on adaptive pruning, and the method comprises the steps: constructing a weighted multi-branch structure on a deep network of a conventional single-branch electroencephalogram deep model, and enabling each branch to be used for fitting the data of a target object and a source object; the self-adaptive weight measures the importance degree of the branches to the target object; furthermore, in order to prevent learning burden caused by too many branches, the branches are flexibly pruned during network training based on adaptive weights. According to the method, the distribution difference of multiple objects can be effectively reduced, the decoding capability of the model on multi-object data is improved, and finally the classification accuracy of the target object is improved.

Description

Technical Field

[0001] The present invention relates to the classification of electroencephalogram (EEG) signals and a network framework for multi-object EEG training, and in particular to a multi-branch multi-object EEG data training method based on adaptive pruning. Background Art

[0002] At present, EEG classification algorithms based on deep models emerge in an endless stream, but the problem of small amount of EEG data for many tasks may cause overfitting of deep models. For example, in the motor imagery task, taking BCICIV-2a data as an example, there are only 288 training data for each object, which is not conducive to the training of deep models. Therefore, many research methods continue to lightweight the deep model to reduce training parameters or use data enhancement algorithms to increase the amount of data. However, the data enhancement algorithm will bring too much repeated information and even noise, further reducing the signal-to-noise ratio of EEG. If the EEG data of other objects can be fully utilized to enhance the training effect of the target object, it is undoubtedly the most potential. However, if the EEG data of all objects are simply trained together, it may even cause a decrease in the accuracy of the target object. The reason for this abnormal phenomenon is that the distribution of EEG signals of different objects is very different. For the target object, the data of other source objects may become noise, that is, the positive impact brought by the expansion of the data volume is not as good as the noise impact brought by this distribution difference. This patent has conducted in-depth research and found that the deep model currently used for EEG classification cannot map their feature space into a cluster when fitting multi-object data, even if the data of different objects have the same classification category. This is also the direct reason why the target object cannot use other object data to improve classification performance. Summary of the invention

[0003] Purpose of the invention: The purpose of the present invention is to propose a multi-branch multi-object EEG data training method based on adaptive pruning to solve the problem that during multi-object data training, the data set cannot be effectively expanded to improve the classification performance of the target object due to the large distribution difference with the target object.

[0004] Technical solution: To solve the above problems, the present invention provides the following technical solutions:

[0005] A multi-branch multi-object EEG data training method based on adaptive pruning, the method comprising the following steps:

[0006] Step 1, constructing a timing module, a channel module and a classification module of the model; the timing module is used to extract features from the time dimension of EEG data, the channel module is used to extract spatial features or spatiotemporal features across channels, and the classification module performs classification based on the extracted features;

[0007] Step 2: construct a base network based on the timing module, and construct a weighted multi-branch network based on the channel module and the classification module;

[0008] Step 3, input all the EEG data of the source objects and the EEG data of the target objects into the base network to obtain the shallow features of each channel;

[0009] Step 4: construct (source, target) sample pairs based on the shallow features of the source object and the target object;

[0010] Step 5: determine whether the adaptive weight of each branch is less than the pruning threshold. If it is less than the threshold, the branch is completely removed;

[0011] Step 6, using the remaining channel modules of each branch to perform deep feature extraction on the (source, target) sample pairs;

[0012] Step 7, input the (source, target) sample pair features obtained by each channel module branch into the corresponding multi-branch classification module to obtain the probability prediction of the (source, target) sample pair;

[0013] Step 8: The classification probabilities of the remaining branches are integrated according to the adaptive weights to obtain the final classification of the target object.

[0014] Furthermore, in step 2, the number of channel module branches is consistent with the number of source objects, and each branch is a channel module initialized in the same way, which is responsible for the feature extraction of each pair of source objects and target objects; the number of classification module branches is consistent with the number of source objects, and each branch is a classification module initialized in the same way, which is used for the classification of each pair of source objects and target objects.

[0015] Further, in step 3, the method of inputting all source objects and target objects into the base network to obtain the temporal features of each channel includes the following steps:

[0016] Step 3.1: Randomly select n source object EEG data and the target object's EEG number X t , the number of each data is b, and the total data X is obtained by superimposing the batch channels st ∈R (n+1)b×C×T , R represents a real number, C is the number of channels, and T is the time series number.

[0017] Step 3.2, X st Input into the base network to obtain the shallow feature f st ∈R (n+1)b×C×l×d , where l is the length of the time series after the convolution of the timing module, and d represents the number of filters in the last layer of the base network.

[0018] Step 3.3, f st Re-divide the shallow features f of n source objects by batch channelss,1 , f s,2 , …, f s,n and the shallow features of the target object f t .

[0019] Furthermore, in step 4, the shallow features are constructed as (source, target) sample pairs, and a one-to-one matching rule between the target object and the source object is adopted to obtain n pairs of sample pairs (f s,1 , f t ), (f s,2 , f t ),…,(f s,n , f t ).

[0020] Furthermore, in step 5, it is determined whether the adaptive weight of each branch is less than the pruning threshold. If the adaptive weight of each branch is less than the threshold, the method of completely removing the branch includes the following steps:

[0021] Step 5.1, use the softmax function to activate the adaptive weights, with the following formula:

[0022]

[0023] where w i represents the weight of the i-th branch, and n is the number of remaining branches.

[0024] Step 5.2, get the historical maximum value of each adaptive weight (from the current training epoch to the past c epochs):

[0025]

[0026] Where c is a hyperparameter, indicating how much historical adaptive weights need to be recorded in the past. Indicates the maximum adaptive weight of the i-th branch from the current j-th epoch to the past c+1 epochs;

[0027] Step 5.3, prune the branches according to the predefined threshold μ. When it is less than μ, the i-th branch is pruned, that is, the i-th branch will not be considered in subsequent training; if a branch is pruned, it is renumbered in the order of branch subscripts, and the number of remaining branches n is updated.

[0028] Furthermore, in step 6, the method of performing spatial feature extraction on the (source, target) sample pair using the channel module of each branch includes the following steps:

[0029] Step 6.1: stack each pair of samples in batches to obtain f st,1 ,f st,2 ,…,fst,n ∈R 2b×C×l×d .

[0030] Step 6.2, f st,1 ,f st,2 ,…,f st,n Input into the channel modules of the branch network respectively to obtain deep features Among them l * is the length of the time series after the channel module convolution, and d * represents the number of filters in the last layer of the channel module in the minute network;

[0031] Step 6.3, Re-divided into source deep features and target deep features: as well as

[0032] Further, in step 7, the (source, target) sample pair features obtained by each channel module branch are input into the corresponding multi-branch classification module to obtain the classification prediction method of the (source, target) sample pair. and Input into the corresponding multi-branch classification module to obtain the classification probability p of the i-th source object s,i And the classification probability p of the target object in the i-th branch t,i ; Classification loss L of source and target objects s and L t They are:

[0033]

[0034]

[0035] Where K represents the number of categories, σ represents the signal function, and y s,ij represents the label of the jth sample of the i-th source object, y t,ij represents the label of the jth sample of the i-th branch of the target object, p s,ij,k represents the probability that the label of the jth sample of the i-th source object is predicted to be the k-th class, p t,ij,k Represents the probability that the label of the jth sample of the i-th branch of the target object is predicted to be the k-th class.

[0036] Furthermore, in step 8, the classification probabilities of the remaining branches are integrated to obtain the final classification method of the target object. The classification loss of the comprehensive decision of the target object is:

[0037]

[0038] p t,i is the output of the remaining i-th branch. The final total loss is:

[0039]

[0040] w i The update method is as follows:

[0041]

[0042] Where η is the learning rate.

[0043] The present invention also provides a computer system, comprising a memory, a processor, and a computer program / instruction stored in the memory and executable on the processor, wherein when the computer program / instruction is executed by the processor, the steps of the multi-branch multi-object EEG data training method based on adaptive pruning are implemented.

[0044] The present invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the multi-branch multi-object EEG data training method based on adaptive pruning.

[0045] Beneficial effects: The present invention provides a multi-branch multi-object EEG data training method based on adaptive pruning. A multi-branch structure is established through the idea of ​​"divide and conquer". Each branch is used to fit the data of a pair of source-target objects. In order to reduce the increase in the number of branches and the increase in model complexity, the present invention further uses adaptive weights to prune the branches corresponding to the adaptive weights that are at a low value for a long time during the network training process. In this way, the source object branches that do not improve the target object much are effectively deleted, and the complexity of the network is reduced, and the training speed of the network is further improved. This method can reuse the training data of multiple objects to improve the classification performance of the target object. In various EEG classification problems, such as motor imagery, action-related potentials, and steady-state visual evoked tasks, the classification accuracy obtained by the present invention is high, and can be used for various classic EEG classifications based on deep models. Compared with traditional data enhancement methods, the present invention has better classification performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a flow chart of a multi-branch multi-object EEG data training method based on adaptive pruning;

[0047] Figure 2 The figure is a detailed flowchart of an implementation example of EEG data classification using EEGNet as a benchmark model. DETAILED DESCRIPTION

[0048] The following will be combined with the embodiments and drawings to clearly and completely describe the concept, specific structure and technical effects of the present invention, so as to fully understand the purpose, scheme and effect of the present invention. It should be noted that the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0049] In order to make full use of the source object data and reduce the distribution differences within them, taking into account the inherent characteristics of EEG signals in the task, the embodiment of the present invention discloses a multi-branch multi-object EEG data training method based on adaptive pruning, such as Figure 1 As shown, the following steps are included:

[0050] Step 1, constructing a timing module, a channel module and a classification module of the model; the timing module is used to extract features from the time dimension of EEG data, the channel module is used to extract spatial features or spatiotemporal features across channels, and the classification module performs classification based on the extracted features;

[0051] Step 2: construct a base network based on the timing module, and construct a weighted multi-branch network based on the channel module and the classification module;

[0052] Step 3: Input all the EEG data of the source objects and the EEG data of the target objects into the base network to obtain the shallow features of each channel;

[0053] Step 4: construct (source, target) sample pairs based on the shallow features of the source object and the target object;

[0054] Step 5: determine whether the adaptive weight of each branch is less than the pruning threshold. If it is less than the threshold, the branch is completely removed;

[0055] Step 6, using the remaining channel modules of each branch to perform deep feature extraction on the (source, target) sample pairs;

[0056] Step 7, input the (source, target) sample pair features obtained by each channel module branch into the corresponding multi-branch classification module to obtain the probability prediction of the (source, target) sample pair;

[0057] Step 8: The classification probabilities of the remaining branches are integrated according to the adaptive weights to obtain the final classification of the target object.

[0058] Specifically, in step 1, the model's timing module, channel module, and classification module can be constructed based on the classic EEG classification network. Some representative EEG classification deep models, such as EEGNet, usually first extract features from the time dimension, and this embodiment defines this part of the network as a timing module; then extract spatial features or spatiotemporal features across channels, and this embodiment defines this part of the network as a channel module; finally, a fully connected layer is used for classification, i.e., a classification module.

[0059] The following takes EEGNet as an example to illustrate the method of building a multi-branch structure:

[0060] like Figure 2 As shown in the figure, the first convolution layer of EEGNet is constructed as a timing module to extract the timing features of EEG data; the second and third convolution layers of EEGNet are constructed as channel modules to extract the spatiotemporal features of EEG data; and the last fully connected layer is constructed as a classification module to output the classification probability distribution of EEG data.

[0061] The timing module extracts shallow features (timing features) of all source objects and one target object, and this embodiment constructs it as a base network. The channel module is expanded into n branches (the number of which is consistent with the number of source objects), that is, each branch is a channel module with the same initialization, responsible for the feature extraction of each pair of source objects and target objects. The classification module is also expanded into n branches, that is, each branch is a classification module with the same initialization, used for the classification of each pair of source objects and target objects.

[0062] Based on the network model constructed in this embodiment, an iterative optimization method of a multi-branch multi-object EEG data training method based on adaptive pruning includes the following steps:

[0063] (1) Randomly select n source object EEG data and the target object's EEG number X t , the number of each data is b, and the total data X is obtained by superposition of batch channels st ∈R (n+1)b×C×T , R represents a real number, C is the number of channels, and T is the time series number.

[0064] (2) X st Input into the base network to obtain the shallow feature f st ∈R (n+1)b×C×l×d , where l is the length of the time series after the convolution of the timing module, and d represents the number of filters in the last layer of the base network.

[0065] (3) f st Re-divide the batch channels into n source object shallow features F s,1 , f s,2 , …, F s,n and the shallow feature F of the target object t .

[0066] (4) The shallow features are constructed as (source, target) sample pairs, using a one-to-one matching rule between the target object and the source object to obtain n pairs of sample pairs (f s,1 , f t ), (f s,2 , f t),…,(f s,n , f t ).

[0067] (5) Superimpose each pair of samples in batches to obtain f st,1 ,f st,2 ,…,f st,n ∈R 2b×c×l×d .

[0068] (6) Determine whether the adaptive weight of each branch is less than the pruning threshold. If it is less than the threshold, the branch is completely removed. First, use the softmax function to activate the adaptive weight, which has the following formula:

[0069]

[0070] where w i represents the weight of the i-th branch (w i Initially initialized to ). Then get the historical maximum value of each adaptive weight (from the current training epoch to the past c epochs):

[0071]

[0072] Where c is a hyperparameter, indicating how much historical adaptive weights need to be recorded in the past. It represents the maximum adaptive weight of the i-th branch from the current j-th epoch to the past c+1 epochs.

[0073] (7) f st,1 ,f st,2 ,…,f st,n Input into the channel modules of the branch network respectively to obtain deep features Among them l * is the length of the time series after the channel module convolution, and d * Indicates the number of filters in the last layer of the channel module in the branch network.

[0074] (8) Re-divided into source deep features and target deep features: as well as

[0075] (9) and Input into the corresponding multi-branch classification module to obtain the classification probability p of source object No. 1 s,1 And the classification probability p of the target object in the first branch t,1 Similarly, we can get p s,2 ,…,p s,nand p t,2 ,…,p t,n Then, the classification loss L of the source object and the target object is s and L t They are:

[0076]

[0077] Where K represents the number of categories, σ represents the signal function, and y s,ij represents the label of the jth sample of the i-th source object, y t,ij represents the label of the jth sample of the i-th branch of the target object, p s,ij,k represents the probability that the label of the jth sample of the i-th source object is predicted to be the k-th class, p t,ij,k Represents the probability that the label of the jth sample of the i-th branch of the target object is predicted to be the k-th class.

[0078] (10) The classification probabilities of the remaining branches are combined to obtain the final classification of the target object. The classification loss of the comprehensive decision of the target object is:

[0079]

[0080] p t,i is the output of the remaining i-th branch. The final total loss is:

[0081]

[0082] w i The update method is as follows:

[0083]

[0084] Where η is the learning rate.

[0085] (11) Calculate the gradient based on the loss and update the base network, multi-branch network (including channel module and classification module) and adaptive weights.

[0086] (12) Repeat steps (1)-(10) until the loss no longer decreases.

[0087] Table 1 shows the basic information of the BCICIV-2a data set based on motor imagery used in this embodiment.

[0088] Table 1. Statistical information of the BCICIV-2a dataset

[0089]

[0090] BCICIV-2a has 9 subjects, and 9 experiments are conducted. In the first experiment, subject No. 1 is selected as the target subject, and the remaining 8 are selected as source subjects; in the second experiment, subject No. 2 is selected as the target subject, and the remaining 8 are selected as source subjects, and so on.

[0091] Table 2 Classification results of the method of the present invention and other classification models

[0092]

[0093] In Table 2, FBCSP+SVM, Shallow ConvNet and EEGNet are comparison methods, and WMB_EEGNet is the method proposed by the present invention. As shown in Table 2, the accuracy of the four methods of Subject 1 is compared as follows: WMB_EEGNet method>ShallowConvNet>EEGNet method>FBCSP+SVM method;

[0094] The accuracy of the four methods of subject 2 was compared: WMB_EEGNet method > EEGNet method > ShallowConvNet method > FBCSP+SVM method;

[0095] The accuracy of the four methods of subject 3 was compared: WMB_EEGNet method > EEGNet method > ShallowConvNet method > FBCSP+SVM method;

[0096] The accuracy of the four methods of subject 4 was compared: WMB_EEGNet method > Shallow ConvNet > EEGNet method > FBCSP+SVM method;

[0097] The accuracy of the four methods of subject 5 was compared: WMB_EEGNet method > EEGNet method > ShallowConvNet method > FBCSP+SVM method;

[0098] The accuracy of the four methods of subject 6 was compared: WMB_EEGNet method > EEGNet method > ShallowConvNet method > FBCSP+SVM method;

[0099] The accuracy of the four methods of subject 7 was compared: WMB_EEGNet method > Shallow ConvNet > EEGNet method > FBCSP+SVM method;

[0100] The accuracy of the four methods of subject 8 was compared: WMB_EEGNet method > EEGNet method > ShallowConvNet method > FBCSP+SVM method;

[0101] The accuracy of the four methods of subject 9 was compared: EEGNet method > WMB_EEGNet method > ShallowConvNet method > FBCSP+SVM method;

[0102] From the above, we can see that our method can effectively utilize multi-object data to improve the classification accuracy of the target object.

[0103] An embodiment of the present invention also discloses a computer system, including a memory, a processor, and a computer program / instruction stored in the memory and executable on the processor. When the computer program / instruction is executed by the processor, the steps of the multi-branch multi-object EEG data training method based on adaptive pruning are implemented.

[0104] The embodiment of the present invention also discloses a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the multi-branch multi-object EEG data training method based on adaptive pruning.

[0105] Although the description of the present invention has been quite detailed and specifically describes several described embodiments, it is not intended to be limited to any of these details or embodiments or any particular embodiment, but should be regarded as providing a broad possible interpretation of these claims in view of the prior art by reference to the appended claims, thereby effectively covering the intended scope of the present invention. In addition, the above description of the present invention is based on the embodiments foreseeable by the inventor, and its purpose is to provide a useful description, and those non-substantial changes to the present invention that have not yet been foreseen may still represent equivalent changes to the present invention.

Claims

1. A multi-branch multi-object EEG data training method based on adaptive pruning, characterized in that: The method comprises the following steps: Step 1, constructing a timing module, a channel module and a classification module of the model; the timing module is used to extract features from the time dimension of EEG data, the channel module is used to extract spatial features or spatiotemporal features across channels, and the classification module performs classification based on the extracted features; Step 2: construct a base network based on the timing module, and construct a weighted multi-branch network based on the channel module and the classification module; Step 3, input all the EEG data of the source objects and the EEG data of the target objects into the base network to obtain the shallow features of each channel; Step 4: construct (source, target) sample pairs based on the shallow features of the source object and the target object; Step 5: determine whether the adaptive weight of each branch is less than the pruning threshold. If it is less than the threshold, the branch is completely removed; Step 6, using the remaining channel modules of each branch to perform deep feature extraction on the (source, target) sample pairs; Step 7, input the (source, target) sample pair features obtained by each channel module branch into the corresponding multi-branch classification module to obtain the probability prediction of the (source, target) sample pair; Step 8: The classification probabilities of the remaining branches are integrated according to the adaptive weights to obtain the final classification of the target object.

2. According to the multi-branch multi-object EEG data training method based on adaptive pruning according to claim 1, it is characterized in that: In step 2, the number of channel module branches is the same as the number of source objects, and each branch is a channel module with the same initialization, responsible for feature extraction of each pair of source and target objects; The number of classification module branches is consistent with the number of source objects, and each branch is an identically initialized classification module, which is used to classify each pair of source and target objects.

3. The multi-branch multi-object EEG data training method based on adaptive pruning according to claim 1, characterized in that: In step 3, the following steps are included: Step 3.1: Randomly select n source object EEG data and the target object EEG data X t , the number of each data is b, and the total data X is obtained by superimposing the batch channels st ∈R (n+1)b×C×T , R represents a real number, C is the number of channels, and T is the time series number; Step 3.2, X st Input into the base network to obtain the shallow feature f st ∈R (n+1)b×C×l×d , where l is the length of the time series after the convolution of the timing module, and d represents the number of filters in the last layer of the base network; Step 3.3, f st Re-divide the shallow features f of n source objects by batch channels s,1 , f s,2 , …, f s,n and the shallow features of the target object f t .

4. The multi-branch multi-object EEG data training method based on adaptive pruning according to claim 1, characterized in that: In step 4, a one-to-one matching rule is adopted between the target object and the source object to obtain n pairs of samples (f s,1 , f t ), (f s,2 , f t ),…,(f s,n , f t ), where f s,1 , f s,2 , …, f s,n are the shallow features of n source objects, f t It is the shallow feature of the target object.

5. The multi-branch multi-object EEG data training method based on adaptive pruning according to claim 1, characterized in that: In step 5, the following steps are included: Step 5.1, use the softmax function to activate the adaptive weights, the formula is as follows: where w i represents the weight of the i-th branch, and n is the number of remaining branches; Step 5.2, get the historical maximum value of each adaptive weight: Where c is a hyperparameter, indicating how many historical adaptive weights need to be recorded in the past. Indicates the maximum adaptive weight of the i-th branch from the current j-th epoch to the past c+1 epochs; Step 5.3, prune the branches according to the predefined threshold μ. When it is less than μ, the i-th branch is pruned, that is, the i-th branch will not be considered in subsequent training; if a branch is pruned, it is renumbered in the order of branch subscripts, and the number of remaining branches n is updated.

6. The multi-branch multi-object EEG data training method based on adaptive pruning according to claim 1, characterized in that: In step 6, the following steps are included: Step 6.1: stack each pair of samples in batches to obtain f st,1 ,f st,2 ,…,f st,n ∈R 2b×C×l×d ; Where n is the number of remaining branches, b is the batch size, C is the number of channels, l is the length of the time series after the convolution of the timing module, and d is the number of filters in the last layer of the base network; Step 6.2, f st,1 ,f st,2 ,…,f st,n Input into the channel modules of the branch network respectively to obtain deep features Among them l * is the length of the time series after the channel module convolution, d * Indicates the number of filters in the last layer of the channel module in the branch network; Step 6.3, Re-divided into source deep features and target deep features: as well as 7. The multi-branch multi-object EEG data training method based on adaptive pruning according to claim 1, characterized in that: In step 7, the source deep features obtained by the channel module are and target deep features Input into the corresponding multi-branch classification module to obtain the classification probability p of the i-th source object s,i And the classification probability p of the target object in the i-th branch t,i ; Classification loss L of source and target objects s and L t They are: Where n is the number of remaining branches, b is the batch size, L is the number of categories, σ is the signal function, and y s,ij represents the label of the kth sample of the i-th source object, y t,ij represents the label of the jth sample of the i-th branch of the target object, p s,ij,k represents the probability that the label of the jth sample of the i-th source object is predicted to be the k-th class, p t,ij,k Represents the probability that the label of the jth sample of the i-th branch of the target object is predicted to be the k-th class.

8. The multi-branch multi-object EEG data training method based on adaptive pruning according to claim 1, characterized in that: In step 8, the classification loss of the comprehensive decision of the target object is: in is the weight w of the i-th branch i The value after activation using the softmax function, p t,i is the predicted probability vector of the remaining i-th branch target object, n is the number of remaining branches, K is the number of categories, σ represents the signal function, b is the batch size, and y t,j is the label of the kth target data sample, is the probability that the jth target data sample is predicted to be class k; the final total loss is: Where L s and L t are the classification losses of the source object and the target object respectively; w i The update method is as follows: Where η is the learning rate.

9. A computer system comprising a memory, a processor, and a computer program / instruction stored in the memory and executable on the processor, characterized in that: When the computer program / instructions are executed by a processor, the steps of a multi-branch multi-object EEG data training method based on adaptive pruning according to any one of claims 1 to 8 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of a multi-branch multi-object EEG data training method based on adaptive pruning according to any one of claims 1 to 8 are implemented.

Images