A Domain Generalization EEG Signal Classification Method Based on Dual Supervision

Abstract

The present invention provides a domain generalization electroencephalogram (EEG) signal classification method based on dual supervision, comprising the following steps: preprocessing the EEG signals; establishing a double-layer supervision mechanism, linearly interpolating the preprocessed EEG data in the time-frequency domain, performing class-related data augmentation to achieve implicit supervision; using a subject-invariant feature learning module to impose weight constraints on the EEG samples, and establishing a more accurate mapping relationship between the invariant class-related EEG features and the corresponding labels to achieve explicit supervision; introducing the double-layer supervision mechanism into a self-supervised model, weighted summing the classification loss and the consistency loss of the self-ensemble model, and combining them with the EEG sample weight constraints to jointly form a double-layer objective optimization model, and alternately updating the weight parameters and network parameters through backpropagation until the accuracy converges to obtain the final classification model. The present invention can improve the accuracy and generalization of EEG signal recognition, and ensure the accurate recognition of EEG signals across different subjects.

Description

Technical Field

[0001] The present invention relates to a domain generalization electroencephalogram (EEG) signal classification method based on dual supervision, belonging to the field of EEG signal recognition. Background Art

[0002] With the rapid development of artificial intelligence technology, deep learning methods have become the mainstream methods in EEG signal recognition tasks. Based on the assumption that the training data and test data follow the same distribution, most current deep learning methods show excellent EEG classification effects. In actual application scenarios, generalizing deep EEG signal recognition methods to EEG signal recognition tasks of new subjects not seen before is an important method to realize the practical application of deep EEG signal recognition methods. However, due to the differences in the distributions between the EEG signals of different subjects, directly applying a deep EEG signal recognition method specific to a certain subject to other subjects will inevitably result in a decline in classification performance. To address this problem, researchers have proposed two major categories of solutions: learning invariant features and data augmentation. Although certain effects have been achieved, both of these two types of strategies are borrowed from the field of machine vision and are not applicable to EEG signal processing. Considering the above reasons, the present invention proposes a domain generalization method based on a dual supervision mechanism, designs a class-related EEG data augmentation supervision strategy and a subject-invariant feature learning supervision strategy, and combines a self-ensemble model to achieve accurate recognition of EEG signals across different subjects. Summary of the Invention

[0003] The present invention provides a domain generalization EEG signal classification method based on dual supervision, designs and introduces a class-related EEG data augmentation supervision module and a subject-invariant feature learning supervision module, and combines a self-ensemble model to achieve accurate recognition of EEG signals across different subjects.

[0004] The present invention adopts the following technical solutions to solve the above problems:

[0005] 1. A domain generalization EEG signal classification method based on dual supervision, comprising the following steps:

[0006] Step 1: Collect EEG data from s subjects;

[0007] Step 2: Preprocess the EEG data of s subjects, including intercepting a specific time period and band-pass filtering, to obtain an EEG data training set where and respectively represent the i-th EEG data of the s-th subject and the corresponding class label;

[0008] Step 3: Randomly select EEG data of the same class as from the EEG data of the remaining subjects, and combine this EEG data with Convert from the time domain to the time-frequency domain, perform data averaging and segmentation along the time domain direction, linearly interpolate their corresponding segmented segments, perform data recombination, and finally convert the newly recombined EEG data from the time-frequency domain to the time domain to obtain the corresponding augmented EEG data until all the EEG data of all subjects are traversed to obtain the augmented EEG data where N represents the sum of the EEG data of s subjects. Repeat step 3 to obtain another set of augmented EEG data Note that here the EEG signals of the training datasets of s subjects are combined, so the superscript of xi is no longer added to distinguish them. Unless otherwise specified, this description will be used hereinafter

[0009] Step 4: Use the EEG signal decoding network Shal low ConvNet as the backbone network of the self-ensemble model, and input the augmented EEG signals and into the student network and the teacher network in the self-ensemble model respectively

[0010] Step 5: The output features of the student network in step 4 pass through the subject-invariant feature learning module to calculate the weights of all EEG samples, thereby removing the dependencies between features

[0011] Step 6: Multiply the probability output obtained by passing the output features of the student network in step 4 through the fully connected layer and the Softmax layer and the true labels by the weights of the corresponding samples, and calculate the classification loss

[0012] Step 7: Multiply the probability output obtained by passing the output features of the student network in step 4 through the fully connected layer and the Softmax layer, and the probability output obtained by the teacher network in the same way, by the weights of the corresponding samples, and calculate the consistency loss

[0013] Step 8: Weightedly sum the classification loss in step 6 and the consistency loss in step 7 to form the final loss function, and the sample weight constraint in step 5, jointly constitute a two-layer optimization problem, and alternately update the weight parameters and network parameters through backpropagation to optimize the entire solution process

[0014] Preferably, the category-related EEG data augmentation strategy is the implicit supervision in the dual supervision of the present invention. Specifically, first, the time-domain EEG signal is converted into time-domain features through the short-time Fourier transform. Then, the time-frequency domain features are segmented, and linear interpolation is performed on each segmented feature and the corresponding segment of the same-class samples, and the interpolated segments are spliced. Finally, the spliced features are converted into time-domain EEG signals through the inverse short-time Fourier transform to obtain the augmented EEG signals. Thus, the category-related EEG data augmentation is realized, which is the implicit supervision in the dual supervision of the present invention

[0015] Preferably, the subject invariant feature learning module is the explicit supervision in the dual supervision of the present invention. Through this module, the weight corresponding to each EEG data is calculated to remove the mutual dependence between different EEG features.

[0016] Preferably, in the domain generalization EEG classification method based on dual supervision, a dual supervision mechanism is introduced into the self-ensemble model. First, the augmented data and are respectively input into the student network and the teacher network in the self-ensemble model. Then, the classification loss and the consistency loss of the self-ensemble model are weighted and summed, and the sample weights calculated by the subject invariant feature learning module are used as constraints to jointly form the following double-layer optimization problem:

[0017]

[0018] where the parameter λ c is used to balance and By alternately updating the parameter α of the weight function and the parameter θ of the student network S , the objective function can be optimized. The parameter θ of the teacher network T is obtained by the exponential smoothing method. and The calculation formulas are:

[0019]

[0020] Here, N represents the sum of the EEG data of s subjects, and respectively represent the cross-entropy loss function, the classification loss, and the consistency loss function. f s and f t respectively represent the feature extraction network of the student model and the feature extraction network of the teacher model. g s and g t respectively represent the classification networks of the two models. Through the above optimization process, the invariant features of the subjects can be better learned, which is the explicit supervision in the dual supervision of the present invention.

[0021] Beneficial effects:

[0022] 1. The present invention introduces an inter-subject category-related data augmentation mechanism. By segmenting, linearly interpolating, and splicing the EEG signals of the same category of multiple subjects in the time-frequency domain, augmented EEG data is finally generated. The proposed new EEG signal augmentation strategy can cross the subject space while retaining the information of the same category.

[0023] 2. The method introduces a subject-invariant feature learning module to decorrelate all EEG features, thereby eliminating the dependence between features and enabling the model to establish a more reasonable mapping relationship between subject-invariant features and corresponding class labels.

[0024] 3. By introducing the above dual-supervision mechanism into the self-ensemble model, the model's ability to learn domain-invariant features is enhanced, and the recognition effect of the model on EEG data of new subjects is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a network framework diagram of the domain generalization EEG signal classification method based on dual supervision in the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0026] The following further explains the present invention with reference to examples.

[0027] The main implementation process of the present invention is as follows. For the relevant process, see Figure 1 .

[0028] Step 1: EEG data collected from s subjects;

[0029] Step 2: Preprocess the EEG data of s subjects, including intercepting a specific time period and band-pass filtering, to obtain an EEG data training set where and respectively represent the i-th EEG data of the s-th subject and the corresponding class label;

[0030] Step 3: Randomly select EEG data of the same category as from the EEG data of the remaining subjects. Convert both of them from the time domain to the time-frequency domain, and successively complete data segmentation, linear interpolation, and data recombination. Finally, convert the newly recombined EEG data from the time-frequency domain to the time domain until all the EEG data of all subjects are traversed to generate augmented EEG signal data. The specific calculation is as follows:

[0031] 1) Randomly select an EEG signal X i , i = 1, 2,..., N, and use the short-time Fourier transform to convert it to a time-frequency representation Here, c, r, and t respectively represent the number of channels, the frequency band range, and the number of time points. Then, along the time dimension, divide T i into K consecutive and non-overlapping segments on average At this time, T i can be re-expressed as:

[0032]

[0033] 2) Select the EEG signal X iThe EEG signal X of any subject in the same category j Perform the processing in step 1) above to obtain T j , use linear interpolation for the k-th segment of the two samples, and the specific method is as follows:

[0034]

[0035] Among them, and respectively represent the k-th segment of T i and T j . λ is the balance coefficient, and its value is taken from the uniform distribution λ~U(0, ρ). The parameter ρ is used to control the augmentation intensity of the sample. For the newly formed data segment Concatenate them in order, and through the inverse short-time Fourier transform, convert the time-frequency domain data to the time domain. Until all the EEG data of all subjects are traversed, a set of augmented EEG signal data

[0036] 3) Repeat the above steps 1) and 2) to obtain another set of augmented EEG signal data

[0037] Step 4: Use the EEG signal decoding network Shallow ConvNet as the backbone network of the self-ensemble model, and input the augmented EEG signals and into the student network and the teacher network in the self-ensemble model respectively;

[0038] Step 5: The output features of the student network in step 4 pass through the subject-invariant feature learning module to obtain the weights of all samples, thereby removing the dependencies between features. The specific calculation is as follows:

[0039] 1) Calculate the independence test statistic of the output features of the student network in step 4 Here represents the partial covariance matrix, and the calculation is as follows:

[0040]

[0041] Among them, represents the output features of the student network, and d represents the dimension of the output features. Z p1 , Z p2 , …, Z pN and Z q1 , Z q2 , …, Z qN are sampled from Z ：，p and Z ：，q distributions respectively. and Among them Denote the random Fourier function space:

[0042]

[0043] Here, \(N(\cdot)\) and \(U(\cdot)\) denote the standard normal distribution and the uniform distribution, respectively. Denote the sample weights, and we have

[0044] 2) Assign weights to each sample feature, and the partial covariance matrix can be re-expressed as follows:

[0045]

[0046] where Denote the sample weights, and we have

[0047] Step 6: Multiply the probability output obtained by passing the output features of the student network in Step 4 through the fully connected layer and the Softmax layer by the corresponding sample weights \(\alpha\) and the true labels, and calculate the classification loss: i (obtained in Step 5)

[0048]

[0049] where Denote the feature extraction network of the student model, Denote the classification network of the student model. \(l\) ce \(l(\cdot)\) denotes the cross-entropy loss function.

[0050] Step 7: Multiply the probability output obtained by passing the output features of the student network in Step 4 through the fully connected layer and the Softmax layer, and the probability output obtained by using the teacher network in the same way, by the corresponding sample weights to calculate the consistency loss:

[0051]

[0052] where Denote the feature extraction network of the teacher model, Denote the classification network of the teacher model. \(l\) c \(l(\cdot)\) denotes the consistency loss function.

[0053] Step 8: Weighted sum the classification loss in Step 6 and the consistency loss in Step 7, and the constraints of the sample weights to form the final objective function, which is expressed as the following bilevel optimization problem:

[0054]

[0055] where the parameter \(\lambda\) c is used to balance and By alternately updating the parameter α of the weight function and the student network parameter θ S the objective function can be optimized. Additionally, the teacher network parameter θ T can be obtained using the exponential smoothing method:

[0056]

[0057] where represents the student network parameter at the t-th iteration. respectively represent the teacher network parameters at the t-th and (t - 1)-th iterations.

[0058] The above are only embodiments of the present invention, and do not limit the patent scope of the present invention accordingly. Any equivalent structure or equivalent process transformation made using the content of the specification and drawings of the present invention, or directly or indirectly applied in the relevant technical fields, shall be similarly included in the patent protection scope of the present invention.

Claims

1. A domain generalization electroencephalogram signal classification method based on dual supervision, characterized in that, The steps are as follows: Step 1: Collect EEG data from S subjects; Step 2: Preprocess the EEG data of S subjects, including intercepting a specific time period and band-pass filtering, to obtain an EEG training dataset where and represent the i-th EEG sample of the S-th subject and the corresponding class label, respectively; Step 3: Randomly select EEG signal data of the same category from the EEG data of the remaining subjects, convert them to the time-frequency domain, segment them, linearly interpolate and reorganize the segmented segments, and then convert the newly reorganized EEG data from the time-frequency domain to the time domain until all the EEG data of all subjects are traversed to obtain augmented EEG signal data Repeat Step 3 to obtain the augmented EEG signal Note that the EEG signals of the training datasets of S subjects are combined here, so the superscript is no longer added to X for distinction; unless otherwise specified, this description is used hereinafter; Repeat Step 3 to obtain the augmented EEG signal i Note that the EEG signals of the training datasets of S subjects are combined here, so the superscript is no longer added to X for distinction; unless otherwise specified, this description is used hereinafter; Step 4: Use the electroencephalogram (EEG) signal decoding network Shallow ConvNet as the backbone network of the self-ensemble model, and input the augmented EEG signals and into the student network and the teacher network in the self-ensemble model respectively; Step 5: The output features of the student network in Step 4 pass through the subject-invariant feature learning module to calculate the weights of all EEG samples, thereby removing the dependencies between features; Step 6: Multiply the probability output obtained by passing the output features of the student network in Step 4 through the fully connected layer and the Softmax layer with the true labels, and calculate the classification loss; Step 7: Multiply the probability output obtained by passing the output features of the student network in Step 4 through the fully connected layer and the Softmax layer, and the probability output obtained by the teacher network in the same way, with the weights of the corresponding samples, and calculate the consistency loss; Step 8: Weightedly sum the classification loss in Step 6 and the consistency loss in Step 7 to form the final loss function, along with the sample weight constraint in Step 5, to jointly constitute a two-layer optimization problem. Update the weight parameters and network parameters alternately through backpropagation until the accuracy converges to obtain the final classification model; Step 9: Input the EEG signals of new subjects into the final classification model to obtain the predicted class labels corresponding to each sample.

2. The domain generalization electroencephalogram signal classification method based on dual supervision according to claim 1, wherein In step 3, the EEG data augmentation strategy related to the category, specifically, first, for the time-domain EEG data X after preprocessing i Use the short-time Fourier transform to convert it from time-domain data to time-frequency domain data T i ; then, along the time dimension, the time-domain EEG data is evenly segmented to obtain K consecutive and non-overlapping segments From the EEG data collected from S subjects, select the EEG signals X of other subjects in the same category as the EEG data X i ; repeat the above steps to obtain j ; for the k-th segment of the new T and T i perform linear interpolation j where and and represent the k-th segments of T i and T j respectively, and λ is the balance coefficient; finally, reorganize in order and convert the time-frequency domain data representation back to time-domain data until all the EEG data of the subjects are traversed, and finally the augmented EEG signal.

3. A method for classifying electroencephalogram signals in domain generalization based on dual supervision according to claim 1, characterized in that, Introduce the described dual supervision mechanism into the self-ensemble model. First, augment the data and are input into the student network and the teacher network in the self-ensemble model respectively. Then, the classification loss and the consistency loss of the self-ensemble model are weighted and summed, and are jointly constrained by the sample weights calculated by the subject-invariant feature learning module to jointly form the following two-layer optimization problem: Among them, the parameter λ c is used to balance and By alternately updating the parameters α of the weight function and the parameters θ of the student network S the objective function can be optimized, and the parameters θ of the teacher network T are obtained by the exponential smoothing method; and The calculation formula of is: Here, N represents the sum of the EEG data of S subjects, l ce (·), and l c (·) represent the cross-entropy loss function, classification loss, and consistency loss function respectively, f s and f t represent the student model feature extraction network and the teacher model feature extraction network respectively, g s and g t represent the classification networks of the two models respectively; through the above optimization process, better learning of the invariant features of the subjects is the explicit supervision in the dual supervision of the present invention.

Images