Method for multi-source domain adaptive electroencephalogram signal classification based on weighted prototype and graph convolution

Abstract

The present application relates to the field of motor imagery electroencephalogram signal classification, and specifically provides a multi-source domain adaptive electroencephalogram signal classification method based on weighted prototypes and graph convolution, comprising the following steps: S1: data preprocessing and data division are performed on the electroencephalogram signal; S2: model training: the features of the target domain and each source domain are respectively aligned on the whole by using MMD loss, and multiple graph convolution operations are designed to assist the alignment; class-level fine-grained alignment is realized by weighting the class prototypes of each source domain to obtain joint source class prototypes; the network is updated in parameters and the optimal parameters are obtained by using the MMD loss of the source domain and the target domain features, the classification cross-entropy loss of multiple source domains and the contrast loss of the target domain features and the joint source domain prototypes; S3: model testing: the optimal parameters obtained in S2 are used to test the unlabeled sample data to be tested. The present scheme can realize sufficient and effective utilization of data.

Description

Technical Field

[0001] This invention relates to the field of motor imagery EEG signal classification technology, and in particular to a multi-source domain adaptive EEG signal classification method based on weighted prototype and graph convolution. Background Technology

[0002] Brain-computer interfaces (BCIs) based on motor imagery EEG signals can directly convert brain activity into command signals, assisting disabled patients and improving their quality of life. Due to their convenience and safety, they have been widely used. Currently, the following are the classifications of EEG signal designs in patents and literature:

[0003] 1. The invention patent application with patent number "202111003357.1" and patent name "A domain adaptation method for solving the feature transfer problem in motor imagery brain-computer interface" uses labeled session data of the same user as the source domain and unlabeled session samples as the target domain. The source domain and target domain samples are mapped separately to ensure that the distribution of the source domain and target domain is consistent after mapping. This can solve the problem of classifier performance degradation caused by feature deviation of different domains in motor imagery brain-computer interface.

[0004] 2. The invention patent application with patent number "202111598515.2" and patent name "A privacy-preserving transfer learning method for motor imagination brain-computer interface" uses data from all previous users as the source domain and unlabeled data from new users as the target domain. The source domain data is used to pre-train the model. After pre-training, the source domain data does not participate in the alignment process with the target domain. The target domain model is directly initialized using the pre-trained model. Then, uncertainty reduction and consistency regularization are considered for retraining. While ensuring the privacy of the source domain data, the distance between the source domain and the target domain is reduced as much as possible to achieve feature alignment.

[0005] 3. The article titled "Deep Representation-Based Domain Adaptation for Nonstationary EEG Classification," published in IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL.32, NO.2, FEBRUARY 2021, and authored by He Zhao, Qingqing Zheng, Kai Ma, Huiqi Li, and Yefeng Zheng, proposes an end-to-end domain adaptation method to address the distributional differences among different users. This paper aggregates all users except the test user into a single source domain, and uses the labeled session data of the current test user as the target domain. An adversarial learning mechanism using a domain discriminator is employed to match deep features between different domains, thereby improving the model's classification performance on unlabeled session data of the test user.

[0006] However, the aforementioned existing technology has the following problems:

[0007] 1. When classifying motor imagery signals in brain-computer interfaces, there are distribution differences between data from different users or different sessions of the same user; existing technologies do not utilize other data within the dataset, leading to information confusion and incomplete information utilization.

[0008] 2. Existing alignment methods do not consider category information or specific structural information that is not shared between different user samples. Forcibly aligning the target domain with multiple sub-source domains can easily lead to negative transfer. Furthermore, some studies only use global alignment methods to reduce inter-domain differences. Although the classification loss model of the source domain is used to constrain the model, the sample category information of each domain is not utilized to the maximum extent.

[0009] In conclusion, designing a method for classifying motor imagery EEG signals that can effectively utilize all data depth is a problem that urgently needs to be solved. Summary of the Invention

[0010] To solve the above problems, the present invention provides a multi-source domain adaptive electroencephalogram signal classification method based on weighted prototypes and graph convolution, which globally aligns the features of the target domain and each source domain using the MMD loss, and designs multiple graph convolution operations to assist in the alignment; realizes class-level fine-grained alignment by weighting the class prototypes of each source domain to obtain a joint source class prototype; updates the parameters of the network through the MMD loss between the source domain and the target domain features, the classification cross-entropy loss of multiple source domains, and the contrast loss between the target domain features and the joint source domain prototypes, and obtains the optimal parameters; this solution can retain the structural information that is not shared between samples in each domain, prevent negative transfer of the domain, avoid weakening of the classification performance of the model on the target domain, and realize the full and effective utilization of data.

[0011] To achieve the above object, the present invention proposes the following technical solutions: A multi-source domain adaptive electroencephalogram signal classification method based on weighted prototypes and graph convolution, comprising the following steps:

[0012] S1: Perform data preprocessing and data partitioning on the electroencephalogram signals; use the unlabeled session data of the current user as the target domain, and use the labeled session data of the current user and the data of other users as multiple individual source domains;

[0013] S2: Model training: Globally align the features of the target domain and each source domain using the MMD loss, and design multiple graph convolution operations to assist in the alignment; realize class-level fine-grained alignment by weighting the class prototypes of each source domain to obtain a joint source class prototype; update the parameters of the network through the MMD loss between the source domain and the target domain features, the classification cross-entropy loss of multiple source domains, and the contrast loss between the target domain features and the joint source domain prototypes, and obtain the optimal parameters;

[0014] S3: Model testing: Test the unlabeled sample data to be measured using the optimal parameters obtained in S2.

[0015] Preferably, the target domain in S1 is denoted as The multiple source domains are denoted as Let the number of source domains be n, then 0 < j < n + 1; where represents the electroencephalogram data collected from the target domain, the j-th source domain, the i-th and j-th electrodes, with the number of electrodes being e and the time series length being t; represents the label of the l-th data in the j-th source domain, and there are cls categories in this label; n t , respectively represent the number of data in the target domain and the j-th source domain.

[0016] Preferably, in S2, globally align the features of the target domain and each source domain using the MMD loss, and design multiple graph convolution operations to assist in the alignment, including the following steps:

[0017] S21: Initialize the network;

[0018] S22: Input n source domain data and 1 target domain data into the feature extractor and obtain n source domain outputs and 1 target domain feature output;

[0019] S23: Input n source domain outputs and 1 target domain output into the corresponding n source domain graph convolution modules and 1 target domain convolution module;

[0020] S24: Concatenate the outputs of the n+1 source and target domain feature extractors with the output of the graph convolution module;

[0021] S25: Pair one target domain feature with n source domain features;

[0022] S26: Calculate the MMD loss using the n pairs of target domain features and source domain features paired in S25, and use the MMD loss for global alignment;

[0023] S27: Calculate the weights based on the distribution differences between the source domain and the target domain, and calculate the joint source domain prototype based on the source domain class prototype and the weights.

[0024] Preferably, the formula for calculating MMD loss in S26 is as follows:

[0025]

[0026] Among them, F η This represents a feature extractor with parameter η. The parameter is ψ j The j-th source domain graph convolutional module, The parameter is ψ t The target domain graph convolution module, This indicates the concatenation of two features.

[0027] Preferably, the weight value γ of each source domain prototype is calculated according to the MMD loss using the following formula. j :

[0028]

[0029] Preferably, the method for calculating the source domain cross-entropy loss in S2 is as follows:

[0030] The concatenated n source domain features are input into n source domain classifiers respectively, and the cross-entropy loss is calculated according to the following formula:

[0031]

[0032] Preferably, before obtaining the contrast loss between the target domain features and the source domain class prototypes in S2, the parameters of each linear layer of the classifier are used as the class prototypes of each category in the current source domain, and the class prototypes of the n domains are weighted and summed according to the following formula to obtain the joint source domain prototype Q:

[0033]

[0034] Q j This represents the prototype of the j-th source domain.

[0035] Preferably, the method for calculating the contrast loss between the target domain features and the source domain class prototype in S2 is as follows: K-means clustering is performed on the concatenated target domain features, and a pseudo-label is assigned to each sample. The contrast loss is then calculated based on the sample, its corresponding pseudo-label, and the joint source domain prototype. The formula for calculating the contrast loss is as follows:

[0036]

[0037] Where f represents the features of the current target domain sample t, and c * This represents the feature in the joint source domain class prototype Q that belongs to the same category as the target feature f, where τ is the temperature factor. <f,c * >), ( <f,c k >) represents the inner product of two vectors.

[0038] Preferably, the total loss in S2 is as follows:

[0039]

[0040] in,

[0041]

[0042]

[0043]

[0044] Where α and β are known hyperparameters, i.e., parameters controlling the proportion of the loss function; η is the parameter of the acquired feature extractor; θ j ψ represents the parameters of the acquired classifier. j Let be the parameters of the obtained source domain graph convolution module, where 0 <j<n+1;ψ t These are the parameters of the target domain graph convolution module obtained.

[0045] Preferably, S3 includes the following sub-steps:

[0046] S31: Input the unlabeled test data into the feature extractor;

[0047] S32: Input the output of the feature extractor into the target domain graph convolution module;

[0048] S33: Concatenate the output of the feature extractor with the output of the target domain graph convolution module;

[0049] S34: Input the concatenated features into n classifiers to obtain the predicted probabilities of n k categories;

[0050] S35: Load the weights γ1, γ2, ... γ1 retained from model training termination in S2. n ;

[0051] S36: The joint probability is obtained by weighting and summing the n weights with the outputs of the n classifiers;

[0052] S37: Infer the classification result of the current test data through joint probability.

[0053] The beneficial effects of this invention are:

[0054] 1. This invention considers the distribution differences between sessions and users. It takes each user other than the user under test and the labeled session data of the user under test as a single source domain, and the unlabeled session of the user under test as the target domain. Through the perspective of multi-source domain adaptation, the features of the target domain and each source domain are globally aligned using MMD loss, which can achieve full and effective use of data.

[0055] 2. In this invention, a graph convolution module with non-shared parameters is set in each domain. Based on prototype and graph convolution, it considers both category information and the unique non-shared inter-sample structure information in each domain. By collecting the non-shared inter-sample structure information through the graph convolution module, it can assist in domain alignment and prevent negative domain transfer. By weighting the class prototypes of the source domain to obtain the joint source class prototype, fine-grained alignment at the class level is achieved, so as to make full use of the category information in each domain in a focused manner.

[0056] 3. This invention assigns pseudo-labels to the target domain using K-means. By fully utilizing the potential category information in each domain during alignment based on the pseudo-labels and weighted source prototypes, the full and effective use of data can be achieved, thereby improving the model's information capture capability. Attached Figure Description

[0057] Figure 1 This is a model training architecture diagram provided in an embodiment of the present invention.

[0058] Figure 2 This is a model testing architecture diagram provided in an embodiment of the present invention.

[0059] Figure 3 This is a flowchart of the model training process provided in an embodiment of the present invention.

[0060] Figure 4 It is the flow chart of model testing provided by an embodiment of the present invention. Specific embodiments

[0061] In order to make the objectives, technical solutions and advantages of the present invention more clear and understandable, the following further elaborates on the present invention in conjunction with Figure 1-4 the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not constitute a limitation to the present invention.

[0062] A multi-source domain adaptive electroencephalogram (EEG) signal classification method based on weighted prototypes and graph convolution includes the following steps:

[0063] S1: Perform data preprocessing on the EEG signals: First, filter the existing EEG signals through a third-order Butterworth band-pass filter to filter the EEG signals to [4 - 38 Hz] to extract the most relevant rhythms; use exponential moving to perform normalization operations on the filtered EEG signals to eliminate unnecessary non-stationarity and abnormal signal fluctuations. <​​​​​​​​​​​​​​​​​​​​​​

[0068] S22: Input n source domain data and 1 target domain data into the feature extractor and obtain n source domain outputs and 1 target domain feature output;

[0069] S23: Input n source domain outputs and 1 target domain output into the corresponding n source domain graph convolution modules and 1 target domain convolution module;

[0070] S24: Concatenate the outputs of the n+1 source and target domain feature extractors with the output of the graph convolution module;

[0071] S25: Pair one target domain feature with n source domain features;

[0072] S26: Calculate the MMD loss using the n pairs of target domain features and source domain features paired in S25, and use the MMD loss for global alignment;

[0073] S27: Calculate the weights based on the distribution differences between the source domain and the target domain, and calculate the joint source domain prototype based on the source domain class prototype and the weights.

[0074] The formula for calculating MMD loss is as follows:

[0075]

[0076] Among them, F η This represents a feature extractor with parameter η. The parameter is ψ j The j-th source domain graph convolutional module, The parameter is ψ t The target domain graph convolution module, This indicates the concatenation of two features.

[0077] The weight value γ of each source domain prototype is calculated according to the MMD loss using the following formula. j :

[0078]

[0079] The method for calculating the source domain cross-entropy loss is as follows: Input the concatenated n source domain features into n source domain classifiers respectively, and calculate the cross-entropy loss according to the following formula (where is the cross-entropy loss of the j-th source domain):

[0080]

[0081] The parameters of each linear layer of the classifier are used as class prototypes for each category in the current source domain, and the class prototypes of the n domains are weighted and summed according to the following formula to obtain the joint source domain prototype:

[0082]

[0083] Q j This represents the prototype of the j-th source domain.

[0084] The method for calculating the contrast loss between target domain features and source domain class prototypes is as follows: K-means clustering is performed on the concatenated target domain features, and a pseudo-label is assigned to each sample. The contrast loss is then calculated based on the sample, its corresponding pseudo-label, and the joint source domain prototype to further constrain the model. The formula for calculating the contrast loss is as follows:

[0085]

[0086] Where f represents the features of the current target domain sample t, and c * This represents the feature in the joint source domain class prototype Q that belongs to the same category as the target feature f, where τ is the temperature factor. <f,c * >), ( <f,c k >) represents the inner product of two vectors.

[0087] The network parameters are updated using source domain cross-entropy loss, MMD loss of source and target domain features, and contrast loss between target domain features and source domain class prototypes. The total loss is as follows:

[0088]

[0089] in,

[0090]

[0091]

[0092] Where α and β are known hyperparameters, i.e., parameters controlling the proportion of the loss function; η is the parameter of the acquired feature extractor; θ j ψ represents the parameters of the acquired classifier. j Let be the parameters of the obtained source domain graph convolution module, where 0 <j<n+1;ψ t These are the parameters of the target domain graph convolution module obtained.

[0093] S3: Model Testing: Using the optimal parameters obtained in S2, the model is tested on the unlabeled sample data to be tested. This includes the following sub-steps:

[0094] S31: Input the unlabeled test data into the feature extractor;

[0095] S32: Input the output of the feature extractor into the target domain graph convolution module;

[0096] S33: Concatenate the output of the feature extractor with the output of the target domain graph convolution module;

[0097] S34: Input the concatenated features into n classifiers to obtain the predicted probabilities of n k categories;

[0098] S35: Load the weights γ1, γ2, ... γ1 retained from model training termination in S2. n ;

[0099] S36: The joint probability is obtained by weighting and summing the n weights with the outputs of the n classifiers;

[0100] S37: Infer the classification result of the current test data through joint probability.

[0101] Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

[0102] The specific embodiments of the present invention described above do not constitute a limitation on the scope of protection of the present invention. Any other corresponding changes and modifications made in accordance with the technical concept of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A multi-source domain adaptive EEG signal classification method based on weighted prototype and graph convolution, characterized in that, Includes the following steps: S1: Perform data preprocessing and data segmentation on EEG signals; use the current user's unlabeled session data as the target domain, and use the current user's labeled session data and other user data as multiple separate source domains; S2: Model Training: The features of the target domain and each source domain are globally aligned using MMD loss, and multiple graph convolution operations are designed to assist the alignment; the class prototypes of each source domain are weighted to obtain a joint source class prototype to achieve fine-grained alignment at the class level; the network parameters are updated and the optimal parameters are obtained by using the MMD loss of source domain and target domain features, the classification cross-entropy loss of multiple source domains, and the contrast loss between target domain features and joint source domain prototypes. S3: Model testing: Test the unlabeled sample data to be tested using the optimal parameters obtained in S2; Among them, the target domain in S1 is denoted as Multiple source domains are denoted as Let the number of source domains be n, then 0 < j < n + 1; where denotes the electroencephalogram data collected from the target domain and the j-th source domain, where the number of electrodes for the i-th and j-th is e, and the length of the time series is t; denotes the label of the l-th data in the j-th source domain, and there are cls categories in total for this label; n t , respectively denote the number of data in the target domain and the j-th source domain; In S2, the features of the target domain and each source domain are globally aligned using MMD loss, and multiple graph convolution operations are designed to assist in the alignment, including the following steps: S21: Initialize the network; S22: Input n source domain data and 1 target domain data into the feature extractor and obtain n source domain outputs and 1 target domain feature output; S23: Input n source domain outputs and 1 target domain output into the corresponding n source domain graph convolution modules and 1 target domain convolution module; S24: Concatenate the outputs of the n+1 source and target domain feature extractors with the output of the graph convolution module; S25: Pair one target domain feature with n source domain features; S26: Calculate the MMD loss using the n pairs of target domain features and source domain features paired in S25, and use the MMD loss for global alignment; S27: Calculate the weights based on the distribution differences between the source domain and the target domain, and calculate the joint source domain prototype based on the source domain class prototype and the weights; The formula for calculating the MMD loss in S26 is as follows: Among them, F η This represents a feature extractor with parameter η. The parameter is ψ j The j-th source domain graph convolutional module, The parameter is ψ t The target domain graph convolution module, This indicates the concatenation of two features.

2. The multi-source domain adaptive EEG signal classification method based on weighted prototyping and graph convolution as described in claim 1, characterized in that, According to the MMD loss, the weight value γ of each source domain prototype is calculated according to the following formula j :

3. The multi-source domain adaptive EEG signal classification method based on weighted prototyping and graph convolution as described in claim 2, characterized in that, The method for calculating the source domain cross-entropy loss in S2 is as follows: The concatenated n source domain features are input into n source domain classifiers respectively, and the cross-entropy loss is calculated according to the following formula:

4. The multi-source domain adaptive EEG signal classification method based on weighted prototyping and graph convolution as described in claim 3, characterized in that, Before obtaining the contrast loss between the target domain features and the source domain class prototypes in S2, the parameters of each linear layer of the classifier are used as the class prototypes of each category in the current source domain. The class prototypes of the n domains are then weighted and summed according to the following formula to obtain the joint source domain prototype: where Q j represents the prototype of the jthsource domain.

5. The multi-source domain adaptive EEG signal classification method based on weighted prototyping and graph convolution as described in claim 4, characterized in that, The method for calculating the contrast loss between the target domain features and the source domain class prototype in S2 is as follows: K-means clustering is performed on the concatenated target domain features, and a pseudo-label is assigned to each sample. The contrast loss is then calculated based on the sample, its corresponding pseudo-label, and the joint source domain prototype. The formula for calculating the contrast loss is as follows: where f is the feature of the current target domain sample t, c * represents the features in the joint source domain class prototype Q that are in the same class as the target feature f, τ is a temperature factor, (<f, c * >) and (<f, c k >) represent the inner product of two vectors.

6. The multi-source domain adaptive EEG signal classification method based on weighted prototyping and graph convolution according to claim 5, characterized in that, The total loss in S2 is as follows: in, wherein a, b are known hyperparameters, i.e. parameters for controlling the proportion of the loss function; η is the parameter of the obtained feature extractor; θ j is the parameter of the obtained classifier; ψ j is the parameter of the obtained source domain graph convolution module, wherein 0 < j < n + 1; ψ t is the parameter of the obtained target domain graph convolution module.

7. The multi-source domain adaptive EEG signal classification method based on weighted prototyping and graph convolution according to any one of claims 1-6, characterized in that, S3 includes the following sub-steps: S31: Input the unlabeled test data into the feature extractor; S32: Input the output of the feature extractor into the target domain graph convolution module; S33: Concatenate the output of the feature extractor with the output of the target domain graph convolution module; S34: Input the concatenated features into n classifiers to obtain the predicted probabilities of n k categories; S35: Load the weights γ1, γ2, ... γ1 retained from model training termination in S2. n ; S36: The joint probability is obtained by weighting and summing the n weights with the outputs of the n classifiers; S37: Infer the classification result of the current test data through joint probability.

Images