A multi-source domain EEG signal analysis method with multi-modal representation

Abstract

The present invention proposes a multi-source domain EEG signal analysis method with multi-modal representation. The present invention first uses multi-manifold mapping to extract the common invariant representation of the multi-source domain and the target domain, while taking into account the low-dimensional structure and multivariate statistical characteristics of the EEG signal. At this stage, the CORAL loss is calculated to guide the model to obtain high-quality common invariant representation. Secondly, the multi-source domain is decomposed and one-to-one feature extraction is performed. At this time, the MMD loss is used to guide the model to obtain high-quality private invariant representation. Finally, a softmax classifier is used for classification. The effectiveness of the method was evaluated on the public MI1 and MI2 datasets, as well as the EEG signal dataset collected by the team. Experimental results show that the present invention performs superiorly in multi-subject scenarios.

Description

Technical Field

[0001] The present invention belongs to the field of signal processing and relates to a multi-source domain electroencephalogram signal analysis method with multi-modality representation. Background Art

[0002] As an important bioelectrical signal, EEG records the electrical activity of neurons at different times and locations. Its advantages include repeatability, non-invasiveness, and low cost, and therefore hold broad application prospects in fields such as rehabilitation medicine and intelligent assistance. However, due to the non-stationary and nonlinear characteristics of EEG signals, as well as the variability between different populations, how to effectively characterize and analyze EEG signals has become a hot and challenging research topic.

[0003] In recent years, artificial intelligence technology has been widely applied and researched in the field of brain-computer interfaces. Traditional machine learning methods have achieved great success in EEG signal recognition. In the field of motor imagery research, the most classic is the Common Space Pattern (CSP). This method converts multi-channel EEG signals into a set of new spatial feature vectors through linear transformation, making it possible to better separate signals related to a specific task. In the field of driver fatigue detection, the short-time Fourier transform (STFT) has been widely used for time-frequency domain analysis and has achieved excellent results.

[0004] These methods have achieved good results in some EEG signal classification tasks. However, these traditional machine learning methods often use manual feature extraction, which leads to certain limitations and deficiencies in feature extraction and classification performance. Unlike traditional machine learning methods, deep learning methods can automatically learn feature expressions from raw data and have strong representation and generalization capabilities. Many deep learning-based methods have emerged to address the problem of EEG signal recognition. These methods have achieved good results in feature extraction and classification tasks of EEG signals, and can effectively reduce the influence of manual intervention and domain knowledge. However, neither traditional machine learning nor deep learning methods have solved the problem of cross-subject EEG recognition well, that is, there are large differences in EEG signals between different individuals, which leads to insufficient generalization ability of the model between different subjects.

[0005] Addressing inter-subject variability and developing a robust EEG signal analysis model has important application value and research implications for future fields such as human-computer interaction and neurorehabilitation. The emergence of transfer learning has effectively addressed this issue. In recent years, an increasing number of researchers have applied transfer learning to EEG classification. H. Kang et al. proposed a composite covariance matrix method by linearly combining the target domain subject covariance matrix with the source domain subject covariance matrix. Summary of the Invention

[0006] In order to improve the generalization capability of multi-source domain models and address the deficiencies of the prior art, the present invention proposes a multi-source domain EEG signal analysis method with multi-modal representation.

[0007] The large variability of EEG data between subjects makes it difficult for models to achieve strong generalization across multiple source domains. Transfer learning can help find invariant representations across subjects. The proposed method constructs a common feature extractor through two-manifold space mapping. This common feature extractor preserves the multivariate statistical characteristics and low-dimensional structural information of EEG signals through polymorphic representation.

[0008] A multi-source domain EEG signal analysis method with multi-modal representation, the method comprising the following steps:

[0009] Step 1: First, preprocess the EEG signal and align the preprocessed data using the CA algorithm. After alignment, the data is mapped to the Riemannian manifold to obtain the multivariate statistical feature information of the EEG signal.

[0010] Step 2: Extract the tangent space features of the entire source and target domains from the mapped multivariate statistical features, and then map them to the Grassmann manifold to obtain the common invariant representation features. At this stage, the CORAL loss between the entire multi-source domain and the target domain is calculated to ensure that the low-dimensional structural information of the EEG signal is extracted while maximizing the common invariant representation between the multi-source domain and the target domain.

[0011] Step 3: Divide the multiple source domains in the public invariant representation features into subdomains, and obtain the private invariant representation of each subdomain and the target domain through a pre-set one-to-one feature extractor. In this stage, the MMD loss and cross entropy loss are calculated between the multiple sub-source domains and the target domain; the MMD loss ensures that the differences between the multiple sub-source domains and the target domain are minimized.

[0012] Step 4: Use multiple softmax classifiers for training, and use cross entropy to reduce loss during the training process. Preferably, the CA algorithm is used in step 1 to perform the first step of alignment on the pre-processed data, specifically dividing the training data into multiple sub-source domains for separate alignment.

[0013] Preferably, in step 2, Grassmann manifold mapping is used, and the mapping dimension is 5-40.

[0014] Preferably, the mapping dimension of the Grassmann manifold mapping is 40.

[0015] Preferably, the tangent space features of the entire source domain and the target domain are extracted from the mapped multivariate statistical features. The expression for tangent space extraction is as follows:

[0016]

[0017] Among them, upper is to take the upper triangular elements of the matrix, P i is the covariance matrix, is the Riemann mean or the Euclidean mean.

[0018] Preferably, the CORAL loss between the entire multi-source domain and the target domain is calculated as follows:

[0019]

[0020] in, represents CORAL loss, d is the feature dimension, C S and C T Represent the covariance matrices of the source domain and target domain data respectively, and i,j are the element indices in the covariance matrix.

[0021] As a preference, the MMD loss between multiple sub-source domains and the target domain is calculated, specifically: the expression is as follows:

[0022]

[0023] Where N represents the number of source domains, A T represents the transformation matrix, C represents the number of categories, n s With n t Represent the total number of trials in the source domain and the target domain, respectively.

[0024] Beneficial effects of the present invention:

[0025] First, the present invention performs a special processing process when using different source domains for training, preserving the public invariant representation and private invariant representation between the multiple source domains and the target domain. Compared with the general method of processing the multi-source domain training set in the original space, it can more effectively increase the generalization ability of the model and narrow the distribution differences between different subjects, which is also not taken into account by traditional domain adaptation algorithms.

[0026] Second, the present invention utilizes multiple manifolds to obtain public invariant representations and private invariant representations of multiple source domains, wherein the Riemannian manifold is utilized to preserve the multivariate statistical feature information of the original EEG data.

[0027] Third, the present invention proposes a multi-source EEG signal analysis method (MMRA) with multi-modal representation. This method uses two manifold space mappings as common feature extractors to extract the common invariant representation of the EEG signal. It then establishes a three-layer MLP to obtain the private invariant representations of N subdomains and the target domain. This method of different modal representations effectively extracts invariant representations across different subjects. By comparing it with the current sota model on a public dataset, it was found that this method performs better when multiple source domains are used as training sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 The current main strategy for training with multiple source domains;

[0029] Figure 2 It is the main framework of the present invention;

[0030] Figure 3 Visualize data distribution;

[0031] Figure 4 Attention maps are drawn for Subject 1 and Subject 2 from the BCI IV IIa dataset; DETAILED DESCRIPTION

[0032] The following is a detailed description of an embodiment of the present invention with reference to the accompanying drawings. This embodiment is implemented using a BCI dataset based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process.

[0033] Existing methods do not fully consider that the EEG data is high-dimensional due to the way it is collected, and contains a large amount of multivariate statistical feature information and low-dimensional structural information. Secondly, when multiple source domains are used as training sets for training, directly inputting multiple source domains will ignore the private invariant representations between the sub-source domains and the target domain. Figure 1 As shown in Figure 1, there are currently two main strategies for training with multiple source domains: a) all source domains are combined into a single, large source domain, which is then used to align the distribution with the target domain; b) multiple source domains are individually aligned with the target domain. Sub-figure a considers the distributional differences between the training and test sets, but fails to account for the significant variability in EEG signals from a training set composed of multiple subjects. Sub-figure b considers the distributional differences between different subjects in the training set, but fails to consider how to identify shared attributes between multiple source domains and the target domain.

[0034] like Figure 2As shown, this implementation patent includes the following steps:

[0035] Step 1: First, perform CA alignment on the data in a Riemannian manifold. A Riemannian manifold is a space with a specific geometric structure that takes into account the multivariate statistical characteristics of the data. EEG signals typically have multiple channels, and these channel signals are preserved on the Riemannian manifold. This is done to preserve channel information while finding common invariant representations across different subjects.

[0036] In step 2, the tangent space features are mapped onto the Grassmann manifold to obtain a common invariant representation between the multiple source domains and the target domain. It is important to note that this module aims to learn a common invariant representation between the entire source domain and the target domain, so in the above steps, multiple source domains are treated as a single source domain. To ensure that the entire source domain and target domain are close on the Grassmann manifold, CORAL is used to measure the distance between the two domains.

[0037] In step 3, after obtaining features from all domains, N single fully connected layers are built to correspond to the N source domains. For each pair of source and target domains, the data is mapped to a unique latent space using a corresponding one-to-one feature extractor, and domain-specific features are then obtained for each branch. To apply DA and bring the two domains close in the latent space, MMD is selected to estimate the distance between the two domains.

[0038] In step 4, the features accepted by the classifier come from the one-to-one feature extractor. After the one-to-one feature extractor, we have N training data, each of which corresponds to a softmax classifier. For each classifier training, cross entropy is used to estimate the classification loss.

[0039] Figure 3 a in the figure is the distribution diagram of the original signal after CSP spatial filtering. It can be seen that the distribution difference between the source domain and the target domain is very obvious. Figure 3 b in the figure is the covariance matrix of the original signal after CA alignment of the target domain and the source domain, and then the t-SNE of the features obtained by the multi-manifold common feature extractor. The data distribution difference after the multi-manifold common feature extractor is reduced, but the generalization ability is still insufficient. Figure 3 The c in the figure is the distribution graph after MMRA processing. Under this algorithm, the marginal distribution and conditional distribution of the target domain and the source domain are well aligned, eliminating the differences between the domains and giving the model strong generalization ability. This effect cannot be achieved by traditional methods.

[0040] like Figure 4As shown, the first and second rows show topographic maps for subject 1 labeled as the left hand. It can be seen that after the Riemannian manifold mapping, the original signal still maintains high activity in channels C3, C4, and Cz, consistent with previously described physiological properties of the human brain. The third and fourth rows show topographic maps for subject 1 labeled as the right hand. Within the purple circles, activity remains on the right side after the Riemannian manifold mapping, reflecting the EEG signal's characteristic of event-related desynchronization (ERD) potentials in the contralateral brain regions and event-related synchronization (ERS) potentials in the ipsilateral brain regions. The last two rows show EEG data for subject 2. Combined with the third row, it can be seen that brain activity levels vary across subjects during motor imagery, but the active regions remain consistent (red circles). Brain topography contains a wealth of interesting information. Varying activity levels may be one of the reasons for the poor performance of cross-subject classification models. This demonstrates that the use of the Riemannian manifold preserves the multivariate statistical characteristics of the EEG signal, which is beneficial for subsequent processing steps.

[0041] In summary, the present invention accepts N source domain data as a training set and a target domain as a test set. CA alignment is used to distribute the original EEG data closer, and then a multi-manifold public feature extractor is used to obtain a multi-source domain feature and a target domain feature, while calculating the CORAL loss between these features. Then, N one-to-one private feature extractors are used to decompose the multi-source domain into N single source domains, which are mapped to the latent space separately with the target domain, while solving the MMD loss of the N source domain features and the target domain features. Finally, the features obtained by the one-to-one private feature extractor are used for classification, and the classification loss of the N source domains is calculated.

Claims

1. A multi-source domain EEG signal analysis method with multi-modal representation, characterized by: The method comprises the following steps: Step 1: First, preprocess the EEG signal and align the preprocessed data using the CA algorithm. After alignment, the data is mapped to the Riemannian manifold to obtain the multivariate statistical feature information of the EEG signal. Step 2: Extract all tangent space features of the source and target domains from the mapped multivariate statistical features, and then map them to the Grassmann manifold to obtain the common invariant representation features. At this stage, calculate the CORAL loss between all source domain signals and the target domain to ensure that the low-dimensional structural information of the EEG signal is extracted while maximizing the common invariant representations between multiple source and target domains. Step 3: Divide the multiple source domains in the public invariant representation features into sub-source domains, and obtain the private invariant representation of each sub-source domain and the target domain through the set one-to-one feature extractor; at this stage, calculate the MMD loss and cross entropy loss between the multiple sub-source domains and the target domain; Step 4: Use multiple softmax classifiers for training, and use cross entropy to reduce losses during the training process.

2. The multi-source domain EEG signal analysis method with multi-modal representation according to claim 1, characterized in that: In step 1, the pre-processed data is aligned using the CA algorithm in the first step, specifically dividing the training data into multiple sub-source domains for separate alignment.

3. The multi-source domain EEG signal analysis method with multi-modal representation according to claim 1, characterized in that: In the step 2, Grassmann manifold mapping is used, and the mapping dimension is 5-40.

4. The multi-source domain EEG signal analysis method with multi-modal representation according to claim 1 or 3, characterized in that: Grassmann manifold mapping, with a mapping dimension of 40.

5. The method for analyzing multi-source domain EEG signals with multi-modal representation according to claim 1, characterized in that: The tangent space features of all source domains and target domains are extracted from the mapped multivariate statistical features. The expression for tangent space extraction is as follows: Among them, upper is to take the upper triangular elements of the matrix, P i is the covariance matrix, is the Riemann mean or the Euclidean mean.

6. The method for analyzing multi-source domain EEG signals with multi-modal representation according to claim 1, characterized in that: The CORAL loss between all source domain signals and the target domain is calculated as follows: in, represents CORAL loss, d is the feature dimension, C S and C T Represent the covariance matrices of the source domain and target domain data respectively, and i,j are the element indices in the covariance matrix.

7. The multi-source domain EEG signal analysis method with multi-modal representation according to claim 1, characterized in that: Calculate the MMD loss between multiple sub-source domains and the target domain. Specifically, the expression is as follows: Where N represents the number of source domains, A T represents the transformation matrix, C represents the number of categories, n s With n t Represent the total number of trials in the source domain and the target domain, respectively.

Images