Co-clustering based cross-subject speech imagery eeg intention decoding domain transfer method

Abstract

The application discloses a cross-subject speech imagery electroencephalogram (EEG) intention decoding domain migration method based on collaborative clustering, which comprises the following steps: step 1, obtaining existing speech imagery EEG data to construct a source domain data set and pre-process; step 2, extracting speech imagery related EEG features in the pre-processed source domain data set; step 3, initializing a sample clustering indication matrix and a feature clustering indication matrix, and solving optimization; step 4, training a semi-supervised speech imagery intention decoding model based on EEG sample and feature collaborative clustering; step 5, collecting target domain data and pre-processing; step 6, determining the best source domain matching of the target domain; step 7, adjusting the coefficient matrix; step 8, obtaining a final optimized sample feature coupling clustering EEG speech imagery intention decoding matrix decomposition model in the target domain, and then the recognition label of the intention decoding can be converted into a clustering indication matrix of the target domain sample and solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electroencephalogram signal processing, and particularly to a cross-subject speech imagination electroencephalogram intention decoding domain migration method based on collaborative clustering. BACKGROUND

[0002] Speech imagination intention decoding is a cutting-edge research topic, and its significance and value lie in its ability to deeply explore the mechanism of human brain processing and generating language, and to open up a new way of brain-computer interface interaction. Speech imagination, i.e. internal pronunciation in a silent state, represents the individual's internal language experience. By decoding these internal sounds, researchers have the opportunity to convert the speech imagined in people's brains into externally recognizable signals or sounds, which has a profound impact in both basic research and practical applications. In the field of basic research, speech imagination intention decoding provides a unique window for us to understand the language preprocessing mechanism of the brain. This technology can reveal how the brain processes, stores and generates language in a silent state, thereby deepening our understanding of cognitive processes. In the field of application, the potential of speech imagination intention decoding technology is even broader. For patients who have lost the ability to speak due to stroke, Parkinson's disease, brain trauma and other diseases, this technology provides new hope for restoring the ability to communicate. By directly converting speech intentions in the brain into sound or text, patients can re-communicate with others without relying on traditional vocal organs. In addition, this technology can also be applied in the fields of encrypted communication and brainprint recognition, providing people with a language-independent communication method. In summary, speech imagination intention decoding technology not only promotes our understanding of brain cognitive mechanisms, but also brings revolutionary application prospects to the fields of medical rehabilitation, silent communication, etc.

[0003] Among them, speech imagination intention decoding based on electroencephalogram signals is a technology that analyzes electroencephalogram (EEG) signals to recognize and decode individual internal language imagination, which has attracted much attention due to its advantages of being inexpensive, easy to use, convenient to collect, and high detection accuracy. The core of this technology lies in using the language processing and generation information contained in the electroencephalogram signals to infer the speech intention of individuals, enabling effective communication even without sound, broadening the application scenarios of brain-computer interface (BCI) systems, and showing broad application prospects and promotional value.

[0004] However, although the brain electrical signal is cheap and easy to obtain, it is a typical non-stationary signal with low signal-to-noise ratio and unobvious data characteristics, and there are significant differences in brain electrical signals across time periods or across subjects. The model trained on the existing speech imagination brain electrical data set often has difficulty in performing the same performance on new subjects or new time periods. In this case, directly applying the trained model to the newly collected brain electrical data often leads to a significant decrease in the accuracy of speech imagination intention decoding, thereby affecting the practical application value of the model.

[0005] In view of the above technical problems, there are mainly two kinds of coping ideas for the known prior art.

[0006] The first one is to retrain a speech imagination intention decoding model for new subjects and new time periods. Although this method can be customized and optimized for new data to improve decoding accuracy, it also has significant disadvantages: retraining the model not only consumes time and effort, but also the sample collection process often consumes a lot of time and human resources, resulting in high time cost. In addition, this method needs to be repeatedly trained for each new subject or time period, which cannot be effectively promoted and applied.

[0007] The other idea is to perform transfer learning based on the existing data set and the trained model, that is, to adjust and optimize the previous model and data based on the newly collected data, so as to adapt to the new data.

[0008] Among them, in transfer learning, the source domain refers to the field in which we have a large amount of labeled data and a trained model, which provides knowledge and experience for training the preliminary model; the target domain is the field to which we want to migrate the source domain model, which may have only limited labeled data or even no labeled data. The task of the target domain is to apply the knowledge obtained from the source domain to the new environment in order to achieve similar learning effect or improve the performance of the target task. However, the feature distribution and task of the target domain may be different from those of the source domain, which is also the challenge of transfer learning. Domain adaptation is an important process in transfer learning, aiming to reduce the distribution difference between the source domain and the target domain. Specifically, domain adaptation tries to adjust or optimize the model so that it can better generalize on the target domain.

[0009] The existing domain adaptation technology mainly has the following two categories:

[0010] Model Adjustment: Based on the model trained in the source domain, adjustment is made for the target domain. Specifically, a small amount of labeled data in the target domain is used for fine-tuning to optimize the model's performance in the target domain. The disadvantage of this technology is that it still relies on a large amount of labeled data for calibration, which still requires high cost and effort in the actual application of electroencephalogram data.

[0011] Distribution Alignment: Through optimization techniques, the original feature distribution of the source domain and the target domain or the distribution in the encoded dimensionality-reduced feature space is adjusted to make them as similar as possible. The probability distribution distance between the source domain and the target domain is minimized to make the source domain and the target domain distribution as similar as possible. Commonly used tools to measure the probability distribution distance include KL divergence, JS distance, and maximum mean difference (also known as MMD distance). Representative technologies of this type include transfer component analysis (TCA), joint distribution adaption (JDA), and transfer joint matching (TJM). However, the problem with this technology is that the source domain and the target domain may not be suitable for alignment, and blind distribution alignment can cause "negative transfer" phenomenon, resulting in unstable transfer learning effect.

[0012] In addition, the domain adaptation process in traditional transfer learning usually involves complex algorithm adjustment, and how to improve the explainability and complexity in the transfer process is still a technical pain point that needs to be solved.

[0013] Based on the above defects, it brings great challenges to speech imagination decoding of electroencephalogram. The non-stationarity of electroencephalogram signals means that their statistical properties (such as mean and variance) change over time, making signal analysis and decoding more complex. Brain activity at different time points may have different electrical activity patterns, leading to changes in signal features, making stable and reliable decoding difficult. Therefore, the features of electroencephalogram signals may change between different experimental sessions, which may be due to changes in experimental conditions, changes in subject state, or other environmental factors. This session variation increases the complexity of model training and decoding, making real-time and long-term stable decoding a major challenge. At the same time, there are natural differences in brain electrical activity patterns among different individuals, which makes it difficult to generalize the decoding model across subjects. Each person's electroencephalogram signal may differ in structure and dynamic characteristics, making the generalization effect of speech imagination intention decoding model based on electroencephalogram signal poor.

[0014] Therefore, it is particularly important to construct an intention decoding model with universality, which can handle individual differences and time period differences of speech imagination electroencephalogram data and has good migration ability in different subjects and different time periods. SUMMARY

[0015] The present application is based on the actual needs, and aims to solve the problems existing in the prior art of speech imagination intention decoding based on electroencephalogram signals. A cross-subject speech imagination electroencephalogram intention decoding domain migration method based on collaborative clustering is proposed, which aims to construct a transfer learning pre-adaptation method to simply and effectively cope with the differences of electroencephalogram signals across time periods and across subjects. Based on the semi-supervised speech imagination intention decoding algorithm based on electroencephalogram sample and feature collaborative clustering, the feature and sample coupling analysis of cross-subject electroencephalogram data distribution alignment is studied to solve the difference of electroencephalogram signals between different individuals and improve the universality of the decoding method. The present application aims to provide technical support for a new brain-computer interface interaction mode based on electroencephalogram signals and speech imagination, and further improve the application potential and practical value of the brain-computer interface.

[0016] To solve the above technical problems, the technical scheme of the present application is as follows:

[0017] A cross-subject speech imagination electroencephalogram intention decoding domain migration method based on collaborative clustering comprises the following steps:

[0018] Step 1, obtaining existing speech imagination electroencephalogram data to construct a source domain data set and pre-process, wherein the source domain data set X s contains m source domain data subsets;

[0019] Step 2, extracting the electroencephalogram features related to speech imagination in the pre-processed source domain data set, and then defining the clustering indication matrix of the features and the clustering indication matrix of the samples according to the label type and the coefficient matrix;

[0020] Step 3, initializing the clustering indication matrix of the samples corresponding to the labeled samples using prior speech imagination intention label information, and initializing the clustering indication matrix of the samples and the clustering indication matrix of the features corresponding to the unlabeled samples in a random initialization manner; after the initialization, the clustering indication matrix of the features, the clustering indication matrix of the samples and the coefficient matrix are solved in an alternating manner using a co-clustering algorithm, so as to obtain the optimized clustering indication matrix of the features, the clustering indication matrix of the samples and the coefficient matrix;

[0021] Step 4, training a semi-supervised speech imagination intention decoding model based on electroencephalogram sample and feature collaborative clustering using the optimized clustering indication matrix of the features, the clustering indication matrix of the samples and the coefficient matrix, to obtain an independent decoding model corresponding to each data subset X si ;

[0022] Step 5, collecting the labeled EEG data of the subject's speech imagination intention label in real time at any time period, i.e. target domain data, and preprocessing, so as to obtain the target domain data set X t ;

[0023] Step 6, traversing all source domain data subsets X si and the corresponding optimized feature clustering indication matrix and coefficient matrix, to determine the best source domain matching of the target domain;

[0024] Step 7, making certain adjustments to the clustering indication matrix and coefficient matrix of the features of the most matched source domain data subset X s* , to obtain the adjusted coefficient matrix;

[0025] Step 8, in the target domain, according to the clustering indication matrix P s* and the adjusted coefficient matrix S s* , obtaining the final optimized sample feature coupling clustering EEG speech imagination intention decoding matrix decomposition model, and then the intention decoding recognition label can be converted into the clustering indication matrix of the target domain sample;

[0026] Step 9, solving the clustering indication matrix of the target domain sample to obtain the decoding result.

[0027] As a preferred, in the step 1, the preprocessing method of the source domain data set includes denoising and filtering, so as to obtain the EEG data with artifacts and noise removed, ensuring the quality and consistency of the data.

[0028] As a preferred, in the step 2, assuming that the data matrix X is composed of labeled samples X l and unlabeled samples X u , wherein the number of labeled samples X l is l, and the number of unlabeled samples X u is u, and the original data matrix is rearranged as The clustering indication matrix Q of the corresponding sample can be expressed as Wherein, Q l ∈Φ l×c , Q u ∈Φ (n-l)×c , wherein c is the number of categories.

[0029] In this co-clustering algorithm:

[0030] · Matrix P represents the feature clustering indication matrix, with a size of d x c, where d is the total number of features and c is the number of clustering categories. Each row of matrix P corresponds to a feature, and each column corresponds to a clustering category. Element P ij indicates the indication value of the second feature in the jth category, which is 1 if the feature belongs to the category, otherwise 0.

[0031] • Matrix Q represents a feature cluster indicator matrix, with size n x c, where n is the total number of samples, and c is the number of cluster categories (consistent with the number of categories in the feature cluster indicator matrix Ppp). Each row of matrix Q corresponds to a sample, and each column corresponds to a cluster category. Element Qij j represents the indicator value of the ith sample in the jth category, which is 1 if the feature belongs to the category, otherwise 0.

[0032] • Matrix S represents a coefficient matrix, usually a diagonal matrix, with size c x c. The diagonal elements of S j represent the weight or importance of each cluster category. Since S ∈ diag, it means that S is a diagonal matrix, with zero non-diagonal elements, and only retains the weight of each cluster category on the diagonal.

[0033] • Parameter c refers to the number of cluster categories. In the speech imagination task, c is usually equal to the number of intent categories. For example, if the categories of speech imagination include "up", "down", "left", and "right" four intents, then c = 4.

[0034] Matrix Q works together with the feature cluster indicator matrix P and the coefficient matrix S to find the best clustering method for features and samples by optimizing the reconstruction error, so that the model can effectively distinguish different speech imagination intent categories.

[0035] As a preferred embodiment, the initialization method of the sample cluster indicator matrix Q of the labeled samples is:

[0036] For the ith labeled sample (i ∈ {1, 2,..., l}), its corresponding indicator vector satisfies:

[0037]

[0038] As a preferred embodiment, the initialization method of the sample cluster indicator matrix Q u of the unlabeled samples is: for the ith labeled sample (i ∈ {l + 1, l + 2,..., n}), its corresponding indicator vector is randomly assigned, satisfying:

[0039]

[0040] where each unlabeled sample only takes the value 1 in one randomly selected category, and takes the value 0 in the remaining categories.

[0041] As a preferred embodiment, the initialization method of the feature cluster indicator matrix is also a random initialization method, which only takes the value 1 in one randomly selected category, and takes the value 0 in the remaining categories.

[0042] As preferred, the solving method of the cluster indication matrix of the features, the cluster indication matrix of the samples, and the coefficient matrix is:

[0043] First, the coefficient matrix is solved with the cluster indication matrix of the features and the cluster indication matrix of the samples fixed, and the formula is as follows:

[0044] s=H -1 r

[0045] wherein H=P T PQ T Q, r=P T XQ, wherein P is the cluster indication matrix of the features, Q is the cluster indication matrix of the samples, and S is the coefficient matrix;

[0046] Then, only the sample cluster indication matrix Q u corresponding to the unlabeled samples is iterated and calculated, while the labeled sample part Ql remains unchanged, and the cluster indication matrix Q u is calculated with the cluster indication matrix of the features and the system matrix fixed.

[0047] Since only one element in each vector q u in Q i equals to 1 and the rest are 0, the solution of Q u can be determined by the following formula:

[0048]

[0049] wherein r k is the kth column of R, and R=PS;

[0050] Finally, the cluster indication matrix of the features is solved with the cluster indication matrix of the samples and the coefficient matrix fixed, and the solving formula is as follows:

[0051]

[0052] wherein l k is the kth row of L, and L=SQ T ;

[0053] The above solving process is repeated to alternately solve the optimization, and iteratively updated until convergence, so as to obtain the optimized cluster indication matrix P i of the features, the cluster indication matrix Q i of the samples, and the coefficient matrix S i .

[0054] As preferred, in step 4, the method for training a semi-supervised speech imagination intention decoding model based on electroencephalogram samples and feature collaborative clustering is:

[0055] Based on a matrix decomposition-based semi-supervised sparse neighbor constraint co-clustering algorithm, the original data matrix X, wherein the number of samples is n and the number of features is d, is subjected to feature clustering and sample clustering, and the optimization objective formula of the co-clustering is as follows:

[0056]

[0057] wherein P represents a clustering indication matrix of features, Q represents a clustering indication matrix of samples, S is a coefficient matrix for connecting P and Q, and for each data subset X si (i∈{12,...,N s} an independent decoding model corresponding thereto is obtained. The symbol T represents a transpose operation of a matrix, the symbol F represents a Frobenius norm, d represents the number of features, c represents the number of clusters of clustering, n represents the number of samples, diag represents a diagonal matrix, and S∈diag represents that S is a diagonal matrix.

[0058] Preferably, in step 5, the preprocessing method of the target domain data is as follows:

[0059] Firstly, the collected electroencephalogram data is down-sampled, and the signals corresponding to the bad electrodes are removed, and regenerated by interpolation of adjacent electrodes; then the electroencephalogram is filtered to 0.1-50 Hz by a band-pass filter to remove noise and remove the interference of noise on the classification accuracy; then the time-frequency-space multi-dimensional features are extracted, connected, sorted and standardized.

[0060] Preferably, in step 6, the speech imagery electroencephalogram data subset collected in the target domain is X t , which is composed of labeled samples X tl and unlabeled samples X tu , the intention decoding label corresponding to the labeled samples is Y tl , and the clustering indication matrix Q tl of the target domain samples is set as Y tl .

[0061] Preferably, in step 6, all source domain data subsets X si and the corresponding matrices P i and S i are traversed, and the F norm between the target domain data X t and the approximation matrix PSQ T output by the decoding model corresponding to the source domain data subset is calculated, that is, The source domain subset X s* that makes the norm minimum is selected, so that the clustering indication matrix P s* of the features and the system matrix S s* corresponding to the source domain data subset X s*, and takes it as the best source domain match for the target domain, and then as the object of domain adaptation.

[0062] Preferably, the source domain data subset X s* The cluster indicator matrix P of the corresponding feature s* and the system matrix S s* The adjustment method is to fix the cluster indicator matrix P of the feature s* , only adjust the coefficient matrix S s* , the adjustment formula is as follows:

[0063] S s* ′=H -1 r

[0064] in

[0065] Preferably, the final optimized sample feature coupled clustering EEG speech imagery intention decoding matrix decomposition model is expressed as:

[0066] X t ≈P s *S′ s *Q t .

[0067] Preferably, in step 8, the identification label intended for decoding can be converted into a cluster indicator matrix Q of the target domain sample. t Solve, the cluster indicator matrix Q of the target domain samples t The solution is: Since Q t Each vector q in i Only one element in Q is equal to 1, and the rest are 0. t The solution can be determined by the following formula:

[0068]

[0069] where r k R=P s *S′ s* The kth column of .

[0070] As an advantage, the method further includes step 10, wherein the clustering indicator matrix Q of the target domain samples is t Visualize the solution results.

[0071] The present invention has the following characteristics and beneficial effects:

[0072] In response to actual needs, the present invention establishes a domain migration system for speech imagery EEG intention decoding across subjects and time periods based on collaborative clustering to address the problems of poor universality and insufficient generalization ability of existing speech imagery intention decoding models.

[0073] Compared with directly applying the trained model or retraining on the speech imagination electroencephalogram data collected from new subjects or new time periods, the application adopts the idea of transfer learning, fully utilizes the source domain information of the existing data set, effectively transfers the knowledge between different domains, improves the accuracy and robustness of the model, and saves the time and resources of model construction and training.

[0074] Compared with the existing transfer learning algorithm, the domain transfer system based on collaborative clustering proposed in the application selects the most matching source domain from the perspective of minimizing the collaborative clustering matrix decomposition loss, improves the transfer effect, and has lower algorithm complexity and higher interpretability.

[0075] Compared with the existing domain adaptation algorithm, the domain transfer system based on collaborative clustering for cross-subject and cross-time period speech imagination electroencephalogram intention decoding proposed in the application does not involve complex multi-round learning and parameter adjustment process during the optimization process of the source domain decoding model for the target domain data, requires less labeled sample amount, and improves the response and optimization speed of domain adaptation. BRIEF DESCRIPTION OF DRAWINGS

[0076] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description only some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0077] Figure 1 The structural schematic diagram of the cross-subject speech imagination electroencephalogram intention decoding domain transfer system based on collaborative clustering of the embodiment of the application.

[0078] Figure 2 The flowchart of the domain selection and domain adaptation module in the cross-subject speech imagination electroencephalogram intention decoding domain transfer method based on collaborative clustering of the embodiment of the application. DETAILED DESCRIPTION

[0079] It should be noted that the embodiments in the application and the features in the embodiments can be combined with each other without conflict.

[0080] In order to make the purpose, technical scheme and advantages of the application more clear, the application will be further described in detail below with reference to the drawings.

[0081] On the contrary, the present application covers any substitutions, modifications, equivalent methods and solutions made within the spirit and scope of the present application defined by the claims. Further, in order to make the public better understand the present application, some specific details are described in the following detailed description of the present application. The present application can also be fully understood without the description of these details by those skilled in the art.

[0082] The present application provides a cross-subject speech imagery electroencephalogram intention decoding domain transfer method based on collaborative clustering, as shown in Figure 1 and Figure 2 The first application of a source domain data preprocessing module is responsible for processing and managing the existing speech imagery electroencephalogram data set, which constitutes the source domain of the entire system. The specific functions include data preprocessing, feature extraction and model training.

[0083] The specific steps are as follows:

[0084] Step 1, obtaining the existing speech imagery electroencephalogram data, thereby constructing the source domain data set and preprocessing, the source domain data set X s contains m source domain data subsets;

[0085] In the preprocessing, the electroencephalogram data is processed by denoising, filtering and other processing steps to remove artifacts and noise, and ensure the quality and consistency of the data.

[0086] Step 2, then, through feature extraction, extract the electroencephalogram features related to speech imagery, and then define the feature clustering indication matrix, sample clustering indication matrix and coefficient matrix according to the label type.

[0087] In this embodiment, it is assumed that the data matrix X is composed of labeled samples X l and unlabeled samples X u , wherein the number of labeled samples X l is l, the number of unlabeled samples X u is u, and the original data matrix is rearranged as The corresponding sample clustering indication matrix Q can be expressed as wherein Q l ∈Φ l×c , Q u ∈Φ (n-l)×c , wherein c is the number of categories.

[0088] Specifically, in this co-clustering algorithm in this embodiment:

[0089] The matrix P represents the feature clustering indication matrix, and the size is d×c, wherein d is the total number of features, and c is the number of clustering categories. Each row of the matrix P corresponds to a feature, and each column corresponds to a clustering category. The element P ijQi,j represents the indicator value of the second feature in the jth category, which is 1 if the feature belongs to the category, otherwise 0.

[0090] The matrix Q represents a feature clustering indicator matrix, with a size of n x c, where n is the total number of samples, and c is the number of clustering categories (consistent with the number of categories in the feature clustering indicator matrix PPP). Each row of the matrix Q corresponds to a sample, and each column corresponds to a clustering category. The element Qi j Qi,j represents the indicator value of the second feature in the jth category, which is 1 if the feature belongs to the category, otherwise 0.

[0091] The matrix S represents a coefficient matrix, usually a diagonal matrix, with a size of c x c. The diagonal elements of S j represent the weight or importance of each clustering category. Since S ∈ diag, it means that S is a diagonal matrix, with non-diagonal elements being zero, and only the weights corresponding to each clustering category on the diagonal are retained.

[0092] The parameter c refers to the number of clustering categories. In the speech imagination task, c is usually equal to the number of intent categories. For example, if the categories of speech imagination include "up", "down", "left", and "right" intents, then c = 4.

[0093] The matrix Q works together with the feature clustering indicator matrix P and the coefficient matrix S to find the best clustering method for the features and samples by optimizing the reconstruction error, so that the model can effectively distinguish different speech imagination intent categories.

[0094] Step 3, in order to fully utilize the label information of speech imagination electroencephalogram data and further improve the quality and interpretability of the clustering results, the optimization solution of P, Q, and S is completed under a semi-supervised framework.

[0095] First, the clustering indicator matrix P of the features and the clustering indicator matrix Q of the samples need to be initialized.

[0096] Specifically, unlike the traditional method of initializing P and Q as any category indicator matrix, this module fully utilizes the prior speech imagination intent label information in the initialization of the Q matrix. First, the number of clustering categories is fixed as the speech imagination intent label categories. In this embodiment, the intent labels of speech imagination are defined as the words "up", "down", "left", and "right", and the number of fixed clustering categories c = 4.

[0097] The prior speech imagination intent label information is used to initialize the clustering indicator matrix Q of the labeled samples l ,

[0098] For the ith labeled sample (i ∈ {1, 2,..., l}), its corresponding indicator vector satisfies:

[0099]

[0100] This initialization method ensures that the class attribution of the labeled samples in the clustering process is consistent with their prior labels.

[0101] Further, the sample clustering indicator matrix and the feature clustering indicator matrix corresponding to the unlabeled samples are initialized by random initialization.

[0102] For the ith labeled sample (i∈{l+1, l+2,..., n}), its corresponding indicator vector is randomly assigned, satisfying:

[0103]

[0104] Wherein, each unlabeled sample only takes the value 1 in a randomly selected class, and takes the value 0 in the remaining classes. Such random initialization provides a starting point for subsequent iterative optimization.

[0105] The initialization method of the feature clustering indicator matrix is also random initialization, only taking the value 1 in a randomly selected class, and taking the value 0 in the remaining classes.

[0106] In this way, the label information of the sample is directly embedded into the co-clustering algorithm, providing a guide for the algorithm, thereby maintaining the sensitivity to the sample intention in the clustering process. This initialization method can better combine the label information of the data and the clustering process, so that the final clustering result not only reflects the internal structure of the data, but also retains and highlights the intention information of the sample.

[0107] After initialization, the feature clustering indicator matrix, the sample clustering indicator matrix, and the coefficient matrix are solved in an alternating manner using the co-clustering algorithm, thereby obtaining the optimized feature clustering indicator matrix, the sample clustering indicator matrix, and the coefficient matrix.

[0108] Specifically, first, the coefficient matrix is solved by fixing the feature clustering indicator matrix and the sample clustering indicator matrix, and the formula is as follows:

[0109] s=H -1 r

[0110] Wherein H=P T PQ T Q, r=P T XQ, wherein P is the feature clustering indicator matrix, Q is the sample clustering indicator matrix, and S is the coefficient matrix.

[0111] Then, only the sample clustering indicator matrix Q u corresponding to the unlabeled samples is iterated and calculated, while the labeled sample part Ql Keeping the cluster indicator matrix of features and the system matrix unchanged, Q u ,

[0112] Since each vector q u in Q i has only one element equal to 1 and the rest are 0, the solution of Q u can be determined by the following formula:

[0113]

[0114] where r k is the kth column of R, R = PS; in this process, x i represents the feature vector of the ith unlabeled sample, and we determine the class membership of the sample by minimizing the Euclidean distance between the sample and all class centers. In this way, by iteratively calculating Q u , the potential structural information of the unlabeled samples can be effectively utilized while keeping the class indication of the labeled samples unchanged, thereby further optimizing the clustering results of the unlabeled samples in the co-clustering process.

[0115] Finally, by fixing the cluster indicator matrix of samples and the coefficient matrix, the cluster indicator matrix of features is solved, and the solution formula is as follows:

[0116]

[0117] where l k is the kth row of L, L = SQ T ;

[0118] Repeat the above solving process to alternately solve the optimization and iteratively update until convergence. The algorithm gradually optimizes the clustering results of features and samples, so that the feature clustering and sample clustering can promote each other, and ultimately achieve better clustering effect.

[0119] After the above steps, the data subset X s corresponding decoding model is obtained by optimization to obtain the final cluster indicator matrix of features P i , the cluster indicator matrix of samples Q i , and the coefficient matrix S i . These matrices not only reflect the internal structure of the data subset, but also retain the speech imagination intention information of the samples.

[0120] Step 4, specifically, using the optimized cluster indicator matrix of features, the cluster indicator matrix of samples, and the coefficient matrix, a semi-supervised speech imagination intention decoding model based on the collaborative clustering of EEG samples and features is trained. For each data subset X si , an independent decoding model is obtained. For each data subset Xsi (i∈{1,2,...,N s}) corresponds to an independent decoding model.

[0121] Specifically, based on a semi-supervised sparse neighbor constraint co-clustering algorithm based on matrix decomposition, the original data matrix X (the number of samples is n, and the number of features is d) is subjected to feature clustering and sample clustering, and the optimization objective formula of the co-clustering is as follows:

[0122]

[0123] Wherein, P represents the clustering indication matrix of the features, Q represents the clustering indication matrix of the samples, and S is a coefficient matrix for connecting P and Q.

[0124] Further settings of the embodiment are that a target domain data collection module is then applied to collect new electroencephalogram data from the target domain, which is usually collected from different subjects or in different time periods than the source domain. The functions of the target domain data collection module include real-time collection of electroencephalogram signals and data preprocessing.

[0125] The specific steps are as follows:

[0126] Step 5, real-time collection of the labeled electroencephalogram data of the subjects with speech imagination intention labels in any time period, i.e., target domain data, and preprocessing, so as to obtain a target domain data set X t

[0127] Specifically, the collected electroencephalogram signals are labeled electroencephalogram data with speech imagination intention labels, and the preprocessing process includes downsampling of the collected electroencephalogram data or existing electroencephalogram data, taking out the signals corresponding to the bad electrodes, and regenerating through interpolation of adjacent electrodes; then, the electroencephalogram is filtered to 0.1-50Hz through a band-pass filter to remove noise and as much as possible to remove the interference of noise on the classification accuracy; then, time-frequency-space multi-dimensional features are extracted, connected, arranged and standardized. This module ensures that high-quality electroencephalogram data is obtained in the target domain as the basis for subsequent transfer learning and model optimization. At the same time, these data also provide an important reference for the domain adaptation process to help adjust and optimize the performance of the model in the target domain.

[0128] Further settings of the embodiment are that the best source domain matching is selected through a domain selection module, and then used as the object of domain adaptation.

[0129] The specific steps are as follows:

[0130] Step 6, set the speech imagination electroencephalogram data subset collected by the target domain as X t , the labeled sample X tl , and the unlabeled sample X tuThe composition, the intention decoding label corresponding to the labeled sample is Y tl , and the clustering indication matrix Q of the target domain sample is determined tl = Y tl . It is known that each data subset X si (i∈{1,2,…,N s}) corresponds to an independent decoding model, and the corresponding P i , Q i and S i matrices are obtained respectively.

[0131] Then, the most suitable source domain is selected by minimizing the reconstruction loss. All source domain data subsets X si and the corresponding matrices P i and S i are traversed, the F norm between the approximation matrix PSQ T output by the decoding model corresponding to the target domain data X t and the source domain data subset is calculated, that is The source domain subset X s*， that makes the norm minimum is selected, so as to obtain the clustering indication matrix P s* and the system matrix S s* of the corresponding features of the source domain data subset X s* , and the source domain data subset X s* is matched as the best source domain of the target domain, and then is used as the object of domain adaptation.

[0132] Further settings of the embodiment, after determining the most matched source domain X s* , the source domain X s* needs to be adjusted in cooperation with the clustering model matrices P s* and S s* for domain adaptation.

[0133] Specifically, the steps are as follows:

[0134] Step 7, in order to simplify the algorithm complexity of the domain adaptation process, the clustering indication matrix P s* of the features is fixed, and only the coefficient matrix S s* is adjusted, and the adjustment formula is as follows:

[0135] S s* ′=H -1 r

[0136] wherein

[0137] Step 8, therefore, on the target domain, the final optimized sample feature coupling clustering electroencephalogram speech imagination intention decoding matrix decomposition model can be represented as:

[0138] X t ≈Ps *S′ s *Q t

[0139] Step 9, then, the identified label of the intent decoding can be converted into the clustering indicator matrix Q of the target domain sample t Solving, the solving process is as follows:

[0140]

[0141] Where r k is the kth column of R = P s *S′ s* .

[0142] Finally, the application result display and decoding application module. This module plays a core role in the final output, evaluation and application in the cross-subject speech imagination EEG intent decoding domain transfer and distribution alignment system based on collaborative clustering. This module will combine the decoding results with the actual application scenarios, through visualization, performance evaluation and decoding scheme deployment, to provide users with intuitive system performance feedback and practical application interface. First, this module includes decoding result visualization function. In the cross-subject speech imagination task, the decoding result usually includes accurate prediction of intent, clustering result of each subject's EEG signal features and its distribution. In order to better understand and analyze these data, this module will provide multi-dimensional visualization tools, including:

[0143] Time series chart: show the change trend of decoded EEG signal in different time periods, help researchers analyze the timing characteristics of different intents.

[0144] Clustering result display: display the clustering results of subject samples and features through two-dimensional or three-dimensional projection chart, intuitively reflect the distribution pattern of different subjects and intents.

[0145] Label comparison chart: compare the matching degree of decoding label and real label, show the decoding accuracy and error distribution of each subject.

[0146] This module provides a variety of evaluation indicators to quantify the model's transfer ability and decoding accuracy across subjects and time periods, including accuracy, confusion matrix, domain adaptation time, model response speed, etc.

[0147] This module provides an interface for application in brain-computer interface (BCI) applications, real-time decoding of user's speech imagination intent, for controlling external devices (such as wheelchair, prosthesis, etc.) or for neural rehabilitation training.

[0148] According to the records of this embodiment, it can be understood that the domain transfer system for speech imagery EEG intention decoding across subjects and time periods established in this embodiment can select the most suitable source domain for transfer from multiple source domains when only a small number of labeled samples in the target domain are available, effectively avoiding the negative transfer phenomenon;

[0149] Establishing a domain transfer system for cross-subject and cross-time period speech imagery EEG intention decoding based on collaborative clustering can transfer domain knowledge and adapt the collaborative clustering matrix decomposition model based on a small number of labeled samples in the target domain, thereby improving the robustness of the cross-subject and cross-time period speech imagery EEG intention decoding algorithm.

[0150] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. It will be apparent to those skilled in the art that various changes, modifications, substitutions, and variations of these embodiments, including components, without departing from the principles and spirit of the present invention are still within the scope of protection of the present invention.

Claims

1. A cross-subject speech imagery EEG intention decoding domain transfer method based on collaborative clustering, characterized by: The steps include: Step 1: Obtain existing speech imagery EEG data to construct a source domain dataset and preprocess it. The source domain dataset X s Contains m source domain data subsets; Step 2: Extract the EEG features related to speech imagination from the preprocessed source domain dataset, and then define the cluster indicator matrix of the features and the cluster indicator matrix of the samples as and coefficient matrix according to the label type; Step 3: Under the semi-supervised framework, the clustering indicator matrix of the features, the clustering indicator matrix of the samples, and the optimization solution of the coefficient matrix are completed; The method for solving the cluster indicator matrix of the features, the cluster indicator matrix of the samples, and the coefficient matrix is ​​as follows: First, the coefficient matrix is ​​solved by fixing the cluster indicator matrix of the features and the cluster indicator matrix of the samples. The formula is as follows: s=H -1 r Where H = P T PQ T Q, r = P T XQ, where P is the cluster indicator matrix of features, Q is the cluster indicator matrix of samples, S is the coefficient matrix, and X is the data matrix; Then, only the sample cluster indicator matrix Q corresponding to the unlabeled samples is u Iterate and calculate, and there is a labeled sample part Q l Keeping the cluster indicator matrix and system matrix unchanged, calculate Q u , Because Q u Each vector q in i Only one element in Q is equal to 1, and the rest are 0. u The solution can be determined by the following formula: where r k is the kth column of R, R = PS; Finally, when the cluster indicator matrix and coefficient matrix of the sample are fixed, the cluster indicator matrix of the feature is solved. The solution formula is as follows: where l k is the kth row of L, L = SQ T ; Repeat the above solution process, alternately solve the optimization, and iteratively update until convergence, so as to obtain the cluster indicator matrix P of the optimized feature i , the cluster indicator matrix Q of the sample i , coefficient matrix S i ; Step 4: Use the optimized feature clustering indicator matrix, sample clustering indicator matrix, and coefficient matrix to train a semi-supervised speech imagery intention decoding model based on EEG samples and feature co-clustering. For each data subset X si Get a corresponding independent decoding model; The method for training a semi-supervised speech imagery intention decoding model based on EEG samples and feature co-clustering is: Based on a semi-supervised sparse neighbor-constrained co-clustering algorithm based on matrix decomposition, the original data matrix X, where the number of samples is n and the number of features is d, is subjected to feature clustering and sample clustering. The optimization objective formula of the co-clustering is as follows: Among them, P represents the cluster indicator matrix of the feature, Q represents the cluster indicator matrix of the sample, and S is the coefficient matrix used to connect P and Q. For each data subset X si (i∈{1,2,...,N s }) obtain a corresponding independent decoding model, the symbol T represents the transpose operation of the matrix, the symbol F represents the Frobenius norm, d represents the number of features, c represents the number of clusters, n represents the number of samples, diag represents the diagonal matrix, and S∈diag represents that S is a diagonal matrix; Step 5: Collect the labeled EEG data of the subject with speech imagination intention labels in real time at any time period, that is, the target domain data, and preprocess it to obtain the target domain dataset X t ; Step 6: Traverse all source domain data subsets X si and its corresponding cluster indicator matrix and coefficient matrix of optimized features to determine the best source domain match for the target domain; Suppose the subset of speech imagery EEG data collected in the target domain is X t , by the labeled sample X tl and unlabeled samples X tu The intent decoding label corresponding to the labeled sample is Y tl , then let the cluster indicator matrix Q of the target domain samples be tl =Y tl ; Traverse all source domain data subsets X si and its corresponding matrix P i and S i , by calculating the target domain data X t The approximate matrix PSQ of the decoding model output corresponding to the subset of source domain data T The F norm between Select the source domain subset X that minimizes the norm s* , thus obtaining the source domain data subset X s* The cluster indicator matrix P of the corresponding feature s* and the system matrix S s* , and use it as the best source domain match for the target domain, and then as the object of domain adaptation; Step 7: For the best matching source domain data subset X s* Make certain adjustments to the cluster indicator matrix and coefficient matrix of the features to obtain the adjusted coefficient matrix; The source domain data subset X s* The cluster indicator matrix P of the corresponding feature s* and the system matrix S s* The adjustment method is: the cluster indicator matrix P of the fixed feature s* , only adjust the coefficient matrix S s* , the adjustment formula is as follows: S s* ′=H -1 r in Step 8: In the target domain, according to the clustering indicator matrix P of the feature s* And the adjusted coefficient matrix S s* The final optimized sample feature coupled clustering EEG speech imagery intention decoding matrix decomposition model is obtained. Then, the recognition label of intention decoding can be converted into the cluster indicator matrix of the target domain sample; Step 9: Solve the cluster indicator matrix of the target domain samples to obtain the decoding results.

2. The cross-subject speech imagery EEG intention decoding domain transfer method based on collaborative clustering according to claim 1 is characterized in that: In step 1, the preprocessing method of the source domain data set includes denoising and filtering, so as to obtain EEG data with artifacts and noise removed to ensure data quality and consistency.

3. The cross-subject speech imagery EEG intention decoding domain transfer method based on collaborative clustering according to claim 1 is characterized in that: In step 2, it is assumed that the data matrix X consists of labeled samples X l and unlabeled samples X u Composition, which contains labeled samples X l The number of unlabeled samples is l, u The number of is u, and the original data matrix is ​​rearranged into The corresponding sample cluster indicator matrix Q can be expressed as Among them, Q l ∈Φ l×c , Q u ∈Φ (n-l)×c , where c is the number of categories.

4. The method for cross-subject speech imagery EEG intention decoding domain transfer based on collaborative clustering according to claim 3 is characterized in that: The sample cluster indicator matrix Q of the labeled samples l The initialization method is: For the i-th labeled sample (i∈{1, 2, ..., l}), its corresponding indicator vector satisfies:

5. The method for cross-subject speech imagery EEG intention decoding domain transfer based on collaborative clustering according to claim 3 is characterized in that: The sample cluster indicator matrix Q of the unlabeled sample u The initialization method is: for the i-th labeled sample (i∈{l+1, l+2, ..., n}), its corresponding indicator vector is randomly assigned to satisfy: Each unlabeled sample takes the value of 1 only for one randomly selected category, and the values ​​of the other categories are 0.

6. The method for cross-subject speech imagery EEG intention decoding domain transfer based on collaborative clustering according to claim 5 is characterized in that: The cluster indicator matrix of the feature is initialized by randomly initializing it, taking the value of 1 only in one randomly selected category and taking the values ​​of 0 in the other categories.

7. The method for cross-subject speech imagery EEG intention decoding domain transfer based on collaborative clustering according to claim 1 is characterized in that: In step 5, the target domain data is preprocessed as follows: First, the collected EEG data is downsampled, and the signals corresponding to the bad electrodes are removed, and regenerated by interpolating adjacent ones; then the EEG is filtered to 0.1-50Hz through a bandpass filter to remove noise and eliminate the interference of noise on the classification accuracy; then the time-frequency-space multi-dimensional features are extracted, connected, sorted and standardized.

8. The method for cross-subject speech imagery EEG intention decoding domain transfer based on collaborative clustering according to claim 1 is characterized in that: The final optimized sample feature coupled clustering EEG speech imagery intention decoding matrix decomposition model is expressed as: X t ≈P s* S′ s* Q t 。 9. The method for cross-subject speech imagery EEG intention decoding domain transfer based on collaborative clustering according to claim 7, characterized in that: In step 8, the identification label intended for decoding can be converted into the cluster indicator matrix Q of the target domain sample t Solve, the cluster indicator matrix Q of the target domain samples t The solution is: Since Q t Each vector q in i Only one element in Q is equal to 1, and the rest are 0. t The solution can be determined by the following formula: where r k R=P s* S′ s* The kth column of .

10. The method for cross-subject speech imagery EEG intention decoding domain transfer based on collaborative clustering according to claim 8 or 9, characterized in that: The step 10 is also included, in which the clustering indicator matrix Q of the target domain samples is t Visualize the solution results.

Images