Semi-supervised speech imagination intention decoding method based on electroencephalogram collaborative clustering

Abstract

The invention discloses a semi-supervised speech imagination intention decoding method based on electroencephalogram collaborative clustering. The method comprises the following steps: step 1, collecting electroencephalogram data of speech imagination; 2, performing feature extraction on the preprocessed electroencephalogram data; 3, performing feature clustering and sample clustering on the original data matrix to obtain a clustering indication matrix of features, a clustering indication matrix of samples and a coefficient matrix; step 4, completing optimization solution of the clustering indication matrix of the features, the clustering indication matrix of the samples and the coefficient matrix under a semi-supervised framework; and 5, displaying a decoding result obtained by the semi-supervised learning model based on the electroencephalogram sample and feature collaborative clustering through a visualization means. The method can fully explore and utilize a bidirectional complex interaction mode of features and samples and label information of speech imagination in speech imagination intention decoding research and system based on electroencephalogram signals.

Description

Technical Field

[0001] The present invention relates to the technical field of electroencephalogram (EEG) signal processing, and in particular to a semi-supervised speech imagery intention decoding method based on EEG collaborative clustering. Background Art

[0002] Brain decoding is understood as mapping brain activity to the stimulus process that generates these activities, and has been an active research field in recent years. Among them, speech expression, as a high-level cognitive behavior, is closely related to brain activity and has become a research focus. Understanding and realizing the conversion from "brain to language" has become a key issue in the field of brain cognition and brain signal processing. In the process of studying speech expression, scientists have discovered a more intrinsic process: speech imagination. It is not only a valuable tool for studying the brain's preprocessing mechanism, but also provides humans with a new type of brain-computer interface interaction method. The application prospects of speech imagination intention decoding technology are very broad, especially for helping patients with speech disorders (such as stroke, Parkinson's disease, brain trauma, cerebral palsy, cerebellar atrophy and other brain damage patients) regain the ability to communicate, which is of great significance. In addition, it also has important application value in encrypted silent communication and brain pattern recognition that do not involve speech utterance.

[0003] Although significant progress has been made in the field of brain decoding in the past few decades, most of these advances have focused on sensory or motor research such as visual imagination, auditory imagination, tactile imagination, olfactory imagination and motor imagination. In contrast, there are relatively few studies on verbal imagination. Unlike simple perceptual imagination and motor imagination, verbal imagination is a special dual mechanism that includes both motor imagination of the vocal organs (i.e., "imagination of speaking") and imagination of speech perception (i.e., "imagination of listening"). Due to the complex neural mechanisms and large amounts of brain neural data involved in verbal imagination, researchers face challenges such as unclear research paradigms, insufficient decoding accuracy, and complex data processing. Existing research on verbal imagination intention decoding is still far from reaching a practical level.

[0004] The existing speech imagery intention decoding methods "from brain to language" at home and abroad can be divided into two categories: invasive and non-invasive, according to the different signal acquisition methods. Invasive brain-computer interface systems require electrodes to be implanted into the brain through surgery, which greatly reduces the impact of movement and other non-neural artifacts. In invasive speech imagery brain-computer interface systems, ECoG signals are widely used because they have high signal-to-noise ratio, temporal resolution, and spatial resolution, and are relatively low in invasiveness. For example, in the latest study, researchers at New York University proposed a decoding framework that maps ECoG signals to interpretable acoustic speech parameters (such as pitch, voice, and formant frequency), and uses these voice parameters to synthesize new voices, realizing the speech-ECoG-speech encoding path. However, such ECoG signal-based research is mainly conducted in patients with epilepsy, and the general population has a low acceptance of invasive technology, which greatly limits its application value. Therefore, non-invasive signal acquisition methods have become the focus of research, including electroencephalogram, functional near-infrared spectroscopy, magnetoencephalography, functional magnetic resonance imaging, and positron emission tomography. Among them, EEG signals are the most popular due to their advantages such as low cost, ease of use, convenient collection and high detection accuracy. EEG technology has greatly broadened the application scenarios of speech imagery intention decoding brain-computer interface systems, making them practical and valuable for promotion among a wider range of people.

[0005] However, most of the existing speech imagery decoding methods are based on machine learning and deep learning methods, and the time-frequency and spatial features extracted from EEG data are used for classification prediction in supervised learning. These methods have some significant defects. First, the transparency and interpretability of the model are poor. Since deep learning models and complex machine learning algorithms are usually "black box" models, it is difficult for researchers and users to understand the internal working mechanism and decision-making process of the model. Secondly, these methods are prone to overfitting. Due to the limitations of EEG signal acquisition, the scale of data obtained from the same subject is limited, which makes it easy for deep learning networks to overfit during training. The model is prone to overfitting the noise and details in the training data during the training process, resulting in the model performing well on the training data but performing poorly on new data.

[0006] In addition, a few speech imagery intention decoding methods based on unsupervised clustering learning only cluster samples, dividing samples into several clusters close to each other, focusing on global patterns. Therefore, it is difficult to find local patterns that are only in specific samples and feature subsets, ignoring important patterns that may exist in feature dimensions in the data, making it difficult to identify complex interaction patterns.

[0007] In addition, if the general sample and feature coupled collaborative clustering method is directly applied to the field of EEG-based speech imagery intention decoding, the intention decoding problem will be completely transformed into an unsupervised learning clustering problem, ignoring the label information of EEG speech imagery, which may lead to clustering to produce meaningless and difficult-to-interpret subsets (clusters). Summary of the invention

[0008] In view of the shortcomings of the existing speech imagery intention decoding methods based on EEG signals, the present invention provides a semi-supervised speech imagery intention decoding method based on EEG collaborative clustering, which can fully explore and utilize the bidirectional complex interaction pattern of features and samples and the label information of speech imagery in the research and system of speech imagery intention decoding based on EEG signals.

[0009] In order to solve the above technical problems, the technical solution of the present invention is:

[0010] A semi-supervised speech imagery intention decoding method based on EEG collaborative clustering includes the following steps:

[0011] Step 1: Collecting EEG data of speech imagination, labeling and preprocessing the collected EEG data according to speech imagination;

[0012] Step 2: Extract features from the preprocessed EEG data. The feature extraction method is as follows: first, extract EEG features from five frequency bands, then extract corresponding time domain features, frequency domain features, time-frequency domain features, and spatial domain features for each frequency band and different EEG electrode channels, and finally connect, organize, and standardize all the extracted features to obtain the original data matrix;

[0013] Step 3: Perform feature clustering and sample clustering on the original data matrix to obtain a feature clustering indicator matrix, a sample clustering indicator matrix and a coefficient matrix;

[0014] Step 4: Under the semi-supervised framework, the clustering indicator matrix of the features, the clustering indicator matrix of the samples and the coefficient matrix are optimized and solved;

[0015] Step 5: Use visualization to display the decoding results obtained by the semi-supervised learning model based on EEG samples and feature co-clustering.

[0016] Preferably, in step 1, when collecting EEG data, the subject performs a speech imagery task when wearing an EEG cap and receiving relevant prompts, and the EEG generated by the subject during speech imagery will be collected as raw EEG data.

[0017] Preferably, the preprocessing method is: firstly down-sample the collected EEG data and remove the signals corresponding to the bad electrodes, and then regenerate them by interpolating adjacent ones; and then filter the EEG data to 0.1-50 Hz by a bandpass filter to remove noise.

[0018] Preferably, the five frequency bands are: Delta band, Theta band, Alpha band, Beta band, and Gamma band.

[0019] Preferably, the time domain features include the centroid, mean absolute difference, peak-to-peak distance, and Shannon entropy on the time axis; the frequency domain features include spectral distance, fundamental frequency, maximum power spectral density, maximum frequency, median frequency, spectral center of gravity, attenuation of spectral amplitude, spectral kurtosis, spectral skewness, spectral slope, spectral variability, spectral positive turning point, spectral start point, power spectrum bandwidth, Fourier transform mean coefficient, and spectral entropy; the time and frequency domain features include wavelet transform features and empirical mode decomposition features; the spatial domain features include using a common spatial pattern algorithm to spatially filter the EEG data and extracting the spatial distribution components of each class from the multi-channel EEG data.

[0020] Preferably, in step 3, the original data matrix is ​​defined as X, the number of samples in the original data matrix X is n, the number of features is d, and the original data matrix X is subjected to feature clustering and sample clustering, and the optimization objective formula of the co-clustering is as follows:

[0021]

[0022] stP∈Φ d×c , Q∈Φ n×c , S∈diag

[0023] Among them, P represents the clustering indicator matrix of features, Q represents the clustering indicator matrix of samples, S is the coefficient matrix connecting P and Q, the symbol T represents the transpose operation of the matrix, the symbol F represents the Frobenius norm, d represents the number of features, c represents the number of clusters, n represents the number of samples, diag represents the diagonal matrix, and S∈diag represents that S is a diagonal matrix.

[0024] Preferably, the original data matrix X includes labeled samples X l and unlabeled samples X u , where there are labeled samples X l The number of unlabeled samples is l, u The number of is u, so the original data matrix X is rearranged into The corresponding sample cluster indicator matrix Q can be expressed as Where Q l ∈Φl×c , Q u ∈Φ (n-l)×c .

[0025] Preferably, step 4 includes initializing a clustering indication matrix P of features and a clustering indication matrix Q of samples.

[0026] Preferably, the clustering indicator matrix Q of the sample is initialized as follows:

[0027] First, the clustering category is fixed to the speech imagery intention label category;

[0028] For the sample cluster indicator matrix Q corresponding to the labeled sample l , for the i-th labeled sample (i∈{1, 2, ..., l}), its corresponding indicator vector satisfies:

[0029]

[0030] Where c refers to the number of language imaginary intention label categories;

[0031] For the sample clustering indicator matrix Q corresponding to the unlabeled sample u , using random initialization, for the i-th unlabeled sample (i∈{l+1,l+2,...,n}), its corresponding indicator vector is randomly assigned to satisfy:

[0032]

[0033] Each unlabeled sample takes the value of 1 only in one randomly selected category and the values ​​of the other categories are 0.

[0034] Preferably, the cluster indicator matrix P of the feature is initialized by using a random initialization method, taking a value of 1 only in a randomly selected category and taking values ​​of 0 in the remaining categories.

[0035] Preferably, in step 4, the optimization solution method is:

[0036] Solve for S with P and Q fixed, and the solution formula is as follows:

[0037] S=H -1 r,

[0038] Where H = P T PQ T Q, and r = P T XQ;

[0039] Calculate Q with P and S fixed u :

[0040] Thanks to Qu Each vector q in i Only one element in Q is equal to 1, and the rest are 0, so Q u The solution can be determined by the following formula:

[0041]

[0042] where r k is the kth column of R, R = PS, x i Represents the feature vector of the i-th unlabeled sample, argmin k Express the selection of category k that minimizes the distance;

[0043] When Q and S are fixed, calculate P and solve the formula as follows:

[0044]

[0045] Among them l k is the kth row of L, L = SQ T , x j Represents the jth eigenvector. Note that j represents the index of the feature, not the index of the sample; argmin k Select the category index k corresponding to the minimum value;

[0046] Repeat the above solution process, alternately solve the optimization, and iteratively update until convergence.

[0047] The present invention has the following characteristics and beneficial effects:

[0048] By adopting the above technical scheme, by simultaneously clustering the samples and features in the EEG data of speech imagery intention decoding, and finding similar subgroups in the sample and feature dimensions, the two-way complex interaction pattern of features and samples and the label information of speech imagery can be fully explored and utilized in the research and system of speech imagery intention decoding based on EEG signals.

[0049] Compared with speech imagery intention decoding methods based on other brain signals (such as ECoG signals and MRI signals), the speech imagery intention decoding method based on EEG data proposed in this patent only uses EEG signals, has lower signal acquisition costs and a wider range of application groups and scenarios.

[0050] Compared with the previous supervised learning-based speech imagery intention decoding classification network method, the collaborative clustering method in this patent has higher interpretability because the method in this patent can reveal the relationship between samples and features, and help understand which features play a key role in a specific sample group.

[0051] Compared with traditional clustering methods (such as K-nearest neighbor methods), the collaborative clustering method in this patent can more effectively mine multi-dimensional information in the data and identify local patterns that have been ignored in the past, that is, patterns that only exist in specific samples and feature subsets. At the same time, in high-channel EEG data, many features may be redundant or irrelevant. By clustering samples and features at the same time, the collaborative clustering method of this patent can reduce the impact of redundant features and thus improve the quality of clustering.

[0052] Compared with the general sample and feature co-clustering method, the semi-supervised speech imagery intention decoding method based on EEG sample and feature co-clustering proposed in this patent has been improved in two aspects. On the one hand, it makes full use of the domain knowledge in the field of speech imagery intention decoding, and mines and utilizes multiple features of speech imagery intention decoding in the time domain, frequency domain, and spatial domain; on the other hand, based on the co-clustering, this patent makes full use of the label information of speech imagery EEG data, further improving the quality and interpretability of the clustering results. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0054] Figure 1 Schematic diagram of a flow chart of an embodiment of the present invention.

[0055] Figure 2 This is a schematic diagram of a semi-supervised speech imagery intention decoding method based on EEG collaborative clustering according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0057] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings.

[0058] On the contrary, the present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention as defined by the claims. Further, in order to make the public have a better understanding of the present invention, some specific details are described in detail in the detailed description of the present invention below. Those skilled in the art can fully understand the present invention without the description of these details.

[0059] The present invention provides a semi-supervised speech imagery intention decoding method based on EEG collaborative clustering, such as Figure 1 As shown, the specific steps are as follows:

[0060] The EEG data is collected through the speech imagery EEG data collection module.

[0061] Specifically, in this embodiment, during the offline training data collection stage, the subjects are required to perform speech imagination tasks when there are relevant prompts such as voice and visual prompts while wearing an EEG cap, and the EEG generated by the subjects during speech imagination will be collected as original data. This section of EEG signals for speech imagination will be marked with a corresponding label. In the real-time EEG data decoding stage, the user performs the speech imagination task within the preset task range while wearing an EEG cap, and the EEG data during this period will be collected and analyzed. In this embodiment, the preset speech imagination task selections are: eating, drinking water, and going to the toilet. The same corresponding EEG data prints the label of the above-mentioned speech imagination task.

[0062] The feature extraction module is used to preprocess and extract features from the collected EEG data.

[0063] Specifically, the preprocessing method is as follows: first, the collected EEG data or the existing EEG data is downsampled, and the signal corresponding to the bad electrode is extracted and regenerated by interpolating the adjacent electrodes; then the EEG is filtered to 0.1-50Hz through a bandpass filter to remove noise and remove the interference of noise on the classification accuracy as much as possible. Then, the time-frequency-space multi-domain features are extracted and fused for analysis.

[0064] Feature extraction method: Feature extraction is performed on the preprocessed EEG data. The feature extraction method is as follows: first, EEG features are extracted from five frequency bands, and then corresponding time domain features, frequency domain features, time-frequency domain features, and spatial domain features are extracted for each frequency band and different channels. Finally, all the extracted features are connected, sorted, and standardized to obtain the original data matrix.

[0065] It should be noted that the "channel" in EEG refers to a specific location or electrode position for measuring and recording the electrical activity of the brain. During the EEG acquisition process, electrodes are placed on specific parts of the scalp to sense the electrical activity of different areas of the brain. These electrodes are connected to the EEG recording device through wires to form a "channel". Each channel represents the potential difference measured between a specific pair of electrodes (or reference points). It can be understood that different channels refer to EEG signals at different electrode positions.

[0066] Specifically, we first extract EEG signals from five classic EEG frequency bands (Delta band (1-3Hz), Theta (4-7Hz), Alpha band (8-13Hz), Beta band (14-30Hz), Gamma band (31-50Hz)), and then extract the corresponding time domain features, frequency domain features, time-frequency domain features and spatial domain features for each frequency band and different EEG electrode channels. The time domain features include the centroid on the time axis, mean absolute difference, peak-to-peak distance, Shannon Entropy; frequency domain features include spectrum distance, fundamental frequency, maximum power spectrum density, maximum frequency, median frequency, spectrum center of gravity, spectrum amplitude attenuation, spectrum kurtosis, spectrum skewness, spectrum slope, spectrum variability, spectrum positive turning point, spectrum opening point, power spectrum bandwidth, Fourier transform mean coefficient, spectrum entropy; time-frequency domain features include wavelet transform features and empirical mode decomposition features; spatial domain features include using the common space mode algorithm to perform spatial filtering on the EEG signal, and extracting the spatial distribution components of each class from the multi-channel EEG data. Finally, after splicing and standardizing these features, the algorithm input data required for use in this embodiment is obtained.

[0067] It should be noted that there are different calculation methods for different features. Because it is not the focus of this patent, only the name of the feature is written, and the specific calculation method is not written. This embodiment does not limit which features are used.

[0068] Time domain feature extraction is to obtain features directly from time signals, usually through the following means:

[0069] Centroid (central moment): The centroid is obtained by calculating the weighted average of the signal strength on the time axis. It is the time distribution characteristic of the signal.

[0070] Mean absolute difference: Calculates the average of the absolute differences between each sampling point of the signal and the mean.

[0071] Peak-to-Peak Distance: Calculates the difference between the maximum and minimum values ​​of a signal.

[0072] Shannon entropy: Use the entropy in information theory to measure the complexity of the signal, specifically by calculating the Shannon entropy formula to process the probability distribution of the signal.

[0073] Frequency domain feature extraction is to analyze the spectral characteristics of the signal by converting the time domain signal to the frequency domain, usually through the following means:

[0074] Fourier transform: The time domain signal is converted into a frequency domain representation through the fast Fourier transform (FFT), from which the spectrum information such as power spectral density (PSD) is obtained.

[0075] Spectral distance, fundamental frequency, maximum power spectral density, etc.: These indicators are calculated from the spectrum diagram and involve the frequency content and power distribution of the signal.

[0076] Spectral center of gravity: the center of mass of the spectrum, which represents the concentration of frequency components in the spectrum by weighted average of the frequencies.

[0077] Spectral entropy, kurtosis, skewness, etc.: used to describe the complexity, symmetry, smoothness, etc. of the spectrum distribution.

[0078] The attenuation and slope of the spectrum amplitude: calculated from the amplitude change trend of the spectrum, can reflect the frequency energy attenuation of the signal.

[0079] The time-frequency domain features combine time information and frequency information and are suitable for the analysis of non-stationary signals, usually through the following means:

[0080] Wavelet transform: Use wavelet transform to decompose the signal into different scales and resolutions to extract multi-scale features, such as using discrete wavelet transform (DWT) or continuous wavelet transform (CWT).

[0081] Empirical Mode Decomposition (EMD): decomposes the signal into several intrinsic mode functions (IMFs) and extracts features from them. It is suitable for processing complex, nonlinear and non-stationary signals.

[0082] Spatial feature extraction is to analyze the spatial distribution characteristics of multi-channel EEG signals, usually through the following means:

[0083] Common Spatial Pattern (CSP): CSP is a very classic spatial filtering algorithm used to extract spatial features of different categories from multi-channel EEG signals. CSP finds discriminative feature components by maximizing and minimizing the variance differences between different categories. These features can be used for classification, such as distinguishing different EEG activity states (e.g. imagining left hand vs. right hand movement).

[0084] The original data matrix is ​​subjected to feature clustering and sample clustering to obtain the feature clustering indicator matrix, sample clustering indicator matrix and coefficient matrix.

[0085] Specifically, this embodiment proposes a semi-supervised sparse neighbor-constrained co-clustering algorithm based on matrix decomposition. Figure 2 As shown, the original data matrix X (the number of samples is n, the number of features is d) is subjected to feature clustering and sample clustering, and the optimization objective formula of the co-clustering is as follows:

[0086]

[0087] stP∈Φ d×c , Q∈Φ n×c , S∈diag

[0088] Among them, P represents the clustering indicator matrix of features, Q represents the clustering indicator matrix of samples, and S is the coefficient matrix connecting P and Q.

[0089] It can be understood that the optimization goal of the co-clustering optimization method proposed in this embodiment is to find the best cluster division in two data dimensions at the same time, so as to maximize a certain criterion or minimize a certain loss between subsets of rows and columns.

[0090] In this embodiment, the label information of speech imagery EEG data is fully utilized to improve the quality and interpretability of clustering results. In this embodiment, the optimization of P, Q, and S is completed under a semi-supervised framework. Assume that the data matrix X consists of labeled samples X l and unlabeled samples X u Composition, which contains labeled samples X l For convenience of representation, in this embodiment, the original data matrix X is rearranged as The corresponding sample cluster indicator matrix Q can be expressed as Where Q l ∈Φ l×c , Q u ∈Φ (n-l)×c .

[0091] Furthermore, the clustering indicator matrix of features, the clustering indicator matrix of samples and the coefficient matrix are optimized under a semi-supervised framework.

[0092] In this embodiment, the optimization solution method is similar to the standard K-means algorithm, and the co-clustering algorithm is solved in an alternating manner.

[0093] The specific steps are as follows:

[0094] (1) P and Q initialization

[0095] Different from the traditional practice of initializing P and Q to arbitrary category indicator matrices, in this embodiment, the prior speech imagery intention label information is fully utilized when initializing the Q matrix. First, the clustering category is fixed to the speech imagery intention label category. In this embodiment, the speech imagery intention labels are the words "up", "down", "left", and "right", so the number of clustering categories c is fixed to 4.

[0096] Furthermore, for the sample clustering indicator matrix Q corresponding to the labeled samples l In this embodiment, it is initialized according to the prior label information. Specifically, for the i-th labeled sample (i∈{1, 2, ..., l}), its corresponding indicator vector satisfies:

[0097]

[0098] The above initialization method ensures that the category affiliation of labeled samples in the clustering process is consistent with their prior labels.

[0099] For the sample clustering indicator matrix Q corresponding to the unlabeled sample u In this embodiment, random initialization is adopted. For the i-th unlabeled sample (i∈{l+1, l+2, ..., n}), its corresponding indicator vector is randomly assigned to satisfy:

[0100]

[0101] Among them, each unlabeled sample takes the value of 1 only in one randomly selected category, and the values ​​of the other categories are 0. Such random initialization provides a starting point for subsequent iterative optimization.

[0102] For the clustering indicator matrix P of the features, in this embodiment, a random initialization method is also adopted, where only a randomly selected category takes a value of 1, and the other categories take values ​​of 0.

[0103] Understandably, the label information of the sample is directly embedded into the co-clustering algorithm, providing a guide for the algorithm to maintain sensitivity to the sample intent during the clustering process. This initialization method can better combine the label information of the data with the clustering process, so that the final clustering result not only reflects the intrinsic structure of the data, but also retains and highlights the sample intent information.

[0104] (2) Solve for S with P and Q fixed:

[0105] P and Q are fixed in the current step, and the optimization objective is transformed into: Where X is the known original data matrix, P and Q are the known cluster indicator matrices in the current step, and S is the matrix we want to optimize so that PSQ T Reconstruct X as much as possible. In the quadratic optimization problem, minimize The solution of can usually be obtained by setting the derivative to zero. Finally, by taking the matrix derivative and setting the derivative to zero, we get the optimal solution formula for S. After expanding the objective function, we can get the following derivative of S:

[0106]

[0107] Setting the derivative to zero gives:

[0108] P T XQ=P T PSQ T Q

[0109] Then the optimal solution of S can be expressed as:

[0110] s=H -1 r,

[0111] Where H = P T PQ T Q, and r = P T XQ.

[0112] (3) Calculate Q when P and S are fixed u :

[0113] In this step, this embodiment only performs clustering on the sample indicator matrix Q corresponding to the unlabeled samples. u Iterate and calculate, and there are labeled sample parts Q l Remain unchanged.

[0114] Thanks to Q u Each vector q in i Only one element in Q is equal to 1, and the rest are 0, so Q u The solution can be determined by the following formula:

[0115]

[0116] where r k is the kth column of R, R = PS, x i Represents the feature vector of the i-th unlabeled sample, argmin k The expression selects the category k that minimizes the distance. The idea of ​​the formula is to minimize the unlabeled sample x i With each category center k The Euclidean distance between them determines x i The category to which it belongs.

[0117] In the above technical solution, only by u The iterative calculation can effectively utilize the potential structural information of unlabeled samples while keeping the category indication of labeled samples unchanged, thereby further optimizing the clustering results of unlabeled samples in the co-clustering process.

[0118] (4) Calculate P when Q and S are fixed:

[0119] Similar to the formula in step (3), the solution formula for P is as follows:

[0120]

[0121] Among them l k is the kth row of L, L = SQ T , x j Represents the jth eigenvector. Note that j represents the index of the feature, not the index of the sample; argmin kSelect the category index k corresponding to the minimum value. This is similar to step (3), and the category attribution of each feature vector is determined by minimizing the Euclidean distance between the feature vector and the category center.

[0122] Repeat steps (2)-(4), alternately solve the optimization, and iterate until convergence. The algorithm gradually optimizes the clustering results of features and samples, so that feature clustering and sample clustering can promote each other and ultimately achieve better clustering results.

[0123] Finally, the decoding results obtained by the semi-supervised learning model based on EEG samples and feature co-clustering are presented through the result display and decoding application module.

[0124] Specifically, on the one hand, from the perspective of model intent classification effectiveness, the clustering indicator matrix Q of unlabeled samples u This means that for the unlabeled sample X u Intent classification decoding recognition results Tracing X by comparison u The real intention Y u The module will generate accurate evaluation indicators, including classification accuracy, recall rate, F1 score, confusion matrix, etc., to quantify the decoding performance of unlabeled samples. On the other hand, users can dynamically view the clustering results of different samples and features through an interactive interface, display the distribution of samples in the feature space through a clustering diagram, and evaluate and display the effect of the model's collaborative clustering in both feature and sample dimensions.

[0125] Furthermore, in this embodiment, the clustering indicator matrix P of the features optimized in module three will be further interpreted and analyzed. Through in-depth analysis of the P matrix, users can understand how the system clusters different features into specific categories, thereby revealing the role and contribution of these features in the speech imagery intention decoding process. This analysis helps to discover which features have higher discrimination ability for decoding tasks and which features may be redundant or irrelevant, promote the understanding of the speech imagery intention decoding process, and provide support for the improvement of experimental paradigms in the field of speech imagery intention decoding.

[0126] In order to make the feature importance obtained from the cluster indicator matrix P of the feature more robust, in this embodiment, analysis is performed from two perspectives:

[0127] Feature contribution: Each row of the P matrix corresponds to a feature, and each column corresponds to a cluster. For each feature i, calculate the sum of its contributions in all clusters:

[0128]

[0129] where s j are the diagonal elements of the S matrix. iThe larger the value is, the higher the contribution and importance of the feature is considered to be.

[0130] Sensitivity of reconstruction error to features: By calculating X≈PSQ T The reconstruction error of can be reduced by zeroing or reducing the influence of certain rows of the P matrix (corresponding to specific features) and observing the impact on the overall reconstruction error. If removing certain features leads to a significant increase in the reconstruction error, then these features are more important.

[0131] Assume that the original reconstruction error is defined as:

[0132]

[0133] Suppose you want to evaluate the importance of a feature i, you can set the i-th row of the P matrix to zero, that is The other rows remain unchanged. The reconstruction error at this time is:

[0134]

[0135] At this time, the importance of feature i can be measured by calculating the change in reconstruction error:

[0136] ΔE i =E featurei -E original

[0137] If ΔE i If it is larger, it means that feature i plays an important role in the reconstruction of the data, so the importance of this feature is higher.

[0138] By ΔE i With C i By ranking the weighted values ​​of , we can get the feature importance ranking, which systematically quantifies their importance in the model:

[0139] Rank i =sort(αΔE i +βC i )

[0140] Where α and β are weight parameters.

[0141] Finally, the model is deployed on the brain-computer interface (BCI) application to support the decoding of real-time collected EEG signals and provide instant feedback on speech imagination intentions based on the decoding results, helping users control external devices or perform other operations through thinking.

[0142] Based on the above technical solution, this embodiment provides the following specific implementation cases:

[0143] Module 1: Collect EEG data through the speech imagination EEG data acquisition module with a sampling frequency of 1000HZ.

[0144] Specifically, in this embodiment, during the offline training data collection stage, the subjects are required to perform a speech imagination task after visual prompts while wearing an EEG cap, and the EEG generated by the subjects during speech imagination will be collected as original data. This section of EEG signals for speech imagination will be marked with a corresponding label. In the real-time EEG data decoding stage, the user performs the speech imagination task within the preset task range while wearing an EEG cap, and the EEG data during this period will be collected and analyzed. In this embodiment, the preset speech imagination task options include: "up", "down", "left", and "right". The same corresponding EEG data prints the label of the above-mentioned speech imagination task.

[0145] Module 2: Use the feature extraction module to preprocess and extract features from the collected EEG data.

[0146] Specifically, the preprocessing method is as follows: first, the collected EEG data or the existing EEG data is downsampled to a frequency of 250HZ, and the signal corresponding to the bad electrode is extracted and regenerated by interpolation of adjacent electrodes; then the EEG is filtered to 0.1-50Hz through a bandpass filter to remove noise and remove the interference of noise on the classification accuracy as much as possible. Then, the time-frequency-space multi-domain features are extracted and fused for analysis.

[0147] Feature extraction method: First, extract EEG signals from five classic EEG frequency bands (Delta band (1-3Hz), Theta (4-7Hz), Alpha band (8-13Hz), Beta band (14-30Hz), Gamma band (31-50Hz)), and then extract the corresponding time domain features, frequency domain features, time-frequency domain features and spatial domain features for each frequency band and different EEG electrode channels. The time domain features include the centroid on the time axis, the mean absolute difference, the peak-to-peak distance, Shannon entropy; frequency domain features include spectrum distance, fundamental frequency, maximum power spectrum density, maximum frequency, median frequency, spectrum center of gravity, spectrum amplitude attenuation, spectrum kurtosis, spectrum skewness, spectrum slope, spectrum variability, spectrum positive turning point, spectrum opening point, power spectrum bandwidth, Fourier transform mean coefficient, spectrum entropy; time-frequency domain features include wavelet transform features and empirical mode decomposition features; spatial domain features include using the common space mode algorithm to perform spatial filtering on the EEG signal, and extracting the spatial distribution components of each class from the multi-channel EEG data. Finally, after splicing and standardizing these features, the algorithm input data required for use in this embodiment is obtained.

[0148] Module 3: Perform feature clustering and sample clustering on the original data matrix to obtain the feature clustering indicator matrix, sample clustering indicator matrix and coefficient matrix.

[0149] Specifically, this embodiment proposes a semi-supervised sparse neighbor-constrained co-clustering algorithm based on matrix decomposition to cluster samples (EEG signal samples) and features (EEG features) simultaneously. The optimization goal is set to minimize the following objective function:

[0150]

[0151] stP∈Φ d×c , Q∈Φ n×c , S∈diag

[0152] Among them, P represents the clustering indicator matrix of features, Q represents the clustering indicator matrix of samples, and S is the coefficient matrix connecting P and Q.

[0153] Through this optimization goal, the algorithm can find the best clustering division in both row and column (sample and feature) dimensions, which is convenient for retaining the intrinsic structure of the data during the decoding process. In order to make full use of the label information of speech imagery EEG data, the quality and interpretability of the clustering results are improved. This embodiment completes the optimization solution of P, Q, and S under a semi-supervised framework. The data matrix X contains 100 labeled samples X l and 200 unlabeled samples X u In this embodiment, the intention labels of speech imagination are the words "up", "down", "left", and "right", so the number of fixed clustering categories c=4. For the convenience of representation, the original data matrix X is rearranged as The corresponding sample cluster indicator matrix Q can be expressed as Where Q l ∈Φ 100×4 , Q u ∈Φ 200×4 Furthermore, the clustering indicator matrix of features, the clustering indicator matrix of samples and the coefficient matrix are optimized under the semi-supervised framework.

[0154] In this embodiment, the optimization solution method is similar to the standard K-means algorithm, and the co-clustering algorithm is solved in an alternating manner.

[0155] The specific steps are as follows:

[0156] (1) P and Q initialization

[0157] Different from the traditional practice of initializing P and Q as arbitrary category indicator matrices, in this embodiment, the prior speech imagery intention label information is fully utilized when initializing the Q matrix. First, the cluster category is fixedly set as the speech imagery intention label category.

[0158] Furthermore, for the sample clustering indicator matrix Q corresponding to the labeled samples lIn this embodiment, it is initialized according to the prior label information. Specifically, for the i-th labeled sample (i∈{1, 2, ..., 100}), its corresponding indicator vector satisfies:

[0159]

[0160] The above initialization method ensures that the category affiliation of labeled samples in the clustering process is consistent with their prior labels.

[0161] For the sample clustering indicator matrix Q corresponding to the unlabeled sample u In this embodiment, random initialization is adopted. For the i-th unlabeled sample (i∈{101, 102, ..., 300}), its corresponding indicator vector is randomly assigned to satisfy:

[0162]

[0163] Among them, each unlabeled sample takes the value of 1 only in one randomly selected category, and the values ​​of the other categories are 0. Such random initialization provides a starting point for subsequent iterative optimization.

[0164] For the clustering indicator matrix P of the features, in this embodiment, a random initialization method is also adopted, where only a randomly selected category takes a value of 1, and the other categories take values ​​of 0.

[0165] Understandably, the label information of the sample is directly embedded into the co-clustering algorithm, providing a guide for the algorithm to maintain sensitivity to the sample intent during the clustering process. This initialization method can better combine the label information of the data with the clustering process, so that the final clustering result not only reflects the intrinsic structure of the data, but also retains and highlights the sample intent information.

[0166] (2) Solve for S with P and Q fixed:

[0167] P and Q are fixed in the current step, and our optimization goal becomes: Where X is the known original data matrix, P and Q are the known cluster indicator matrices in the current step, and S is the matrix we want to optimize so that PSQ T Reconstruct X as much as possible. In the quadratic optimization problem, minimize The solution of can usually be obtained by setting the derivative to zero. Finally, by taking the matrix derivative and setting the derivative to zero, we get the optimal solution formula for S. After expanding the objective function, we can get the following derivative of S:

[0168]

[0169] Setting the derivative to zero gives:

[0170] P T XQ=P T PSQ T Q

[0171] Then the optimal solution of S can be expressed as:

[0172] s=H -1 r,

[0173] Where H = P T PQ T Q, and r = P T XQ.

[0174] (3) Calculate Q when P and S are fixed u :

[0175] In this step, this embodiment only performs clustering on the sample indicator matrix Q corresponding to the unlabeled samples. u Iterate and calculate, and there are labeled sample parts Q l Remain unchanged.

[0176] Thanks to Q u Each vector q in i Only one element in Q is equal to 1, and the rest are 0, so Q u The solution can be determined by the following formula:

[0177]

[0178] where r k is the kth column of R, R = PS, x i Represents the feature vector of the i-th unlabeled sample, argmin k The expression selects the category k that minimizes the distance. The idea of ​​the formula is to minimize the unlabeled sample x i With each category center k The Euclidean distance between them determines x i The category to which it belongs.

[0179] In the above technical solution, only by u The iterative calculation can effectively utilize the potential structural information of unlabeled samples while keeping the category indication of labeled samples unchanged, thereby further optimizing the clustering results of unlabeled samples in the co-clustering process.

[0180] (4) Calculate P when Q and S are fixed:

[0181] Similar to the formula in step (3), the solution formula for P is as follows:

[0182]

[0183] Among them lk is the kth row of L, L = SQ T , x j Represents the jth eigenvector. Note that j represents the index of the feature, not the index of the sample; argmin k Select the category index k corresponding to the minimum value. This is similar to step (3), and the category attribution of each feature vector is determined by minimizing the Euclidean distance between the feature vector and the category center.

[0184] Repeat steps (2)-(4), alternately solve the optimization, and iterate until convergence. The algorithm gradually optimizes the clustering results of features and samples, so that feature clustering and sample clustering can promote each other and ultimately achieve better clustering results.

[0185] Finally, in module 4, the decoding results obtained by the semi-supervised learning model based on EEG samples and feature co-clustering are presented through the result display and decoding application module.

[0186] Specifically, on the one hand, from the perspective of model intent classification effectiveness, the clustering indicator matrix Qu of the unlabeled sample means the clustering indicator matrix Q of the unlabeled sample X u Intent classification decoding recognition results Tracing X by comparison u The real intention Y u The module will generate accurate evaluation indicators, including classification accuracy, recall rate, F1 score, confusion matrix, etc., to quantify the decoding performance of unlabeled samples. On the other hand, users can dynamically view the clustering results of different samples and features through an interactive interface, display the distribution of samples in the feature space through a clustering diagram, and evaluate and display the effect of the model's collaborative clustering in both feature and sample dimensions.

[0187] Furthermore, in this embodiment, the clustering indicator matrix P of the features optimized in module three will be further interpreted and analyzed. Through in-depth analysis of the P matrix, users can understand how the system clusters different features into specific categories, thereby revealing the role and contribution of these features in the speech imagery intention decoding process. This analysis helps to discover which features have higher discrimination ability for decoding tasks and which features may be redundant or irrelevant, promote the understanding of the speech imagery intention decoding process, and provide support for the improvement of experimental paradigms in the field of speech imagery intention decoding.

[0188] In order to make the feature importance obtained from the cluster indicator matrix P of the feature more robust, in this embodiment, analysis is performed from two perspectives:

[0189] Feature contribution: Each row of the P matrix corresponds to a feature, and each column corresponds to a cluster. For each feature i, calculate the sum of its contributions in all clusters:

[0190]

[0191] where s j are the diagonal elements of the S matrix. i The larger the value is, the higher the contribution and importance of the feature is considered to be.

[0192] Reconstruction error sensitivity ΔE i : By calculating X≈PSQ T The reconstruction error of can be reduced by zeroing or reducing the influence of certain rows of the P matrix (corresponding to specific features) and observing the impact on the overall reconstruction error. If removing certain features leads to a significant increase in the reconstruction error, then these features are more important.

[0193] Assume that the original reconstruction error is defined as:

[0194]

[0195] Assuming that the importance of a feature i needs to be evaluated, the i-th row of the P matrix can be set to zero, that is, The other rows remain unchanged. The reconstruction error at this time is:

[0196]

[0197] At this time, the importance of feature i can be measured by calculating the change in reconstruction error:

[0198] ΔE i =E featurei -E original

[0199] If ΔE i If it is larger, it means that feature i plays an important role in the reconstruction of the data, so the importance of this feature is higher.

[0200] By ΔE i With C i By ranking the weighted values ​​of , we can get the feature importance ranking, which systematically quantifies their comprehensive feature importance in the model:

[0201] Rank i =sort(αΔE i +βC i )

[0202] Where α and β are weight parameters.

[0203] In the data set of this embodiment, the top ten features and their corresponding feature contribution C i , Reconstruction error sensitivity ΔE i And the comprehensive feature importance I i (Ii =0.6*ΔE i +0.4*C i ) is as follows:

[0204]

[0205]

[0206] After comprehensive analysis, Gamma energy, Theta frequency band spectral center of gravity and Gamma average power show high importance in this task. These features show strong explanatory power in the clustering and data reconstruction of the model, and are suitable as core decoding features, while other features such as Theta peak frequency and frequency band entropy also have a high auxiliary effect on reconstruction and intention decoding.

[0207] In order to verify the effectiveness of the decoding model, this embodiment uses the decoding accuracy of unlabeled samples to measure the model performance. The following are the detailed steps of the evaluation process: Unlabeled sample decoding results generated by the co-clustering algorithm and with the true intent label Y u Comparison. The decoding performance of unlabeled samples is used to evaluate the generalization ability of the model to unlabeled data. The evaluation indicators include classification accuracy (indicates the proportion of correctly classified samples in all samples, used to measure the overall prediction accuracy of the model), recall (indicates the proportion of all true positive classes that are correctly predicted as positive classes, used to measure the recognition ability of the model for a specific task), and F1 score (the harmonic mean of classification accuracy and recall, which comprehensively reflects the classification ability of the model). To verify the effectiveness of this semi-supervised co-clustering method, we compared it with the traditional MLP model. The following are the performance comparison results of the experiment:

[0208]

[0209]

[0210] The results show that this embodiment achieves effective improvement in feature selection and unlabeled data utilization through the semi-supervised co-clustering method. Compared with the MLP model trained directly on labeled data, it significantly improves the accuracy and robustness of speech imagery intention decoding.

[0211] Finally, the model is deployed on the brain-computer interface (BCI) application to support the decoding of real-time collected EEG signals and provide instant feedback on speech imagination intentions based on the decoding results, helping users control external devices or perform other operations through thinking.

[0212] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions and variations of these embodiments including components are made without departing from the principles and spirit of the present invention, and still fall within the scope of protection of the present invention.

Claims

1. A semi-supervised speech imagery intention decoding method based on EEG collaborative clustering, characterized in that: The steps include: Step 1: Collecting EEG data of speech imagination, labeling and preprocessing the collected EEG data according to speech imagination; Step 2: Extract features from the preprocessed EEG data. The feature extraction method is as follows: first, extract EEG features from five frequency bands, then extract corresponding time domain features, frequency domain features, time-frequency domain features, and spatial domain features for each frequency band and different EEG electrode channels, and finally connect, organize, and standardize all the extracted features to obtain the original data matrix; Step 3: Perform feature clustering and sample clustering on the original data matrix to obtain a feature clustering indicator matrix, a sample clustering indicator matrix and a coefficient matrix; Step 4: Under the semi-supervised framework, the clustering indicator matrix of the features, the clustering indicator matrix of the samples and the coefficient matrix are optimized and solved; Step 5: Use visualization to display the decoding results obtained by the semi-supervised learning model based on EEG samples and feature co-clustering.

2. According to the semi-supervised speech imagery intention decoding method based on EEG collaborative clustering according to claim 1, it is characterized in that: In the step 1, when collecting EEG data, the subject performs a speech imagery task when wearing an EEG cap and receiving relevant prompts, and the EEG generated by the subject during speech imagery will be collected as raw EEG data.

3. According to the semi-supervised speech imagery intention decoding method based on EEG collaborative clustering according to claim 1, it is characterized in that: The preprocessing method is: firstly, down-sampling the collected EEG data and removing the signals corresponding to the bad electrodes, and then regenerating them by interpolating adjacent ones; and then filtering the EEG data to 0.1-50 Hz by a bandpass filter to remove noise.

4. The semi-supervised speech imagery intention decoding method based on EEG collaborative clustering according to claim 1 is characterized in that: The five frequency bands are: Delta band, Theta band, Alpha band, Beta band, and Gamma band.

5. The semi-supervised speech imagery intention decoding method based on EEG collaborative clustering according to claim 1 is characterized in that: The time domain features include the center of mass, mean absolute difference, peak-to-peak distance, and Shannon entropy on the time axis; the frequency domain features include spectral distance, fundamental frequency, maximum power spectral density, maximum frequency, median frequency, spectral center of gravity, attenuation of spectral amplitude, spectral kurtosis, spectral skewness, spectral slope, spectral variability, spectral positive turning point, spectral start point, power spectrum bandwidth, Fourier transform mean coefficient, and spectral entropy; the time and frequency domain features include wavelet transform features and empirical mode decomposition features; the spatial domain features include using a common spatial pattern algorithm to spatially filter the EEG data and extracting the spatial distribution components of each class from the multi-channel EEG data.

6. A semi-supervised speech imagery intention decoding method based on EEG collaborative clustering according to any one of claims 1 to 5, characterized in that: In step 3, the original data matrix is ​​defined as X, the number of samples in the original data matrix X is n, the number of features is d, and the original data matrix X is subjected to feature clustering and sample clustering, and the optimization objective formula of the co-clustering is as follows: s.t.P∈Φ d×c ,Q∈Φ n×c ,S∈diag Among them, P represents the clustering indicator matrix of features, Q represents the clustering indicator matrix of samples, S is the coefficient matrix used to connect P and Q, the symbol T represents the transpose operation of the matrix, the symbol F represents the Frobenius norm, d represents the number of features, c represents the number of clusters, n represents the number of samples, diag represents the diagonal matrix, and S∈diag represents that S is a diagonal matrix.

7. The semi-supervised speech imagery intention decoding method based on EEG collaborative clustering according to claim 6 is characterized in that: The original data matrix X includes labeled samples X l and unlabeled samples X u , where there are labeled samples X l The number of unlabeled samples is l, u The number of is u, so the original data matrix X is rearranged into The corresponding sample cluster indicator matrix Q can be expressed as Where Q l ∈Φ l×c , Q u ∈Φ (n-l)×c .

8. The semi-supervised speech imagery intention decoding method based on EEG collaborative clustering according to claim 7, characterized in that: The step 4 includes initializing the clustering indicator matrix P of the features and the clustering indicator matrix Q of the samples.

9. The semi-supervised speech imagery intention decoding method based on EEG collaborative clustering according to claim 8, characterized in that: The initialization method of the cluster indicator matrix Q of the sample is: First, the clustering category is fixed to the speech imagery intention label category; For the sample cluster indicator matrix Q corresponding to the labeled sample l , for the i-th labeled sample (i∈{1,2,...,l}), its corresponding indicator vector satisfies: Where c refers to the number of language imaginary intention label categories; For the sample clustering indicator matrix Q corresponding to the unlabeled sample u , using random initialization, for the i-th unlabeled sample (i∈{l+1,l+2,...,n}), its corresponding indicator vector is randomly assigned to satisfy: Each unlabeled sample takes the value of 1 only in one randomly selected category and the values ​​of the other categories are 0.

10. The semi-supervised speech imagery intention decoding method based on EEG collaborative clustering according to claim 9, characterized in that: The cluster indicator matrix P of the feature is initialized by using a random initialization method, where only a randomly selected category takes a value of 1, and the other categories take values ​​of 0.

11. The semi-supervised speech imagery intention decoding method based on EEG collaborative clustering according to claim 10, characterized in that: In step 4, the optimization solution method is: Solve for S with P and Q fixed, and the solution formula is as follows: S=H -1 r, Where H = P T PQ T Q, and r = P T XQ; Calculate Q with P and S fixed u : Thanks to Q u Each vector q in i Only one element in Q is equal to 1, and the rest are 0, so Q u The solution can be determined by the following formula: where r k is the kth column of R, R = PS, x i Represents the feature vector of the i-th unlabeled sample, argmin k Express the selection of category k that minimizes the distance; When Q and S are fixed, calculate P and solve the formula as follows: Among them l k is the kth row of L, L = SQ T , x j Represents the jth eigenvector. Note that j represents the index of the feature, not the index of the sample; argmin k Select the category index k corresponding to the minimum value; Repeat the above solution process, alternately solve the optimization, and iteratively update until convergence.

Images