A method for classifying and recognizing group motor imagery based on electroencephalogram multi-view decoding

Abstract

The present invention proposes a method for classifying and recognizing group motor imagery based on multi-view decoding of electroencephalogram (EEG). By using common spatial patterns to mine the discriminative spatial features of multi-channel EEG, the spatial view relationship oriented to the intra-individual EEG characteristics is constructed. At the same time, channel selection is carried out based on the Granger causality network, and the coupling information between individuals is extracted through a graph convolutional network to establish a coupling relationship view of group motor imagery EEG. Then, through a multi-view model and a fusion strategy, the advantages of each view are explored to the greatest extent, and a subspace clustering algorithm based on self-representation learning is used to jointly decode the two types of view representation information. By constructing a multi-view representation that combines the spatio-temporal features of single-person motor imagery EEG and the cross-brain coupling features of the group, the spatial view relationship of the intra-individual EEG characteristics is realized, and thus the connections between various channels of multiple brains are fully considered, thereby improving the classification accuracy of EEG motor imagery.

Description

Technical Field

[0001] The present invention relates to a neuroelectrophysiological signal analysis technology in the field of brain cognition, and is a method for classifying and recognizing group motor imagery based on electroencephalogram multi-view decoding. Background Art

[0002] A motor imagery brain-computer interface (MI-BCI) refers to a situation where the body does not perform actual movements, but instead uses the brain's thinking to imagine actual limb movements. Finally, through a controller, the brain can achieve actual operation of a computer or other electronic devices. Motor imagery signals are generally collected through electroencephalogram (EEG) signals.

[0003] The characteristic of a group brain-computer interface is to record and analyze the EEG signals of multiple individuals to promote information transmission and collaborative control among groups. It not only solves the challenges in signal decoding and control in a single-user brain-computer interface, but also solves the problems of cooperation, decision-making fusion, and interaction among group members. This innovative paradigm enables direct communication and cooperation among multiple individuals through EEG signals, thereby overcoming the limitations of existing single-user brain-computer interfaces and opening up broader application and research avenues. Currently, the group brain-computer interface decoding technology is still in its early stage, and existing decoding methods are often direct adaptations of traditional single-person models.

[0004] For example, in the invention patent "Group Motor Imagery Brain-Computer Interface Decoding Method and System Based on Feature Interaction" by Liu Riheng of Hangzhou Dianzi University, this method considers the current state of the user and dynamically adjusts the weights of different users, enhancing the accuracy of the system. However, in practice, during the execution of a multi-brain motor imagery task, multiple regions of multiple brains need to be involved, and there must be interconnections among these regions. The above method does not consider these factors, resulting in low accuracy. Summary of the Invention

[0005] To address the above deficiencies, the present invention proposes a method for classifying and recognizing group motor imagery based on multi-view decoding of electroencephalogram (EEG). The method utilizes common spatial patterns to mine discriminative spatial features of multi-channel EEG, thereby constructing a spatial view relationship oriented to the intra-individual EEG characteristics. Meanwhile, channel selection is performed based on the Granger causality network, and the coupling information between individuals is extracted through a graph convolutional network to establish a coupling relationship view of group motor imagery EEG. Furthermore, by designing an appropriate multi-view model and fusion strategy, the method explores how to maximize the advantages of each view. Considering that the advantage of self-representation learning lies in capturing the internal structure and relationships in EEG signals, and the ability of subspace clustering algorithms lies in effectively distinguishing the characteristics and patterns of different participants, a subspace clustering algorithm based on self-representation learning is implemented to jointly decode the two types of view representations.

[0006] In a first aspect, the present invention provides a method for classifying and recognizing group motor imagery based on multi-view decoding of EEG, comprising the following steps:

[0007] Step 1: Acquisition of group motor imagery EEG data;

[0008] Collect group EEG signal data of at least two individuals under motor imagery tasks, where the motor imagery tasks include left-hand motor imagery, right-hand motor imagery, and idle-state tasks;

[0009] Step 2: Preprocessing of the EEG signals collected in Step 1;

[0010] The preprocessing includes downsampling and artifact removal;

[0011] Step 3: Extraction of spatio-temporal features of single-person motor imagery EEG;

[0012] For the single-person EEG signals preprocessed in Step 2, the feature vectors with the maximum discrimination are extracted through common spatial patterns (CSP) as the spatio-temporal features of single-person motor imagery EEG;

[0013] Step 4: Extraction of group cross-brain coupling features;

[0014] 4-1 Cross-brain channel selection;

[0015] For the group EEG signals preprocessed in Step 2, calculate the Granger causality values of the same channels between individuals; then, arrange the group EEG signals in descending order according to the Granger causality values, and select the top channel pairs of data with mutual causal relationships from the group EEG signals; combine the channel pairs of data into a tensor form;

[0016] 4-2 Group coupling feature extraction;

[0017] Integrate the learning mechanism of graph convolution, and for the top Each channel of the data is regarded as a node in the graph of the graph convolutional model. Therefore, the number of nodes should be n represents the number of individuals;

[0018] The adjacency matrix of each node is composed of the phase synchronization values between nodes;

[0019] The feature matrix of a node includes the degree, power spectral density (PSD), and differential entropy (DE) features of the node;

[0020] By using the information of the feature matrices of neighboring nodes to update the feature matrix of the current node, cross-brain features with coupling information can be finally obtained, which are the group cross-brain coupling features;

[0021] Step Five: Multi-view representation and learning of combined single-person and inter-individual coupling information;

[0022] 5-1 Multi-view subspace self-representation

[0023] Project the single-person motor imagery EEG spatio-temporal features and group cross-brain coupling features into a low-dimensional subspace; then, calculate the representation matrices of the single-person motor imagery EEG spatio-temporal features and group cross-brain coupling features on the low-dimensional subspace respectively. This representation matrix contains the linear combination coefficients of the features in the low-dimensional subspace, and each representation matrix represents a view subspace self-representation;

[0024] 5-2 Regard the view subspace self-representations of all features as a tensor, and impose a low-rank constraint on this tensor; then, calculate the affinity matrix according to the tensor with the above low-rank constraint; obtain the eigenvalues of the Laplacian matrix L according to the affinity matrix, and cluster the above eigenvalues to finally obtain the classification result of group motor imagery.

[0025] Furthermore, Step Three specifically includes the following sub-steps:

[0026] Assume that the preprocessed EEG signal of any individual is represented as X ∈ R C×S , where C represents the number of channels and S represents the number of sampling points; divide X into X 1 and X 2 according to the left and right hand motor imagery tasks. X i (i = 1, 2) represents the EEG signals corresponding to the left and right hand motor imagery tasks;

[0027] To calculate the covariance matrix of X, assume C < S. In the case of two tasks, X 1 and X 2 are respectively:

[0028]

[0029] Among them, S 1 and S 2 represent the sampling points of left - hand and right - hand motor imagery, and S M represents the source signal common to S 1 and S 2 ; C M represents the common spatial pattern corresponding to S M ; C i represents the common spatial pattern corresponding to S i ;

[0030] The X 1 and X 2 of the spatio - temporal signal, the normalized covariance matrices ∑ 1 and ∑ 2 :

[0031]

[0032] Among them, trace() represents the sum of the elements on the diagonal of the matrix;

[0033] Next, solve for ∑ in the mixed space:

[0034]

[0035] where represents the average value of ∑ 1 ;

[0036] Perform eigenvalue decomposition on ∑:

[0037] ∑ = AμA T (4)

[0038] where A is the eigenvector matrix and μ is the diagonal matrix of eigenvalues arranged in descending order;

[0039] Whiten the transformation of A to obtain the whitening matrix Y:

[0040]

[0041] Apply equation (5) to C 1 and C 2 to get:

[0042] S i = Y∑ i Y T , (i = 1, 2) (6)

[0043] S 1 and S 2 have common eigenvectors, and there exist two diagonal matrices μ 1 , μ 2 and the same eigenvector matrix H, for S 1and S 2 Performing principal component decomposition gives:

[0044] S i = Hμ i H T , (i = 1, 2) (7)

[0045] and

[0046] μ 1 + μ 2 = I (8)

[0047] where I represents the identity matrix;

[0048] Using the matrix H, the projection matrix D can be obtained:

[0049] D = H T Y (9)

[0050] where the projection matrix D is the spatial filter corresponding to the matrix H;

[0051] Filtering X 1 and X 2 through the spatial filter D gives W 1 and W 2 :

[0052] W i = D × X i , (i = 1, 2) (10)

[0053] The characteristics f 1 and f 2 of X 1 and X 2 can be obtained as:

[0054]

[0055] where var represents the variance of the matrix;

[0056] Furthermore, step 4-1 specifically includes the following sub-steps:

[0057] 4-1-1 Calculate the cross-channel Granger causality for the EEG data between individuals;

[0058] Arbitrarily take the EEG time signals X a (t) and X b (t) of two individuals. Assume that X b (t) is composed of X a (t) and its lag terms X a (t - 1), X a (t - 2), …, X aIt is linearly combined by (t - p), and then an error term ε(t) is added; p represents the lag time; in this way, the expression of X b can be written as:

[0059]

[0060] Then, calculate X b The probability of occurrence given X a is estimated through a logistic regression model, that is, the probability estimate of binary classification for X is made given X b and its lag terms; a ;

[0061]

[0062] Finally, the Granger causality value:

[0063]

[0064] According to the Granger causality value, perform a descending order sorting, and take the top channel pairs;

[0065] The channel pair refers to the EEG signals of two individuals a and b in the same channel;

[0066] Furthermore, step 4-2 specifically includes the following sub-steps:

[0067] 4-2-1, Obtain the top channel pairs obtained in step 4-1. Consider each channel as a node in the graph, so graph data of nodes is obtained; among them, the graph data includes an adjacency matrix and a node feature matrix;

[0068] a. The adjacency matrix is composed of the phase-locking values between nodes; the phase-locking value PLV is a phase-based functional connectivity method, and what it actually measures is the phase difference between two channel signals. The calculation method is as follows:

[0069]

[0070] where sn represents the maximum moment, respectively represent the phase angles of the channel pair X a , X b at time point t, and i represents the imaginary unit.

[0071] The value range of PLV is [0, 1], and the larger the value, the stronger the phase synchronization degree between the two signals;

[0072] b. The node feature matrix includes the degree, power spectrum density (PSD), and differential entropy (DE) features of the nodes;

[0073] For the degree of the node, set the threshold ξ = 0.7 according to the PLV value calculated in sub-step 4-2-1(a); PLV i,j (between the i-th node and the j-th node) ≥ ξ, degree i,j = 1; Similarly, when PLV i,j < ξ, degree i,j = 0;

[0074] The power spectrum density (PSD) is a method for calculating the power of each frequency band of a signal. The periodogram method is used to calculate the power spectrum density of the EEG signal. The specific process is as follows:

[0075] For a finite-length discontinuous signal X(t), t = 0, 1, …, N0 - 1, where X(t) represents the t-th discrete signal value in the time domain; first perform the discrete Fourier transform, and its formula is:

[0076]

[0077] In the formula, U(k) represents the k-th frequency component in the frequency domain. W N0 = e -i2π / N , k = 0, 1, …, N0 - 1; kt represents the weighted sum of each time point of the signal, where the frequency k determines the weighting factor of each time point; i represents the imaginary number, and W N0 has periodicity, so U(k) also has periodicity, and its period size is N0. Then, square the obtained U(k) and divide it by N0 to get the power spectrum density:

[0078] PSD = U 2 (k) / N0 (17)

[0079] c. Extract DE features from the channel pairs obtained in step 4-1 in five frequency bands: δ band (1 - 3 Hz), θ band (4 - 7 Hz), α band (8 - 13 Hz), β band (14 - 30 Hz), γ band (31 - 50 Hz):

[0080] Use the Hamming window to filter the channel pairs obtained in step 4-1, perform the fast Fourier transform on the filtered data, and calculate the differential entropy of the above five frequency bands;

[0081] The definition method of differential entropy is as follows:

[0082] Let the value of the random variable Z be Z = {z 1 , z2 ,…,z n1}(n1≥1), and the corresponding probability is According to the definition method of Shannon information entropy, the information quantity of this non-deterministic system is shown in Equation (18):

[0083]

[0084] Replace the state probability p in the time domain in the above formula i with the power spectral density in the frequency domain defined based on the fast Fourier transform From this, the definition of differential entropy is shown in Equation (19):

[0085]

[0086] where represents the power spectral density;

[0087] By using the feature matrix information of neighbor nodes to update the feature matrix of the current node, the feature matrix of the current node can finally be used as the cross-brain feature with coupling information, that is, the group cross-brain coupling feature;

[0088] Furthermore, Step 5 specifically includes the following sub-steps:

[0089] 5-1, Multi-view subspace self-representation;

[0090] Take the single-person motor imagery EEG spatio-temporal features and group cross-brain coupling features generated in Steps 3 and 4 as different views and input them into the multi-view learning model; the method for processing multi-view data is as follows:

[0091]

[0092] s.t. X (v) = X (v) Z (v) + E (v) , v = 1, 2, …, V, (20)

[0093] where, X (v) represents the data matrix of the v-th view, corresponding to the single-person motor imagery EEG spatio-temporal features or group cross-brain coupling features, Z (v) and E (v) represent the subspace representation matrix and reconstruction error matrix of the v-th view respectively; λ v is a hyperparameter that controls the loss penalty intensity of the v-th view; V is the number of views; R represents the regularization term;

[0094] 5-2, Low-rank tensor constraint;

[0095] Use tensors to constrain the correlation between different views; among them, the tensor nuclear norm is defined as:

[0096]

[0097] Among them, ξ m is a constant that satisfies ξ i > 0 and ; is an M - order tensor, and Z (m) is a matrix obtained by unfolding the tensor along the m - th mode, and this mode is defined as ||·|| * represents the nuclear norm, which is used to enforce the tensor under the low - rank constraint; essentially, the nuclear norm of a tensor is the convex combination of the nuclear norms of all matrices along each mode; under the low - rank tensor constraint, the objective function (20) of sub - step 5 - 1 is formulated as:

[0098]

[0099] Among them, Ψ(·) constructs the tensor (v) by combining different representations Z into a 3 - order tensor with dimensions N2×N2×V; N2 represents the dimension of Z (v) ; E = [E (1) , E (2) , …, E (v) is formed by vertically concatenating the error columns corresponding to each view; ||·|| 2,1 is the l 2,1 norm, which encourages the columns of E to be zero; λ represents the hyperparameter;

[0100] Substituting formula (21) into formula (22), therefore, formula (22) is transformed into:

[0101]

[0102] Among them encodes the strength of the low - rank tensor constraint, and the constraint is the same as in (22); Z (v) is the subspace representation matrix corresponding to the v - th view, and Z (m) is the m - th mode unfolding matrix of ;

[0103] 5 - 3, Affinity matrix and spectral clustering;

[0104] According to formula (23), the affinity matrix B can be obtained:

[0105]

[0106] Calculate the degree matrix Q using the following formula:

[0107]

[0108] where B ij represents the element in the i-th row and j-th column of the affinity matrix B, and Q is a diagonal matrix of q i constituting an n3*n3; Calculate the Laplacian matrix L:

[0109] L = Q - B (26)

[0110] Calculate the matrix eigenvalues of L, sort the matrix eigenvalues from smallest to largest, take the first k matrix eigenvalues, and calculate the eigenvectors u 1 , u 2 , …, u k ; Form a matrix with the k column vectors:

[0111] U = {u 1 , u 2 , …, u k}, U ∈ R n3*k (27)

[0112] Let ve i ∈ R k be the vector of the i-th row of U, where i = 1, 2, …, n3;

[0113] Use the k-means algorithm to cluster the new sample points VE = {ve 1 , ve 2 , …, ve n3} into clusters C 1 , C 2 , …, C k .

[0114] In a second aspect, the present invention provides a group motor imagery classification and recognition system, including:

[0115] A data acquisition and preprocessing module for acquiring group motor imagery EEG data and preprocessing the collected EEG signals;

[0116] A single-person feature extraction module for extracting the spatio-temporal features of single-person motor imagery EEG;

[0117] A group feature extraction module for extracting group cross-brain coupling features;

[0118] A classification and recognition module for inputting the spatio-temporal features of single-person motor imagery EEG and group cross-brain coupling features as different views into a multi-view learning model for multi-view representation and learning, and finally obtaining a classification result.

[0119] In a third aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed on a computer, the computer is made to execute the method described above.

[0120] In a fourth aspect, the present invention provides a computing device, including a memory and a processor. An executable code is stored in the memory. When the processor executes the executable code, the method described above is implemented.

[0121] The technical effect of the present invention is:

[0122] The present invention proposes a multi-view brain decoding method for a group motor imagery brain-computer interface. By constructing a multi-view representation that combines the spatio-temporal features of single-person motor imagery EEG and the cross-brain coupling features of the group, the spatial view relationship of the EEG characteristics within an individual is realized, and then the connections between various channels of multiple brains are fully considered, thereby improving the classification accuracy of EEG motor imagery.

[0123] In addition, compared with other decoding methods for motor imagery, the classification accuracy of motor imagery of the model based on multi-view learning in the present invention has been improved, and the accuracy reaches 75.48% ± 06.34%. BRIEF DESCRIPTION OF THE DRAWINGS

[0124] Figure 1 is a module flowchart of the method according to an embodiment of the present invention;

[0125] Figure 2 is a multi-view representation and learning flowchart of the method according to an embodiment of the present invention;

[0126] Figure 3 is an experimental paradigm of the method according to an embodiment of the present invention;

[0127] Figure 4 is the channel position for collecting EEG signals of the method according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0128] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0129] As Figure 1 , Figure 2 shown, this embodiment provides a group motor imagery classification method, which specifically includes the following steps:

[0130] Step (1) Acquisition of group motor imagery EEG dataset, as Figure 3 .

[0131] Step (1-1) Select 3 different motor imagery tasks from the material library, namely motor imagery of the left hand, right hand, and idle state;

[0132] In step (1-2), imagine 375 trials for each experiment; there is a 1-second cue before each imagination, the imagination time for each time is 4 seconds, and there is a 1-second break after each imagination ends.

[0133] In step (1-3), collect the group EEG signal data of 10 subjects under the motor imagery task. Figure 4 This is the channel position for collecting EEG signals in the method of the embodiment of the present invention.

[0134] Step (2) EEG signal preprocessing and spatio-temporal feature extraction

[0135] In step (2-1), downsample the original signal to 100 Hz and remove the signals interfered by electrooculogram and electromyogram;

[0136] Next, we perform band-pass filtering (4-40 Hz) on the EEG, and then calculate the motor imagery EEG features through the CSP algorithm, and set the number of components to 10.

[0137] Step (3) Extraction of single-person motor imagery EEG spatio-temporal features;

[0138] Assume that the single-person EEG signal is represented as X∈R C×S , where C represents the number of channels and S represents the number of sampling points. Divide the training data into X 1 and X 2 according to the left-hand and right-hand motor imagery tasks, and X i (i = 1, 2) represents the EEG signals corresponding to different categories of tasks. To calculate its covariance matrix, assume C < S. In the case of two tasks, X 1 and X 2 are respectively:

[0139]

[0140] Among them, S 1 and S 2 represent two categories of tasks, S M represents the source signal common to S 1 and S 1 , C M represents the common spatial pattern corresponding to S M ; C i represents the common spatial pattern corresponding to S i .

[0141] The normalized covariance matrices ∑ 1 and ∑ 2 of the spatio-temporal signals X 1 and ∑ 2 are:

[0142]

[0143] Among them, trace() represents the sum of the elements on the diagonal of the matrix. Next, solve for ∑ in the mixed space:

[0144]

[0145] Perform eigenvalue decomposition on ∑:

[0146] ∑ = AμA T (4)

[0147] Among them, A is the eigenvector matrix, and μ is the diagonal matrix of the eigenvalues arranged in descending order. Whitening transformation of A gives the whitening matrix Y:

[0148]

[0149] Apply equation (5) to C 1 and C 2 to get:

[0150] S 1 = Y∑ i Y T , (i = 1, 2) (6)

[0151] S 1 and S 2 have common eigenvectors, and there exist two diagonal matrices μ 1 , μ 2 and the same eigenvector matrix H. Perform principal component decomposition on S 1 and S 2 to obtain:

[0152] S i = Hμ i H T , (i = 1, 2) (7)

[0153] And

[0154] μ 1 + μ 2 = I (8)

[0155] The projection matrix D can be obtained using the matrix H:

[0156] D = H T Y (9)

[0157] Among them, the projection matrix D is the spatial filter corresponding to the matrix H.

[0158] Filter the EEG matrices X 1 and X 2 of the training set through the filter D to obtain W 1 and W 2 :

[0159] W i = D × X i , (i = 1, 2) (10)

[0160] The characteristic f can be obtained 1 and f 2 are as follows:

[0161]

[0162] For the test set X n , its feature vector f n is extracted as follows:

[0163]

[0164] Step (4) Extraction of group cross-brain coupling features;

[0165] Step (4-1), Normalize the EEG signals using z-score, and the normalization formula is as shown in Equation (13):

[0166]

[0167] where X is the EEG signal on each channel, is the average value of the EEG signal on each channel, and S is the standard deviation of the EEG signal on each channel.

[0168] Step (4-2) Cross-brain channel selection. For all individuals, analyze and calculate the Granger causality values of each pair of the same channels of the brains. Let any two preprocessed multi-brain EEG signals X a (t) and X b (t) (62 channels * 400 time points) be taken, and calculate the Granger causality values of the same channels of the two brains.

[0169] Assume that X b (t) is linearly combined by X a (t) and its lag terms X a (t-1), X a (t-2), …, X a (t-p), plus an error term ε(t); p represents the lag time; thus, the expression of X b (t) can be written:

[0170]

[0171] Then, calculate the probability of X b appearing given X a is estimated through a logistic regression model, that is, given X b ​In the case of it and its lag terms, the probability estimation of binary classification for X a is carried out;

[0172]

[0173] Finally, the Granger causality value:

[0174]

[0175] Then, select those channel pairs with mutual causality, and their causality values are all greater than 0. Combine the information of these channel pairs into a tensor form with a dimension of where represents the number of channel pairs with mutual causality.

[0176] Step (4-3) Group coupling feature extraction. Integrate the learning mechanism of graph convolution, regard each channel as a node in the graph, so the number of nodes should be 2X (where X is the number of channel selections). The adjacency matrix of each node is composed of the phase synchronization values between nodes. The feature matrix includes the degree, power spectral density (PSD), and differential entropy (DE) features of the nodes. By using the information of neighbor nodes to update the feature representation of the current node, cross-brain features with coupling information can be obtained finally.

[0177] Obtain the channel pairs of brain X and brain Y obtained in step three, regard each channel as a node in the graph, so graph data of nodes are obtained. Among them, the graph data contains an adjacency matrix A and a node feature matrix X.

[0178] a. The adjacency matrix is composed of the phase-locking values between nodes; the phase-locking value PLV is a phase-based functional connection method, and what it actually measures is the phase difference between the signals of two channels. The calculation method is as follows:

[0179]

[0180] where sn represents the maximum moment, respectively represent the phase angles of channel pair X a and X b at time point t, and i represents the imaginary unit.

[0181] The value range of PLV is [0,1], and the larger the value, the stronger the phase synchronization degree between the two signals;

[0182] b. The node feature matrix includes the degree, power spectral density (PSD), and differential entropy (DE) features of the nodes;

[0183] The degree of the node sets the threshold ξ = 0.7 according to the PLV value calculated in sub-step 4-2-1(a); PLV i,j (located at the i-th node and the j-th node) ≥ ξ, degree i,j = 1; Similarly, when PLV i,j < ξ, degree i,j = 0;

[0184] The Power Spectrum Density (PSD) is a method for calculating the power of each frequency band of a signal. The periodogram method is used to calculate the power spectrum density of the EEG signal. The specific process is as follows:

[0185] For a finite-length discontinuous signal X(t), t = 0, 1, …, N0 - 1, where X(t) represents the t-th discrete signal value in the time domain; first, perform a discrete Fourier transform, and its formula is:

[0186]

[0187] In the formula, U(k) represents the k-th frequency component in the frequency domain. W N0 = e -i2π / N , k = 0, 1, …, N0 - 1; kt represents the weighted sum of each time point of the signal, where the frequency k determines the weighting factor of each time point; i represents the imaginary number, where W N0 has periodicity, so U(k) also has periodicity, and its period size is N0. Then, square the obtained U(k) and divide it by N0 to get the power spectrum density:

[0188] PSD = U 2 (k) / N0 (19)

[0189] c. Extract DE features from the channel pairs obtained in step 4-1 in 5 frequency bands: δ band (1 - 3 Hz), θ band (4 - 7 Hz), α band (8 - 13 Hz), β band (14 - 30 Hz), γ band (31 - 50 Hz):

[0190] Use a Hamming window to filter the channel pairs obtained in step 4-1, perform a fast Fourier transform on the filtered data, and calculate the differential entropy of the above five frequency bands;

[0191] The definition method of differential entropy is as follows:

[0192] Let the value of the random variable Z be Z = {z 1 , z 2 , …, z n1} (n1 ≥ 1), and the corresponding probability is According to the definition method of Shannon information entropy, the information quantity of this non-deterministic system is shown in Equation (18):

[0193]

[0194] Replace the state probability p in the time domain in the above formula i with the power spectral density in the frequency domain defined based on the fast Fourier transform Thus, the definition of differential entropy is introduced as shown in Equation (20):

[0195]

[0196] where represents the power spectral density.

[0197] By using the feature matrix information of neighbor nodes to update the feature matrix of the current node, the feature matrix of the current node can finally be used as the cross-brain feature with coupling information, that is, the group cross-brain coupling feature.

[0198] Step (5) Multi-view representation and learning method combining single-person and inter-individual coupling information. This step mainly includes two parts: multi-view subspace self-representation and tensor low-rank constraint. Among them, multi-view subspace self-representation means projecting each data sample into a low-dimensional subspace through dimensionality reduction. Then, for each data sample, calculate its representation matrix in this subspace, and this representation matrix contains the linear combination coefficients of the data sample in the subspace. At the same time, tensor low-rank constraint means regarding these representation matrices as a tensor, imposing a low-rank constraint on this tensor, which well models the cross information between different views, effectively reduces the redundancy of learning subspace representation, and improves the accuracy of clustering. Then, obtain the affinity matrix according to the subspace self-representation matrix and input it into the spectral clustering algorithm to generate the final clustering result.

[0199] Step (5-1) Multi-view subspace self-representation.

[0200] Input the single-person motor imagery EEG spatio-temporal features and group cross-brain coupling features generated in Steps 3 and 4 into the multi-view learning model. The method for processing multi-view data is as follows:

[0201]

[0202] s.t. X (v) =X (v) Z (v) +E (v) ,v=1,2,…,V, (22)

[0203] where, X (v) 、Z (v) and E (v)respectively represent the data matrix, subspace representation matrix, and reconstruction error matrix of the v-th view. λ v is a hyperparameter that controls the loss penalty intensity of the v-th view. V is the number of views. R represents the regularization term.

[0204] Step (5-2) Low-rank tensor constraint. However, the method in sub-step (5-1) only considers each view independently and ignores the correlation between different views. To solve this problem, this method uses a tensor to constrain these affinity matrices. Among them, the tensor nuclear norm is defined as:

[0205]

[0206] where ξ m is a constant that satisfies ξ i > 0 and . is an M-order tensor, and Z (m) is a matrix obtained by unfolding the tensor along the m-th mode, and this mode is defined as The nuclear norm ||·|| * forces the tensor under the low-rank constraint condition. Essentially, the nuclear norm of the tensor is a convex combination of the nuclear norms of all matrices along each mode. Under the low-rank tensor constraint, the objective function of step (5-1) is formulated as:

[0207]

[0208] where Ψ(·) constructs the tensor (v) by combining different representations Z into a 3-order tensor with a dimension of N2×N2×V; N2 represents the dimension of Z (v) ; E = [E (1) , E (2) , …, E (v) is formed by vertically concatenating the error columns corresponding to each view together; ||·|| 2,1 is the l 2,1 norm, which encourages the columns of E to be zero; λ represents the hyperparameter; note that this method normalizes the data matrices of different views to force the errors of different views to have the same scale, which can reduce the variation of the error magnitudes between different views.

[0209] Substitute formula (23) into formula (24), so formula (24) is converted to:

[0210]

[0211] where Encode the strength of the low-rank tensor constraint, and the constraint is the same as in (24). Z (v) is the subspace representation matrix corresponding to the v-th view, while Z (m) is 's m-th mode unfolding matrix.

[0212] Step (5-3) Affinity matrix and spectral clustering.

[0213] According to formula (25), the affinity matrix B can be obtained:

[0214]

[0215] Calculate the degree matrix Q using the following formula:

[0216]

[0217] where B ij represents the element in the i-th row and j-th column of the affinity matrix B, and Q is the n3*n3 diagonal matrix composed of q i ;

[0218] Calculate the Laplacian matrix:

[0219] L = D - S (28)

[0220] Calculate the matrix eigenvalues of L, sort the matrix eigenvalues from smallest to largest, take the first k eigenvalues, and calculate the eigenvectors u1, u2,..., u k . Form a matrix with the k column vectors:

[0221] U = {u 1 , u 2 ,..., u k}, U ∈ R n3*k (29)

[0222] Let y ∈ R k be the vector of the i-th row of U, where i = 1, 2,..., n3.

[0223] Use the k-means algorithm to cluster the new sample points VE = {ve 1 , ve 2 ,..., ve n3} into clusters C 1 , C 2 ,..., C k .

[0224] Step (6) Define the input and output of the model. The input of the model is the single-person spatio-temporal CSP feature in Step 3 and the cross-brain coupling feature in Step 4. The model consists of two outputs, namely the tensor with low-rank constraint obtained in Step 5 and the probability that the sample belongs to each category.

[0225] Step (7) Define the objective function. The final objective function of this method consists of two loss functions. First, impose a low-rank constraint on the tensor formed by the self-representation matrix. Since rank() is non-convex and difficult to solve in an optimization problem, we use its convex approximation, the nuclear norm ||W||*, to approximate the constraint. Second, impose the L2,1 norm on the tensor formed by the reconstruction error matrix between the self-representation matrix and the original matrix. This step constrains most of the elements in the reconstruction error tensor to be zero, thus achieving sparsity.

[0226] Step (8) Training and testing. First, we use the input data generated in Step Six to train the model. During the training process, the model improves its performance by learning the features of the data and the relationships between them. Once the model is trained, we use the samples in the test set to test it. During the testing process, we make predictions on the inputs of the test samples by the model and obtain the classification results. Next, we use these classification results to evaluate the performance of the model and calculate its accuracy in the classification task.

[0227] Step (9) Evaluate the results of group motor imagery decoding using the average classification accuracy.

[0228] Let TP be the number of samples correctly predicted as the positive class, TN be the number of samples correctly predicted as the negative class, FP be the number of samples wrongly predicted as the positive class, and FN be the number of samples wrongly predicted as the negative class;

[0229] Evaluate the model using accuracy; accuracy is the proportion of samples with correct classification among the total number of samples; a total of 10 subjects participated in this experiment, and each subject conducted three experiments. In one experiment, 375 single attempts of motor imagery tasks were performed. The formula for calculating the accuracy of the i-th sample is shown in Equation (30):

[0230]

[0231] Then the average accuracy of the 10 subjects in 3 experiments is shown in Equation (31)

[0232]

[0233] The standard deviation of this experiment is shown in Equation (32)

[0234]

[0235] Obtain the evaluation results according to the above calculations.

[0236] This method is applicable to multi-person scenarios. However, limited by the number of devices and personnel, the final results will be limited to calculating the causal values and accuracy rates of the fusion of two subjects. The accuracy rates of Subject 1 and Subject 2 are only for comparison with this method.

[0237] Table 1

[0238] Accuracy rate (%) Standard deviation (%) Subject 1 62.47 07.48 Subject 2 56.46 06.95 This method 75.37 05.16

[0239] Based on the above results, the accuracy rate of this method has increased by 15% compared to the single-brain average, and the standard deviation has decreased by approximately 2%. This indicates that when processing EEG signals, this method can more effectively capture relevant features by utilizing the correlations among different individuals in each channel, reduce the differences among different individuals, and thus improve the classification accuracy, showing obvious advantages in group motor imagery classification and recognition.

Claims

1. A group motor imagery classification and recognition method based on EEG multi-view decoding, characterized in that It includes the following steps: Step 1: Acquisition of group motor imagery EEG data; Collect group EEG signal data of at least 2 individuals under motor imagery tasks, where the motor imagery tasks include left hand motor imagery, right hand motor imagery, and idle state tasks; Step 2: Preprocess the EEG signals collected in Step 1; Step 3: Extraction of single-person motor imagery EEG spatio-temporal features; For the single-person EEG signals preprocessed in Step 2, extract the feature vectors with the maximum discrimination through the common spatial pattern (CSP) as the single-person motor imagery EEG spatio-temporal features; Step 4: Extraction of group cross-brain coupling features; 4-1 Cross-brain channel selection; For the group EEG signals preprocessed in step 2, the Granger causality value of the same channel between individuals is calculated; then the group EEG signals are sorted in descending order according to the Granger causality value, and the foreground with mutual causal relationship is screened out from the group EEG signals. Channel pair data; The channel pairs of data are combined into a tensor form; 4-2 Group coupling feature extraction; The learning mechanism of graph convolution is integrated into the previous Each channel of the data is regarded as a node in the graph in the graph convolution model, so the number of nodes should be n represents the number of individuals; The adjacency matrix of each node is composed of the phase synchronization values between nodes; The feature matrix of a node includes the degree of the node, power spectral density, and differential entropy features; By using the feature matrix information of neighbor nodes to update the feature matrix of the current node, finally obtain the cross-brain features with coupling information, which are the group cross-brain coupling features; Step 5: Multi-view representation and learning by combining single-person and inter-individual coupling information.

2. The method according to claim 1, characterized in that: Specifically, Step 3 is: Assume that the preprocessed EEG signal of any individual is represented by X∈R C×S , where C represents the number of channels and S represents the sampling points; X is divided into X1 and X2 according to the left-hand and right-hand motor imagery tasks, i=1,2 represents the left-hand and right-hand motor imagery tasks respectively; To calculate the covariance matrix of X, assume C < S, and X1 and X2 are respectively: Among them, S1 and S2 represent the sampling points of left-hand and right-hand motor imagination, S M represents the source signal shared by S1 and S2, C M Indicates that S M The corresponding co-spatial pattern; C i Indicates S i The corresponding co-spatial pattern; The normalized covariance matrices ∑1 and ∑2 of X1 and X2 of spatio-temporal signals: where trace() represents the sum of the elements on the diagonal of the matrix; Then solve ∑ for the mixed space: in represents the average value of ∑1; Perform eigen-decomposition on ∑: ∑=AμA T (4) where A is the eigenvector matrix, and μ is the diagonal matrix of the eigenvalues arranged in descending order; Whiten and transform A to obtain the whitening matrix Y: Apply Equation (5) to C1 and C2 to get: S i =Y∑ i Y T ,(i=1,2) (6) S1 and S2 have common eigenvectors, and there exist two diagonal matrices μ1, μ2 and the same eigenvector matrix H. Perform principal component decomposition on S1 and S2 to obtain: S i =Hμ i H T ,(i=1,2) (7) and μ1 + μ2 = I (8) where I represents the identity matrix; Use the matrix H to obtain the projection matrix D: D=H T Y (9) where the projection matrix D is the spatial filter corresponding to the matrix H; Filter X1 and X2 through the spatial filter D to obtain W1 and W2: IN i =D×X i ,(i=1,2) (10) The features f1 and f2 of X1 and X2 can be obtained as: where var represents the variance of the matrix.

3. The method according to claim 1, characterized in that: Specifically, Step 4-1 includes the following sub-steps: Calculate the cross-brain channel Granger causality for the inter-individual EEG data; Take the EEG time signals X of two individuals at random a (t) and X b (t), assuming X b (t) By X a (t) and its lag term X a (t-1),X a (t-2),…,X a (tp) linear combination, plus an error term ε(t); p represents the lag time; therefore, X b The expression of (t) is as follows: Then, calculate X b At a given X a The probability of occurrence It is estimated by the logistic regression model, that is, given X b and its lag term, for X a Estimation of the probability of binary classification; Finally, the Granger causality value: Sort in descending order according to the Granger causality value, and take the first θ pairs of channel pairs.

4. The method according to claim 1, characterized in that: Specifically, Step 4-2 is: Obtain the first θ pairs of channel pairs obtained in Step 4-1. Consider each channel as a node in the graph, so as to obtain the graph data of nθ nodes; among them, the graph data includes an adjacency matrix and a node feature matrix; the adjacency matrix is composed of the phase locking values between nodes; the node feature matrix includes the degree of the node, power spectral density, and differential entropy features.

5. The method according to claim 4, characterized in that: The phase locking value in Step 4-2 is a phase-based functional connectivity method, and its calculation method is as follows: Where PLV represents the phase lock value, sn represents the maximum time, Represents the channel pair X a , X b The phase angle at time t, i represents the imaginary unit.

6. The method according to claim 1, characterized in that: Specifically, Step 5 is: 5-1 Multi-view subspace self-representation Project the spatiotemporal EEG features of individual motion imagery and the cross-brain coupling features of a group into a low-dimensional subspace. Then, calculate the representation matrix of each on the low-dimensional subspace for the spatiotemporal EEG features of individual motion imagery and the cross-brain coupling features of a group. This representation matrix contains the linear combination coefficients of the features in the low-dimensional subspace. Each representation matrix represents a view subspace self-representation. 5-2 The view subspace self-representation of all features is regarded as a tensor, and a low-rank constraint is imposed on the tensor; then, the affinity matrix is ​​calculated based on the tensor with a low-rank constraint imposed on it; the eigenvalues ​​of the Laplace matrix L are obtained based on the affinity matrix, the eigenvalues ​​are clustered, and finally the classification results of group motion imagination are obtained.

7. The method according to claim 6, characterized in that: Step 5 is as follows: 5-1, multi-view subspace self-representation; The single-person motor imagery EEG spatiotemporal features and group cross-brain coupling features generated in steps 3 and 4 are used as different views and input into the multi-view learning model. The method for processing multi-view data is as follows: s.t.X (v) =X (v) Z (v) +E (v) ,v=1,2,…,V, (16) Among them, X (v) The data matrix of the vth view corresponds to the spatiotemporal characteristics of single-person motor imagery EEG or the cross-brain coupling characteristics of a group, Z (v) and E (v) Respectively represent the subspace representation matrix and reconstruction error matrix of the v-th view; λ v is a hyperparameter that controls the strength of the loss penalty for the vth view; V is the number of views; R represents the regularization term; 5-2, low-rank tensor constraints; Tensors are used to constrain the correlation between different views; the tensor nuclear norm is defined as: Among them, ξ m is to satisfy i >0 and The constant of is an M-order tensor, Z (m) is done by passing the tensor The matrix obtained by expanding along the mth mode is defined as ||·|| * represents the nuclear norm, which is used to force the tensor under the low-rank constraint. In essence, the nuclear norm of a tensor is the convex combination of the nuclear norms of all matrices expanded along each mode. Under the low-rank tensor constraint, the objective function (16) of substep 5-1 is formulated as: where Ψ(·) is expressed by different (v) Merge into a 3rd-order tensor of dimension N2×N2×V to construct the tensor N2 represents Z (v) Dimension; E = [E (1) ,E (2) ,…,E (v) ] is formed by vertically concatenating the error columns corresponding to each view; ||·|| 2,1 yes norm, which encourages the columns of E to be zero; λ represents a hyperparameter; Substitute formula (17) into formula (18), so formula (18) is converted to: in encodes the strength of the low-rank tensor constraint, and the constraints are the same as in (18); Z (v) is the subspace representation matrix corresponding to the vth view, and Z (m) yes The mth mode expansion matrix of ; 5-3, affinity matrix and spectral clustering; According to formula (19), the affinity matrix B is obtained: The degree matrix Q is calculated using formula (21): Among them B ij represents the i-th row and j-th column element of affinity matrix B, and the degree matrix Q is q i The n3*n3 diagonal matrix composed of; Calculate the Laplacian matrix L: L=QB (22) Calculate the matrix eigenvalues ​​of L, sort the matrix eigenvalues ​​from small to large, take the first k matrix eigenvalues, and calculate the eigenvectors u1, u2, …, u of the first k matrix eigenvalues k ; Form a matrix of k column vectors: U={u1,u2,…,u k },U∈R n3*k (23) Let ve i ∈R k is the vector of the i-th row of U, where i = 1, 2, …, n3; Use k-means algorithm to transform VE={ve1,ve2,…,ve n3 }Clustering into clusters C1, C2, …, C k .

8. A group movement imagery classification and recognition system implementing the method described in any one of claims 1 to 7, characterized in that include: Data acquisition and preprocessing module, used to acquire group motor imagery EEG data and preprocess the collected EEG signals; Single-person feature extraction module, used to extract the spatiotemporal features of single-person motor imagery EEG; Group feature extraction module, used to extract group cross-brain coupling features; The classification and recognition module is used to input the spatiotemporal characteristics of single-person motor imagery EEG and the cross-brain coupling characteristics of a group as different views into the multi-view learning model for multi-view representation and learning, and finally obtain the classification results.

9. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 7.

10. A computing device, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 7 is implemented.

Images