Multi-dimensional feature fusion electroencephalogram signal decoding method and system, medium and equipment

Abstract

The invention discloses a multi-dimensional feature fusion electroencephalogram signal decoding method, system, medium and equipment, and the method comprises the steps: collecting an electroencephalogram signal, preprocessing the collected signal, further carrying out the multi-band filtering of the preprocessed electroencephalogram signal, constructing a deep feature extraction model, and carrying out the decoding of the deep feature extraction model. Training is carried out until the performance of the model stops improving, multi-dimensional fusion features are obtained, then a machine learning method and a Wrapper type recursive feature elimination feature selection method are combined, redundant features are screened out, and decoding of electroencephalogram signals is achieved. According to the method, the multi-dimensional fusion features of the signals in the Euclidean space and the Riemannian manifold are gradually extracted, the signals are deeply analyzed from the two dimensions of the space-time frequency domain and the geometric structure, the information extraction completeness can be effectively improved, and then the electroencephalogram signal decoding precision is improved. In addition, a feature selection strategy is further designed, redundant features can be further screened out, the calculation complexity is reduced, and the actual deployment requirement is met.

Description

Technical Field

[0001] The present invention relates to the technical field of electroencephalogram (EEG) signal decoding, and particularly to a multi-dimensional feature fusion EEG signal decoding method, system, medium and device. Background Art

[0002] As a non-invasive neurophysiological signal, EEG signals have shown important application values in the fields of brain-computer interfaces, neurological disease diagnosis, intelligent control, etc., due to their advantages of low cost, high temporal resolution, portability, and non-invasiveness. However, affected by their weak physiological characteristics and complex acquisition environments, the original EEG signals have a significantly low signal-to-noise ratio, which poses a great challenge to establishing accurate decoding models.

[0003] To improve the decoding accuracy of EEG signals, existing methods mainly visualize EEG signals and combine technologies such as deep learning to extract spatio-temporal-frequency features of EEG signals to improve decoding accuracy. Although such methods can effectively capture the local time-frequency characteristics of signals, they are limited by the linear hypothesis in Euclidean space and are difficult to accurately represent the geometric attributes of EEG signals. Another study constructs a feature extraction model based on Riemannian geometry theory and effectively represents the geometric structure of signals through techniques such as geodesic distance measurement between manifolds. However, although such methods can represent the overall state of brain network connections, they may also lead to the loss of local details. In addition, the manifold space features and the time-frequency features of the original signal space belong to different mathematical spaces, and traditional feature splicing or weighting strategies have redundant information, resulting in low efficiency of collaborative utilization of multi-dimensional information. Summary of the Invention

[0004] In view of this, this paper provides a multi-dimensional feature fusion EEG signal decoding method, system, medium and device. The aims are as follows:

[0005] 1) Decompose EEG signals into multiple frequency bands according to neurophysiological mechanisms, and combine with a convolutional neural network to perform spatio-temporal filtering on the signals in a data-driven manner to extract spatio-temporal-frequency domain features of the signals; map the filtered signals to the Riemannian manifold space in the form of covariance matrices, and use the affine-invariant geometric metric in the space of symmetric positive definite matrices to analyze the geometric characteristics of the signals in the non-Euclidean space; transform the manifold features into tangent vectors in the Euclidean space through tangent space projection, and construct geometrically interpretable feature vectors for feature selection.

[0006] 2) Adopt a Wrapper-based feature selection strategy, and through the collaborative optimization of the forward recursive feature elimination algorithm and the support vector machine, establish a feature weight evaluation model, iteratively screen out a feature subset with significant discrimination, realize the compression of feature dimensions, control the number of model parameters, thereby reducing the computational complexity to meet the real-time requirements of embedded system deployment.

[0007] The above technical object of the present invention is achieved through the following technical solutions: A multi-dimensional feature fusion electroencephalogram signal decoding method, comprising the following steps:

[0008] S1: Collect electroencephalogram signals, and preprocess the collected signals, including removing power supply noise, and removing eye movement and electromyogram artifacts;

[0009] S2: Perform multi-band filtering on the preprocessed electroencephalogram signals based on neurophysiological principles;

[0010] S3: Construct a deep feature extraction model, perform multiple iterative trainings on the model until the model loss no longer decreases, and output multi-dimensional fusion features. The deep feature extraction model takes the multi-band filtered signals as input, and iteratively updates with the goal of reducing the loss between the predicted result of the signal category by the model and the true label of the signal, and finally outputs the multi-dimensional fusion features extracted by the model.

[0011] S4: Combine machine learning methods with a feature selection method of Wrapper-style recursive feature elimination to screen out redundant features and achieve the decoding of electroencephalogram signals.

[0012] Further, in step S1, collecting electroencephalogram signals includes:

[0013] Collecting multi-channel electroencephalogram signals using an electrode cap and performing discretization processing in the time dimension to obtain the original electroencephalogram signal x(t):

[0014] x(t) = {x 1 (t), x 2 (t), x 3 (t), …, x C (t)}

[0015] where C represents the number of channels of the electroencephalogram signal, and x c (t) represents the signal collected at channel c of the electrode cap, that is:

[0016] x c (t) = {x c (t 1 ), x c (t 2 ), x c (t 3 ), …, x c (t E )}

[0017] where t represents the discretized time series, including E time sampling points obtained according to the sampling rate f sa and x c (t e ) represents the signal at channel c at time te The collected EEG signal values;

[0018] Preprocess the signal, including removing power noise, eye movement and EMG artifacts. The main process is as follows:

[0019] S11: Filter the signal to remove high-frequency noise, low-frequency drift and power noise;

[0020] S12: Use independent component analysis to separate the signal into multiple independent source signals, and determine the independent source corresponding to the EEG signal according to the waveform, spectrum and spatial distribution characteristics of eye movement and EMG artifacts;

[0021] S13: Remove the independent source signal corresponding to the artifact, and reconstruct the EEG signal to obtain the EEG signal x pp (t) after removing artifacts and power noise;

[0022] Furthermore, in step S2, perform multi-band filtering on the preprocessed EEG signal based on neurophysiological principles, including:

[0023] Determine the frequency bands related to the decoding task based on neurophysiological principles: {[f s1 , f e1 , [f s2 , f e2 , [f s3 , f e3 , …, [f sG , f eG}, where G represents the total number of frequency bands;

[0024] Design a band-pass filter f sg , f eg (t) based on each frequency band [f g (t) to construct a bank of band-pass filters f(t) for multi-band decomposition of the signal x pp (t):

[0025] x fg (t) = f g (t) ⊙ x pp (t)

[0026] where ⊙ represents the convolution operation, and x fg (t) is the EEG signal obtained after filtering by the band-pass filter f g (t);

[0027] Obtain the multi-band EEG signal x f (t):

[0028] x f (t) = {x f1 (t), x f2(t), x f3 (t), …, x fG (t)}

[0029] Furthermore, in step S3, a deep feature extraction model is constructed, which is set as follows:

[0030] Filtering module - used to perform spatio-temporal filtering on multi-band EEG signals, filter out noise, and extract spatio-temporal features;

[0031] Riemannian module - maps the filtered signal to the Riemannian manifold and performs Riemannian geometric operations on the mapped matrix to extract more discriminative multi-dimensional features as output;

[0032] Furthermore, the filtering module includes several spatio-temporal filtering layers, and each layer includes several two-dimensional convolutional kernels K of size h*w to perform spatio-temporal convolution operations on each frequency band signal to obtain a multi-band spatio-temporal filtered signal x ff (t):

[0033] x ff (t) = {x ff1 (t), x ff2 (t), x ff3 (t), …, x ffG (t)}

[0034] where x ffg (t) is the signal obtained by performing spatio-temporal convolution operation on the g-th frequency band signal x f (t) of x fg (t):

[0035]

[0036] where x fg (i + m, j + n) represents the signal value of the signal x fg at the (i + m)-th channel and the (j + n)-th time point, K(m, n) is the value of the convolutional kernel at the position (m, n), and x ffg (i, j) represents the signal value of the signal x ffg at the i-th channel and the j-th time point;

[0037] Furthermore, the Riemannian module includes a Riemannian manifold mapping layer, a bilinear mapping layer, a geometric center extraction layer, and a tangent space mapping layer, and proceeds through the following steps:

[0038] S31: Input the output signal x ff (t) of the filtering module into the Riemannian manifold mapping layer. First, based on the principles of neurophysiology, in the time dimension, x ff(t) is cut into L non-overlapping blocks, and the covariance matrix X of each block signal is calculated. X is a symmetric positive definite square matrix of size [C, C]. At this time, the signal has been projected onto the Riemannian manifold;

[0039] S32: Input all covariance matrices into the bilinear mapping layer, and perform A bilinear mappings and activation operations on each matrix respectively. Among them, the a-th operation is as follows:

[0040] First, perform a bilinear mapping on the matrix:

[0041] X a = M·X a-1 ·M

[0042] where M is a learnable transformation matrix of size [C a , C a-1 , and C 0 = C, X 0 = X;

[0043] Subsequently, activate X a :

[0044] X r = Bmax(ηI, V)B

[0045] where B and V are the eigenvector matrix and eigenvalue matrix of X a respectively, max() represents the maximum value operation, η is a threshold, and I is the identity diagonal matrix;

[0046] After the conversion of the bilinear mapping layer, the sizes of all covariance matrices are converted to [C A , C A . The total size of the features output by the bilinear mapping layer is [G, L, C A ,, C A , where G is the total number of frequency bands, and if C A < C, the feature dimension reduction operation can be achieved;

[0047] S33: Input the matrices {X r} L output by the L bilinear mapping layers of each frequency band into the geometric center extraction layer respectively, and extract the central matrix X c of each frequency band based on the Riemannian geometry method;

[0048] S34: Input the central matrix X c of each frequency band extracted into the tangent space mapping layer, and perform a tangent space mapping on X c :

[0049]

[0050] Among them, upper represents the operation of obtaining the upper triangular elements of a matrix, logm represents the matrix logarithm operation, D is a reference matrix, which can be set as the identity diagonal matrix I, Q represents the number of eigenvalues, is the q-th eigenvalue of x rm .

[0051] Furthermore, in step S4, a feature selection method combining a machine learning method with a wrapper-style recursive feature elimination is used to screen out redundant features, including:

[0052] S41: Initialize the support vector machine classification model;

[0053] S42: Train the support vector machine model and obtain the importance coefficients [λ 1 , λ 2 , λ 3 , …, λ Q that reflect the importance of features to the prediction result. Features with larger coefficients indicate greater contributions to the decision boundary.

[0054] S43: According to the feature importance scores provided by the model, delete the feature with the smallest coefficient and retrain the model using the remaining features. Then, re-evaluate the importance of the remaining features [λ 1 , λ 2 , λ 3 , …, λ P .

[0055] S44: Recursively delete features and retrain the model, each time deleting the feature with the smallest importance coefficient until the remaining features reach a predetermined number or the performance of the model no longer improves. Construct the final feature selection strategy and set the finally obtained support vector machine model as the classification model.

[0056] Perform feature selection on the output x rm of the deep model in step S3 according to the final feature selection strategy to obtain the optimal feature subset x s , and input x s into the support vector machine classification model obtained in step S44 to get the predicted signal category, realizing the decoding of electroencephalogram signals.

[0057] An electroencephalogram signal decoding system with multi-dimensional feature fusion includes:

[0058] Data acquisition module: Use an electrode cap to collect multi-channel electroencephalogram signals within a predetermined time at a certain sampling rate, denoted as the original electroencephalogram signals;

[0059] Data preprocessing module: Remove noise and artifacts from the original electroencephalogram signals and perform multi-band filtering on the electroencephalogram signals to obtain the corresponding multi-band electroencephalogram signals;

[0060] Feature extraction module: Use a deep feature extraction model to extract multi-dimensional features of multi-band EEG signals in Euclidean space and Riemannian manifold;

[0061] Feature selection module: Screen the extracted multi-dimensional features according to the feature selection strategy, eliminate redundant features, and obtain features with significant classification contributions;

[0062] Classification module: Use a support vector machine model to classify the signals and obtain the decoding results of the EEG signal categories.

[0063] Another aspect of the present invention provides a computer-readable storage medium, on which a computer program or instruction is stored. When the above computer program or instruction is executed by a processor, the steps of the above method are implemented.

[0064] Another aspect of the present invention provides an electronic device, including: one or more processors; a memory for storing one or more computer programs, wherein the above one or more processors execute the above one or more computer programs to implement the steps of the above method.

[0065] Compared with the prior art, the present invention has the following advantages:

[0066] 1. The present invention not only uses a convolutional neural network to perform spatio-temporal filtering on signals in Euclidean space, but also projects the filtered signals of different time windows onto the Riemannian manifold space based on neurophysiological principles. Using Riemannian geometry theory and combining deep learning methods, multi-dimensional features of the signals in Euclidean space and Riemannian manifold are further extracted, realizing deeper feature extraction and improving the completeness of information extraction;

[0067] 2. By using a bilinear mapping layer, more discriminative features can be extracted on the Riemannian manifold and feature dimensionality reduction can be achieved. At the same time, while reducing the feature dimension, the relationship of the Riemannian distance in the original manifold space can be maintained; the geometric center extraction layer can fuse the features of different time windows, improving the method's perception ability in the time dimension while further reducing the feature dimension, effectively avoiding the problems of overfitting and large computational complexity;

[0068] 3. Use the feature selection method of Wrapper-style recursive feature elimination to screen the multi-dimensional features, eliminate redundant features, compress the feature dimension, effectively reduce the computational complexity, and improve the decoding efficiency. Brief Description of the Drawings

[0069] Figure 1 It is a schematic flowchart of a method for decoding EEG signals with multi-dimensional feature fusion provided by the present invention.

[0070] Figure 2Schematic flow diagram of an electroencephalogram signal decoding system with multi-dimensional feature fusion provided by the present invention.

[0071] Figure 3 Schematic structural diagram of an electronic device provided by the present invention. Detailed implementation manners

[0072] The present invention will be further described in detail below in conjunction with embodiments and the accompanying drawings, but the implementation manners of the present invention are not limited thereto.

[0073] Embodiment 1:

[0074] As Figure 1 shown, a method for decoding electroencephalogram signals with multi-dimensional feature fusion is characterized in that the specific steps are as follows:

[0075] S1: Collect electroencephalogram signals and preprocess the collected electroencephalogram signals, including removing power noise, eye movement, and electromyogram artifacts.

[0076] S2: Perform multi-band filtering on the preprocessed electroencephalogram signals based on neurophysiological principles.

[0077] S3: Construct a deep feature extraction model, perform multiple iterative trainings on the model until the model loss no longer decreases, and output multi-dimensional fusion features; the deep feature extraction model takes the multi-band filtered signal as the input and iteratively updates with the goal of reducing the loss between the predicted result of the signal category by the model and the true label of the signal, and finally outputs the multi-dimensional fusion features extracted by the model.

[0078] S4: Combine machine learning methods with a feature selection method of Wrapper-style recursive feature elimination to screen out redundant features to achieve the decoding of electroencephalogram signals.

[0079] By gradually extracting the fusion features of the signal in the Euclidean space and the Riemannian manifold, and deeply analyzing the signal from two dimensions of time-frequency domain and geometric structure respectively, the present invention can effectively improve the completeness of information extraction, and further improve the accuracy of signal decoding. At the same time, the feature selection strategy can further screen out redundant features and reduce the computational complexity to meet the requirements of actual deployment.

[0080] Furthermore, the above-mentioned collection of electroencephalogram signals specifically includes:

[0081] Collect multi-channel electroencephalogram signals using an electrode cap and perform discretization processing in the time dimension to obtain the original electroencephalogram signal x(t):

[0082] x(t) = {x 1 (t), x 2 (t), x 3 (t), …, x C (t)}

[0083] Among them, C represents the number of channels of the electroencephalogram signal, and x c (t) represents the signal collected at channel c of the electrode cap, that is:

[0084] x c (t) = {x c (t 1 ), x c (t 2 ), x c (t 3 ), …, x c (t E )}

[0085] Among them, t represents the discretized time series, including E time sampling points obtained according to the sampling rate f sa , and x c (t e ) represents the electroencephalogram signal value collected at channel c at time t e ;

[0086] The above preprocessing of the electroencephalogram signal, including the removal of power supply noise and the removal of eye movement and electromyogram artifacts, mainly has the following processes:

[0087] S11: Use a fourth-order Butterworth band-pass filter (0.5 - 100 Hz) to eliminate the low-frequency drift (<0.5 Hz) and high-frequency noise (>100 Hz) of the signal, and superimpose a 50 Hz notch filter to eliminate the power frequency noise of the signal;

[0088] S12:: Use the FastICA algorithm to decompose the signal into N independent components (N is equal to the number of electrode channels), and identify the artifact components in combination with the following criteria: a) Eye movement artifact: The proportion of the component power spectral density in the 1 - 4 Hz frequency band is ≥65%, and the projection weight on the frontal electrodes (FP1 / FP2) > 0.7; b) Electromyogram artifact: The energy of the component in the 20 - 100 Hz frequency band suddenly increases by ≥3 times the standard deviation compared with the baseline level, and shows a high correlation (Pearson coefficient > 0.8) in the temporal region electrodes (T7 / T8);

[0089] S13: Remove the independent source signal corresponding to the artifact, and reconstruct the electroencephalogram signal to obtain the electroencephalogram signal x pp (t) without artifacts and power supply noise.

[0090] The present invention improves the signal-to-noise ratio of the EEG signal through signal preprocessing, enables subsequent feature extraction operations to more accurately capture effective information, reduces the interference of error information on the decoding result, and thus improves the robustness and stability of the overall decoding method.

[0091] Further, in the above step S2, multi-band filtering is performed on the preprocessed EEG signals based on neurophysiological principles, including:

[0092] Determine the frequency bands related to the decoding task based on neurophysiological principles: {delta wave: [0.5, 4], theta wave: [4, 8], alpha wave: [8, 13], beta wave: [13, 30], gamma wave: [30, 100]}

[0093] Design a band-pass filter f g (t) to construct a bank of band-pass filters f(t) for multi-band decomposition of the signal x pp (t), such as for the delta band:

[0094] x fg (t) = f g (t) ⊙ x pp (t)

[0095] where ⊙ represents the convolution operation, and x fg (t) is the EEG signal obtained after filtering by the band-pass filter f g (t);

[0096] Obtain the multi-band EEG signal x f (t):

[0097] x f (t) = {x f1 (t), x f2 (t), x f3 (t), …, x fG (t)}

[0098] Therefore, the size of x f (t) is: [5 (number of frequency bands), C (number of channels), E (number of time sampling points)].

[0099] In the present invention, spatio-temporal filtering of signals is performed using a convolutional neural network in Euclidean space, and based on neurophysiological principles, the filtered signals in different time windows are projected onto a Riemannian manifold space. Using Riemannian geometry theory and combining deep learning methods, multi-dimensional features of the signals in Euclidean space and Riemannian manifold are further extracted, realizing deeper feature extraction and improving the completeness of information extraction.

[0100] Further, in step S3, constructing a deep feature extraction model includes:

[0101] Filtering module - used to perform spatio-temporal filtering on multi-band EEG signals, filter out noise, and extract spatio-temporal features;

[0102] Riemann module - maps the filtered signal to a Riemannian manifold, performs Riemannian geometric operations on the mapped matrix, and extracts more discriminative multi-dimensional features as output;

[0103] Furthermore, the filtering module includes several layers of spatio-temporal filtering layers. Each layer contains C two-dimensional convolutional kernels K of size C*13 to perform spatio-temporal convolution operations on signals in each frequency band, obtaining multi-band spatio-temporal filtered signals x ff (t):

[0104] x ff (t) = {x ff1 (t), x ff2 (t), x ff3 (t), …, x ffG (t)}

[0105] where x ffg (t) is the signal of the g-th frequency band of x f (t), and the signal obtained after spatio-temporal convolution operation on x fg (t):

[0106]

[0107] where x fg (i + m, j + n) represents the signal value of the signal x fg at the (i + m)-th channel and the (j + n)-th time point, K(m, n) is the value of the convolutional kernel at the position (m, n), and x ffg (i, j) represents the signal value of x ff at the g-th frequency band, the i-th channel, and the j-th time point;

[0108] The Riemann module includes a Riemannian manifold mapping layer, a bilinear mapping layer, a geometric center extraction layer, and a tangent space mapping layer, and proceeds through the following steps:

[0109] S31: Input the output signal x ff (t) of the filtering module into the Riemannian manifold mapping layer. First, based on the principle of neurophysiology, cut x ff (t) non-overlappingly into L blocks in the time dimension, and calculate the covariance matrix X of each block. X is a symmetric positive definite square matrix of size [C, C]. At this time, the signal has been projected onto the Riemannian manifold, and the feature size extracted from the signals of each frequency band is [L, C, C]. Therefore, the total size of the features output by the filtering module is [5, L, C, C], where 5 is the total number of frequency bands;

[0110] S32: Input all covariance matrices X into the bilinear mapping layer, and perform A bilinear mapping and activation operations on each matrix respectively. Among them, the a-th operation is as follows:

[0111] First, perform a bilinear mapping on the matrix:

[0112] X a = M·X a-1 ·M

[0113] where M is a learnable transformation matrix of size [C a , C a-1 , and C 0 = C, X 0 = X;

[0114] Subsequently, activate X a :

[0115] X r = Bmax(ηI, V)B

[0116] where B and V are the eigenvector matrix and eigenvalue matrix of X a respectively, max() represents the maximum operation, η is a threshold, and I is the identity diagonal matrix.

[0117] After the conversion by the bilinear mapping layer, the size of all covariance matrices is converted to [C A , C A , and the total size of the features output by the bilinear mapping layer is [5, L, C A ,, C A , where 5 is the total number of frequency bands, and if C A < C, the feature dimension reduction operation can be achieved;

[0118] S33: Input the matrices {X r} L output by the L bilinear mapping layers of each frequency band into the geometric center extraction layer respectively, and extract the center matrix X c of each frequency band based on Riemannian geometry;

[0119] X c has a size of [C A ,, C A , so the total size of the features output by the geometric center extraction layer becomes [5, C A ,, C A , where 5 is the total number of frequency bands;

[0120] S34: Input the extracted center X c of each frequency band matrix into the tangent space mapping layer, and perform tangent space mapping on X c :

[0121]

[0122] Among them, upper represents the operation of obtaining the upper triangular elements of the matrix, logm represents the matrix logarithm operation, D is the reference matrix, which can be set as the identity diagonal matrix I, Q represents the number of eigenvalues, is the q-th eigenvalue of x rm ; and Q = C A ·(C A +1) / 2, is the Q-th eigenvalue of x rm .

[0123] Subsequently, the model is iteratively trained multiple times until the model loss no longer decreases, including:

[0124] (a) Input x rm into a fully connected layer and obtain the prediction result of the model for the signal category in combination with the softmax function;

[0125] (b) Use the cross-entropy loss function to calculate the loss between the model prediction result and the true label of the signal, and use the gradient descent algorithm to backpropagate the loss to update the model parameters;

[0126] (c) Repeat the above process to iteratively update the deep feature extraction model multiple times until the model loss no longer decreases.

[0127] Output the feature x rm output after the signal is filtered by the model filtering module and the features are extracted by the Riemann module.

[0128] Existing research constructs a feature extraction model based on the theory of Riemannian geometry and effectively represents the geometric structure of the signal through techniques such as geodesic distance measurement between manifolds. However, although such methods can represent the overall state of brain network connections, they may also lead to the loss of local details. The proposed method for decoding EEG signals with multi-dimensional feature fusion in the present invention deeply analyzes the signal from two dimensions of spatio-temporal frequency domain and geometric structure by gradually extracting the fusion features of the signal in the Euclidean space and Riemannian manifold, which can effectively improve the completeness of information extraction and thus improve the accuracy of signal decoding.

[0129] In step S4, a feature selection method combining a machine learning method and a Wrapper-style recursive feature elimination method is used to screen out redundant features, including:

[0130] S41: Initialize the support vector machine classification model;

[0131] S42: Train the support vector machine model and obtain the importance coefficients [λ 1 , λ 2 , λ 3 , …, λ Q that reflect the importance of features to the prediction result. Features with larger coefficients indicate greater contributions to the decision boundary.

[0132] S43: According to the feature importance scores provided by the model, delete the feature with the smallest coefficient and retrain the model using the remaining features. Then, re-evaluate the importance of the remaining features [λ 1 , λ 2 , λ 3 , …, λ P ;

[0133] S44: Recursively delete features and retrain the model. Each time, delete the feature with the smallest importance coefficient until the remaining features reach a predetermined number or the performance of the model no longer improves. Construct the final feature selection strategy and set the finally obtained support vector machine model as the classification model;

[0134] Perform feature selection on the output x of the deep model in step S3 according to the final feature selection strategy rm to obtain the optimal feature subset x s , and input x s into the support vector machine classification model obtained in step S44 to obtain the predicted signal category, realizing the decoding of electroencephalogram signals.

[0135] According to the embodiments of the present invention, the electroencephalogram signal decoding method with multi-dimensional feature fusion adopts a Wrapper-style feature selection strategy. Through the collaborative optimization of the forward recursive feature elimination algorithm and the support vector machine, a feature weight evaluation model is established, and a feature subset with significant discrimination is iteratively selected to realize the compression of the feature dimension and control the number of model parameters, thereby reducing the computational complexity to meet the real-time requirements of embedded system deployment.

[0136] Embodiment 2:

[0137] As Figure 2 shown, this embodiment provides an electroencephalogram signal decoding system for realizing multi-dimensional feature fusion, including:

[0138] Data acquisition module: Arranged according to the 10-20 international standard system using a high-density electrode cap (such as a 64-channel Ag / AgCl electrode), collect the original electroencephalogram signals at a sampling rate of ≥1000 Hz. The module is built-in with a reference electrode drive circuit to eliminate common-mode interference, synchronously collect environmental noise signals for subsequent filtering, and support online impedance detection and dynamic shielding of abnormal channels to ensure that the signal acquisition quality meets the standards;

[0139] Data preprocessing module: A fourth-order Butterworth bandpass filter (0.5 - 100 Hz) is used to eliminate the low-frequency drift (<0.5 Hz) and high-frequency noise (>100 Hz) of the signal, and a 50 Hz notch filter is superimposed to eliminate the power frequency noise of the signal; the FastICA algorithm is used to remove signal artifacts and the inverse ICA method is used to reconstruct the signal; through a zero-phase IIR filter bank (δ: 0.5 - 4 Hz, θ: 4 - 8 Hz, α: 8 - 13 Hz, β: 13 - 30 Hz, γ: 30 - 100 Hz), multi-band segmentation conforming to the characteristics of neural rhythms is achieved to obtain the corresponding multi-band EEG signals;

[0140] Feature extraction module: Use the trained deep feature extraction model to extract multi-dimensional features of the multi-band EEG signals in the Euclidean space and Riemannian manifold;

[0141] Feature selection module: Screen the extracted multi-dimensional features according to the feature selection strategy, eliminate redundant features, and select features with significant classification contributions;

[0142] Classification module: Use the trained support vector machine model to classify the signal and obtain the decoding result of the EEG signal category.

[0143] In this embodiment, the feature selection module includes a filtering module and a Riemannian module. The filtering module contains several layers of spatio-temporal filtering layers for performing spatio-temporal convolution operations on the multi-band EEG signals. The Riemannian module contains a manifold mapping layer, a bilinear mapping layer, a geometric center extraction layer, and a tangent space mapping layer. This module projects the obtained multi-band spatio-temporal filtered signals onto the Riemannian manifold, combines Riemannian geometric methods and convolutional neural network methods, and extracts the geometric structure features of the EEG signals in a data-driven manner. Finally, a multi-dimensional fusion feature vector x containing spatio-temporal frequency characteristics in the Euclidean space and geometric structure characteristics on the Riemannian manifold is output rm

[0144] In the EEG signal decoding system that realizes multi-dimensional feature fusion in this embodiment, by using the bilinear mapping layer, more discriminative features can be extracted on the Riemannian manifold, and feature dimensionality reduction can be achieved. At the same time, while reducing the feature dimension, the Riemannian distance relationship of the original manifold space can be maintained; by using the geometric center extraction layer, features of different time windows can be fused, which can improve the time dimension perception ability of the method and further reduce the feature dimension, effectively avoiding the problems of overfitting and large computational complexity.

[0145] Embodiment 3:

[0146] As Figure 3 shown, this embodiment provides an electronic device for implementing a method for decoding EEG signals with multi-dimensional feature fusion.

[0147] The electronic device 1 may include a communication interface 10, a memory 11, a data preprocessor 12, and a feature extraction processor 13.

[0148] The communication interface 10 is used for data transmission with external devices (such as sensors, computers, or other electronic devices). The communication interface can support multiple protocols, such as USB, Wi-Fi, Bluetooth, Ethernet, etc., and can flexibly access different types of devices. The communication interface can receive the collected raw signal data and transmit it to the memory.

[0149] The memory 11 includes various storage media such as RAM, flash memory, and hard disks, and appropriate storage methods can be selected according to application requirements. The memory is not only used to store the running programs required by the processor, but also stores raw data, preprocessed data, feature data extracted from the feature extraction processor, and model parameters. This memory can provide high-speed data reading and writing, and support the storage and management of large-capacity data.

[0150] The data preprocessor 12 performs preprocessing operations on the raw data received from the communication interface, including but not limited to removing high-frequency noise, low-frequency drift, and power supply noise, removing eye movement and electromyogram artifacts, and multi-band filtering of signals. The preprocessor can appropriately convert or adjust the signals according to different data types and application requirements to improve the accuracy of subsequent processing. The data preprocessor can also adjust its processing strategy according to real-time feedback to adapt to different signal characteristics.

[0151] The feature extraction processor 13 extracts the multi-space fusion features of the data according to the computer programs, model parameters, and preprocessed data stored in the memory, and iteratively updates the model parameters through the loss function to ensure that the extracted features have high accuracy and effectiveness in the subsequent decoding process.

[0152] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories and optical memories, etc.) containing computer-usable program code.

[0153] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate a means for implementing the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or a means for implementing the functions specified in multiple blocks.

[0154] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including an instruction means, and the instruction means implements the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or a means for implementing the functions specified in multiple blocks.

[0155] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or a means for implementing the functions specified in multiple blocks.

[0156] Those of ordinary skill in the art can understand that all or part of the processes of implementing the above-described embodiment methods can be completed by instructing relevant hardware through a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-described method embodiments. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory, or a random access memory, etc.

[0157] The above is only the preferred embodiment of the present invention and is not used to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for decoding EEG signals by multi-dimensional feature fusion, characterized in that: The specific steps are as follows: S1: Collect EEG signals and pre-process the collected EEG signals: S2: Multi-band filtering of preprocessed EEG signals based on neurophysiological principles; S3: construct a deep feature extraction model, perform multiple iterations of model training until the model loss no longer decreases, and output multi-dimensional fusion features; the deep feature extraction model takes a multi-band filtered signal as input, performs iterative updates with the goal of reducing the loss between the model's prediction result of the signal category and the signal's true label, and finally outputs the multi-dimensional fusion features extracted by the model; S4: Combine machine learning methods with the feature selection method of wrapper-style recursive feature elimination to screen out redundant features to achieve EEG signal decoding.

2. The method for decoding EEG signals by multi-dimensional feature fusion according to claim 1, characterized in that: The step S1 of collecting EEG signals includes: The electrode cap is used to collect multi-channel EEG signals and discretize them in the time dimension to obtain the original EEG signal x(t): x(t)={x1(t),x2(t),x3(t),…,x C (t)} Among them, C represents the number of channels of EEG signal, x c (t) represents the signal collected at channel c of the electrode cap, that is: x c (t)={x c (t1),x c (t2),x c (t3),…,x c (t E )} Among them, t represents the discretized time series, including the sampling rate f sa The E time sampling points obtained, x c (t e ) represents the time t on channel c e The collected EEG signal value; The EEG signal preprocessing in step S1 includes the removal of power supply noise, eye movement and myoelectric artifacts. The main steps are: S11: Filter the signal to remove high-frequency noise, low-frequency drift and power supply noise; S12: Use independent component analysis to separate the signal into multiple independent source signals, and determine the independent source corresponding to the EEG signal based on the waveform, spectrum and spatial distribution characteristics of eye movement and electromyography artifacts; S13: Remove the independent source signal corresponding to the artifact and reconstruct the EEG signal to obtain the EEG signal x without artifacts and power supply noise pp (t).

3. The EEG signal decoding method of multi-dimensional feature fusion according to claim 1, characterized in that: In step S2, multi-band filtering is performed on the preprocessed EEG signal based on neurophysiological principles, including: Based on neurophysiological principles, the frequency bands related to the decoding task are determined: {[f s1 ,f e1 ],[f s2 ,f e2 ],[f s3 ,f e3 ],…,[f sG ,f eG ]}, where G represents the total number of frequency bands; Based on each frequency band [f sg ,f eg ] Design a bandpass filter f g (t) constructs a bandpass filter bank f(t) for the signal x pp (t) and perform multi-band decomposition: x fg (t)=f g (t)⊙x pp (t) Among them, ⊙ represents the convolution operation, x fg (t) is the value of the bandpass filter f g (t) EEG signal obtained after filtering; Obtain multi-band EEG signals x f (t): x f (t)={x f1 (t),x f2 (t),x f3 (t),…,x fG (t)} 4. The method for decoding EEG signals by multi-dimensional feature fusion according to claim 1, characterized in that: In step S3, a deep feature extraction model is constructed, including: The filtering module is used to perform spatiotemporal filtering on multi-band EEG signals, filter out noise, and extract spatiotemporal features; Riemann module, which maps the filtered signal to a Riemann manifold and performs Riemannian geometry operations on the mapped matrix to extract more discriminative multi-dimensional features as output; The filtering module includes several spatiotemporal filtering layers, each of which includes several two-dimensional convolution kernels K of size h*w to perform spatiotemporal convolution operations on the signals of each frequency band to obtain a multi-band spatiotemporal filtering signal x ff (t): x ff (t)={x ff1 (t),x ff2 (t),x ff3 (t),…,x ffG (t)} Among them, x ffg (t) is x f (t) g-th frequency band signal, x fg (t) Signal obtained after spatiotemporal convolution operation: Among them, x fg (i+m,j+n) represents the signal x fg The signal value at the i+mth channel and j+nth time point, K(m,n) is the value of the convolution kernel at position (m,n), x ffg (i,j) represents x ff Signal value in the g-th frequency band, i-th channel, and j-th time point.

5. The method for decoding EEG signals by multi-dimensional feature fusion according to claim 4, characterized in that: The Riemann module includes a Riemann manifold mapping layer, a bilinear mapping layer, a geometric center extraction layer, and a tangent space mapping layer, and is performed through the following steps: S31: Output signal x of the filter module ff (t) Input Riemann manifold mapping layer, first based on the principle of neurophysiology, x is transformed in the time dimension ff (t) Cut into L blocks without overlapping, and calculate the covariance matrix X of each block signal, where X is a symmetric positive definite matrix of size [C, C]. At this time, the signal has been projected onto the Riemann manifold; S32: All covariance matrices X are input into the bilinear mapping layer, and each matrix is ​​subjected to A bilinear mapping and activation operations, wherein the a-th operation is as follows: First, perform a bilinear mapping on the matrix: X a =M·X a-1 ·M Where M is the size of [C a ,C a-1 ] is a learnable transformation matrix, and C 0 =C,X 0 =X; Then X a To activate: X r =Bmax(ηI,V)B Among them, B and V are X a The eigenvector matrix and eigenvalue matrix of , max() represents the maximum value operation, η is a threshold, and I is a unit diagonal matrix; After the bilinear mapping layer, the size of all covariance matrices is converted to [C A ,C A ], the total size of the output feature of the bilinear mapping layer is [G,L,C A , ,C A ], where G is the total number of frequency bands, and if C A <C can realize feature dimension reduction operation; S33: The matrix {X r } L Input the geometric center extraction layer respectively, and extract the center matrix X of each frequency band based on Riemann geometry c ; S34: extract the center X of each frequency band matrix c Input tangent space mapping layer, for X c Perform tangent space mapping: Among them, upper represents the operation of obtaining the upper triangular elements of the matrix, logm represents the matrix logarithm operation, D is the reference matrix, which can be set to the unit diagonal matrix I, and Q represents the number of eigenvalues. For x rm The Qth eigenvalue of .

6. The EEG signal decoding method of multi-dimensional feature fusion according to claim 1, characterized in that: In step S4, the feature selection method combining the machine learning method with the wrapper-type recursive feature elimination method is used to screen out redundant features, including: S41: Initialize the support vector machine classification model; S42: Train the support vector machine model and obtain the coefficients [λ1,λ2,λ3,…,λ Q ], features with large coefficients represent greater contributions to the decision boundary; S43: According to the feature importance scores provided by the model, delete the feature with the smallest coefficient and retrain the model using the remaining features, and re-evaluate the importance of the remaining features [λ1,λ2,λ3,…,λ P ]; S44: recursively delete features and retrain the model, deleting the feature with the smallest importance coefficient each time, until the remaining features reach a predetermined number or the performance of the model is no longer improved, constructing a final feature selection strategy, and setting the final support vector machine model as a classification model; According to the final feature selection strategy, the output x of the deep model in step S3 is rm Perform feature selection to obtain the best feature subset x s , and x s The support vector machine classification model obtained in step S44 is input to obtain the predicted signal category, thereby decoding the EEG signal.

7. A multi-dimensional feature fusion EEG signal decoding system, characterized in that: include Data acquisition module: The data acquisition module uses an electrode cap to collect multi-channel EEG signals within a predetermined time according to a certain sampling rate and records them as raw EEG signals; Data preprocessing module: The data preprocessing module removes noise and artifacts from the original EEG signal, and performs multi-band filtering on the EEG signal to obtain the corresponding multi-band EEG signal; Feature extraction module: The feature extraction module uses a deep feature extraction model to extract multi-dimensional features of multi-band EEG signals in Euclidean space and Riemann manifold; Feature selection module: The feature selection module screens the extracted multi-dimensional features according to the feature selection strategy, removes redundant features, and obtains features with significant classification contribution; Classification module: The classification module uses the support vector machine model to classify the signal and obtain the decoding result of the EEG signal category.

8. The EEG signal decoding system with multi-dimensional feature fusion according to claim 7, characterized in that: The feature selection module includes a filtering module and a Riemann module. The filtering module includes several layers of spatiotemporal filtering layers. The spatiotemporal filtering layers perform spatiotemporal convolution operations on multi-band EEG signals to obtain multi-band spatiotemporal filtering signals x ff (t); The Riemann module includes a Riemann manifold mapping layer, a bilinear mapping layer, a geometric center extraction layer, and a tangent space mapping layer. The Riemann module projects the multi-band spatiotemporal filtering signal onto the Riemann manifold, combines the Riemannian geometry method with the convolutional neural network method, and extracts the geometric structure features of the EEG signal in a data-driven manner, and finally outputs a multi-dimensional fusion feature vector x containing the spatiotemporal frequency characteristics on the Euclidean space and the geometric structure characteristics on the Riemann manifold. rm .

9. A computer-readable storage medium, wherein the computer-readable storage medium can store computer-executable instructions, characterized in that When the executable instructions are executed by the control processor, the EEG signal decoding method with multi-dimensional feature fusion as claimed in any one of claims 1 to 6 is implemented.

10. A device for an electroencephalogram signal decoding method, characterized in that: It includes at least one data preprocessor and at least one signal decoding processor, and also includes a storage device for communicating with the at least one data preprocessor and the at least one signal decoding processor; the storage device stores instructions that can be executed by the at least one data preprocessor and instructions that can be executed by at least one signal decoder, the data preprocessing instructions are executed by the at least one data preprocessor, and the signal decoder instructions are executed by the signal decoder processor, so that the at least one control processor can execute the EEG signal decoding method as described in any one of claims 1 to 6.

Images