Motor imagery EEG signal decoding method based on the interaction between spectral and spatial domains

Abstract

The present invention discloses a method for decoding motor imagery EEG signals based on the interaction of spectral and spatial domains, comprising: acquiring motor imagery EEG signals and preprocessing them; extracting mixed domain features and spatial guidance features based on a dual-branch structure, and then using the spatial guidance features to enhance spatial expression capabilities; performing spectral domain interaction on the enhanced spatial domain features based on an attention mechanism; performing spectral-correlated time-domain aggregation on the spectral domain interaction features based on an attention mechanism; and utilizing a fully connected layer to output the probability that the motor imagery EEG signal belongs to a certain motor imagery category based on the time-domain aggregation features, thereby ultimately decoding the motor imagery EEG signal. This method conducts an in-depth analysis of the non-grid nature of the spatial domain features of motor imagery EEG signals, while also modeling the cross-frequency coupling phenomenon of motor imagery EEG signals, thereby achieving complete time-domain, spectral-domain, and spatial-domain feature extraction for motor imagery EEG signals.

Description

Technical Field

[0001] The present invention relates to the technical field of motor imagery EEG signal decoding, and in particular to a motor imagery EEG signal decoding method based on the interaction between spectral domain and spatial domain. Background Art

[0002] Brain-computer interface (BCI) systems offer a novel approach to human-computer interaction by directly interpreting a user's brain signals and converting them into control commands for peripheral devices. Brain signals exist in various forms, with EEG signals increasingly gaining favor among researchers due to their non-invasiveness, low cost, and high temporal resolution. Three main BCI paradigms based on EEG signals exist: motor imagery, P300, and steady-state visual evoked potentials. Of these three paradigms, motor imagery allows users to spontaneously perform corresponding neurophysiological activities without external stimulation, and therefore has attracted increasing research attention. Motor imagery refers to the mental rehearsal of specific limb movements without actually performing them. During motor imagery, the ε and β rhythms in the sensorimotor cortex undergo energy modulation, a phenomenon known as event-related desynchronization and event-related synchronization. BCI systems based on motor imagery EEG signals have been applied in a variety of healthcare applications, such as neurorehabilitation, assisting in the recovery of lost or impaired limb function, and controlling robotic wheelchairs.

[0003] A BCI system based on motor imagery EEG signals typically consists of three components: motor imagery EEG signal acquisition, motor imagery EEG signal decoding, and external devices. Motor imagery EEG signal decoding is crucial for the development of BCI systems based on motor imagery EEG signals. The performance of the decoding method directly impacts the effectiveness of the BCI system. Typically, the use of a BCI system based on motor imagery EEG signals consists of two phases: offline training and online testing. During the offline training phase, the user performs motor imagery and generates corresponding EEG signals, which are used solely to train the decoding method. During the online testing phase, the user performs motor imagery and generates corresponding EEG signals, which are directly input into the trained decoding method, which then outputs commands to control external devices. Because the process of acquiring motor imagery EEG signals is cumbersome and the EEG signals generated by subjects performing the same motor imagery task on different days vary significantly, the motor imagery EEG signal decoding method must have sufficient generalization capabilities while remaining lightweight.

[0004] However, current methods for decoding motor imagery EEG signals still have many limitations. They can be broadly categorized into two main categories: machine learning-based methods and deep learning-based methods. Machine learning-based methods overly rely on prior neurophysiological knowledge and exhibit poor robustness to noise. Regarding deep learning-based methods, first, current decoding methods ignore the significant changes in temporal fluctuation patterns caused by mental fatigue during motor imagery. Second, most decoding methods use convolutional neural networks to extract spatial features of motor imagery EEG signals. However, the spatially uneven distribution of electrodes used to collect motor imagery EEG signals makes it difficult for convolutional neural networks to fully extract spatial features of motor imagery EEG signals. Furthermore, convolutional neural networks struggle to model the interactions between different brain regions during motor imagery. Finally, current decoding methods ignore cross-frequency coupling during motor imagery and fail to effectively model these interactions.

[0005] Although a few deep learning-based motor imagery EEG signal decoding methods have begun to attempt to use graph neural networks to model the interactions between different brain regions during subjects' motor imagery, there are still many limitations. First, the adjacency matrix defined by most graph neural network-based motor imagery EEG signal decoding methods cannot fully reflect the interactions between different brain regions. Second, the internal computational mechanism of graph neural networks greatly increases the number of parameters required for the decoding method, making it difficult to lightweight graph neural network-based motor imagery EEG signal decoding methods. Finally, most graph neural network-based motor imagery EEG signal decoding methods treat the brain network structure of subjects performing motor imagery as static. Relevant studies have shown that the brain network structure of subjects will undergo significant changes when performing different motor imagery tasks. Existing graph neural network-based motor imagery EEG signal decoding methods ignore the dynamic characteristics of the subjects' brain network structure. Summary of the Invention

[0006] The purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and propose a method for decoding motor imagery EEG signals based on the interaction of spectral domain and spatial domain, which can enhance the spatial expression ability based on the non-grid nature of the spatial domain characteristics of motor imagery EEG signals. In addition, the cross-frequency coupling of motor imagery EEG signals is modeled through spectral domain interaction based on the attention mechanism. Finally, the significant changes in the time domain fluctuation pattern caused by mental fatigue when the subjects perform motor imagery are modeled through spectral correlation time domain aggregation based on the attention mechanism.

[0007] To achieve the above objectives, the present invention provides a technical solution: a method for decoding motor imagery EEG signals based on the interaction between spectral domain and spatial domain, comprising the following steps:

[0008] S1: Acquire motor imagery EEG signals and preprocess them to obtain multi-band motor imagery EEG signals with specific frequency components;

[0009] S2: Design a dual-branch structure consisting of a hybrid domain branch and a spatial guidance branch. This dual-branch structure takes multi-band motor imagery EEG signals as input, uses the hybrid domain branch to extract hybrid domain features of the multi-band motor imagery EEG signals, which include time domain features, spectral domain features, and spatial domain features. At the same time, the spatial guidance branch uses the non-grid nature of the spatial domain features of the motor imagery EEG signals to extract spatial guidance features of the multi-band motor imagery EEG signals. Finally, the spatial guidance features are used to enhance the spatial expression ability of the hybrid domain features and output enhanced spatial domain features.

[0010] S3: The enhanced spatial domain features are subjected to spectral domain interaction based on the attention mechanism. First, the spectral-level global information of the enhanced spatial domain features is aggregated. Then, the coefficients of the spectral-level global information are reconstructed based on the cross-correlation technology. The coefficient reconstruction is based on the global information of each spectral channel. The reconstructed coefficients are recombined in a cross-correlation manner and output. The enhanced spatial domain features are feature corrected based on the reconstructed coefficients, thereby achieving spectral domain interaction and outputting spectral domain interaction features.

[0011] S4: Aggregate the spectral domain interaction features in the spectral-correlated time domain based on the attention mechanism, assign weights to each subsequence in different spectral channels in the spectral domain interaction features along the time domain, and then aggregate the spectral domain interaction features along the time domain according to the weights based on the attention mechanism and output the time domain aggregated features;

[0012] S5: Use the fully connected layer to output the probability that the motor imagery EEG signal belongs to a certain motor imagery category based on the time domain aggregation features;

[0013] S6: Use the maximum index function to output the final motor imagery EEG signal decoding result according to the probability that the motor imagery EEG signal belongs to a certain motor imagery category.

[0014] Further, in step S1, for the input motor imagery EEG signal x raw ∈R C×T , where R represents the real number domain, C represents the number of electrodes used to collect motor imagery EEG signals, and T represents the number of sampling points contained in the motor imagery EEG signals; a bandpass filter bank is used for preprocessing and outputs a multi-band motor imagery EEG signal with specific frequency components. Among them, N bRepresents the number of bandpass filters in the bandpass filter group; the bandpass frequencies of the bandpass filter group are: 4-8 Hz, 8-12 Hz, 12-16 Hz, 16-20 Hz, 20-24 Hz, 24-28 Hz, 28-32 Hz, 32-36 Hz, and 36-40 Hz.

[0015] Furthermore, in step S2, the input multi-band motor imagery EEG signal x fb ∈R Nb×C×T , where R represents the real number field, N b Represents the number of band-pass filters in the band-pass filter group, C represents the number of electrodes for collecting motor imagery EEG signals, and T represents the number of sampling points contained in the motor imagery EEG signals; the mixed domain features and spatial guidance features will be extracted in the mixed domain branch and the spatial guidance branch respectively;

[0016] The specific operation steps of the hybrid domain branch are as follows:

[0017] S211: For the input multi-band motor imagery EEG signal, the m*C*1 convolutional layer is first used to extract the spatial domain features containing richer spectral domain semantic information, where m represents the number of spectral channels. The spatial domain features x are then input into the BN layer and the exponential linear unit activation function to obtain the spatial domain features x. ssf ∈R m×T :

[0018] x ssf =σ(F bn (F sconv (x fb )))

[0019] Where, F sconv represents the m*C*1 convolutional layer, F bn represents the BN layer, σ represents the exponential linear unit activation function;

[0020] S212: Use the time window to split each spectral channel of the spatial domain feature into multiple subsequences, so as to accurately represent the change of the time domain pattern of each spectral channel in the spatial domain feature, and then calculate the logarithmic variance of each subsequence to extract the mixed domain feature x tssf ∈R m×t , the calculation process is as follows:

[0021]

[0022] Where x tssf [a,b] represents the bth element in the ath spectral channel of the mixed domain feature, log represents the logarithmic function, w represents the length of the subsequence, x ssf [a,i] represents the i-th element in the a-th spectral channel of the spatial domain feature, μ ssf[a,b] represents the average value of the b-th subsequence in the a-th spectral channel of the spatial domain feature. represents the number of subsequences, represents round down;

[0023] The specific operation steps of the spatial guidance branch are as follows:

[0024] S221: For the input multi-band motor imagery EEG signal, in order to ensure that the spatial guidance feature can effectively enhance the spatial expression ability of the mixed domain feature, it is necessary to ensure that the mixed domain feature and the spatial guidance feature are compatible in the time domain. The spectral aggregation feature x is obtained by sequentially using a 1*1*1 convolutional layer, a BN layer, and an exponential linear unit activation function. sf ∈R C×T ; Use the time window to split the spectral aggregation feature into multiple subsequences; then calculate the logarithmic variance of each subsequence to extract the subsequence spectral aggregation feature x tsf ∈R C×t , the calculation process is as follows:

[0025] x sf =σ(F bn (F pconv (x fb )))

[0026]

[0027] Where, F pconv represents a 1*1*1 convolutional layer, x tsf [d,e] represents the element in the dth row and eth column of the subsequence spectrum aggregation feature, x sf [d,j] represents the element in the dth row and jth column of the spectral aggregation feature, μ sf [d,e] represents the average value of the e-th subsequence in the d-th row of the spectral aggregation feature;

[0028] S222: For the subsequence spectral aggregation feature, in order to realize the non-grid nature of the spatial domain characteristics of the motor imagery EEG signal, the spatial guidance features of the multi-band motor imagery EEG signal are extracted and the graph motor imagery EEG signal is constructed. The definition formula of the graph motor imagery EEG signal is as follows:

[0029] G=(V,E,A,H)

[0030] Where G represents graph motor imagery EEG signal, V represents node, E represents edge, A represents adjacency matrix, and H represents node feature;

[0031] In the process of constructing the motor imagery EEG signal into a graph motor imagery EEG signal, the node is defined as the electrode that collects the motor imagery EEG signal, the adjacency matrix is defined as the phase locking value between the motor imagery EEG signals collected by each electrode that collects the motor imagery EEG signal, the definition of the edge is essentially the same as that of the adjacency matrix, and the node feature is defined as the subsequence spectral aggregation feature; the calculation formula of the adjacency matrix is as follows:

[0032]

[0033] Where A[f,g] represents the adjacency matrix value between the electrode collecting the f-th motor imagery EEG signal and the electrode collecting the g-th motor imagery EEG signal, and PLV fg represents the phase locking value between the motor imagery EEG signal collected by the f-th electrode collecting motor imagery EEG signals and the motor imagery EEG signal collected by the g-th electrode collecting motor imagery EEG signals, and threshold represents an artificially defined threshold;

[0034] For graph motion imagery EEG signals, a graph attention network is used to dynamically model spatial domain interactions in a non-Euclidean domain. The graph attention network adaptively calculates attention weights based on the node features between nodes, thereby achieving dynamic feature aggregation. Since the calculation process of the graph attention network only involves local neighborhoods, it is suitable for graph data with unclear structures. The inherent complex network properties of the human brain make graph motion imagery EEG signals belong to graph data with unclear structures. After the graph attention network is used to process the graph motion imagery EEG signals, they are sequentially input into the BN layer and the exponential linear unit activation function. The calculation process is as follows:

[0035] x nf =σ(F bn (F gatv2 (G)))

[0036] Where, F gatv2 represents the graph attention network, x nf ∈R C×t Represents node embedding;

[0037] For node embedding, global sum pooling is used as the readout function, and the calculation process is as follows:

[0038] x sgf =F readout (x nf )

[0039] Where, F readout Represents the readout function, x sgf ∈R t represents the spatial guidance feature;

[0040] S23: After obtaining the hybrid domain features and spatial guidance features, the spatial guidance features are used to enhance the spatial expression capability of the hybrid domain features and output enhanced spatial domain features. The calculation process is as follows:

[0041] x estf =x tssf +x sgf

[0042] Where x estf ∈R m×t Represents enhanced spatial domain features.

[0043] Furthermore, the specific operation steps of step S3 are as follows:

[0044] S31: Enhanced spatial domain features x for the input estf ∈R m×t , where R represents the real number domain, m represents the number of spectral channels, and t represents the number of subsequences; the maximum pooling layer and the average pooling layer are used to extract the spectral level global information z based on the maximum pooling. maxpool ∈R m and spectral-level global information z based on average pooling avgpool ∈R m :

[0045] z maxpool =F maxpool (x estf )

[0046] z avgpool =F avgpool (x estf )

[0047] Where, F maxpool represents the maximum pooling layer, F avgpool represents the average pooling layer;

[0048] S32: Based on the cross-correlation technology, coefficient reconstruction of the spectral level global information based on maximum pooling and the spectral level global information based on average pooling is realized respectively. The coefficient reconstruction recombines the global information of each spectral channel based on the spectral level global information based on maximum pooling and the spectral level global information based on average pooling in a cross-correlation manner and outputs the reconstructed coefficients, and outputs the spectral level context descriptor u based on maximum pooling maxpool ∈R m and the spectral level context descriptor u based on average pooling avgpool ∈R m , the specific calculation process is as follows:

[0049]

[0050] Where u maxpool[h] represents the hth element of the spectral level context descriptor based on maximum pooling, weight maxpool ∈R l represents the learnable weight vector, l represents the scope of the manually defined coefficient reconstruction, represents the first learnable weight vector elements, z maxpool [hk] represents the hkth element of the spectral level global information based on maximum pooling, u avgpool [h] represents the hth element of the spectral level context descriptor based on average pooling, weight avgpool ∈R l represents the learnable weight vector, represents the first learnable weight vector elements, z avgpool [hk] represents the hk-th element of the spectral-level global information based on average pooling;

[0051] S33: After performing coefficient reconstruction and obtaining a spectral context descriptor based on maximum pooling and a spectral context descriptor based on average pooling, the two spectral context descriptors are fused using point-by-point addition:

[0052] u=φ(u maxpool +u avgpool )

[0053] Where u∈R m represents the fused spectral level context descriptor, and φ represents the Sigmoid activation function;

[0054] S34: By fusing spectral-level contextual descriptors with enhanced spatial domain features x estf ∈R m×t The spectral-level multiplication of is used to achieve spectral-level feature correction, thereby realizing spectral domain interaction. In addition, the introduction of residual connections helps to quickly converge during the training process. The specific calculation formula is as follows:

[0055]

[0056] Where x acsif ∈R m×t represents the spectral domain interaction feature, Represents spectral level multiplication.

[0057] Further, in step S4, for the input spectral domain interaction feature x acsif ∈R m×t, where R represents the real number domain, m represents the number of spectral channels, and t represents the number of subsequences. Since the temporal fluctuation patterns of each subsequence in different spectral channels in the spectral domain are significantly different, the spectral correlation time-domain aggregation based on the attention mechanism is used to assign weights to each subsequence in different spectral channels in the spectral domain interaction features along the time domain. Then, the spectral domain interaction features are aggregated along the time domain according to the weights based on the attention mechanism and the time-domain aggregated features x are output. asdtf ∈R m , the specific calculation formula is as follows:

[0058]

[0059] Where x asdtf [p] represents the output value of the pth spectral channel, α pq represents the weight of the qth subsequence in the pth spectral channel, x acsif [p,q] represents the value of the qth subsequence in the pth spectral channel of the spectral domain interaction feature, attn pq ∈R represents the learnable parameters of the qth subsequence in the pth spectral channel, attn pr ∈R represents the learnable parameters of the rth subsequence in the pth spectral channel.

[0060] Furthermore, in step S5, for the input time domain aggregation feature x asdtf ∈R m , where R represents the real number domain and m represents the number of spectral channels. The fully connected layer is used to output the probability that the motor imagery EEG signal belongs to a certain motor imagery category:

[0061] output=F fc (x asdtf )

[0062] Where, F fc represents the fully connected layer, represents the probability that the motor imagery EEG signal belongs to a certain motor imagery category, N c Represents the number of types of motor imagery.

[0063] Further, in step S6, the probability that the motor imagery EEG signal belongs to a certain motor imagery category is N c Represents the number of types of motor imagery, and uses the maximum index function to output the final motor imagery EEG signal decoding result:

[0064] result = maxindex(output)

[0065] Where result represents the final motor imagery EEG signal decoding result, and maxindex represents the maximum index function.

[0066] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0067] 1. The present invention designs a dual-branch structure, which can realize the modeling of the non-grid properties of the spatial domain characteristics of motor imagery EEG signals in the non-Euclidean domain based on the graph motor imagery EEG signals without significantly increasing the number of parameters, and at the same time realizes the dynamic modeling of the complex network structure inside the subject's brain.

[0068] 2. The present invention designs a spectral domain interaction based on the attention mechanism, extracts the spectral-level global information of the enhanced spatial domain through the maximum pooling layer and the average pooling layer, and then realizes the modeling of the cross-frequency coupling phenomenon of motor imagery EEG signals based on the cross-correlation technology.

[0069] 3. The present invention designs a spectral-correlated time-domain aggregation based on the attention mechanism, taking into account the significant changes in the time-domain fluctuation pattern caused by the mental fatigue of the subjects when performing motor imagination, and realizes the modeling of the evolution of the time-domain pattern when the subjects perform motor imagination.

[0070] 4. The motor imagery EEG signal decoding method designed by the present invention has strong generalization ability and requires only a small number of training parameters. In the future, it can be deployed in the brain-computer interface system based on motor imagery EEG signals to improve the performance of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 It is a framework diagram of the method of the present invention; raw is the EEG signal of motor imagery, x fb is the multi-band motor imagery EEG signal, x tssf is the mixed domain feature, x sgf is the spatial guidance feature, is addition, x estf To enhance the spatial domain features, x acsif is the spectral domain interaction feature, x asdtf is the time domain aggregation feature, output is the probability that the motor imagery EEG signal belongs to a certain motor imagery category, and result is the final motor imagery EEG signal decoding result.

[0072] Figure 2 It is a schematic diagram of hybrid domain branch; fb is the multi-band motor imagery EEG signal, m is the number of spectral channels, C is the number of electrodes collecting motor imagery EEG signals, σ is the exponential linear unit activation function, x ssf is the spatial domain feature, w is the length of the subsequence, xtssf is the mixed domain feature, and t is the number of subsequences.

[0073] Figure 3 This is a schematic diagram of the space guidance branch; fb is the multi-band motor imagery EEG signal, σ is the exponential linear unit activation function, C is the number of electrodes collecting motor imagery EEG signals, w is the length of the subsequence, x sf is the spectral aggregation feature, t is the number of subsequences, x tsf is the subsequence spectral aggregation feature, G is the motor imagery EEG signal, x nf is the node embedding, x sgf It is the spatial guidance feature.

[0074] Figure 4 Schematic diagram of spectral domain interaction based on attention mechanism; x estf To enhance the spatial domain features, m is the number of spectral channels, t is the number of subsequences, and z maxpool is the spectral level global information based on maximum pooling, z avgpool is the spectral level global information based on average pooling, u maxpool is the spectral level context descriptor based on maximum pooling, u avgpool is the spectral context descriptor based on average pooling, u is the fused spectral context descriptor, φ is the Sigmoid activation function, is the spectral level multiplication, is addition, x acsif is the spectral domain interaction feature.

[0075] Figure 5 Schematic diagram of spectral correlation time-domain aggregation based on attention mechanism; x acsif is the spectral domain interaction feature, m is the number of spectral channels, ⊙ is the element-by-element multiplication, t is the number of subsequences, α is the weight of the subsequence, and x asdtf It is the time domain aggregation feature. DETAILED DESCRIPTION

[0076] The present invention will be described in further detail below with reference to the embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0077] like Figure 1 As shown, this embodiment discloses a method for decoding motor imagery EEG signals based on the interaction between spectral domain and spatial domain, the details of which are as follows:

[0078] 1) For the input motor imagery EEG signal x raw ∈R C×T, where R represents the real number domain, C represents the number of electrodes used to collect motor imagery EEG signals, and T represents the number of sampling points contained in the motor imagery EEG signals. A bandpass filter bank is used for preprocessing and outputs a multi-band motor imagery EEG signal with specific frequency components. where N b Represents the number of bandpass filters in the bandpass filter bank. The bandpass frequencies of the bandpass filter bank are: 4-8 Hz, 8-12 Hz, 12-16 Hz, 16-20 Hz, 20-24 Hz, 24-28 Hz, 28-32 Hz, 32-36 Hz, and 36-40 Hz.

[0079] 2) A dual-branch structure consisting of a hybrid domain branch and a spatial guidance branch is designed. The dual-branch structure takes multi-band motor imagery EEG signals as input, uses the hybrid domain branch to extract hybrid domain features of the multi-band motor imagery EEG signals including time domain features, spectral domain features and spatial domain features, while uses the spatial guidance branch to extract spatial guidance features of the multi-band motor imagery EEG signals based on the non-grid nature of the spatial domain features of the motor imagery EEG signals. Finally, the spatial guidance features are used to enhance the spatial expression ability of the hybrid domain features and output enhanced spatial domain features.

[0080] For the input multi-band motor imagery EEG signal where N b represents the number of bandpass filters in the bandpass filter bank, C represents the number of electrodes used to collect motor imagery EEG signals, and T represents the number of sampling points contained in the motor imagery EEG signals. The mixed domain features and spatial guidance features are extracted simultaneously in the mixed domain branch and the spatial guidance branch, respectively.

[0081] like Figure 2 As shown, the specific operation steps of the hybrid domain branch are as follows:

[0082] 2.1.1) For the input multi-band motor imagery EEG signal, we first use the m*C*1 convolutional layer to extract the spatial domain features containing richer spectral domain semantic information, where m represents the number of spectral channels. Then, we input it into the BN layer and the exponential linear unit activation function in sequence to obtain the spatial domain features x ssf ∈R m×T :

[0083] x ssf =σ(F bn (F sconv (x fb )))

[0084] Among them, F sconv represents the m*C*1 convolutional layer, F bn represents the BN layer, and σ represents the exponential linear unit activation function.

[0085] 2.1.2) Use the time window to split each spectral channel of the spatial domain feature into multiple subsequences to accurately represent the changes in the temporal pattern of each spectral channel in the spatial domain feature. Then calculate the logarithmic variance of each subsequence to extract the mixed domain feature x tssf ∈R m×t , the calculation process is as follows:

[0086]

[0087] Among them, x tssf [a,b] represents the bth element in the ath spectral channel of the mixed domain feature, log represents the logarithmic function, w represents the length of the subsequence, x ssf [a,i] represents the i-th element in the a-th spectral channel of the spatial domain feature, μ ssf [a,b] represents the average value of the b-th subsequence in the a-th spectral channel of the spatial domain feature. represents the number of subsequences, Represents round down.

[0088] like Figure 3 As shown, the specific operation steps of the spatial guidance branch are as follows:

[0089] 2.2.1) For the input multi-band motor imagery EEG signal, in order to ensure that the spatial guidance feature can effectively enhance the spatial expression ability of the mixed domain feature, it is necessary to ensure that the mixed domain feature and the spatial guidance feature are compatible in the time domain. The spectral aggregation feature x is obtained by sequentially using a 1*1*1 convolutional layer, a BN layer, and an exponential linear unit activation function. sf ∈R C×T The spectral aggregation feature is divided into multiple subsequences using a time window. The logarithmic variance of each subsequence is then calculated to extract the subsequence spectral aggregation feature x tsf ∈R C×t , the calculation process is as follows:

[0090] x sf =σ(F bn (F pconv (x fb )))

[0091]

[0092] Among them, F pconv represents a 1*1*1 convolutional layer, F bn represents the BN layer, σ represents the exponential linear unit activation function, x tsf [d,e] represents the element in the dth row and eth column of the subsequence spectral aggregation feature, log represents the logarithmic function, w represents the length of the subsequence, x represents the length of the subsequence, sf[d,j] represents the element in the dth row and jth column of the spectral aggregation feature, μ sf [d,e] represents the average value of the e-th subsequence in the d-th row of the spectral aggregation feature. represents the number of subsequences, Represents round down.

[0093] 2.2.2) For the subsequence spectral aggregation feature, in order to realize the non-grid nature of the spatial domain characteristics of the motor imagery EEG signal, the spatial guidance features of the multi-band motor imagery EEG signal are extracted and the graph motor imagery EEG signal is constructed. The definition formula of the graph motor imagery EEG signal is as follows:

[0094] G=(V,E,A,H)

[0095] Among them, G represents graph motor imagery EEG signal, V represents node, E represents edge, A represents adjacency matrix, and H represents node feature.

[0096] When constructing motor imagery EEG signals into a graph, nodes are defined as electrodes that collect motor imagery EEG signals. The adjacency matrix is defined as the phase lock values between motor imagery EEG signals collected by each electrode. The definition of edges is essentially the same as that of the adjacency matrix. Node features are defined as subsequence spectral aggregation features. The adjacency matrix is calculated as follows:

[0097]

[0098] Among them, A[f,g] represents the adjacency matrix value between the electrode collecting the f-th motor imagery EEG signal and the electrode collecting the g-th motor imagery EEG signal, and PLV fg represents the phase locking value between the motor imagery EEG signal collected by the f-th electrode collecting motor imagery EEG signals and the motor imagery EEG signal collected by the g-th electrode collecting motor imagery EEG signals, and threshold represents an artificially defined threshold.

[0099] For graph motion imagery EEG signals, a graph attention network is used to dynamically model spatial domain interactions in a non-Euclidean domain. The graph attention network adaptively calculates attention weights based on node features between nodes to achieve dynamic feature aggregation. Since the calculation process of the graph attention network only involves local neighborhoods, it is suitable for graph data with unclear structures. However, the inherent complex network properties within the human brain make graph motion imagery EEG signals belong to graph data with unclear structures. After the graph attention network is used to process the graph motion imagery EEG signals, they are then input into the BN layer and the exponential linear unit activation function in sequence. The calculation process is as follows:

[0100] x nf =σ(Fbn (F gatv2 (G)))

[0101] Among them, G represents the EEG signal of motor imagery, F gatv2 represents graph attention network, F bn represents the BN layer, σ represents the exponential linear unit activation function, x nf ∈R C×t Represents node embedding.

[0102] For node embedding, global sum pooling is used as the readout function, and the calculation process is as follows:

[0103] x sgf =F readout (x nf )

[0104] Among them, F readout Represents the readout function, x sgf ∈R t Represents the spatial guidance feature.

[0105] 2.3) After obtaining the hybrid domain features and spatial guidance features, the spatial guidance features are used to enhance the spatial expression capability of the hybrid domain features and ultimately output the enhanced spatial domain features. The calculation process is as follows:

[0106] x estf =x tssf +x sgf

[0107] Among them, x tssf ∈R m×t represents the mixed domain feature, x sgf ∈R t represents the spatial guidance feature, x estf ∈R m×t Represents enhanced spatial domain features.

[0108] 3) If Figure 4 As shown in Figure 1, the enhanced spatial domain features are used for spectral domain interaction based on the attention mechanism. The specific steps are as follows:

[0109] 3.1) For the input enhanced spatial domain feature x estf ∈R m×t , where m represents the number of spectral channels and t represents the number of subsequences. The maximum pooling layer and average pooling layer are used to extract the spectral level global information z based on the maximum pooling. maxpool ∈R m and spectral-level global information z based on average pooling avgpool ∈R m :

[0110] z maxpool=F maxpool (x estf )

[0111] z avgpool =F avgpool ∈(x estf )

[0112] Among them, F maxpool represents the maximum pooling layer, F avgpool Represents the average pooling layer.

[0113] 3.2) Based on the cross-correlation technology, coefficient reconstruction of the spectral level global information based on maximum pooling and the spectral level global information based on average pooling is realized respectively. The coefficient reconstruction recombines the global information of each spectral channel based on the spectral level global information based on maximum pooling and the spectral level global information based on average pooling in a cross-correlation manner and outputs the reconstructed coefficients, and outputs the spectral level context descriptor u based on maximum pooling maxpool ∈R m and the spectral level context descriptor u based on average pooling avgpool ∈R m , the specific calculation process is as follows:

[0114]

[0115] Among them, u maxpool [h] represents the hth element of the spectral level context descriptor based on maximum pooling, weight maxpool ∈R l represents the learnable weight vector, l represents the scope of the manually defined coefficient reconstruction, represents the first learnable weight vector elements, z maxpool [hk] represents the hkth element of the spectral level global information based on maximum pooling, u avgpool [h] represents the hth element of the spectral level context descriptor based on average pooling, weight avgpool ∈R l represents the learnable weight vector, represents the first learnable weight vector elements, z avgpool [hk] represents the hk-th element of spectral-level global information based on average pooling.

[0116] 3.3) After coefficient reconstruction and obtaining the spectral context descriptor based on maximum pooling and the spectral context descriptor based on average pooling, the two spectral context descriptors are fused using point-by-point addition:

[0117] u=φ(u maxpool +uavgpool )

[0118] Where u∈R m represents the fused spectral level context descriptor, and φ represents the Sigmoid activation function;

[0119] 3.4) By fusing spectral level contextual descriptors with enhanced spatial domain features x estf ∈R m×t The spectral-level multiplication of is used to achieve spectral-level feature correction and thus spectral domain interaction. In addition, the introduction of residual connections helps to quickly converge during the training process. The specific calculation formula is as follows:

[0120]

[0121] Among them, x acsif ∈R m×t represents the spectral domain interaction feature, Represents spectral level multiplication.

[0122] 4) If Figure 5 As shown, for the input spectral domain interaction feature x acsif ∈R m×t , where m represents the number of spectral channels and t represents the number of subsequences. Since the temporal fluctuation patterns of each subsequence in different spectral channels in the spectral domain are significantly different, the spectral correlation time domain aggregation based on the attention mechanism is used to assign weights to each subsequence in different spectral channels in the spectral domain interaction features along the time domain. Then, the spectral domain interaction features are aggregated along the time domain according to the weights based on the attention mechanism and the time domain aggregated features x are finally output. asdtf ∈R m , the specific calculation formula is as follows:

[0123]

[0124] Among them, x asdtf [p] represents the output value of the pth spectral channel, α pq represents the weight of the qth subsequence in the pth spectral channel, x acsif [p,q] represents the value of the qth subsequence in the pth spectral channel of the spectral domain interaction feature, attn pq ∈R represents the learnable parameters of the qth subsequence in the pth spectral channel, attn pr ∈R represents the learnable parameters of the rth subsequence in the pth spectral channel.

[0125] 5) For the input time domain aggregation feature x asdtf ∈R m , where m represents the number of spectral channels, and a fully connected layer is used to output the probability that the motor imagery EEG signal belongs to a certain motor imagery category:

[0126] output=F fc (x asdtf )

[0127] Among them, F fc represents the fully connected layer, represents the probability that the motor imagery EEG signal belongs to a certain motor imagery category, N c Represents the number of types of motor imagery.

[0128] 6) For the probability that the motor imagery EEG signal belongs to a certain motor imagery category, the maximum index function is used to output the final motor imagery EEG signal decoding result:

[0129] result = maxindex(output)

[0130] Among them, result represents the final motor imagery EEG signal decoding result, and maxindex represents the maximum index function.

[0131] The BCIC-IV-2a dataset contains motor imagery EEG signals from nine subjects. The motor imagery paradigm in this dataset covers four areas: the left hand, right hand, tongue, and foot. The researchers used 22 electrodes to collect EEG signals at a sampling rate of 250 Hz. After acquisition, the EEG signals were preprocessed using a bandpass filter with a bandpass frequency of 0.5 to 100 Hz. Each subject's motor imagery EEG signals were collected from two sessions, each consisting of 288 motor imagery trials, each lasting 4 seconds.

[0132] The OpenBMI dataset contains motor imagery EEG signals from 54 subjects. The motor imagery paradigm in this dataset covers two areas: the left hand and the right hand. The researchers used 62 electrodes to collect EEG signals at a sampling rate of 1000 Hz. Only 20 electrodes located in the sensorimotor cortex were selected for analysis, and the signals were downsampled to 250 Hz. For each subject, motor imagery EEG signals were collected from two sessions, each consisting of 200 motor imagery trials, each lasting 4 seconds.

[0133] The details of the BCIC-IV-2a dataset and OpenBMI dataset are shown in Table 1.

[0134] Table 1 Dataset details

[0135]

[0136] To verify the effectiveness of the proposed method, the decoding performance was compared with that of Shallow ConvNet, Deep ConvNet, EEGNet, FBCNet, FBMSNet, IFNet, and ADFCNN methods under session-independent and session-dependent conditions on two commonly used datasets, the BCIC-IV-2a dataset and the OpenBMI dataset.

[0137] Among them, accuracy and F1-score are used to represent decoding performance.

[0138] The experimental results are shown in Tables 2 and 3.

[0139] Table 2 Decoding performance comparison on the BCIC-IV-2a dataset

[0140]

[0141] Table 3 Decoding performance comparison on the OpenBMI dataset

[0142]

[0143] Experimental results show that the motor imagery EEG signal decoding performance of the method of the present invention is better than other methods. For the BCIC-IV-2a dataset, the accuracy and F1-score of the method of the present invention under session-independent conditions are 2.2% and 2.29% higher than those of the best comparison method, while the accuracy and F1-score under session-dependent conditions are 4.65% and 4.81% higher than those of the best comparison method; for the OpenBMI dataset, the accuracy and F1-score of the method of the present invention under session-independent conditions are 4.56% and 5.88% higher than those of the best comparison method, while the accuracy and F1-score under session-dependent conditions are 12.53% and 13.39% higher than those of the best comparison method, verifying the effectiveness of the method of the present invention.

[0144] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A motor imagery EEG signal decoding method based on the interaction between spectral domain and spatial domain, characterized in that: The following steps are involved: S1: Acquire motor imagery EEG signals and preprocess them to obtain multi-band motor imagery EEG signals with specific frequency components; S2: Design a dual-branch structure consisting of a hybrid domain branch and a spatial guidance branch. This dual-branch structure takes multi-band motor imagery EEG signals as input, uses the hybrid domain branch to extract hybrid domain features of the multi-band motor imagery EEG signals, which include time domain features, spectral domain features, and spatial domain features. At the same time, the spatial guidance branch uses the non-grid nature of the spatial domain features of the motor imagery EEG signals to extract spatial guidance features of the multi-band motor imagery EEG signals. Finally, the spatial guidance features are used to enhance the spatial expression ability of the hybrid domain features and output enhanced spatial domain features. S3: The enhanced spatial domain features are subjected to spectral domain interaction based on the attention mechanism. First, the spectral-level global information of the enhanced spatial domain features is aggregated. Then, the coefficients of the spectral-level global information are reconstructed based on the cross-correlation technology. The coefficient reconstruction is based on the global information of each spectral channel. The reconstructed coefficients are recombined in a cross-correlation manner and output. The enhanced spatial domain features are feature corrected based on the reconstructed coefficients, thereby achieving spectral domain interaction and outputting spectral domain interaction features. S4: Aggregate the spectral domain interaction features in the spectral-correlated time domain based on the attention mechanism, assign weights to each subsequence in different spectral channels in the spectral domain interaction features along the time domain, and then aggregate the spectral domain interaction features along the time domain according to the weights based on the attention mechanism and output the time domain aggregated features; S5: Use the fully connected layer to output the probability that the motor imagery EEG signal belongs to a certain motor imagery category based on the time domain aggregation features; S6: Use the maximum index function to output the final motor imagery EEG signal decoding result according to the probability that the motor imagery EEG signal belongs to a certain motor imagery category.

2. The method for decoding motor imagery EEG signals based on the interaction between spectral domain and spatial domain according to claim 1, characterized in that: In step S1, the input motor imagery EEG signal x raw ∈R C×T , where R represents the real number domain, C represents the number of electrodes used to collect motor imagery EEG signals, and T represents the number of sampling points contained in the motor imagery EEG signals; a bandpass filter bank is used for preprocessing and outputs a multi-band motor imagery EEG signal with specific frequency components. Among them, N b Represents the number of bandpass filters in the bandpass filter group; the bandpass frequencies of the bandpass filter group are: 4-8 Hz, 8-12 Hz, 12-16 Hz, 16-20 Hz, 20-24 Hz, 24-28 Hz, 28-32 Hz, 32-36 Hz, and 36-40 Hz.

3. The method for decoding motor imagery EEG signals based on interaction between spectral domain and spatial domain according to claim 1, characterized in that: In step S2, the input multi-band motor imagery EEG signal Among them, R represents the real number field, N b Represents the number of band-pass filters in the band-pass filter group, C represents the number of electrodes for collecting motor imagery EEG signals, and T represents the number of sampling points contained in the motor imagery EEG signals; the mixed domain features and spatial guidance features will be extracted in the mixed domain branch and the spatial guidance branch respectively; The specific operation steps of the hybrid domain branch are as follows: S211: For the input multi-band motor imagery EEG signal, the m*C*1 convolutional layer is first used to extract the spatial domain features containing richer spectral domain semantic information, where m represents the number of spectral channels. The spatial domain features x are then input into the BN layer and the exponential linear unit activation function to obtain the spatial domain features x. ssf ∈R m×T : x ssf =σ(F bn (F sconv (x fb ))) Where, F sconv represents the m*C*1 convolutional layer, F bn represents the BN layer, σ represents the exponential linear unit activation function; S212: Use the time window to split each spectral channel of the spatial domain feature into multiple subsequences, so as to accurately represent the change of the time domain pattern of each spectral channel in the spatial domain feature, and then calculate the logarithmic variance of each subsequence to extract the mixed domain feature x tssf ∈R m×t , the calculation process is as follows: Where x tssf [a,b] represents the bth element in the ath spectral channel of the mixed domain feature, log represents the logarithmic function, w represents the length of the subsequence, x ssf [a,i] represents the i-th element in the a-th spectral channel of the spatial domain feature, μ ssf [a,b] represents the average value of the b-th subsequence in the a-th spectral channel of the spatial domain feature. represents the number of subsequences, represents round down; The specific operation steps of the spatial guidance branch are as follows: S221: For the input multi-band motor imagery EEG signal, in order to ensure that the spatial guidance feature can effectively enhance the spatial expression ability of the mixed domain feature, it is necessary to ensure that the mixed domain feature and the spatial guidance feature are compatible in the time domain. The spectral aggregation feature x is obtained by sequentially using a 1*1*1 convolutional layer, a BN layer, and an exponential linear unit activation function. sf ∈R C×T ; Use the time window to split the spectral aggregation feature into multiple subsequences; then calculate the logarithmic variance of each subsequence to extract the subsequence spectral aggregation feature x tsf ∈R C×t , the calculation process is as follows: x sf =σ(F bn (F pconv (x fb ))) Where, F pconv represents a 1*1*1 convolutional layer, x tsf [d,e] represents the element in the dth row and eth column of the subsequence spectrum aggregation feature, x sf [d,j] represents the element in the dth row and jth column of the spectral aggregation feature, μ sf [d,e] represents the average value of the e-th subsequence in the d-th row of the spectral aggregation feature; S222: For the subsequence spectral aggregation feature, in order to realize the non-grid nature of the spatial domain characteristics of the motor imagery EEG signal, the spatial guidance features of the multi-band motor imagery EEG signal are extracted and the graph motor imagery EEG signal is constructed. The definition formula of the graph motor imagery EEG signal is as follows: G=(V,E,A,H) Where G represents graph motor imagery EEG signal, V represents node, E represents edge, A represents adjacency matrix, and H represents node feature; In the process of constructing the motor imagery EEG signal into a graph motor imagery EEG signal, the node is defined as the electrode that collects the motor imagery EEG signal, the adjacency matrix is defined as the phase locking value between the motor imagery EEG signals collected by each electrode that collects the motor imagery EEG signal, the definition of the edge is essentially the same as that of the adjacency matrix, and the node feature is defined as the subsequence spectral aggregation feature; the calculation formula of the adjacency matrix is as follows: Where A[f,g] represents the adjacency matrix value between the electrode collecting the f-th motor imagery EEG signal and the electrode collecting the g-th motor imagery EEG signal, and PLV fg represents the phase locking value between the motor imagery EEG signal collected by the f-th electrode collecting motor imagery EEG signals and the motor imagery EEG signal collected by the g-th electrode collecting motor imagery EEG signals, and threshold represents an artificially defined threshold; For graph motion imagery EEG signals, a graph attention network is used to dynamically model spatial domain interactions in a non-Euclidean domain. The graph attention network adaptively calculates attention weights based on the node features between nodes, thereby achieving dynamic feature aggregation. Since the calculation process of the graph attention network only involves local neighborhoods, it is suitable for graph data with unclear structures. The inherent complex network properties of the human brain make graph motion imagery EEG signals belong to graph data with unclear structures. After the graph attention network is used to process the graph motion imagery EEG signals, they are sequentially input into the BN layer and the exponential linear unit activation function. The calculation process is as follows: x nf =σ(F bn (F gatv2 (G))) Where, F gatv2 represents the graph attention network, x nf ∈R C×t Represents node embedding; For node embedding, global sum pooling is used as the readout function, and the calculation process is as follows: x sgf =F readout (x nf ) Where, F readout Represents the readout function, x sgf ∈R t represents the spatial guidance feature; S23: After obtaining the hybrid domain features and spatial guidance features, the spatial guidance features are used to enhance the spatial expression capability of the hybrid domain features and output enhanced spatial domain features. The calculation process is as follows: x estf =x tssf +x sgf Where x estf ∈R m×t Represents enhanced spatial domain features.

4. The method for decoding motor imagery EEG signals based on interaction between spectral domain and spatial domain according to claim 1, characterized in that: The specific operation steps of step S3 are as follows: S31: Enhanced spatial domain features x for the input estf ∈R m×t , where R represents the real number domain, m represents the number of spectral channels, and t represents the number of subsequences; the maximum pooling layer and the average pooling layer are used to extract the spectral level global information z based on the maximum pooling. maxpool ∈R m and spectral-level global information z based on average pooling avgpool ∈R m : z maxpool =F maxpool (x estf ) z avgpool =F avgpool (x estf ) Where, F maxpool represents the maximum pooling layer, F avgpool represents the average pooling layer; S32: Based on the cross-correlation technology, coefficient reconstruction of the spectral level global information based on maximum pooling and the spectral level global information based on average pooling is realized respectively. The coefficient reconstruction recombines the global information of each spectral channel based on the spectral level global information based on maximum pooling and the spectral level global information based on average pooling in a cross-correlation manner and outputs the reconstructed coefficients, and outputs the spectral level context descriptor u based on maximum pooling maxpool ∈R m and the spectral level context descriptor u based on average pooling avgpool ∈R m , the specific calculation process is as follows: Where u maxpool [h] represents the hth element of the spectral level context descriptor based on maximum pooling, weight maxpool ∈R l represents the learnable weight vector, l represents the scope of the manually defined coefficient reconstruction, represents the first learnable weight vector elements, z maxpool [hk] represents the hkth element of the spectral level global information based on maximum pooling, u avgpool [h] represents the hth element of the spectral level context descriptor based on average pooling, weight avgpool ∈R l represents the learnable weight vector, represents the first learnable weight vector elements, z avgpool [hk] represents the hk-th element of the spectral-level global information based on average pooling; S33: After performing coefficient reconstruction and obtaining a spectral context descriptor based on maximum pooling and a spectral context descriptor based on average pooling, the two spectral context descriptors are fused using point-by-point addition: u=φ(u maxpool +in avgpool ) Where u∈R m represents the fused spectral level context descriptor, and φ represents the Sigmoid activation function; S34: By fusing spectral-level contextual descriptors with enhanced spatial domain features x estf ∈R m×t The spectral-level multiplication of is used to achieve spectral-level feature correction, thereby realizing spectral domain interaction. In addition, the introduction of residual connections helps to quickly converge during the training process. The specific calculation formula is as follows: Where x acsif ∈R m×t represents the spectral domain interaction feature, Represents spectral level multiplication.

5. The method for decoding motor imagery EEG signals based on the interaction between spectral domain and spatial domain according to claim 1, characterized in that: In step S4, for the input spectral domain interaction feature x acsif ∈R m×t , where R represents the real number domain, m represents the number of spectral channels, and t represents the number of subsequences. Since the temporal fluctuation patterns of each subsequence in different spectral channels in the spectral domain are significantly different, the spectral correlation time-domain aggregation based on the attention mechanism is used to assign weights to each subsequence in different spectral channels in the spectral domain interaction features along the time domain. Then, the spectral domain interaction features are aggregated along the time domain according to the weights based on the attention mechanism and the time-domain aggregated features x are output. asdtf ∈R m , the specific calculation formula is as follows: Where x asdtf ]p] represents the output value of the pth spectral channel, α pq represents the weight of the qth subsequence in the pth spectral channel, x acsif [p,q] represents the value of the qth subsequence in the pth spectral channel of the spectral domain interaction feature, attn pq ∈R represents the learnable parameters of the qth subsequence in the pth spectral channel, attn pr ∈R represents the learnable parameters of the rth subsequence in the pth spectral channel.

6. The method for decoding motor imagery EEG signals based on interaction between spectral domain and spatial domain according to claim 1, characterized in that: In step S5, for the input time domain aggregation feature x asdtf ∈R m , where R represents the real number domain and m represents the number of spectral channels. The fully connected layer is used to output the probability that the motor imagery EEG signal belongs to a certain motor imagery category: output=F fc (x asdtf ) Where, F fc represents the fully connected layer, represents the probability that the motor imagery EEG signal belongs to a certain motor imagery category, N c Represents the number of types of motor imagery.

7. The method for decoding motor imagery EEG signals based on interaction between spectral domain and spatial domain according to claim 1, characterized in that: In step S6, the probability that the motor imagery EEG signal belongs to a certain motor imagery category is N c Represents the number of types of motor imagery, and uses the maximum index function to output the final motor imagery EEG signal decoding result: result = maxindex(output) Where result represents the final motor imagery EEG signal decoding result, and maxindex represents the maximum index function.

Images