Motor imagery electroencephalogram signal decoding method based on interaction of spectral domain and spatial domain

Abstract

The invention discloses a motor imagery electroencephalogram signal decoding method based on interaction of a spectral domain and a spatial domain. The motor imagery electroencephalogram signal decoding method comprises the following steps: acquiring a motor imagery electroencephalogram signal and preprocessing the motor imagery electroencephalogram signal; respectively extracting a mixed domain feature and a spatial guidance feature based on a double-branch structure, and then enhancing the spatial expression ability by using the spatial guidance feature; spectral domain interaction based on an attention mechanism is carried out on the enhanced spatial domain features; spectrum correlation time domain aggregation based on an attention mechanism is carried out on the spectrum domain interaction features; and outputting the probability that the motor imagery electroencephalogram signal belongs to a certain motor imagery category according to the time domain aggregation feature by utilizing a full connection layer, and finally realizing the decoding of the motor imagery electroencephalogram signal. According to the method, deep analysis is carried out on non-grid properties of spatial domain features of the motor imagery electroencephalogram signals, meanwhile, modeling is carried out on the cross-frequency coupling phenomenon of the motor imagery electroencephalogram signals, and complete time domain, spectral domain and spatial domain feature extraction for the motor imagery electroencephalogram signals is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of motor imagery EEG signal decoding, and in particular to a motor imagery EEG signal decoding method based on the interaction of spectral domain and spatial domain. Background Art

[0002] The brain-computer interface system provides users with a novel human-computer interaction method by directly interpreting the user's brain signals and converting them into control commands for peripherals. Brain signals exist in many forms, among which EEG signals are gradually favored by researchers due to their advantages of non-invasiveness, low cost and high temporal resolution. There are three main brain-computer interface paradigms based on EEG signals: motor imagery, P300 and steady-state visual evoked potential. Among the three paradigms mentioned above, motor imagery allows users to spontaneously perform corresponding neurophysiological activities without external stimulation, so motor imagery is increasingly attracting the attention of researchers. Motor imagery refers to the subject's mental rehearsal of specific limb movements without actual execution. During the process of motor imagery, the ε rhythm and β rhythm in the sensorimotor cortex undergo energy modulation, and the above phenomenon is called event-related desynchronization and event-related synchronization. Brain-computer interface systems based on motor imagery EEG signals have been used in a variety of health care applications, such as neurorehabilitation, assisting in the recovery of lost or damaged limb functions, and controlling robot wheelchairs.

[0003] The brain-computer interface system based on motor imagery EEG signals usually consists of three parts: motor imagery EEG signal acquisition, motor imagery EEG signal decoding and external devices. Among them, motor imagery EEG signal decoding is crucial to the development of the brain-computer interface system based on motor imagery EEG signals. The performance of the decoding method will directly affect the effectiveness of the use of the brain-computer interface system. Generally, the use of the brain-computer interface system based on motor imagery EEG signals consists of two stages: offline training and online testing. In the offline training stage, the user performs motor imagery and generates corresponding EEG signals, which are only used to train the decoding method. In the online testing stage, the user performs motor imagery and generates corresponding EEG signals, which are directly input into the trained decoding method and output instructions to control external devices. Since the process of acquiring motor imagery EEG signals is relatively cumbersome and there are significant differences between the EEG signals generated by subjects performing the same motor imagery task on different days, this requires that the motor imagery EEG signal decoding method needs to have sufficient generalization ability and should be lightweight.

[0004] However, the current motor imagery EEG signal decoding methods still have many limitations. The current motor imagery EEG signal decoding methods can be mainly divided into two categories: motor imagery EEG signal decoding methods based on machine learning and motor imagery EEG signal decoding methods based on deep learning. The motor imagery EEG signal decoding methods based on machine learning overly rely on neurophysiological prior knowledge and have poor robustness to noise. For the motor imagery EEG signal decoding methods based on deep learning. First, the current decoding methods ignore the significant changes in the time domain fluctuation pattern caused by the mental fatigue of the subjects during motor imagery; secondly, most decoding methods use convolutional neural networks to extract the spatial domain features of motor imagery EEG signals. However, the spatial unevenness of the electrodes that collect motor imagery EEG signals makes it difficult for convolutional neural networks to fully extract the spatial domain features of motor imagery EEG signals, and it is also difficult for convolutional neural networks to model the interactions between various brain regions during the subject's motor imagery process; finally, the current decoding methods ignore the cross-frequency coupling phenomenon during the subject's motor imagery process and fail to achieve effective modeling of cross-frequency interactions.

[0005] Although there are a few deep learning-based motor imagery EEG signal decoding methods that have begun to try to use graph neural networks to model the interactions between brain regions during the subject's motor imagery process. However, there are still many limitations. First, the adjacency matrix defined by most graph neural network-based motor imagery EEG signal decoding methods cannot fully reflect the interaction relationship between brain regions; second, the internal computational mechanics of graph neural networks greatly increase the number of parameters required for the decoding method, making it difficult to lightweight the motor imagery EEG signal decoding method based on graph neural networks; finally, most graph neural network-based motor imagery EEG signal decoding methods treat the brain network structure of the subject when performing motor imagery as static. Relevant studies have shown that when the subject performs different motor imagery tasks, the brain network structure of the subject will change significantly. The existing graph neural network-based motor imagery EEG signal decoding methods ignore the dynamic characteristics of the subject's brain network structure. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art, and proposes a method for decoding motor imagery EEG signals based on the interaction of spectral domain and spatial domain, which can enhance the spatial expression ability according to the non-grid nature of the spatial domain characteristics of motor imagery EEG signals. In addition, the modeling of cross-frequency coupling of motor imagery EEG signals is realized through spectral domain interaction based on the attention mechanism. Finally, the modeling of significant changes in time domain fluctuation patterns caused by mental fatigue of subjects during motor imagery is realized through spectral correlation time domain aggregation based on the attention mechanism.

[0007] To achieve the above object, the technical solution provided by the present invention is: a method for decoding motor imagery EEG signals based on the interaction between spectral domain and spatial domain, comprising the following steps:

[0008] S1: Acquire motor imagery EEG signals and preprocess them to obtain multi-band motor imagery EEG signals with specific frequency components;

[0009] S2: Design a dual-branch structure including a hybrid domain branch and a spatial guidance branch. The dual-branch structure takes the multi-band motor imagery EEG signal as input, uses the hybrid domain branch to extract the hybrid domain features of the multi-band motor imagery EEG signal including time domain features, spectral domain features and spatial domain features, and uses the spatial guidance branch to extract the spatial guidance features of the multi-band motor imagery EEG signal based on the non-grid nature of the spatial domain features of the motor imagery EEG signal. Finally, the spatial guidance features are used to enhance the spatial expression ability of the hybrid domain features and output the enhanced spatial domain features.

[0010] S3: The enhanced spatial domain features are subjected to spectral domain interaction based on the attention mechanism. First, the spectral level global information of the enhanced spatial domain features is aggregated. Then, the coefficient reconstruction of the spectral level global information is realized based on the cross-correlation technology. The coefficient reconstruction is based on the global information of each spectral channel. The reconstructed coefficients are recombined in a cross-correlation manner and output. The enhanced spatial domain features are feature corrected based on the reconstructed coefficients, thereby realizing spectral domain interaction and outputting spectral domain interaction features.

[0011] S4: Aggregate the spectral domain interaction features in the spectral-related time domain based on the attention mechanism, assign weights to each subsequence in different spectral channels in the spectral domain interaction features along the time domain, and then aggregate the spectral domain interaction features along the time domain according to the weights based on the attention mechanism and output the time domain aggregated features;

[0012] S5: Use the fully connected layer to output the probability that the motor imagery EEG signal belongs to a certain motor imagery category based on the time domain aggregation features;

[0013] S6: Use the maximum index function to output the final motor imagery EEG signal decoding result according to the probability that the motor imagery EEG signal belongs to a certain motor imagery category.

[0014] Further, in step S1, for the input motor imagery EEG signal x raw ∈R C×T , where R represents the real number domain, C represents the number of electrodes for collecting motor imagery EEG signals, and T represents the number of sampling points contained in the motor imagery EEG signals; a bandpass filter bank is used for preprocessing and a multi-band motor imagery EEG signal with specific frequency components is output. Among them, N bRepresents the number of bandpass filters in the bandpass filter group; the bandpass frequencies of the bandpass filter group are: 4-8Hz, 8-12Hz, 12-16Hz, 16-20Hz, 20-24Hz, 24-28Hz, 28-32Hz, 32-36Hz and 36-40Hz.

[0015] Further, in step S2, for the input multi-band motor imagery EEG signal x fb ∈R Nb×C×T , where R represents the real number domain, N b represents the number of band-pass filters in the band-pass filter group, C represents the number of electrodes for collecting motor imagery EEG signals, and T represents the number of sampling points contained in the motor imagery EEG signals; the mixed domain features and the spatial guidance features are extracted in the mixed domain branch and the spatial guidance branch respectively;

[0016] The specific operation steps of the hybrid domain branch are as follows:

[0017] S211: For the input multi-band motor imagery EEG signal, the m*C*1 convolutional layer is first used to extract the spatial domain features containing richer spectral domain semantic information, where m represents the number of spectral channels. Then, the convolutional layer is sequentially input into the BN layer and the exponential linear unit activation function to obtain the spatial domain features x ssf ∈R m×T :

[0018] x ssf =σ(F bn (F sconv (x fb )))

[0019] In the formula, F sconv represents the m*C*1 convolutional layer, F bn represents the BN layer, σ represents the exponential linear unit activation function;

[0020] S212: Use a time window to split each spectral channel of the spatial domain feature into multiple subsequences, so as to accurately represent the change of the time domain pattern of each spectral channel in the spatial domain feature, and then calculate the logarithmic variance of each subsequence to extract the mixed domain feature x tssf ∈R m×t , the calculation process is as follows:

[0021]

[0022] In the formula, x tssf [a,b] represents the bth element in the ath spectral channel of the mixed domain feature, log represents the logarithmic function, w represents the length of the subsequence, x ssf [a,i] represents the i-th element in the a-th spectral channel of the spatial domain feature, μ ssf[a,b] represents the average value of the b-th subsequence in the a-th spectral channel of the spatial domain feature. represents the number of subsequences, represents round down;

[0023] The specific operation steps of the space guidance branch are as follows:

[0024] S221: For the input multi-band motor imagery EEG signal, in order to ensure that the spatial guidance feature can effectively enhance the spatial expression ability of the mixed domain feature, it is necessary to ensure that the mixed domain feature and the spatial guidance feature are compatible in the time domain. The spectral aggregation feature x is obtained by using a 1*1*1 convolutional layer, a BN layer, and an exponential linear unit activation function in sequence. sf ∈R C×T ; Use the time window to split the spectral aggregation feature into multiple subsequences; then calculate the logarithmic variance of each subsequence to extract the subsequence spectral aggregation feature x tsf ∈R C×t , the calculation process is as follows:

[0025] x sf =σ(F bn (F pconv (x fb )))

[0026]

[0027] In the formula, F pconv represents a 1*1*1 convolutional layer, x tsf [d,e] represents the element in the dth row and eth column of the subsequence spectral aggregation feature, x sf [d,j] represents the element in the dth row and jth column of the spectral aggregation feature, μ sf [d,e] represents the average value of the e-th subsequence in the d-th row of the spectral aggregation feature;

[0028] S222: For the subsequence spectral aggregation features, in order to realize the non-grid nature of the spatial domain characteristics of the motor imagery EEG signal, the spatial guidance features of the multi-band motor imagery EEG signal are extracted, and the graph motor imagery EEG signal is constructed. The definition formula of the graph motor imagery EEG signal is as follows:

[0029] G=(V,E,A,H)

[0030] In the formula, G represents the graph motor imagery EEG signal, V represents the node, E represents the edge, A represents the adjacency matrix, and H represents the node feature;

[0031] In the process of constructing the motor imagery EEG signal into a graph motor imagery EEG signal, the node is defined as the electrode that collects the motor imagery EEG signal, the adjacency matrix is ​​defined as the phase locking value between the motor imagery EEG signals collected by each electrode that collects the motor imagery EEG signal, the definition of the edge is essentially the same as the adjacency matrix, and the node feature is defined as the subsequence spectral aggregation feature; the calculation formula of the adjacency matrix is ​​as follows:

[0032]

[0033] Where A[f,g] represents the adjacency matrix value between the f-th electrode collecting motor imagery EEG signals and the g-th electrode collecting motor imagery EEG signals, and PLV fg represents the phase locking value between the motor imagery EEG signal collected by the f-th electrode collecting the motor imagery EEG signal and the motor imagery EEG signal collected by the g-th electrode collecting the motor imagery EEG signal, and threshold represents an artificially defined threshold;

[0034] For the EEG signals of graph motion imagery, the graph attention network is used to dynamically model the spatial domain interaction in the non-Euclidean domain. The graph attention network adaptively calculates the attention weights according to the node features between nodes, thereby realizing dynamic feature aggregation. Since the calculation process of the graph attention network only involves the local neighborhood, the graph attention network is suitable for graph data with unclear structures. The inherent complex network properties of the human brain make the EEG signals of graph motion imagery belong to graph data with unclear structures. After the graph attention network is used to process the EEG signals of graph motion imagery, they are input into the BN layer and the exponential linear unit activation function in turn. The calculation process is as follows:

[0035] x nf =σ(F bn (F gatv2 (G)))

[0036] In the formula, F gatv2 represents the graph attention network, x nf ∈R C×t Represents node embedding;

[0037] For node embedding, global sum pooling is used as the readout function, and the calculation process is as follows:

[0038] x sgf =F readout (x nf )

[0039] In the formula, F readout represents the readout function, x sgf ∈R t represents the spatial guidance feature;

[0040] S23: After obtaining the mixed domain features and the spatial guidance features, the spatial guidance features are used to enhance the spatial expression capability of the mixed domain features and output the enhanced spatial domain features. The calculation process is as follows:

[0041] x estf =x tssf +x sgf

[0042] In the formula, x estf ∈R m×t Represents enhanced spatial domain features.

[0043] Further, the specific operation steps of step S3 are as follows:

[0044] S31: Enhanced spatial domain features x for the input estf ∈R m×t , where R represents the real number domain, m represents the number of spectral channels, and t represents the number of subsequences; the maximum pooling layer and the average pooling layer are used to extract the spectral level global information z based on the maximum pooling. maxpool ∈R m and the spectral level global information z based on average pooling avgpool ∈R m :

[0045] z maxpool =F maxpool (x estf )

[0046] z avgpool =F avgpool (x estf )

[0047] In the formula, F maxpool represents the maximum pooling layer, F avgpool represents the average pooling layer;

[0048] S32: Based on the cross-correlation technology, coefficient reconstruction of the spectral level global information based on maximum pooling and the spectral level global information based on average pooling is respectively realized. The coefficient reconstruction recombines the global information of each spectral channel based on the spectral level global information based on maximum pooling and the spectral level global information based on average pooling in a cross-correlation manner and outputs the reconstructed coefficients, and outputs the spectral level context descriptor u based on maximum pooling maxpool ∈R m and the spectral level context descriptor u based on average pooling avgpool ∈R m , the specific calculation process is as follows:

[0049]

[0050] In the formula, u maxpool[h] represents the hth element of the spectral context descriptor based on maximum pooling, weight maxpool ∈R l represents the learnable weight vector, l represents the scope of artificially defined coefficient reconstruction, represents the learnable weight vector elements, z maxpool [hk] represents the hkth element of the spectral level global information based on maximum pooling, u avgpool [h] represents the hth element of the spectral context descriptor based on average pooling, weight avgpool ∈R l represents the learnable weight vector, represents the learnable weight vector elements, z avgpool [hk] represents the hkth element of the spectral-level global information based on average pooling;

[0051] S33: After coefficient reconstruction is performed and a spectral level context descriptor based on maximum pooling and a spectral level context descriptor based on average pooling are obtained, the two spectral level context descriptors are fused using point-by-point addition:

[0052] u=φ(u maxpool +u avgpool )

[0053] Where u∈R m represents the fused spectral level context descriptor, and φ represents the Sigmoid activation function;

[0054] S34: By fusing spectral level contextual descriptors with enhanced spatial domain features x estf ∈R m×t The spectral level multiplication of is used to realize spectral level feature correction, and then realize spectral domain interaction. In addition, the introduction of residual connection helps to converge quickly during the training process. The specific calculation formula is as follows:

[0055]

[0056] In the formula, x acsif ∈R m×t represents the spectral domain interaction feature, Represents spectral level multiplication.

[0057] Further, in step S4, for the input spectral domain interaction feature x acsif ∈R m×t, where R represents the real number domain, m represents the number of spectral channels, and t represents the number of subsequences. Since the temporal fluctuation patterns of each subsequence in different spectral channels in the spectral domain are significantly different, the spectral correlation time-domain aggregation based on the attention mechanism is used to assign weights to each subsequence in different spectral channels in the spectral domain interaction features along the time domain. Then, the spectral domain interaction features are aggregated along the time domain according to the weights based on the attention mechanism and the time-domain aggregated features x are output. asdtf ∈R m , the specific calculation formula is as follows:

[0058]

[0059] In the formula, x asdtf [p] represents the output value of the pth spectral channel, α pq represents the weight of the qth subsequence in the pth spectral channel, x acsif [p,q] represents the value of the qth subsequence in the pth spectral channel of the spectral domain interaction feature, attn pq ∈R represents the learnable parameters of the qth subsequence in the pth spectral channel, attn pr ∈R represents the learnable parameters of the rth subsequence in the pth spectral channel.

[0060] Further, in step S5, for the input time domain aggregation feature x asdtf ∈R m , where R represents the real number domain, m represents the number of spectral channels, and a fully connected layer is used to output the probability that the motor imagery EEG signal belongs to a certain motor imagery category:

[0061] output=F fc (x asdtf )

[0062] In the formula, F fc represents the fully connected layer, represents the probability that the motor imagery EEG signal belongs to a certain motor imagery category, N c Represents the number of types of motor imagery.

[0063] Further, in step S6, the probability that the motor imagery EEG signal belongs to a certain motor imagery category is N c Represents the number of types of motor imagery, and uses the maximum index function to output the final motor imagery EEG signal decoding result:

[0064] result = maxindex(output)

[0065] Where result represents the final motor imagery EEG signal decoding result, and maxindex represents the maximum value index function.

[0066] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0067] 1. The present invention designs a dual-branch structure, which can realize the modeling of the non-grid nature of the spatial domain characteristics of motor imagery EEG signals in the non-Euclidean domain based on the graph motor imagery EEG signals without significantly increasing the number of parameters, and at the same time realizes the dynamic modeling of the complex network structure inside the subject's brain.

[0068] 2. The present invention designs a spectral domain interaction based on the attention mechanism, extracts the spectral-level global information of the enhanced spatial domain through the maximum pooling layer and the average pooling layer, and then realizes the modeling of the cross-frequency coupling phenomenon of motor imagery EEG signals based on the cross-correlation technology.

[0069] 3. The present invention designs a spectrally correlated time-domain aggregation based on an attention mechanism, taking into account the significant changes in the time-domain fluctuation pattern caused by the mental fatigue of the subjects when performing motor imagination, and realizes the modeling of the evolution of the time-domain pattern when the subjects perform motor imagination.

[0070] 4. The motor imagery EEG signal decoding method designed by the present invention has strong generalization ability and requires only fewer training parameters. In the future, it can be deployed to the brain-computer interface system based on motor imagery EEG signals to improve the performance of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 It is a framework diagram of the method of the present invention; raw is the EEG signal of motor imagery, x fb is the multi-band motor imagery EEG signal, x tssf is the mixed domain feature, x sgf is the spatial guidance feature, For addition, x estf To enhance the spatial domain features, x acsif is the spectral domain interaction feature, x asdtf is the time domain aggregation feature, output is the probability that the motor imagery EEG signal belongs to a certain motor imagery category, and result is the final motor imagery EEG signal decoding result.

[0072] Figure 2 It is a schematic diagram of hybrid domain branch; fb is the multi-band motor imagery EEG signal, m is the number of spectral channels, C is the number of electrodes for collecting motor imagery EEG signals, σ is the exponential linear unit activation function, x ssf is the spatial domain feature, w is the length of the subsequence, xtssf is the mixed domain feature, and t is the number of subsequences.

[0073] Figure 3 This is a schematic diagram of the space guidance branch; fb is the multi-band motor imagery EEG signal, σ is the exponential linear unit activation function, C is the number of electrodes for collecting motor imagery EEG signals, w is the length of the subsequence, x sf is the spectral aggregation feature, t is the number of subsequences, x tsf is the subsequence spectral aggregation feature, G is the motor imagery EEG signal, x nf is the node embedding, x sgf It is the spatial guidance feature.

[0074] Figure 4 Schematic diagram of spectral domain interaction based on attention mechanism; x estf To enhance the spatial domain features, m is the number of spectral channels, t is the number of subsequences, and z maxpool is the spectral level global information based on maximum pooling, z avgpool is the spectral level global information based on average pooling, u maxpool is the spectral level context descriptor based on maximum pooling, u avgpool is the spectral context descriptor based on average pooling, u is the fused spectral context descriptor, φ is the Sigmoid activation function, is the spectral level multiplication, For addition, x acsif It is the spectral domain interaction feature.

[0075] Figure 5 Schematic diagram of spectral correlation time-domain aggregation based on attention mechanism; x acsif is the spectral domain interaction feature, m is the number of spectral channels, ⊙ is the element-by-element multiplication, t is the number of subsequences, α is the weight of the subsequence, and x asdtf It is the time domain aggregation feature. DETAILED DESCRIPTION

[0076] The present invention is further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0077] like Figure 1 As shown, this embodiment discloses a method for decoding motor imagery EEG signals based on the interaction between spectral domain and spatial domain, and the specific details are as follows:

[0078] 1) For the input motor imagery EEG signal x raw ∈R C×T, where R represents the real number domain, C represents the number of electrodes used to collect motor imagery EEG signals, and T represents the number of sampling points contained in the motor imagery EEG signals. Use a bandpass filter bank to preprocess and output a multi-band motor imagery EEG signal with specific frequency components Where N b Represents the number of bandpass filters in the bandpass filter group. The bandpass frequencies of the bandpass filter group are: 4-8Hz, 8-12Hz, 12-16Hz, 16-20Hz, 20-24Hz, 24-28Hz, 28-32Hz, 32-36Hz and 36-40Hz.

[0079] 2) Design a dual-branch structure consisting of a mixed domain branch and a spatial guidance branch. The dual-branch structure takes the multi-band motor imagery EEG signal as input, uses the mixed domain branch to extract the mixed domain features of the multi-band motor imagery EEG signal including time domain features, spectral domain features and spatial domain features, and uses the spatial guidance branch to extract the spatial guidance features of the multi-band motor imagery EEG signal based on the non-grid nature of the spatial domain features of the motor imagery EEG signal. Finally, the spatial guidance features are used to enhance the spatial expression ability of the mixed domain features and output the enhanced spatial domain features.

[0080] For the input multi-band motor imagery EEG signal Where N b represents the number of bandpass filters in the bandpass filter group, C represents the number of electrodes for collecting motor imagery EEG signals, and T represents the number of sampling points contained in the motor imagery EEG signals. The mixed domain features and spatial guidance features are extracted simultaneously in the mixed domain branch and the spatial guidance branch, respectively.

[0081] like Figure 2 As shown in the figure, the specific operation steps of the hybrid domain branch are as follows:

[0082] 2.1.1) For the input multi-band motor imagery EEG signal, the m*C*1 convolutional layer is first used to extract the spatial domain features containing richer spectral domain semantic information, where m represents the number of spectral channels. Then, the convolutional layer is sequentially input into the BN layer and the exponential linear unit activation function to obtain the spatial domain features x ssf ∈R m×T :

[0083] x ssf =σ(F bn (F sconv (x fb )))

[0084] Among them, F sconv represents the m*C*1 convolutional layer, F bn represents the BN layer, and σ represents the exponential linear unit activation function.

[0085] 2.1.2) Use the time window to split each spectral channel of the spatial domain feature into multiple subsequences to accurately represent the changes in the temporal pattern of each spectral channel in the spatial domain feature. Then calculate the logarithmic variance of each subsequence to extract the mixed domain feature x tssf ∈R m×t , the calculation process is as follows:

[0086]

[0087] Among them, x tssf [a,b] represents the bth element in the ath spectral channel of the mixed domain feature, log represents the logarithmic function, w represents the length of the subsequence, x ssf [a,i] represents the i-th element in the a-th spectral channel of the spatial domain feature, μ ssf [a,b] represents the average value of the b-th subsequence in the a-th spectral channel of the spatial domain feature. represents the number of subsequences, Represents round down.

[0088] like Figure 3 As shown, the specific operation steps of the space guidance branch are as follows:

[0089] 2.2.1) For the input multi-band motor imagery EEG signal, in order to ensure that the spatial guidance feature can effectively enhance the spatial expression ability of the mixed domain feature, it is necessary to ensure that the mixed domain feature and the spatial guidance feature are compatible in the time domain. The spectral aggregation feature x is obtained by sequentially using a 1*1*1 convolutional layer, a BN layer, and an exponential linear unit activation function. sf ∈R C×T The spectral aggregation feature is divided into multiple subsequences using a time window. Then the logarithmic variance of each subsequence is calculated to extract the subsequence spectral aggregation feature x tsf ∈R C×t , the calculation process is as follows:

[0090] x sf =σ(F bn (F pconv (x fb )))

[0091]

[0092] Among them, F pconv represents a 1*1*1 convolutional layer, F bn represents the BN layer, σ represents the exponential linear unit activation function, x tsf [d,e] represents the element in the dth row and eth column of the subsequence spectral aggregation feature, log represents the logarithmic function, w represents the length of the subsequence, x represents the length of the subsequence, sf[d,j] represents the element in the dth row and jth column of the spectral aggregation feature, μ sf [d,e] represents the average value of the e-th subsequence in the d-th row of the spectral aggregation feature. represents the number of subsequences, Represents round down.

[0093] 2.2.2) For the subsequence spectral aggregation features, in order to realize the non-grid nature of the spatial domain characteristics of the motor imagery EEG signal, the spatial guidance features of the multi-band motor imagery EEG signal are extracted, and the graph motor imagery EEG signal is constructed. The definition formula of the graph motor imagery EEG signal is as follows:

[0094] G=(V,E,A,H)

[0095] Among them, G represents graph motor imagery EEG signal, V represents node, E represents edge, A represents adjacency matrix, and H represents node feature.

[0096] In the process of constructing the motor imagery EEG signal into a graph motor imagery EEG signal, the node is defined as the electrode that collects the motor imagery EEG signal. The adjacency matrix is ​​defined as the phase locking value between the motor imagery EEG signals collected by each electrode that collects the motor imagery EEG signal. The definition of the edge is essentially the same as the adjacency matrix. The node feature is defined as the subsequence spectral aggregation feature. The calculation formula of the adjacency matrix is ​​as follows:

[0097]

[0098] Among them, A[f,g] represents the adjacency matrix value between the electrode collecting the f-th motor imagery EEG signal and the electrode collecting the g-th motor imagery EEG signal, and PLV fg represents the phase locking value between the motor imagery EEG signal collected by the f-th electrode collecting motor imagery EEG signals and the motor imagery EEG signal collected by the g-th electrode collecting motor imagery EEG signals, and threshold represents an artificially defined threshold.

[0099] For the EEG signals of graph motion imagery, the graph attention network is used to dynamically model the spatial domain interaction in the non-Euclidean domain. The graph attention network adaptively calculates the attention weights based on the node features between nodes to achieve dynamic feature aggregation. Since the calculation process of the graph attention network only involves the local neighborhood, the graph attention network is suitable for graph data with unclear structure. However, the inherent complex network properties of the human brain make the EEG signals of graph motion imagery belong to graph data with unclear structure. After the graph attention network is used to process the EEG signals of graph motion imagery, they are input into the BN layer and the exponential linear unit activation function in turn. The calculation process is as follows:

[0100] x nf =σ(Fbn (F gatv2 (G)))

[0101] Among them, G represents the EEG signal of motor imagery, F gatv2 represents graph attention network, F bn represents the BN layer, σ represents the exponential linear unit activation function, x nf ∈R C×t Represents node embedding.

[0102] For node embedding, global sum pooling is used as the readout function, and the calculation process is as follows:

[0103] x sgf =F readout (x nf )

[0104] Among them, F readout represents the readout function, x sgf ∈R t Represents the spatial guidance feature.

[0105] 2.3) After obtaining the mixed domain features and spatial guidance features, the spatial guidance features are used to enhance the spatial expression ability of the mixed domain features and finally output the enhanced spatial domain features. The calculation process is as follows:

[0106] x estf =x tssf +x sgf

[0107] Among them, x tssf ∈R m×t represents the mixed domain feature, x sgf ∈R t represents the spatial guidance feature, x estf ∈R m×t Represents enhanced spatial domain features.

[0108] 3) If Figure 4 As shown in the figure, the enhanced spatial domain features are interacted with the spectral domain based on the attention mechanism. The specific steps are as follows:

[0109] 3.1) For the input enhanced spatial domain feature x estf ∈R m×t , where m represents the number of spectral channels and t represents the number of subsequences. The maximum pooling layer and average pooling layer are used to extract the spectral level global information z based on the maximum pooling. maxpool ∈R m and the spectral level global information z based on average pooling avgpool ∈R m :

[0110] z maxpool=F maxpool (x estf )

[0111] z avgpool =F avgpool ∈(x estf )

[0112] Among them, F maxpool represents the maximum pooling layer, F avgpool Represents an average pooling layer.

[0113] 3.2) Based on the cross-correlation technology, the coefficient reconstruction of the spectral level global information based on the maximum pooling and the spectral level global information based on the average pooling is realized respectively. The coefficient reconstruction recombines the global information of each spectral channel based on the spectral level global information based on the maximum pooling and the spectral level global information based on the average pooling in a cross-correlation manner and outputs the reconstructed coefficients, and outputs the spectral level context descriptor u based on the maximum pooling maxpool ∈R m and the spectral level context descriptor u based on average pooling avgpool ∈R m , the specific calculation process is as follows:

[0114]

[0115] Among them, u maxpool [h] represents the hth element of the spectral context descriptor based on maximum pooling, weight maxpool ∈R l represents the learnable weight vector, l represents the scope of artificially defined coefficient reconstruction, represents the learnable weight vector elements, z maxpool [hk] represents the hkth element of the spectral level global information based on maximum pooling, u avgpool [h] represents the hth element of the spectral context descriptor based on average pooling, weight avgpool ∈R l represents the learnable weight vector, represents the learnable weight vector elements, z avgpool [hk] represents the hk-th element of spectral-level global information based on average pooling.

[0116] 3.3) After coefficient reconstruction and obtaining the spectral context descriptor based on maximum pooling and the spectral context descriptor based on average pooling, the two spectral context descriptors are fused using point-by-point addition:

[0117] u=φ(u maxpool +uavgpool )

[0118] Where u∈R m represents the fused spectral level context descriptor, and φ represents the Sigmoid activation function;

[0119] 3.4) By fusing the spectral level contextual descriptor with the enhanced spatial domain feature x estf ∈R m×t The spectral level multiplication of is used to realize spectral level feature correction and thus realize spectral domain interaction. In addition, the introduction of residual connection helps to converge quickly during the training process. The specific calculation formula is as follows:

[0120]

[0121] Among them, x acsif ∈R m×t represents the spectral domain interaction feature, Represents spectral level multiplication.

[0122] 4) If Figure 5 As shown, for the input spectral domain interaction feature x acsif ∈R m×t , where m represents the number of spectral channels and t represents the number of subsequences. Since the temporal fluctuation patterns of each subsequence in different spectral channels in the spectral domain are significantly different, the spectral correlation time-domain aggregation based on the attention mechanism is used to assign weights to each subsequence in different spectral channels in the spectral domain interaction features along the time domain, and then the spectral domain interaction features are aggregated along the time domain according to the weights based on the attention mechanism and finally the time-domain aggregated features x are output. asdtf ∈R m , the specific calculation formula is as follows:

[0123]

[0124] Among them, x asdtf [p] represents the output value of the pth spectral channel, α pq represents the weight of the qth subsequence in the pth spectral channel, x acsif [p,q] represents the value of the qth subsequence in the pth spectral channel of the spectral domain interaction feature, attn pq ∈R represents the learnable parameters of the qth subsequence in the pth spectral channel, attn pr ∈R represents the learnable parameters of the rth subsequence in the pth spectral channel.

[0125] 5) For the input time domain aggregation feature x asdtf ∈R m , where m represents the number of spectral channels, and a fully connected layer is used to output the probability that the motor imagery EEG signal belongs to a certain motor imagery category:

[0126] output=F fc (x asdtf )

[0127] Among them, F fc represents the fully connected layer, represents the probability that the motor imagery EEG signal belongs to a certain motor imagery category, N c Represents the number of types of motor imagery.

[0128] 6) For the probability that the motor imagery EEG signal belongs to a certain motor imagery category, the maximum index function is used to output the final motor imagery EEG signal decoding result:

[0129] result = maxindex(output)

[0130] Among them, result represents the final motor imagery EEG signal decoding result, and maxindex represents the maximum value index function.

[0131] The BCIC-IV-2a dataset contains the motor imagery EEG signals of 9 subjects. The motor imagery paradigm of this dataset includes 4 parts: left hand, right hand, tongue and foot. The researchers of this dataset used 22 electrodes to collect EEG signals with a sampling rate of 250Hz. After collecting the EEG signals, a bandpass filter with a bandpass frequency of 0.5-100Hz was used to pre-process the EEG signals. Each subject contains 2 sessions of motor imagery EEG signals, each session contains 288 motor imagery trials, and each trial lasts 4s.

[0132] The OpenBMI dataset contains motor imagery EEG signals of 54 subjects. The motor imagery paradigm of this dataset includes two parts: left hand and right hand. The researchers of this dataset used 62 electrodes to collect EEG signals with a sampling rate of 1000Hz. Only 20 electrodes located in the sensorimotor cortex were selected for analysis and the signals were downsampled to 250Hz. Each subject contains motor imagery EEG signals of 2 sessions, each session contains 200 motor imagery trials, and each trial lasts 4s.

[0133] The details of the BCIC-IV-2a dataset and OpenBMI dataset are shown in Table 1.

[0134] Table 1 Dataset details

[0135]

[0136] To verify the effectiveness of the method of the present invention, the decoding performance was compared with that of Shallow ConvNet, Deep ConvNet, EEGNet, FBCNet, FBMSNet, IFNet, and ADFCNN methods on two commonly used datasets, BCIC-IV-2a dataset and OpenBMI dataset, under session-independent and session-dependent conditions.

[0137] Among them, accuracy and F1-score are used to represent the decoding performance.

[0138] The experimental results are shown in Tables 2 and 3.

[0139] Table 2 Decoding performance comparison on the BCIC-IV-2a dataset

[0140]

[0141] Table 3 Decoding performance comparison on the OpenBMI dataset

[0142]

[0143] Experimental results show that the motor imagery EEG signal decoding performance of the method of the present invention is better than other methods. For the BCIC-IV-2a dataset, the accuracy and F1-score of the method of the present invention under session-independent conditions are 2.2% and 2.29% higher than the best comparison method, while the accuracy and F1-score under session-related conditions are 4.65% and 4.81% higher than the best comparison method; for the OpenBMI dataset, the accuracy and F1-score of the method of the present invention under session-independent conditions are 4.56% and 5.88% higher than the best comparison method, while the accuracy and F1-score under session-related conditions are 12.53% and 13.39% higher than the best comparison method, which verifies the effectiveness of the method of the present invention.

[0144] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A motor imagery EEG signal decoding method based on the interaction between spectral domain and spatial domain, characterized in that: The following steps are involved: S1: Acquire motor imagery EEG signals and preprocess them to obtain multi-band motor imagery EEG signals with specific frequency components; S2: Design a dual-branch structure including a hybrid domain branch and a spatial guidance branch. The dual-branch structure takes the multi-band motor imagery EEG signal as input, uses the hybrid domain branch to extract the hybrid domain features of the multi-band motor imagery EEG signal including time domain features, spectral domain features and spatial domain features, and uses the spatial guidance branch to extract the spatial guidance features of the multi-band motor imagery EEG signal based on the non-grid nature of the spatial domain features of the motor imagery EEG signal. Finally, the spatial guidance features are used to enhance the spatial expression ability of the hybrid domain features and output the enhanced spatial domain features. S3: The enhanced spatial domain features are subjected to spectral domain interaction based on the attention mechanism. First, the spectral level global information of the enhanced spatial domain features is aggregated. Then, the coefficient reconstruction of the spectral level global information is realized based on the cross-correlation technology. The coefficient reconstruction is based on the global information of each spectral channel. The reconstructed coefficients are recombined in a cross-correlation manner and output. The enhanced spatial domain features are feature corrected based on the reconstructed coefficients, thereby realizing spectral domain interaction and outputting spectral domain interaction features. S4: Aggregate the spectral domain interaction features in the spectral-related time domain based on the attention mechanism, assign weights to each subsequence in different spectral channels in the spectral domain interaction features along the time domain, and then aggregate the spectral domain interaction features along the time domain according to the weights based on the attention mechanism and output the time domain aggregated features; S5: Use the fully connected layer to output the probability that the motor imagery EEG signal belongs to a certain motor imagery category based on the time domain aggregation features; S6: Use the maximum index function to output the final motor imagery EEG signal decoding result according to the probability that the motor imagery EEG signal belongs to a certain motor imagery category.

2. The method for decoding motor imagery EEG signals based on the interaction between spectral domain and spatial domain according to claim 1, characterized in that: In step S1, for the input motor imagery EEG signal x raw ∈R C×T , where R represents the real number domain, C represents the number of electrodes for collecting motor imagery EEG signals, and T represents the number of sampling points contained in the motor imagery EEG signals; a bandpass filter bank is used for preprocessing and a multi-band motor imagery EEG signal with specific frequency components is output. Among them, N b Represents the number of bandpass filters in the bandpass filter group; the bandpass frequencies of the bandpass filter group are: 4-8Hz, 8-12Hz, 12-16Hz, 16-20Hz, 20-24Hz, 24-28Hz, 28-32Hz, 32-36Hz and 36-40Hz.

3. The method for decoding motor imagery EEG signals based on the interaction between spectral domain and spatial domain according to claim 1, characterized in that: In step S2, the input multi-band motor imagery EEG signal Among them, R represents the real number field, N b represents the number of band-pass filters in the band-pass filter group, C represents the number of electrodes for collecting motor imagery EEG signals, and T represents the number of sampling points contained in the motor imagery EEG signals; the mixed domain features and the spatial guidance features are extracted in the mixed domain branch and the spatial guidance branch respectively; The specific operation steps of the hybrid domain branch are as follows: S211: For the input multi-band motor imagery EEG signal, the m*C*1 convolutional layer is first used to extract the spatial domain features containing richer spectral domain semantic information, where m represents the number of spectral channels. Then, the convolutional layer is sequentially input into the BN layer and the exponential linear unit activation function to obtain the spatial domain features x ssf ∈R m×T : x ssf =σ(F bn (F sconv (x fb ))) In the formula, F sconv represents the m*C*1 convolutional layer, F bn represents the BN layer, σ represents the exponential linear unit activation function; S212: Use a time window to split each spectral channel of the spatial domain feature into multiple subsequences, so as to accurately represent the change of the time domain pattern of each spectral channel in the spatial domain feature, and then calculate the logarithmic variance of each subsequence to extract the mixed domain feature x tssf ∈R m×t , the calculation process is as follows: In the formula, x tssf [a,b] represents the bth element in the ath spectral channel of the mixed domain feature, log represents the logarithmic function, w represents the length of the subsequence, x ssf [a,i] represents the i-th element in the a-th spectral channel of the spatial domain feature, μ ssf [a,b] represents the average value of the b-th subsequence in the a-th spectral channel of the spatial domain feature. represents the number of subsequences, represents round down; The specific operation steps of the space guidance branch are as follows: S221: For the input multi-band motor imagery EEG signal, in order to ensure that the spatial guidance feature can effectively enhance the spatial expression ability of the mixed domain feature, it is necessary to ensure that the mixed domain feature and the spatial guidance feature are compatible in the time domain. The spectral aggregation feature x is obtained by using a 1*1*1 convolutional layer, a BN layer, and an exponential linear unit activation function in sequence. sf ∈R C×T ; Use the time window to split the spectral aggregation feature into multiple subsequences; then calculate the logarithmic variance of each subsequence to extract the subsequence spectral aggregation feature x tsf ∈R C×t , the calculation process is as follows: x sf =σ(F bn (F pconv (x fb ))) In the formula, F pconv represents a 1*1*1 convolutional layer, x tsf [d,e] represents the element in the dth row and eth column of the subsequence spectral aggregation feature, x sf [d,j] represents the element in the dth row and jth column of the spectral aggregation feature, μ sf [d,e] represents the average value of the e-th subsequence in the d-th row of the spectral aggregation feature; S222: For the subsequence spectral aggregation features, in order to realize the non-grid nature of the spatial domain characteristics of the motor imagery EEG signal, the spatial guidance features of the multi-band motor imagery EEG signal are extracted, and the graph motor imagery EEG signal is constructed. The definition formula of the graph motor imagery EEG signal is as follows: G=(V,E,A,H) In the formula, G represents the graph motor imagery EEG signal, V represents the node, E represents the edge, A represents the adjacency matrix, and H represents the node feature; In the process of constructing the motor imagery EEG signal into a graph motor imagery EEG signal, the node is defined as the electrode that collects the motor imagery EEG signal, the adjacency matrix is ​​defined as the phase locking value between the motor imagery EEG signals collected by each electrode that collects the motor imagery EEG signal, the definition of the edge is essentially the same as the adjacency matrix, and the node feature is defined as the subsequence spectral aggregation feature; the calculation formula of the adjacency matrix is ​​as follows: Where A[f,g] represents the adjacency matrix value between the f-th electrode collecting motor imagery EEG signals and the g-th electrode collecting motor imagery EEG signals, and PLV fg represents the phase locking value between the motor imagery EEG signal collected by the f-th electrode collecting the motor imagery EEG signal and the motor imagery EEG signal collected by the g-th electrode collecting the motor imagery EEG signal, and threshold represents an artificially defined threshold; For the EEG signals of graph motion imagery, the graph attention network is used to dynamically model the spatial domain interaction in the non-Euclidean domain. The graph attention network adaptively calculates the attention weights according to the node features between nodes, thereby realizing dynamic feature aggregation. Since the calculation process of the graph attention network only involves the local neighborhood, the graph attention network is suitable for graph data with unclear structures. The inherent complex network properties of the human brain make the EEG signals of graph motion imagery belong to graph data with unclear structures. After the graph attention network is used to process the EEG signals of graph motion imagery, they are input into the BN layer and the exponential linear unit activation function in turn. The calculation process is as follows: x nf =σ(F bn (F gatv2 (G))) In the formula, F gatv2 represents the graph attention network, x nf ∈R C×t Represents node embedding; For node embedding, global sum pooling is used as the readout function, and the calculation process is as follows: x sgf =F readout (x nf ) In the formula, F readout represents the readout function, x sgf ∈R t represents the spatial guidance feature; S23: After obtaining the mixed domain features and the spatial guidance features, the spatial guidance features are used to enhance the spatial expression capability of the mixed domain features and output the enhanced spatial domain features. The calculation process is as follows: x estf =x tssf +x sgf In the formula, x estf ∈R m×t Represents enhanced spatial domain features.

4. The method for decoding motor imagery EEG signals based on the interaction between spectral domain and spatial domain according to claim 1, characterized in that: The specific operation steps of step S3 are as follows: S31: Enhanced spatial domain features x for the input estf ∈R m×t , where R represents the real number domain, m represents the number of spectral channels, and t represents the number of subsequences; the maximum pooling layer and the average pooling layer are used to extract the spectral level global information z based on the maximum pooling. maxpool ∈R m and the spectral level global information z based on average pooling avgpool ∈R m : z maxpool =F maxpool (x estf ) z avgpool =F avgpool (x estf ) In the formula, F maxpool represents the maximum pooling layer, F avgpool represents the average pooling layer; S32: Based on the cross-correlation technology, coefficient reconstruction of the spectral level global information based on maximum pooling and the spectral level global information based on average pooling is respectively realized. The coefficient reconstruction recombines the global information of each spectral channel based on the spectral level global information based on maximum pooling and the spectral level global information based on average pooling in a cross-correlation manner and outputs the reconstructed coefficients, and outputs the spectral level context descriptor u based on maximum pooling maxpool ∈R m and the spectral level context descriptor u based on average pooling avgpool ∈R m , the specific calculation process is as follows: In the formula, u maxpool [h] represents the hth element of the spectral context descriptor based on maximum pooling, weight maxpool ∈R l represents the learnable weight vector, l represents the scope of artificially defined coefficient reconstruction, represents the learnable weight vector elements, z maxpool [hk] represents the hkth element of the spectral level global information based on maximum pooling, u avgpool [h] represents the hth element of the spectral context descriptor based on average pooling, weight avgpool ∈R l represents the learnable weight vector, represents the learnable weight vector elements, z avgpool [hk] represents the hkth element of the spectral-level global information based on average pooling; S33: After coefficient reconstruction is performed and a spectral level context descriptor based on maximum pooling and a spectral level context descriptor based on average pooling are obtained, the two spectral level context descriptors are fused using point-by-point addition: u=φ(u maxpool +in avgpool ) Where u∈R m represents the fused spectral level context descriptor, and φ represents the Sigmoid activation function; S34: By fusing spectral level contextual descriptors with enhanced spatial domain features x estf ∈R m×t The spectral level multiplication of is used to realize spectral level feature correction, and then realize spectral domain interaction. In addition, the introduction of residual connection helps to converge quickly during the training process. The specific calculation formula is as follows: In the formula, x acsif ∈R m×t represents the spectral domain interaction feature, Represents spectral level multiplication.

5. The method for decoding motor imagery EEG signals based on the interaction between spectral domain and spatial domain according to claim 1, characterized in that: In step S4, for the input spectral domain interaction feature x acsif ∈R m×t , where R represents the real number domain, m represents the number of spectral channels, and t represents the number of subsequences. Since the temporal fluctuation patterns of each subsequence in different spectral channels in the spectral domain are significantly different, the spectral correlation time-domain aggregation based on the attention mechanism is used to assign weights to each subsequence in different spectral channels in the spectral domain interaction features along the time domain. Then, the spectral domain interaction features are aggregated along the time domain according to the weights based on the attention mechanism and the time-domain aggregated features x are output. asdtf ∈R m , the specific calculation formula is as follows: In the formula, x asdtf ]p] represents the output value of the pth spectral channel, α pq represents the weight of the qth subsequence in the pth spectral channel, x acsif [p,q] represents the value of the qth subsequence in the pth spectral channel of the spectral domain interaction feature, attn pq ∈R represents the learnable parameters of the qth subsequence in the pth spectral channel, attn pr ∈R represents the learnable parameters of the rth subsequence in the pth spectral channel.

6. The method for decoding motor imagery EEG signals based on the interaction between spectral domain and spatial domain according to claim 1, characterized in that: In step S5, for the input time domain aggregation feature x asdtf ∈R m , where R represents the real number domain, m represents the number of spectral channels, and a fully connected layer is used to output the probability that the motor imagery EEG signal belongs to a certain motor imagery category: output=F fc (x asdtf ) In the formula, F fc represents the fully connected layer, represents the probability that the motor imagery EEG signal belongs to a certain motor imagery category, N c Represents the number of types of motor imagery.

7. The method for decoding motor imagery EEG signals based on the interaction between spectral domain and spatial domain according to claim 1, characterized in that: In step S6, the probability that the motor imagery EEG signal belongs to a certain motor imagery category is N c Represents the number of types of motor imagery, and uses the maximum index function to output the final motor imagery EEG signal decoding result: result = maxindex(output) Where result represents the final motor imagery EEG signal decoding result, and maxindex represents the maximum value index function.

Images