Multi-view electroencephalogram and myoelectricity fusion decoding method for action classification

Abstract

The invention provides a multi-view electroencephalogram and myoelectricity fusion decoding method for action classification, which combines the advantages of time-frequency graph feature extraction and graph representation feature extraction. Firstly, EEG and sEMG signals are converted into a time-frequency image through short-time Fourier transform (STFT), space and time features in the time-frequency image are extracted through a 3D convolutional neural network, in order to further improve the expression ability of the features, a convolutional attention module (CBAM) is adopted to dynamically adjust the importance of a feature channel and a space region, and the time-frequency image is extracted through the 3D convolutional neural network. In order to further improve the precision of information aggregation, the method introduces a graph attention network (GAT) to enhance the learning of key information, constructs graph representation of EEG and sEMG signals based on standardized mutual permutation information (SPMI), uses time-frequency image features as node features, and uses a graph convolutional network (GCN) to carry out information propagation and capture spatial features between nodes, so as to further improve the precision of information aggregation. The method comprises the following steps: adaptively calculating association weights among nodes, optimizing a propagation process of node information, finally, carrying out multi-view feature fusion on extracted time-frequency features and graph structure features, obtaining fused features through series operation, and finally, applying the fused features to action classification.

Description

Technical Field

[0001] The present invention belongs to the field of using brain and myoelectricity fusion for action classification and decoding, and is used to improve the accuracy of action classification. Background Art

[0002] With the rapid development of brain-computer interface (BCI) technology, motion classification based on electroencephalogram (EEG) and electromyogram (sEMG) signals has gained widespread application in fields such as rehabilitation medicine, robotic control, and intelligent systems. As two primary bioelectrical signals, EEG and sEMG signals can reflect a person's motor intentions and muscle activity. Therefore, effectively decoding these signals and accurately classifying motions is a key research topic in BCI technology. However, motion classification using EEG and sEMG signals faces many challenges, including signal noise, the complexity of spatiotemporal features, and the correlation between different modal signals.

[0003] Traditional EEG and sEMG signal processing methods primarily rely on single feature extraction and classification techniques, such as time-domain analysis, frequency-domain analysis, or time-frequency analysis. These methods often overlook the inherent correlation and complementarity between EEG and sEMG signals. While time-frequency representation methods such as the short-time Fourier transform (STFT) can capture the signal's spectral characteristics and time series information, single-modality time-frequency images often fail to fully reflect the complex patterns and high-order dependencies within the signal, limiting classification effectiveness.

[0004] In recent years, graph neural network technologies such as graph convolutional networks (GCNs) and graph attention networks (GATs) have demonstrated strong capabilities in processing graph-structured data, effectively capturing spatial relationships and high-order dependencies between nodes. Applying GCNs and GATs to the multimodal fusion of EEG and sEMG signals can further improve the expressiveness of signal features and classification performance. Summary of the Invention

[0005] First, most current methods fuse single-dimensional feature layers, which cannot fully utilize the complementary characteristics of the two signals. Therefore, the purpose of the present invention is to provide a multi-view EMG fusion decoding method for action classification, aiming to improve the accuracy of fusion signal classification. The technical solution of the present invention is:

[0006] It includes the following steps:

[0007] 1. A multi-view brain and myoelectricity fusion decoding method for action classification, characterized by comprising the following steps:

[0008] Step 1: Design an experimental paradigm for synchronous data collection based on EEG motor imagery and muscle execution.

[0009] Step 2: Perform preprocessing operations including denoising, normalization, signal segmentation and reconstruction on the raw EEG and sEMG data obtained in step 1.

[0010] Step 3: Use short-time Fourier transform to perform time-frequency representation of EEG and sEMG signals and generate time-frequency images to extract spectral information and time series features from the signals.

[0011] Step 4: EEG-sEMG graph representation based on normalized permuted mutual information, with the temporal signals of EEG and sEMG signals as node features and the normalized permuted mutual information as edge weights.

[0012] Step 5: A 3D convolution-attention module is proposed, using the 3D CNN model as a feature extractor, and the convolutional attention module is introduced into the model.

[0013] Step 6: A graph convolution-attention fusion module is proposed, which uses graph convolution for information propagation and graph attention to adjust the association weights between nodes, thus improving the model's ability to process complex graph data.

[0014] Step 7: After obtaining the time-frequency features and graph representation features, these feature matrices are serially connected to perform multi-view feature fusion to finally obtain the action classification results.

[0015] 2. A multi-view brain-myography fusion decoding method for action classification according to claim 1, characterized in that: Step 1: This experimental design uses a synchronous experimental paradigm to collect EEG and sEMG signals to study the classification of three action categories, including walking, standing, and sitting. Each action consists of two parts: action execution and motor imagination. The subjects perform motor imagination simultaneously while performing the action. The EEG signal uses a Brain Products 32-lead EEG cap as the EEG acquisition device, and the sEMG signal uses a Delsys device by placing electrodes on the main muscle groups. Both signals are synchronously collected at a sampling frequency of 500Hz. Each action lasts for 5 seconds, with an interval of 5 seconds between actions, and is repeated 30 times.

[0016] 3. The multi-view brain electromyography fusion decoding method for action classification according to claim 1 is characterized in that step 2: preprocessing of EEG and sEMG signals includes denoising, standardization, signal segmentation and reconstruction. The EEG signal is subjected to bandpass filtering to remove low-frequency and high-frequency noise, independent component analysis (ICA) is used to remove motion artifacts and eye movement artifacts, and normalization is performed. The sEMG signal is subjected to high-pass and bandpass filtering to remove noise, maximum normalization and envelope demodulation are performed, and effective electromyographic signals are extracted. Both signals are segmented using overlapping sliding windows to ensure that the signals of each action and motor imagery are effectively processed.

[0017] 4. The multi-view EEG and myoelectric fusion decoding method for action classification according to claim 1, wherein step 3 comprises: performing a time-frequency representation of the preprocessed EEG and sEMG signals based on a short-time Fourier transform. EEG channels C3, C4, and Cz, commonly used for motor imagery, and an sEMG channel are selected, and each signal is converted into a time-frequency image using a short-time Fourier transform (STFT).

[0018] The STFT is defined as:

[0019]

[0020] f∈R represents frequency, x(t)∈L 2 (R) represents the long-term EEG and sEMG signals, τ∈R represents the given time, g(t)∈L 2 (R) represents the window function.

[0021] STFT captures signal variations in time and frequency through framing and Fourier transform, generating time-frequency features for each channel. This reveals the dynamic nature of neural and muscular activity during motor imagery and action execution. The time-frequency images for each channel are stacked, and by aligning the time-frequency maps of different channels, they are stacked to form a sequence of time-frequency maps. This stacking process fuses the time-frequency features of multiple channels, enhancing the expressive power of the fused signal in the time-frequency domain and providing rich information for subsequent feature extraction and classification.

[0022] 5. According to the multi-view EMG fusion decoding method for action classification described in claim 1, step 4: graph representation based on normalized permutation mutual information, specifically:

[0023] (1) Node representation:

[0024] Each channel (EEG and sEMG signal) is a node in the graph. There are C channels in total, each channel has a feature vector Vi, and the feature vector of the i-th channel is Vi∈R d , where d is the dimension of the channel feature.

[0025] V={V1,V2,...,V C}

[0026] Where each Vi represents the feature of the i-th channel, and we use the original time signal as the feature vector.

[0027] (2) Edge representation and adjacency matrix construction:

[0028] The edges in the graph represent the correlation between channels, which is calculated by Standardized Permutation Mutual Information (SPMI). Assume there are two channels V i and Vj , the similarity between them can be expressed as:

[0029] A ij =SPMI(V i ,V j )

[0030] Among them, A ij An element in the adjacency matrix represents the similarity or correlation between the i-th channel and the j-th channel. SPMI measures the statistical dependence between two channels. The SPMI value range is [0, 1], with higher values indicating stronger correlation between channels. By connecting channels with high correlation based on a threshold, we can capture the connectivity between channels and thus construct edges in the graph.

[0031] (3) Standardization of adjacency matrix:

[0032] In order to eliminate the influence of node degree in the graph, the adjacency matrix needs to be normalized. The normalization of the adjacency matrix can be calculated by the following formula:

[0033]

[0034] Where A is the original adjacency matrix and D is the degree matrix, defined as D ii =∑ j A ij , that is, the degree of the i-th channel. By normalizing the adjacency matrix Eliminates the degree differences of nodes.

[0035] (4) Construct the final graph:

[0036] According to the above calculations, the final graph G can be represented as a graph containing nodes and edges, where: the node set V = {V1, V2, ..., V C}, representing the features of each channel. Adjacency matrix Represents the similarity or relationship between channels. The final graph structure can be expressed as:

[0037] 6. According to the multi-view EEG fusion decoding method for action classification described in claim 1, in step 5: we propose a 3D CNN model as a feature extractor. This model consists of three 3D convolutional stacks, three maximum pooling layers, and one fully connected layer. In the first convolutional stack, the convolution kernel size is 6*3*3, and the pooling layer size is 1*2*2. In the other two convolutional stacks, the convolutional and pooling layers are 1*1*1. All convolutional layers use the RELU activation function, and batch normalization is used to increase the model's generalization ability, prevent overfitting, and speed up training. To better extract key features, this method introduces a convolutional block attention module (CBAM) into the model. CBAM applies different attention to the channel and spatial dimensions of the data, dynamically adjusting the importance of each channel and spatial region, highlighting the learning of key information. This attention mechanism can enhance feature selection and improve the accuracy and robustness of action classification.

[0038] 7. According to the multi-view EEG fusion decoding method for action classification as described in claim 1, step 6: the EEG-sEMG graph structure obtained in step 4 is used, and GCN captures spatial domain features by aggregating information of adjacent nodes. The basic format of graph convolution is given by the following equation:

[0039]

[0040] By stacking multiple graph convolution layers, higher-level features can be extracted layer by layer. Assuming there are L layers of graph convolution, the final feature representation is:

[0041]

[0042] Among them, H l Represents the node feature matrix of the lth layer, W l The weight matrix of this layer, is the normalized adjacency matrix, and σ is the activation function. The output of each layer is the transformation result of the features of the previous layer, through the adjacency matrix and the weight matrix W lPropagating features. By stacking multiple graph convolutional layers, GCN is able to capture higher-level features layer by layer, thereby improving the expressive power of the model. GCN can be viewed as a form of Laplace smoothing, and deeper layers can lead to over-smoothing, so the number of layers should not be too deep. Two GCN layers are used in the model to balance the depth of feature extraction and avoid the risk of over-smoothing, thereby fully leveraging the advantages of graph convolutional networks to extract efficient and discriminative node features. Based on the second layer of graph convolution, a graph attention layer is added to calculate the attention weights between nodes and neighbors, thereby more accurately aggregating information from neighboring nodes.

[0043] Formula for graph attention layer:

[0044]

[0045] Where: N(i) represents the set of neighbor nodes of node i, α ij is the attention weight of node i to neighbor node j, W l The weight matrix of this layer, σ, is the activation function. Graph convolution aggregates node information using the adjacency matrix, while the graph attention mechanism calculates attention weights based on the characteristics of different neighboring nodes, making information aggregation more flexible and dynamic. By adding the graph attention layer, the model can adaptively adjust the relationship weights between nodes, thereby improving representation and classification performance.

[0046] 8. The multi-view EMG fusion decoding method for action classification according to claim 1, characterized in that, in step 7, after obtaining the time-frequency features and the graph representation features, these feature matrices are subjected to a series operation to perform multi-view feature fusion. The series operation is defined as follows:

[0047] X=X S ||X G

[0048] Among them, X G 、X S where represents the graph representation and the time-frequency representation, respectively, and || represents the concatenation operation. Finally, we feed the fused features into a 256-dimensional fully connected layer. After the SoftMax activation layer, the output is divided into three action categories.

[0049] The advantages and beneficial effects of the present invention are as follows:

[0050] The main advantage of the present invention is that it proposes a multi-view brain electromyography fusion decoding method for action classification. The multi-view fusion of this method realizes a comprehensive analysis of EEG and sEMG signals by simultaneously combining time-frequency features and graph structure features. The time-frequency image provides rich time-frequency information, while the graph structure captures the spatial and relational information between each signal channel. By connecting the features of these two views in series, the present method can effectively fuse the advantages of multimodal signals and maximize the use of the time-frequency features and graph structure features of the signal, thereby providing richer input data for classification. This combination of multiple views not only improves the expressive power of features, but also enhances the model's ability to learn complex data patterns, effectively improving performance in action classification tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a multi-view EMG fusion decoding method for action classification. DETAILED DESCRIPTION

[0052] The following will describe the technical solutions in the embodiments of the present invention in detail with reference to the accompanying drawings. The described embodiments are only a part of the embodiments of the present invention.

[0053] As shown in the figure, this embodiment provides a multi-view brain and myoelectric fusion decoding method for action classification, including the following steps:

[0054] Step 1: Design an experimental paradigm for synchronous data collection based on EEG motor imagery and muscle execution.

[0055] Step 2: Perform preprocessing operations including denoising, normalization, signal segmentation and reconstruction on the raw EEG and sEMG data obtained in step 1.

[0056] Step 3: Use short-time Fourier transform to perform time-frequency representation of EEG and sEMG signals and generate time-frequency images to extract spectral information and time series features from the signals.

[0057] Step 4: EEG-sEMG graph representation based on normalized permuted mutual information, with the temporal signals of EEG and sEMG signals as node features and the normalized permuted mutual information as edge weights.

[0058] Step 5: A 3D convolution-attention module is proposed, using the 3D CNN model as a feature extractor, and the convolutional attention module is introduced into the model.

[0059] Step 6: A graph convolution-attention fusion module is proposed, which uses graph convolution for information propagation and graph attention to adjust the association weights between nodes, thus improving the model's ability to process complex graph data.

[0060] Step 7: After obtaining the time-frequency features and graph representation features, these feature matrices are serially connected to perform multi-view feature fusion to finally obtain the action classification results.

Claims

1. A multi-view brain and myoelectricity fusion decoding method for action classification, characterized in that: The following steps are involved: Step 1: Design an experimental paradigm for synchronous data collection based on EEG motor imagery and muscle execution; Step 2: Perform preprocessing operations including denoising, normalization, signal segmentation and reconstruction on the raw EEG and sEMG data obtained in step 1; Step 3: Use short-time Fourier transform to perform time-frequency representation on EEG and sEMG signals and generate time-frequency images to extract spectral information and time series features from the signals; Step 4: EEG-sEMG graph representation based on normalized permuted mutual information, using the temporal signals of EEG and sEMG signals as node features and the normalized permuted mutual information as edge weights; Step 5: A 3D convolution-attention module is proposed, using the 3D CNN model as a feature extractor and introducing the convolution-attention module into the model; Step 6: A graph convolution-attention fusion module is proposed, which uses graph convolution for information propagation and graph attention to adjust the association weights between nodes, improving the model's ability to process complex graph data. Step 7: After obtaining the time-frequency features and graph representation features, these feature matrices are serially connected to perform multi-view feature fusion to finally obtain the action classification results.

2. The multi-view brain and myoelectricity fusion decoding method for action classification according to claim 1, characterized in that: Step 1: This experimental design uses a synchronous experimental paradigm to collect EEG and sEMG signals to study the classification of three action categories, including walking, standing, and sitting. Each action includes two parts: action execution and motor imagery. The subjects simultaneously perform motor imagery while performing the action. EEG signals use a Brain Products 32-lead EEG cap as an EEG acquisition device, and sEMG signals use a Delsys device by placing electrodes on major muscle groups. Both signals are synchronously acquired at a sampling frequency of 500 Hz. Each action lasts for 5 seconds, with a 5-second interval between actions, and is repeated 30 times.

3. The multi-view brain and myoelectricity fusion decoding method for action classification according to claim 1, characterized in that: Step 2: Preprocessing of EEG and sEMG signals includes denoising, standardization, signal segmentation, and reconstruction. EEG signals are bandpass filtered to remove low- and high-frequency noise, and independent component analysis (ICA) is used to remove motion and eye movement artifacts. Normalization is then performed. sEMG signals are high-pass and bandpass filtered to remove noise, followed by maximum normalization and envelope demodulation to extract valid electromyographic signals. Both signals are segmented using overlapping sliding windows to ensure that each action and motor imagery signal is effectively processed.

4. According to the multi-view brain and myoelectric fusion decoding method for action classification described in claim 1, step 3: based on the time-frequency representation of the preprocessed EEG and sEMG signals using short-time Fourier transform, the EEG channels C3, C4, and Cz commonly used in motor imagery, and the sEMG channel are selected, and each signal is converted into a time-frequency image using short-time Fourier transform (STFT). The STFT is defined as: f∈R represents frequency, x(t)∈L 2 (R) represents the long-term EEG and sEMG signals, τ∈R represents the given time, g(t)∈L 2 (R) represents the window function; STFT captures signal variations in time and frequency through framing and Fourier transform, generating time-frequency features for each channel. This reveals the dynamic nature of neural and muscular activity during motor imagery and action execution. The time-frequency images for each channel are stacked, and by aligning the time-frequency maps of different channels, they are stacked to form a sequence of time-frequency maps. This stacking process fuses the time-frequency features of multiple channels, enhancing the expressive power of the fused signal in the time-frequency domain and providing rich information for subsequent feature extraction and classification.

5. According to a multi-view brain and myoelectric fusion decoding method for action classification described in claim 1, the step 4: graph representation based on normalized permutation mutual information, specifically: (1) Node representation: Each channel (EEG and sEMG signal) is a node in the graph. There are C channels in total, each channel has a feature vector Vi, and the feature vector of the i-th channel is Vi∈R d , where d is the dimension of channel features, V={V1,V2,...,V C } Where each Vi represents the feature of the i-th channel, and we use the original time signal as the feature vector; (2) Edge representation and adjacency matrix construction: The edges in the graph represent the correlation between channels, which is calculated by Standardized Permutation Mutual Information (SPMI). Assume there are two channels V i and V j , the similarity between them can be expressed as: A ij =SPMI(V i ,V j ) in, A ij It is an element in the adjacency matrix, representing the similarity or correlation between the i-th channel and the j-th channel. SPMI measures the statistical dependence between the two channels. The value range of SPMI is [0,1]. The higher the value, the stronger the correlation between the channels. By connecting channels with higher correlation according to the threshold, the connectivity between channels can be captured, thereby constructing the edges of the graph; (3) Standardization of adjacency matrix: In order to eliminate the influence of node degree in the graph, the adjacency matrix needs to be standardized. The standardization of the adjacency matrix can be calculated by the following formula: Where A is the original adjacency matrix and D is the degree matrix, defined as D ii =∑ j A ij , that is, the degree of the i-th channel, by normalizing the adjacency matrix Eliminates node degree differences; (4) Construct the final graph: According to the above calculations, the final graph G can be represented as a graph containing nodes and edges, where: the node set V = {V1, V2, ..., V C }, representing the features of each channel, the adjacency matrix Representing the similarity or relationship between channels, the final graph structure can be expressed as:

6. According to the multi-view brain and electromyography fusion decoding method for action classification described in claim 1, step 5: we propose a 3D CNN model as a feature extractor, which has three 3D convolution stacks, three maximum pooling layers and one fully connected layer. In the first convolution stack, the convolution kernel size is 6*3*3, and the pooling layer size is 1*2*2. In the other two convolution stacks, the size of the convolution layer and the pooling layer is 1*1*1. All convolution layers use the RELU activation function, and Batch Normalization is used to increase the generalization ability of the model, suppress model overfitting, and improve the training speed of the model. In order to better extract key features, this method introduces a convolutional block attention module (CBAM) into the model. CBAM dynamically adjusts the importance of each channel and spatial area by applying different attention to the channel and spatial dimensions of the data, highlighting the learning of key information. This attention mechanism can enhance feature selection and improve the accuracy and robustness of action classification.

7. According to the multi-view EEG fusion decoding method for action classification according to claim 1, step 6: the EEG-sEMG graph structure obtained in step 4 is used, and GCN captures spatial domain features by aggregating information of adjacent nodes. The basic format of graph convolution is given by the following equation: By stacking multiple graph convolution layers, higher-level features can be extracted layer by layer. Assuming there are L layers of graph convolution, the final feature representation is: in, H l Represents the node feature matrix of the lth layer, W l The weight matrix of this layer, is the normalized adjacency matrix, and σ is the activation function. The output of each layer is the transformation result of the features of the previous layer, through the adjacency matrix and the weight matrix W l Propagation features, by stacking multiple graph convolution layers, GCN can capture higher-level features layer by layer, thereby improving the expressive power of the model. GCN can be regarded as a form of Laplace smoothing, and deeper layers will lead to over-smoothing, so the number of layers should not be too deep. Two GCN layers are used in the model to balance the depth of feature extraction and avoid the risk of over-smoothing, thereby making full use of the advantages of graph convolutional networks to extract efficient and discriminative node features. On the basis of the second layer of graph convolution, a graph attention layer is added to calculate the attention weights between nodes and neighbors, so as to more accurately aggregate the information of neighboring nodes. Formula for graph attention layer: Where: N(i) represents the set of neighbor nodes of node i, α ij is the attention weight of node i to neighbor node j, W l The weight matrix of this layer, σ is the activation function. Graph convolution aggregates node information through the adjacency matrix, and the graph attention mechanism calculates the attention weight according to the characteristics of different neighbor nodes, making information aggregation more flexible and dynamic. By adding the graph attention layer, the model can adaptively adjust the relationship weights between nodes, thereby improving representation ability and classification performance.

8. The multi-view brain and myoelectricity fusion decoding method for action classification according to claim 1, characterized in that: Step 7: After obtaining the time-frequency features and the graph representation features, these feature matrices are serially operated to perform multi-view feature fusion. The serial operation is defined as follows: X=X S ||X G Among them, X G 、X S They represent graph representation features and time-frequency representation features respectively, || is the connection operation. Finally, we feed the fused features into a 256-dimensional fully connected layer. After the SoftMax activation layer, the output results are divided into 3 action categories.

Images