Electroencephalogram emotion recognition method based on hierarchical multi-scale map neural network

Abstract

The invention discloses an electroencephalogram emotion recognition method based on a hierarchical multi-scale map neural network, and belongs to the technical field of emotion calculation and bio-electricity signal processing. According to the method, structural innovation is carried out on the basis of a traditional graph neural network, a multi-scale graph network model with layer-by-layer feature enhancement from local to global is constructed, and the core lies in optimization and improvement of graph structure modeling, a feature fusion mechanism and global dependency capture capability. According to the model, local graph convolution is combined with a random walk normalized matrix, and inter-electrode spatial dependence modeling is enhanced; a virtual brain region center and an SE channel attention mechanism are introduced into the mesoscale layer, and cross-brain region emotional features are effectively extracted; a simplified Transform structure is adopted in a global layer to capture a long-distance cross-time step feature relationship, and finally, the feature expression capability and robustness are improved in combination with a diffusion generation network. According to the method, the model weight is kept, meanwhile, high-precision modeling of the complex emotion mode of the EEG signal is achieved, the accuracy, stability and cross-subject generalization ability of emotion recognition are remarkably improved, and the novel efficient emotion recognition method suitable for the multi-source physiological signal scene is provided.

Description

Technical Field

[0001] The present invention relates to the technical field of emotion recognition and graph neural network modeling, and in particular to an EEG emotion recognition method based on a hierarchical multi-scale graph neural network. Background Art

[0002] With the rapid development of artificial intelligence and human-computer interaction technologies, emotion recognition, as an important means of understanding and responding to human emotions, has shown great potential in application scenarios such as health monitoring, intelligent customer service, driver assistance, and virtual reality. Electroencephalography (EEG), a non-invasive, real-time neurophysiological signal that can reflect the characteristics of brain activity in individuals under different emotional states, has been widely used in emotion recognition research. Existing emotion recognition methods are mainly divided into traditional machine learning methods and deep learning methods. Traditional methods such as support vector machines (SVM) and K-nearest neighbor (KNN) rely on manually designed features and have poor adaptability. Deep learning methods such as convolutional neural networks (CNN), recurrent neural networks (RNN), and Transformer architectures have improved automatic feature extraction capabilities. In recent years, graph neural networks (GNNs) have gradually become a research hotspot in EEG emotion recognition due to their ability to model the spatial structure and functional connectivity between EEG channels.

[0003] Although graph neural networks have made some progress in EEG signal modeling, they still face many challenges. First, existing methods often use statically constructed adjacency matrices, which make it difficult to reflect the dynamic changes in interactions between individual brain regions, limiting the model's adaptability. Second, with the superposition of graph convolutional layers, node features tend to converge, resulting in oversmoothing, which weakens the model's discriminative ability. Furthermore, most models only extract features within local or medium-scale ranges and lack effective modeling of long-distance dependencies across time steps and between brain regions. Furthermore, EEG signals are susceptible to noise such as eye movements, electromyography, and environmental electromagnetic interference. Existing methods often use deterministic modeling, which makes it difficult to capture the uncertainty and complexity in the data, resulting in insufficient model robustness and affecting the accuracy and stability of emotion recognition.

[0004] Therefore, there is an urgent need for an EEG emotion recognition method that can extract multi-scale features, model local-to-global brain region interactions, and balance data uncertainty processing and robustness to improve the system's accuracy, generalization, and practical adaptability. This paper proposes a hierarchical multi-scale graph neural network EEG emotion recognition method in this context, aiming to address these issues and enhance the model's recognition capabilities in complex emotional states. Summary of the Invention

[0005] The present invention addresses the problems of weak modeling capabilities, insufficient cross-brain region dependency modeling, and poor robustness in EEG emotion recognition tasks, and proposes an EEG emotion recognition method based on a hierarchical multi-scale graph neural network. This method improves the model's feature expression capabilities at different time steps and between brain regions by designing a local graph convolution structure, a meso-level aggregation module, a global Transformer modeling structure, and a diffusion generation network, thereby enhancing the model's robustness in recognizing complex emotional states and improving the model's generalization capabilities across different individuals and experimental environments. The present invention can be widely used in scenarios such as affective computing, human-computer interaction, and intelligent health monitoring, and is particularly suitable for high-precision, high-stability EEG emotion recognition systems.

[0006] The EEG emotion recognition method proposed by the present invention comprises the following steps:

[0007] Step (1): EEG data acquisition and preprocessing;

[0008] Step (2): Construct a hierarchical multi-scale graph neural network model;

[0009] Step (3): Model training;

[0010] Step (4): Emotion recognition and classification.

[0011] The local graph convolution module is used to model the channel-level adjacency structure characteristics of EEG signals. Its core idea is to combine the Laplacian matrix and the random walk matrix to perform two-stage graph convolution processing to improve the spatial dependency modeling ability between nodes.

[0012] First, the input EEG data is transformed into a two-dimensional feature matrix Represented by N, where N represents the number of EEG channels and d is the feature dimension of each channel. The input features are projected into the latent space through the learnable linear transformation matrix W, and we get:

[0013] X′=XW

[0014] Then, by normalizing the Laplace matrix Implement graph convolution operation:

[0015]

[0016] in A is the adjacency matrix and D is the degree matrix.

[0017] Then, the random walk matrix P=D is introduced -1 A acts as a structural diffusion mechanism to further weight feature propagation:

[0018] H (2) =PH (1)

[0019] Finally, the original input, convolution output, and diffusion output are dimensionally concatenated to form a local multi-part feature tensor:

[0020] H local =[X||H (1) ||H (2) ]

[0021] This structure can effectively enhance the expressiveness of inter-node features and improve the model's depth of capturing spatial structural information. It is particularly suitable for modeling scenarios where emotional information has local discharge area characteristics.

[0022] The mesoscale subgraph aggregation module is used to abstract functional brain region-level feature expressions from the local node level, enhancing the model's ability to model mesoscale spatial dependencies. This module primarily includes subgraph partitioning, virtual center node construction, intra-subgraph attention aggregation, and weighted center coordinate calculation.

[0023] Specifically, the entire brain map is divided into k functional subgraphs (e.g., 7 subgraphs), each of which consists of multiple channel nodes. Assume that the input feature is Where B is the batch dimension, N is the number of channels, and d is the feature dimension. The node features in each subgraph are first linearly transformed to obtain the transformed representation:

[0024] H′ i =X i W

[0025] in represents the node features of the i-th subgraph, and W is the shared linear transformation matrix. Next, self-attention calculation is performed between all nodes in each subgraph to obtain the attention weight matrix:

[0026] A i =Softmax(H′ i ·(H′ i ) T )

[0027] Based on the attention matrix A i , the subgraph node features are weighted and aggregated into a virtual central node representation:

[0028]

[0029] At the same time, in order to preserve the spatial information, the three-dimensional coordinates of the nodes are weighted fused with the same weight to generate the spatial position of the central node:

[0030]

[0031] Finally, the central features output by all sub-images are concatenated to form a mesoscale feature tensor:

[0032] H meso =[h center,1 ||h center,2 ||…||h center,k ]

[0033] In order to further improve the expression weight difference of different channels after mesoscale aggregation and enhance the responsiveness of key channels, a channel attention mechanism (Squeeze-and-Excitation Block) is introduced.

[0034] Specifically, let the input feature be Where C is the number of channels and L is the feature length. The SE module first performs global average pooling on each channel:

[0035]

[0036] Get channel description vector Then it is mapped to the channel attention coefficient through two fully connected layers:

[0037] s=σ(W2·δ(W1·z))

[0038] Where δ is the ReLU activation function, σ is the Sigmoid function, W1 and W2 are linear transformation weights. Finally, the original features are scaled according to the channel attention coefficient:

[0039] H′=H·s

[0040] This structure can suppress invalid channels, enhance the response of key brain areas, and improve the sparsity and discriminability of feature expression.

[0041] To model global emotional dependencies across brain regions, a simplified Transformer module was introduced. This module uses a single-layer multi-head self-attention structure to model fully connected information interactions across all nodes, improving the model's ability to perceive long-range dependencies.

[0042] Assume the input feature is Generate query, key, and value vectors through linear transformation:

[0043] Q=XW Q ,K=XW K ,V=XW V

[0044] Calculate attention weights:

[0045]

[0046] The output features are added to the input residual to form the final global expression:

[0047] Xout =X+Attention(Q,K,V)

[0048] This module does not contain positional encoding and feedforward networks to reduce computational complexity and is suitable for global modeling scenarios with graph structures.

[0049] To enhance the model's robustness to EEG signal noise and outliers, a one-dimensional diffusion residual feature enhancement structure is proposed. This structure draws on the design of the U-Net residual network and constructs a feature enhancement path through convolutional layers, nonlinear gating, normalization, and skip connections.

[0050] The input features are first reduced in dimension through a one-dimensional convolution:

[0051] h1=Conv1(x)

[0052] Then through gate activation and normalization processing:

[0053] h2=Norm1(h1),h3=h2·σ(h2)

[0054] Then enter the second layer of convolution and form a residual connection:

[0055] h4=Conv2(h3),x out =x+h4

[0056] This module has the functions of feature enhancement, interference suppression and edge preservation, and is suitable for the reconstruction and stable expression of complex emotional signals.

[0057] Compared with the existing technology, the above technical solution has the following beneficial effects:

[0058] (1) Starting from the perspective of spatial modeling of EEG, the present invention optimizes the structure of the emotion recognition model in a hierarchical manner and proposes an emotion recognition structure with multi-scale graph modeling as the core. By introducing the local graph convolution module, the propagation and fusion capabilities of structural information between nodes are enhanced; the mesoscale subgraph aggregation module models the hierarchical structure of functional brain regions through virtual central nodes, improving the model's ability to express regional features; the simplified Transformer module constructs global dependencies across brain regions, and the diffusion enhancement module further enhances the model's robustness to weak signals and noise. The synergistic effect of multiple modules effectively improves the model's overall recognition performance and structural understanding of emotional states.

[0059] (2) The multi-scale graph neural network structure proposed in this invention has good scalability and portability, can adapt to different types of EEG channel layouts (such as 32 channels, 62 channels), supports a variety of public emotion datasets such as SEED and SEED-IV, and achieves accurate recognition while ensuring a streamlined structure. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 This is an overall flow chart of an EEG emotion recognition method based on a hierarchical multi-scale graph neural network provided in an embodiment of the present application;

[0061] Figure 2 is the EEG signal dataset used in the embodiments of this application;

[0062] Figure 3 This is a schematic diagram of the local graph convolution module in this application;

[0063] Figure 4 is a schematic diagram of the mesoscale subgraph aggregation module in this application;

[0064] Figure 5 This is a simplified diagram of the Transformer module of this application;

[0065] Figure 6 This is a schematic diagram of the diffusion enhancement module in this application. DETAILED DESCRIPTION

[0066] The following is a clear and complete description of the technical solutions in the embodiments of the present invention, in conjunction with the accompanying drawings. The examples provided are only some of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments without creative effort are within the scope of protection of the present invention.

[0067] See also Figure 1 An embodiment of the present invention provides an EEG emotion recognition method based on a hierarchical multi-scale graph neural network, which specifically includes the following steps:

[0068] Step (1): Dataset and graph structure construction.

[0069] First, we used publicly available EEG emotion recognition datasets, such as the SEED and SEED-IV datasets, to collect multi-channel EEG signals encompassing a variety of emotional states. In the original datasets, each subject had 62 EEG channels, sampled at a 200Hz frequency, and the experimental stimuli included a variety of positive, negative, and neutral emotion-inducing materials.

[0070] To improve the quality of the original data, bandpass filtering (0.5 Hz to 50 Hz) was used to remove high-frequency noise and DC drift, and the amplitude of each EEG signal was normalized using minimum-maximum normalization.

[0071] Then, according to the electrode spatial layout (10-20 system standard), the coordinate information of each channel in three-dimensional space is extracted to construct the distance matrix between channels. Calculate the adjacency matrix based on distance relationships:

[0072]

[0073] where c i ,c j is the channel space coordinate, and σ is the distance scale factor. The constructed adjacency matrix is used for subsequent graph neural network feature propagation. The normalized Laplacian matrix L and random walk matrix P are further calculated to provide the structural foundation for the subsequent local graph convolution and diffusion propagation modules.

[0074] Step (2): Build a multi-scale graph neural network model.

[0075] The emotion recognition model of the present invention includes a local graph convolution module, a mid-scale brain region subgraph aggregation module, a channel attention module, a global Transformer structure and a diffusion residual enhancement module.

[0076] In the local graph convolution module, each EEG sample input is a three-dimensional tensor Where d is the feature dimension of each channel. First, transform X through linear projection, and then use the Laplacian matrix L to propagate the features:

[0077] H (1) =σ(LXW1)

[0078] Then the random walk matrix P is introduced for secondary propagation:

[0079] H (2) =σ(PH (1) W2)

[0080] The original input and the results of the two propagations are spliced into local multi-level fusion features as the basis for subsequent modeling.

[0081] The data processed by the local graph convolution module is input into the mesoscale brain region subgraph aggregation module. The mesoscale brain region subgraph aggregation module divides the 62 channels into 7 subgraphs (such as the frontal lobe, parietal lobe, and occipital lobe) based on brain region function. Each subgraph has a virtual central node. The nodes within the subgraph are aggregated using the following attention mechanism:

[0082]

[0083] The output subgraph center node is represented as:

[0084]

[0085] At the same time, the spatial positions of the nodes are also fused with the same weights to form the subgraph center coordinates. The Squeeze-and-Excitation module is introduced to generate weight coefficients for each channel feature after global average pooling. The original channel features are scaled according to s to improve the key channel response:

[0086] s=σ(W2·δ(W1·GAP(H)))

[0087] The fused multi-scale features are input into the self-attention module and calculated as follows:

[0088]

[0089] This module omits positional encoding and retains only the multi-head attention backbone structure to model long-range dependencies between brain regions. A one-dimensional diffusion residual structure is introduced at the end of the model to perform nonlinear reconstruction of full-image features. The specific formula is as follows:

[0090] y=x+Conv2(δ(Norm1(Conv1(x))))

[0091] Step (3): Train the multi-scale graph neural network model.

[0092] The model was trained on a Windows 11 operating system, an NVIDIA RTX 4090 graphics card, PyCharm development environment, and the PyTorch deep learning framework. The initial learning rate was set to 0.01, and AdamW was used as the optimizer, with a cosine decay strategy for adjusting the learning rate. Training was performed for 300 iterations with a batch size of 64. No pretrained weights were used for any of the networks during training.

[0093] At the same time, in order to evaluate the performance of the algorithm, this paper selects two experimental settings, subject-dependent and subject-independent, as the evaluation methods of the model to measure the accuracy and generalization ability of the model in the same-subject and cross-subject emotion recognition tasks.

[0094] A subjective-dependent experiment involves training and testing each subject separately. The model only needs to model the mapping relationship between EEG and emotion labels within a single subject, reflecting the model's upper limit of recognition in an ideal intra-subject environment. A subjective-independent experiment involves training the model on one group of subjects and testing it on another, unseen group of subjects to assess the model's cross-subject generalization capabilities.

[0095] The performance indicators used in this invention are average recognition accuracy (Accuracy) and standard deviation (StandardDeviation), which are specifically defined as follows:

[0096]

[0097] TP represents the number of samples that are actually positive and predicted correctly, TN represents the number of samples that are actually negative and predicted correctly, FP represents the number of samples that are incorrectly predicted as positive, and FN represents the number of samples that are incorrectly predicted as negative.

[0098]

[0099] in is the average accuracy of all test rounds, Acc i is the accuracy rate obtained in the i-th experiment.

[0100] The proposed multi-scale graph neural network model was then trained using the aforementioned training parameters. To evaluate the model's performance, experiments were conducted on the SEED and SEED-IV datasets. Finally, based on the accuracy and stability of the training results, the model structure and key module parameters were further optimized, ultimately achieving an optimal model structure suitable for widespread application in various EEG emotion recognition tasks.

[0101] Step (4) emotion recognition and classification.

[0102] After training, the model is used for emotion recognition tasks. The model input is a normalized and structured multi-channel EEG signal, and the output is the corresponding emotion category label, specifically a binary classification (positive / negative), a three-category classification (positive / neutral / negative), or a four-category classification (happy / calm / angry / sad).

[0103] The model output is passed through the MLP layer to complete the emotion prediction. Let the final extracted global feature be tensor After the fully connected layer, it is converted into emotion probability distribution:

[0104] y = Softmax(W2·δ(W1·X))

[0105] Where W1 and W2 are fully connected weights, δ is the LeakyReLU activation function, and Softmax outputs the probability of the final emotion class.

[0106] The model recognition results include: the emotion label; the confidence level (Softmax probability) of each emotion; and optional auxiliary information, such as brain region response intensity maps, time window emotion curves, etc.

[0107] The final output can be used for terminal display (emotional feedback graphs), intelligent interaction (such as human-computer dialogue system adjustment), or medical assistance (such as emotional disorder monitoring systems). In addition, the model structure supports online fine-tuning and individual adaptation, which can further optimize the recognition effect for specific users or scenarios.

Claims

1. A method for EEG emotion recognition based on a hierarchical multi-scale graph neural network, characterized in that: The following steps are involved: Step (1) collecting and preprocessing EEG signals to construct a graph structure input containing spatial coordinates and adjacency relationships; Step (2) designing a multi-scale graph neural network model including a local graph convolution layer, a mid-scale subgraph aggregation layer, a global Transformer layer and a residual diffusion network; Step (3) Use SEED and SEED-IV public datasets to train the model and optimize the emotion recognition accuracy; Step (4) uses a multi-layer perceptron to classify the fused features and output the final emotion recognition results.

2. The method for EEG emotion recognition based on a hierarchical multi-scale graph neural network according to claim 1, characterized in that: The EEG structure input in step (1) is composed of the following processes: feature construction based on 62-channel EEG signals, artifact removal using bandpass filtering, generation of three-dimensional position data based on predefined electrode spatial coordinates, and construction of an adjacency matrix using the inverse square of the spatial distance between nodes, which is further normalized into a Laplace matrix and a random walk matrix.

3. The method for EEG emotion recognition based on a hierarchical multi-scale graph neural network according to claim 1, characterized in that: The local graph convolution layer in step (2) includes: based on the original feature input, using the normalized Laplacian matrix to perform graph convolution operation, and propagating the features through the random walk normalized matrix, combining the LeakyReLU activation function and the linear mapping layer to improve the local feature expression ability, and integrating the original, convolutional and propagated local features through multi-scale splicing.

4. The method for EEG emotion recognition based on a hierarchical multi-scale graph neural network according to claim 1, characterized in that: The mesoscale subgraph aggregation layer in step (2) includes: dividing the nodes into several mesoscale subgraphs according to the functional division of brain regions, and realizing feature aggregation within each subgraph by generating virtual central nodes; at the same time, introducing the Squeeze-and-Excitation module to dynamically weight the feature channels to enhance the expressive ability of emotional signals in key brain regions.

5. The method for EEG emotion recognition based on a hierarchical multi-scale graph neural network according to claim 1, characterized in that: The global Transformer layer in step (2) includes: constructing a simplified Transformer architecture that removes position encoding and multi-layer stacking structure, retaining the multi-head attention mechanism, modeling the long-distance dependencies between nodes across time steps and brain regions through the feature similarity matrix, and improving the model's ability to model global emotional patterns.

6. The method for EEG emotion recognition based on a hierarchical multi-scale graph neural network according to claim 1, characterized in that: The model training described in step (3) is based on the PyTorch deep learning framework, and the hyperparameters include: the training platform is Python 3.8, CUDA 11.3 and PyTorch 1.12.0, and the hardware is accelerated by NVIDIA GeForce RTX 4090 graphics card; the training parameters include 300 training rounds, an initial learning rate of 0.01, a batch size of 64, a dropout rate of 0.4, an L2 regularization term weight of 5e-4, and optimization using the cross entropy loss function. A random seed of 222 is used to ensure the reproducibility of training.

Images