Electroencephalogram emotion recognition method based on micro-state brain function network

Abstract

The invention relates to an electroencephalogram emotion recognition method based on a micro-state brain function network. The method comprises the following steps: firstly, preprocessing an EEG signal; based on a GFP peak value and a K-means clustering method, a group-level micro-state template is obtained, micro-state analysis is carried out, and emotion-related micro-state selection is carried out; in the screened micro-state time period, constructing a brain function network by utilizing PLV; dividing the electroencephalogram signals into a plurality of frequency bands, extracting differential entropy and power spectral density features, taking the differential entropy and power spectral density features as node features, and forming graph data by combining a PLV matrix; and finally, a graph attention network model is adopted, a multi-head attention mechanism is utilized to adaptively learn mutual influence of brain intervals, and emotion recognition is completed. According to the invention, the influence of time window length setting on identification precision can be effectively avoided; the correspondence and timeliness between the brain interval function connection and the emotional state are remarkably improved, and accurate modeling of the electroencephalogram signals under the high time resolution is achieved.

Description

Technical Field

[0001] This invention belongs to the field of signal processing technology research, and more specifically, it relates to an EEG emotion recognition method based on microstate brain functional networks, which can be used to classify and identify an individual's emotional state. Background Technology

[0002] Emotions are a crucial component of human psychological activity, significantly influencing an individual's cognitive abilities, behavioral decision-making, social interactions, and physical health. Fluctuations in emotions reflect an individual's mental health; emotional stability helps maintain psychological balance, while frequent or intense mood swings may be early warning signs of mental disorders such as anxiety and depression. In social interactions, emotional stability directly impacts the quality of interpersonal relationships and the effectiveness of communication. Research indicates that emotional states play a vital regulatory role in decision-making, and emotional fluctuations can lead to irrational, impulsive, or even avoidant behavioral choices. Therefore, studying the dynamic processes of emotional change is essential for understanding the mechanisms of individual mental health and social behavior.

[0003] In emotion recognition tasks, compared to non-physiological signals such as facial expressions, tone of voice, and body posture, which are easily controlled by subjective factors, physiological signals such as electroencephalography (EEG), electromyography (EMG), and electrocardiography (ECG) are more reliable sources for identifying genuine emotional changes due to their objectivity and stability. Among them, EEG signals have advantages such as being non-invasive, portable, easy to operate, and having high temporal resolution, and can capture the characteristics of brain neural activity during emotional changes in real time, making it a key data source for current research on emotion recognition.

[0004] However, identifying stable states and coordinated change patterns between brain regions in EEG signals during rapid emotional shifts remains a challenge. Existing research primarily employs a sliding time window method to construct brain functional networks based on functional connections between brain regions, and then uses complex network theory to analyze the network structure and dynamic change patterns. This method can characterize the temporal changes in brain network connectivity patterns during emotional state transitions to some extent, but there is a trade-off in the choice of time scale: while a too-short time window can improve temporal resolution, it may lead to increased fluctuations and noise amplification during feature extraction; conversely, a too-long time window may obscure key neural activity features in short-term emotional changes, blurring the boundaries of emotional states and affecting recognition accuracy. Furthermore, the sliding window construction method often forcibly divides time series into fixed intervals, lacking adaptability to the natural fluctuation rhythms of EEG signals, thus limiting its ability to characterize rapid and nonlinear emotional dynamics.

[0005] The patent publication number "CN 117609863 A" discloses a "Long-Term EEG Emotion Recognition Method Based on EEG Microstates," which includes: establishing a feature extractor, comprising a multi-scale feature acquisition module and a Transformer encoder; the Transformer encoder comprising a multi-head attention mechanism module and a feedforward network module; the multi-head attention mechanism module comprising a local self-attention module and a global self-attention module; setting training set data and test set data as the source domain and target domain respectively, and training the feature extractor using a domain adversarial network; inputting the long-term EEG signals of test set subjects into the trained feature extractor, extracting key features, and inputting them into an emotion classifier for emotion recognition. Its core is to process microstate sequences through a multi-scale feature extractor (including a Transformer encoder) and to use domain adversarial training (Domain Adversarial Network) to solve the problem of cross-subject differences. This method directly inputs the microstate sequence into the Transformer module, uses a local-global attention mechanism to extract temporal features, and finally outputs the recognition result through an emotion classifier. The existing technical problem is: The proposed solution only treats EEG microstates as static feature inputs and directly serializes them into the Transformer encoder for processing. It does not combine the functional division of microstate fragments with dynamic and continuous features for structured modeling. It still cannot get rid of the limitation of relying on a fixed structure model window and cannot fully capture the transient brain activity response under rapid emotional changes. Therefore, it cannot completely replace the traditional sliding time window strategy.

[0006] The lack of differentiation between microstates and emotions, and the uniform treatment of all microstates, cannot ensure that subsequent analysis focuses on microstates closely related to emotions, making it difficult to extract specific neural patterns in the process of emotional change.

[0007] Self-attention simulation of spatial relationships relying on Transformers lacks the structural support of real brain region connections, ignores the physiological topology of neural networks, and is difficult to form a brain network basis that has both physiological interpretability and stability.

[0008] Using only temporal features and ignoring frequency domain information leads to a disconnect between interbrain connectivity and neural frequency features, resulting in technical problems such as weak multimodal feature fusion capabilities and limited recognition accuracy. Summary of the Invention

[0009] This invention proposes an EEG emotion recognition method based on microstate brain functional networks to solve the problem that existing technologies cannot effectively capture the instantaneous interaction features of brain regions during rapid emotional changes.

[0010] To achieve the above objectives, the technical solution of the present invention is: a brainwave emotion recognition method based on microstate brain functional networks, comprising the following steps: Step 1: EEG data preprocessing: The raw EEG data is preprocessed to obtain preprocessed EEG signals; Step 2, GFP peak detection: Calculate the global field power (GFP) of the preprocessed EEG signal at each time point; select the EEG topography map corresponding to the peak point of the GFP curve and input it into the clustering model; use the K-means algorithm to cluster its spatial distribution to obtain the microstate template at the population level. Step 3: Microstate analysis: Determine the optimal number of microstates using Global Explained Variance (GEV) and Cross-Validation (CV) metrics. Backfit the optimal microstate template back to the preprocessed EEG data to obtain the microstate time series. Extract microstate time features from the microstate time series. Step 4: Selection of emotion-related microstates: Screening out microstates that are significantly related to emotional states to provide neurophysiological basis for emotion recognition; Step 5: Constructing a brain functional network based on microstates: For the selected microstates, calculate the phase lock values ​​between EEG channels within their corresponding time periods, and construct a PLV connection matrix based on the microstates as the structural basis of the brain functional network reflecting the strength of functional connections between brain regions; perform sparsification processing on the PLV connection matrix to construct a sparse undirected weighted graph. Step 6: Node Feature Extraction: Extract the differential entropy (DE) and power spectral density (PSD) of each channel in the five frequency bands δ, θ, α, β and γ as frequency domain features; Step 7: Construct graph data: Using a sparse PLV matrix as edge weights and frequency domain features as node features, construct the graph data structure G=(V,E,X); Step 8, Emotion Recognition: Input the graph data structure into the Graph Attention Network (GAT), and use the attention mechanism to achieve adaptive feature aggregation and information weighting in the graph structure; after multiple GAT layers, use a fully connected layer and a Softmax classifier to output the final emotion category, and use the cross-entropy loss function as the objective function for training.

[0011] Furthermore, the preprocessing steps in step one above include, in sequence: downsampling, bandpass filtering, removal of power frequency noise, eye movement artifacts, bad lead interpolation, and whole-brain average reference.

[0012] Furthermore, the clustering in step two above includes two steps: the first clustering is performed at the individual subject level, and the second clustering is performed across the subject level.

[0013] Furthermore, in step five above, when sparsifying the PLV connection matrix, a threshold is set to retain the edge weights of the top 30% of the connection strengths, while the rest are set to 0.

[0014] Furthermore, in step six above, the EEG signal is divided into five frequency bands: δ (1–4 Hz), θ (4–8 Hz), α (8–14 Hz), β (14–31 Hz), and γ (31–50 Hz).

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) A method for segmenting EEG signals based on EEG microstate sequences is proposed. Microstate templates are generated based on global field power (GFP) peak clustering, and the natural physiological rhythm-driven segmentation of EEG signals is achieved through inverse fitting, completely replacing the traditional sliding time window strategy and avoiding the masking of transient emotional responses by a fixed window length. This method dynamically segments the signal according to the natural changes in EEG activity, more accurately extracting key time segments corresponding to emotional changes, and effectively avoiding the impact of time window length settings on recognition accuracy. (2) In the statistical test-driven screening process proposed in this invention, three types of dynamic features—duration, frequency, and coverage—are extracted for microstates. These features are then combined with a self-rating emotion scale (valence / arousal) to perform group difference tests, thereby screening out microstates that are significantly associated with emotions. This ensures that subsequent analysis focuses on microstates closely related to emotions. Based on the statistical test screening of microstates related to emotion changes, and by selecting the corresponding microstate time periods to construct brain functional networks, the correspondence and timeliness between brain region functional connectivity and emotional states are significantly improved, overcoming the problem of insufficient modeling of short-term brain region collaborative activities in traditional methods.

[0016] (3) Based on the selected microstate time segments, this invention calculates the phase lock value (PLV) between brain channels, constructs a functional connectivity matrix reflecting the synchronicity of brain regions, and eliminates noise interference by sparsifying the first 30% of strong connections, forming a brain network foundation that combines physiological interpretability and stability. Simultaneously, this invention not only utilizes temporal features but also integrates frequency domain information, extracting differential entropy (DE) and power spectral density (PSD) as node features in the δ, θ, α, β, and γ frequency bands, constructing a brain functional map data structure that integrates spatial connectivity and frequency domain dynamic features. By selecting microstate-related time segments, connection patterns closely related to emotional changes are selectively retained, reducing interference from irrelevant redundant information and improving feature quality. Furthermore, GAT is used to process the constructed graph data, adaptively allocating connection weights between nodes through an attention mechanism, emphasizing key nodes and functional connections that contribute significantly to the emotion recognition task, and effectively mining the interactive relationship features between brain regions.

[0017] (4) The brain functional network provided by this invention, combined with the characteristic of EEG microstates reflecting the instantaneous stability of the brain's overall activity, constructs a functional connectivity network within the naturally stable segments defined by the microstates, thereby achieving accurate modeling of EEG signals at high temporal resolution. Compared with traditional methods based on fixed sliding time windows, microstate analysis can naturally divide time segments according to the dynamic changes of neural activity itself, fully preserving the rapid state transition characteristics and transient synchronization patterns in EEG signals, and solving the problems of temporal granularity loss and information averaging caused by fixed time windows. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the implementation of an EEG emotion recognition method based on microstate brain functional networks. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the present invention clearer, the specific embodiments of the present invention will be further described in detail below with reference to the examples and accompanying drawings.

[0020] A brainwave-based emotion recognition method based on microstate brain functional networks. For example... Figure 1 As shown, the method includes the following steps: Step 1: EEG data preprocessing: The raw EEG data of the subjects were preprocessed to obtain preprocessed EEG signals. The preprocessing steps used in this example are as follows: downsampling, bandpass filtering, power line noise removal, eye movement artifact removal, bad lead interpolation, and whole-brain average reference. These steps can effectively improve the quality of EEG data signals. Step 2, GFP peak detection: The GFP of the preprocessed EEG signal at each time point was calculated to quantify the overall activity intensity of the potential distribution; Topological topography extraction: EEG topography is extracted from the peak points of the GFP curve, that is, the topological topography with high signal-to-noise ratio is used as the time segment with the most stable signal features to input the clustering model.

[0021] K-Means clustering: Clustering is performed on the selected topology. The clustering includes two steps: the first clustering is performed at the individual subject level, and the second clustering is performed across the subject level to obtain a microstate template at the population level. All of the above clustering uses the K-Means clustering algorithm.

[0022] Step 3: Microstate analysis: 3.1 After obtaining the microstate template at the group level, the optimal number of microstates is determined using the Global Explained Variance (GEV) and Cross-Validation (CV) metrics. In this example, the optimal number of microstates is determined to be 4, which means that this embodiment has determined four types of microstate templates with the optimal number of microstates.

[0023] 3.2 Based on the obtained four types of microstate templates, the final microstate templates are backfitted back to the preprocessed EEG data. Each time point is assigned to one type of microstate. Noise and invalid states are removed by time smoothing and global map dissimilarity (GMD) to obtain a microstate time series covering the whole brain activity. In this embodiment, when performing time smoothing on the microstate segments, high-frequency jitter segments shorter than 30 ms are removed.

[0024] 3.3 Extract microstate time features from the microstate time series obtained by inverse fitting, including duration, occurrence frequency and coverage, to quantify the time proportion and activation intensity of different microstates.

[0025] Step 4: Selecting Emotion-Related Microstates: This example categorizes emotion-induced EEG data into low-level and high-level emotion groups based on self-assessment levels from 32 participants. The eigenvalues ​​of the three extracted microstate time features are analyzed using independent t-tests or Wilcoxon rank-sum tests to assess significant differences, thus achieving statistical association modeling between emotional state and microstate dynamic features. Microstates significantly correlated with emotional state are selected using a p-value <0.05; in this example, two of these microstates are identified as being emotion-related.

[0026] Step 5: Constructing a brain functional network based on microstates: 5.1 For the two selected emotion-related microstates, the phase lock value between EEG channels is calculated within their corresponding time periods. A symmetrical phase lock value (PLV) connectivity matrix is ​​constructed based on the microstates rather than a fixed time window. In this step, the EEG signals corresponding to the selected emotion-related microstates for the corresponding time periods are extracted, and the phase lock value (PLV) between all channel pairs is calculated. The phase lock value (PLV) can measure the degree of phase synchronization between different channels, thereby generating a symmetrical phase lock value (PLV) connectivity matrix. This matrix serves as the basis for the brain functional network structure reflecting the strength of functional connectivity between brain regions, and more accurately captures the synchronization patterns between brain regions at the moment of emotional change. 5.2 The PLV matrix is ​​sparsified to construct a sparse PLV matrix. In this step, a threshold is set to retain the edge weights of the top 30% of the connections, and the rest are set to 0, resulting in an undirected weighted graph structure after removing weak connections, which is the sparse PLV matrix. This improves the stability of the network topology, reduces noise interference, and preserves the brain region connection patterns that are most discriminative for the recognition task.

[0027] Step 6: Node Feature Extraction The differential entropy (DE) and power spectral density (PSD) of each channel of the EEG signal in five frequency bands are extracted as frequency domain features. In this embodiment, the EEG signal is divided into five commonly used frequency bands: δ (1–4 Hz), θ (4–8 Hz), α (8–14 Hz), β (14–31 Hz), and γ (31–50 Hz). The differential entropy (DE) and power spectral density (PSD) features are extracted in these frequency bands respectively to capture neural frequency band patterns related to emotional changes. DE can measure the complexity of the signal, while PSD reflects the energy distribution. These frequency domain features will be used as feature inputs for each channel node in the figure.

[0028] Step 7: Construct graph data: Combining the sparse PLV matrix and frequency domain features generated in step six, and using the sparse PLV matrix as edge weights and the frequency domain features as node features, a graph data structure G=(V,E,X) is constructed. The node set V corresponds to the EEG channel, the edge set E is the non-zero PLV connection pair, and the node feature matrix X is the combination of DE and PSD features of the channel in five frequency bands. This forms graph data that can simultaneously reflect spatial connectivity and frequency domain features, and integrates the inter-brain connectivity relationship with the local feature information of each node.

[0029] Step 8: Emotion Recognition 8.1 Inputting graph data structures into Graph Attention Networks (GAT) leverages attention mechanisms to achieve adaptive feature aggregation and weighted information transfer within the graph data structure. Graph Attention Networks (GAT) dynamically learn the weights between each node and its neighbors through attention mechanisms, thereby strengthening highly correlated connections and suppressing weakly correlated connections. Furthermore, by aggregating various relational expressions through multi-head attention mechanisms, GAT enhances the model's adaptability to complex brain network structures and strengthens its ability to perceive key pathways and important connections.

[0030] 8.2 After the graph attention network GAT, a fully connected layer and a softmax classifier are used to predict the emotional state. During the training process, the cross-entropy loss function is used as the objective function to minimize the classification error, thereby improving the model's generalization ability and robustness, and improving the accuracy of emotion recognition.

[0031] The method proposed in this invention is as follows: First, the raw EEG signals are analyzed using microstate analysis to generate a microstate time series covering the entire brain activity, achieving dynamic segmentation based on the natural changes in EEG activity. Second, the dynamic features of the microstate sequence (including duration, frequency of occurrence, and coverage) are further extracted. Combined with self-emotional rating results, statistical tests are used to screen out microstates (MS3 and MS4) that are significantly related to emotional states, providing a neurophysiological basis for emotion recognition. For the screened emotion-related microstates, the phase lock value (PLV) between EEG channels is calculated within their corresponding time periods. A functional connectivity network is constructed based on the microstates, which can more accurately capture the synchronization patterns of brain regions at the moment of emotional change. Finally, differential entropy (DE) features are extracted as node attributes, and the PLV matrix is ​​thresholded to determine edge weights, constructing a graph data structure. The connectivity relationships between brain regions and the local feature information of each node are fused, and the fused graph data is then input into a graph attention network. The attention mechanism is used to achieve adaptive feature aggregation and weighted information transfer in the graph structure, enhancing the model's ability to perceive key channels and important connections.

[0032] The effectiveness of this invention is demonstrated by comparison with existing technologies. These existing technologies include: Du et al. used a 1-second non-overlapping time window to extract DE features and proposed an Attention-based Domain Discriminant Network (ATDD-LSTM) for emotion recognition. Gao et al. used a 1-second non-overlapping time window to extract two different features based on the time domain (Hjorth, DE, sample entropy) and the frequency domain (Power Spectral Density (PSD)), and used CNN + SVM to classify emotions. Li et al. used a 1-second non-overlapping time window to construct a Frontal Lobe Dual-Duel Deep Q-Network (FLD3QN) for emotion recognition. Yin et al. used a 6-second non-overlapping time window to extract DE features and fused Graph Convolutional Neural Network (GCNN) and Long Short-Term Memory Neural Network (LSTM) for emotion recognition. Yao et al. used a 6-second non-overlapping time window to construct an Expanded Bottleneck Convolutional Neural Network (DBCN) for emotion recognition. Dhara et al. used a 2-second time window with a 1-second sliding step and proposed a deep learning method based on fuzzy ensemble for emotion recognition. Wu et al. used a 3-second time window with a 1-second sliding step to extract DE and PSD from each frequency band as complementary frequency domain features. They then constructed a dynamic brain functional network to analyze spatial connectivity patterns related to emotion representation for emotion recognition. Table 1 shows a comparison with existing methods for feature extraction using sliding time windows and for constructing brain functional networks. Table 1: Comparison with existing methods author Publication time method Valence Arousal Du et al. 2020 1-second time window 90.91 Gao et al. 2022 1-second time window 80.52 Li et al. 2023 1-second time window 98.35 98.17 Yin et al. 2021 6-second time window 90.45 90.60 Yao et al. 2022 6-second time window 90.93 89.67 Dhara et al. 2024 2-second time window, step size 1 90.84 91.65 Wu et al.

[52] 2024 3-second time window, step size 1 94.92 97.01 Ours 2024 Determining the time window based on microstate 99.26 99.19 As can be seen, traditional methods for constructing brain functional networks using sliding time windows neglect the rapid changes in EEG signals. Due to the highly dynamic nature of EEG signals across different time scales, traditional time-window methods cannot adequately capture the rapid fluctuations and abrupt changes in signals within a short period. This invention supplements the shortcomings of traditional brain functional networks in the time dimension by constructing brain functional networks using microstates, thereby capturing instantaneous neural activity characteristics. This invention demonstrates a significant advantage in emotion recognition accuracy.

[0033] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A brainwave emotion recognition method based on microstate brain functional networks, characterized in that: Includes the following steps: Step 1: EEG data preprocessing: The raw EEG data is preprocessed to obtain preprocessed EEG signals; Step 2, GFP peak detection: Calculate the global field power of the preprocessed EEG signal at each time point; The EEG topography corresponding to the peak point of the GFP curve is selected as input into the clustering model, and the spatial distribution is clustered using the K-means algorithm to obtain the microstate template at the population level. Step 3: Microstate analysis: Determine the optimal number of microstates using global explained variance and cross-validation index, backfit the optimal microstate template back to the preprocessed EEG data to obtain the microstate time series, and extract microstate time features from the microstate time series. Step 4: Selection of emotion-related microstates: Screening out microstates that are significantly related to emotional states to provide neurophysiological basis for emotion recognition; Step 5: Constructing a brain functional network based on microstates: For the selected microstates, calculate the phase lock value between EEG channels within their corresponding time periods, and construct a PLV connection matrix based on the microstates as the structural basis of the brain functional network reflecting the strength of functional connections between brain regions. The PLV connection matrix is ​​then sparsified to construct a sparse undirected weighted graph. Step 6: Node Feature Extraction: Extract the differential entropy (DE) and power spectral density (PSD) of each channel in the five frequency bands δ, θ, α, β and γ as frequency domain features; Step 7: Construct graph data: Using a sparse PLV matrix as edge weights and frequency domain features as node features, construct the graph data structure G=(V,E,X); Step 8, Emotion Recognition: Input the graph data structure into the Graph Attention Network (GAT), and use the attention mechanism to achieve adaptive feature aggregation and information weighting in the graph structure; after multiple GAT layers, use a fully connected layer and a Softmax classifier to output the final emotion category, and use the cross-entropy loss function as the objective function for training.

2. The EEG emotion recognition method based on microstate brain functional networks according to claim 1, characterized in that: The preprocessing steps in step one include, in sequence: downsampling, bandpass filtering, removal of power frequency noise, eye movement artifacts, bad lead interpolation, and whole-brain average reference.

3. The EEG emotion recognition method based on microstate brain functional networks according to claim 2, characterized in that: The clustering in step two includes two steps: the first clustering is performed at the individual subject level, and the second clustering is performed across the subject level.

4. The EEG emotion recognition method based on microstate brain functional networks according to claim 3, characterized in that: In step five, when sparsifying the PLV connection matrix, a threshold is set to retain the edge weights of the top 30% of the connection strengths, and the rest are set to 0.

5. The EEG emotion recognition method based on microstate brain functional networks according to claim 4, characterized in that: In step six, the EEG signal is divided into five frequency bands: δ (1–4 Hz), θ (4–8 Hz), α (8–14 Hz), β (14–31 Hz), and γ (31–50 Hz).

Images