Time-frequency space electroencephalogram emotion recognition method based on three-dimensional space position embedding

Abstract

The invention belongs to the field of electroencephalogram signal processing, and provides a time-frequency space electroencephalogram emotion recognition method based on three-dimensional space position embedding, which comprises the following steps of: firstly, constructing a three-dimensional electrode space position matrix based on an international 10-20 system standard, determining a space adjacency relation between electrodes, and calculating a phase locking value to obtain a functional connection matrix; then, deep feature fusion of an electrode spatial position matrix and a functional connection matrix is realized by adopting a hierarchical cross Transform architecture, the spatial position matrix represents spatial distribution features of a cerebral cortex region, and the functional connection matrix quantifies phase synchronization features of cross-brain region neural oscillation and simulates a brain spatial topological structure; and finally, extracting time, frequency and spatial features of the electroencephalogram signals through combination of a graph attention network and bidirectional long-short-term memory with an attention mechanism for emotion recognition. The method can effectively extract space structure information highly related to the emotional state, and significantly improves the accuracy of emotion recognition.

Description

Technical Field

[0001] This invention belongs to the research field of electroencephalogram (EEG) signal processing technology, specifically involving a time-frequency-space EEG emotion recognition method based on three-dimensional spatial location embedding, which realizes the classification and recognition of individual emotional states. Background Technology

[0002] Emotion recognition, a cutting-edge research area in affective computing, has been widely applied in human-computer interaction, intelligent healthcare, traffic safety monitoring, and intelligent education. Its main goal is to infer an individual's current emotional state by analyzing their physiological or non-physiological signals, thereby enabling artificial intelligence systems to perceive and understand human emotions. It is considered one of the key supporting technologies for achieving natural human-computer interaction. Existing emotion recognition methods can be mainly divided into two categories: those based on non-physiological signals and those based on physiological signals. Non-physiological signals, such as facial expressions, voice tone, and body movements, have advantages such as convenient collection and flexible application; however, they are easily affected by external environmental interference or human manipulation, limiting the accuracy of emotion recognition. In contrast, physiological signals are autonomously generated by the central nervous system and include skin electrical activity, heart rate changes, blood pressure levels, respiratory rhythm, electroencephalography (EEG), and magnetoencephalography (MEG). These signals can more realistically and objectively reflect an individual's emotional state and have significant research value in emotion recognition tasks.

[0003] The brain is the core of the human nervous system, governing a variety of higher functions such as perception, movement, memory, emotion, and thinking. Among these, emotion processing relies on the activity of specific cortical regions, such as the prefrontal cortex and amygdala, which play crucial roles in the generation and regulation of emotions. To study the dynamic changes in the brain under emotional states, electroencephalography (EEG), as a means of recording brain electrical activity, is widely used in emotion recognition tasks due to its advantages such as high temporal resolution, non-invasiveness, and low cost. EEG can reflect the brain's functional state in real time, providing dynamic neural signals closely related to emotional changes. With the continuous development of neuroscience and computing technology, emotion recognition methods based on EEG signals have made continuous progress and are gradually becoming a research hotspot in fields such as affective computing and mental health monitoring.

[0004] However, relying solely on signal features to fully extract emotional information from EEG signals has limitations. In recent years, increasing research has focused on the functional connectivity between different brain regions to reveal deeper patterns of neural activity. The brain consists of multiple functional areas, and the topological connections and activation patterns between these areas reflect their complex interaction mechanisms. Although these functional areas are physically connected, this does not always mean they are functionally highly interconnected. Therefore, studying the connectivity structure of the brain requires moving beyond traditional Euclidean analysis and exploring more complex functional networks. In recent years, modeling EEG channels as graph structures has become a natural and effective approach. In this graph, nodes typically represent the signal characteristics (such as power or entropy) carried by each channel, while edges characterize structural connectivity information such as correlations or causal relationships between channels. Most graph-based EEG modeling methods construct graph structures through the functional connectivity between signals and combine them with graph neural networks to mine potential functional connections and feature representations between brain regions. For example, patent CN202410567568.5 discloses an EEG emotion recognition method and system based on multi-task and attention mechanisms. First, the original signal is processed by frequency band division. After calculating the differential entropy and power spectral density of each frequency band, the features are mapped onto a two-dimensional map. Then, the multi-band features are stacked to obtain a three-dimensional structure. This type of method mainly focuses on modeling the functional connectivity features between channels; however, it does not adequately consider the actual positional relationships of EEG channels in three-dimensional brain space. Another example is patent CN202310264515.1, which discloses an EEG emotion recognition method based on identity pattern verification. It directly constructs EEG channel mapping relationships and channel connection diagrams based on the spatial locations of electrodes specified in the international standard 10-20 system, forming three-dimensional spatial spectral features through the channel mapping relationships. Essentially, this method still obtains three-dimensional spatial spectral features by stacking the spectral features of a two-dimensional mapping matrix. The methods mentioned above all suffer from the problem of missing three-dimensional spatial information. This lack of spatial information makes it difficult for the model to fully capture the topological structure of EEG signals, thus limiting its performance in tasks such as emotion state recognition. Summary of the Invention

[0005] To address the above issues, this invention proposes a time-frequency spatial EEG emotion recognition method based on three-dimensional spatial location embedding, which can solve the problems of strong dependence on functional connectivity features and insufficient utilization of spatial information in existing brain network research.

[0006] The technical solution of the present invention is as follows:

[0007] A time-frequency spatial EEG emotion recognition method based on three-dimensional spatial location embedding includes the following steps:

[0008] Step 1: Based on the international 10-20 system standard, construct a three-dimensional spatial coordinate system for EEG electrodes, and establish a spatial position matrix based on the spatial distance between nodes to represent the spatial adjacency relationship between electrodes. At the same time, divide the EEG signal into five frequency bands: δ, θ, α, β, and γ, and calculate the phase locking value (PLV) between different node pairs to form a multi-band functional connectivity matrix.

[0009] Step 2: Input the spatial location matrix of the EEG nodes and the functional connectivity matrix of each frequency band into the hierarchical cross-transformer module to perform deep feature fusion and construct a multi-band fusion graph structure that characterizes the spatial topology of the brain.

[0010] Step 3: Based on the constructed multi-band fusion graph structure, design a cross-band spatial information fusion mechanism to dynamically weight and fuse the spatial features of EEG node pairs in different frequency bands;

[0011] Step 4: Extract the power spectral density (PSD) and differential entropy (DE) as the basic features of the graph nodes;

[0012] Step 5: Input the fused graph structure and node features into the Graph Attention Network (GAT), and use the attention mechanism to calculate the attention weights between nodes to achieve effective aggregation of spatial and frequency domain information;

[0013] Step 6: Input the node feature sequence output by GAT into a bidirectional long short-term memory (Bi-LSTM) network with an attention mechanism to extract the time-dependent patterns of EEG signals and enhance the feature capture ability of key time segments through a frame-level attention mechanism.

[0014] Step 7: Input the representation after fusing the time-frequency-space multidimensional features into the classification module to achieve accurate identification of different emotional states.

[0015] Furthermore, in step one, when establishing the spatial location matrix, based on the three-dimensional MNI coordinates of the EEG electrodes, four graph structures (A-Graph, M-Graph, S-Graph, I-Graph) are defined. The adjacency matrix is ​​calculated using average distance, minimum neighbor distance, simplified self-connection, and inverse squared distance, and then normalized to construct the three-dimensional electrode spatial location matrix. The distance set is L = {d} ij |(x i x j )∈V 2, where \(i\neq j\), and \(d\) ij is the Euclidean distance between node \(x\) i and \(x\) j in three-dimensional coordinates. The four graph structure formulas are respectively shown in formulas (1)-(4):

[0016]

[0017] where \(A(L)\) represents the average value of the distance set \(L\), and \(A(d\) ik \(|d\) ik \(<A(L), k\in[1,n])\) represents the average value of the distances of all adjacent nodes that are connected to node \(x\) i and have a distance less than \(A(L)\), and \(Min(d\) io \(|o\in[1,n])\) represents the minimum value of the distances of all adjacent nodes that are connected to node \(x\) i and have a distance less than \(A(L)\).

[0018] Furthermore, the specific steps of step two include:

[0019] Feature serialization: First, the channel dimensions of the two types of original graph structures are extended to a high-dimensional space through a linear mapping operation. Subsequently, position encoding is introduced to provide the position information of each element in the sequence. Finally, the features are standardized through an application layer normalization operation;

[0020] Hierarchical cross-fusion attention: First, the two types of graph features are used as inputs respectively, and a hierarchical cross-fusion module is introduced. Each layer of the fusion structure consists of two cross-fusion attention units. The multi-head attention mechanism is used to weight the feature sequence. One branch is used as the query input, and the other branch is used as the key and value inputs. Subsequently, the roles of the two are exchanged to complete bidirectional cross-fusion, and the deep coupling of spatial structure information is achieved through layer-by-layer iteration;

[0021] Feature mapping: After the fused feature map undergoes layer normalization and a feed-forward network operation, the position feature map \(X\) m1 and the functional feature map \(X\) m2 are added together, and matrix fusion is performed through the multi-head attention mechanism.

[0022] Furthermore, in step three, assume that \(F(\delta)\), \(F(\theta)\), \(F(\alpha)\), \(F(\beta)\), \(F(\gamma)\) respectively represent the fusion matrices of the \(\delta\), \(\theta\), \(\alpha\), \(\beta\), \(\gamma\) frequency electroencephalogram signals. The cross-entropy loss function is used to iteratively update the values of the weight coefficients in the backpropagation. \(p\) δ , \(p\) θ , \(p\) α , \(p\) β , \(p\) γ are respectively the weight coefficients of the fusion matrices in the \(\delta\), \(\theta\), \(\alpha\), \(\beta\), \(\gamma\) frequency bands during the feature fusion process. The feature fusion process is represented by formulas (5)-(6):

[0023] F f =p δ F (δ) +p θ F (θ) +p α F (α) +p β F (β) +p γ F (γ) (5)

[0024] p δ +p θ +p α +p β +p γ =1 (6)

[0025] Furthermore, in the step described above, the feature matrix is ​​the fusion matrix F. f The input features of each node i are N is the number of nodes, j is the feature dimension, and GAT calculates the node j∈N based on the masking self-attention mechanism. i Calculate the attention score, where N i is a neighboring node of node i in the adjacency matrix, and its calculation is expressed by formula (7):

[0026]

[0027] Here, 'a' is a single-layer feedforward neural network consisting of a weight vector. After parameterization and using a LeakyReLU nonlinearity with a negative slope of 0.2, the attention score is calculated using Equation (8) after full expansion:

[0028]

[0029] After obtaining the normalized attention score, GAT aggregates the information of neighboring nodes to the target node through a weighted summation operation, and uses a nonlinear function σ to update the feature representation of each node. The specific calculation is expressed by formula (9):

[0030]

[0031] Introducing a multi-head attention mechanism, that is, using K independent attention mechanisms to perform the transformation as shown in Equation (9), the final feature is expressed by Equation (10):

[0032]

[0033] Where || represents the concatenation operation. W represents the attention weight of the m-th head. mThis represents the weight parameter matrix for the m-th head.

[0034] Furthermore, in step six, the input sequence of the Bi-LSTM is X = {x1, x2, ..., x...} T}, where T represents the number of time steps. The Bi-LSTM network computes the hidden state h at each time step in both forward and backward directions. t The hidden states in these two directions are then concatenated to obtain the combined hidden state for each time step, which is expressed by formulas (11)-(12):

[0035]

[0036] Among them, h t c represents the hidden state at time step t. t Let fwd represent the memory state at time step t, fwd represent the forward direction, and bwd represent the backward direction. Finally, the combined hidden state at each time step t can be represented as H = {h1, h2, ..., h...}. T}, where each position t is represented as It is a concatenation of the forward and backward hidden states;

[0037] Next, a self-attention mechanism is used to capture the dependencies between different time steps in the sequence through multiple attention heads. The output H of the LSTM will be used as the input of the self-attention layer. First, a query, key, and value matrix is ​​generated through a linear layer. The calculation formula is expressed by formula (13):

[0038] Q = HW Q K = HW K V = HW v (13)

[0039] Among them, W Q W K and W v This is the learning weight matrix for each head. For each head, the attention score is calculated, and the attention weights are obtained through Softmax, as expressed by formula (14):

[0040]

[0041] Finally, the outputs of all heads are concatenated and projected through a linear layer to form the final self-attention output, which is expressed by formula (15):

[0042] Y = Concat(A1, A2, ..., A h W O (15)

[0043] Among them, A iW is the output of the i-th attention head. O It is the weight matrix of the linear projection.

[0044] Furthermore, in step seven, during emotion recognition, the output is nonlinearly transformed through a fully connected network. Subsequently, the classification probability is calculated using the softmax function, and the maximum value is taken as the final classification result. Finally, the error of the sample is evaluated using the cross-entropy loss function.

[0045] Compared with existing technologies, the temporal-frequency-spatial EEG emotion recognition method based on three-dimensional spatial location embedding described in this technical solution has the following advantages:

[0046] (1) This invention proposes a graph structure construction method based on the spatial location of EEG electrodes. This method effectively enhances the expressive ability of spatial physical adjacency relationships in EEG by integrating the three-dimensional spatial location information of the electrodes, providing a stable structural prior for the feature fusion process; based on the three-dimensional spatial information of EEG electrodes, four node location representation graphs are designed: A-Graph, M-Graph, S-Graph, and I-Graph, integrating the spatial information of nodes into the graph representation of EEG signals, which not only enhances the interaction between signals, but also improves the expressive ability of brain functional areas.

[0047] (2) This invention designs a method for joint modeling of spatial location matrix and functional connectivity matrix. This method utilizes the electrode spatial location matrix and multi-band functional connectivity matrix, and achieves deep fusion of multi-dimensional features through a hierarchical cross-Transformer architecture to construct an EEG model that more closely resembles the actual topology of the brain; the fusion of the two is promoted through the attention mechanism. The spatial location matrix provides a stable spatial reference by preserving the spatial layout of EEG nodes. The functional connectivity matrix is ​​dynamically generated based on the characteristics of each subject's EEG signal, reflecting the functional connectivity characteristics between brain regions under specific emotional states. The relationship between the two signals is related not only to physical location but also to functional connectivity; therefore, by combining the two, the model can take into account both stable structure and individual differences, thus achieving better generalization performance among different subjects and more comprehensively explaining the spatial characteristics of EEG signals.

[0048] (3) This invention proposes a time-frequency-space EEG emotion recognition method based on three-dimensional spatial location embedding. It constructs a graph structure by combining the spatial location information and functional connectivity information of EEG signals, and extracts the time-frequency-space features of EEG signals through GAT and Bi-LSTM to achieve emotion recognition and increase the accuracy of emotion recognition. Attached Figure Description

[0049] Figure 1 This is a flowchart illustrating the implementation of a time-frequency spatial EEG emotion recognition method based on three-dimensional spatial location embedding.

[0050] Figure 2 Flowchart for joint modeling of spatial location matrix and functional connection matrix. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the present invention clearer, the specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0052] A time-frequency spatiotemporal EEG emotion recognition method based on 3D spatial location embedding. For example... Figure 1 As shown, the method includes the following steps:

[0053] Step 1: Constructing the spatial location matrix and functional connectivity matrix: Based on the three-dimensional MNI coordinates of the EEG electrodes, four graph structures (A-Graph, M-Graph, S-Graph, I-Graph) are defined. The adjacency matrix is ​​calculated using the average distance, minimum neighbor distance, simplified self-connectivity, and inverse square distance, and then normalized to construct the three-dimensional electrode spatial location matrix. Multi-channel EEG signals are acquired using an EEG acquisition device with a 10-20 electrode system layout. The sampling frequency is 128Hz. After filtering and despoofing, the signals are divided into five frequency bands: δ (1-4Hz), θ (4-8Hz), α (8-14Hz), β (14-31Hz), and γ (31-50Hz). The PLV matrix of each frequency band is calculated based on the preprocessed signal segments to quantify the phase synchronization degree between brain regions. The functional connectivity matrix is ​​constructed using the PLV value as the edge weight. The PLV calculation process is expressed by formula (1):

[0054]

[0055] in, This represents the phase difference between signals x and y at time t. N represents the total number of time slice samples.

[0056] Next, the elements in the functional connection diagram are determined based on the PLV. The calculation process is expressed by formula (2):

[0057]

[0058] Step Two: Layered Cross-Transformer Feature Fusion: The functional connectivity matrix and electrode spatial location matrix of each frequency band are input into the layered cross-transformer module. Through feature serialization, layered cross-fusion attention, and feature mapping, information from different source graphs is fused to form a unified spatial topology graph structure; for example... Figure 2 As shown:

[0059] Feature serialization includes expanding and encoding spatial features. First, linear mapping is used to expand the channel dimensions of the two original graph structures to a higher dimension to enhance feature representation. Then, positional encoding is introduced to provide positional information of each element in the sequence. Finally, layer normalization is applied to standardize the features, improving the stability of network training and providing a unified input representation for subsequent feature interaction and fusion. The feature serialization process can be represented by formula (3):

[0060]

[0061] Where PCM represents the positional connectivity matrix, FCM represents the functional connectivity matrix, LP represents the linear projection operation, PE represents the positional encoding operation, LN represents the layer normalization operation, and X represents the layer normalization operation. f1 X f2 ∈R D×N This represents the features after serialization.

[0062] Layered cross-fusion attention: This includes bi-branch cross-modeling of the spatial location matrix and the functional connectivity matrix. Through the layered cross-fusion attention mechanism, the two branches are used alternately as query, key, and value inputs to achieve multi-head attention fusion between features. The multi-head attention fusion operation is expressed by formula (4):

[0063]

[0064] Where Q2, K1, and V1 represent query, key, and value, respectively, and A represents the output feature of multi-head attention fusion. A branch X res The query input Q is used as the first cross-fusion attention, while the other branch X... s These serve as the key and value inputs for this cross-fusion attention. Subsequently, the second cross-fusion attention branches out to X. s As the query input, the output of the first cross-fusion attention is used as its key and value input. Therefore, the two cross-fusion attentions together constitute a layer of hierarchical cross-fusion attention, which is mathematically expressed by formula (5):

[0065]

[0066] Here, w = (1,2,...,w) represents the number of layers in the cross-fusion attention, w-1 is the previous layer, and w is the current layer. This mechanism effectively enhances the information interaction between different feature maps, and can dynamically adjust the weights according to the features of the input data, enabling the model to adaptively select the most relevant spatial information and effectively model the spatial relationships between different brain regions.

[0067] Feature mapping: This includes normalization, feature mapping, and dimensionality compression operations on the fused features. After layer normalization and feedforward network operations, the fused feature map X is... m1 and X m2 The features are summed and fused using a multi-head attention mechanism. To simplify subsequent network model processing and reduce computational complexity, the feature mapping module converts the feature dimensions to h×w×D, then reduces the number of feature channels through convolution operations, ultimately obtaining a fusion matrix Fused_Matrix∈R with the same dimensions as the location map and functional connectivity map. h×w×c The feature mapping operation is represented by formulas (6)-(7):

[0068]

[0069] Fused_Matrix = FM(MultiHead(X) m1 +X m2 (7)

[0070] Where LN represents layer normalization, FFN represents feedforward network, FM represents feature mapping, and MultiHead represents multi-head attention mechanism.

[0071] Step 3: Cross-band Spatial Information Fusion: For the fused graph structure, its spatial features across five frequency bands are dynamically weighted and fused to enhance its frequency band selectivity and discriminative ability. Let F(δ), F(θ), F(α), F(β), and F(γ) represent the fusion matrices of EEG signals at frequencies δ, θ, α, β, and γ, respectively. The cross-entropy loss function is used to iteratively update the weight coefficients in the backpropagation, p. δ p θ p α p β p γ These are the weight coefficients of the δ, θ, α, β, and γ band fusion matrices during the feature fusion process. The feature fusion process is represented by formulas (8)-(9):

[0072] F f =p δ F (δ) +p θ F (θ) +p α F (α) +p β F (β) +p γ F (γ) (8)

[0073] p δ +p θ +p α +p β +pγ =1 (9)

[0074] Step 4: Extract PSD and DE from the EEG segments of each frequency band as feature vectors for graph nodes. These features effectively reflect the changes in signal power and information entropy. Assume x ~ N(μ,σ) 2 The DE calculation process is represented by formula (10):

[0075]

[0076] PSD describes the energy distribution of a signal at different frequencies; it is the distribution of the signal's frequency components along the frequency axis. The Welch method reduces noise through multi-segment averaging and is applicable to non-stationary signals. The PSD calculation process is expressed by formulas (11)-(12):

[0077]

[0078] Where, x k w(n) represents the k-th signal segment, and w(n) is a window function (such as a Hamming or Hanning window). It is a normalization factor used to adjust the effect of the window on power, and K is the total number of segments.

[0079] Step 5: GAT Modeling: The fused graph structure and node features are input into the GAT network. A multi-head attention mechanism is used to calculate node attention weights, and neighbor node features are aggregated to generate a joint spatial frequency representation. Graph attention networks are an improvement on graph neural networks, achieving weighted aggregation of neighboring nodes by introducing a self-attention mechanism. The model input consists of an adjacency matrix and node features. In this paper, the feature matrix is ​​the fusion matrix F. f The input features of each node i are (N is the number of nodes, j is the feature dimension). First, to obtain higher expressive power and capture the relationships between nodes, GAT calculates the relationship between nodes j∈N based on a masking self-attention mechanism. i Calculate the attention score, where N i Let i be a neighboring node of node i in the adjacency matrix, and its calculation is expressed by formula (13):

[0080]

[0081] Here, 'a' is a single-layer feedforward neural network consisting of a weight vector. Parameterization is performed using a LeakyReLU nonlinearity with a negative slope of 0.2 (i.e., α = 0.2). After full expansion, the attention score is calculated using Equation (14):

[0082]

[0083] Where T represents the transpose operation, and || represents the concatenation operation. It is a weight matrix (by all (shared), e ij This reflects the importance of node j to node i. To make the attention scores comparable among adjacent nodes, GAT uses the Softmax function to normalize the attention scores of all nodes. The fully expanded form of the attention mechanism is expressed by formula (15):

[0084]

[0085] After obtaining the normalized attention score, GAT aggregates the information of neighboring nodes to the target node through a weighted summation operation, and uses a nonlinear function σ to update the feature representation of each node. The specific calculation is expressed by formula (16):

[0086]

[0087] To stabilize the self-attention learning process, a multi-head attention mechanism is introduced, that is, K independent attention mechanisms are used to perform the transformation as shown in Equation (16), and the final features are expressed by Equation (17):

[0088]

[0089] Where || represents the concatenation operation. W represents the attention weight of the m-th head. m This represents the weight parameter matrix for the m-th head.

[0090] Step Six: Bi-LSTM Modeling: The node sequence output by GAT is input into the Bi-LSTM network step by step to bidirectionally encode EEG temporal dependencies and emphasize the recognition contribution of key frames through a frame-level attention mechanism. The input sequence of Bi-LSTM is X = {x1, x2, ..., x...} T}, where T represents the number of time steps. The Bi-LSTM network computes the hidden state h at each time step in both forward and backward directions. t The hidden states in these two directions are then concatenated to obtain the comprehensive hidden state for each time step, thus providing a more comprehensive representation of the dependencies between time series. This is expressed by formulas (18)-(19):

[0091]

[0092] Among them, h t c represents the hidden state at time step t. tLet fwd represent the memory state at time step t, fwd represent the forward direction, and bwd represent the backward direction. Ultimately, the combined hidden state at each time step t can be represented as H = {h1, h2, ..., h...}. T}, where each position t is represented as It is a concatenation of forward and backward hidden states.

[0093] Next, a self-attention mechanism is used to capture the dependencies between different time steps in the sequence through multiple attention heads. The output H of the LSTM will be used as the input of the self-attention layer. First, a query, key, and value matrix is ​​generated through a linear layer, and the calculation formula is expressed by formula (20):

[0094] Q = HW Q K = HW K V = HW v (20)

[0095] Among them, W Q W K and W v This is the learning weight matrix for each head. For each head, the attention score is calculated, and the attention weights are obtained through Softmax, as expressed by formula (21):

[0096]

[0097] Finally, the outputs of all heads are concatenated and projected through a linear layer to form the final self-attention output, which is expressed by formula (22):

[0098] Y = Concat(A1, A2, ..., A...) h W O (twenty two)

[0099] Among them, A i W is the output of the i-th attention head. O It is the weight matrix of the linear projection.

[0100] Step 7: Emotion Recognition: The output is non-linearly transformed through a fully connected network to obtain a deeper feature representation. Then, the softmax function is used to calculate the classification probability, and the maximum value is taken as the final classification result. Finally, the error of the sample is evaluated using the cross-entropy loss function, expressed by formula (23):

[0101]

[0102] Among them, y ic Indicates true emotional tags, This represents the predicted sentiment label, where N is the total number of samples and C is the total number of categories.

[0103] The method proposed in this invention is as follows: First, a node location representation map is constructed using the three-dimensional spatial information of EEG electrodes, effectively integrating spatial information into the graph structure representation of EEG signals, thereby enhancing the modeling ability of brain functional regions. Second, by combining the electrode spatial location matrix and the multi-band functional connectivity matrix, a hierarchical cross-Transformer architecture is used to achieve deep fusion of multi-dimensional features, constructing an EEG model that more closely resembles the real brain topology, and introducing an attention mechanism to promote the synergistic fusion of spatial and functional information. Based on this, the attention mechanisms of GAT and Bi-LSTM are integrated to comprehensively extract temporal, frequency, and spatial features of emotional states, achieving comprehensive modeling and recognition of emotional states, significantly improving the accuracy of emotion recognition.

[0104] The effectiveness of this invention is demonstrated by comparison with existing technologies. Liu et al. proposed a global-to-local feature aggregation network (GLFANet), which constructs a topological graph structure using the spatial location and frequency domain features of channels, and learns more representative topological features through GLFANet. Chen et al. calculated the Euclidean distance adjacency matrix using Ordinary Differential Equation (ODE) and constructed a temporal functional adjacency matrix using the Dynamic Time Wrapping (DTW) algorithm. They further fused this matrix with two parallel GCN and LSTM modules to extract inter-channel relational features, achieving recognition accuracies of 96.21% and 97.92% on the SEED and DEAP datasets, respectively. Kong et al. constructed a functional connectivity graph using Granger causality (GC) between channels and extracted emotion features for recognition analysis using depthwise separable convolution. Xu et al. proposed an attention-based Multiple Dimensions EEG Transformer (AMDET), which employs a self-attention mechanism in the channel dimension to capture brain functional connectivity, thereby generating spatial feature representations. These representations are then fused with frequency and temporal features to achieve feature complementarity. The results, compared with existing graph-based methods, are shown in Table 1.

[0105] Table 1 Comparison of relevant methods and experimental results

[0106]

[0107] In summary, most current EEG emotion recognition studies neglect the positional connectivity between channels during graph structure construction, and their methods for fusing electrode space and functional connectivity graphs are overly simplistic, failing to fully leverage the complementarity and redundancy of spatial information. This invention, by introducing a hierarchical cross-attention fusion module, effectively integrates electrode positional connectivity and functional connectivity information, simulating the complex relationships between EEG electrodes from a multi-dimensional perspective, optimizing information fusion strategies, and thus extracting more discriminative spatial features. Compared to existing graph structure methods, this invention demonstrates significant advantages in both emotion recognition accuracy and spatial feature representation capabilities.

[0108] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A time-frequency spatial EEG emotion recognition method based on three-dimensional spatial location embedding, characterized in that, Includes the following steps: Step 1: Based on the international 10-20 system standard, construct a three-dimensional spatial coordinate system for EEG electrodes, and establish a spatial position matrix based on the spatial distance between nodes to represent the spatial adjacency relationship between electrodes. At the same time, divide the EEG signal into five frequency bands: δ, θ, α, β, and γ, and calculate the phase locking value (PLV) between different node pairs to form a multi-band functional connectivity matrix. Step 2: Input the spatial location matrix of the EEG nodes and the functional connectivity matrix of each frequency band into the hierarchical cross-transformer module to perform deep feature fusion and construct a multi-band fusion graph structure that characterizes the spatial topology of the brain. Step 3: Based on the constructed multi-band fusion graph structure, design a cross-band spatial information fusion mechanism to dynamically weight and fuse the spatial features of EEG node pairs in different frequency bands; Step 4: Extract the power spectral density (PSD) and differential entropy (DE) as the basic features of the graph nodes; Step 5: Input the fused graph structure and node features into the Graph Attention Network (GAT), and use the attention mechanism to calculate the attention weights between nodes to achieve effective aggregation of spatial and frequency domain information; Step 6: Input the node feature sequence output by GAT into a bidirectional long short-term memory (Bi-LSTM) network with an attention mechanism to extract the time-dependent patterns of EEG signals and enhance the feature capture ability of key time segments through a frame-level attention mechanism. Step 7: Input the representation after fusing the time-frequency-space multidimensional features into the classification module to achieve accurate identification of different emotional states.

2. The time-frequency spatial EEG emotion recognition method based on three-dimensional spatial location embedding according to claim 1, characterized in that: In step one, when establishing the spatial location matrix, based on the three-dimensional MNI coordinates of the EEG electrodes, four graph structures (A-Graph, M-Graph, S-Graph, I-Graph) are defined. The adjacency matrix is ​​calculated using average distance, minimum neighbor distance, simplified self-connectivity, and inverse squared distance, and then normalized to construct the three-dimensional electrode spatial location matrix. The distance set is L = {d} ij |(x i x j )∈V 2 , i≠j}, where d ij For node x i and x j In three-dimensional coordinates, the Euclidean distance formulas for the four graph structures are equations (1)-(4): where, A(L) represents the average value of the distance from the set L, A(d ik |d ik <A(L), k ∈ [1, n]) represents the average value of the distances of all adjacent nodes connected to the node x i and with distances less than A(L), Min(d io |o ∈ [1, n]) represents the minimum value of the distances of the nodes connected to the node x i and with connections.

3. The time-frequency spatial EEG emotion recognition method based on three-dimensional spatial location embedding according to claim 2, characterized in that, Step two specifically includes: Feature serialization first extends the channel dimension of the two original graph structures to a high-dimensional space through a linear mapping operation. Then, positional encoding is introduced to provide positional information of each element in the sequence. Finally, the features are standardized by applying layer normalization operations. Layered cross-fusion note: First, the two types of graph features are used as inputs and introduced into the layered cross-fusion module. Each layer of the fusion structure consists of two cross-fusion attention units. A multi-head attention mechanism is used to weight the feature sequence. One branch is used as the query input and the other branch is used as the key and value input. Then, the roles of the two branches are exchanged to complete the bidirectional cross-fusion. The deep coupling of spatial structure information is achieved through layer-by-layer iteration. Feature mapping: After the fused feature map undergoes layer normalization and feedforward network operations, the location feature map X is mapped. m1 and functional feature diagram X m2 They are added together and then fused into a matrix using a multi-head attention mechanism.

4. The time-frequency spatial EEG emotion recognition method based on three-dimensional spatial location embedding according to claim 3, characterized in that: In step three, assuming F(δ), F(θ), F(α), F(β), and F(γ) represent the fusion matrices of EEG signals at frequencies δ, θ, α, β, and γ, respectively, the cross-entropy loss function is used to iteratively update the weight coefficients in the backpropagation. δ p θ p α p β p γ These are the weight coefficients of the δ, θ, α, β, and γ band fusion matrices during the feature fusion process. The feature fusion process is represented by formulas (5)-(6): F f =p δ F (δ) +p θ F (θ) +p α F (α) +p β F (β) +p γ F (γ) (5) p δ +p θ +p α +p β +p γ =1 (6)。 5. The time-frequency spatial EEG emotion recognition method based on three-dimensional spatial location embedding according to claim 4, characterized in that: In the aforementioned steps, the feature matrix is ​​the fusion matrix F. f The input features of each node i are N is the number of nodes, j is the feature dimension, and GAT calculates the node j∈N based on the masking self-attention mechanism. i Calculate the attention score, where N i is a neighboring node of node i in the adjacency matrix, and its calculation is expressed by formula (7): Here, 'a' is a single-layer feedforward neural network consisting of a weight vector. After parameterization and using a LeakyReLU nonlinearity with a negative slope of 0.2, the attention score is calculated using Equation (8) after full expansion: After obtaining the normalized attention score, GAT aggregates the information of neighboring nodes to the target node through a weighted summation operation, and uses a nonlinear function σ to update the feature representation of each node. The specific calculation is expressed by formula (9): Introducing a multi-head attention mechanism, that is, using K independent attention mechanisms to perform the transformation as shown in Equation (9), the final feature is expressed by Equation (10): Where || represents the concatenation operation. W represents the attention weight of the m-th head. m This represents the weight parameter matrix for the m-th head.

6. The time-frequency spatial EEG emotion recognition method based on three-dimensional spatial location embedding according to claim 5, characterized in that: In step six, the input sequence of Bi-LSTM is X = {x1, x2, ..., x...} T }, where T represents the number of time steps. The Bi-LSTM network computes the hidden state h at each time step in both forward and backward directions. t The hidden states in these two directions are then concatenated to obtain the combined hidden state for each time step, which is expressed by formulas (11)-(12): Among them, h t c represents the hidden state at time step t. t Let fwd represent the memory state at time step t, fwd represent the forward direction, and bwd represent the backward direction. Finally, the combined hidden state at each time step t can be represented as H = {h1, h2, ..., h...}. T }, where each position t is represented as It is a concatenation of the forward and backward hidden states; Next, a self-attention mechanism is used to capture the dependencies between different time steps in the sequence through multiple attention heads. The output H of the LSTM will be used as the input of the self-attention layer. First, a query, key, and value matrix is ​​generated through a linear layer. The calculation formula is expressed by formula (13): Q=HW Q ,K=HW K ,V=HW v (13) Among them, W Q W K and W v This is the learning weight matrix for each head. For each head, the attention score is calculated, and the attention weights are obtained through Softmax, as expressed by formula (14): Finally, the outputs of all heads are concatenated and projected through a linear layer to form the final self-attention output, which is expressed by formula (15): Y=Concat(A1,A2,…,A h )W O (15) Among them, A i W is the output of the i-th attention head. O It is the weight matrix of the linear projection.

7. The time-frequency spatial EEG emotion recognition method based on three-dimensional spatial location embedding according to claim 5, characterized in that: In step seven, during emotion recognition, the output undergoes a nonlinear transformation through a fully connected network. Subsequently, the softmax function is used to calculate the classification probability, and the maximum value is taken as the final classification result. Finally, the cross-entropy loss function is used to evaluate the sample error.

Images