Electroencephalogram emotion recognition method and device and storage medium

Abstract

The application provides an electroencephalogram emotion recognition method and device and a storage medium. The method comprises the following steps: calling a plurality of electrodes attached to the head of a user to collect multi-channel electroencephalogram signals of the user; extracting time features from the electroencephalogram signals to obtain electroencephalogram time sequence features; constructing an electroencephalogram space-time topology according to the correlation between the plurality of electroencephalogram time sequence features and the spatial distribution information of the plurality of electrodes; extracting target electroencephalogram space-time features in the electroencephalogram space-time topology according to the electroencephalogram time sequence features and the attention of the electrodes in geometry; and recognizing the emotional information of the user according to the target electroencephalogram space-time features. The embodiment realizes the collaborative modeling of space-time features, can intuitively reflect the relative importance of each brain area in the emotion recognition task, improves the accuracy and stability of emotion recognition, and enhances the generalization and physiological interpretability of the model.

Description

Technical Field

[0001] This invention belongs to the technical field of deep learning, and in particular relates to a brainwave emotion recognition method, device and storage medium. Background Technology

[0002] Emotion recognition based on electroencephalography (EEG) signals identifies emotional states by analyzing the brain's electrical activity. It can respond to human emotions in a more nuanced way and can be applied to scenarios such as mental health monitoring, human-computer interaction, and adaptive learning.

[0003] For example, it can help monitor the emotional state of patients with mood disorders and provide real-time feedback during treatment, thereby assisting clinicians in tailoring interventions to individual needs.

[0004] For example, in interactive applications such as games, virtual reality, and personalized learning, the system identifies users' emotional states to dynamically adjust and enhance user engagement.

[0005] Typically, EEG-based emotion recognition relies on machine learning models that classify emotional states from extracted features that capture complex spatiotemporal patterns in EEG signals.

[0006] Currently, deep learning models such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and Transformer-based architectures are commonly used to learn hierarchical features from raw EEG signals and perform emotion recognition accordingly.

[0007] However, these deep learning models mainly focus on temporal features, have a relatively single dimension, and have low accuracy in emotion recognition. Summary of the Invention

[0008] In view of this, the present invention provides a brainwave emotion recognition method, device and storage medium to improve the accuracy of emotion recognition.

[0009] A first aspect of the present invention provides a brainwave emotion recognition method, comprising:

[0010] Multiple electrodes attached to the user's head are used to collect multi-channel electroencephalogram (EEG) signals from the user;

[0011] Temporal features are extracted from the electroencephalogram (EEG) signals to obtain EEG temporal features.

[0012] A spatiotemporal topology map of brainwaves is constructed based on the correlation between the multiple temporal features of brainwaves and the spatial distribution information of the multiple electrodes;

[0013] Based on the EEG temporal features and the attention sensed geometrically by the electrodes, the target EEG spatiotemporal features are extracted from the EEG spatiotemporal topology map.

[0014] The user's emotional information is identified based on the target EEG spatiotemporal features.

[0015] A second aspect of the present invention provides a brainwave emotion recognition device, comprising:

[0016] The EEG signal acquisition module is used to call multiple electrodes attached to the user's head to acquire multi-channel EEG signals from the user;

[0017] The EEG temporal feature extraction module is used to extract temporal features from the EEG signal to obtain EEG temporal features.

[0018] The EEG spatiotemporal topology map construction module is used to construct an EEG spatiotemporal topology map based on the correlation between multiple EEG temporal features and the spatial distribution information of multiple electrodes;

[0019] The target EEG spatiotemporal feature extraction module is used to extract the target EEG spatiotemporal features in the EEG spatiotemporal topology map based on the EEG temporal features and the attention sensed geometrically by the electrodes.

[0020] The emotion information recognition module is used to recognize the user's emotion information based on the target EEG spatiotemporal features.

[0021] A third aspect of the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the EEG emotion recognition method as described in the first aspect above.

[0022] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the EEG emotion recognition method as described in the first aspect above.

[0023] A fifth aspect of the present invention provides a computer program product that, when run on a computer, causes the computer to perform the EEG emotion recognition method as described in the first aspect above.

[0024] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0025] In this embodiment, multiple electrodes attached to the user's head are used to collect multi-channel EEG signals. Temporal features are extracted from the EEG signals to obtain EEG temporal features. A spatiotemporal topology map of the EEG is constructed based on the correlation between the multiple EEG temporal features and the spatial distribution information of the electrodes. Target spatiotemporal features of the EEG are extracted from the spatiotemporal topology map based on the EEG temporal features and the geometrically perceived attention of the electrodes. The user's emotional information is then identified based on the target spatiotemporal features. This embodiment extracts the temporal dynamic features of the electrodes and uses geometrically perceived attention to model the spatial dependencies between the electrodes, achieving a joint representation of the spatiotemporal features of the EEG signals. This provides a more comprehensive characterization of emotion-related brain activity patterns and achieves collaborative modeling of spatiotemporal features. It can intuitively reflect the relative importance of each brain region in the emotion recognition task, improving the accuracy and stability of emotion recognition, and enhancing the model's generalization and physiological interpretability. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a schematic diagram of an EEG emotion recognition method provided in an embodiment of the present invention;

[0028] Figure 2 This is an example diagram of an electrode and its adjacency matrix provided in an embodiment of the present invention;

[0029] Figure 3 This is a schematic diagram of the structure of an emotion detection module provided in an embodiment of the present invention;

[0030] Figure 4 This is a schematic flowchart of a geometric perception self-attention mechanism provided in an embodiment of the present invention;

[0031] Figure 5 This is a schematic diagram of an EEG emotion recognition device provided in an embodiment of the present invention;

[0032] Figure 6 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0033] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the present invention. However, those skilled in the art will recognize that the present application may be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted to avoid unnecessary detail that could obscure the description of the present application.

[0034] The technical solution of the present invention will be illustrated below through specific embodiments.

[0035] Reference Figure 1 The diagram illustrates a brainwave emotion recognition method provided by an embodiment of the present invention, which may specifically include the following steps:

[0036] Step 101: Use multiple electrodes attached to the user's head to collect multi-channel EEG signals from the user.

[0037] In this embodiment, as Figure 2 As shown, the brain-computer interface device can be worn on the user's head. The brain-computer interface provides multiple electrodes, which are attached to the corresponding positions on the user's head according to the specifications.

[0038] Among them, multiple electrodes can be either wet electrodes or dry electrodes.

[0039] Taking the 10-20 system as an example of a brain-computer interface device, the electrodes in the 10-20 system include 19 recording electrodes and 2 reference electrodes.

[0040] Two lines are marked on the scalp surface: one is the 100% mark connecting the anterior and posterior ends of the line from the root of the nose to the external occipital protuberance, and the other is the 100% mark connecting the left and right sides of the line between the anterior fossae of both ears. The intersection of these two lines at the top of the head is the location of the Cz electrode. 10% backward from the root of the nose is the FPz (frontal midline). From FPz backward, each 20% mark is an electrode location, namely Fz (frontal midline), Cz (central midline), Pz (parietal midline), and Oz (occipital midline).

[0041] The distance between Oz and the external occipital protuberance is 10%. The T3 (left middle temporal) electrode is located 10% of the distance from the left anterior temporal fossa along the line connecting the two anterior auricles. Electrodes are then placed every 20% to the right: C3 (left central), Cz, C4 (right central), and T4 (right middle temporal). T4 is 10% of the distance from the right anterior temporal fossa. The line connecting FPz through T3 to Oz is the left temporal line. 10% to the left of FPz is FP1 (left frontal pole). Electrodes are then placed every 20% posteriorly from FP1: F7 (left anterior temporal), T3 (left middle temporal), T5 (left posterior temporal), and O1 (left occipital). T3 is the intersection of this line and the line connecting the two anterior auricles, and O1 is 10% of the distance from Oz. The line connecting the right temporal poles corresponds to this, and from front to back, they are FP2 (right temporal pole), F8 (right anterior temporal pole), T4 (right middle temporal pole), T6 (right posterior temporal pole), and O2 (right occipital pole).

[0042] Draw lines from FP1 to O1 and from FP2 to O2, forming left and right parasagittal lines. Electrode sites are located 20% of the distance from FP1 and FP2 onwards. On the left, these are F3 (left frontal), C3 (left central), P3 (left parietal), and O1 (left occipital); on the right, they are F4 (right frontal), C4 (right central), P4 (right parietal), and O2 (right occipital). In the 10-20 system, FPz and Oz are not included within the 19 recording sites.

[0043] like Figure 3 As shown, during the user's business operations (such as treating mood disorders, playing games, etc.), multiple electrodes can be invoked to collect multi-channel EEG signals from the user along the time axis.

[0044] For the raw EEG signals, preprocessing such as bandpass filtering, resampling, and segmentation can be performed sequentially to remove power frequency interference, motion artifacts, and low-frequency drift.

[0045] The bandpass filter has a frequency range of 4Hz to 45Hz, and the sampling rate is 128Hz after resampling.

[0046] The preprocessed EEG signal is segmented into segments according to a fixed time window (e.g., 2 seconds) to obtain EEG signal fragments.

[0047] The preprocessed EEG signal segments are input into an instance feature encoder to extract robust features to represent the EEG signal, or differential entropy features are extracted from five frequency bands (δ, θ, α, β, γ) to represent the EEG signal.

[0048] Step 102: Extract temporal features from the EEG signal to obtain EEG temporal features.

[0049] In this embodiment, an emotion detection model can be pre-built and trained for the emotion recognition task. The emotion detection model is used to detect the user's emotional information based on EEG signals.

[0050] like Figure 3 As shown, the emotion detection model includes structures such as a Channel-Wise Feature Encoder, a Geometry-Aware Graph Transformer (EEGGraphormer), and a classifier.

[0051] Among them, the channel feature encoder is used to extract features of EEG signals from the time dimension, the geometric perceptual map transformer is used to further extract features of EEG signals by combining the time and space dimensions, and the post-processing structure is used to identify the user's emotional information.

[0052] In this embodiment, multi-channel EEG signals can be input into a channel feature encoder, which is responsible for extracting temporal features from the multi-channel EEG signals to obtain EEG temporal features.

[0053] The channel feature encoder stacks one or more convolutional mixer blocks to extract multi-size temporal features layer by layer. Each convolutional mixer block includes structures such as depthwise convolution and pointwise convolution.

[0054] Deep convolutional layers are responsible for performing independent convolutions on the input channels, with each channel using an independent convolutional kernel, and the number of output channels is equal to the number of input channels.

[0055] Point convolutional layers use 1×1 kernels to fuse inter-channel features from the output of deep convolutional layers, adjusting the number of channels.

[0056] Then, the first candidate temporal features of each channel are input into the deep convolutional layer. The convolutional kernel of each channel slides along the time axis to extract local features of the time series signal (i.e., EEG signal) of that channel, thereby obtaining the first single-channel EEG feature, thus capturing the short-term fluctuation pattern of EEG signal in the time dimension, such as brain wave rhythm, instantaneous changes and energy distribution patterns.

[0057] For the stacked case, the first candidate temporal feature is initially the EEG signal, and the second candidate temporal feature is not initially the output of the previous convolutional mixed block.

[0058] Deep convolutional layers operate independently on each channel, without mixing information between different channels, enabling the extraction of temporal patterns within each EEG. This operation ensures that the temporal locality within individual channels is preserved, thereby maintaining the independence and integrity of the features of each channel, while significantly reducing computational cost and the number of parameters.

[0059] Each of the first single-channel EEG features is sequentially nonlinearly activated and normalized to obtain the second single-channel EEG features.

[0060] For example, the nonlinear activation is GELU (Gaussian Error Linear Unit), and the normalization is BatchNorm.

[0061] For the same channel, the EEG signal is concatenated with the second single-channel EEG feature to form a third single-channel EEG feature, thereby constructing a residual connection to ensure smooth information flow and alleviate the gradient vanishing problem.

[0062] By stacking the above structures in multiple layers, it is possible to capture the temporal dependencies between short-range and long-range data, forming a hierarchical representation of temporal features.

[0063] The third single-channel EEG features from multiple channels are input into a convolutional layer for interaction. The feature information of different channels is fused point by point in the time dimension to promote feature integration between channels and realize cross-channel feature projection and mapping to obtain cross-channel EEG features.

[0064] Pointwise convolutional layers can establish inter-channel relationships without changing the temporal resolution, and embed the features of different channels into a unified high-dimensional space, providing an input basis for subsequent spatial modeling.

[0065] Nonlinear activation, normalization, and dropout processing are sequentially performed on the cross-channel EEG features of each channel to obtain second candidate temporal features, thereby enhancing the nonlinear expressive power of the channel feature encoder, improving training stability, and preventing overfitting.

[0066] In the case of stacking, the second candidate temporal feature output by the non-final convolutional hybrid block is the first candidate temporal feature input to the next convolutional hybrid block, and the second candidate temporal feature output by the final convolutional hybrid block is the EEG temporal feature.

[0067] For example, the nonlinear activation is GELU, and the normalization is BatchNorm.

[0068] This combined structure constitutes a highly efficient depthwise separable convolution, which can approximate the effect of standard convolution with significantly lower computational overhead. During training, it can automatically balance the weights of different features, enabling the channel feature encoder to learn the temporal patterns of EEG signals more robustly and efficiently extract dynamic features within the channels.

[0069] Step 103: Construct a spatiotemporal topology map of brainwaves based on the correlation between multiple EEG temporal features and the spatial distribution information of multiple electrodes.

[0070] Electroencephalogram (EEG) signals typically exhibit two dimensions: temporal evolution and spatial distribution across electrodes. The spatial dimension reflects the anatomical arrangement of electrodes on the scalp, which correspond to different functional brain regions, while the temporal dimension captures the dynamic changes in neural activity.

[0071] like Figure 2 As shown, a graph structure can be constructed based on the correlation between multiple EEG temporal features and the spatial distribution information of multiple electrodes, which is denoted as the EEG spatiotemporal topology graph.

[0072] In practical applications, electrodes are modeled as nodes in a non-Euclidean structure. That is, electrodes are converted into nodes in the spatiotemporal topology of EEG. For the same electrode, the EEG temporal features of the corresponding channel of the electrode can be written into the node corresponding to the node as the node feature.

[0073] This graphical representation captures the spatial relationships between electrodes, thereby capturing the complex spatial dependencies between them.

[0074] Furthermore, in the spatiotemporal topology map of EEG, constructing edges between two nodes and configuring the distances between the edges according to the spatial positional relationship between the two electrodes can more realistically preserve the irregular spatial topology of EEG and provide greater flexibility for capturing physiologically sound relationships between electrodes.

[0075] like Figure 2 As shown, each electrode can be traversed to determine whether there is an adjacency between any two electrodes. The adjacency represents the connection strength or correlation between electrodes (nodes), which can include physical electrode structure connection methods (such as the lead method) or functional connections between electrodes constructed through signal analysis.

[0076] Generally, whether there is an adjacency relationship between two electrodes is prior knowledge. The adjacency relationship between two electrodes can form an adjacency matrix. The adjacency matrix provides a mathematical tool for studying brain networks and helps to understand the dynamic characteristics of brain electrical activity.

[0077] Structural connectivity refers to the strength of white matter fiber connections between different brain regions (or electrodes) constructed using techniques such as diffusion-weighted imaging (DWI), reflecting physical structural connectivity.

[0078] Functional connectivity refers to the analysis of time series of EEG signals to calculate the statistical dependence (such as correlation) between electrodes, reflecting the functional interactions between brain regions.

[0079] In the adjacency matrix, a value of 1 indicates that there is an edge between two nodes, and a value of 0 indicates that there is no edge between two nodes.

[0080] If two electrodes are spatially adjacent, the two-dimensional coordinates of each node on the user's head are queried based on prior knowledge. The shortest path length (SPH) between the two nodes is calculated based on the two-dimensional coordinates of the two nodes on the user's head, and the shortest path length is assigned as the distance of the edge.

[0081] If there is no spatial adjacency between the two electrodes, the specified value (i.e., a constant) is assigned the distance of the edge.

[0082] All of these values ​​(i.e., constants) are greater than the shortest path length.

[0083] If we use a matrix to record the distances between edges between any two nodes, we can form a distance matrix (D). In this case, the distance matrix D can be represented as D = [d...]. ij ] N×N , where d ij Let N be the distance between the edge between the i-th node and the j-th node, and N be the number of nodes.

[0084] Step 104: Extract the target EEG spatiotemporal features from the EEG spatiotemporal topology map based on the EEG temporal characteristics and the attention sensed by the electrodes in geometry.

[0085] like Figure 3 As shown, the spatiotemporal topology map of EEG is input into the geometric perception map transformer. By explicitly considering the geometric relationship between electrodes, the spatial dependence between electrodes is captured. Based on the temporal characteristics of EEG and the attention perceived by the electrodes in geometry, the spatiotemporal characteristics of the target EEG are extracted. This enables the learning of more meaningful spatial representations and captures both local and global dependencies.

[0086] In one embodiment of the present invention, step 104 may include the following steps:

[0087] Step 1041: Construct geometric decay elements for the multi-head attention mechanism.

[0088] In this embodiment, as Figure 3As shown, the geometric-aware graph transformer stacks one or more graph encoding layers, progressively enhancing the characterization of spatial dependencies by node representations. The graph encoding layer ultimately outputs a high-dimensional representation for each node, which simultaneously integrates channel temporal information and geometric constraints between electrodes. This representation can be used by subsequent classifiers for sentiment discrimination or for interpretability analysis.

[0089] A multi-head attention mechanism is provided in the geometric perception graph transformer, with multiple head structures applying the self-attention mechanism in each graph coding layer. Optionally, the number of graph coding layers is 2, the number of head structures in each graph coding layer is 4, and the embedding dimension is 64.

[0090] Each head structure includes structures such as LayerNorm, Geometric-aware Self-Attention, and Feed Forward Network.

[0091] In this embodiment, a geometric decay element can be constructed for the multi-head attention mechanism. The geometric decay element is used to adjust the degree of influence of spatial distance. There is a one-to-one correspondence between the geometric decay element and the edge in the EEG spatiotemporal topology graph. The geometric decay element is negatively correlated with the distance of the edge. That is, the larger the distance of the edge, the smaller the geometric decay element, and vice versa.

[0092] For example, the geometric attenuation element has a base of an exponential attenuation factor that controls the attenuation intensity and an exponent of the distance between its edges. In this case, the geometric attenuation element can be represented as M. (i,j) =γ dij , of which M (i,j) Let d be the geometric decay element corresponding to the edge between the i-th node and the j-th node. ij Let γ be the distance between the edge between the i-th node and the j-th node, and let γ be the exponential decay factor.

[0093] The exponential decay factor is a hyperparameter. To promote spatial diversity among different attention heads, an independent exponential decay factor can be assigned to each head structure.

[0094] For example, logarithmic interpolation can be used to define a series of exponential decay factors γ, γ = 1-2, for multiple head structures. -5-arange(0,h) ∈R, where arange(0,h) is used to generate an array between 0 and h, and R is the set of real numbers.

[0095] In this example, the exponential decay factor ranges from (0, 1), and the distance of the edge ranges from (0, 1). Therefore, the geometric decay element of an edge with an adjacency relationship is greater than the geometric decay element of an edge without an adjacency relationship.

[0096] If we use a matrix to record the geometric decay elements corresponding to the edges between each pair of nodes, we can form a distance decay matrix. In this case, the distance matrix M can be represented as M∈R N×N , where R is the set of real numbers and N is the number of nodes.

[0097] Step 1042: Normalize the first intermediate spatiotemporal features.

[0098] like Figure 3 As shown, the first intermediate spatiotemporal features can be normalized using methods such as layer normalization.

[0099] In the case of stacking, the first intermediate spatiotemporal feature is initially the EEG spatiotemporal topology map (the EEG temporal features in each node, and the second intermediate spatiotemporal feature is not initially the output of the encoder of the previous map).

[0100] Step 1043: If normalization is completed, then in the EEG spatiotemporal topology map, the first intermediate spatiotemporal feature is updated to the first candidate EEG spatiotemporal feature using a multi-head attention mechanism based on the geometric decay element.

[0101] In this embodiment, as Figure 3 As shown, the spatiotemporal topology of EEG can be aligned with the geometric decay elements. Based on the geometric decay elements, a multi-head attention mechanism is used to update the first intermediate spatiotemporal features of the corresponding nodes (i.e., the nodes connected by the edges corresponding to the geometric decay elements) to the first candidate spatiotemporal features of EEG.

[0102] Generally, spatially adjacent nodes have greater weight, while the influence of distant nodes on attention scores is suppressed.

[0103] In specific implementations, such as Figure 4 As shown, a linear layer can be used to map the first intermediate spatiotemporal feature (i.e., the node feature) to the query vector Q, key vector K, and value vector V in the attention mechanism.

[0104] Where Q=HW q K=HW k V=HW v H is the first intermediate spatiotemporal feature, H∈R N×d W q W k With W v Both are learnable weight matrices, W q The dimension is d×d q W k The dimension is d×d k With W vThe dimension is d×d v N is the number of nodes, and d is the original dimension of the first intermediate spatiotemporal feature. q Let d be the dimension of the first intermediate spatiotemporal feature in the query direction. k Let d be the dimension of the first intermediate spatiotemporal feature in the bond direction. v Let be the dimension of the first intermediate spatiotemporal feature in the value direction.

[0105] By traversing each node, for the current node, the similarity between the query vector of the current node and the key vector of other nodes can be calculated using matrix multiplication MatMul and other methods. The similarity between the query vector of the current node and the key vector of other nodes is scaled using the geometric decay elements corresponding to the edges between the current node and other nodes to obtain the attention score, making the attention score more consistent with physiological structural characteristics.

[0106] For example, the attention score is represented as:

[0107] ;

[0108] Where node i is the current node, node j is other nodes, score(i,j) is the attention score between node i and node j, and Q is the sum of the attention scores between them. i Let K be the query vector for node i. j Let d be the key vector of node j. k Let M(i,j) be the vector dimension, M(i,j) be the geometric decay element corresponding to the edge between node i and node j, and T denote the transpose.

[0109] Iterate through the attention scores between the current node and other nodes, and use Softmax (a sigmoid function) to normalize the attention scores between the current node and other nodes to obtain the attention weights.

[0110] For example, attention weights can be represented as follows:

[0111] ;

[0112] Where Attention(i,j) is the attention weight between node i and node j, exp is the natural exponential function, score(i,j) is the attention score between node i and node j, and N is the number of nodes.

[0113] Iterate through the attention weights between the current node and other nodes, and use matrix multiplication such as MatMul to multiply the attention weight of the current node with the value vectors of each other node. Add the products of the attention weight of the current node with the value vectors of the other nodes to obtain the first candidate spatiotemporal features of EEG.

[0114] For example, the first candidate spatiotemporal EEG feature can be represented as:

[0115] ;

[0116] in, V represents the first candidate spatiotemporal EEG feature of node i, Attention(i,j) represents the attention weight between node i and node j, and V j Let j be the value vector of node j.

[0117] After traversing all nodes, the first candidate EEG spatiotemporal features of each node can be recorded in matrix form, resulting in a new node feature matrix H'. .

[0118] Step 1044: Merge the first intermediate spatiotemporal features before normalization with the first candidate EEG spatiotemporal features to form the second candidate EEG spatiotemporal features.

[0119] like Figure 3 As shown, functions such as Concat (concatenation) and Add (addition) can be used to fuse the first intermediate spatiotemporal feature before normalization with the first candidate EEG spatiotemporal feature into the second candidate EEG spatiotemporal feature, thereby constructing a residual connection.

[0120] Step 1045: Normalize the spatiotemporal features of the second candidate EEG.

[0121] In this embodiment, as Figure 3 As shown, layer normalization (LayerNorm) and other methods can be used to normalize the spatiotemporal features of the second candidate EEG.

[0122] Step 1046: If normalization is completed, perform nonlinear transformation on the second candidate EEG spatiotemporal features to map them to the third candidate EEG spatiotemporal features.

[0123] When normalizing the second candidate EEG spatiotemporal features, the second candidate EEG spatiotemporal features can be input into a feedforward network for nonlinear transformation, thereby mapping the second candidate EEG spatiotemporal features to the third candidate EEG spatiotemporal features.

[0124] Step 1047: Merge the second candidate EEG spatiotemporal features and the third candidate EEG spatiotemporal features before normalization into a second intermediate spatiotemporal feature.

[0125] like Figure 3 As shown, functions such as Concat (concatenation) and Add (addition) can be used to fuse the second candidate EEG spatiotemporal features and the third candidate EEG spatiotemporal features before normalization into a second intermediate spatiotemporal feature, thereby constructing a residual connection.

[0126] In the case of stacking, the second intermediate spatiotemporal feature output by the non-final graph coding layer becomes the first intermediate spatiotemporal feature input to the next graph coding layer, and the second intermediate spatiotemporal feature output by the final graph coding layer becomes the target EEG spatiotemporal feature.

[0127] If a graph neural network (GNN) is used to process the spatial structure of EEG signals, it represents the multi-channel EEG signals as a graph structure, where electrodes are nodes and spatial distances or functional correlations are edges. Graph-based models, such as graph convolutional networks (GCN) and graph attention networks (GAT), can capture the spatial dependencies between electrodes and achieve performance improvements in EEG emotion recognition tasks by utilizing non-Euclidean spatial information.

[0128] GCN aggregates features of neighboring nodes based on a predefined adjacency matrix, making it difficult to capture dependencies between distant electrodes, especially in high-density EEG settings where the graph structure becomes large and complex. To model such long-range dependencies, multiple graph convolutional layers are stacked, which can lead to over-smoothing and degradation of discriminative features.

[0129] GAT partially mitigates this problem by assigning learnable attentional weights to adjacent nodes, but the attention mechanisms in GAT are typically based on feature similarity and local structure, failing to explicitly incorporate the geometric relationships between electrodes. Therefore, its attentional weights may not adequately reflect the underlying neurophysiological organization of the brain.

[0130] The aforementioned limitations highlight the need for a more expressive and geometrically aware framework that can simultaneously model local and global spatial dependencies in EEG signals and explicitly incorporate electrode topology into the learning process. To this end, this embodiment proposes a geometrically aware graph transformer model that directly embeds spatial prior information derived from electrode locations into a self-attention mechanism. This enhances the model's ability to learn physiologically meaningful spatial representations while maintaining the modeling flexibility of the transformer architecture.

[0131] Unlike graph convolutional networks that rely on fixed neighborhood aggregations, this embodiment enables attention weights to be dynamically modulated based on the spatial arrangement of electrodes, allowing the model to emphasize spatially relevant interactions in a data-driven and interpretable manner. Furthermore, by simultaneously capturing short-range and long-range dependencies through a multi-head attention mechanism with geometrically perceptual bias, the model can better utilize the non-Euclidean structure of the EEG topology. This facilitates more effective integration of spatial information, significantly improving the accuracy of decoding emotional states involving distributed neural processes across different functional brain regions.

[0132] Step 105: Identify the user's emotional information based on the target EEG spatiotemporal characteristics.

[0133] In this embodiment, a classification operation can be performed based on the spatiotemporal characteristics of the target EEG to identify the user's emotional information, such as positive, neutral, negative, etc. The emotional information has a confidence score or intensity score, which is used to reflect the degree of continuous change in the user's emotional state.

[0134] In specific implementations, such as Figure 3 As shown, the target EEG spatiotemporal features are input into a classifier to perform binary or multi-class classification operations to obtain the user's emotional information.

[0135] For example, classifiers include logistic regression layers, support vector machines (SVM), multilayer perceptrons (MLP), and so on.

[0136] Users' emotional information can be output to the application layer. In the application layer, the user's emotional change trend can be displayed in real time on the user interface based on the user's emotional information (especially confidence or intensity scores). Users' emotional information can also be input to subsequent modules (such as large language models, emotion intervention systems, or mental health assessment platforms) to achieve closed-loop monitoring and adaptive feedback of emotional state.

[0137] In this embodiment, multiple electrodes attached to the user's head are used to collect multi-channel EEG signals. Temporal features are extracted from the EEG signals to obtain EEG temporal features. A spatiotemporal topology map of the EEG is constructed based on the correlation between the multiple EEG temporal features and the spatial distribution information of the electrodes. Target spatiotemporal features of the EEG are extracted from the spatiotemporal topology map based on the EEG temporal features and the geometrically perceived attention of the electrodes. The user's emotional information is then identified based on the target spatiotemporal features. This embodiment extracts the temporal dynamic features of the electrodes and uses geometrically perceived attention to model the spatial dependencies between the electrodes, achieving a joint representation of the spatiotemporal features of the EEG signals. This provides a more comprehensive characterization of emotion-related brain activity patterns and achieves collaborative modeling of spatiotemporal features. It can intuitively reflect the relative importance of each brain region in the emotion recognition task, improving the accuracy and stability of emotion recognition, and enhancing the model's generalization and physiological interpretability.

[0138] This embodiment evaluates the performance of EEGGraphormer on two public benchmark datasets for emotion recognition based on EEG signals, DEAP and MAHNOB-HCI. The results show that its performance consistently outperforms existing methods based on CNN, RNN and GCN. These results confirm the effectiveness of incorporating spatial priors into attention-based models to complete EEG decoding tasks.

[0139] The DEAP dataset contains multimodal physiological signals, including EEG, facial expressions, and skin conductance response (GSR). This dataset was recorded from data collected by 32 participants (16 men and 16 women) while watching 40 one-minute music videos. For each participant, the EEG signals were divided into 40 trials, each corresponding to one music video. Each trial lasted one minute and included a 3-second pre-trial baseline period. After each trial, participants completed a questionnaire, reporting their emotional state on a 9-point discrete scale across the dimensions of arousal, valence, dominance, and liking. EEG signals were acquired using a 32-channel device at a sampling rate of 512 Hz.

[0140] MAHNOB-HCI is a multimodal dataset containing 27 participants (11 men and 16 women). Participants were monitored while watching video clips using a camera, a head-mounted microphone, an eye tracker, and physiological sensors that measured electrocardiograms, electroencephalograms (32 channels), respiratory amplitude, and skin temperature.

[0141] Each participant watched 20 film clips, ranging in length from 35 to 117 seconds. EEG signals were acquired using 32 electrodes based on the 10-20 international standard, with a sampling frequency of 256 Hz. For each trial, valence, arousal, dominance, and predictability were denoted using four self-reported integer values ​​from 1 to 9.

[0142] Binary classification tasks were performed on both the DEAP and MAHNOB-HCI datasets to predict low / high arousal levels and negative / positive valence. Trial-level 10-fold cross-validation was used for participant-dependent emotion recognition, meaning that training and testing data came from the same subjects, and data splitting was performed at the trial level to prevent potential data leakage. Trial-level performance was evaluated in all experiments.

[0143] Specifically, the EEG signal for each trial is segmented into 2-second non-overlapping segments. These segments are then fed into a sentiment detection model for feature extraction and sentiment prediction. The final prediction for each trial is obtained through a majority voting mechanism, which aggregates the predicted labels of all segments to determine the final sentiment information for that trial. Evaluation metrics include accuracy and the mean and standard deviation of the F1 score.

[0144] In the channel feature encoder, the depthwise convolution kernel size is 7, followed by pointwise convolution, and the dropout rate is set to 0.5 to prevent overfitting. The graph transform encoder contains two graph encoding layers, each with four head structures and an embedding dimension of 64. The sentiment detection model is trained using the Adam (Adaptive Moment Estimator) optimizer with a learning rate of 1e. -3 The batch size is 32, the training period is 200 rounds, and early stopping is performed based on the validation loss.

[0145] The emotion detection model was evaluated for subject-dependent emotion recognition on the DEAP and MAHNOB-HCI datasets. The performance of EEGGraphormer was compared with CNN, RNN and GNN-based methods, including DeepConvNet, EEGNet, ACRNN, DGCNN, TSception and LGGNet.

[0146] For the DEAP dataset, EEGGraphormer achieves an accuracy of 62.88% in arousal prediction and an F1 score of 63.73%; for valence prediction, it achieves an accuracy of 62.25% and an F1 score of 63.34%. These results demonstrate that EEGGraphormer significantly outperforms baseline methods (including DeepConvNet, EEGNet, ACRNN, DGCNN, TSception, and LGGNet). These performance improvements indicate the effectiveness of the geometry-aware self-attention mechanism in capturing spatial dependencies between EEG electrodes and enhancing the model's ability to learn meaningful spatial representations.

[0147] For the MAHNOB-HCI dataset, EEGGraphormer consistently outperforms all comparable methods on both arousal (63.06% accuracy, 41.72% F1 score) and valence (62.52% accuracy, 42.36% F1 score) classification tasks, further demonstrating EEGGraphormer's robustness and superior spatial modeling capabilities across different EEG emotion recognition benchmarks.

[0148] It should be noted that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0149] Reference Figure 5 The diagram shows a schematic of an EEG emotion recognition device provided in an embodiment of the present invention, which may specifically include the following modules:

[0150] The EEG signal acquisition module 501 is used to call multiple electrodes attached to the user's head to acquire multi-channel EEG signals from the user.

[0151] The EEG timing feature extraction module 502 is used to extract temporal features from the EEG signal to obtain EEG timing features.

[0152] The EEG spatiotemporal topology construction module 503 is used to construct an EEG spatiotemporal topology based on the correlation between the multiple EEG temporal features and the spatial distribution information of the multiple electrodes;

[0153] The target EEG spatiotemporal feature extraction module 504 is used to extract the target EEG spatiotemporal features in the EEG spatiotemporal topology map based on the EEG temporal features and the attention sensed geometrically by the electrodes.

[0154] The emotion information recognition module 505 is used to recognize the user's emotion information based on the target EEG spatiotemporal features.

[0155] In one embodiment of the present invention, the EEG timing feature extraction module 502 includes:

[0156] The first single-channel EEG feature extraction module is used to input the first candidate temporal features of each channel into the deep convolutional layer to extract the first single-channel EEG features; the first candidate temporal features are initially the EEG signal, and if not initially, they are the previous second candidate temporal features.

[0157] The second single-channel EEG feature generation module is used to sequentially nonlinearly activate and normalize each of the first single-channel EEG features to obtain the second single-channel EEG features.

[0158] The third single-channel EEG feature generation module is used to fuse the EEG signal with the second single-channel EEG feature for the same channel into a third single-channel EEG feature.

[0159] The cross-channel EEG feature generation module is used to input the third single-channel EEG features from multiple channels into a convolutional layer for interaction to obtain cross-channel EEG features.

[0160] The EEG timing feature generation module is used to sequentially perform nonlinear activation, normalization and real-time inactivation processing on the cross-channel EEG features of each channel to obtain the second candidate timing feature; the final second candidate timing feature is the EEG timing feature.

[0161] In one embodiment of the present invention, the EEG spatiotemporal topology map construction module 503 includes:

[0162] A node construction module is used to convert the electrodes into nodes in the EEG spatiotemporal topology graph, and to write the EEG temporal features of the corresponding channels of the electrodes into the nodes;

[0163] An edge construction module is used to construct an edge between two nodes in the EEG spatiotemporal topology graph and configure the distance of the edge according to the spatial positional relationship between the two electrodes.

[0164] In one embodiment of the present invention, the edge construction module includes:

[0165] The first distance assignment module is used to calculate the shortest path length between two nodes based on the two-dimensional coordinates of the two nodes on the user's head if there is a spatial adjacency between the two electrodes, and assign the shortest path length as the distance of the edge.

[0166] The second distance assignment module is used to assign a specified value as the distance of the edge if there is no spatial adjacency between the two electrodes; the values ​​are all greater than the shortest path length.

[0167] In one embodiment of the present invention, the target EEG spatiotemporal feature extraction module 504 includes:

[0168] A geometric decay element construction module is used to construct geometric decay elements for multi-head attention mechanisms; each head structure in the multi-head attention mechanism is configured with an independent exponential decay factor, and the geometric decay element is based on the exponential decay factor and the distance of the edge is the exponent.

[0169] The topology normalization module is used to normalize the first intermediate spatiotemporal feature; the first intermediate spatiotemporal feature is initially the EEG temporal feature in the EEG spatiotemporal topology map, and is otherwise the previous second intermediate spatiotemporal feature;

[0170] The node feature update module is used to update the first intermediate spatiotemporal feature to the first candidate EEG spatiotemporal feature in the spatiotemporal topology graph of the EEG according to the geometric decay element and the multi-head attention mechanism if normalization is completed.

[0171] The EEG spatiotemporal feature fusion module is used to fuse the EEG spatiotemporal topology map before normalization with the first candidate EEG spatiotemporal feature into a second candidate EEG spatiotemporal feature.

[0172] The EEG spatiotemporal feature normalization module is used to normalize the second candidate EEG spatiotemporal features;

[0173] The nonlinear transformation module for spatiotemporal features of EEG is used to perform nonlinear transformation on the second candidate spatiotemporal features of EEG after normalization is completed, so as to map them into the third candidate spatiotemporal features of EEG.

[0174] The target EEG spatiotemporal feature fusion module is used to fuse the second candidate EEG spatiotemporal feature before normalization with the third candidate EEG spatiotemporal feature into a second intermediate spatiotemporal feature; the final second intermediate spatiotemporal feature is the target EEG spatiotemporal feature.

[0175] In one embodiment of the present invention, the node feature update module includes:

[0176] The feature mapping module is used to map the first intermediate spatiotemporal features into query vectors, key vectors, and value vectors respectively in the head structure of the multi-head attention mechanism;

[0177] The attention score calculation module is used to modulate the similarity between the query vector of the current node and the key vector of other nodes using the geometric decay element to obtain an attention score;

[0178] The attention weight calculation module is used to normalize the attention score of the current node to obtain the attention weight;

[0179] The weighted summation module is used to add the product of the attention weight of the current node and the value vector of the other nodes to obtain the first candidate EEG spatiotemporal feature.

[0180] In one embodiment of the present invention, the attention score is represented as:

[0181] ;

[0182] Where score(i,j) is the attention score between node i and node j, Q i Let K be the query vector for node i. j Let d be the key vector of node j. k Let M(i,j) be the vector dimension, M(i,j) be the geometric decay element between node i and node j, and T denote the transpose.

[0183] The present invention provides an EEG emotion recognition device, which can realize the steps in the aforementioned EEG emotion recognition method embodiments.

[0184] It should be noted that the module division in the various EEG emotion recognition devices provided in the above embodiments is illustrative and only represents a logical functional division. In actual implementation, other division methods may also be used. Furthermore, the functional modules in the various embodiments of this invention can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0185] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the technical solution of the embodiments of the present invention can be embodied in the form of a computer program product, which is stored in a computer storage medium and includes several instructions to cause an electronic device or processor to execute all or part of the steps of the methods in the various embodiments of the present invention. The aforementioned computer storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0186] Furthermore, the EEG emotion recognition device and the EEG emotion recognition method provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0187] Reference Figure 6 The diagram illustrates an electronic device according to an embodiment of the present invention. Figure 6 As shown, the electronic device in this embodiment of the invention includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps in the above-described EEG emotion recognition method embodiment. Alternatively, when the processor executes the computer program, it implements the functions of each module in the above-described EEG emotion recognition device embodiment.

[0188] For example, the computer program may be divided into one or more modules, which are stored in the memory and executed by the processor to complete this application. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which can be used to describe the execution process of the computer program in the electronic device.

[0189] The electronic device may be a desktop computer, a cloud server, or other computing device. The electronic device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 6 This is merely one example of an electronic device and does not constitute a limitation on the electronic device. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, the electronic device may also include input / output devices, network access devices, buses, etc.

[0190] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0191] The memory can be an internal storage unit of the electronic device, such as a hard drive or RAM. Alternatively, it can be an external storage device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc. Furthermore, the memory can include both internal and external storage units. The memory is used to store the computer program and other programs and data required by the electronic device. The memory can also be used to temporarily store data that has been output or will be output.

[0192] This invention also discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the EEG emotion recognition method as described in the foregoing embodiments.

[0193] This invention also discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements the EEG emotion recognition method as described in the foregoing embodiments.

[0194] This invention also discloses a computer program product that, when run on a computer, causes the computer to execute the EEG emotion recognition method described in the foregoing embodiments.

[0195] The embodiments described above are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A brainwave emotion recognition method, characterized in that, include: Multiple electrodes attached to the user's head are used to collect multi-channel electroencephalogram (EEG) signals from the user; Temporal features are extracted from the electroencephalogram (EEG) signals to obtain EEG temporal features. A spatiotemporal topology map of brainwaves is constructed based on the correlation between the multiple temporal features of brainwaves and the spatial distribution information of the multiple electrodes; Based on the EEG temporal features and the attention sensed geometrically by the electrodes, the target EEG spatiotemporal features are extracted from the EEG spatiotemporal topology map. Identify the user's emotional information based on the target EEG spatiotemporal features; The step of constructing a spatiotemporal topology map of brainwaves based on the correlation between the multiple brainwave temporal features and the spatial distribution information of the multiple electrodes includes: The electrodes are converted into nodes in the spatiotemporal topology of the EEG, and the EEG temporal features of the corresponding channels of the electrodes are written into the nodes. In the spatiotemporal topology of the EEG, an edge is constructed between two nodes, and the distance of the edge is configured according to the spatial positional relationship between the two electrodes; The extraction of target EEG spatiotemporal features based on the EEG temporal features and the attention perceived geometrically by the electrodes in the EEG spatiotemporal topology map includes: A geometric decay element is constructed for the multi-head attention mechanism; each head structure in the multi-head attention mechanism is configured with an independent exponential decay factor, and the geometric decay element is based on the exponential decay factor and the distance of the edge is the exponent. The first intermediate spatiotemporal feature is normalized; the first intermediate spatiotemporal feature is initially the EEG temporal feature in the EEG spatiotemporal topology map, and if it is not initially, it is the previous second intermediate spatiotemporal feature. If normalization is completed, then in the EEG spatiotemporal topology graph, the first intermediate spatiotemporal feature is updated to the first candidate EEG spatiotemporal feature using the multi-head attention mechanism based on the geometric decay element. The first intermediate spatiotemporal feature before normalization is fused with the first candidate EEG spatiotemporal feature to form the second candidate EEG spatiotemporal feature. Normalize the spatiotemporal features of the second candidate EEG; If normalization is completed, the second candidate EEG spatiotemporal features are subjected to nonlinear transformation to be mapped to the third candidate EEG spatiotemporal features. The second candidate EEG spatiotemporal features before normalization are fused with the third candidate EEG spatiotemporal features to form a second intermediate spatiotemporal feature; the final second intermediate spatiotemporal feature is the target EEG spatiotemporal feature.

2. The method according to claim 1, characterized in that, The extraction of temporal features from the EEG signal to obtain EEG temporal features includes: The first candidate temporal features of each channel are respectively input into a deep convolutional layer to extract the first single-channel EEG features; the first candidate temporal features are initially the EEG signals, and otherwise are the previous second candidate temporal features. Each of the first single-channel EEG features is sequentially nonlinearly activated and normalized to obtain the second single-channel EEG features; For the same channel, the EEG signal is fused with the second single-channel EEG feature to form a third single-channel EEG feature; The third single-channel EEG features from multiple channels are input into a convolutional layer for interaction to obtain cross-channel EEG features. The cross-channel EEG features of each channel are sequentially subjected to nonlinear activation, normalization, and dropout processing to obtain the second candidate temporal features; the final second candidate temporal features are the EEG temporal features.

3. The method according to claim 1, characterized in that, The step of configuring the edge distance according to the spatial positional relationship between the two electrodes includes: If two electrodes are spatially adjacent, the shortest path length between the two nodes is calculated based on their two-dimensional coordinates on the user's head, and the shortest path length is assigned as the distance of the edge. If there is no spatial adjacency between the two electrodes, the specified value is assigned as the distance of the edge; all values ​​are greater than the shortest path length.

4. The method according to claim 3, characterized in that, In the EEG spatiotemporal topology map, updating the first intermediate spatiotemporal feature with a first candidate EEG spatiotemporal feature using the multi-head attention mechanism based on the geometric decay element includes: In the head structure of the multi-head attention mechanism, the first intermediate spatiotemporal features are mapped as query vector, key vector and value vector respectively; The similarity between the query vector of the current node and the key vector of other nodes is modulated using the geometric decay element to obtain an attention score; The attention score of the current node is normalized to obtain the attention weight; The attention weight of the current node is added to the product of the value vectors of the other nodes to obtain the first candidate EEG spatiotemporal feature.

5. The method according to claim 4, characterized in that, The attention score is expressed as: ； Where score(i,j) is the attention score between node i and node j, Q i Let K be the query vector for node i. j Let d be the key vector of node j. k Let M(i,j) be the vector dimension, M(i,j) be the geometric decay element between node i and node j, and T denote the transpose.

6. The method according to any one of claims 1-5, characterized in that, The step of identifying the user's emotional information based on the target EEG spatiotemporal features includes: The target EEG spatiotemporal features are input into a classifier to perform a classification operation, thereby obtaining the user's emotional information.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the EEG emotion recognition method as described in any one of claims 1-6.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the EEG emotion recognition method as described in any one of claims 1-6.

Images