Double-branch electroencephalogram emotion recognition method and system based on brain region topology and space-time

Abstract

The invention belongs to the field of artificial intelligence and electroencephalogram emotion recognition, and provides a double-branch electroencephalogram emotion recognition method and system based on brain region topology and time-space, and the method comprises the steps: preprocessing a to-be-recognized electroencephalogram signal to obtain a plurality of electroencephalogram fragments, and extracting a difference entropy sequence of each electroencephalogram fragment and a Spearman correlation coefficient matrix between channels; based on the Spearman correlation coefficient matrix, utilizing a bridging dynamic graph attention network module to extract topological features of a brain region; processing the differential entropy sequence by using a multi-scale space-time mixed attention module to obtain multi-scale space-time features; carrying out residual mutual cross attention fusion on the topological features of the brain region and the multi-scale spatial-temporal features to obtain fusion features; and performing classification based on the fusion features, and determining an emotion recognition result corresponding to the electroencephalogram signal. According to the method, the accuracy and robustness of emotion recognition are improved by utilizing the spatial topology characteristics and the multi-topology time dynamic characteristics of the electroencephalogram signals, and the defects of modeling spatial dependence and time dynamic are overcome.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and brainwave emotion recognition technology, specifically relating to a bi-branch brainwave emotion recognition method and system based on brain region topology and spatiotemporal factors. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Objective identification of human emotional states is of significant value in interpersonal communication, mental health assessment, and intelligent human-computer interaction. Electroencephalograms (EEGs), as direct electrophysiological responses to brain neural activity, provide crucial objective physiological evidence for emotion recognition. EEG-based emotion recognition (EER) technology aims to infer the emotional state of a subject by analyzing collected EEG signals. With the practical development of technologies such as brain-computer interfaces, intelligent health monitoring, and virtual reality, the demand for efficient and stable EER systems is increasingly urgent. Technically, EER is typically structured as a classification task, mapping input EEG signals to predefined emotion categories.

[0004] The current technical bottlenecks of EER systems mainly lie in feature extraction and spatial structure modeling between brain regions. Traditional methods often rely on manually designed time-domain (mean, amplitude, etc.), frequency-domain, or time-frequency features. However, EEG signals are highly non-stationary, multi-channel distributed, and have complex brain region dependencies. Furthermore, emotional responses involve the coordination of multiple brain regions, leading to insufficient discriminative power and robustness of handcrafted features, thus affecting recognition accuracy. In addition, EEG signals are acquired through multiple channels, each corresponding to different brain regions. The spatial topology and functional connectivity information contained within these channels are crucial for emotion discrimination. At the same time, emotional states evolve dynamically over time, and effectively capturing their time-varying characteristics is also a technical challenge. Existing methods still suffer from technical deficiencies in achieving high-precision, cross-subject, and cross-time-period emotion recognition, such as weak model generalization ability and insufficient modeling of brain region associations and temporal dynamics. They cannot more effectively integrate the multi-channel spatial structure of EEG signals, nor can they simultaneously capture their temporal dynamic features. Summary of the Invention

[0005] To address the aforementioned issues, this invention proposes a bi-branch EEG emotion recognition method and system based on brain region topology and spatiotemporal dynamics. This invention fully utilizes the spatial topological features and multi-scale spatiotemporal dynamic features contained in EEG signals, thereby improving the accuracy and robustness of emotion recognition.

[0006] According to some embodiments, the first aspect of the present invention provides a two-branch EEG emotion recognition method based on brain region topology and spatiotemporal factors, employing the following technical solution: A two-branch EEG emotion recognition method based on brain region topology and spatiotemporal factors includes: The EEG signals to be identified are preprocessed to obtain multiple EEG segments, and the differential entropy sequence of each EEG segment and the Spearman correlation coefficient matrix between channels are extracted. Based on the Spearman correlation coefficient matrix, the topological features of brain regions are extracted using the bridging dynamic graph attention network module; Multi-scale spatiotemporal features were obtained by processing the differential entropy sequence of each EEG segment using a multi-scale spatiotemporal hybrid attention module. The brain region topological features and multi-scale spatiotemporal features are fused by residual cross-attention fusion to obtain fused features; Based on the fusion features, the classification determines the emotion recognition result corresponding to the EEG signal.

[0007] Furthermore, the extraction of brain region topological features based on the Spearman correlation coefficient matrix using a bridging dynamic graph attention network module includes: Using the Spearman correlation coefficient matrix as a dynamic adjacency matrix, channels as nodes, and the correlation between channels as the weights of edges, a brain region topology map is constructed. Based on the brain region topology map, three sequentially connected dynamic graph attention network layers are used to perform feature aggregation and update on any pair of nodes in the brain region topology map in turn, so as to obtain the updated features of all brain region nodes. The brain region node update features output by the previous dynamic graph attention network layer are added element-wise to the brain region node update features output by the current dynamic graph attention network layer to obtain the brain region node residual features of the current dynamic graph attention network layer. The residual features of brain region nodes in each layer are nonlinearly mapped to obtain the brain region node mapping features in each layer. The mapping features of each brain region node are weighted and fused, and then multiplied element-wise with the residual features of the last brain region node to obtain the initial brain region topological features. The global features of the initial brain region topological features are then extracted to obtain the brain region topological features.

[0008] Furthermore, the multi-scale spatiotemporal hybrid attention module is used to process the differential entropy sequence of each EEG segment to obtain multi-scale spatiotemporal features, including: The differential entropy feature matrix is ​​composed of differential entropy sequences of each EEG segment. The differential entropy feature matrix is ​​then convolved, and the result of the convolution is then subjected to depthwise separable convolution to obtain the temporal dynamic features. By dividing the time-dynamic features along the channel dimension, the query vector, key vector, and value vector required by the self-attention mechanism are obtained. Multi-scale temporal attention is used to perform multi-scale temporal modeling on value vectors to obtain temporal feature representations; The query vector, key vector, and temporal feature representation are fused and then convolved. The self-attention module is then used to extract the initial multi-scale spatiotemporal features from the convolutional result. Global features are then extracted from the initial multi-scale spatiotemporal features to obtain the multi-scale spatiotemporal features.

[0009] Furthermore, a multi-scale temporal attention module is used to perform multi-scale temporal modeling on the value vector to obtain a temporal feature representation, including: Spatial filtering and convolution are performed on the value vector, and the convolutional feature tensor is divided into three value convolutional feature components. By using three parallel attention head branches to process the three value convolutional feature components, three multi-scale features are obtained; The multi-scale feature representations of the three branches are processed by self-attention to obtain three multi-scale attention features; Based on each multi-scale attention feature, determine the corresponding attention weight coefficient; All multi-scale attention features are weighted and summed based on attention weight coefficients, and then linearly mapped and fused with features through convolution to obtain temporal feature representations.

[0010] Furthermore, the residual cross-attention fusion of brain region topological features and multi-scale spatiotemporal features to obtain fused features includes: Brain region topological features and multi-scale spatiotemporal features are concatenated along the feature dimension, and the concatenation result is mapped to a unified dimension to obtain the concatenated features. A cross-attention mechanism was used to bidirectionally model brain region topological features and multi-scale spatiotemporal features to obtain preliminary fusion features. The initial fused features are processed by depthwise separable convolution and then concatenated with the initial fused features. Two linear transformation layers are used to process the concatenated features to obtain the fused enhanced features. The residuals of the spliced ​​features and the fusion enhancement features are added together to obtain the fusion features.

[0011] Furthermore, the method employs a cross-attention mechanism to bidirectionally model brain region topological features and multi-scale spatiotemporal features, obtaining preliminary fusion features, including: Using brain region topological features as queries and multi-scale spatiotemporal features as keys and values, attention calculation is performed to obtain positive cross-branch attention features. Using multi-scale spatiotemporal features as queries and brain region topological features as keys and values, attention calculation is performed to obtain inverse cross-branch attention features. The positive cross-branch attention features and the negative cross-branch attention features are weighted and fused to obtain the preliminary fused features.

[0012] According to some embodiments, a second aspect of the present invention provides a dual-branch EEG emotion recognition system based on brain region topology and spatiotemporal factors, employing the following technical solution: A bi-branch EEG emotion recognition system based on brain region topology and spatiotemporal factors includes: The EEG signal processing module is used to preprocess the EEG signal to be identified to obtain multiple EEG segments, and extract the differential entropy sequence of each EEG segment and the Spearman correlation coefficient matrix between channels; The brain region topology extraction module is used to extract brain region topological features based on the Spearman correlation coefficient matrix and the bridging dynamic graph attention network module. The multi-scale spatiotemporal extraction module is used to process the differential entropy sequence of each EEG segment using the multi-scale spatiotemporal hybrid attention module to obtain multi-scale spatiotemporal features. The feature fusion module is used to perform residual cross-attention fusion of brain region topological features and multi-scale spatiotemporal features to obtain fused features. The fusion classification module is used to classify based on fusion features and determine the emotion recognition result corresponding to the EEG signal.

[0013] According to some embodiments, a third aspect of the present invention provides a computer-readable storage medium.

[0014] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the bi-branch EEG emotion recognition method based on brain region topology and spatiotemporal as described in the first embodiment above.

[0015] According to some embodiments, a fourth aspect of the present invention provides a computer device.

[0016] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the bi-branch EEG emotion recognition method based on brain region topology and spatiotemporal as described in the first embodiment above.

[0017] According to some embodiments, a fifth aspect of the present invention provides a computer program product or computer program.

[0018] A computer program product or computer program includes computer instructions stored in a computer-readable storage medium, wherein a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps in the bi-branch EEG emotion recognition method based on brain region topology and spatiotemporal as described in the first embodiment above.

[0019] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention employs a dual-branch parallel processing architecture, technically solving the problem that existing methods struggle to simultaneously consider the spatial topological relationships and complex temporal dynamics of EEG signals. The first branch constructs a functional connectivity graph based on Spearman's correlation coefficient using differential entropy, quantifying and structuring the nonlinear collaborative relationships between brain regions. The second branch, through a multi-scale temporal attention module, simultaneously captures both instantaneous changes and long-term trends in emotion-related EEG signals. This collaborative design ensures the complete representation of emotion-related neural patterns, achieving collaborative modeling of spatial functional connectivity and multi-scale temporal dynamics of brain regions.

[0020] Compared to methods that rely on fixed brain region priors or simple spatial convolutions, the Bridged Dynamic Graph Attention Network (BDGAT) employed in this invention represents a key technological advancement. This network can dynamically learn the importance weights of functional connections between brain regions under different emotional states and enhance information exchange between distant brain regions using bridging mechanisms. This allows for a more accurate characterization of the activity patterns of distributed brain networks closely related to emotion processing, significantly improving the discriminative power of spatial features and enhancing the accuracy and adaptability of spatial feature extraction.

[0021] The multi-scale temporal attention (MTA) module of this invention extracts temporal features of different granularities through parallel convolutional kernels and adaptively focuses on key time segments most relevant to emotions using a multi-head attention mechanism. This effectively overcomes the interference caused by signal non-stationarity, enhances the model's understanding of the emotional evolution process, and strengthens the capture of key information of multi-scale temporal patterns.

[0022] The Residual Cross-Attention (RMCA) module designed in this invention goes beyond simple feature concatenation or addition, creatively achieving deep interaction and complementarity between features of two branches. This module allows spatial and temporal features to guide and enhance each other, thereby generating more robust fused representations. The introduction of residual connections ensures training stability, enabling the model to build deeper network layers to improve performance, and achieves information complementarity and performance gain through a deep feature fusion mechanism. Attached Figure Description

[0023] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0024] Figure 1 This is a flowchart of the bi-branch EEG emotion recognition method based on brain region topology and spatiotemporal in an embodiment of the present invention; Figure 2 This is a flowchart of the bi-branch EEG emotion recognition method based on brain region topology and spatiotemporal structure provided in this embodiment of the invention. Figure 3 This is a schematic diagram of the internal structure of the bridging dynamic diagram network module provided in the embodiment of the present invention; Figure 4 This is a schematic diagram of the overall structure of the multi-scale spatiotemporal hybrid attention module provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the internal structure of the multi-scale temporal attention module provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the internal structure of the residual cross-attention fusion module provided in an embodiment of the present invention. Detailed Implementation

[0025] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0026] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0027] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0028] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0029] Example 1 like Figure 1 and Figure 2As shown, this embodiment provides a bi-branch EEG emotion recognition method based on brain region topology and spatiotemporal data. This embodiment uses the application of this method to a server as an example for illustration. It is understood that this method can also be applied to terminals, and can also be applied to systems including terminals, servers, and other components, and can be implemented through interaction between the terminal and the server. The server can be an independent physical server, a server cluster composed of multiple physical servers, or a distributed system. It can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network servers, cloud communication, middleware services, domain name services, CDN security services, and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein. In this embodiment, the method includes the following steps: Step S1: Preprocess the EEG signal to be identified to obtain multiple EEG segments, and extract the differential entropy sequence of each EEG segment and the Spearman correlation coefficient matrix between channels, specifically including: Step S11: Acquire the EEG signal to be identified and preprocess it to obtain multiple EEG segments, as follows: The EEG signal to be identified is denoised, including removing power line interference, electromyographic noise and eye movement artifacts; Bandpass filtering is performed on the denoised EEG signal to preserve specific frequency bands. band; The filtered EEG signal is segmented into multiple EEG segments according to a fixed window; the segmentation process ensures that the signal in each window has sufficient time information.

[0030] Specifically, raw EEG signals are collected by a multi-channel EEG acquisition device while the subject is watching or listening to emotionally stimulating materials (such as videos, music, or pictures). The sampling frequency is preferably 200Hz to 1000Hz to ensure time resolution.

[0031] The raw EEG signal is subjected to artifact removal (denoising), bandpass filtering, and segmentation. Preferably, artifacts such as eye movement, electromyography, and power supply interference can be removed by independent component analysis (ICA); then, a bandpass filter of 0.5–75 Hz is used to remove low-frequency drift and high-frequency noise. Subsequently, the continuous EEG signal is divided into several EEG segments according to a sliding window of fixed duration for subsequent feature extraction.

[0032] Step S12: Calculate the frequency bands of each channel within each EEG segment. The differential entropy features are used to form the differential entropy sequence of each EEG segment; this is used to characterize the energy distribution of EEG signals at different frequency rhythms.

[0033] Among them, the differential entropy features of each frequency band The calculation formula is as follows:

[0034] in, This represents the mean of the signal within the time window. Indicates standard deviation, The probability distribution function representing the electroencephalogram (EEG) signal. This represents the sampled value of the electroencephalogram (EEG) signal. Indicates the start and end points of the time window. Indicates signal It follows a normal distribution.

[0035] Based on the differential entropy characteristics of all channels across all frequency bands within each EEG segment, a differential entropy sequence for each EEG segment is constructed. , Indicates the first The differential entropy sequence of EEG segments, with a size of This contains the differential entropy feature sequence of each channel across all frequency bands; these differential entropy features Used to characterize the complexity of the signal in each frequency band.

[0036] Step S13: Based on the differential entropy sequence of each EEG segment, calculate the Spearman correlation coefficient matrix between any two channels within each EEG segment, including: The differential entropy feature sequences of each channel in each differential entropy sequence are sorted by rank to obtain the rank value of each channel; The Spearman correlation coefficient between channels is determined by calculating the difference between the rank values ​​of any two channels, in order to quantify the nonlinear dependence between channels; For different channels and In a certain brainwave segment The difference entropy feature sequences on are denoted as follows: and And sort them by rank respectively to obtain and This represents the rank of the two channels within the same time segment. Then, the difference between the two rank values ​​is calculated:

[0037] The Spearman correlation coefficient measures the monotonic relationship between the ranks of the differential entropy sequences of two channels after sorting transformation. The calculation formula is as follows:

[0038] in, Indicates channel and channels The rank difference of the corresponding difference entropy feature sequences, This indicates the number of channels involved in the calculation.

[0039] The Spearman correlation coefficients between all channels are used to construct a Spearman correlation coefficient matrix that varies over time.

[0040] Step S2: Based on the Spearman correlation coefficient matrix, extract brain region topological features using the bridging dynamic graph attention network module, including: The first branch extracts brain region topological features from the Spearman correlation coefficient matrix. A bridging dynamic graph attention network module is used to extract these features. This module comprises a dynamic graph attention network layer, residual connection layers, and bridging blocks. It models the topological dependencies between brain regions within the dynamic graph structure, mitigating the over-smoothing problem of graph features during multi-layer propagation and enhancing the discriminativeness and diversity of brain region features. Figure 3 As shown, the dynamic graph attention network has 3 layers, and each dynamic graph attention network layer has 4 attention heads. After each dynamic graph attention network layer, a residual connection layer and a bridging block are connected.

[0041] Step S21: Using the Spearman correlation coefficient matrix as a dynamic adjacency matrix, with channels as nodes and the correlation between channels as edge weights, construct a brain region topology map. Specifically, the Spearman correlation coefficient matrix middle, Number of EEG signal channels Represents the number of EEG segments; each EEG segment matrix Viewed as a dynamic adjacency matrix of a graph, in each EEG segment matrix Each element Indicates channel The Spearman correlation coefficient between the brainwave segment and other channels. This value measures the correlation between the brainwave segment and other channels. In the middle, the first Correlation between channels.

[0042] Step S22: Based on the brain region topology map, use three sequentially connected dynamic graph attention network layers to perform feature aggregation and update on any pair of nodes in the brain region topology map in sequence to obtain the updated features of all brain region nodes. The processing procedure of a single dynamic graph attention network layer is illustrated as follows: By utilizing each dynamic graph attention network layer, the initial node features of nodes in the brain region topology map are aggregated and updated to obtain multi-head aggregated node features. The process is as follows: Using each attention head for any node pair in the brain region topology map The attention weights are calculated as the weights of the edges between nodes, using the following formula:

[0043] in, Represents the self-attention mapping function. The weight matrix is ​​a learnable matrix. It is for nodes and nodes Initial node features and Through the weight matrix The result after linear transformation. It is a node and nodes The weight of the edge between nodes represents the weight of the node. and nodes The correlation between them.

[0044] Attention coefficients are obtained by normalizing the edges between nodes using softmax. The specific formula is as follows:

[0045] Subsequently, a weighted aggregation update is performed based on the attention coefficients to obtain the node features, as shown in the following formula:

[0046] in, express The attention coefficients after softmax normalization represent the actual weights between nodes. Represents a node and nodes Attention weights between them; Represents a node The updated feature representation is obtained by analyzing all nodes. Initial node features The result is obtained by weighted summation.

[0047] Aggregate all node features to obtain the output of each dynamic graph attention network layer, which serves as the node's output. Multi-head aggregation node characteristics The formula is as follows:

[0048] in, This indicates a feature concatenation operation. For the number of attention heads, Indicates the first Nodes in each attention head and nodes Attention coefficients between brain regions; This embodiment uses a multi-head dynamic graph attention mechanism to further capture brain region dependencies, thereby improving the robustness of the self-attention mechanism.

[0049] Step S23: Add the brain region node update features output by the previous dynamic graph attention network layer to the brain region node update features output by the current dynamic graph attention network layer element-wise to obtain the brain region node residual features of the current dynamic graph attention network layer. ,as follows:

[0050]

[0051] in, It is the first Pay attention to the brain region node update features output by the layer dynamic graph. It is the first Pay attention to the brain region node update features output by the layer dynamic graph. It is the first Residual features of brain region nodes in the layer. A residual connection structure is introduced between each dynamic graph attention network layer, adding the output of the previous layer to the result of the current layer element-wise, in order to alleviate the over-smoothing phenomenon in deep graph networks and maintain the stability and information diversity of node features.

[0052] Step S24: Perform nonlinear mapping on the residual features of brain region nodes in each layer to obtain the mapped features of brain region nodes in each layer. ,as follows:

[0053] in, The learnable parameter matrix of the bridging layer. For activation function, This is a layer normalization operation.

[0054] After each dynamic graph attention network layer, a connection bridging block is set up to perform nonlinear mapping on the residual features of each brain region node, which is used to enhance the discriminativeness of brain region features and improve the nonlinear expression ability of deep features.

[0055] Step S25: Weight and fuse the mapping features of brain region nodes in each layer, and multiply them element-wise with the residual features of the last brain region node to obtain the initial brain region topological features. The calculation formula is as follows:

[0056] in, This indicates an element-wise multiplication operation, here In the dynamic graph, note the number of network layers; here it's set to 3. Represents the residual features of brain region nodes in the last layer. Initial brain region topological features. It can integrate topological information at different levels and use it as input to the subsequent Transformer module for further global feature modeling to obtain brain region topological features. .

[0057] Step S3: Process the differential entropy sequence of each EEG segment using a multi-scale spatiotemporal hybrid attention module to obtain multi-scale spatiotemporal features. Specific steps are as follows: Figure 4 As shown, it includes: Step S31: Based on the differential entropy sequence of each EEG segment, a differential entropy feature matrix is ​​formed. The differential entropy feature matrix is ​​convolved, and the result after convolution is then subjected to depthwise separable convolution to obtain the temporal dynamic features.

[0058] For the differential entropy feature matrix Perform convolutional encoding, where, Indicates the number of EEG segments, Indicates the number of brainwave channels. This represents the number of frequency bands. First, a... The convolutional layers expand the input features through channels, thereby improving the resolution and expressive power of the spatial relationships in EEG signals. Subsequently, depthwise separable convolutions are used to extract temporal features within each channel, yielding temporal dynamic features. The calculation process can be expressed as follows:

[0059] in, express Convolutional layer This indicates a depthwise separable convolutional layer.

[0060] Step S32: Utilize time-dynamic features By dividing along the channel dimension, we obtain the query vector, key vector, and value vector required by the self-attention mechanism. The query vector and key vector are normalized and then fused to obtain the attention map. Time dynamic characteristics By partitioning along the channel dimension, we obtain the query, key, and value vectors required for the self-attention mechanism, as shown in the following formula:

[0061] in, This indicates a partitioning operation along the channel dimension.

[0062] Step S33: Perform multi-scale temporal modeling on the value vector using multi-scale temporal attention to obtain temporal feature representation. ,include: like Figure 5 As shown, a multi-scale temporal attention (MTA) module is introduced to address the value vector. Multi-scale temporal modeling is performed to comprehensively capture short-term and long-term dependencies in EEG signals, thereby obtaining more context-informed temporal feature representations. .like Figure 5 As shown, the MTA module first performs spatial filtering on the input features to reduce the channel dimension, thereby reducing redundant information and improving computational efficiency; subsequently, through... The convolutional layer restores the channel dimension to maintain the consistency of the overall feature mapping. Based on this, the convolutional feature tensor is divided into three value convolutional feature components; three parallel attention head branches are then used to process these three value convolutional feature components. Among them, attention head Each attention head branch corresponds to a temporal convolution kernel of a different scale, used to extract dynamic features within different time ranges.

[0063] Specifically, each attention head branch employs one-dimensional temporal convolutions with kernel sizes of 2, 4, and 6 to capture the short-, medium-, and long-term temporal dependencies of EEG signals. After convolution, the output features are sequentially subjected to layer normalization and exponential linear unit activation (ELU) operations to enhance the model's nonlinear expressive power and stability, thereby obtaining normalized multi-scale features. The specific process is as follows:

[0064]

[0065] in, This indicates a partitioning operation along the channel dimension. It represents multi-scale features.

[0066] The multi-scale feature representations output from the three branches are then processed through self-attention calculations to obtain multi-scale attention features. This serves as the final output feature of different attention head branches.

[0067] A soft-gating fusion mechanism is employed to adaptively fuse the output features of different attention head branches, thereby fully utilizing multi-scale feature information and enhancing feature representation capabilities. Specifically, for each attention head... First, global average pooling is used to extract... The overall feature distribution is then used to calculate the corresponding attention weight coefficients through a linear transformation with learnable parameters and a sigmoid activation function. The calculation process is as follows:

[0068] in, This indicates a global average pooling operation. and These are the learnable weight matrix and the bias term, respectively. This is the Sigmoid activation function.

[0069] Multiscale attention features of all heads We perform weighted summation and linear mapping with 1×1 convolution to fuse features, thus obtaining the temporal feature representation. The calculation process is as follows:

[0070] in, This represents element-wise multiplication. express Convolution operations are used to reproject weighted multi-head features onto a unified embedding dimension; Step S34: After fusing the attention map and temporal feature representation, perform convolution processing, and then use a self-attention module to extract multi-scale spatiotemporal features from the convolution result, including: Will Compared with the previously generated and The initial multi-scale spatiotemporal features are obtained by inputting them into the self-attention module. The specific formula is as follows:

[0071]

[0072] in, This represents an unbiased 1×1 convolutional layer. This represents the softmax function. It is a learnable scaling parameter. The resulting initial multi-scale spatiotemporal features The input is then fed into the Transformer module to extract global dependencies, yielding the final output of the second branch, which is the multi-scale spatiotemporal features. .

[0073] Step S4: Perform residual cross-attention fusion on brain region topological features and multi-scale spatiotemporal features to obtain fused features. This is implemented based on the residual cross-attention fusion module. Specific steps are as follows: Figure 6As shown, it includes: Step S41: Concatenate the brain region topological features and multi-scale spatiotemporal features along the feature dimension, and map the concatenation result to a unified dimension to obtain the concatenated features; The brain region topological features of EEG signals are denoted as... The multi-scale spatiotemporal characteristics of EEG signals are denoted as ,in Indicates the number of brainwave channels. and These represent the feature dimensions of the two branches. First, the two sets of features are concatenated along their feature dimensions, and then mapped to a unified dimension using a fully connected layer. The specific formula is as follows:

[0074] in, It is a splicing feature.

[0075] Step S42: Employ a cross-attention mechanism to perform bidirectional modeling of brain region topological features and multi-scale spatiotemporal features to obtain preliminary fusion features; To capture the complementary dependencies between the two branches, a cross-attention mechanism is used to model the features bidirectionally.

[0076] First, using brain region topological features as queries and multi-scale spatiotemporal features as keys and values, attention calculation is performed to obtain positive cross-branch attention features. ; Next, in the reverse computation, multi-scale spatiotemporal features are used as queries, and brain region topological features are used as keys and values ​​to perform attention calculations and obtain reverse cross-branch attention features. ; Finally, the positive cross-branch attention features are... and reverse cross-branch attention features Weighted fusion was performed to obtain preliminary fusion characteristics. .

[0077] The attention calculation can be performed using any attention formula in the existing technology, such as scaled dot product attention.

[0078] Step S43: Perform depthwise separable convolution on the initial fused features and then concatenate them with the initial fused features. Use two linear transformation layers to process the concatenated features to obtain the fused enhanced features. ; To enhance the nonlinear modeling capability and local spatial awareness capability of features, The input is fed into the inverse residual multilayer perceptron module (IRMLP), through a network with residual connections. The system employs depthwise separable convolutions and two linear transformation layers to expand and then compress the channels, followed by a non-linear transformation using the GELU activation function along the expanded channel dimension. The computation process is as follows:

[0079] in, It's a Dropout operation to prevent the model from overfitting; Indicates a fully connected operation; This represents depthwise convolution.

[0080] By designing the inverted residual structure, the model can introduce nonlinear transformations and enhance local feature responses while maintaining consistency in feature dimensions, thereby improving the expressive richness of the fused features.

[0081] Step S44: Add the residuals of the spliced ​​features and the fusion enhancement features to obtain the fusion features; To ensure complete information transmission and enhance feature stability, the splicing features in step S41 will be... With fusion enhancement features By summing the residuals, the fusion features are obtained. The specific formula is as follows:

[0082] This residual connection not only ensures dimensionality consistency but also effectively fuses linear and nonlinear feature information, further improving the model's ability to capture cross-branch associations and its performance in discriminating sentiment features. The final fused features... Used for final sentiment classification.

[0083] Step S5: Classify based on fusion features to determine the emotion recognition result corresponding to the EEG signal, including: Fusion features The data is flattened and fed into a fully connected network, transforming the output into a one-dimensional vector. Specifically, the flattened features are sequentially input into a three-layer fully connected network for non-linear mapping, and finally pass through a Softmax network layer for EEG emotion classification. The Softmax network layer is a general network structure that represents the output in the interval [0, 1], finding the dimension with the highest probability as the final output result, i.e., the emotion recognition result corresponding to the EEG signal. :

[0084] in, Indicates a fully connected operation. This represents the final emotion recognition result of the model.

[0085] Example 2 This embodiment provides a two-branch EEG emotion recognition system based on brain region topology and spatiotemporal factors, including: The EEG signal processing module is used to preprocess the EEG signal to be identified to obtain multiple EEG segments, and extract the differential entropy sequence of each EEG segment and the Spearman correlation coefficient matrix between channels; The brain region topology extraction module is used to extract brain region topological features based on the Spearman correlation coefficient matrix and the bridging dynamic graph attention network module. The multi-scale spatiotemporal extraction module is used to process the differential entropy sequence of each EEG segment using the multi-scale spatiotemporal hybrid attention module to obtain multi-scale spatiotemporal features. The feature fusion module is used to perform residual cross-attention fusion of brain region topological features and multi-scale spatiotemporal features to obtain fused features. The fusion classification module is used to classify based on fusion features and determine the emotion recognition result corresponding to the EEG signal.

[0086] The examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1 above. It should be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer-executable instructions.

[0087] The descriptions of each embodiment in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0088] The proposed system can be implemented in other ways. For example, the system embodiments described above are merely illustrative, and the division of modules described above is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed.

[0089] Example 3 This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the bi-branch EEG emotion recognition method based on brain region topology and spatiotemporal as described in Embodiment 1 above.

[0090] Example 4 This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the bi-branch EEG emotion recognition method based on brain region topology and spatiotemporal as described in Embodiment 1 above.

[0091] Example 5 This embodiment provides a computer program product or computer program, including computer instructions stored in a computer-readable storage medium. The processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps in the bi-branch EEG emotion recognition method based on brain region topology and spatiotemporal as described in Embodiment 1 above.

[0092] Those skilled in the art will understand that embodiments of the present invention can provide methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0093] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0094] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0095] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0096] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0097] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A bi-branch EEG emotion recognition method based on brain region topology and spatiotemporal factors, characterized in that, include: The EEG signals to be identified are preprocessed to obtain multiple EEG segments, and the differential entropy sequence of each EEG segment and the Spearman correlation coefficient matrix between channels are extracted. Based on the Spearman correlation coefficient matrix, the topological features of brain regions are extracted using the bridging dynamic graph attention network module; Multi-scale spatiotemporal features were obtained by processing the differential entropy sequence of each EEG segment using a multi-scale spatiotemporal hybrid attention module. The brain region topological features and multi-scale spatiotemporal features are fused by residual cross-attention fusion to obtain fused features; Based on the fusion features, the classification determines the emotion recognition result corresponding to the EEG signal.

2. The bi-branch EEG emotion recognition method based on brain region topology and spatiotemporal factors as described in claim 1, characterized in that, The method for extracting brain region topological features based on the Spearman correlation coefficient matrix and utilizing a bridging dynamic graph attention network module includes: Using the Spearman correlation coefficient matrix as a dynamic adjacency matrix, channels as nodes, and the correlation between channels as the weights of edges, a brain region topology map is constructed. Based on the brain region topology map, three sequentially connected dynamic graph attention network layers are used to perform feature aggregation and update on any pair of nodes in the brain region topology map in turn, so as to obtain the updated features of all brain region nodes. The brain region node update features output by the previous dynamic graph attention network layer are added element-wise to the brain region node update features output by the current dynamic graph attention network layer to obtain the brain region node residual features of the current dynamic graph attention network layer. The residual features of brain region nodes in each layer are nonlinearly mapped to obtain the brain region node mapping features in each layer. The mapping features of each brain region node are weighted and fused, and then multiplied element-wise with the residual features of the last brain region node to obtain the initial brain region topological features. The global features of the initial brain region topological features are then extracted to obtain the brain region topological features.

3. The bi-branch EEG emotion recognition method based on brain region topology and spatiotemporal factors as described in claim 1, characterized in that, The method of using a multi-scale spatiotemporal hybrid attention module to process the differential entropy sequence of each EEG segment yields multi-scale spatiotemporal features, including: The differential entropy feature matrix is ​​composed of differential entropy sequences of each EEG segment. The differential entropy feature matrix is ​​then convolved, and the result of the convolution is then subjected to depthwise separable convolution to obtain the temporal dynamic features. By dividing the time-dynamic features along the channel dimension, the query vector, key vector, and value vector required by the self-attention mechanism are obtained. Multi-scale temporal attention is used to perform multi-scale temporal modeling on value vectors to obtain temporal feature representations; The query vector, key vector, and temporal feature representation are fused and then convolved. The self-attention module is then used to extract the initial multi-scale spatiotemporal features from the convolutional result. Global features are then extracted from the initial multi-scale spatiotemporal features to obtain the multi-scale spatiotemporal features.

4. The bi-branch EEG emotion recognition method based on brain region topology and spatiotemporal factors as described in claim 3, characterized in that, Multi-scale temporal attention modules are used to perform multi-scale temporal modeling on value vectors to obtain temporal feature representations, including: Spatial filtering and convolution are performed on the value vector, and the convolutional feature tensor is divided into three value convolutional feature components. By using three parallel attention head branches to process the three value convolutional feature components, three multi-scale features are obtained; The multi-scale feature representations of the three branches are processed by self-attention to obtain three multi-scale attention features; Based on each multi-scale attention feature, determine the corresponding attention weight coefficient; All multi-scale attention features are weighted and summed based on attention weight coefficients, and then linearly mapped and fused with features through convolution to obtain temporal feature representations.

5. The bi-branch EEG emotion recognition method based on brain region topology and spatiotemporal factors as described in claim 1, characterized in that, The process of fusing brain region topological features and multi-scale spatiotemporal features through residual cross-attention fusion to obtain fused features includes: Brain region topological features and multi-scale spatiotemporal features are concatenated along the feature dimension, and the concatenation result is mapped to a unified dimension to obtain the concatenated features. A cross-attention mechanism was used to bidirectionally model brain region topological features and multi-scale spatiotemporal features to obtain preliminary fusion features. The initial fused features are processed by depthwise separable convolution and then concatenated with the initial fused features. Two linear transformation layers are used to process the concatenated features to obtain the fused enhanced features. The residuals of the spliced ​​features and the fusion enhancement features are added together to obtain the fusion features.

6. The bi-branch EEG emotion recognition method based on brain region topology and spatiotemporal factors as described in claim 5, characterized in that, The method employs a cross-attention mechanism to bidirectionally model brain region topological features and multi-scale spatiotemporal features, obtaining preliminary fused features, including: Using brain region topological features as queries and multi-scale spatiotemporal features as keys and values, attention calculation is performed to obtain positive cross-branch attention features. Using multi-scale spatiotemporal features as queries and brain region topological features as keys and values, attention calculation is performed to obtain inverse cross-branch attention features. The positive cross-branch attention features and the negative cross-branch attention features are weighted and fused to obtain the preliminary fused features.

7. A dual-branch EEG emotion recognition system based on brain region topology and spatiotemporal factors, characterized in that, include: The EEG signal processing module is used to preprocess the EEG signal to be identified to obtain multiple EEG segments, and extract the differential entropy sequence of each EEG segment and the Spearman correlation coefficient matrix between channels; The brain region topology extraction module is used to extract brain region topological features based on the Spearman correlation coefficient matrix and the bridging dynamic graph attention network module. The multi-scale spatiotemporal extraction module is used to process the differential entropy sequence of each EEG segment using the multi-scale spatiotemporal hybrid attention module to obtain multi-scale spatiotemporal features. The feature fusion module is used to perform residual cross-attention fusion of brain region topological features and multi-scale spatiotemporal features to obtain fused features. The fusion classification module is used to classify based on fusion features and determine the emotion recognition result corresponding to the EEG signal.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the bi-branch EEG emotion recognition method based on brain region topology and spatiotemporal as described in any one of claims 1-6.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the bi-branch EEG emotion recognition method based on brain region topology and spatiotemporal as described in any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps in the bi-branch EEG emotion recognition method based on brain region topology and spatiotemporal as described in any one of claims 1-6.

Images