An electroencephalogram signal emotion recognition method and system, a computer device and a medium

Abstract

The application provides an electroencephalogram emotion recognition method, system, computer device and medium, and belongs to the field of emotion computing, and comprises the following steps: obtaining a data-driven adjacency matrix by calculating the self-similarity of an input feature tensor corresponding to each electrode node; fusing the data-driven adjacency matrix and a structure prior adjacency matrix to obtain a sparse dynamic adjacency matrix; performing diffusion convolution on the sparse dynamic adjacency matrix to obtain a spatial feature tensor; dividing the spatial feature tensor into two complementary feature components; obtaining a time sequence feature by multi-level time sequence convolution, performing diffusion convolution on the time sequence feature to obtain a space-time coupling feature of the feature components; taking the space-time coupling feature of the feature components as a gating weight to modulate the feature components, so as to obtain a space-time cross fusion feature component of the feature components; and determining an emotion category prediction probability distribution corresponding to the electroencephalogram according to the space-time cross fusion feature. The method effectively captures complementary enhancement information between space-time features, and improves the precision of electroencephalogram emotion recognition.

Description

Technical Field

[0001] This invention belongs to the field of affective computing, specifically relating to a method, system, computer device, and medium for emotion recognition based on electroencephalogram (EEG) signals. Background Technology

[0002] Emotion recognition is a core research area in affective computing, with significant applications in human-computer interaction, psychology, and clinical medicine. Electroencephalograms (EEGs), which directly reflect brain neural activity and are unaffected by subjective will, serve as a crucial physiological carrier for emotion recognition.

[0003] To address the need for emotion recognition from EEG signals, existing technologies employ deformable convolution and attention mechanisms to extract and fuse multi-scale features from EEG signals, combined with bidirectional gated recurrent units (GRUs) to capture temporal semantic information. However, EEG signals are essentially products of spatiotemporal coupling; temporal changes in EEG signals inevitably accompany changes in spatial topology, and spatial connections inherently carry information about temporal evolution. GRUs first extract global temporal dependencies using a temporal network, then use the output of the temporal network as input to a spatial network for spatial feature extraction. When temporal features are compressed and independently transmitted to the spatial network, the dynamic information in the temporal dimension loses its real-time correspondence with the spatial topology. This results in the spatial network being unable to perceive the driving effect of temporal changes on spatial activation patterns, effectively separating temporal and spatial features into two isolated stages. Consequently, the complementary and enhancing information between the two cannot be effectively captured, leading to limited accuracy in emotion recognition. Summary of the Invention

[0004] To address the limited accuracy of existing emotion recognition methods, this invention provides a method, system, computer device, and medium for emotion recognition based on electroencephalogram (EEG) signals.

[0005] To achieve the above objectives, the present invention provides the following technical solution: A method for emotion recognition using electroencephalogram (EEG) signals includes: The system acquires EEG signals from multiple electrode channels; constructs an input feature tensor based on the EEG signals; inputs the input feature tensor into DSTCFNet to obtain the probability distribution of emotion category prediction. DSTCFNet includes a dynamic graph construction module, a spatiotemporal cross-fusion module, and a classification output module. The sparse dynamic adjacency matrix is ​​constructed from the input feature tensor by the dynamic graph construction module, and the spatial feature tensor is obtained by graph convolution based on the sparse dynamic adjacency matrix. The spatial feature tensor is divided into two complementary feature components through a spatiotemporal cross-fusion module. For each feature component, a temporal feature is obtained through multi-level temporal convolution. The temporal feature is then subjected to a diffusion convolution to obtain the spatiotemporal coupling feature of the feature component. Using the spatiotemporal coupling feature of each feature component as the Tanh gate weight, the other feature component is modulated element-wise to obtain the spatiotemporal cross-fusion feature component of the feature component. The two spatiotemporal cross-fusion feature components are then fused to obtain the spatiotemporal cross-fusion feature. The classification output module determines the predicted probability distribution of the emotion category corresponding to the EEG signal based on the spatiotemporal cross-fusion characteristics.

[0006] Optionally, the EEG signal emotion recognition method provided by the present invention further includes: The input feature tensor is aggregated along the time dimension to obtain the node feature matrix; the self-similarity between each electrode node is determined by the inner product of the node feature matrix; the self-similarity between each electrode node is normalized by Softmax to obtain the data-driven adjacency matrix. The sparse dynamic adjacency matrix is ​​obtained by fusing the data-driven adjacency matrix with the structural prior adjacency matrix determined based on the spatial positional relationship between each electrode node. Diffusion convolution is performed on the sparse dynamic adjacency matrix to obtain the spatial feature tensor.

[0007] Optionally, the EEG signal emotion recognition method provided by the present invention further includes: Using a sparse dynamic adjacency matrix as weights, the input feature tensor is subjected to multiple directed diffusions to obtain multiple multi-scale spatial feature tensors. Multiple multi-scale spatial feature tensors are concatenated along the channel dimension to obtain a concatenated multi-scale spatial feature tensor. Convolution is then performed on the concatenated multi-scale spatial feature tensor to obtain a multi-scale spatial feature tensor with reduced dimensionality after convolution. The multi-scale spatial feature tensor after convolutional dimensionality reduction is sequentially subjected to Dropout regularization, concatenated with the residual of the input feature tensor, and activated by the ReLU function to obtain the spatial feature tensor.

[0008] Optionally, the EEG signal emotion recognition method provided by the present invention further includes: The attention weights of the two branches are determined using a lightweight convolutional network; The spatial feature tensor is weighted and grouped according to the attention weights to obtain the complementary feature components corresponding to the two branches. For each feature component, the feature component is processed by two-level multi-scale temporal convolution to obtain temporal features. The kernel sizes of the two-level multi-scale temporal convolution are different. A dynamic graph is constructed based on temporal features to obtain a sample-specific sparse dynamic adjacency matrix. Multi-order directed diffusion convolution is then performed on the sample-specific sparse dynamic adjacency matrix to obtain the spatiotemporal coupling features of the feature components.

[0009] Optionally, the EEG signal emotion recognition method provided by the present invention further includes: The spatiotemporal coupling feature of one feature component is activated by the Tanh function to obtain the Tanh gate weight of another feature component. The feature component is then modulated according to the Tanh gate weight to obtain the interactively modulated feature component. Temporal graph convolution and dynamic graph diffusion are performed on the interactively modulated feature components to obtain the spatiotemporally cross-fused feature components after dynamic graph diffusion. The spatiotemporal cross-fusion feature components after dynamic graph diffusion of each branch are distributed and calibrated through the feature components after interactive modulation of another branch to obtain the spatiotemporal cross-fusion feature components.

[0010] Optionally, the EEG signal emotion recognition method provided by the present invention further includes: Convolutional processing is performed on the spatiotemporal cross-fusion feature components to obtain the spatiotemporal cross-fusion feature components with adjusted dimensions; Average pooling is performed on the spatiotemporal cross-fusion feature components after dimensional adjustment to obtain spatiotemporal cross-fusion feature components aligned with the time dimension. We obtain time-aligned spatiotemporal cross-fusion features by weighted fusion of time-dimensional aligned spatiotemporal cross-fusion components; and we obtain spatiotemporal cross-fusion features by linking time-dimensional aligned spatiotemporal cross-fusion features with input feature tensor residuals.

[0011] Optionally, the EEG signal emotion recognition method provided by the present invention further includes: The spatiotemporal cross-fusion features are filtered by gated linear units to obtain the filtered features; The filtered features are linked with the spatiotemporal cross-fusion feature residuals, and then global average pooling is used to obtain the global feature vector. The global feature vector is processed sequentially by a fully connected layer and activated by the Softmax function to obtain the probability distribution of the predicted emotion category corresponding to the EEG signal.

[0012] The present invention also provides an EEG signal emotion recognition system, comprising: The data acquisition module is used to acquire EEG signals from multiple electrode channels; construct an input feature tensor based on the EEG signals; input the input feature tensor into DSTCFNet to obtain the probability distribution of emotion category prediction. DSTCFNet includes a dynamic graph construction module, a spatiotemporal cross-fusion module, and a classification output module.

[0013] The feature extraction module constructs a sparse dynamic adjacency matrix from the input feature tensor through the dynamic graph construction module, and performs graph convolution on the input feature tensor based on the sparse dynamic adjacency matrix to obtain the spatial feature tensor.

[0014] The fusion module is used to segment the spatial feature tensor into two complementary feature components through the spatiotemporal cross-fusion module. For each feature component, a temporal feature is obtained through multi-level temporal convolution, and a diffusion convolution is performed on the temporal feature to obtain the spatiotemporal coupling feature of the feature component. Using the spatiotemporal coupling feature of each feature component as the Tanh gate weight, the other feature component is modulated element-wise to obtain the spatiotemporal cross-fusion feature component of the feature component. The two spatiotemporal cross-fusion feature components are fused to obtain the spatiotemporal cross-fusion feature.

[0015] The emotion recognition module is used to determine the predicted probability distribution of the emotion category corresponding to the EEG signal based on the spatiotemporal cross-fusion features through the classification output module.

[0016] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement any of the steps in a brainwave signal emotion recognition method.

[0017] The present invention also provides a computer-readable storage medium storing a computer program that, when loaded by a processor, can execute any step of a brainwave signal emotion recognition method.

[0018] The EEG signal emotion recognition method provided by this invention has the following beneficial effects: This invention adaptively segments the feature tensor into two complementary components using an attention mechanism. Each branch carries effective information from different aspects of the original feature. Then, each branch independently performs a cascaded operation of temporal convolution and dynamic graph diffusion, ensuring that the output of each branch is itself a spatiotemporally coupled feature representation, rather than an isolated temporal or spatial feature. Furthermore, considering that temporal changes inevitably lead to shifts in spatial activation patterns, and that spatial connectivity changes with the selection of the time window, the complete spatiotemporally coupled feature of one branch is used as a gating signal to perform bidirectional element-wise modulation and fusion of the original feature of the other branch. The information flow forms a closed-loop feedback between the two branches, ensuring that the spatiotemporally complementary information is fully mixed, effectively capturing the dynamic correlation and complementary enhancement information between spatiotemporal features, and significantly improving the accuracy of EEG emotion recognition. Attached Figure Description

[0019] To more clearly illustrate the embodiments and design schemes of the present invention, the accompanying drawings required for this embodiment will be briefly described below. The drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of an EEG signal emotion recognition method provided in an embodiment of the present invention; Figure 2 This is an example of the overall process of DSTCFNet provided in the embodiments of the present invention; Figure 3 This is an example of a DSTCFNet network architecture diagram provided in an embodiment of the present invention; Figure 4 This is an example of the dynamic graph construction module process provided in an embodiment of the present invention; Figure 5 This is an example of the process of a multi-order directed diffusion convolution module provided in an embodiment of the present invention; Figure 6 This is an example of the spatiotemporal cross-fusion module process provided in an embodiment of the present invention; Figure 7 This is an example of a subject-dependent experiment classification accuracy distribution chart provided in an embodiment of the present invention; Figure 8 An example of the distribution of independent experimental classification accuracy of subjects provided in an embodiment of the present invention. Detailed Implementation

[0021] To enable those skilled in the art to better understand and implement the technical solutions of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and should not be construed as limiting the scope of protection of the present invention.

[0022] Example 1 This invention provides a method for emotion recognition using electroencephalogram (EEG) signals, specifically as follows: Figure 1 As shown, it includes the following steps: Step 1: Acquire EEG signals from multiple electrode channels; construct an input feature tensor based on the EEG signals.

[0023] Step 2: Input the input feature tensor into DSTCFNet to obtain the probability distribution of predicted sentiment categories. DSTCFNet includes a dynamic graph construction module, a spatiotemporal cross-fusion module, and a classification output module. The dynamic graph construction module constructs a sparse dynamic adjacency matrix from the input feature tensor. Based on the sparse dynamic adjacency matrix, graph convolution is performed on the input feature tensor to obtain a spatial feature tensor. The self-similarity of the input feature tensor corresponding to each electrode node is calculated to obtain a data-driven adjacency matrix. The data-driven adjacency matrix is ​​fused with the structural prior adjacency matrix determined based on the spatial positional relationship between each electrode node to obtain a sparse dynamic adjacency matrix. The sparse dynamic adjacency matrix is ​​then subjected to diffusion convolution to obtain a spatial feature tensor.

[0024] The sparse dynamic adjacency matrix can be generated through the following steps: Step 21: Aggregate the input feature tensor along the time dimension to obtain the node feature matrix.

[0025] Step 22: Determine the self-similarity between each electrode node by the self-inner product of the node feature matrix.

[0026] Step 23: The self-similarity between each electrode node is normalized using Softmax to obtain the data-driven adjacency matrix.

[0027] Specifically, such as Figure 2 As shown, firstly, time-series data collected from multiple electrode channels are acquired as multi-channel EEG signals to generate an input feature tensor. Then, a structural prior adjacency matrix based on electrode spatial coordinates and a data-driven adjacency matrix based on input feature self-similarity are fused. This is then adaptively weighted and fused through a learnable fully connected layer, followed by Top-K sparsification to generate a sparse dynamic adjacency matrix. Through this approach, the dynamic graph can adaptively overcome static presuppositions: K-order diffusion enhances spatial topology modeling; bi-branch gated fusion alleviates spatiotemporal isolation; and hierarchical integration improves multi-scale representation and cross-modal generalization.

[0028] For example, firstly, according to the international 10-20 electrode positioning system standard, the coordinates of electrode i in three-dimensional space are: At this time, the Euclidean distance between electrode i and electrode j As shown in formula (1): (1); Then, the distance is converted into similarity using a Gaussian kernel function, based on the prior adjacency matrix elements. The calculation formula is shown in formula (2): (2); Among them, when hour, =0, The bandwidth parameter can be determined by the average distance of all electrode pairs, as shown in formula (3): (3); Where N is the total number of electrodes. Then, the prior adjacency matrix is ​​row-normalized so that the sum of the connection weights of each node is 1. The normalization process is shown in formula (4): (4); in, This is the normalized structural prior adjacency matrix. Considering that the structural prior adjacency matrix reflects the inherent anatomical spatial structure between electrodes and usually remains unchanged, the structural prior adjacency matrix can be determined before model training and kept constant during training and inference.

[0029] Input feature tensors constructed based on EEG signals Along the time dimension The node feature matrix is ​​obtained by performing mean aggregation. The calculation process is shown in formula (5): (5); in, Input feature tensor A specific input feature tensor For the number of feature channels or feature dimensions, This is the batch size. Then, it is processed through the node feature matrix. The self-inner product is used to calculate the feature similarity between nodes. The data-driven adjacency matrix is ​​then obtained by Softmax normalization. The calculation process is shown in formulas (6) and (7): (6); (7); in, It is a data-driven adjacency matrix specific to each sample, which changes dynamically with the input data.

[0030] When the structural prior adjacency matrix and data-driven adjacency matrix After the build is complete, and The concatenation matrix is ​​obtained by concatenating along the dimensions. Then, adaptive weighted fusion is performed through a learnable fully connected layer to obtain the adaptively weighted fused matrix. The specific calculation process is shown in formulas (8) and (9): (8); (9); in, and For learnable fusion parameters, Using the Sigmoid activation function, the fusion weights are mapped to the [0,1] interval. To make the structural prior adjacency matrix The dimension from Expand to The matrix after that, To drive the adjacency matrix with data The dimension from Expand to The matrix after that, This involves performing tensor concatenation along the extended dimension. The resulting matrix is ​​then adaptively weighted and fused. The Top-K sparsification process is shown in formula (10): (10); in, It is a sparse dynamic adjacency matrix, which is obtained by retaining the k neighbors with the highest association strength of each node through the Top-K operation and setting the remaining connections to zero. It can suppress redundant weak connections while retaining key topological connections.

[0031] The spatial feature tensor can be generated through the following steps: Step 24: Using the sparse dynamic adjacency matrix as weights, perform multiple directed diffusions on the input feature tensor to obtain multiple multi-scale spatial feature tensors.

[0032] Step 25: Concatenate multiple multi-scale spatial feature tensors along the channel dimension to obtain a concatenated multi-scale spatial feature tensor. Convolve the concatenated multi-scale spatial feature tensor to obtain a multi-scale spatial feature tensor with reduced dimensionality after convolution.

[0033] Step 26: Perform Dropout regularization on the multi-scale spatial feature tensor after convolutional dimensionality reduction, connect it with the residual of the input feature tensor, and activate it using the ReLU function to obtain the spatial feature tensor.

[0034] Specifically, when the sparse dynamic adjacency matrix is ​​obtained... Then, relying on the sparse dynamic adjacency matrix K-order directed diffusion iterative calculations are performed to extract multi-scale spatial correlation features between electrode nodes. After multi-scale feature splicing and dimensionality reduction fusion, the spatial feature tensor is output.

[0035] For example, let the diffusion order hyperparameter be... ,by Using the initial input, iteratively perform directed diffusion, and the k-th diffusion feature... The calculation is shown in formula (11): (11); in, That is, sparse dynamic adjacency matrix The middle represents the sample Middle node To the node The directed correlation strength, Let K be the number of channels. Through K iterations, K multi-scale spatial feature tensors are obtained. K multi-scale spatial feature tensors The multi-scale spatial feature tensor is obtained by stitching along the channel dimension. The concatenated multi-scale spatial feature tensor is subjected to convolutional dimensionality reduction to obtain the convolutionally dimensionality-reduced multi-scale spatial feature tensor. The specific calculations are shown in formulas (12) and (13): (12); (13); Subsequently, for the multi-scale spatial feature tensor after convolutional dimensionality reduction... The feature tensor is obtained by introducing Dropout regularization. , feature tensor With original features That is, the input feature tensor is subjected to residual connection and ReLU activation to obtain a spatial feature tensor with enhanced feature representation capability. Specifically, as shown in formula (14): (14); In summary, this invention captures the directional propagation characteristics of EEG signals through a multi-stage directional diffusion mechanism, and residual connections ensure that the original node's own characteristics are not lost during the diffusion process.

[0036] Step 3: The spatial feature tensor is divided into two complementary feature components by the spatiotemporal cross-fusion module; for each feature component, a temporal feature is obtained by multi-level temporal convolution, and the spatiotemporal coupling feature of the feature component is obtained by diffusion convolution; the spatiotemporal coupling feature of each feature component is used as the Tanh gate weight to modulate the other feature component element by element to obtain the spatiotemporal cross-fusion feature component of the feature component.

[0037] The spatiotemporal coupling characteristics can be determined through the following steps: Step 31: Determine the attention weights of the two branches using a lightweight convolutional network.

[0038] Step 32: Weight the spatial feature tensor according to the attention weights to obtain the complementary feature components corresponding to the two branches.

[0039] Step 33: For each feature component, process the feature component through two-level multi-scale temporal convolution to obtain temporal features. The kernel sizes of the two-level multi-scale temporal convolution are different.

[0040] Step 34: Construct a dynamic graph based on temporal features to obtain a sample-specific sparse dynamic adjacency matrix. Perform multi-order directed diffusion convolution on the sample-specific sparse dynamic adjacency matrix to obtain the spatiotemporal coupling features of the feature components.

[0041] Specifically, when the spatial feature tensor is determined Subsequently, this invention adaptively divides the feature tensor into two complementary branches through an attention mechanism. The two branches perform multi-scale temporal convolution and dynamic graph convolution in parallel, and then achieve bidirectional cross-branch feature modulation fusion through the Tanh nonlinear gating mechanism.

[0042] Specifically, the group attention weights are first calculated using a lightweight convolutional network. As shown in formula (15): (15); in, The spatiotemporal feature tensor input to the spatiotemporal cross-fusion module, i.e., the spatial feature tensor. And based on group attention weights By weighting and grouping the original features, two complementary branches are obtained as shown in formulas (16) and (17): (16); (17); in, and These are the feature components of two complementary branches. Then, in each of the two complementary branches, two levels of multi-scale temporal convolution are performed, with the two convolution kernel sizes being [size missing]. and By capturing local transient features and global trend features respectively, the temporal features of each branch are obtained. Specifically, as shown in formula (18): (18); in, for and Any feature component in the graph. Then, using the dynamic graph construction method provided in the above implementation, which is similar to the method for constructing a sparse dynamic adjacency matrix, the temporal features are... Generate a sample-specific sparse dynamic adjacency matrix Furthermore, by employing the K-order directed diffusion convolution method provided in the above embodiments, which is similar to the method for generating spatial feature tensors, spatiotemporal coupled features are obtained. Specifically, as shown in formula (19): (19); The spatiotemporal cross-fusion feature components can be determined through the following steps: Step 35: Activate the spatiotemporal coupling feature of one feature component using the Tanh function to obtain the Tanh gate weight of another feature component. Modulate the feature component according to the Tanh gate weight to obtain the interactively modulated feature component.

[0043] Step 36: Perform temporal graph convolution on the interactively modulated feature components and then perform dynamic graph diffusion to obtain the spatiotemporal cross-fusion feature components after dynamic graph diffusion.

[0044] Step 37: The spatiotemporal cross-fusion feature components after the dynamic graph diffusion of each branch are distributed and calibrated through the feature components after interactive modulation of another branch to obtain the spatiotemporal cross-fusion feature components.

[0045] Specifically, when the spatiotemporal coupling characteristics are determined Then, the spatiotemporal coupling features of one branch are activated by Tanh and used as gating weights to modulate the original features of the other branch element by element, as shown in formulas (20) and (21): (20); (twenty one); in, and These are the gating weights of the two branches after Tanh activation. and For the interactively modulated feature components, Tanh activation maps the gated values ​​to The interval, thus when the spatiotemporal coupling characteristics are close to High-value features are preserved and enhanced when spatiotemporal coupling features are close to Time-suppressed interference features. Then, the interactively modulated feature components... and After symmetrical temporal convolution and dynamic graph diffusion operations, the spatiotemporal cross-fusion feature components after dynamic graph diffusion are obtained. and And residual adjustment convolution is introduced for distribution calibration, as shown in formulas (22) and (23): (twenty two); (twenty three); in, and These are two spatiotemporal cross-fusion feature components.

[0046] Step 4: Fuse the two spatiotemporal cross-fusion feature components to obtain the spatiotemporal cross-fusion feature; determine the predicted probability distribution of the emotion category corresponding to the EEG signal through the classification output module based on the spatiotemporal cross-fusion feature.

[0047] The spatiotemporal cross-fusion feature can be determined through the following steps: Step 41: Perform convolution processing on the spatiotemporal cross-fusion feature components to obtain the spatiotemporal cross-fusion feature components with adjusted dimensions.

[0048] Step 42: Perform average pooling on the spatiotemporal cross-fusion feature components after dimension adjustment to obtain spatiotemporal cross-fusion feature components aligned with the time dimension.

[0049] Step 43: Weighted fusion of the time-aligned spatiotemporal cross-fusion feature components to obtain time-aligned spatiotemporal cross-fusion features; link the time-aligned spatiotemporal cross-fusion features with the input feature tensor residual to obtain spatiotemporal cross-fusion features.

[0050] Specifically, after determining the two spatiotemporal cross-fusion feature components, this invention generates a unified high-dimensional spatiotemporal feature representation through dimension alignment, weighted average fusion, and residual connection.

[0051] For example, when you get and Then, dimensionality adjustment is performed using 1×1 convolution to obtain the dimensionality-adjusted spatiotemporal cross-fusion feature components. As shown in formula (24): (twenty four); Then, the spatiotemporal cross-fusion feature components of each branch are... Adaptive average pooling aligns the time dimension to the target size, resulting in time-aligned spatiotemporal cross-fusion feature components. As shown in formula (25): (25); in, , That is, all dimensions from index 2 to the end. For the input features to perform multi-scale hierarchical integration, i.e. and Then, the spatiotemporal cross-fusion feature components aligned across multiple time dimensions are weighted, averaged, and fused with the original input residual to obtain the spatiotemporal cross-fusion feature. As shown in formula (26): (26); The probability distribution for predicting sentiment categories can be determined through the following steps: Step 44: Filter the spatiotemporal cross-fusion features using gated linear units to obtain the filtered features.

[0052] Step 45: Link the filtered features with the spatiotemporal cross-fusion feature residuals, and obtain the global feature vector through global average pooling.

[0053] Step 46: Process the global feature vector sequentially with a fully connected layer and activate it with the Softmax function to obtain the predicted probability distribution of the emotion category corresponding to the EEG signal.

[0054] Specifically, after the spatiotemporal cross-fusion features are unified, they are filtered by gated linear units, compressed by global average pooling, and then input into a fully connected layer. The predicted probabilities of each sentiment category are output through the Softmax function to complete sentiment recognition.

[0055] For example, firstly, spatiotemporal cross-fusion features are obtained through gated linear units. Perform adaptive filtering to obtain filtered features As shown in formula (27): (27); in, This involves performing a linear transformation on the input features through convolution to generate candidate feature tensors; To generate gated weight tensors through another set of convolutions, The Sigmoid activation function is used. Then, residual connections are introduced to preserve basic information, and global average pooling is used to compress the spatiotemporal dimension to the channel dimension, resulting in a global feature vector. As shown in formulas (28) and (29): (28); (29); in, To introduce spatiotemporal cross-fusion features that preserve basic information by incorporating residual connections, a fully connected layer and a softmax function are then used to output the predicted probability distribution of each sentiment category. As shown in formula (30): (30); in, This is the weight matrix of the fully connected layer. For a learnable bias vector, the elements Indicates the first The predicted probability that a sample belongs to the k-th sentiment category.

[0056] In summary, this invention achieves dynamic adaptation of the graph structure to input samples by fusing the prior adjacency matrix of the fusion structure and the data-driven adjacency matrix, and then generating a sparse dynamic adjacency matrix by Top-K sparsification after adaptive weighted fusion using a learnable fully connected layer. This overcomes the limitation of existing technologies that rely on static preset graph structures.

[0057] Furthermore, by constructing a dual-branch parallel structure, each branch synchronously executes temporal convolution and dynamic graph convolution, and designs a Tanh nonlinear gating mechanism to achieve bidirectional cross-branch feature modulation fusion, this invention effectively captures the dynamic correlation and complementary enhancement information between spatiotemporal features, thus solving the technical problem of isolated modeling of spatiotemporal features in the prior art.

[0058] Furthermore, this invention, through the cascading of three-level spatiotemporal cross-fusion units and the integration of multi-branch hierarchical modules, simultaneously covers local details and global spatiotemporal dependencies, effectively improving the multi-scale expressive ability of features, thereby enhancing the accuracy and cross-subject generalization ability of EEG emotion recognition.

[0059] Example 2 Building upon Example 1, this example designs a dual-branch spatiotemporal cross-fusion network (DSTCFNet) for EEG emotion recognition tasks. It achieves sample-adaptive spatial topology modeling through dynamic graph construction and diffusing convolution, and achieves complementary enhancement of spatiotemporal features through dual-branch parallel processing and gated cross-fusion. This effectively solves the technical problems of isolated spatiotemporal feature modeling and static graph structure pre-setting in existing technologies. For example, Figure 3 As shown, this embodiment uses the DSTCFNet model as the emotion recognition network. The model is composed of a dynamic graph construction module, a multi-level diffusion convolution module, a spatiotemporal cross-fusion module, a multi-branch hierarchical integration module, and a classification output module. The model takes 62-channel, 200Hz EEG signals as input, and outputs the predicted probabilities of each emotion category after being processed by each module in sequence.

[0060] exist Figure 3Based on the DSTCFNet model shown, this embodiment uses the SEED and SEED-IV publicly available EEG emotion recognition datasets as examples to further illustrate the implementation process of the present invention and its advantages compared to existing models. The SEED dataset was collected from 15 subjects (7 men and 8 women). It was sampled at 1000Hz using an ESI NeuroScan 62-channel EEG acquisition device. Positive, neutral, and negative emotions were induced by watching 15 emotional film clips. The signals were downsampled to 200Hz and preprocessed with a 0-75Hz bandpass filter. The SEED-IV dataset also came from 15 subjects and was collected using a 62-channel device. Four emotions—happiness, sadness, fear, and neutral—were induced through 24 emotional video clips. The signals were downsampled to 200Hz and preprocessed with a 0-75Hz bandpass filter. , Bandpass filtering preprocessing. The input feature tensor dimension is represented as... ,in For batch size, The number of feature channels, For the number of electrodes, For time step.

[0061] After that, as Figure 4 As shown, the dynamic graph construction module of the DSTCFNet model uses a dual-path strategy to calculate the adjacency matrix, and the structural prior branches are based on... Each electrode follows international standards. The system's standard three-dimensional coordinates The Euclidean distance between the electrode pairs is calculated according to formula (1). Then, according to formula (2), the distance is mapped to the Gaussian kernel function. Interval similarity, where the bandwidth parameter Take the mean distance of all electrode pairs according to formula (3), set the diagonal elements to zero to eliminate self-loops, and perform row normalization according to formula (4) to obtain the prior adjacency matrix. Prior adjacency matrix The calculation is performed once before training to reflect the prior knowledge of brain anatomy and serve as a fixed knowledge injection model. The data-driven branch performs mean aggregation on the input feature tensor X along the time dimension T according to formula (5), and calculates the node feature self-inner product matrix according to formulas (6) and (7), and then normalizes it with Softmax to obtain the data-driven adjacency matrix. The data is dynamically updated based on the input samples. The dual-matrix adaptive fusion is performed according to formulas (8) and (9). and After splicing along the last dimension, the parameters are passed. , The weighted fusion of learnable fully connected layers, after Sigmoid activation, yields the fusion matrix. Execute according to formula (10). Top-K sparsification process preserves the strongest values ​​of each electrode node. Connecting each neighbor generates the final sparse dynamic adjacency matrix. .

[0062] like Figure 5 As shown, the multi-order diffusing convolution module of the DSTCFNet model uses a dynamic adjacency matrix. and input feature tensor For input, execute The first-order directed diffusion iteration. According to formula (11), the... Secondary diffusion nodes Its characteristic is that all its predecessor nodes are the first Rank features Weighted aggregation of weights, weights , , , Direct characterization of neural signals from electrodes Transfer to electrodes The probability. After After the iteration, the two multi-scale spatial feature tensors are converted according to formula (12). and By splicing along the channel dimension, we obtain Then, according to formula (13) Convolution kernel dimensionality reduction and fusion The weights of features at different scales are adaptively assigned through parameterized learning. Finally, the weights are introduced according to formula (14). Dropout regularization is applied to the residuals and then activated by ReLU to obtain the final spatial feature tensor. .

[0063] like Figure 6 As shown, the spatiotemporal cross-fusion module of the DSTCFNet model first calculates the group attention weights of the input features through a lightweight convolutional network according to formula (15). Then, according to formulas (16) and (17), the features are adaptively segmented into two complementary branches. and The sum of the two is the original feature, achieving complementary information separation. Each branch performs two-level multi-scale temporal convolution: convolution kernel... Covering short-term local windows to capture transient features of mood shifts, convolution kernels Overlaying a long-term window to capture trends in sentiment changes; then invoking the dynamic graph construction module based on the current temporal features. Generate a sample-specific sparse dynamic adjacency matrix, and input it into the diffusion convolution module for execution. The order-directed diffusion is followed by residual concatenation with the original temporal features according to formula (18) to obtain the spatiotemporal coupling features. Gated cross-fusion is implemented according to formula (19). For example, the spatiotemporal coupling feature of one branch is activated by Tanh and multiplied element-wise with the original feature of another branch. Tanh maps the gated value to... The range, close to Enhance high-value characteristics, close to Time-suppressed interference characteristics; characteristics after cross-modulation and After undergoing symmetric temporal convolution and graph diffusion operations again, the feature distribution is calibrated by adjusting the convolution with residuals according to formulas (20) and (21), and the final spatiotemporal cross-fusion feature is output. The three-level spatiotemporal cross-fusion units are cascaded sequentially, reducing the receptive field layer by layer to extract multi-scale information.

[0064] Subsequently, the multi-branch hierarchical integration module of the DSTCFNet model receives six feature streams from the high / low group outputs of the three-level spatiotemporal cross-fusion units. These features are then processed according to formulas (22) and (23). The convolution is adjusted in dimension, and the time dimension is aligned to the target size using adaptive average pooling according to formula (24). ); Select four paths from the second and third levels of the six features ( , , , After weighted fusion according to formula (25), residual connection is performed with the original input to output a unified high-dimensional spatiotemporal feature representation. The classification output module of the DSTCFNet model first adaptively filters the integrated features through gated linear units according to formula (26), and the candidate features are... With gate weights The vectors are generated by two sets of convolutions, and then multiplied element-wise after Sigmoid activation. Residual connections are introduced according to formula (27) to preserve the original basic information. Global average pooling is performed according to formula (28) to compress the spatial and temporal mean into a channel-dimensional global vector. Finally, pass through formula (29). The fully connected layer and the Softmax function output the predicted probabilities of each sentiment category. The category with the highest probability is taken as the emotion recognition result.

[0065] Furthermore, model training can employ a joint loss function combining weighted cross-entropy loss and graph regularization loss. The weighted cross-entropy loss is obtained through class weights. To balance the contributions of each category, the weights are inversely proportional to the sample size of each category. Specifically, as shown in formula (31): (31); The graph regularization loss imposes a sparsity constraint on the dynamic adjacency matrix through the L2 norm, preventing overfitting of the graph structure, as shown in formula (32): (32); in, Describing the Frobenius norm, Let be the regularization coefficient. Combining formulas (31) and (32), the total loss is... As shown in formula (33): (33); The training hyperparameters of the model are determined by those skilled in the art based on actual needs, such as batch size. Learning rate , Optimizer ( ), Regularization coefficient Maximum number of iterations Stop early and be patient. Graph regularization coefficient Hidden dimension diffusion steps , Retention ratio The Dropout rate for the graph module is 0.1, and the Dropout rate for the feature extraction module is 0.2.

[0066] In validating the SEED and SEED-IV datasets, this embodiment designs two evaluation scenarios: subject-dependent and subject-independent. The subject-dependent experiment includes... Each of the 100 subjects, respectively Data as training set, As a test set, dedicated models were trained to evaluate their ability to model individual internal EEG emotional patterns; Subject-Independent experiments included a leave-one-subject cross-validation strategy (LOSO-CV), with each round selecting... All data from the subjects were used as the test set, and the remaining data were used as the test set. Data from [number] subjects were used as the training set, totaling [number] subjects. The evaluation round assesses the model's ability to generalize across individuals when no new subjects are encountered.

[0067] The classification accuracy of the DSTCFNet model provided by this invention is compared with existing models such as SVM and GSCCA. The results are shown in Table 1 and Table 2, respectively: Table 1. Comparison of Subject-Dependent Classification Accuracy Based on SEED and SEED-IV Datasets

[0068] Table 2. Comparison of Independent Classification Accuracy of Subjects Based on SEED and SEED-IV Datasets

[0069] As shown in Table 1 and Figure 7 As shown, in the subject-dependent experiment, DSTCFNet achieved an average accuracy of 100% on the SEED dataset. On the SEED-IV dataset, the average accuracy reached , In most of the subjects, the success rate was over 95%, demonstrating the model's effective ability to model individual-specific spatiotemporal EEG emotional characteristics. (See Table 2 and...) Figure 8 As shown, in independent trials with subjects, DSTCFNet achieved an average accuracy of [percentage missing] on the SEED dataset. On the SEED-IV dataset, the average accuracy reached All of these results outperform mainstream benchmark methods such as SVM, GSCCA, DBN, and DGCNN. In summary, in both subject-dependent and independent scenarios on the SEED and SEED-IV benchmark datasets, this invention achieves [performance targets]. and The accuracy rate is better than that of existing mainstream methods, and the ablation experiments verify the effectiveness of each core component.

[0070] Example 3 The present invention also provides an EEG signal emotion recognition system, comprising: The data acquisition module is used to acquire EEG signals from multiple electrode channels; construct an input feature tensor based on the EEG signals; input the input feature tensor into DSTCFNet to obtain the probability distribution of emotion category prediction. DSTCFNet includes a dynamic graph construction module, a spatiotemporal cross-fusion module, and a classification output module.

[0071] The feature extraction module constructs a sparse dynamic adjacency matrix from the input feature tensor through the dynamic graph construction module, and performs graph convolution on the input feature tensor based on the sparse dynamic adjacency matrix to obtain the spatial feature tensor.

[0072] The fusion module is used to segment the spatial feature tensor into two complementary feature components through the spatiotemporal cross-fusion module. For each feature component, a temporal feature is obtained through multi-level temporal convolution, and a diffusion convolution is performed on the temporal feature to obtain the spatiotemporal coupling feature of the feature component. Using the spatiotemporal coupling feature of each feature component as the Tanh gate weight, the other feature component is modulated element-wise to obtain the spatiotemporal cross-fusion feature component of the feature component. The two spatiotemporal cross-fusion feature components are fused to obtain the spatiotemporal cross-fusion feature.

[0073] The emotion recognition module is used to determine the predicted probability distribution of the emotion category corresponding to the EEG signal based on the spatiotemporal cross-fusion features through the classification output module.

[0074] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps in an embodiment of an EEG signal emotion recognition method. Specific implementation methods can be found in the method embodiments, and will not be repeated here.

[0075] Furthermore, the present invention also provides a non-transitory computer-readable storage medium containing instructions on which a computer program is stored. For example, a memory containing instructions that can be executed by a processor of a computer device to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc. When the computer program is executed by the processor, it can implement the steps in an embodiment of an EEG signal emotion recognition method. Specific implementation methods can be found in the method embodiments, which will not be repeated here.

[0076] Those skilled in the art will understand that embodiments of the present invention can provide methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0077] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0078] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0079] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0080] It should be noted that the specific embodiments described above enable those skilled in the art to more fully understand the present invention, but do not limit the present invention in any way. Therefore, although the present invention has been described in detail in this specification and embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the present invention; and all technical solutions and improvements that do not depart from the spirit and scope of the present invention are covered within the protection scope of the present invention patent. No reference numerals in the claims should be construed as limiting the scope of the claims. Any simple variations or equivalent substitutions of technical solutions that can be readily obtained by those skilled in the art within the scope of the technology disclosed in the present invention are within the protection scope of the present invention.

Claims

1. A method for emotion recognition using electroencephalogram (EEG) signals, characterized in that, include: Acquire EEG signals from multiple electrode channels; Construct an input feature tensor based on the EEG signals; The input feature tensor is input into DSTCFNet to obtain the probability distribution of the predicted sentiment category. The DSTCFNet includes a dynamic graph construction module, a spatiotemporal cross-fusion module, and a classification output module. The dynamic graph construction module constructs a sparse dynamic adjacency matrix from the input feature tensor, and performs graph convolution on the input feature tensor based on the sparse dynamic adjacency matrix to obtain a spatial feature tensor. The spatiotemporal cross-fusion module divides the spatial feature tensor into two complementary feature components. For each feature component, a temporal feature is obtained through multi-level temporal convolution. The temporal feature is then subjected to diffusion convolution to obtain the spatiotemporal coupling feature of the feature component. Using the spatiotemporal coupling feature of each feature component as the Tanh gate weight, the other feature component is modulated element-wise to obtain the spatiotemporal cross-fusion feature component. The two spatiotemporal cross-fusion feature components are then fused to obtain the spatiotemporal cross-fusion feature. The classification output module determines the predicted probability distribution of the emotion category corresponding to the EEG signal based on the spatiotemporal cross-fusion features.

2. The method for emotion recognition using electroencephalogram (EEG) signals according to claim 1, characterized in that, The dynamic graph construction module constructs a sparse dynamic adjacency matrix from the input feature tensor, and performs graph convolution on the input feature tensor based on the sparse dynamic adjacency matrix to obtain a spatial feature tensor including: The input feature tensor is aggregated along the time dimension to obtain a node feature matrix; the self-similarity between each electrode node is determined by the inner product of the node feature matrix; the self-similarity between each electrode node is normalized by Softmax to obtain a data-driven adjacency matrix. The data-driven adjacency matrix is ​​fused with the structural prior adjacency matrix determined based on the spatial positional relationship between each electrode node to obtain a sparse dynamic adjacency matrix. The sparse dynamic adjacency matrix is ​​subjected to diffusion convolution to obtain the spatial feature tensor.

3. The method for emotion recognition using electroencephalogram (EEG) signals according to claim 1, characterized in that, Based on the sparse dynamic adjacency matrix, graph convolution is performed on the input feature tensor to obtain a spatial feature tensor including: Using the sparse dynamic adjacency matrix as weights, the input feature tensor is subjected to multiple directed diffusions to obtain multiple multi-scale spatial feature tensors. Multiple multi-scale spatial feature tensors are concatenated along the channel dimension to obtain a concatenated multi-scale spatial feature tensor. The concatenated multi-scale spatial feature tensor is then convolved to obtain a multi-scale spatial feature tensor with reduced dimensionality after convolution. The multi-scale spatial feature tensor after convolutional dimensionality reduction is sequentially subjected to Dropout regularization, concatenated with the residual of the input feature tensor, and activated by the ReLU function to obtain the spatial feature tensor.

4. The method for emotion recognition using electroencephalogram (EEG) signals according to claim 1, characterized in that, The spatiotemporal coupling features of the feature components obtained by performing diffusion convolution on the temporal features include: The attention weights of the two branches are determined using a lightweight convolutional network; The spatial feature tensor is weighted and grouped according to the attention weights to obtain the complementary feature components corresponding to the two branches respectively. For each feature component, the feature component is processed by two-level multi-scale temporal convolution to obtain the temporal feature, wherein the kernel size of the two-level multi-scale temporal convolution is different; A dynamic graph is constructed based on the temporal features to obtain a sample-specific sparse dynamic adjacency matrix. Multi-order directed diffusion convolution is performed on the sample-specific sparse dynamic adjacency matrix to obtain the spatiotemporal coupling features of the feature components.

5. The method for emotion recognition using electroencephalogram (EEG) signals according to claim 4, characterized in that, Using the spatiotemporal coupling features of each feature component as Tanh gate weights, another feature component is modulated element-wise to obtain the spatiotemporal cross-fusion feature components of the feature components, including: The spatiotemporal coupling feature of one feature component is activated by the Tanh function to obtain the Tanh gate weight of another feature component. The feature component is then modulated according to the Tanh gate weight to obtain the interactively modulated feature component. The interactively modulated feature components are subjected to temporal graph convolution and dynamic graph diffusion to obtain the spatiotemporally cross-fused feature components after dynamic graph diffusion. The spatiotemporal cross-fusion feature components after the dynamic graph diffusion of each branch are distributed and calibrated through the feature components after interactive modulation of another branch to obtain the spatiotemporal cross-fusion feature components.

6. The method for emotion recognition using electroencephalogram (EEG) signals according to claim 5, characterized in that, The spatiotemporal cross-fusion feature obtained by fusing two spatiotemporal cross-fusion feature components includes: The spatiotemporal cross-fusion feature components are convolved to obtain the spatiotemporal cross-fusion feature components with adjusted dimensions. Average pooling is performed on the spatiotemporal cross-fusion feature components after the dimension adjustment to obtain spatiotemporal cross-fusion feature components aligned with the time dimension. The time-aligned spatiotemporal cross-fusion feature components are weighted and fused to obtain the time-aligned spatiotemporal cross-fusion feature; the time-aligned spatiotemporal cross-fusion feature is linked with the input feature tensor residual to obtain the spatiotemporal cross-fusion feature.

7. The method for emotion recognition using electroencephalogram (EEG) signals according to claim 1, characterized in that, The classification output module determines the probability distribution of the predicted emotion category corresponding to the EEG signal based on the spatiotemporal cross-fusion features, including: The spatiotemporal cross-fusion features are filtered using gated linear units to obtain the filtered features; The filtered features are linked with the spatiotemporal cross-fusion feature residuals, and then global average pooling is used to obtain a global feature vector. The global feature vector is processed sequentially by a fully connected layer and activated by the Softmax function to obtain the predicted probability distribution of the emotion category corresponding to the EEG signal.

8. A brainwave signal emotion recognition system, characterized in that, include: The data acquisition module is used to acquire EEG signals from multiple electrode channels; Construct an input feature tensor based on the EEG signals; The input feature tensor is input into DSTCFNet to obtain the probability distribution of the predicted sentiment category. The DSTCFNet includes a dynamic graph construction module, a spatiotemporal cross-fusion module, and a classification output module. The feature extraction module constructs a sparse dynamic adjacency matrix from the input feature tensor through the dynamic graph construction module, and performs graph convolution on the input feature tensor based on the sparse dynamic adjacency matrix to obtain a spatial feature tensor. The fusion module is used to segment the spatial feature tensor into two complementary feature components through the spatiotemporal cross-fusion module; for each feature component, a temporal feature is obtained through multi-level temporal convolution, and a spatiotemporal coupling feature is obtained by performing a diffusion convolution on the temporal feature; using the spatiotemporal coupling feature of each feature component as a Tanh gate weight, the other feature component is modulated element-wise to obtain the spatiotemporal cross-fusion feature component of the feature component; the two spatiotemporal cross-fusion feature components are fused to obtain the spatiotemporal cross-fusion feature; The emotion recognition module is used to determine the predicted probability distribution of the emotion category corresponding to the EEG signal based on the spatiotemporal cross-fusion features through the classification output module.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the EEG signal emotion recognition method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is loaded by the processor, it is able to perform the steps of the EEG signal emotion recognition method according to any one of claims 1 to 7.

Images