Multi-domain constraint emotion recognition system inspired by self-adaptive cognitive map and emotion recognition method

Abstract

The invention discloses a multi-domain constraint emotion recognition system inspired by an adaptive cognitive map and an emotion recognition method, which are applied to the field of electroencephalogram signal processing. According to a traditional emotion classification method, after electroencephalogram signals are subjected to manual feature extraction, key features are screened and input into traditional classifiers such as a support vector machine (SVM), but professional priori knowledge is needed for manual feature extraction; most of the methods mainly focus on feature selection and time information utilization, so that space information is not fully utilized; according to the method, a space-guided dynamic graph constraint learning module is designed, a topological structure is generated in a self-adaptive mode, the function relation of the topological structure is constrained, and complex spatial domain feature extraction overcoming cognitive priori information is achieved; meanwhile, a time-excited sequence attention module is provided, and global time sequence dependence characteristics are captured by paying attention to more important time in the sequence; furthermore, a new multi-domain interaction attention module is designed, traditional static cross-domain interaction is overcome, and constraint of multi-domain difference information and aggregation of complementary information are achieved. Besides, in order to better utilize information migration between different domains, a constraint is introduced to reduce distribution difference between the source domain and target domain features, and similarity and mobility of cross-domain features are improved.

Description

Technical Field

[0001] The invention belongs to the field of electroencephalogram signal processing, and in particular relates to an emotion classification technology. Background Art

[0002] In neurobiology, the generation of emotions is closely related to brain activity. When external stimuli are transmitted to the brain, they trigger corresponding brain activities, and human emotions are identified by capturing the reactions generated by these activities. Accurate recognition of emotions is of great significance in areas such as mental health tracking and human-computer interaction. Emotional reactions can be expressed through physiological signals and non-physiological signals. Compared with non-physiological signals such as facial expressions and voices that are susceptible to subjective influences, physiological signals are more popular because they are not easy to disguise. Among these physiological signals, electroencephalogram (EEG) signals are widely used in emotion research due to their portability, non-invasiveness, and high temporal resolution.

[0003] The traditional emotion classification method is to manually extract features from EEG signals and then select key features and input them into traditional classifiers such as support vector machines (SVM). However, manual feature extraction requires professional prior knowledge, and most methods focus on feature selection and the use of temporal information, which makes insufficient use of spatial information. Therefore, with the deepening of emotion recognition research, more and more deep learning methods are used for emotion detection. High temporal resolution and spatial resolution are the basic characteristics of EEG, so some studies have explored the importance of temporal information and spatial information respectively. For example, Du et al. proposed the ATDD-LSTMATDD-LSTM (Attention-based LSTM with Domain Discriminator, a deep learning model for emotion recognition) framework, which applies LSTM (Long Short-Term Memory) to EEG channel sequences to characterize the nonlinear relationship between multi-channel EEG signals. The dynamic graph convolutional neural network proposed by Song et al. uses a dynamic adjacency matrix to learn more spatial discriminant features. In addition, many studies focus on capturing a combination of specific spatial topological information and time-dependent representations. For example, Ding et al. proposed a spatiotemporal feature extraction using multi-scale convolution. Ning et al. proposed a spatial-spectral-temporal attention-based 3D dense network (SST-Net) and focused on integrated high-dimensional emotion features through self-attention. Liang et al. proposed a hybrid unsupervised generative adversarial network (EEGFuseNet) that was trained in an unsupervised manner to automatically characterize emotion features covering spatial and temporal dynamics.

[0004] Fundamentally, these works do not take into account the impact of the interaction of spatiotemporal cross-domain information on emotion recognition. Based on this problem, Cheng et al. proposed a hybrid network that uses cross-attention to interactively extract spatiotemporal features. However, the complexity and unknowability of brain activity still hinder efficient emotion recognition. In neurophysiology, the neural structure of the brain determines that there is interaction and cooperation between spatial and temporal processing in different regions. Therefore, how to efficiently decode spatiotemporal interactive information becomes a problem that needs to be solved.

[0005] From the perspective of improving the decoding of emotional states, many works based on graph neural networks have achieved some results through data-driven methods. For example, Pan et al. proposed a spatiotemporal self-constructing graph neural network to appropriately reflect the connections between various brain regions. Qu et al. proposed to construct edge connection features to further improve the emotion recognition effect. However, they are based on cognitive prior information such as phase locking value (PLV) and Pearson correlation coefficient (PCC) as the graph structure, which results in the structural information being fixed and making it difficult to capture important information from the complex domain structure. Therefore, dynamically mining the brain's emotional cognitive structure and effectively depicting emotional representations may provide more room for further improving the performance of emotion recognition systems. Summary of the invention

[0006] To solve the above technical problems, the present invention proposes a multi-domain constraint learning system (AC-DCL) and an emotion recognition method inspired by adaptive cognitive graphs.

[0007] One of the technical solutions adopted by the present invention is: a multi-domain constraint emotion recognition system inspired by an adaptive cognitive map, comprising: a dynamic graph constraint learning module, a sequence attention module and a multi-domain interactive attention module; the dynamic graph constraint learning module extracts the spatial domain features of the EEG signal based on the constructed brain topology and the dynamically generated topology; the sequence attention module is used to extract the time domain features of the EEG signal; the multi-domain interactive attention module obtains the emotion recognition results based on the spatial domain features of the EEG signal and the time domain features of the EEG signal.

[0008] The second technical solution adopted by the present invention is: a multi-domain constrained emotion recognition method inspired by an adaptive cognitive map, comprising:

[0009] S1, extract the differential entropy features of EEG signals, and construct the brain topology structure based on the differential entropy features;

[0010] S2, randomly generate a driving structure of the same size as the brain topology constructed in step S1;

[0011] S3, obtaining the spatial domain features of the EEG signal based on the driving structure and the brain topology structure constructed in step S1;

[0012] S4, and using position coding to add position information to the differential entropy feature extracted in step S1;

[0013] S5, the result obtained in step S4 is passed through the self-attention mechanism and the feedforward network to obtain the time domain features of the EEG signal;

[0014] S6, the temporal and spatial features of EEG signals are fused through the cross-attention mechanism;

[0015] S7. Obtain an emotion recognition result based on the fusion result obtained in step S6.

[0016] Beneficial effects of the present invention: The present invention proposes a multi-domain constraint learning system (AC-DCL) inspired by adaptive cognitive graphs, which embeds spatial and temporal representative knowledge simultaneously to improve the performance of emotion recognition systems. In AC-DCL, a spatially guided dynamic graph constraint learning module is first carefully designed. By adaptively generating topological structures and constraining the functional relationships of topological structures, complex spatial domain feature extraction that overcomes cognitive prior information is achieved. At the same time, a time-inspired sequence attention module is proposed to capture global temporal dependency features by focusing on more important times in the sequence. Further, the present invention designs a new multi-domain interactive attention module, which overcomes the traditional static cross-domain interaction and realizes the constraint of multi-domain difference information and the aggregation of complementary information. In addition, in order to better utilize the information migration between different domains, the present invention also introduces a constraint to reduce the distribution difference between source domain and target domain features, and improve the similarity and transferability of cross-domain features. The present invention uses three public emotion datasets to evaluate the proposed AC-DCL learning system, and consistently proves the advanced nature of its decoding performance in three experiments of subject dependency, inter-subject and cross-session. The advantages of the AC-DCL learning system of the present invention are as follows:

[0017] (1) By designing a spatially guided dynamic graph constraint learning module, the AC-DCL learning system is able to adaptively generate and constrain the functional relationships of topological structures in emotion recognition tasks for the first time. This breaks through the limitations of traditional reliance on prior knowledge and more effectively captures complex domain structure information.

[0018] (2) A time-inspired sequence attention module is proposed to capture the global temporal dependency features by focusing on the more important times in the sequence.

[0019] (3) By designing a new multi-domain interactive attention module, AC-DCL effectively overcomes the traditional static cross-domain interaction and realizes the constraint of multi-domain difference information and the aggregation of complementary information.

[0020] (4) The proposed AC-DCL learning system captures stable cognitive functions from complex and dynamic cognitive structures for the first time, providing a potential solution for further revealing the emotional cognitive functions of the brain. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a system architecture diagram of the present invention. DETAILED DESCRIPTION

[0022] To facilitate those skilled in the art to understand the technical content of the present invention, the present invention is further explained below with reference to the accompanying drawings.

[0023] The present invention makes efficient use of the spatial topological information and time series information of EEG data and designs a new method (AC-DCL) for emotion recognition. The overall architecture of the model is as follows: Figure 1 As shown in Figure 2, the proposed architecture consists of three main functional modules: dynamic graph constraint learning module, time series attention module, and cross-domain cross attention module. After explaining the symbols and nomenclature, each proposed submodule is described in detail.

[0024] The EEG signal of each sample is assumed to be S = {(Xi, Yi), i = 1, 2, ..., n}, where X i ∈R C×T , which is a 2D matrix representing the i-th trial of the preprocessed EEG sample with C channels and T time sampling points. n represents the total number of EEG trials. The corresponding true label of X is denoted by Y i ∈R n×1 , where the EEG emotion classification task has h categories.

[0025] In the whole framework, the extracted differential entropy (DE) features are used as the input of the model, and its dimension is expressed as [N, D, C, F]. Here, D represents the number of features from X i The feature dimensions obtained after extracting DE are as follows. For example, each EEG sample X i ∈R C×t After extracting DE features, its dimension will be converted to X d ∈R D×C×F Then, the present invention will X d As the input of the model, the spatial topological features and the temporal features of attention enhancement are obtained respectively. Finally, the spatiotemporal features of the obtained EEG signal are fused using the multi-domain interactive attention module, and the final prediction label is obtained through the classification layer.

[0026] The following is a detailed description of each module:

[0027] 1. Dynamic graph constraint learning module

[0028] The EEG signal is first preprocessed, that is, the original EEG data is first downsampled to a frequency of 200Hz, and then a bandpass filter between 1Hz and 75Hz is used to process the data to filter noise and eliminate artifacts. After this, the EEG signal is divided into several non-overlapping time periods by a sliding window, each time period lasting t seconds, and t is 4 seconds in this embodiment. Subsequently, DE features are extracted from the preprocessed signal as the input of the model. The topological structure of the brain is crucial for emotion recognition. Usually PCC and PLV are used to construct brain networks, but they are static connections obtained through prior knowledge. Therefore, the goal of this module is to learn adaptive topological structures to reflect the differences in brain functional connection patterns under different emotional states. The present invention introduces learnable parameters to adaptively generate topological structures, which are expressed as follows:

[0029]

[0030] Where W A Q and W A K is the query (Query:XW A Q ) and key (Key:XW A K ) are learnable parameters. is a scaling factor used to prevent the value from being too large; T represents transposition. Therefore, the attention-based score matrix A is obtained through linear transformation and self-attention adj ∈R C×C , that is, the adaptively generated topology.

[0031] Then, on this basis, the present invention sets a screening parameter b for the connectivity strength of the brain structure (the value range of b is 0.1, 0.2, ..., 0.9), and the value of b in this embodiment is 0.8; based on the screening parameter b, A adj Screen and get f adj .

[0032]

[0033] The module then randomly generates a drive structure s adj ∈R C×C , and then the brain topology f adj ∈R C×C The randomly generated drive structure recalibrates itself separately to facilitate the acquisition of stable modes.

[0034]

[0035] Among them, Conv is a one-dimensional convolution, so the final attention-based matrix f′ is obtained adj ∈R C×C and drive structure s′ adj ∈R C×C .

[0036] Given a graph G = (V, E, A), where V (|V| = V) is the set of channel nodes, E is the set of graph edges, and A is the graph adjacency matrix. Although traditional graph convolution usually relies on the graph Laplacian matrix, the dynamically generated topological structure is directly used as the adjacency matrix for convolution here, emphasizing the model's dependence on the relationship structure in the adjacency matrix and making full use of the flexibility and expressiveness of the attention mechanism. In addition, in order to improve operating efficiency, the present invention introduces Chebyshev polynomials to accelerate graph convolution calculations, while being able to handle a wider range of adjacency relationships to extract richer graph structure information. Each order polynomial corresponds to the local features of the graph, and the calculation process of the kth order Chebyshev polynomial is shown in formula (4):

[0037]

[0038] The initial conditions of the Chebyshev polynomial are T0(X) = 1, T1(X) = X; that is, the first-order Chebyshev polynomial T1(X) = X, the second-order Chebyshev polynomial

[0039] Given X l-1 As the input of the lth graph convolutional layer, X l-1 That is, the output of the DE feature after the first l-1 graph convolutional layers. It is the normalized adjacency matrix in the graph convolution operation, which is the result of preprocessing the adjacency matrix A of the graph. The purpose is to improve the numerical stability and convergence effect of the graph convolution. The output calculation of the l-th layer of graph convolution is shown in formula (5):

[0040]

[0041] where θ l k is the trainable parameter of the kth order of the corresponding graph convolution layer; σ represents the activation function. As the number of iterations increases, the node features propagate in the graph structure, and finally the global spatial information aggregation representation X is obtained. S In order to avoid information loss, the present invention directly sums the output of each graph convolution layer without performing graph pooling, thereby integrating the features of all nodes into a graph-level representation.

[0042] By taking f′ adj and s′adj As the adjacency matrix of graph convolution, based on the above Chebyshev polynomials, such as Figure 1 As shown, adaptive topological features and driving structure features can be obtained respectively; these two features are fused to obtain the spatial domain features of the EEG signal.

[0043] It should be emphasized that traditional GNNs (graph neural networks) usually directly rely on fixed input graph structures. However, in practical applications, graph topology is often extremely complex and may contain implicit information, which is difficult to capture by prior knowledge. In this case, the performance of graph neural networks may be significantly affected. Therefore, the present invention aims to adaptively adjust the topological structure of the graph during the learning process to better adapt to the characteristics of the data. Here, the present invention designs a new method L that can directly constrain the relationship between the feature matrix and the topological structure matrix C , so that the model can more effectively integrate node features and topological information. Moreover, enhancing the correlation representation helps improve the model's understanding of node relationships. At the same time, due to the optimization of the topological structure, the robustness of the model is enhanced, which enables it to perform better when processing actual complex data. Specifically, as shown in formula (6):

[0044] L C =||X S ·X S T -f′ adj ·A adj +(1-A adj )·s adj || (6)

[0045] Here, ‖·‖ represents the Frobenius norm. The core idea of ​​this method is to constrain the relationship between the autocorrelation of the node feature matrix and the adaptive topological structure to force the model to learn a topological structure that can better represent the relationship between nodes. This structure not only relies on the original topological structure but also introduces a driving matrix, which further enhances the model's ability to express potential complex relationships.

[0046] 2. Time-Excited Sequence Converter

[0047] Studies have shown that brain patterns under the same emotional state also vary significantly over time, so this module will focus more on the time segments in the EEG sequence that are more relevant to emotions. d Adding position information can add time information to the input embedding vector, allowing the model to capture the relative and absolute position relationship between sequences during training and inference, thereby improving the model's ability to understand the sequence structure. The calculation is as follows:

[0048]

[0049] Output(X)=Droput(X d +PE) (8)

[0050] Formula (7) represents the time information in the sequence, where d model is the dimension of the model, p is the position, and i is the dimension index. Then use formula (8) to calculate the input X d Add position encoding information, where is the position encoding matrix. The self-attention mechanism then captures the dependencies between the input sequences, and then performs nonlinear transformations through a feedforward neural network. For each time step t, the self-attention mechanism is calculated as follows:

[0051] Q=ψ(XW Q ),K t =ψ(XW K ),V t =ψ(XW V ) (9)

[0052]

[0053] where ψ(·) function represents linear transformation; W Q ∈R F×C ,W K ∈R F×C ,W V ∈R F×C is the learnable parameter of self-attention; d k is a coefficient factor. In this invention, multi-head self-attention is adopted, allowing the model to calculate attention in parallel in different representation subspaces:

[0054] MultiHead(Q,K,V)=Concat(head1,head2,…,head h )W O (11)

[0055] in is a learnable parameter matrix, h represents the number of heads. Each head i Is an independent self-attention mechanism:

[0056]

[0057] After the multi-head self-attention mechanism and feedforward network, the output is:

[0058] Output(X t)=LN(LN(X+MultiHead(Q,K,V))+FFN(LN(X+MultiHead(Q,K,V))) (13)

[0059] Among them, LN is a layer normalization, which is used to reduce the impact caused by parameter changes between layers during training, thereby accelerating convergence and stabilizing training. FFN is a feedforward neural network that contains two fully connected layers and an activation function, which performs nonlinear transformation on each input position to enhance the representation ability of the model.

[0060] 3. Multi-domain interactive attention module

[0061] In order to fully explore the complex interactive information between the time domain and the spatial domain in the EEG signal, the present invention proposes a multi-domain interactive attention module based on graph attention cross-attention. The core idea of ​​this model is to generate a cross-domain adjacency matrix through a cross-attention mechanism, and use the graph attention network to further enhance the expressiveness of features. The Graph Attention Network (GAT) can capture the implicit associations and non-local dependencies between nodes by assigning adaptive attention weights to different nodes, thereby effectively modeling complex spatial structural information. This design enables the model to dynamically adjust the feature weight distribution between different domains, thereby improving the capture capability and overall expression effect of key features in EEG signals.

[0062] This module first integrates the temporal and spatial features of EEG signals through a cross-attention mechanism to achieve cross-domain information integration. In this process, the cross-attention mechanism can effectively focus on and extract more important information in each module, thereby strengthening the expression of key features.

[0063]

[0064] Where W Q 1 ,W K 1 ,W V 1 ,W Q 2 ,W K 2 ,W V 2 is a learnable weight matrix, and ψ(·) represents a linear transformation. The cross-domain fused features O1 and O2 are obtained according to the above formula.

[0065] After obtaining the fused features, the model generates a cross-domain adjacency matrix for the graph attention network. This adjacency matrix not only contains the association information between the time domain and the spatial domain, but also captures the complex interactions between different brain regions. The adjacency matrix after fusion of features is generated by the following formula, which represents the weighted relationship between different nodes (i.e., different time points or brain regions), where ψ(O) represents a linear transformation.

[0066]

[0067] Next, the present invention introduces multiple heads into GAT, that is, calculates the output of each attention head and then concatenates these outputs. For each attention head i, the calculation process is as follows: first, feature transformation is performed:

[0068] H i =W i O (17)

[0069] Among them, W i is the weight matrix of the i-th attention head, O is the input feature, here is the fused features O1 and O2. For each node i, its new feature representation H i 'Calculated by the following formula:

[0070] H i ′=ELU(∑ j∈N(i) α i,j H i ) (18)

[0071] ELU is the activation function, and N(i) represents the set of neighbor nodes of node i. i,j is the attention weight between node i and node j, which can be expressed as:

[0072]

[0073] Among them, a is a learnable attention weight vector, ∥ represents the concatenation operation of the vector, and Adj is Adj1 and Adj2, which represent the adjacency matrix obtained after cross attention. Through this formula, the model can adaptively assign weights to each neighbor node, thereby dynamically adjusting the node feature representation. Finally, by concatenating the output features of each head, the output is obtained:

[0074] H=Concat(H1′,H2′,H2′,…,H n ′) (20)

[0075] The concatenated features are then processed through an additional graph attention layer:

[0076]

[0077] Finally, the two are combined to form the final spatiotemporal feature representation H out . In summary, by combining the cross-fusion of time domain and space domain and the graph attention mechanism, the model can effectively learn the complex spatiotemporal interaction information in EEG signals on the graph structure. GAT can not only capture the local connection relationship between nodes, but also extract global context information through multiple attention heads, thereby further improving the accuracy and robustness of feature expression. Finally, the multi-domain interactive attention module can effectively extract and characterize the complex spatiotemporal features in EEG signals, providing richer and more accurate feature representation for subsequent classification tasks.

[0078] 4. Emotion decoding process

[0079] This method transforms the representation H from the multi-domain interactive attention module out Sentiment classification is achieved through the fully connected layer and the Softmax layer, as shown in formula (22). and i They represent the predicted label and true label of the i-th emotion respectively.

[0080]

[0081] In order to further improve the robustness of decoding emotional states, this paper introduces a learning strategy to measure the feature differences between the source domain and the target domain. This strategy evaluates the similarity between the source domain features and the target domain features by calculating the F norm between them. The specific calculation formula is as follows:

[0082]

[0083] Among them, X source and X target Represent the feature matrices of the source domain and the target domain respectively, N source is the number of samples in the source domain, and tr(·) represents the trace of the matrix, i.e., the sum of the diagonal elements of the matrix. The purpose of this strategy is to reduce the feature space difference between the source domain and the target domain, thereby helping the model learn a more robust feature representation and improving the stability of the model training process.

[0084] Therefore, the optimization strategy of the AC-GCL learning system consists of three main parts:

[0085] L=λ1L cls +λ2L C +λ3L diff (twenty four)

[0086] Among them, λ1, λ2 and λ3 are coefficients for adjusting the influence of each part. In the present invention, these coefficients are used to balance the weights of different parts to achieve the best classification accuracy and feature relevance. Such a design effectively captures the feature relevance between different domains, and the introduction of transfer learning provides support for enhancing the generalization ability and robustness of the model.

[0087] Calculate the gradient of loss L with respect to the parameters, and the optimizer Adam updates the parameters:

[0088]

[0089] Among them, η represents the learning rate, Represents the gradient of loss L with respect to parameter θ;

[0090] In this embodiment, the number of iterations is 300.

[0091] The effect of the present invention is described below in conjunction with specific data:

[0092] 1) In this embodiment, the technical effects of the present invention are verified based on the following data sets:

[0093] DREAMER is a multimodal database consisting of electroencephalogram (EEG) and electrocardiogram (ECG) signals recorded when emotions are evoked through audiovisual stimulation. The researchers recorded signals from 23 participants (14 males and 9 females, age mean = 26.6, sd = 2.7) watching 18 movie clips, and also recorded the participants' self-assessment of their emotional state after each stimulus with a score of 1-5, including valence, arousal and dominance, to transform the problem into a binary classification problem. The Emotivo epoc system was used to record at a sampling rate of 128Hz using 14 of the 16 channels (the left 2 are references). The preprocessing method in this paper was followed to remove artifacts from the EEG data, retaining only the last 60s of each clip, and further processing the EEG data using a window with an overlap of 1s for five bands. For each band, the DE feature was calculated.

[0094] The FACED dataset is a fine-grained emotion category affective computing EEG dataset released by Zhang Dan's research group at the Department of Psychology, Tsinghua University. A total of 123 subjects (75 females, average age = 23.2, ranging from 17 to 38 years old) were recruited for this dataset. The dataset includes 32-channel EEG data and self-reported emotion scores when the subjects watched 28 emotion-inducing video clips for nine emotions (anger, fear, disgust, sadness, entertainment, encouragement, joy, tenderness, and neutral emotions), including four negative emotions, four positive emotions, and three neutral emotions, so they can be further divided into three categories. The DE features are calculated from 5 frequency bands, following the practice of the SEED dataset. The present invention selects the first 30 subjects to evaluate the proposed method.

[0095] The SEED-IV dataset is an emotional EEG dataset of four emotions from Shanghai Jiao Tong University. The dataset includes 15 subjects, each of whom watched 24 video clips and recorded EEG data. These video clips are divided into 4 emotions, namely happiness, sadness, fear, and neutral. The experiment records 62 EEG channel data generated by each subject, watching the video clip for about 2 minutes. First, the EEG signal is downsampled to 200Hz, and the signal is filtered into 5 frequency bands, namely δ (1-4Hz), θ (4-8Hz), α (8-14Hz), β (14-31Hz), and γ (31-50Hz). Then, the DE features are extracted with a 4-second time window, which does not overlap in the 5 frequency bands.

[0096] 2) Experimental setup

[0097] In the subject-dependent experiment, the present invention uses a five-fold cross-validation for both the DREAMER and FACED datasets, that is, 80% of the data is randomly selected for training and 20% for testing. In the subject-independent experiment, the subject-independent experiment with the leave-one-subject protocol is used for both the DREAMER and FACED datasets, that is, one subject is used for testing and the remaining subjects are used for training, and the cycle ends when all subjects are used for testing. In order to verify the performance of the proposed method across sessions, the present invention uses the leave-one-session protocol to conduct cross-session experiments on SEED-IV, using the first and second sessions for training and the third session for testing to verify the generalization of the model.

[0098] The method proposed in the present invention is implemented in Python environment using Pytorch library. The experimental platform of the present invention is based on AMD Ryzen 7 3700X 8-Core 3.59GHz processor and 16GB memory. Adam optimizer is used in the training process, and the specific hyperparameter settings are shown in Table 1.

[0099] Table 1 Hyperparameter settings

[0100]

[0101] In Table 1, Hyperparameters represents hyperparameters.

[0102] In the model, the optimal learning rate and regularization parameters are dynamically selected.

[0103] 3) The prior art compared with the method of the present invention in this embodiment includes:

[0104] (1) ATDD-LSTM: Attention-based LSTM with domain discriminator (ATDD-LSTM) to characterize the nonlinear relationship between different electrodes. For details, please refer to "DU X, MAC, ZHANG G, et al. An Efficient LSTM Network for Emotion Recognition From Multichannel EEG Signals [J / OL]. IEEE Transactions on Affective Computing, 2022: 1528-1540. http: / / dx.doi.org / 10.1109 / taffc .2020.3013711.

[0105] DOI:10.1109 / taffc.2020.3013711.1”

[0106] (2) STGATE: It applies Transformer encoder to capture time-frequency features, which are fed into the spatiotemporal graph attention for sentiment classification. For details, please refer to "J.Li, W.Pan, H.Huang, J.Pan, and F.Wang," STGATE:

[0107] Spatial-temporal graph attention network with a transformer encoder for EEG-based emotion recognition, "Frontiers Human Neurosci., vol.17, Apr.2023, Art.no.1169949."

[0108] (3) Attention Conv-LSTM Network: A convolutional encoder is used to extract spatial features. After using the attention mechanism to assign adaptive weights to different frequency bands, a long short-term memory network is used to extract temporal features, and the temporal attention mechanism is used to obtain key temporal information. For details, please refer to "Y. Zhang, Y. Zhang, and S. Wang," An attention-based hybrid deep learning model for EEG emotion recognition," Signal, Image Video Process., vol. 17, no. 5, pp. 2305-2313, Jul. 2023."

[0109] (4) DGCNN: Use the adjacency matrix to dynamically learn the relationship between different EEG channels during training. For details, please refer to "SONG T, ZHENG W, SONG P, et al. EEG Emotion Recognition Using Dynamical Graph Convolutional Neural Networks [J / OL]. IEEE Transactions on Affective Computing, 2020: 532-541.

[0110] http: / / dx.doi.org / 10.1109 / taffc.2018.2817622.DOI:10.1109 / taffc.2018.2817622.”

[0111] (5) SparseDGCNN: Sparse constraints are introduced into the graph G to modify DGCNN. For details, please refer to "ZHANG G, YU M, LIUY J, et al. SparseDGCNN: Recognizing Emotion From Multichannel EEG Signals[J / OL]. IEEE Transactions on Affective Computing, 2023: 537-548. http: / / dx.doi.org / 10.1109 / taffc.2021.3051332.

[0112] DOI:10.1109 / taffc.2021.3051332.1”

[0113] (6) EEG-GCN: An adaptive brain network adjacency matrix is ​​designed to quantify the connection strength between channels and adaptively capture the significant sequential segment and spatial position information in the EEG signal. For details, please refer to "GAO Y, FU X, OUYANG T, et al. EEG-GCN: Spatio-Temporal and Self-Adaptive Graph Convolutional Networks for Single and Multi-View EEG-Based Emotion Recognition [J / OL]. IEEE Signal Processing Letters,

[0114] 2022:1574-1578.http: / / dx.doi.org / 10.1109 / lsp.2022.3179946.DOI:10.1109 / lsp.2022.3179946.”

[0115] (7) Siam-GCAN: Use multi-head attention mechanism and twin neural network to cluster the output of GCN according to Euclidean distance. For details, please refer to "Zeng H, Wu Q, **Y, et al. Siam-GCAN: A Siamese graph convolutional attention network for EEG emotion recognition [J]. IEEE Transactions on Instrumentation and Measurement, 2022, 71: 1-9."

[0116] (8) Hybrid Network: A hybrid network based on dynamic graph convolution and temporal self-attention. Dynamic graph convolution is used to obtain spatial features, self-attention is used to obtain temporal features, and then the representative knowledge of spatial topology and temporal context is integrated. For details, please refer to "C.Cheng, Z.Yu, Y.Zhang and L.Feng,"Hybrid Network Using Dynamic GraphConvolution and Temporal Self-Attention for EEG-Based Emotion Recognition," in IEEE Transactions on Neural Networks and Learning Systems, doi:10.1109 / TNNLS.2023.3319315."

[0117] (9) FBSTCNet: A spatiotemporal convolutional network that integrates power and connectivity features to recognize emotions by extracting power and / or connectivity features from EEG. For details, see “Huang W, Wang W, Li Y, et al. FBSTCNet: ASpatio-Temporal Convolutional Network Integrating Power and Connectivity Features for EEG-Based Emotion Decoding[J]. IEEE Transactions on Affective Computing, 2024.”

[0118] (10) MS-FRAN: A multi-source domain adaptation method that is effective in aligning the distribution of each pair of source and target domains, and can also reduce the distribution differences between multiple source domains. For details, see "W. Li, W. Huan, S. Shao, B. Hou and A. Song," MS-FRAN: A Novel Multi-Source Domain Adaptation Method for EEG-Based Emotion Recognition," in IEEE Journal of Biomedical and Health Informatics, vol. 27, no. 11, pp.

[0119] 5302-5313,Nov.2023,doi:10.1109 / JBHI.2023.3311338.”

[0120] 4) Verification on the DREAMER dataset

[0121] In this embodiment, the present invention conducts subject-dependent experiments and subject-independent experiments on the DREAMER dataset, and demonstrates the classification accuracy of 23 subjects on three emotion labels (arousal, pleasure, and dominance), as shown in Table 2.

[0122] Table 2 Experiments on the DREAMER dataset

[0123]

[0124] In Table 2, subject refers to the subject, subject_dependent refers to the subject-dependent experiment, and subject_independent refers to the subject-independent experiment.

[0125] The data in Table 2 show the specific accuracy of each subject under different emotion labels in the within-subject experiment, as well as the corresponding accuracy in the cross-subject experiment. The average accuracy of the within-subject experiment was 88.56% for arousal, 87.94% for pleasure, and 88.23% for dominance. These results outline the intensity of each subject's emotional response under the same conditions. In the cross-subject experiment, the accuracy of emotion label classification varied greatly among the subjects, and the average accuracy was lower than that of the within-subject experiment, which was 72.31% for arousal, 74.99% for pleasure, and 73.23% for dominance.

[0126] Further analysis shows that subject 4 showed consistent high emotional responses in the cross-subject experiment, with arousal, pleasure, and dominance being 82.42%, 85.35%, and 81.37%, respectively. In contrast, subject 13 had lower accuracy rates in pleasure and dominance, both at 65.52%, indicating that the subject's responses on these emotional dimensions were relatively bland. In addition, there was an imbalance in the accuracy of emotional labels among different subjects. For example, subject 19 had higher accuracy rates in arousal and dominance (84.38% and 76.66%), but lower accuracy rates in pleasure, at only 64.84%. This imbalance reflects the complex emotional responses of individuals to experimental situations.

[0127] Overall, the average score of pleasure across subjects in the experiment was slightly higher than that of other emotional dimensions, which may indicate that in the experimental situation, most subjects are more inclined to experience pleasant emotional reactions. This finding provides important insights into the emotional response patterns in the field of affective computing.

[0128] This embodiment also compares and analyzes the method proposed by the present invention with the current advanced emotion recognition methods, and the experimental results are shown in Table 3.

[0129] Table 3 Performance comparison of different methods on the DREAMER dataset

[0130]

[0131] In Table 3, Method represents a method.

[0132] Table 3 shows the comparison between the research method of the present invention and the methods in recent years on the DREAMER dataset, and evaluates the performance of each method in subject-dependent and subject-independent experiments, covering the three dimensions of arousal, valence and dominance. As can be seen from Table 3, the method of the present invention has achieved the highest classification accuracy in all three emotional dimensions, which is significantly better than other comparison methods. At the same time, it can be seen that the accuracy of the model of the present invention is close to that of the Hybrid Network. This is because the method also interacts with time and space, but the graph attention interaction effect adopted by the present invention is better. In addition, it can be clearly observed that the difference between the method of the present invention and STGATE is the largest, which may be because the method only captures the time-frequency feature input graph attention and does not extract spatial features. These methods use different strategies in spatial and temporal feature extraction, resulting in differences in performance.

[0133] In general, the proposed method shows superior classification ability on the DREAMER dataset, especially in complex cross-subject situations, further proving its effectiveness and robustness in emotion recognition tasks. This advantage is mainly attributed to the extraction of spatial features by the dynamic graph constraint learning module proposed in the present invention and the effective fusion mechanism adopted for multiple domains, which can effectively capture the complex features in EEG signals, especially the information integration of time and space domains, which plays a key role in emotion recognition.

[0134] 5) Verification on the FACED dataset

[0135] In this embodiment, the present invention conducts subject-dependent and subject-independent experiments on the FACED dataset, and demonstrates the sentiment classification accuracy of 30 subjects, as shown in Table 4.

[0136] Table 4 Experimental accuracy on the FACED dataset

[0137]

[0138] Overall, the average accuracy of the dataset is high, indicating that the model is relatively stable in distinguishing the three types of labels. However, the experimental results also reveal certain individual differences, reflecting that the different characteristics of different subjects may affect the classification effect.

[0139] In the subject-dependent experiment, some subjects performed very well, such as subject 26, whose accuracy reached 100%, indicating that the model is highly adaptable to the data of these subjects. This may be because the data features of these subjects are clearer, with less noise or significant feature differences between different categories. At the same time, the accuracy of some subjects is lower, such as subject 24, which is only 66.62%, suggesting that the model faces greater challenges in processing the data of these subjects, which may be due to the noise in the data or the complexity of the features. Overall, the accuracy of most subjects is concentrated between 85% and 98%, indicating that the model has strong classification ability on the data of these subjects and the overall performance is relatively stable.

[0140] In the independent experiments of the subjects, the accuracy fluctuated greatly, reflecting the performance differences of different subjects in the emotion classification task. For example, the accuracy of subjects 7 and 23 was 72.08% and 71.17%, respectively, which was the highest among all subjects. This may mean that their physiological signal characteristics are clearer, or their emotional expressions are more consistent, which is easy for the model to recognize. In contrast, the accuracy of subject 15 was only 48.35%, which is far below the average level. It may be due to the large data noise or the unique emotional response pattern, which makes it difficult for the model to accurately classify. Similarly, the accuracy of subjects 8 and 2 was also relatively low, at 50.19% and 51.87%, respectively, which may reflect that their physiological signals in the experiment had large individual differences.

[0141] Table 5 shows the classification accuracy of different methods in the intra-subject and independent subject experiments on the FACED dataset. By comparing the performance of these methods, the generalization ability and feature extraction effect of the model in the emotion classification task can be better evaluated. As can be seen from the table, in the intra-subject experiment, the accuracy of various methods is above 80%, indicating that these models show strong classification ability when processing the intra-subject classification task of the FACED dataset. It is worth noting that the method proposed in the present invention achieved the highest accuracy in the subject-dependent experiment, reaching 92.29%, showing its significant advantages in EEG signal feature extraction and cross-domain feature fusion. In contrast, the accuracy of Hybrid Network is 91.56%, which is slightly lower than the method of the present invention, but still performs well. This shows the importance of the interaction between time domain and spatial domain features in improving model performance in emotion classification. Methods such as SparseDGCNN and EEG-GCN also performed well, achieving accuracies of 85.62% and 87.38% respectively. This shows that the feature extraction strategy based on graph structure has better adaptability to EEG signals, but there is still a slight gap compared with the method of the present invention.

[0142] Table 5 Accuracy of FACED dataset experiments under different methods

[0143]

[0144] In the independent experiment of the subjects, the method of the present invention also performed well, with an accuracy of 72.29%, which is higher than all the comparison methods. This further proves the strong adaptability of the dynamic graph constraint learning module and its cross-domain feature fusion mechanism proposed in the present invention in dealing with individual differences. The Hybrid Network also performed well in the independent experiment of the subjects, with an accuracy of 69.81%, indicating that its interaction with spatial features and temporal features contributes to the generalization ability of the model. In contrast, the performance of ATDD-LSTM and STGAT in the independent experiment of the subjects was significantly lower than that of other models, with accuracy rates of 50.96% and 51.34%, respectively, which shows that a single temporal modeling or attention mechanism has weak adaptability to the independent situation of the subjects and it is difficult to effectively capture the feature differences between different individuals.

[0145] In general, the method of the present invention shows significant advantages in both subject-dependent and subject-independent experiments, especially in complex subject-independent situations, showing strong generalization ability.

[0146] 6) Verification on SEED-IV dataset

[0147] In order to further verify the generalization ability of the model, the present invention conducted a cross-session experiment on the SEED-IV dataset. The experimental results are shown in Table 6.

[0148] Table 6 Cross-session experiments on the SEED-IV dataset

[0149]

[0150] From the experimental results, the average classification accuracy of the model in the cross-session setting is 72.29%, indicating that it has a certain generalization ability between different sessions. However, the accuracy rates of the subjects vary significantly, with the lowest being 62.73% and the highest being 87.48%. This difference may reflect the changes in the emotional characteristics of different subjects between different sessions. Subject 15 has the highest classification accuracy, indicating that the emotional characteristics of this subject are highly consistent between different sessions, allowing the model to classify well. At the same time, the accuracy rates of subjects 4 and 7 are also significantly higher than the average level, further verifying that the model has good generalization ability on these data. However, the accuracy rates of some subjects are low, which may be due to the large fluctuations in the emotional characteristics of these subjects in different sessions, making it difficult for the model to effectively capture their characteristics.

[0151] In addition, the present invention compares the cross-session experimental results of the method proposed in the present invention with other advanced methods on the SEED-IV dataset. The specific results are shown in Table 7.

[0152] Table 7 Cross-session comparison experiments with other methods on the SEED-IV dataset

[0153]

[0154] The comparison results show that the method proposed in the present invention performs best among all the comparison methods, with an accuracy of 72.29%. This result shows the superiority of the method of the present invention in the cross-session classification task. In contrast, the existing methods often ignore the comprehensive fusion of these features when extracting the spatial or temporal features of EEG signals, resulting in less than ideal feature extraction effects. The method of the present invention not only effectively captures the complex spatial features of EEG signals, but also improves the generalization ability of the model through a reasonable fusion mechanism, thereby performing well in the cross-session sentiment classification task.

[0155] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the scope of protection of the present invention is not limited to such specific statements and embodiments. For those skilled in the art, the present invention may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A multi-domain constrained emotion recognition system inspired by adaptive cognitive graphs, characterized by: include: Differential entropy extraction module, dynamic graph constraint learning module, sequence attention module, and multi-domain interactive attention module; The differential entropy extraction module is used to extract the differential entropy features of EEG signals; The dynamic graph constraint learning module includes: a brain topology structure construction unit, a drive structure generation unit, a calibration unit, and a first feature extraction unit; the brain topology structure construction unit adaptively generates a brain topology structure based on differential entropy features and learnable parameters; the drive structure generation unit is used to generate a random drive structure of the same size as the brain topology structure; the calibration unit calibrates the brain topology structure based on the random drive structure; the first feature extraction unit extracts the spatial domain features of the EEG signal through the calibrated brain topology structure and the drive structure according to the differential entropy features; The sequence attention module includes a differential entropy extraction unit, a position encoding unit and a second feature extraction unit; the position encoding unit is used to add position information to the differential entropy feature, and the second feature extraction unit extracts the time domain feature of the EEG signal according to the differential entropy after adding the position information; The multi-domain interactive attention module obtains emotion recognition results based on the spatial domain features of EEG signals and the temporal domain features of EEG signals.

2. A multi-domain constrained emotion recognition system inspired by adaptive cognitive graphs according to claim 1, characterized in that: The first feature extraction unit is implemented using a graph neural network.

3. The multi-domain constrained emotion recognition system inspired by adaptive cognitive graph according to claim 2, characterized in that: The second feature extraction unit includes a self-attention mechanism and a feedforward neural network; the self-attention mechanism is used to capture the dependencies between input sequences, and the feedforward neural network performs a nonlinear transformation on the dependencies obtained by the self-attention mechanism; finally, based on the dependencies obtained by the self-attention mechanism and the nonlinear transformation results of the feedforward neural network, the spatial domain characteristics of the EEG signal are obtained.

4. The multi-domain constrained emotion recognition system inspired by adaptive cognitive graph according to claim 3, characterized in that: The multi-domain interactive attention module fuses the spatial domain features of the EEG signal with the temporal domain features of the EEG signal through a cross-attention mechanism to obtain fused features.

5. A multi-domain constrained emotion recognition method inspired by adaptive cognitive graphs, characterized in that include: S1, extracting differential entropy features of EEG signals, and constructing brain topology structures based on the extracted differential entropy features; S2, randomly generate a driving structure of the same size as the brain topology constructed in step S1; S3, obtaining the spatial domain features of the EEG signal based on the driving structure and the brain topology structure constructed in step S1; S4, using position encoding to add position information to the extracted differential entropy features; S5, the result obtained in step S4 is passed through the self-attention mechanism and the feedforward network to obtain the time domain features of the EEG signal; S6, the temporal and spatial features of EEG signals are fused through the cross-attention mechanism; S7. Obtain an emotion recognition result based on the fusion result obtained in step S6.

6. The multi-domain constrained emotion recognition method inspired by adaptive cognitive graph according to claim 5, characterized in that: Step S1 adaptively generates a brain topology structure by introducing learnable parameters; the adaptively generated brain topology structure is expressed as: Where W A Q is the learnable parameter of the query in self-attention, W A K is the learnable parameter of the key in self-attention, is the scaling factor used to prevent the value from being too large; T represents transpose, and X represents the EEG signal.

7. The method of multi-domain constrained emotion recognition inspired by adaptive cognitive graph according to claim 6, characterized in that: The implementation process of step S3 is: Set the screening parameter b to screen the adaptively generated brain topology structure, so as to obtain the screened brain topology structure f adj ; Let the randomly generated drive structure be s adj ; f adj 、s adj Perform calibration; the calibration expression is: Among them, f′ adj is the attention-based brain topology matrix obtained after calibration, s′ adj is the attention-based driving structure matrix obtained after calibration.

Images