An Emotion Recognition System and Method with Multi-Domain Constraints Inspired by Adaptive Cognitive Maps

Abstract

The present invention discloses a multi-domain constrained emotion recognition system and an emotion recognition method inspired by adaptive cognitive maps, which are applied to the field of electroencephalogram (EEG) signal processing. In view of the fact that traditional emotion classification methods extract features from EEG signals manually and then select key features and input them into traditional classifiers such as support vector machines (SVMs). However, manual feature extraction requires professional prior knowledge, and most methods mainly focus on feature selection and utilization of time information, thus insufficiently utilizing spatial information. The present invention designs a space-guided dynamic graph constraint learning module, which realizes complex spatial domain feature extraction that overcomes cognitive prior information by adaptively generating a topological structure and constraining the functional relationships of the topological structure. At the same time, a time-stimulated sequential attention module is proposed to capture global temporal dependence features by focusing on more important times in the sequence. Further, the present invention designs a new multi-domain interaction attention module, which overcomes traditional static cross-domain interaction and realizes the constraint of multi-domain difference information and the aggregation of complementary information. In addition, in order to better utilize information transfer between different domains, the present invention also introduces a constraint to reduce the distribution difference between source domain and target domain features, and improve the similarity and transferability of cross-domain features.

Description

Technical Field

[0001] The present invention belongs to the field of electroencephalogram signal processing, and particularly relates to an emotion classification technology. Background Art

[0002] In neurobiology, the generation of emotions is closely related to brain activities. After external stimuli are transmitted to the brain, corresponding brain activities will be triggered, and human emotions can be recognized by capturing the responses generated by these activities. Accurate recognition of emotions is of great significance in the fields of mental health tracking and human-computer interaction. Emotional responses can be manifested through physiological signals and non-physiological signals. Compared with non-physiological signals such as facial expressions and voices that are easily subject to subjective influences, physiological signals are more popular due to their advantage of being difficult to disguise. Among these physiological signals, electroencephalogram (EEG) signals are widely used in emotion research due to their portability, non-invasiveness, and high temporal resolution characteristics.

[0003] Traditional emotion classification methods are to manually extract features from EEG signals and then select key features and input them into traditional classifiers such as Support Vector Machine (SVM). However, manual feature extraction requires professional prior knowledge, and most methods mainly focus on feature selection and utilization of time information, thus insufficiently utilizing spatial information. Therefore, with the deepening of research on emotion recognition, more and more deep learning methods have been used for emotion detection. High temporal resolution and spatial resolution are the basic characteristics of electroencephalogram. Therefore, some studies have separately explored the importance of temporal information and spatial information. For example, Du et al. proposed the ATDD-LSTM (Attention-based LSTM with Domain Discriminator, a deep learning model for emotion recognition) framework, which applied LSTM (Long Short-Term Memory) to EEG channel sequences to characterize the non-linear relationship between multi-channel EEG signals. The dynamic graph convolutional neural network proposed by Song et al. uses a dynamic adjacency matrix to learn more spatial discriminative features. In addition, many studies focus on capturing the combination of specific spatial topological information and temporal dependence representations. For example, Ding et al. proposed a spatio-temporal feature extraction using multi-scale convolution. Ning et al. proposed a three-dimensional dense network based on spatial-spectral-temporal attention (SST-Net) and integrated high-dimensional emotion features through self-attention. Liang et al. proposed a hybrid unsupervised generative adversarial network (EEGFuseNet), which is trained in an unsupervised manner to automatically represent emotion features covering spatial and temporal dynamics.

[0004] Fundamentally speaking, these works do not consider the impact of spatio-temporal cross-domain information interaction on emotion recognition. Based on this problem, Cheng et al. proposed a hybrid network that uses cross-attention to interactively extract spatio-temporal features. However, the complexity and unpredictability of brain activities still pose obstacles to efficient emotion recognition. In neurophysiology, the neural structure of the brain determines that there is interaction and cooperation between spatial processing and temporal processing in different regions. Therefore, how to efficiently decode spatio-temporal interaction information becomes a problem that needs to be solved.

[0005] From the perspective of improving the decoding of emotional states, many works based on graph neural networks have achieved some results through data-driven methods. For example, Pan et al. proposed a spatio-temporal self-constructing graph neural network to appropriately reflect the connections between different regions of the brain. Qu et al. proposed constructing edge-connected features to further improve the emotion recognition effect. However, they use cognitive prior information such as the phase locking value (PLV) and Pearson correlation coefficient (PCC) as the graph structure, which causes the structural information to be fixed and makes it difficult to capture important information from complex domain structures. Therefore, dynamically mining the emotional cognitive structure of the brain and effectively depicting emotional representations may provide more room for further improving the performance of emotion recognition systems. Summary of the Invention

[0006] To solve the above technical problems, the present invention proposes a multi-domain constrained learning system (AC-DCL) and an emotion recognition method inspired by an adaptive cognitive map.

[0007] One of the technical solutions adopted by the present invention is: a multi-domain constrained emotion recognition system inspired by an adaptive cognitive map, including: a dynamic graph constraint learning module, a sequential attention module, and a multi-domain interaction attention module; the dynamic graph constraint learning module extracts the spatial domain features of EEG signals based on the constructed brain topology structure and the dynamically generated topology structure; the sequential attention module is used to extract the temporal domain features of EEG signals; the multi-domain interaction attention module obtains the emotion recognition result based on the spatial domain features of EEG signals and the temporal domain features of EEG signals.

[0008] Another technical solution adopted by the present invention is: a multi-domain constrained emotion recognition method inspired by an adaptive cognitive map, including:

[0009] S1. Extract the differential entropy features of EEG signals, and construct a brain topology structure according to the differential entropy features;

[0010] S2. Randomly generate a driving structure with the same size as the brain topology structure constructed in step S1;

[0011] S3, obtaining the spatial domain features of the EEG signal based on the driving structure and the brain topology constructed in step S1;

[0012] S4, and using position encoding to add position information to the differential entropy feature extracted in step S1;

[0013] S5, the result obtained in step S4 is passed through the self-attention mechanism and the feedforward network to obtain the time domain features of the EEG signal;

[0014] S6. Fusion of temporal and spatial features of EEG signals through a cross-attention mechanism;

[0015] S7. Obtain an emotion recognition result based on the fusion result obtained in step S6.

[0016] Beneficial effects of the present invention: The present invention proposes a multi-domain constraint learning system (AC-DCL) inspired by adaptive cognitive graphs, which simultaneously embeds spatiotemporal representative knowledge to improve the performance of emotion recognition systems. In AC-DCL, a spatially guided dynamic graph constraint learning module is first carefully designed. By adaptively generating topological structures and constraining the functional relationships of topological structures, it achieves complex spatial domain feature extraction that overcomes cognitive prior information. At the same time, a time-inspired sequence attention module is proposed to capture global temporal dependency features by focusing on more important times in the sequence. Furthermore, the present invention designs a new multi-domain interaction attention module, which overcomes the traditional static cross-domain interaction and realizes the constraint of multi-domain difference information and the aggregation of complementary information. In addition, in order to better utilize information transfer between different domains, the present invention also introduces a constraint to reduce the distribution difference between source domain and target domain features, thereby improving the similarity and transferability of cross-domain features. The present invention uses three public emotion datasets to evaluate the proposed AC-DCL learning system and consistently demonstrates the advanced decoding performance in three experiments: subject dependence, between subjects, and cross-session. The advantages of the AC-DCL learning system of the present invention are as follows:

[0017] (1) By designing a space-guided dynamic graph constraint learning module, the AC-DCL learning system is able to adaptively generate and constrain the functional relationships of topological structures in emotion recognition tasks for the first time. This breaks through the limitations of traditional reliance on prior knowledge and more effectively captures complex domain structure information.

[0018] (2) A time-inspired sequence attention module is proposed to capture global temporal dependency features by focusing on the more important time in the sequence.

[0019] (3) By designing a new multi-domain interaction attention module, AC-DCL effectively overcomes the traditional static cross-domain interaction and realizes the constraint of multi-domain difference information and the aggregation of complementary information.

[0020] (4) The proposed AC-DCL learning system captures stable cognitive functions from complex and dynamic cognitive structures for the first time, providing a potential solution for further revealing the emotional cognitive functions of the brain. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is the system architecture diagram of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0022] To facilitate those skilled in the art to understand the technical content of the present invention, the content of the present invention will be further explained below with reference to the accompanying drawings.

[0023] The present invention makes efficient use of the spatial topological information and time series information of EEG data, and designs a new method (AC-DCL) for emotion recognition. The overall architecture of the model is as Figure 1 shown. This architecture includes three main functional modules: a dynamic graph constraint learning module, a time series attention module, and a cross-domain cross-attention module. After explaining the symbols and nomenclature here, each of the proposed sub-modules is described in detail.

[0024] Assume the EEG signal of each sample is S = {(Xi, Yi), i = 1, 2,..., n}, where X i ∈R C×T , which is a 2D matrix representing the i-th trial of the preprocessed EEG sample with C channels and T time sampling points. n represents the total number of EEG trials. The corresponding true label of X is denoted as Y i ∈R n×1 , where the EEG emotion classification task has h categories.

[0025] In the entire framework, the extracted differential entropy (DE) features are used as the input of the model, and its dimension is denoted as [N, D, C, F]. Here, D represents the feature dimension obtained after extracting DE from X i . For example, for each EEG sample X i ∈R C×T , after extracting the DE features, its dimension will be converted to X d ∈R D×C×F . Subsequently, the present invention uses X d as the input of the model to obtain spatial topological features and attention-enhanced time features respectively. Finally, the spatio-temporal features of the obtained EEG signal are fused using the multi-domain interaction attention module, and the final predicted label is obtained through the classification layer

[0026] The following will explain each module in detail:

[0027] 1. Dynamic Graph Constraint Learning Module

[0028] The EEG signal is first pre - processed. That is, the original EEG data is first down - sampled to a frequency of 200 Hz, and then the data is processed using a band - pass filter between 1 Hz and 75 Hz to filter out noise and eliminate artifacts. After that, the EEG signal is divided into several non - overlapping time periods by a sliding window, and the duration of each time period is t seconds. In this embodiment, t is taken as 4 seconds. Subsequently, the DE features are extracted from the pre - processed signal as the input of the model. The topological structure of the brain is crucial for emotion recognition. Usually, PCC and PLV, etc., are used to construct the brain network, but they are static connections obtained through prior knowledge. Therefore, the goal of this module is to learn an adaptive topological structure to reflect the differences in brain functional connectivity patterns under different emotional states. The present invention introduces learnable parameters to adaptively generate the topological structure, which is expressed as follows:

[0029]

[0030] where \(W\) A Q and \(W\) A K are the learnable parameters for calculating the query (Query: \(XW\) A Q ) and the key (Key: \(XW\) A K ) in self - attention. is a scaling factor used to prevent numerical values from being too large; \(T\) represents the transpose. Therefore, through linear transformation and self - attention, the attention - based score matrix \(A\) adj \(\in R\) C×C , that is, the adaptively generated topological structure.

[0031] Then, based on this, the present invention sets a screening parameter \(b\) (the value range of \(b\) is 0.1, 0.2,..., 0.9), and in this embodiment, the value of \(b\) is 0.8; based on the screening parameter \(b\), \(A\) adj is screened to obtain \(f\) adj .

[0032]

[0033] This module then also randomly generates a driving structure \(s\) adj \(\in R\) C×C , and then the screened brain topological structure \(f\) adj \(\in R\) C×C and the randomly generated driving structure are respectively recalibrated to themselves to promote the acquisition of stable patterns.

[0034]

[0035] Among them, Conv is a one-dimensional convolution. Therefore, the final attention-based matrix f' is obtained. adj ∈R C×C and the driving structure s' adj ∈R C×C .

[0036] Given a graph G=(V, E, A), where V(|V| = V) is the set of channel nodes, E is the set of graph edges, and A is the graph adjacency matrix. Although traditional graph convolutions usually rely on the graph Laplacian matrix, here a dynamically generated topological structure is directly used as the adjacency matrix for convolution, emphasizing the model's dependence on the relational structure in the adjacency matrix and making full use of the flexibility and expressive power of the attention mechanism. In addition, to improve the running efficiency, the present invention introduces Chebyshev polynomials to accelerate the graph convolution calculation, and at the same time can handle a wider range of adjacency relationships to extract richer graph structure information. Each order of the polynomial corresponds to the local features of the graph, and the calculation process of the k-th order Chebyshev polynomial is shown in formula (4):

[0037]

[0038] The initial conditions of the Chebyshev polynomials are T0(X)=1, T1(X)=X; that is, the first-order Chebyshev polynomial T1(X)=X, and the second-order Chebyshev polynomial <s

[0039] Given X l-1 as the input of the l-th graph convolution layer, X [[ID=U27]] l-1 which is also the output of the DE features after passing through the previous l-1 graph convolution layers, is the normalized adjacency matrix in the graph convolution operation, which is the result of preprocessing the graph adjacency matrix A, aiming to improve the numerical stability and convergence effect of the graph convolution. The output calculation of the l-th layer graph convolution is shown in formula (5):

[0040]

[0041] where θ l k are the trainable parameters of the k-th order of the corresponding graph convolution layer; σ represents the activation function. As the number of iterations increases, the node features propagate in the graph structure, and finally the global spatial information aggregation representation X S is obtained. To avoid information loss, the present invention directly sums the outputs of each graph convolution layer without performing a graph pooling operation, thus integrating the features of all nodes into a graph-level representation

[0042] By combining f' adj and s'adj As the adjacency matrix of graph convolution respectively, based on the above Chebyshev polynomials, as Figure 1 shown, the adaptive topological features and driving structure features can be obtained respectively; the spatial domain features of EEG signals are obtained after fusing these two features.

[0043] It should be emphasized that traditional GNNs (Graph Neural Networks) usually directly rely on the fixed input graph structure. However, in practical applications, the graph topology is often extremely complex and may contain implicit information, which is difficult to be captured by prior knowledge. In this case, the performance of graph neural networks may be significantly affected. Therefore, the present invention aims to adaptively adjust the topology of the graph during the learning process to better adapt to the characteristics of the data. Here, the present invention designs a new method L C that can directly constrain the relationship between the feature matrix and the topology matrix, enabling the model to more effectively integrate node features and topological information. Moreover, enhancing the correlation representation helps to improve the model's understanding of node relationships. At the same time, due to the optimization of the topology, the robustness of the model is enhanced, which enables it to perform better when dealing with actual complex data. Specifically, as shown in formula (6):

[0044] L C = ||X S ·X S T - f′ adj ·A adj +(1 - A adj )·s adj || (6)

[0045] Among them, ||·|| represents the Frobenius norm. The core idea of this method is to force the model to learn a topology that can better represent the relationship between nodes by constraining the relationship between the autocorrelation of the node feature matrix and the adaptive topology. This structure not only depends on the original topology but also introduces a driving matrix, further enhancing the model's ability to express potential complex relationships.

[0046] 2. Temporal excitation sequence converter

[0047] Research shows that there are also obvious differences in the brain patterns under the same emotional state over time. Therefore, in this module, more attention will be paid to the time segments in the EEG sequence that are more related to emotions. First, adding positional information to X d using positional encoding can add time information to the input embedding vector, allowing the model to capture the relative and absolute positional relationships between sequences during training and inference, thereby improving the model's understanding ability of sequence structure. The calculation is as follows:

[0048]

[0049] Output(X) = Dropout(X d + PE)(8)

[0050] Equation (7) represents the time information in the sequence, where d model is the dimension of the model, p is the position, and i is the dimension index. Then, the input X is added with position encoding information using Equation (8), where d is the position encoding matrix. Subsequently, the self-attention mechanism is used to capture the dependencies between the input sequences, and then a non-linear transformation is performed through a feed-forward neural network. For each time step t, the self-attention mechanism is calculated as follows:

[0051] Q = ψ(XW Q ), K t = ψ(XW K ), V t = ψ(XW V )(9)

[0052]

[0053] where the function ψ(·) represents a linear transformation; W Q ∈ R F×C , W K ∈ R F×C , W V ∈ R F×C are the learnable parameters of self-attention; d k is the coefficient factor. In this invention, multi-head self-attention is adopted, allowing the model to compute attention in parallel in different representation subspaces:

[0054] MultiHead(Q, K, V) = Concat(head1, head2,..., head h )W O (11)

[0055] where is the learnable parameter matrix, and h represents the number of heads. Each head head i is an independent self-attention mechanism:

[0056]

[0057] After passing through the multi-head self-attention mechanism and the feed-forward network, the output is:

[0058] Output(X t) = LN(LN(X + MultiHead(Q, K, V)) + FFN(LN(X + MultiHead(Q, K, V))))(13)

[0059] Among them, LN is a layer normalization, which is used to reduce the impact caused by the parameter changes between layers during the training process, thereby accelerating convergence and stabilizing training. FFN is a feed-forward neural network, which contains two fully connected layers and an activation function, and performs a non-linear transformation on each input position to enhance the representation ability of the model.

[0060] 3. Multi-domain interaction attention module

[0061] In order to fully exploit the complex interaction information in the time domain and spatial domain of EEG signals, the present invention proposes a multi-domain interaction attention module based on graph attention cross-attention. The core idea of this model is to generate a cross-domain adjacency matrix through the cross-attention mechanism and further enhance the feature expression ability using a graph attention network. The Graph Attention Network (GAT) can capture the implicit associations and non-local dependencies between nodes by assigning adaptive attention weights to different nodes, thereby effectively modeling complex spatial structure information. This design enables the model to dynamically adjust the feature weight distribution between different domains and improve the capture ability of key features and the overall expression effect in EEG signals.

[0062] This module first integrates the time-domain and spatial-domain features of EEG signals through the cross-attention mechanism to achieve cross-domain information integration. During this process, the cross-attention mechanism can effectively focus on and extract more important information in each module, thereby strengthening the expression of key features.

[0063]

[0064] Among them, W Q 1 , W K 1 , W V 1 , W Q 2 , W K 2 , W V 2 are learnable weight matrices, and ψ(·) represents a linear transformation. The cross-domain fused features O1 and O2 are obtained according to the above formula.

[0065] After obtaining the fused features, the model generates a cross - domain adjacency matrix for the graph attention network. This adjacency matrix not only contains the correlation information between the time domain and the spatial domain but also can capture the complex interactions between different brain regions. The adjacency matrix after generating the fused features is generated by the following formula, which represents the weighted relationship between different nodes (i.e., different time points or brain regions), where ψ(·) represents a linear transformation.

[0066]

[0067] Next, the present invention introduces multi - heads into GAT, that is, calculates the output of each attention head and then concatenates these outputs. For each attention head i, its calculation process is as follows. First, perform a feature transformation:

[0068] H i =W i O (17)

[0069] where W i is the weight matrix of the i - th attention head, O is the input feature, here it is the fused features O1 and O2. For each node i, its new feature representation H i ' is calculated by the following formula:

[0070] H i ′=ELU(∑ j∈N(i) α i,j H i ) (18)

[0071] ELU is the activation function, N(i) represents the set of neighbor nodes of node i. α i,j is the attention weight between node i and node j, which can be expressed as:

[0072]

[0073] where a is the learnable attention weight vector, || represents the vector concatenation operation, Adj is Adj1 and Adj2, representing the adjacency matrix obtained after cross - attention. Through this formula, the model can adaptively assign weights to each neighbor node, thereby dynamically adjusting the node feature representation. Finally, by concatenating the output features of each head, the output is obtained:

[0074] H=Concat(H1′,H2′,H2′,…,H n ′) (20)

[0075] The concatenated features are further processed by an additional graph attention layer:

[0076]

[0077] Finally, the two are combined to form the final spatio-temporal feature representation H out . In summary, by combining the cross-fusion of the time domain and the space domain and the graph attention mechanism, the model can effectively learn the complex spatio-temporal interaction information in EEG signals on the graph structure. GAT can not only capture the local connection relationships between nodes, but also extract global context information through multiple attention heads, thereby further improving the accuracy and robustness of feature representation. Finally, the multi-domain interaction attention module can effectively extract and characterize the complex spatio-temporal features in EEG signals, providing a richer and more accurate feature representation for subsequent classification tasks.

[0078] 4. Emotional decoding process

[0079] This method uses the representation H from the multi-domain interaction attention module out to perform emotion classification through a fully connected layer and a Softmax layer, as shown in formula (22). Where and y i represent the predicted label and the true label of the i-th type of emotion respectively.

[0080]

[0081] To further improve the robustness of decoding the emotional state, the present invention introduces a learning strategy for measuring the feature difference between the source domain and the target domain. This strategy evaluates the similarity by calculating the F-norm between the source domain features and the target domain features. The specific calculation formula is as follows:

[0082]

[0083] where, X source and X target represent the feature matrices of the source domain and the target domain respectively, N source is the number of samples in the source domain, and tr(·) represents the trace of the matrix, that is, the sum of the diagonal elements of the matrix. The purpose of this strategy is to reduce the feature space difference between the source domain and the target domain, thereby helping the model learn a more robust feature representation and improving the stability of the model training process at the same time.

[0084] Therefore, the optimization strategy of the AC-GCL learning system consists of three main parts:

[0085] L = λ1L cls + λ2L C + λ3L diff (24)

[0086] Among them, λ1, λ2, and λ3 are coefficients for adjusting the influence of each part. In the present invention, these coefficients are used to balance the weights of different parts to achieve the best classification accuracy and feature correlation. Such a design effectively captures the feature correlation between different domains, and at the same time, the introduction of transfer learning provides support for enhancing the generalization ability and robustness of the model.

[0087] Calculate the gradient of the loss L with respect to the parameters, and the optimizer Adam updates the parameters:

[0088]

[0089] Among them, η represents the learning rate, represents the gradient of the loss L with respect to the parameter θ;

[0090] In this embodiment, the number of iterations is 300.

[0091] The effects of the present invention will be described below with specific data:

[0092] 1). In this embodiment, the technical effects of the present invention are verified based on the following datasets respectively

[0093] DREAMER is a multimodal database composed of electroencephalogram (EEG) and electrocardiogram (ECG) signals, which records the signals when emotions are stimulated by audiovisual stimuli. Researchers recorded the signals of 23 participants (14 males and 9 females, average age = 26.6, sd = 2.7) watching 18 movie clips. At the same time, the self-assessment of the participants' emotional state after each stimulus with a score of 1-5, including valence, arousal, and dominance, was also recorded, and the problem was transformed into a binary classification problem. Recorded at a sampling rate of 128 Hz using a 14-channel (2 left channels are references) emotivo epoc system with 16 channels. The present invention follows the preprocessing method in to remove artifacts in the EEG data, only retaining the last 60 s of each clip, and further processing the EEG data using a window with an overlap of 1 s in five bands. For each band, DE features are calculated.

[0094] The FACED dataset is a fine-grained emotion category affective computing EEG dataset released by the Zhang Dan research group of the Department of Psychology at Tsinghua University. A total of 123 subjects were recruited for this dataset (75 females, average age = 23.2, ranging from 17 to 38 years old). This dataset includes 32-channel electroencephalogram data and self-reported emotion scores of the subjects when watching 28 emotion-inducing video clips, which are targeted at nine emotions (anger, fear, disgust, sadness, amusement, inspiration, joy, tenderness, and neutral emotion), including four negative emotions, four positive emotions, and three neutral emotions, so it can be further divided into three classifications. The DE features are calculated from 5 frequency bands, following the practice of the SEED dataset. In this invention, the first 30 subjects are selected for evaluating the proposed method.

[0095] The SEED-IV dataset is an emotion EEG dataset of four emotions from Shanghai Jiao Tong University. This dataset includes 15 subjects, and each subject watches 24 video clips and records EEG data. These video clips are divided into 4 emotions, namely happiness, sadness, fear, and neutral. The experiment records 62-channel EEG data generated by each subject and watches the video clips for about 2 minutes. First, the EEG signal is downsampled to 200Hz, and the signal is filtered into 5 frequency bands, namely δ (1 - 4Hz), θ (4 - 8Hz), α (8 - 14Hz), β (14 - 31Hz), and γ (31 - 50Hz). Then, DE features are extracted with a 4-second time window without overlap in the 5 frequency bands.

[0096] 2) Experimental settings

[0097] In the subject-dependent experiment, this invention uses five-fold cross-validation for both the DREAMER and FACED datasets, that is, randomly select 80% of the data for training and 20% for testing. In the subject-independent experiment, a subject-independent experiment using the leave-one-subject protocol is adopted for both the DREAMER and FACED datasets, that is, one subject is used for testing, and the remaining subjects are used for training, and the cycle continues until all subjects have been used as tests and then ends. In order to verify the performance of the proposed method across sessions, this invention uses the leave-one-session protocol for cross-session experiments on SEED-IV, using the first and second sessions for training and the third session for testing to verify the generalization of the model.

[0098] The method proposed in this invention is implemented using the Pytorch library in a Python environment. The experimental platform of this invention is based on an AMD Ryzen 7 3700X 8-Core 3.59GHz processor and 16GB of memory. The Adam optimizer is used during the training process, and the specific hyperparameter settings are shown in Table 1.

[0099] Table 1 Hyperparameter setting table

[0100]

[0101] In Table 1, Hyperparameters represents hyperparameters.

[0102] In the model, the optimal learning rate and regularization parameter are dynamically selected.

[0103] 3), The prior arts compared with the method of the present invention in this embodiment include:

[0104] (1) ATDD-LSTM: An attention-based LSTM with a domain discriminator (ATDD-LSTM) that characterizes the non-linear relationship between different electrodes. Specifically refer to "DU X, MAC, ZHANG G, et al. An Efficient LSTM Network for Emotion Recognition From Multichannel EEG Signals[J / OL]. IEEE Transactions on Affective Computing, 2022: 1528-1540. http: / / dx.doi.org / 10.1109 / taffc .2020.3013711.

[0105] DOI: 10.1109 / taffc.2020.3013711.1”

[0106] (2) STGATE: It applies a Transformer encoder to capture time-frequency features, which are fed into a spatio-temporal graph attention for emotion classification. Specifically refer to "J.Li, W.Pan, H.Huang, J.Pan, and F.Wang, "STGATE:

[0107] Spatial-temporal graph attention network with a transformer encoder for EEG-based emotion recognition," Frontiers Human Neurosci., vol. 17, Apr. 2023, Art. no. 1169949.”

[0108] (3)Attention Conv-LSTM Network: Extract spatial features using a convolutional encoder. After allocating adaptive weights to different frequency bands using the attention mechanism, use a long short-term memory network to extract temporal features, and use the temporal attention mechanism to obtain key temporal information. For details, refer to "Y.Zhang, Y.Zhang, and S.Wang, "An attention-based hybrid deep learning model for EEG emotion recognition," Signal, Image, Video Process., vol. 17, no. 5, pp. 2305-2313, Jul. 2023.”

[0109] (4)DGCNN: Dynamically learn the relationships between different EEG channels during training using an adjacency matrix. For details, refer to "SONG T, ZHENG W, SONG P, et al. EEG Emotion Recognition Using Dynamical Graph Convolutional Neural Networks[J / OL]. IEEE Transactions on Affective Computing, 2020: 532-541.

[0110] http: / / dx.doi.org / 10.1109 / taffc.2018.2817622. DOI: 10.1109 / taffc.2018.2817622.”

[0111] (5)SparseDGCNN: Modify DGCNN by introducing a sparse constraint in graph G. For details, refer to "ZHANG G, YU M, LIU Y J, et al. SparseDGCNN: Recognizing Emotion From Multichannel EEG Signals[J / OL]. IEEE Transactions on Affective Computing, 2023: 537-548. http: / / dx.doi.org / 10.1109 / taffc.2021.3051332.

[0112] DOI: 10.1109 / taffc.2021.3051332.1”

[0113] (6) EEG-GCN: An adaptive brain network adjacency matrix was designed to quantify the connection strength between channels, and the significant sequential segments and spatial location information in EEG signals were captured adaptively. For specific reference, see "GAO Y, FU X, OUYANG T, et al. EEG-GCN: Spatio-Temporal and Self-Adaptive Graph Convolutional Networks for Single and Multi-View EEG-Based Emotion Recognition[J / OL]. IEEE Signal Processing Letters,

[0114] 2022: 1574 - 1578. http: / / dx.doi.org / 10.1109 / lsp.2022.3179946. DOI: 10.1109 / lsp.2022.3179946.”

[0115] (7) Siam-GCAN: The multi-head attention mechanism and siamese neural network were used to cluster the output of the GCN according to the Euclidean distance. For specific reference, see "Zeng H, Wu Q, **Y, et al. Siam-GCAN: A Siamese graph convolutional attention network for EEG emotion recognition[J]. IEEE Transactions on Instrumentation and Measurement, 2022, 71: 1 - 9.”

[0116] (8) Hybrid Network: A hybrid network based on dynamic graph convolution and temporal self-attention. Dynamic graph convolution was used to obtain spatial features, and self-attention was used to obtain temporal features. Then, the representative knowledge of spatial topology and temporal context was integrated. For specific reference, see "C. Cheng, Z. Yu, Y. Zhang and L. Feng, "Hybrid Network Using Dynamic Graph Convolution and Temporal Self-Attention for EEG-Based Emotion Recognition," in IEEE Transactions on Neural Networks and Learning Systems, doi: 10.1109 / TNNLS.2023.3319315.”

[0117] (9) FBSTCNet: A spatio-temporal convolutional network that integrates power and connectivity features to identify emotions by extracting power and / or connectivity features from EEG. For details, refer to "Huang W, Wang W, Li Y, et al. FBSTCNet: A Spatio-Temporal Convolutional Network Integrating Power and Connectivity Features for EEG-Based Emotion Decoding[J]. IEEE Transactions on Affective Computing, 2024."

[0118] (10) MS-FRAN: A multi-source domain adaptation method that is effective in aligning the distributions of each pair of source and target domains and can also reduce the distribution differences between multiple source domains. For details, refer to "W. Li, W. Huan, S. Shao, B. Hou and A. Song, "MS-FRAN: A Novel Multi-Source Domain Adaptation Method for EEG-Based Emotion Recognition," in IEEE Journal of Biomedical and Health Informatics, vol. 27, no. 11, pp.

[0119] 5302 - 5313, Nov. 2023, doi: 10.1109 / JBHI.2023.3311338."

[0120] 4) Validation on the DREAMER dataset

[0121] In this embodiment, the present invention conducts subject-dependent and subject-independent experiments on the DREAMER dataset and presents the classification accuracies of 23 subjects on three emotion labels (arousal, valence, and dominance), as shown in Table 2.

[0122] Table 2 Experiments on the DREAMER dataset

[0123]

[0124] In Table 2, subject represents the subject, subject_dependent represents the subject-dependent experiment, and subject_independent represents the subject-independent experiment.

[0125] The data in Table 2 show the specific accuracies of each subject under different emotion labels in the within-subject experiment, as well as the corresponding accuracies in the between-subject experiment. The average accuracies in the within-subject experiment are: arousal 88.56%, pleasure 87.94%, and dominance 88.23%. These results outline the intensity of emotional responses of each subject under the same conditions. In the between-subject experiment, there are significant differences in the accuracy of emotion label classification among subjects, and the average accuracy is lower than that in the within-subject experiment, which are arousal 72.31%, pleasure 74.99%, and dominance 73.23% respectively.

[0126] Further analysis shows that Subject 4 shows consistent high emotional responses in the between-subject experiment, with arousal, pleasure, and dominance being 82.42%, 85.35%, and 81.37% respectively. In contrast, Subject 13 has relatively low accuracies in pleasure and dominance, both being 65.52%, indicating that this subject has a relatively dull response in these emotional dimensions. In addition, there are also imbalances in the accuracies of different subjects in emotion labels. For example, Subject 19 has relatively high accuracies in arousal and dominance (84.38%, 76.66%), but a relatively low accuracy in pleasure, only 64.84%. This imbalance reflects the complex emotional responses of individuals to the experimental situation.

[0127] Overall, the average score of pleasure in the between-subject experiment is slightly higher than other emotional dimensions, which may indicate that in the experimental situation, most subjects are more inclined to experience pleasant emotional responses. This finding provides important insights into the emotional response patterns in the field of affective computing.

[0128] This embodiment also conducts a comparative analysis of the method proposed in the present invention with current advanced emotion recognition methods. The experimental results are shown in Table 3.

[0129] Table 3 Performance comparison of different methods on the DREAMER dataset

[0130]

[0131] In Table 3, Method represents the method.

[0132] Table 3 shows the comparison between the research method of the present invention and recent methods on the DREAMER dataset. The performance of each method in subject-dependent and subject-independent experiments is evaluated respectively, covering three dimensions of arousal, valence, and dominance. It can be seen from Table 3 that the method of the present invention has achieved the highest classification accuracy in all three emotional dimensions, significantly outperforming other comparison methods. At the same time, it can be seen that the accuracy of the model of the present invention is relatively close to that of the Hybrid Network. This is because this method also conducts the interaction of time and space, but the graph attention adopted by the present invention has a better interaction effect. In addition, it can be clearly observed that the difference between the method of the present invention and STGATE is the largest. This may be because this method only captures the time-frequency features and inputs them into the graph attention, without extracting the features in space. These methods adopt different strategies in the extraction of spatial and temporal features, resulting in differences in performance.

[0133] Generally speaking, the method proposed by the present invention shows superior classification ability on the DREAMER dataset, especially in complex cross-subject scenarios, further proving its effectiveness and robustness in the emotion recognition task. This advantage is mainly attributed to the extraction of spatial features by the dynamic graph constraint learning module proposed by the present invention and the effective fusion mechanism adopted for multiple domains. This mechanism can effectively capture the complex features in EEG signals, especially the information integration in the time domain and spatial domain, which plays a key role in emotion recognition.

[0134] 5) Verification on the FACED dataset

[0135] In this embodiment, the present invention conducts subject-dependent and subject-independent experiments on the FACED dataset and shows the emotion classification accuracy of 30 subjects, as shown in Table 4.

[0136] Table 4 Experimental accuracy on the FACED dataset

[0137]

[0138] Overall, the average accuracy of this dataset is relatively high, indicating that the model performs relatively stably in distinguishing three types of labels. However, the experimental results also reveal certain individual differences, reflecting that the different characteristic features of different subjects may affect the classification effect.

[0139] In the subject-dependent experiment, some subjects performed extremely well. For example, the accuracy rate of subject 26 reached 100%, indicating that the model has extremely strong adaptability to the data of these subjects. This may be because the data features of these subjects are relatively clear, with less noise or significant feature differences between different categories. At the same time, the accuracy rates of some subjects were relatively low. For example, the accuracy rate of subject 24 was only 66.62%, which implies that the model faces greater challenges when processing the data of these subjects, possibly due to noise or feature complexity in the data. Generally speaking, the accuracy rates of most subjects concentrated between 85% and 98%, indicating that the model has strong classification ability on the data of these subjects and its overall performance is relatively stable.

[0140] In the subject-independent experiment, the accuracy rates fluctuated greatly, reflecting the performance differences of different subjects in the emotion classification task. For example, the accuracy rates of subject 7 and subject 23 were 72.08% and 71.17% respectively, which were the highest among all subjects. This may mean that their physiological signal features are clearer, or their emotional expressions are more consistent, making it easier for the model to recognize. In contrast, the accuracy rate of subject 15 was only 48.35%, far lower than the average level, possibly due to greater data noise or a relatively unique emotional response pattern, resulting in the model having difficulty classifying accurately. Similarly, the accuracy rates of subject 8 and subject 2 were also relatively low, 50.19% and 51.87% respectively, which may reflect the large individual differences in their physiological signals during the experiment.

[0141] Table 5 shows the classification accuracy rates of different methods in the within-subject and independent-subject experiments on the FACED dataset. By comparing the performances of these methods, the generalization ability and feature extraction effect of the model in the emotion classification task can be better evaluated. As can be seen from the table, in the within-subject experiment, the accuracy rates of all methods were above 80%, indicating that these models showed strong classification ability when dealing with the within-subject classification task of the FACED dataset. It is worth noting that the method proposed in this invention achieved the highest accuracy rate of 92.29% in the subject-dependent experiment, showing its significant advantages in EEG signal feature extraction and cross-domain feature fusion. In contrast, the accuracy rate of the Hybrid Network was 91.56%. Although slightly lower than the method of this invention, it still performed excellently. This shows the importance of the interaction between time-domain and space-domain features in improving the model performance in emotion classification. The methods such as SparseDGCNN and EEG-GCN also performed relatively prominently, achieving accuracy rates of 85.62% and 87.38% respectively. This indicates that the graph structure-based feature extraction strategy has good adaptability to EEG signals, but there is still a slight gap compared with the method of this invention.

[0142] Table 5 Accuracy Rates of Experiments on the FACED Dataset under Different Methods

[0143]

[0144] In the subject-independent experiment, the method of the present invention also performed excellently, with an accuracy rate of 72.29%, higher than all comparative methods. This further proves the strong adaptability of the dynamic graph constraint learning module and its cross-domain feature fusion mechanism proposed by the present invention in dealing with individual differences. The Hybrid Network also performed well in the subject-independent experiment, with an accuracy rate of 69.81%, indicating that the interaction between its spatial features and temporal features contributes to the generalization ability of the model. In contrast, the performance of ATDD-LSTM and STGAT in the subject-independent experiment was significantly lower than that of other models, with accuracy rates of 50.96% and 51.34% respectively, indicating that a single temporal modeling or attention mechanism has a weak adaptability to the subject-independent situation and is difficult to effectively capture the feature differences between different individuals.

[0145] Generally speaking, the method of the present invention has shown significant advantages in both subject-dependent and subject-independent experiments, especially demonstrating strong generalization ability in complex subject-independent situations.

[0146] 6) Verification on the SEED-IV dataset

[0147] To further verify the generalization ability of the model, the present invention conducted a cross-session experiment on the SEED-IV dataset, and the experimental results are shown in Table 6.

[0148] Table 6 Cross-session experiment on the SEED-IV dataset

[0149]

[0150] From the experimental results, the average classification accuracy rate of the model in the cross-session setting is 72.29%, indicating that it has a certain generalization ability between different sessions. However, the accuracy rate differences between subjects are significant, with the lowest being 62.73% and the highest being 87.48%. This difference may reflect the changes in the emotional characteristics of different subjects between different sessions. The classification accuracy rate of Subject 15 is the highest, indicating that the emotional characteristics of this subject are more consistent between different sessions, enabling the model to perform better classification. At the same time, the accuracy rates of Subject 4 and Subject 7 are also significantly higher than the average level, further verifying that the model has good generalization ability on these data. However, the accuracy rates of some subjects are relatively low, probably because the emotional characteristics of these subjects fluctuate greatly between different sessions, making it difficult for the model to effectively capture their features.

[0151] In addition, the present invention compared the experimental results of the method proposed by the present invention with those of other advanced methods in the cross-session experiment on the SEED-IV dataset, and the specific results are shown in Table 7.

[0152] Cross-session comparison experiments with other methods on the SEED-IV dataset in Table 7

[0153]

[0154] The comparison results show that the method proposed in this invention performs best among all the comparison methods, with an accuracy rate of 72.29%. This result demonstrates the superiority of the method of this invention in cross-session classification tasks. In contrast, when existing methods extract spatial or temporal features of EEG signals, they often neglect to comprehensively fuse these features, resulting in unsatisfactory feature extraction effects. The method of this invention not only effectively captures the complex spatial features of EEG signals, but also improves the generalization ability of the model through a reasonable fusion mechanism, thus performing excellently in cross-session emotion classification tasks.

[0155] Those of ordinary skill in the art will realize that the embodiments described herein are for the purpose of assisting the reader in understanding the principles of this invention, and it should be understood that the protection scope of this invention is not limited to such specific statements and embodiments. For those skilled in the art, various changes and modifications can be made to this invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this invention shall be included within the scope of the claims of this invention.

Claims

1. A multi-domain constrained emotion recognition system inspired by adaptive cognitive graphs, characterized by: include: Differential entropy extraction module, dynamic graph constraint learning module, sequence attention module, and multi-domain interactive attention module; The differential entropy extraction module is used to extract the differential entropy features of EEG signals; The dynamic graph constraint learning module includes: a brain topology structure construction unit, a driving structure generation unit, a calibration unit, and a first feature extraction unit; the brain topology structure construction unit adaptively generates a brain topology structure based on differential entropy features and learnable parameters, sets a screening parameter b, and screens the adaptively generated brain topology structure to obtain a screened brain topology structure f adj ; The driving structure generation unit is used to generate the brain topology structure f adj Random drive structures of the same size adj ; Calibration unit for random drive structure s adj and brain topology adj Perform calibration, the calibration expression is: Among them, f′ adj is the attention-based brain topology matrix obtained after calibration, s′ adj is the attention-based driving structure matrix obtained after calibration; The first feature extraction unit extracts the spatial domain features of the EEG signal based on the calibrated brain topology and driving structure; Also included: Optimizing the brain topology based on the following formula: L C =||X S ·X S T -f′ adj ·A adj +(1-A adj )·s adj || Among them, L C is a constraint, X S is the graph-level representation of the features of all nodes, T represents transposition, A adj is the adaptively generated brain topology, a sdj is the driving structure, ‖·‖ represents the Frobenius norm; The sequence attention module includes a position encoding unit and a second feature extraction unit; the position encoding unit is used to add position information to the differential entropy feature, and the second feature extraction unit extracts the time domain features of the EEG signal based on the differential entropy after adding the position information; The multi-domain interactive attention module obtains emotion recognition results based on the spatial domain features of EEG signals and the temporal domain features of EEG signals.

2. The multi-domain constrained emotion recognition system inspired by adaptive cognitive graph according to claim 1, characterized in that: The first feature extraction unit is implemented using a graph neural network.

3. The multi-domain constrained emotion recognition system inspired by adaptive cognitive graph according to claim 2, characterized in that: The second feature extraction unit includes a self-attention mechanism and a feedforward neural network; the self-attention mechanism is used to capture the dependencies between input sequences, and the feedforward neural network performs a nonlinear transformation on the dependencies obtained by the self-attention mechanism; finally, based on the dependencies obtained by the self-attention mechanism and the nonlinear transformation results of the feedforward neural network, the time domain characteristics of the EEG signal are obtained.

4. The multi-domain constrained emotion recognition system inspired by adaptive cognitive graph according to claim 3, characterized in that: The multi-domain interactive attention module fuses the spatial domain features of the EEG signal with the temporal domain features of the EEG signal through a cross-attention mechanism to obtain fused features.

5. A multi-domain constrained emotion recognition method inspired by adaptive cognitive graphs, characterized by: include: S1. Extract the differential entropy features of the EEG signal and adaptively generate the brain topology structure based on the extracted differential entropy features and learnable parameters; Set the screening parameter b to screen the adaptively generated brain topology structure, thereby obtaining the screened brain topology structure f adj ; S2, randomly generate a brain topology f constructed in step S1 adj Drive structures of equal size adj ; f adj 、s adj Perform calibration; the calibration expression is: Among them, f′ adj is the attention-based brain topology matrix obtained after calibration, s′ adj is the attention-based driving structure matrix obtained after calibration; Also included: Optimizing the brain topology based on the following formula: L C =||X S ·X S T -f′ adj ·A adj +(1-A adj )·s adj || Among them, L C is a constraint, X S is the graph-level representation of the features of all nodes, T represents transposition, A adj For the adaptively generated brain topology, s adj is the driving structure, ‖·‖ represents the Frobenius norm; S3, based on f′ adj and s′ adj , obtain the spatial domain features of the EEG signal; S4, using position encoding to add position information to the extracted differential entropy features; S5, the result obtained in step S4 is passed through the self-attention mechanism and the feedforward network to obtain the time domain features of the EEG signal; S6. The temporal and spatial domain features of EEG signals are fused through a multi-domain interactive attention module based on graph attention and cross-attention. Specifically, the temporal and spatial domain features of EEG signals are fused through the cross-attention mechanism to achieve cross-domain information integration and strengthen the expression of key features. Secondly, after obtaining the fused features, the model generates a cross-domain adjacency matrix for the graph attention network. This adjacency matrix contains the correlation information between the temporal and spatial domains and can capture the complex interactions between different brain regions. Next, multiple heads are introduced into the graph attention, that is, the output of each attention head is calculated and then these outputs are spliced together. S7. Obtain an emotion recognition result based on the fusion result obtained in step S6.

6. The multi-domain constrained emotion recognition method inspired by adaptive cognitive graph according to claim 5, characterized in that: Step S1 adaptively generates a brain topology structure by introducing learnable parameters; the adaptively generated brain topology structure is expressed as: Where W A Q is the learnable parameter of the query in self-attention, W A K is the learnable parameter of the key in self-attention, is the scaling factor used to prevent the value from being too large; X represents the EEG signal.

Images