An electroencephalogram emotion recognition method and system based on a deep neural network

Abstract

The application relates to the technical field of electroencephalogram signal processing, and discloses an electroencephalogram emotion recognition method and system based on a deep neural network. The method comprises the following steps: collecting multi-channel EEG signals and preprocessing; constructing a graph data structure, taking the electroencephalogram channel as a node, extracting multi-domain features, and constructing an adaptive dynamic adjacency matrix; constructing a graph convolution long short-term memory network, learning spatial features by GNN and time dependence by LSTM; using a multi-scale time-frequency feature fusion method, combining STFT and CWT to enhance emotion capture; combining brain network features to construct an FCN to obtain global topological information of brain regions; and outputting emotion indexes such as alertness by a classification model. The application combines graph structure learning and time sequence modeling, optimizes EEG signal emotion recognition, and improves individual adaptation and emotion recognition accuracy.

Description

Technical Field

[0001] This invention relates to the field of electroencephalogram (EEG) signal processing technology, and in particular to an EEG emotion recognition method and system based on deep neural networks. Background Technology

[0002] In recent years, brain-computer interface technology has received widespread attention in the field of EEG emotion recognition, and related research has gradually deepened into the acquisition and processing of EEG signals. EEG signal acquisition is the first step in emotion recognition, typically achieved by placing electrodes on the scalp to monitor the brain's electrical activity. However, because EEG signals are inherently weak, the acquired signals often require amplification. Furthermore, these signals are susceptible to noise from electrooculography (EOG), electrocardiography (ECG), electromagnetic interference, and electromyography (EMG), which can mask the target signal and reduce the accuracy of emotion recognition. Therefore, signal preprocessing becomes a crucial step.

[0003] Early preprocessing of EEG signals typically relied on manual observation and interference removal, a method that was not only time-consuming but also had limited effectiveness for weak signals. With ongoing research, automated noise removal algorithms have gradually developed, among which filtering and independent component analysis (ICA) are widely used, effectively purifying EEG signals and thus improving the accuracy and stability of EEG data processing.

[0004] Besides signal processing, feature extraction and model building are also crucial components of emotion recognition systems. In recent years, with the rapid development of deep learning technology, researchers have gradually integrated feature extraction and model building into an end-to-end process. By training deep learning models, the most representative features can be automatically extracted from the raw signals, resulting in more accurate emotion recognition. This end-to-end integration method not only improves the accuracy of recognition but also reduces human intervention, opening up new prospects for the application of EEG interface technology in the field of emotion recognition.

[0005] Temporal features are an intuitive and important type of feature in EEG signal processing, especially in the study of time-related phase (ERP), where the P1 and N170 components have received widespread attention for their application in emotion recognition. Although temporal features can show how signals change over time, they have certain limitations in extracting higher-order features of EEG signals, mainly because they cannot effectively capture frequency information.

[0006] To fully extract information from EEG signals, researchers have begun exploring methods for extracting frequency domain features. For example, Li and his team used Short-Time Fourier Transform (STFT) to convert EEG signals into a frequency domain representation and extracted the power spectral density (PSD) of each frequency band as a feature, achieving an average recognition rate of 82.29%. Furthermore, Xiang and his team constructed a feature matrix using differential entropy, achieving an average recognition accuracy of 90.45% and a wake-up recognition accuracy of 90.60%.

[0007] Deep learning algorithms have demonstrated powerful capabilities in emotion classification tasks, particularly in handling large-scale datasets and automatically extracting complex features. For example, Maheshwari et al. constructed an EEG emotion classification model with eight one-dimensional convolutional layers, achieving 98.99% accuracy on the DEAP dataset. Zhu et al. proposed a model containing five layers of two-dimensional CNN and GRU to extract spatial and temporal features, achieving arousal and valence recognition rates of 87.86% and 88.69%, respectively.

[0008] Besides convolutional neural networks (CNNs), other deep learning models have also achieved good classification results in the field of EEG emotion recognition. For example, Zheng et al. used deep belief networks (DBNs) to achieve an emotion classification accuracy of 86.08% on the SEED dataset.

[0009] Because there are numerous EEG channels, not all of them are crucial for emotion recognition, and redundant information can negatively impact algorithm performance. Therefore, selecting channels closely related to emotion recognition has become an important task in emotion recognition research. For example, Tong et al. proposed a superior combination of EEG channels; Wang et al. proposed the most suitable channel combinations based on the needs of emotion valence and arousal classification; and Xiang et al. proposed five EEG channels suitable for emotion recognition.

[0010] Based on the above analysis, the shortcomings of the existing technology are as follows:

[0011] EEG signals are susceptible to interference. The acquisition of EEG signals is affected by environmental factors, and relatively pure data in a laboratory environment may not fully reflect signals in the real world.

[0012] Individual differences. Each person's brain structure and function are different, which means that even when faced with the same stimuli, different individuals may exhibit different brain electrical signals. This inter-individual variability requires analysis and learning through large amounts of data to establish a broader correspondence between emotions and brain electrical signals.

[0013] Research on brain channel correlations is insufficient. Complex interactions exist between different brain regions, which manifest as channel correlations in electroencephalogram (EEG) signals. However, current research rarely considers these channel correlations in deep learning models. A deeper understanding of brain channel correlations is crucial for building more accurate EEG signal models. Further research and exploration are needed in this area. Summary of the Invention

[0014] The purpose of this invention is to provide a brainwave emotion recognition method and system based on deep neural networks. By combining the spatial, temporal, and time-frequency features of brainwave signals, a graph convolutional long short-term memory network and a multi-scale time-frequency feature fusion mechanism are constructed to achieve more accurate emotion recognition.

[0015] To achieve the above objectives, the following technical solution is adopted:

[0016] In a first aspect, the present invention provides a brainwave emotion recognition method based on a deep neural network, the method comprising:

[0017] An adaptive dynamic adjacency matrix is ​​constructed by treating all EEG channels as nodes; the EEG signals of time series, the statistical characteristics of EEG signals, and / or the frequency domain characteristics of EEG signals are used as node features.

[0018] The nodes, the adaptive dynamic adjacency matrix, and the node features are constructed into graph features;

[0019] A GNN-LSTM-MSTF model is constructed, which includes a graph convolutional network, a long short-term memory network, and a multi-scale time-frequency feature fusion network. The GNN-LSTM-MSTF model responds to the input graph features, uses a graph convolutional network to extract spatial features of the graph features, combines a long short-term memory network to extract temporal features, and combines multi-scale time-frequency feature fusion to enhance the modeling of instantaneous emotional changes, thereby obtaining the temporal encoding representation and time-frequency fusion features of the entire sequence.

[0020] Functional connectivity networks are used to extract topological information between brain regions as brain network features;

[0021] The brain network features are concatenated with the output of the GNN-LSTM-MSTF model to form the final feature vector.

[0022] Based on the final feature vector, a multilayer perceptron classification model is used to output the emotion recognition result, which is one or a combination of alertness, arousal, dominance, and liking.

[0023] Furthermore, the GNN-LSTM-MSTF model responds to the input graph features by employing a graph convolutional network to extract spatial features of the graph features, combining a long short-term memory network to extract temporal features, and combining multi-scale time-frequency feature fusion to enhance the modeling of instantaneous emotional changes. The methods for obtaining the temporal encoding representation and time-frequency fusion features of the entire sequence include:

[0024] Graph convolution operations are used to embed the constructed graph features into the graph, capturing the spatial dependencies between different EEG channels. Let the graph features be... ,in, Representing node characteristics, Let N be the set of real numbers, N be the number of channels, and V be the set of nodes. Let be the set of edges. F The feature dimension is and the adjacency matrix is ​​. The graph convolution calculation is shown in equation (1):

[0025]

[0026] in, It is the first The node feature matrix input to the layer has a dimension of . , It is the first The node representation matrix is ​​output by the layer. It is the first The learnable weight matrix of the layer, To add the adjacency matrix after adding self-loops, yes The degree matrix, Initially, node features, For the first Layer learnable weights For activation functions;

[0027] The embedding vector of each node output by graph convolution is used as a temporal input to capture the dynamic dependence of EEG signals between different time windows; for the time input sequence The recursive form of LSTM is:

[0028]

[0029]

[0030]

[0031]

[0032]

[0033] in, The time-coded representation of the entire sequence; Indicates the input gate at time step activation value, This represents the input weight matrix of the input gate. This indicates the hidden state at the current time step. This indicates the hidden state at the previous moment. This represents the bias term of the input gate. This represents the input weight matrix of the forget gate. The hidden state weight matrix represents the forget gate. The bias term representing the forget gate. Indicates the forget gate at time step activation value, Indicates the output gate at time step activation value, This represents the hidden state weight matrix of the output gate. This represents the bias term of the output gate. This indicates the cell state at the current time step. This indicates the cell state at the previous time step. The input weight matrix represents the candidate matrix. The hidden state weight matrix represents the candidate state. Bias terms representing candidate states;

[0034] Use STFT to obtain time-frequency spectrum Capture fixed-frequency domain information through a sliding window:

[0035]

[0036] in, Indicates at time ,frequency The STFT transformation results on the surface, This indicates the original time signal at time point. The value of , The window function is centered at... , Denotes the base of the natural logarithm. Represents the imaginary unit. Represents frequency variables. Represents the integral variable, and represents a point on the entire time axis. Indicates the current time position in the analysis;

[0037] Extracting short-time dynamic features using continuous wavelet transform:

[0038]

[0039] In the formula, The result of the wavelet transform is represented by the scale. and time Local features of the signal below, Indicates the scale parameter. Indicates time and location. Indicates the time of the input signal The possible values ​​of ;

[0040] Will and After convolutional encoding, and Fusion yields time-frequency fusion features. .

[0041] Furthermore, methods for extracting topological information of brain regions as brain network features using functional connectivity networks include:

[0042] A functional connectivity network is constructed, and global topological features are extracted from graph features as brain network features using the following formula. These global topological features include one of centrality, clustering coefficient, and path length, where the formula for calculating centrality is:

[0043]

[0044] in, express s arrive t The number of shortest paths, Indicates the process v The number of paths, s , t , v This represents a node in the graph.

[0045] Furthermore, the brain network features are concatenated with the output of the GNN-LSTM-MSTF model to form the final feature vector. :

[0046]

[0047] In the formula, Indicates time-coded features. This represents the encoding result after multi-scale fusion. Indicates brain network characteristics;

[0048] The fused feature Z is input into a multilayer perceptron to obtain the output emotion recognition result:

[0049]

[0050] In the formula, This represents the emotion recognition result vector predicted by the model. This represents a multilayer perceptron.

[0051] Furthermore, the adaptive dynamic adjacency matrix is ​​constructed as follows:

[0052] Calculate the spatial distance and signal similarity between EEG channels;

[0053] For any two EEG channels and Calculate Euclidean distance using three-dimensional coordinates:

[0054]

[0055] Forming a spatial matrix ,in, , , It is the first The coordinates of each channel in three-dimensional space, where N is the number of channels. For two EEG channels and Euclidean distance;

[0056] The Pearson correlation coefficient is used to measure the signal correlation between any two channels. Form a similarity matrix :

[0057]

[0058] In the formula, Indicates channel and channels Covariance between Indicates channel standard deviation Indicates channel Standard deviation;

[0059] Initial edge weights are defined based on spatial distance and signal similarity. for:

[0060]

[0061] in, These are weighting coefficients. To prevent division by zero of small constants;

[0062] Adaptive weighting is used to dynamically adjust the edge weights of the adjacency matrix: making the adjacency matrix a learnable parameter matrix. During model training, updates are performed using gradient descent:

[0063]

[0064] in, For learning rate, For the overall identification loss function, This is the descent gradient.

[0065] Thus, the generated Adaptive Dynamic Adjacency Matrix (ADAM) can flexibly adjust the graph structure according to the spatial topology and dynamic temporal characteristics of EEG signals, avoiding information loss caused by static adjacency. This improves the modeling accuracy of graph neural networks for dependencies between EEG channels, thereby enhancing the accuracy and generalization ability of emotion classification.

[0066] Furthermore, the multi-scale time-frequency feature fusion method includes:

[0067] Calculate the short-time Fourier transform to extract the steady-state frequency domain features of the EEG signal;

[0068] Calculate continuous wavelet transform to extract short-time dynamic features of EEG signals;

[0069] By combining the above features through a feature fusion mechanism, the ability to identify emotion patterns at different time scales can be enhanced.

[0070] Furthermore, the nodes, adjacency matrix, and node features are constructed into graph features, including:

[0071] The EEG channels are viewed as nodes in a graph, with each node corresponding to one EEG channel. Node features include: time-domain features, frequency-domain features, and wavelet transform features.

[0072] An adaptive dynamic adjacency matrix is ​​adopted, and the edge weights of the adjacency matrix are dynamically adjusted based on adaptive weighting.

[0073] Optimize the adjacency matrix by combining physical distance and relevance information;

[0074] By integrating the adjacency matrix, node features, and brain network features, graph structure data is obtained.

[0075] Furthermore, the graph convolutional network and the long short-term memory network are combined to form a graph convolutional long short-term memory network, which includes a graph convolutional layer, a long short-term memory network, and a multilayer perceptron. The graph convolutional layer is used to extract brain network features from the graph features, the long short-term memory network is used to extract time-frequency features, and the multilayer perceptron is used to fuse the brain network features and the time-frequency features and output the emotion recognition result.

[0076] Furthermore, the graph convolutional layer is configured as three layers, each of which includes a convolutional layer, a linear activation function layer, and a random deactivation layer connected in sequence. The output of the three graph convolutional layers is aggregated through a global pooling layer and then input into the long short-term memory network.

[0077] Secondly, the present invention provides a brainwave emotion recognition system based on a deep neural network, the system comprising:

[0078] The node feature determination module is configured to use all EEG channels as nodes to construct an adaptive dynamic adjacency matrix; and to use time-series EEG signals, statistical features of EEG signals, and / or frequency domain features of EEG signals as node features.

[0079] The graph feature construction module is configured to construct graph features from the nodes, the adaptive dynamic adjacency matrix, and the node features;

[0080] The graph feature processing module is configured to construct a GNN-LSTM-MSTF model, which includes a graph convolutional network, a long short-term memory network, and a multi-scale time-frequency feature fusion network. In response to the input graph features, the GNN-LSTM-MSTF model uses a graph convolutional network to extract spatial features of the graph features, combines a long short-term memory network to extract temporal features, and combines multi-scale time-frequency feature fusion to enhance the modeling of instantaneous emotional changes, thereby obtaining the temporal encoding representation and time-frequency fusion features of the entire sequence.

[0081] The brain network feature extraction module is configured to use functional connectivity networks to extract topological information of brain regions as brain network features.

[0082] The feature concatenation module is configured to concatenate brain network features with the output of the GNN-LSTM-MSTF model to form the final feature vector. ;

[0083] The classification output module is configured to output emotion recognition results based on the final feature vector using a multilayer perceptron classification model. The emotion recognition results are one or a combination of alertness, arousal, dominance, and liking.

[0084] The beneficial effects of this invention are:

[0085] This invention constructs an EEG emotion recognition method based on graph neural networks by combining spatial, temporal, and frequency domain features. Adaptive Adjacency Matrix (ADMA) is used to optimize channel connectivity, and a Functional Connectivity Network (FCN) is incorporated to improve the model's generalization ability. In experiments, the DEAP dataset was selected, and the data was divided into training, validation, and test sets in a 6:1:1 ratio. After feature fusion, combining frequency and spatial features, a convolutional neural network was used for training, ultimately achieving an accuracy of 93%, demonstrating a satisfactory emotion recognition performance. Attached Figure Description

[0086] Figure 1A structural diagram of an EEG emotion recognition framework according to an embodiment of the present invention is shown.

[0087] Figure 2 A flowchart of a brainwave emotion recognition method based on a deep neural network according to an embodiment of the present invention is shown.

[0088] Figure 3 A flowchart illustrating emotion recognition based on a graph convolutional long short-term memory network according to an embodiment of the present invention is shown.

[0089] Figure 4 A schematic diagram of the alertness dimension according to an embodiment of the present invention is shown; wherein, (a) is an accuracy image generated when the point feature is a time-series EEG signal and the edge feature is distance; and (b) is an accuracy image generated when the point feature is a time-series EEG signal and the edge feature is Pearson coefficient.

[0090] Figure 5 A schematic diagram of the arousal dimension according to an embodiment of the present invention is shown; wherein, (a) is an accuracy image generated when the point feature is a time-series EEG signal and the edge feature is distance; and (b) is an accuracy image generated when the point feature is a time-series EEG signal and the edge feature is Pearson coefficient.

[0091] Figure 6 A schematic diagram of the degree of dominance dimension according to an embodiment of the present invention is shown; wherein, (a) is an accuracy image generated when the point feature is a time-series EEG signal and the edge feature is distance; and (b) is an accuracy image generated when the point feature is a time-series EEG signal and the edge feature is Pearson coefficient.

[0092] Figure 7 A schematic diagram of the liking dimension according to an embodiment of the present invention is shown; wherein, (a) is an accuracy image generated when the point feature is a time-series EEG signal and the edge feature is distance; and (b) is an accuracy image generated when the point feature is a time-series EEG signal and the edge feature is Pearson coefficient.

[0093] Figure 8 A schematic diagram of the degree of domination loss function according to an embodiment of the present invention is shown.

[0094] Figure 9 A structural diagram of an EEG emotion recognition system based on a deep neural network according to an embodiment of the present invention is shown. Detailed Implementation

[0095] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0096] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0097] Emotions play an increasingly important role in modern people's daily lives, especially in the field of human-computer interaction, where emotion recognition is becoming increasingly important. Electroencephalography (EEG), as an effective means, can implicitly monitor users' emotional and perceptual states in multimedia interactions. Due to its simplicity and high stability, EEG signal acquisition is widely used in emotion recognition. Therefore, this invention provides an EEG-based emotion recognition method based on deep neural networks.

[0098] This invention provides an EEG emotion recognition framework, the structure of which is as follows: Figure 1 As shown in the diagram. In this EEG emotion recognition framework, EEG signals are first acquired. Acquisition methods include, but are not limited to, using existing EEG signal acquisition systems, such as TDT, Neuralynx, Blackrock Microsystems, Plexon, and Open Ephys. The acquired EEG signals are then preprocessed to generate feature maps. A graph convolutional long short-term memory network is then used for feature extraction and classification. The extracted features include brain network features and time-frequency features. Finally, a prediction score is used to output the prediction result. The brain network features include the connection matrix, brain network structure, and network attributes. The time-frequency features include time-domain and frequency-domain features. The classification results include, but are not limited to, the following: Figure 1 It displays two emotions: happiness and sadness.

[0099] This deep neural network-based EEG emotion recognition method can, for example, Figure 1 The EEG emotion recognition framework shown achieves emotion recognition. Specifically, as... Figure 2 As shown, the brainwave emotion recognition method based on deep neural networks includes steps S100 to S500, which are described in detail below.

[0100] S100. Construct an adaptive dynamic adjacency matrix by treating all EEG channels as nodes; use time-series EEG signals, statistical characteristics of EEG signals, and / or frequency domain characteristics of EEG signals as node features.

[0101] In some embodiments, an adaptive dynamic adjacency matrix is ​​constructed as follows:

[0102] Calculate the spatial distance and signal similarity between EEG channels;

[0103] For any two EEG channels and Calculate Euclidean distance using three-dimensional coordinates:

[0104]

[0105] Forming a spatial matrix ,in, , , It is the first The coordinates of each channel in three-dimensional space, where N is the number of channels. For two EEG channels and Euclidean distance;

[0106] The Pearson correlation coefficient is used to measure the signal correlation between any two channels. Form a similarity matrix :

[0107]

[0108] In the formula, Indicates channel and channels Covariance between Indicates channel standard deviation Indicates channel Standard deviation;

[0109] Initial edge weights are defined based on spatial distance and signal similarity. for:

[0110]

[0111] in, These are weighting coefficients. To prevent division by zero of small constants;

[0112] Adaptive weighting is used to dynamically adjust the edge weights of the adjacency matrix: making the adjacency matrix a learnable parameter matrix. During model training, updates are performed using gradient descent:

[0113]

[0114] in, For learning rate, For the overall identification loss function, This is the descent gradient.

[0115] Thus, the generated Adaptive Dynamic Adjacency Matrix (ADAM) can flexibly adjust the graph structure according to the spatial topology and dynamic temporal characteristics of EEG signals, avoiding information loss caused by static adjacency. This improves the modeling accuracy of graph neural networks for dependencies between EEG channels, thereby enhancing the accuracy and generalization ability of emotion classification.

[0116] S200, The nodes, the adaptive dynamic adjacency matrix, and the node features are constructed into graph features.

[0117] S300. Construct a GNN-LSTM-MSTF model, which includes a graph convolutional network, a long short-term memory network, and a multi-scale time-frequency feature fusion network. The GNN-LSTM-MSTF model responds to the input graph features by using a graph convolutional network to extract spatial features of the graph features, combining a long short-term memory network to extract temporal features, and combining multi-scale time-frequency feature fusion to enhance the modeling of instantaneous emotional changes, thereby obtaining the temporal encoding representation and time-frequency fusion features of the entire sequence.

[0118] In some embodiments, the GNN-LSTM-MSTF model responds to the input graph features by employing a graph convolutional network to extract spatial features of the graph features, combining a long short-term memory network to extract temporal features, and combining multi-scale time-frequency feature fusion to enhance the modeling of instantaneous emotional changes. The methods for obtaining the temporal encoding representation and time-frequency fusion features of the entire sequence include:

[0119] Graph convolution operations are used to embed the constructed graph features into the graph, capturing the spatial dependencies between different EEG channels. Let the graph features be... ,in, Representing node characteristics, Let N be the set of real numbers, N be the number of channels, and V be the set of nodes. Let be the set of edges. F The feature dimension is and the adjacency matrix is ​​. The graph convolution calculation is shown in equation (1):

[0120]

[0121] in, It is the first The node feature matrix input to the layer has a dimension of , It is the first The layer outputs a matrix of node representations. It is the first The learnable weight matrix of the layer, To add the adjacency matrix after adding self-loops, yes The degree matrix, Initially, node features, For the first Layer learnable weights For activation functions;

[0122] The embedding vector of each node output by graph convolution is used as a temporal input to capture the dynamic dependence of EEG signals between different time windows; for the time input sequence The recursive form of LSTM is:

[0123]

[0124]

[0125]

[0126]

[0127]

[0128] in, The time-coded representation of the entire sequence; Indicates the input gate at time step activation value, This represents the input weight matrix of the input gate. This indicates the hidden state at the current time step. This indicates the hidden state at the previous moment. This represents the bias term of the input gate. This represents the input weight matrix of the forget gate. This represents the hidden state weight matrix of the forget gate. The bias term representing the forget gate. Indicates the forget gate at time step activation value, Indicates the output gate at time step activation value, This represents the hidden state weight matrix of the output gate. This represents the bias term of the output gate. This indicates the cell state at the current time step. This indicates the cell state at the previous time step. The input weight matrix represents the candidate matrix. The hidden state weight matrix represents the candidate state. Bias terms representing candidate states;

[0129] Use STFT to obtain time-frequency spectrum Capture fixed-frequency domain information through a sliding window:

[0130]

[0131] in, Indicates at time ,frequency The STFT transformation results on the surface, This indicates the original time signal at time point. The value of , The window function is centered at... , Denotes the base of the natural logarithm. Represents the imaginary unit. Represents frequency variables. Represents the integral variable, and represents a point on the entire time axis. Indicates the current time position of the analysis.

[0132] Extracting short-time dynamic features using continuous wavelet transform:

[0133]

[0134] In the formula, Indicates at time ,frequency The STFT transformation results on the surface, This indicates the original time signal at time point. The value of , The window function is centered at... , Denotes the base of the natural logarithm. Represents the imaginary unit. Represents frequency variables. Represents the integral variable, and represents a point on the entire time axis. Indicates the current time position in the analysis;

[0135] Will and After convolutional encoding, and Fusion yields time-frequency fusion features. .

[0136] like Figure 3The diagram shows a flowchart of emotion recognition based on a graph convolutional long short-term memory (LSTM) network. The LSTM network is the GNN-LSTM module in the GNN-LSTM-MSTF model. During emotion recognition, the input is a signal acquired by an EEG signal acquisition system. This signal is processed using existing algorithms to obtain EEG features, which are then used as the EEG signal. Steps S100-S300 above preprocess the EEG signal to obtain an EEG image. Finally, the LSTM network responds to the input EEG image by processing it through graph convolutional layers, a LSTM network, and a multilayer perceptron (MLP) to obtain the final output. The graph convolutional layers consist of three layers, each comprising a convolutional layer, a linear activation function layer, and a random deactivation layer connected in sequence. The outputs of the three graph convolutional layers are aggregated through a global pooling layer before being input into the LSTM network. Figure 3 As shown, the convolutional layer, linear activation function layer, and random deactivation layer of the first graph convolutional layer are represented as GConv1, Relu1, and Dropout1, respectively; the convolutional layer, linear activation function layer, and random deactivation layer of the second graph convolutional layer are represented as GConv2, Relu2, and Dropout2, respectively; and the convolutional layer, linear activation function layer, and random deactivation layer of the third graph convolutional layer are represented as GConv3, Relu3, and Dropout3, respectively.

[0137] In the experimental setup, the batch size was set to 128, the learning rate to 0.001, ReLU was used as the activation function, MSELoss as the loss function, and Adam as the optimizer. During data processing, the raw data was converted into a graph format, and time-domain and frequency-domain features were extracted.

[0138] To handle these two different features, two graph convolutional long short-term memory networks (GCNs) can be used. Each GCN consists of three graph convolutional layers (GCN layers), one long short-term memory network (LSTM) layer, and one multilayer perceptron (MLP) layer. The graph convolutional layers process the graph structure data, the LSTM layer captures the dynamic characteristics of the time series data, and the MLP layer generates the final prediction result (emotion recognition result). Specifically, the graph convolutional layers extract structural information from the graph, the LSTM layer processes the time series data, and the MLP layer combines the outputs of the graph convolutional layers and the LSTM layer to generate the final prediction result. This multi-layered model structure enables the model to effectively handle complex data features and improves its performance in tasks such as sentiment analysis.

[0139] When building the model, the number of input channels of the first layer of graph convolution was set to 8064 and the number of output channels was set to 4032. A ReLU activation function was added to increase the non-linearity of the model. A dropout layer was added with the dropout probability set to 0.2, randomly deactivating 20% ​​of neurons to reduce the risk of overfitting.

[0140] After this processing layer, the 32×8064 data is transformed into 32×4032 data. Then, we use a global pooling layer to aggregate the features of the entire graph for graph classification. After global pooling, the data changes from 32×512 to 1×512. Subsequently, we introduce an LSTM layer with parameters set to 2, 3, 2, meaning the LSTM layer contains two hidden states, each with three units. After processing by the LSTM layer, the data is transformed into 1×1536. Finally, an MLP is used to generate 1×1 predictions.

[0141] S400 employs a functional connectivity network to extract topological information of brain regions as brain network features.

[0142] In some embodiments, the method of using functional connectivity networks to extract topological information of brain regions as brain network features includes:

[0143] A functional connectivity network is constructed, and global topological features are extracted from graph features as brain network features using the following formula. These global topological features include one of centrality, clustering coefficient, and path length, where the formula for calculating centrality is:

[0144]

[0145] in, express s arrive t The number of shortest paths, Indicates the process v The number of paths, s , t , v This represents a node in the graph.

[0146] S500: Concatenate the brain network features with the output of the GNN-LSTM-MSTF model to form the final feature vector. .

[0147] In some embodiments, brain network features are concatenated with the output of the GNN-LSTM-MSTF model to form the final feature vector. :

[0148]

[0149] In the formula, Indicates time-coded features. This represents the encoding result after multi-scale fusion. This indicates characteristics of brain networks.

[0150] S600. Based on the final feature vector, a multilayer perceptron classification model is used to output the emotion recognition result, which is one or a combination of alertness, arousal, dominance, and liking.

[0151] In some embodiments, the fused feature Z is input into a multilayer perceptron to obtain the output emotion recognition result:

[0152]

[0153] In the formula, This represents the emotion recognition result vector predicted by the model. This represents a multilayer perceptron.

[0154] In some embodiments, the nodes, adjacency matrix, and node features are constructed as graph features by: treating EEG channels as nodes of a graph, with each node corresponding to one EEG channel; node features including: time-domain features, frequency-domain features, and wavelet transform features; employing an adaptive dynamic adjacency matrix, dynamically adjusting the edge weights of the adjacency matrix based on adaptive weighting; optimizing the adjacency matrix by combining physical distance and correlation information; and integrating the adjacency matrix, node features, and brain network features to obtain graph structure data.

[0155] The feasibility and progressiveness of this invention will be fully illustrated below with a specific implementation case.

[0156] In this implementation, the dataset is randomly divided into training, validation, and test sets in a 6:1:1 ratio. This partitioning helps ensure the model's generalization ability because the test set, being an independent dataset not used in training, allows for a more accurate evaluation of the model's true performance. We use the `train_test_split()` function from the sklearn library for simple cross-validation. This function can split the dataset into training, validation, and test sets in a specific ratio to evaluate the performance and generalization ability of the machine learning model. By splitting the data into independent training and test sets, we can effectively evaluate the model's performance on unseen data, thereby better understanding the model's generalization ability and stability, and improving the model's reliability and practicality. For example, the algorithm's performance on the training and validation sets is as follows: Figure 8 As shown.

[0157] This embodiment uses accuracy as the evaluation metric. The model output is the predicted score. For accuracy, this embodiment sets a threshold; if the difference between the predicted score and the true value is not greater than this threshold, the prediction is considered valid.

[0158] In the experiment, this embodiment used a total of two edge features and three point features, resulting in a total of six combinations. Each combination was trained separately to obtain four corresponding models. Figures 4 to 7 This demonstrates the case where time-series EEG signals are used as point features and Pearson correlation coefficients are used as edge features.

[0159] When using EEG signals as point features and Pearson correlation coefficient as edge features to train the model, as shown in the attached figure... Figures 4 to 7 As shown, the accuracy rate was 75% in the alertness dimension, 83% in the arousal dimension, 78% in the dominance dimension, and 72% in the liking dimension.

[0160] This shows that the model proposed in this embodiment performs best in the arousal dimension under this combination, while the accuracy in the other three dimensions is not high. This indicates that this feature combination has high applicability in arousal prediction, but its performance in other dimensions is relatively average.

[0161] In terms of point features, when frequency domain features are used as point features, the accuracy is significantly higher than the other two point features, regardless of whether the edge features are distance or Pearson correlation coefficient. Therefore, frequency domain features can be preferred as point features to achieve emotion recognition.

[0162] This invention also provides an EEG emotion recognition system based on deep neural networks. Please refer to [link to relevant documentation]. Figure 9 The system includes:

[0163] The node feature determination module 901 is configured to use all EEG channels as nodes to construct an adaptive dynamic adjacency matrix; and to use time-series EEG signals, statistical features of EEG signals, and / or frequency domain features of EEG signals as node features.

[0164] Graph feature construction module 902 is configured to construct graph features from the nodes, the adaptive dynamic adjacency matrix, and node features;

[0165] The graph feature processing module 903 is configured to construct a GNN-LSTM-MSTF model, which includes a graph convolutional network, a long short-term memory network, and a multi-scale time-frequency feature fusion network. In response to the input graph features, the GNN-LSTM-MSTF model uses a graph convolutional network to extract spatial features of the graph features, combines a long short-term memory network to extract temporal features, and combines multi-scale time-frequency feature fusion to enhance the modeling of instantaneous emotional changes, thereby obtaining the temporal encoding representation and time-frequency fusion features of the entire sequence.

[0166] The brain network feature extraction module 904 is configured to use a functional connectivity network to extract topological information of brain regions as brain network features.

[0167] Feature concatenation module 905 is configured to concatenate brain network features with the output of the GNN-LSTM-MSTF model to form the final feature vector. ;

[0168] The classification output module 906 is configured to output emotion recognition results based on the final feature vector using a multilayer perceptron classification model. The emotion recognition results are one or a combination of alertness, arousal, dominance, and liking.

[0169] It should be noted that the deep neural network-based EEG emotion recognition system and the previously described method belong to the same technical concept, have the same technical principles and beneficial effects, so they will not be repeated here.

[0170] The above embodiments are only used to illustrate the present invention and are not intended to limit the present invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, all equivalent technical solutions also fall within the scope of the present invention, and the patent protection scope of the present invention should be defined by the claims.

Claims

1. A brainwave emotion recognition method based on deep neural networks, characterized in that, The method includes: An adaptive dynamic adjacency matrix is ​​constructed by treating all EEG channels as nodes; the EEG signals of time series, the statistical characteristics of EEG signals, and / or the frequency domain characteristics of EEG signals are used as node features. The nodes, the adaptive dynamic adjacency matrix, and the node features are constructed into graph features; A GNN-LSTM-MSTF model is constructed, which includes a graph convolutional network, a long short-term memory network, and a multi-scale time-frequency feature fusion network. The GNN-LSTM-MSTF model responds to the input graph features, uses a graph convolutional network to extract spatial features of the graph features, combines a long short-term memory network to extract temporal features, and combines multi-scale time-frequency feature fusion to enhance the modeling of instantaneous emotional changes, thereby obtaining the temporal encoding representation and time-frequency fusion features of the entire sequence. Functional connectivity networks are used to extract topological information of brain regions as brain network features; the brain network features are global topological features extracted from graph features. The brain network features are concatenated with the output of the GNN-LSTM-MSTF model to form the final feature vector. ; Based on the final feature vector, a multilayer perceptron classification model is used to output the emotion recognition result, which is one or a combination of alertness, arousal, dominance, and liking. The GNN-LSTM-MSTF model responds to the input graph features, employs a graph convolutional network to extract spatial features of the graph features, combines a long short-term memory network to extract temporal features, and integrates multi-scale time-frequency feature fusion to enhance the modeling of instantaneous emotional changes. The methods for obtaining the temporal encoding representation and time-frequency fusion features of the entire sequence include: Graph convolution operations are used to embed the constructed graph features into the graph, capturing the spatial dependencies between different EEG channels. Let the graph features be... ,in, Represents the node feature matrix, Let N be the set of real numbers, N be the number of channels, and V be the set of nodes. Let be the set of edges. F The feature dimension is and the adjacency matrix is ​​. The graph convolution calculation is shown in equation (1): in, It is the first The node feature matrix input to the layer has a dimension of , It is the first The layer outputs a matrix of node representations. It is the first The learnable weight matrix of the layer, To add the adjacency matrix after adding self-loops, yes The degree matrix, Initially, node features, For the first Layer learnable weights For activation functions; The embedding vector of each node output by graph convolution is used as a temporal input to capture the dynamic dependence of EEG signals between different time windows; for the time input sequence The recursive form of LSTM is: in, The time-coded representation of the entire sequence; Indicates the input gate at time step activation value, This represents the input weight matrix of the input gate. This indicates the hidden state at the current time step. This indicates the hidden state at the previous moment. This represents the bias term of the input gate. This represents the input weight matrix of the forget gate. This represents the hidden state weight matrix of the forget gate. The bias term representing the forget gate. Indicates the forget gate at time step activation value, Indicates the output gate at time step activation value, This represents the hidden state weight matrix of the output gate. This represents the bias term of the output gate. This indicates the cell state at the current time step. This indicates the cell state at the previous time step. The input weight matrix represents the candidate matrix. The hidden state weight matrix represents the candidate state. Bias terms representing candidate states; Use STFT to obtain time-frequency spectrum Capture fixed-frequency domain information through a sliding window: in, Indicates at time ,frequency The STFT transformation results on the surface, This indicates the original time signal at time point. The value of , The window function is centered at... , Denotes the base of the natural logarithm. Represents the imaginary unit. Represents frequency variables. Represents the integral variable, and represents a point on the entire time axis. Indicates the current time position in the analysis; Extracting short-time dynamic features using continuous wavelet transform: In the formula, The result of the wavelet transform is represented by the scale. and time Local features of the signal below, Indicates the scale parameter. Indicates time and location. Indicates the time of the input signal The value of ; Will and After convolutional encoding, and Fusion yields time-frequency fusion features. .

2. The method as described in claim 1, characterized in that, Methods for extracting topological information of brain regions as brain network features using functional connectivity networks include: A functional connectivity network is constructed, and global topological features are extracted from graph features as brain network features using the following formula. These global topological features include one of centrality, clustering coefficient, and path length, where the formula for calculating centrality is: in, express s arrive t The number of shortest paths, Indicates the process v The number of paths, s , t , v This represents a node in the graph.

3. The method as described in claim 1, characterized in that, The brain network features are concatenated with the output of the GNN-LSTM-MSTF model to form the final feature vector. : In the formula, Indicates time-coded features. This represents the encoding result after multi-scale fusion. Indicates brain network characteristics; The fused feature Z is input into a multilayer perceptron to obtain the output emotion recognition result: In the formula, This represents the emotion recognition result vector predicted by the model. This represents a multilayer perceptron.

4. The method as described in claim 1, characterized in that, An adaptive dynamic adjacency matrix can be constructed as follows: Calculate the spatial distance and signal similarity between EEG channels; For any two EEG channels and Calculate Euclidean distance using three-dimensional coordinates: Forming a spatial matrix ,in, , , It is the first The coordinates of each channel in three-dimensional space, where N is the number of channels. For two EEG channels and Euclidean distance; The Pearson correlation coefficient is used to measure the signal correlation between any two channels. Form a similarity matrix : In the formula, Indicates channel and channels Covariance between Indicates channel standard deviation Indicates channel Standard deviation; Initial edge weights are defined based on spatial distance and signal similarity. for: in, These are weighting coefficients. To prevent division by zero of small constants; Adaptive weighting is used to dynamically adjust the edge weights of the adjacency matrix: making the adjacency matrix a learnable parameter matrix. During model training, updates are performed using gradient descent: in, For learning rate, For the overall identification loss function, This is the descent gradient.

5. The method as described in claim 1, characterized in that, The multi-scale time-frequency feature fusion method includes: Calculate the short-time Fourier transform to extract the steady-state frequency domain features of the EEG signal; Calculate continuous wavelet transform to extract short-time dynamic features of EEG signals; By combining the above features through a feature fusion mechanism, the ability to identify emotion patterns at different time scales can be enhanced.

6. The method as described in claim 1, characterized in that, The nodes, adjacency matrix, and node features are constructed into graph features, including: The EEG channels are considered as nodes in a graph, with each node corresponding to one EEG channel. Node features include: time-domain features, frequency-domain features, and wavelet transform features. An adaptive dynamic adjacency matrix is ​​adopted, and the edge weights of the adjacency matrix are dynamically adjusted based on adaptive weighting. Optimize the adjacency matrix by combining physical distance and relevance information; By integrating the adjacency matrix, node features, and brain network features, graph structure data is obtained.

7. The method as described in claim 1, characterized in that, The graph convolutional network and the long short-term memory network are combined to form a graph convolutional long short-term memory network. The graph convolutional long short-term memory network includes a graph convolutional layer, a long short-term memory network, and a multilayer perceptron. The graph convolutional layer is used to extract brain network features from the graph features, the long short-term memory network is used to extract time-frequency features, and the multilayer perceptron is used to fuse the brain network features and time-frequency features and output the emotion recognition result.

8. The method as described in claim 7, characterized in that, The graph convolutional layer is configured as three layers. Each graph convolutional layer includes a convolutional layer, a linear activation function layer, and a random deactivation layer connected in sequence. The output of the three graph convolutional layers is aggregated through a global pooling layer and then input into the long short-term memory network.

9. A brainwave emotion recognition system based on deep neural networks, characterized in that, The system includes: The node feature determination module is configured to use all EEG channels as nodes to construct an adaptive dynamic adjacency matrix; and to use time-series EEG signals, statistical features of EEG signals, and / or frequency domain features of EEG signals as node features. The graph feature construction module is configured to construct graph features from the nodes, the adaptive dynamic adjacency matrix, and the node features; The graph feature processing module is configured to construct a GNN-LSTM-MSTF model. This GNN-LSTM-MSTF model includes a graph convolutional network, a long short-term memory network, and a multi-scale time-frequency feature fusion network. Responding to the input graph features, the GNN-LSTM-MSTF model uses a graph convolutional network to extract spatial features, combines a long short-term memory network to extract temporal features, and integrates multi-scale time-frequency feature fusion to enhance the modeling of instantaneous emotional changes, resulting in the temporal encoding representation and time-frequency fusion features of the entire sequence. Specifically, this includes: Graph convolution operations are used to embed the constructed graph features into the graph, capturing the spatial dependencies between different EEG channels. Let the graph features be... ,in, Represents the node feature matrix, Let N be the set of real numbers, N be the number of channels, and V be the set of nodes. Let be the set of edges. F The feature dimension is and the adjacency matrix is ​​. The graph convolution calculation is shown in equation (1): in, It is the first The node feature matrix input to the layer has a dimension of , It is the first The layer outputs a matrix of node representations. It is the first The learnable weight matrix of the layer, To add the adjacency matrix after adding self-loops, yes The degree matrix, Initially, node features, For the first Layer learnable weights For activation functions; The embedding vector of each node output by graph convolution is used as a temporal input to capture the dynamic dependence of EEG signals between different time windows; for the time input sequence The recursive form of LSTM is: in, The time-coded representation of the entire sequence; Indicates the input gate at time step activation value, This represents the input weight matrix of the input gate. This indicates the hidden state at the current time step. This indicates the hidden state at the previous moment. This represents the bias term of the input gate. This represents the input weight matrix of the forget gate. The hidden state weight matrix represents the forget gate. The bias term representing the forget gate. Indicates the forget gate at time step activation value, Indicates the output gate at time step activation value, This represents the hidden state weight matrix of the output gate. This represents the bias term of the output gate. This indicates the cell state at the current time step. This indicates the cell state at the previous time step. The input weight matrix represents the candidate matrix. The hidden state weight matrix represents the candidate state. Bias terms representing candidate states; Use STFT to obtain time-frequency spectrum Capture fixed-frequency domain information through a sliding window: in, Indicates at time ,frequency The STFT transformation results on the surface, This indicates the original time signal at time point. The value of , The window function is centered at... , Denotes the base of the natural logarithm. Represents the imaginary unit. Represents frequency variables. Represents the integral variable, and represents a point on the entire time axis. Indicates the current time position in the analysis; Extracting short-time dynamic features using continuous wavelet transform: In the formula, The result of the wavelet transform is represented by the scale. and time Local features of the signal below, Indicates the scale parameter. Indicates time and location. Indicates the time of the input signal The value of ; Will and After convolutional encoding, and Fusion yields time-frequency fusion features. ; The brain network feature extraction module is configured to use a functional connectivity network to extract topological information of brain regions as brain network features; the brain network features are global topological features extracted from graph features. The feature concatenation module is configured to concatenate brain network features with the output of the GNN-LSTM-MSTF model to form the final feature vector. ; The classification output module is configured to output emotion recognition results based on the final feature vector using a multilayer perceptron classification model. The emotion recognition results are one or a combination of alertness, arousal, dominance, and liking.

Images