Electroencephalogram emotion recognition method and system based on deep neural network

Abstract

The invention relates to the technical field of electroencephalogram signal processing, and discloses an electroencephalogram emotion recognition method and system based on a deep neural network. The method comprises the following steps: collecting and preprocessing a multi-channel EEG signal; constructing a graph data structure, extracting multi-domain features by taking electroencephalogram channels as nodes, and constructing a self-adaptive dynamic adjacency matrix; constructing a graph convolution long and short-term memory network, learning spatial features by GNN, and extracting time dependence by LSTM; enhancing emotion capture by using a multi-scale time-frequency feature fusion method in combination with STF and CWT; constructing global topological information of an FCN brain extraction region in combination with brain network features; and outputting alertness and other emotion indexes by means of the classification model. According to the method, graph structure learning and time sequence modeling are combined, EEG signal emotion recognition is optimized, and personalized adaptation and emotion recognition accuracy is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electroencephalogram signal processing, and particularly relates to an electroencephalogram emotion recognition method and system based on a deep neural network. BACKGROUND

[0002] In recent years, brain-computer interface technology has attracted widespread attention in the field of electroencephalogram emotion recognition, and related research has gradually deepened into the acquisition and processing of electroencephalogram signals. Electroencephalogram signal acquisition is the first step in emotion recognition, usually through the placement of electrodes on the scalp to monitor the electrical activity of the brain. However, due to the weak nature of electroencephalogram signals, the collected signals often need to be amplified. At the same time, these signals are easily affected by noise such as electrooculogram, electrocardiogram, electromagnetic interference, and electromyogram, which can mask the target signal and thus reduce the accuracy of emotion recognition. Therefore, signal preprocessing is a crucial step.

[0003] Early electroencephalogram signal preprocessing usually relies on manual observation and interference removal, which is not only time-consuming but also has limited effectiveness in processing weak signals. With the continuous deepening of research, automated noise removal algorithms have gradually developed, among which filtering and independent component analysis (ICA) methods are widely used, which can effectively purify electroencephalogram signals and thus improve the accuracy and stability of emotion recognition.

[0004] In addition to signal processing, feature extraction and model construction are also key links in the emotion recognition system. In recent years, with the rapid development of deep learning technology, researchers have gradually integrated feature extraction and model construction into an end-to-end process, through training deep learning models to automatically extract the most representative features from raw signals, thus achieving more accurate emotion recognition. This end-to-end integration method not only improves the accuracy of recognition but also reduces human intervention, opening up new prospects for the application of brain-computer interface technology in the field of emotion recognition.

[0005] Time-domain features are an intuitive and important feature type in electroencephalogram signal processing, especially when studying time-dependent phases (ERP), the application of P1 and N170 components in emotion recognition has attracted widespread attention. Although time-domain features can show the changes of signals over time, they have certain limitations in extracting high-order features of electroencephalogram signals, mainly because they cannot effectively capture frequency information.

[0006] To fully exploit the information in EEG signals, researchers have started exploring methods for extracting frequency-domain features. For example, Li and his team used short-time Fourier transform (STFT) to convert EEG signals into frequency-domain representations and extracted the power spectral density (PSD) of each frequency band as a feature, achieving an average recognition rate of 82.29%. In addition, Xiang and his team constructed a feature matrix through differential entropy, achieving an average recognition accuracy of 90.45% and an arousal recognition accuracy of 90.60%.

[0007] Deep learning algorithms have shown strong capabilities in emotion classification tasks, especially in handling large-scale datasets and automatically extracting complex features. For example, Maheshwari et al. constructed an EEG emotion classification model containing 8 one-dimensional convolutional layers, achieving an accuracy of 98.99% on the DEAP dataset. Zhu et al. proposed a model containing 5 layers of two-dimensional CNN and GRU for extracting spatial and temporal features, achieving arousal and valence recognition rates of 87.86% and 88.69%, respectively.

[0008] In addition to convolutional neural networks (CNN), other deep learning models have also achieved good classification results in EEG emotion recognition. For example, Zheng et al. used deep belief networks (DBN) to achieve an emotion classification accuracy of 86.08% on the SEED dataset.

[0009] Due to the large number of EEG channels, not all channels play an important role in emotion recognition, and redundant information may affect the performance of the algorithm. Therefore, screening channels closely related to emotion recognition has become an important task in emotion recognition research. For example, Tong et al. proposed a good combination of EEG channels; Wang et al. proposed the most suitable channel combination according to the needs of emotion valence and arousal classification; Xiang et al. proposed 5 EEG channels suitable for emotion recognition.

[0010] Based on the above analysis, the shortcomings of existing technologies are as follows: EEG signals are susceptible to interference. The collection of EEG signals is affected by environmental factors, and the relatively pure data in the laboratory environment may not fully reflect the signals in the real world.

[0011] Individual differences. The structure and function of each person's brain are different, which leads to differences in EEG signals even when facing the same stimulus. This inter-individual variability needs to be analyzed and learned through a large amount of data to establish a more extensive correspondence between emotions and EEG signals.

[0012] The correlation between brain channels is not well studied. There are complex interactions between different regions of the brain, which are reflected in the correlation between channels in electroencephalogram signals. However, current research rarely considers these inter-channel correlations in deep learning models. A deeper understanding of the correlation between brain channels is crucial for building more accurate electroencephalogram signal models. However, this area still needs further research and exploration. SUMMARY

[0013] The purpose of the present application is to provide a deep neural network-based electroencephalogram emotion recognition method and system, abandoning the traditional three-classification method, adopting the form of prediction score output, combining spatial features and temporal features, in order to achieve more accurate emotion recognition.

[0014] In order to achieve the above purpose, the technical scheme adopted is as follows: In a first aspect, the present application provides a deep neural network-based electroencephalogram emotion recognition method, comprising: All electroencephalogram channels are taken as nodes to construct an adaptive dynamic adjacency matrix; time series electroencephalogram signals, statistical features of electroencephalogram signals, and / or frequency domain features of electroencephalogram signals are taken as node features; The nodes, adaptive dynamic adjacency matrix, and node features are constructed into graph features; A GNN-LSTM-MSTF model is constructed, which includes a graph convolution network, a long short-term memory network, and a multi-scale time-frequency feature fusion network. The GNN-LSTM-MSTF model responds to the input graph features, extracts spatial features of the graph features using the graph convolution network, extracts temporal features using the long short-term memory network, and combines the multi-scale time-frequency feature fusion to enhance the modeling of instantaneous emotional changes, obtaining the time coding representation and time-frequency fusion features of the entire sequence; A functional connectivity network is used to extract the topological structure information between brain regions as brain network features; The brain network features and the output of the GNN-LSTM-MSTF model are spliced to form a final feature vector

[0015] Based on the final feature vector, a multi-layer perceptron classification model is used to output an emotion recognition result, which is one of alertness, arousal, dominance, and liking, and combinations thereof.

[0016] Further, the GNN-LSTM-MSTF model responds to the input graph features, extracts spatial features of the graph features using the graph convolution network, extracts temporal features using the long short-term memory network, and combines the multi-scale time-frequency feature fusion to enhance the modeling of instantaneous emotional changes, obtaining the time coding representation and time-frequency fusion features of the entire sequence in the following manner: Graph convolution operations are used to embed the constructed graph features into the graph, capturing the spatial dependencies between different EEG channels. Let the graph features be... ,in, Representing node characteristics, Let N be the set of real numbers, N be the number of channels, and V be the set of nodes. Let be the set of edges. F The feature dimension is and the adjacency matrix is ​​. The graph convolution calculation is shown in equation (1): (1) in, It is the first The node feature matrix input to the layer has a dimension of . , It is the first The node representation matrix is ​​output by the layer. It is the first The learnable weight matrix of the layer, To add the adjacency matrix after adding self-loops, yes The degree matrix, Initially, node features, For the first Layer learnable weights For activation functions; The embedding vector of each node output by graph convolution is used as a temporal input to capture the dynamic dependence of EEG signals between different time windows; for the time input sequence The recursive form of LSTM is: (2) (3) (4) (5) (6) in, The time-coded representation of the entire sequence; Indicates the input gate at time step activation value, This represents the input weight matrix of the input gate. This indicates the hidden state at the current time step. This indicates the hidden state at the previous moment. This represents the bias term of the input gate. This represents the input weight matrix of the forget gate. The hidden state weight matrix represents the forget gate. The bias term representing the forget gate. Indicates the forget gate at time step activation value, Indicates the output gate at time step activation value, This represents the hidden state weight matrix of the output gate. This represents the bias term of the output gate. This indicates the cell state at the current time step. This indicates the cell state at the previous time step. The input weight matrix represents the candidate matrix. The hidden state weight matrix represents the candidate state. Bias terms representing candidate states; Use STFT to obtain time-frequency spectrum Capture fixed-frequency domain information through a sliding window: (7) in, Indicates at time ,frequency The STFT transformation results on the surface, This indicates the original time signal at time point. The value of , The window function is centered at... , Denotes the base of the natural logarithm. Represents the imaginary unit. Represents frequency variables. Represents the integral variable, and represents a point on the entire time axis. Indicates the current time position in the analysis; Extracting short-time dynamic features using continuous wavelet transform: (8) In the formula, The result of the wavelet transform is represented by the scale. and time Local features of the signal below, Indicates the scale parameter. Indicates time and location. Indicates the time of the input signal The value of ; Will and After convolutional encoding, and Fusion yields time-frequency fusion features. .

[0017] Furthermore, methods for extracting topological information of brain regions as brain network features using functional connectivity networks include: A functional connectivity network is constructed, and global topological features are extracted from graph features as brain network features using the following formula. These global topological features include one of centrality, clustering coefficient, and path length, where the formula for calculating centrality is: (9) in, express s arrive t The number of shortest paths, Indicates the process v The number of paths, s , t , v This represents a node in the graph.

[0018] Furthermore, the brain network features are concatenated with the output of the GNN-LSTM-MSTF model to form the final feature vector. : (10) In the formula, Indicates time-coded features. This represents the encoding result after multi-scale fusion. Indicates brain network characteristics; The fused feature Z is input into a multilayer perceptron to obtain the output emotion recognition result: (11) In the formula, This represents the emotion recognition result vector predicted by the model. This represents a multilayer perceptron.

[0019] Furthermore, the adaptive dynamic adjacency matrix is ​​constructed as follows: Calculate the spatial distance and signal similarity between EEG channels; For any two EEG channels and Calculate Euclidean distance using three-dimensional coordinates: (12) Forming a spatial matrix ,in, , , It is the first The coordinates of each channel in three-dimensional space, where N is the number of channels. For two EEG channels and Euclidean distance; The Pearson correlation coefficient is used to measure the signal correlation between any two channels. Form a similarity matrix : (13) wherein, denotes the covariance between channels and channels , denotes the standard deviation of channels , denotes the standard deviation of channels ; The initial edge weight is defined according to the spatial distance and signal similarity as follows: (14) wherein, is a weighting coefficient, is a small constant to prevent division by zero; The adaptive weighted dynamic adjustment of the edge weight of the adjacency matrix is adopted: the adjacency matrix becomes a learnable parameter matrix , which is updated by gradient descent during model training: (15) wherein, is a learning rate, is the overall recognition loss function, is the descending gradient.

[0020] At this point, the generated adaptive dynamic adjacency matrix (ADAM) can flexibly adjust the graph structure according to the spatial topology and dynamic time sequence characteristics of the EEG signal, avoiding information loss caused by static adjacency. Improve the modeling accuracy of the dependence relationship between EEG channels by the graph neural network, thereby enhancing the accuracy and generalization ability of emotion classification.

[0021] Further, the multi-scale time-frequency feature fusion method comprises: calculating the short-time Fourier transform to extract the steady-state frequency domain features of the EEG signal; calculating the continuous wavelet transform to extract the short-time dynamic features of the EEG signal; combining the above features through a feature fusion mechanism to enhance the recognition ability of different time scale emotion patterns.

[0022] Further, the node, adjacency matrix and node feature are constructed as a graph feature, comprising: the EEG channels are regarded as nodes of the graph, and each node corresponds to an EEG channel. The node features include: time domain features, frequency domain features, wavelet transform features; an adaptive dynamic adjacency matrix is adopted to dynamically adjust the edge weight of the adjacency matrix based on adaptive weighting; the adjacency matrix is optimized in combination with physical distance and correlation information; The adjacency matrix, node features and brain network features are integrated to obtain graph structure data.

[0023] Further, the graph convolution network and the long short-term memory network are combined to form a graph convolution long short-term memory network, and the graph convolution long short-term memory network comprises a graph convolution layer, a long short-term memory network and a multilayer perceptron; wherein the graph convolution layer is used to extract brain network features in the graph features, the long short-term memory network is used to extract time-frequency features, and the multilayer perceptron is used to fuse the brain network features and the time-frequency features and output an emotion recognition result.

[0024] Further, the graph convolution layer is set to three layers, each layer of the graph convolution layer comprises a convolution layer, a linear activation function layer and a random inactivation layer connected in sequence, and the outputs of the three layers of the graph convolution layer are aggregated by a global pooling layer and then input to the long short-term memory network.

[0025] In a second aspect, the present application provides an electroencephalogram emotion recognition system based on a deep neural network, and the system comprises: A node feature determination module is configured to take all electroencephalogram channels as nodes to construct an adaptive dynamic adjacency matrix; and take time series electroencephalogram signals, statistical features of the electroencephalogram signals and / or frequency domain features of the electroencephalogram signals as node features. A graph feature construction module is configured to construct graph features from the nodes, the adaptive dynamic adjacency matrix and the node features. A graph feature processing module is configured to construct a GNN-LSTM-MSTF model, and the GNN-LSTM-MSTF model comprises a graph convolution network, a long short-term memory network and a multi-scale time-frequency feature fusion network; the GNN-LSTM-MSTF model extracts spatial features of the graph features by using the graph convolution network in response to input graph features, extracts time features by combining the long short-term memory network, and enhances modeling of instantaneous emotion changes by combining the multi-scale time-frequency feature fusion, to obtain time coding representation and time-frequency fusion features of the entire sequence. A brain network feature extraction module is configured to extract topological structure information between brain regions as brain network features by using a functional connection network. A feature splicing module is configured to splice the brain network features and the output of the GNN-LSTM-MSTF model to form a final feature vector. A classification output module is configured to output an emotion recognition result by using a multilayer perceptron classification model based on the final feature vector, and the emotion recognition result is one of alertness, arousal, degree of being dominated, and like degree and combinations thereof. The present application has the following beneficial effects:

[0026] ​The application constructs an EEG emotion recognition method based on a graph neural network by combining the characteristics of space, time and frequency domain. The channel connection relationship is optimized through an adaptive adjacency matrix (ADMA), and the model generalization capability is improved in combination with a brain functional connection network (FCN). In the experiment, the DEAP data set is selected, and the data is divided into a training set, a verification set and a test set according to a ratio of 6:1:1. After feature fusion, the frequency domain and spatial features are combined, and a convolutional neural network is used for training, and finally an accuracy of 93% is obtained, and the emotion recognition effect is relatively ideal. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 A structural diagram of an electroencephalogram emotion recognition framework according to an embodiment of the application is shown.

[0028] Figure 2 A flowchart of an electroencephalogram emotion recognition method based on a deep neural network according to an embodiment of the application is shown.

[0029] Figure 3 A flowchart of emotion recognition based on a graph convolutional long short-term memory network according to an embodiment of the application is shown.

[0030] Figure 4 A schematic diagram of the arousal dimension according to an embodiment of the application is shown; wherein (a), the point feature is a time series electroencephalogram signal, and the edge feature is a distance to generate an accuracy image; (b), the point feature is a time series electroencephalogram signal, and the edge feature is a Pearson coefficient to generate an accuracy image.

[0031] Figure 5 A schematic diagram of the arousal dimension according to an embodiment of the application is shown; wherein (a), the point feature is a time series electroencephalogram signal, and the edge feature is a distance to generate an accuracy image; (b), the point feature is a time series electroencephalogram signal, and the edge feature is a Pearson coefficient to generate an accuracy image.

[0032] Figure 6 A schematic diagram of the degree of dominance dimension according to an embodiment of the application is shown; wherein (a), the point feature is a time series electroencephalogram signal, and the edge feature is a distance to generate an accuracy image; (b), the point feature is a time series electroencephalogram signal, and the edge feature is a Pearson coefficient to generate an accuracy image.

[0033] Figure 7 A schematic diagram of the degree of dominance dimension according to an embodiment of the application is shown; wherein (a), the point feature is a time series electroencephalogram signal, and the edge feature is a distance to generate an accuracy image; (b), the point feature is a time series electroencephalogram signal, and the edge feature is a Pearson coefficient to generate an accuracy image.

[0034] Figure 8A schematic diagram of a degree-of-influence loss function according to an embodiment of the application is shown.

[0035] Figure 9 A structural diagram of a deep neural network-based electroencephalogram emotion recognition system according to an embodiment of the application is shown. DETAILED DESCRIPTION

[0036] The present application can be implemented or applied in other different specific embodiments, and the details in the specification can be modified or changed based on different views and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0037] The specific embodiments of the present application are described in detail below with reference to the accompanying drawings and embodiments.

[0038] Emotion plays an increasingly important role in the daily life of modern people, especially in the field of human-computer interaction, the importance of emotion recognition is increasing. Electroencephalogram (EEG) as an effective means, can be in the multimedia interaction implicit monitoring of user's emotion and perception state. Because of its simple operation and high stability, electroencephalogram signal acquisition is widely used in emotion recognition. Therefore, the embodiment of the present application provides a deep neural network-based electroencephalogram emotion recognition method.

[0039] In order to improve the performance of the deep neural network-based emotion recognition method in accuracy and robustness. The embodiment proposes a combination of graph convolutional neural network (GCN) and long short-term memory network (LSTM) electroencephalogram emotion recognition method, namely graph convolutional long short-term memory network (GC-LSTNet). The method extracts spatial, temporal and frequency domain features. And use all 32 electroencephalogram channel signals in the original data. These channels are used as nodes in the graph, and the adjacency matrix is constructed by the spatial distance vector and the correlation between channels. At the same time, statistical features and frequency domain signals are extracted from the time series as node features. In the experiment, DEAP dataset is selected, and the data is divided into training set, validation set and test set according to the ratio of 6:1:1. After feature fusion, combining frequency domain and spatial features, convolutional neural network is used for training, and finally 93% accuracy is obtained. The effect is ideal.

[0040] The embodiment of the present application provides an electroencephalogram emotion recognition framework, the structure of which is as follows Figure 1The EEG emotion recognition framework is shown. In the EEG emotion recognition framework, the EEG signal is collected first, and the collection method includes but is not limited to using an existing EEG signal collection system to collect, and common EEG signal collection systems include TDT, Neuralynx, Blackrock Microsystems, Plexon and Open Ephys. Subsequently, the collected EEG signal is preprocessed to generate a feature map, and then a graph convolutional long short-term memory network is used for feature extraction and classification recognition, the extracted features include brain network features and time-frequency features, and finally a prediction score is used to output the prediction result. Among them, the brain network features include connection matrix, brain network and network attribute, and the time-frequency features include time domain features and frequency domain features. The classification recognition result includes but is not limited to, for example Figure 1 The two emotions of happiness and sadness are shown.

[0041] The EEG emotion recognition method based on the deep neural network can realize emotion recognition through the EEG emotion recognition framework as shown in Figure 1 Specifically, as shown in Figure 2 The EEG emotion recognition method based on the deep neural network includes steps S100 to S500, which are described in detail as follows.

[0042] S100, all EEG channels are taken as nodes to construct an adaptive dynamic adjacency matrix; the time series EEG signal, the statistical feature of the EEG signal and / or the frequency domain feature of the EEG signal are taken as node features.

[0043] In some embodiments, the adaptive dynamic adjacency matrix is constructed in the following way: The spatial distance and signal similarity between the EEG channels are calculated; For any two EEG channels and , the Euclidean distance is calculated using three-dimensional coordinates: (12) The spatial matrix is formed, wherein , , is the coordinate value of the first channel in three-dimensional space, N is the number of channels, is the Euclidean distance of the two EEG channels and ; The Pearson correlation coefficient is used to measure the signal correlation between any two channels , and the similarity matrix is formed: (13) In the formula,​ covariance between channels, and channels, covariance between channels, standard deviation of channels, standard deviation of channels, standard deviation of channels; define initial edge weights according to spatial distance and signal similarity (14) wherein, is a weighting coefficient, is a small constant to prevent division by zero; adopt adaptive weighting to dynamically adjust the edge weights of the adjacency matrix: let the adjacency matrix become a learnable parameter matrix , and update through gradient descent during model training: (15) wherein, is a learning rate, is the overall recognition loss function, is the descending gradient.

[0044] At this point, the generated adaptive dynamic adjacency matrix (ADAM) can flexibly adjust the graph structure according to the spatial topology and dynamic time sequence characteristics of the EEG signal, avoiding information loss caused by static adjacency. Improve the modeling accuracy of the dependence relationship between the channels of the electroencephalogram by the graph neural network, thereby enhancing the accuracy and generalization ability of emotion classification.

[0045] S200, construct the node, adaptive dynamic adjacency matrix and node feature into a graph feature.

[0046] S300, construct a GNN-LSTM-MSTF model, the GNN-LSTM-MSTF model includes a graph convolutional network, a long short-term memory network and a multi-scale time-frequency feature fusion network, the GNN-LSTM-MSTF model responds to the input graph feature, adopts the graph convolutional network to extract the spatial feature of the graph feature, combines the long short-term memory network to extract the time feature, and combines the multi-scale time-frequency feature fusion to enhance the modeling of instantaneous emotional changes, obtains the time coding representation and time-frequency fusion feature of the entire sequence.

[0047] In some embodiments, the GNN-LSTM-MSTF model responds to the input graph feature, adopts the graph convolutional network to extract the spatial feature of the graph feature, combines the long short-term memory network to extract the time feature, and combines the multi-scale time-frequency feature fusion to enhance the modeling of instantaneous emotional changes, obtains the time coding representation and time-frequency fusion feature of the entire sequence. The way includes: ​​Graph convolution operations are used to embed the constructed graph features into the graph, capturing the spatial dependencies between different EEG channels. Let the graph features be... ,in, Representing node characteristics, Let N be the set of real numbers, N be the number of channels, and V be the set of nodes. Let be the set of edges. F The feature dimension is and the adjacency matrix is ​​. The graph convolution calculation is shown in equation (1): (1) in, It is the first The node feature matrix input to the layer has a dimension of . , It is the first The node representation matrix is ​​output by the layer. It is the first The learnable weight matrix of the layer, To add the adjacency matrix after adding self-loops, yes The degree matrix, Initially, node features, For the first Layer learnable weights For activation functions; The embedding vector of each node output by graph convolution is used as a temporal input to capture the dynamic dependence of EEG signals between different time windows; for the time input sequence The recursive form of LSTM is: (2) (3) (4) (5) (6) in, The time-coded representation of the entire sequence; Indicates the input gate at time step activation value, This represents the input weight matrix of the input gate. This indicates the hidden state at the current time step. This indicates the hidden state at the previous moment. This represents the bias term of the input gate. This represents the input weight matrix of the forget gate. The hidden state weight matrix represents the forget gate. The bias term representing the forget gate. denotes the activation value of the forget gate at time step , denotes the activation value of the output gate at time step , denotes the hidden state weight matrix of the output gate, denotes the bias term of the output gate, denotes the cell state of the current time step, denotes the cell state of the previous time step, denotes the input weight matrix of the candidate matrix, denotes the hidden state weight matrix of the candidate state, denotes the bias term of the candidate state; Using STFT to obtain time-frequency spectrum , capture the fixed frequency domain information by sliding window: (7) wherein, denotes, denotes, denotes, denotes, denotes, denotes, denotes, denotes; Using continuous wavelet transform to extract short-time dynamic characteristics: (8) wherein, denotes the STFT transform result at time , frequency , denotes the value of the original time signal at time point , denotes the window function, and the center is located at , denotes the base of natural logarithm, denotes the imaginary unit, denotes the frequency variable, denotes the integral variable, and denotes the point on the entire time axis, denotes the current analysis time position; After convolutional encoding of and , and fusion with , time-frequency fusion features are obtained.

[0048] As Figure 3As shown, the flowchart for implementing emotion recognition based on a graph convolutional long short-term memory network. Among them, the graph convolutional long short-term memory network is the GNN-LSTM module in the GNN-LSTM-MSTF model. When performing emotion recognition, the input is the signal collected by the electroencephalogram signal acquisition system, and the electroencephalogram feature is obtained through the existing algorithm processing, and the electroencephalogram feature is taken as the electroencephalogram signal. Through the above steps S100-S300, the preprocessing of the electroencephalogram signal is realized, and the EEG image is obtained. Finally, the graph convolutional long short-term memory network responds to the input EEG image through the processing of the graph convolutional layer, the long short-term memory network (LSTM) and the multi-layer perception (MLP), and obtains the final output (output). The graph convolutional layer is set to three layers, and each layer of the graph convolutional layer includes a convolutional layer, a linear activation function layer and a random inactivation layer connected in turn. The output of the three-layer graph convolutional layer is aggregated through the global pooling layer and then input to the long short-term memory network. As shown in Figure 3 The convolutional layer, linear activation function layer and random inactivation layer of the first layer of graph convolutional layer are represented as GConv1, Relu1 and Dropout1 respectively, the convolutional layer, linear activation function layer and random inactivation layer of the second layer of graph convolutional layer are represented as GConv2, Relu2 and Dropout2 respectively, and the convolutional layer, linear activation function layer and random inactivation layer of the third layer of graph convolutional layer are represented as GConv3, Relu3 and Dropout3 respectively.

[0049] In the experimental setup, the batch size is set to 128, the learning rate is set to 0.001, ReLU is used as the activation function, MSELoss is used as the loss function, and Adam is used as the optimizer. In the data processing stage, the original data is converted into a graph form, and the time domain features and frequency domain features are extracted.

[0050] In order to process these two different features, two graph convolutional long short-term memory networks can be used. Each graph convolutional long short-term memory network consists of three layers of graph convolutional layers (GCN layers), one layer of long short-term memory network (LSTM) layers and one layer of multi-layer perceptron (MLP) layers. The graph convolutional layer is used to process the graph structure data, the LSTM layer is used to capture the dynamic characteristics of the time series data, and the MLP layer is used to generate the final prediction result (emotion recognition result). Among them, the graph convolutional layer is responsible for extracting the structural information in the graph, the LSTM layer is responsible for processing the time series data, and the MLP layer is responsible for combining the outputs of the graph convolutional layer and the LSTM layer to generate the final prediction result. This multi-level model structure enables the model to effectively process complex data features and improve its performance in tasks such as emotion analysis.

[0051] In constructing the model, the input channel number of the first layer graph convolution is set to 8064, the output channel number is set to 4032, a ReLU activation function is added to increase the nonlinearity of the model, and a dropout layer is added with a dropout probability of 0.2 to randomly inactivate 20% of the neurons to reduce the risk of overfitting.

[0052] After this layer of processing, the 32x8064 data is converted to 32x4032 data. Then we use the global pooling layer to aggregate the features of the entire graph for graph classification. After global pooling, the data changes from 32x512 to 1x512. Subsequently, we introduce the LSTM layer, and the parameter setting of LSTM is 2, 3, 2, which means that the LSTM layer contains two hidden states, each with three units. After processing by the LSTM layer, the data is converted to 1x1536. Finally, the MLP is used to generate a 1x1 prediction value.

[0053] S400, adopt a functional connection network to extract the topological structure information between brain regions as brain network features.

[0054] In some embodiments, the method of adopting a functional connection network to extract the topological structure information between brain regions as brain network features includes; Construct a functional connection network, and extract global topological features as brain network features from the graph features by the following formula, wherein the global topological features include one of centrality, clustering coefficient, and path length, and the calculation formula of the centrality is: (9) wherein, represents s the number of shortest paths from t to , v represents the number of paths passing through s , t , v represents a node in the graph.

[0055] S500, concatenate the brain network features with the output of the GNN-LSTM-MSTF model to form a final feature vector .

[0056] In some embodiments, the method of concatenating the brain network features with the output of the GNN-LSTM-MSTF model to form a final feature vector : (10) wherein, represents a time encoding feature, represents a multi-scale fusion encoding result, representing a brain network feature.

[0057] S600, based on the final feature vector, adopting a multilayer perceptron classification model, outputting an emotion recognition result, the emotion recognition result being one of alertness, arousal, degree of being dominated, and like degree and a combination thereof.

[0058] In some embodiments, the fusion feature Z is input into a multilayer perceptron to obtain an output emotion recognition result: (11) In the formula, representing a model predicted output emotion recognition result vector, representing a multilayer perceptron.

[0059] In some embodiments, the way of constructing the node, the adjacency matrix and the node feature into a graph feature is: regarding the EEG channel as a node of the graph, each node corresponding to an EEG channel, the node feature including: time domain feature, frequency domain feature, wavelet transform feature; adopting an adaptive dynamic adjacency matrix, based on adaptive weighted dynamic adjustment of the edge weight of the adjacency matrix; combining physical distance and correlation information, optimizing the adjacency matrix; integrating the adjacency matrix, the node feature and the brain network feature to obtain graph structure data.

[0060] The feasibility and progressiveness of the present application will be further illustrated below in combination with specific examples.

[0061] In the experiment, two edge features and three point features were used in total, and six combinations were obtained through permutation and combination. For each combination, four corresponding models were obtained through training. Figures 4 to 7 The case when the time series electroencephalogram signal is used as the point feature and the Pearson correlation coefficient is used as the edge feature combination is shown.

[0062] Therefore, the graph convolutional long short-term memory network (GC-LSTNet) provided by the present application can simultaneously extract the time sequence features and spatial features of the electroencephalogram signals, realize more accurate emotion recognition, and give the output in the form of a prediction score. Specifically, the present application adopts an LSTM network. The performance of the RNN is optimized, the limitation of parameter sharing in the RNN is broken, and the problems of gradient disappearance and gradient explosion are effectively alleviated. In the LSTM, each gate unit is equipped with a dedicated parameter set, so that the network can more flexibly control the flow of information. The GCN network adopts a graph convolutional neural network that gives up a fixed-size convolution kernel and aggregates the features of adjacent nodes according to the spatial structure, can effectively process non-Euclidean space data, and opens up a new road for the application of deep learning in the field of complex network analysis. The present application combines the two to create a graph convolutional long short-term memory network (GC-LSTNet) to simultaneously extract the time sequence features and spatial features of the electroencephalogram signals and realize more accurate emotion recognition.

[0063] The feasibility and progressiveness of the present application will be fully illustrated below by combining a specific implementation case.

[0064] In this implementation case, the data set is randomly divided into a training set, a validation set and a test set according to a ratio of 6:1:1. This division method helps to ensure the generalization ability of the model, because the test set is an independent data set that has not participated in the training and can more accurately evaluate the real performance of the model. We use the train_test_split() function in the sklearn library for simple cross-validation. This function can divide the data set into a training set, a validation set and a test set according to a certain ratio, so as to evaluate the performance and generalization ability of the machine learning model. By dividing the data into independent training and test sets, the performance of the model on unseen data can be effectively evaluated, so as to better understand the generalization ability and stability of the model, improve the reliability and practicality of the model. Taking the dominance dimension as an example, the performance of the algorithm on the training set and the validation set is as shown in the following table. Figure 8

[0065] The present embodiment adopts accuracy (Accuracy) as an evaluation index. The output of the model is a predicted score. For accuracy, the present embodiment sets a threshold, and when the difference between the predicted score and the true value is not greater than the threshold, the prediction is considered valid.

[0066] When the electroencephalogram signal is selected as the point feature and the Pearson correlation coefficient is selected as the edge feature to train the model, as shown in the following table, the accuracy in the alertness dimension is 75%, the accuracy in the arousal dimension is 83%, the accuracy in the dominance dimension is 78%, and the accuracy in the liking dimension is 72%. Figures 4 to 7 ​​​

[0067] It can be seen that the model proposed in the embodiment performs best in the alertness dimension under the combination, and the accuracy rate does not reach a high level in the other three dimensions. This shows that the feature combination has high applicability in arousal prediction, but the performance in the other dimensions is relatively general.

[0068] In terms of point features, when the frequency domain features are used as the point features, the accuracy rate is significantly higher than that of the other two point features, regardless of whether the edge features are distances or Pearson correlation coefficients. Therefore, the frequency domain features can be preferably selected as the point features to realize emotion recognition.

[0069] In order to verify the advancement of the method proposed in the application, the method proposed in the application is compared with other electroencephalogram emotion algorithms. The comparative algorithms are MLP algorithm and GRU algorithm.

[0070] Table 1 shows the results after comparison of several algorithms. The result of the MLP algorithm is relatively low because it needs a large amount of data for calculation, and the electroencephalogram dataset is relatively small and is prone to overfitting. The average recognition accuracy of our algorithm in the alertness, arousal, dominance and likeability dimensions of the DEAP dataset is 6.98%, 9.28% and 8.94% higher than that of GRU, respectively. Through the above comparison, it can be seen that the algorithm proposed has the best classification accuracy.

[0071] Table 1 Comparative algorithm results (Acc%) based on DEAP dataset , The embodiment of the application also provides an electroencephalogram emotion recognition system based on a deep neural network. Please refer to Figure 9 The system comprises: A node feature determination module 901 configured to construct an adaptive dynamic adjacency matrix by taking all electroencephalogram channels as nodes; and to take the time series electroencephalogram signals, statistical features of the electroencephalogram signals and / or frequency domain features of the electroencephalogram signals as node features; A graph feature construction module 902 configured to construct graph features from the nodes, the adaptive dynamic adjacency matrix and the node features; A graph feature processing module 903 configured to construct a GNN-LSTM-MSTF model, wherein the GNN-LSTM-MSTF model comprises a graph convolution network, a long short-term memory network and a multi-scale time-frequency feature fusion network, the GNN-LSTM-MSTF model responds to input graph features, extracts spatial features of the graph features by using the graph convolution network, extracts time features in combination with the long short-term memory network, and enhances modeling of instantaneous emotion changes in combination with the multi-scale time-frequency feature fusion, to obtain time coding representation and time-frequency fusion features of the entire sequence; The brain network feature extraction module 904 is configured to use a functional connectivity network to extract topological information of brain regions as brain network features. Feature concatenation module 905 is configured to concatenate brain network features with the output of the GNN-LSTM-MSTF model to form the final feature vector. ; The classification output module 906 is configured to output emotion recognition results based on the final feature vector using a multilayer perceptron classification model. The emotion recognition results are one or a combination of alertness, arousal, dominance, and liking.

[0072] It should be noted that the deep neural network-based EEG emotion recognition system and the previously described method belong to the same technical concept, have the same technical principles and beneficial effects, so they will not be repeated here.

[0073] The above embodiments are only used to illustrate the present invention and are not intended to limit the present invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, all equivalent technical solutions also fall within the scope of the present invention, and the patent protection scope of the present invention should be defined by the claims.

Claims

1. A deep neural network-based electroencephalogram emotion recognition method, characterized in that, The method comprises: all brain electrical channels are taken as nodes to construct an adaptive dynamic adjacency matrix; time series of the electroencephalogram signals, statistical characteristics of the electroencephalogram signals and / or frequency domain characteristics of the electroencephalogram signals are taken as node characteristics; the nodes, the adaptive dynamic adjacency matrix and the node characteristics are constructed into graph characteristics; a GNN-LSTM-MSTF model is constructed, the GNN-LSTM-MSTF model comprises a graph convolution network, a long short-term memory network and a multi-scale time-frequency feature fusion network, the GNN-LSTM-MSTF model is responsive to input graph characteristics, adopts the graph convolution network to extract spatial characteristics of the graph characteristics, combines the long short-term memory network to extract time characteristics, and combines the multi-scale time-frequency feature fusion to enhance modeling of instantaneous emotional changes, to obtain time coding representation and time-frequency fusion characteristics of the entire sequence; a functional connection network is adopted to extract topological structure information between brain regions as brain network characteristics; concatenate the brain network features with the output of the GNN-LSTM-MSTF model to form a final feature vector ; based on the final feature vector, a multi-layer perception classification model is adopted to output an emotional recognition result, the emotional recognition result is one of alertness, arousal, degree of being dominated, like degree and combinations thereof.

2. The method of claim 1, wherein, The GNN-LSTM-MSTF model is responsive to input graph characteristics, adopts the graph convolution network to extract spatial characteristics of the graph characteristics, combines the long short-term memory network to extract time characteristics, and combines the multi-scale time-frequency feature fusion to enhance modeling of instantaneous emotional changes, to obtain time coding representation and time-frequency fusion characteristics of the entire sequence in the following manner: The graph convolution operation is used for embedding learning of the constructed graph features, so as to capture the spatial dependence relationship between different EEG channels, and the graph features are denoted as wherein, denotes a node feature matrix, is a real set, N is a channel number, V is a node set, is an edge set, F is a feature dimension, and an adjacency matrix is The graph convolution calculation is shown in formula (1): （1） wherein, is the node feature matrix of the input of the layer, , is the node representation matrix of the output of the layer, is the learnable weight matrix of the layer, is the adjacency matrix after adding self-loops, is the degree matrix of , is initialized as the node feature, is the learnable weight of the layer, is an activation function. Each node embedding vector output by the graph convolution is taken as a time input to capture the dynamic dependencies between different time windows of the EEG signal; for the time input sequence The recursive form of the LSTM is （2） （3） （4） （5） （6） wherein, represents a temporal encoding representation of the entire sequence; represents the activation value of the input gate at time step , represents the input weight matrix of the input gate, represents the hidden state of the current time step, represents the hidden state of the previous time step, represents the bias term of the input gate, represents the input weight matrix of the forget gate, represents the hidden state weight matrix of the forget gate, represents the bias term of the forget gate, represents the activation value of the forget gate at time step , represents the activation value of the output gate at time step , represents the hidden state weight matrix of the output gate, represents the bias term of the output gate, represents the cell state of the current time step, represents the cell state of the previous time step, represents the input weight matrix of the candidate matrix, represents the hidden state weight matrix of the candidate state, represents the bias term of the candidate state; Acquiring time-frequency spectrum using STFT Capturing frequency-domain information by sliding window （7） wherein denotes the STFT result at time , frequency , denotes the value of the original time signal at time point , denotes a window function centered at , denotes the base of the natural logarithm, denotes the imaginary unit, denotes the frequency variable, denotes the integration variable, denoting a point on the entire time axis, denotes the current analyzed time position; short-time dynamic characteristics are extracted using continuous wavelet transform: （8） wherein denotes the result of the wavelet transform, denotes the local features of the signal at scale and time denotes the scale parameter, denotes the time position, denotes the value of the input signal at time ;​ Will And After convolutional encoding, with Fusion, get time-frequency fusion features .

3. The method of claim 1, wherein, the functional connection network is adopted to extract topological structure information between brain regions as brain network characteristics in the following manner: a functional connection network is constructed, and global topological characteristics are extracted from the graph characteristics as brain network characteristics by the following formula, the global topological characteristics comprise one of centrality, clustering coefficient and path length, wherein the calculation formula of the centrality is: （9） wherein, represents s to t the shortest path number, represents the path number through v , s , t , v represents a node in the graph.

4. The method of claim 1, wherein, concatenate the brain network features with the output of the GNN-LSTM-MSTF model to form a final feature vector : （10） In the formula, representing time coding features, representing the coding result after multi-scale fusion, representing brain network features; the fusion characteristics Z are input into the multi-layer perception to obtain an output emotional recognition result: （11） In the formula, an emotion recognition result vector representing a model prediction output, represents a multi-layer perceptron.

5. The method of claim 1, wherein, the adaptive dynamic adjacency matrix is constructed in the following manner: the spatial distance and signal similarity between the EEG channels are calculated; For any two EEG channels and The Euclidean distance is calculated using three-dimensional coordinates: （12） Forming a spatial matrix ,in, , , It is the first The coordinates of each channel in three-dimensional space, where N is the number of channels. For two EEG channels and Euclidean distance; The signal correlation between any two channels is measured using the Pearson correlation coefficient , forming a similarity matrix : （13） wherein denotes the covariance between channels and channels , denotes the standard deviation of channels , denotes the standard deviation of channels . According to the spatial distance and the signal similarity, an initial edge weight is defined is defined as: （14） wherein, is a weighting factor, is a small constant to prevent division by zero; Adopting self-adaptive weighting to dynamically adjust the edge weight of the adjacency matrix: let the adjacency matrix become a learnable parameter matrix Update by gradient descent during model training: （15） wherein, is the learning rate, is the overall recognition loss function, is the descending gradient.

6. The method of claim 1, wherein, the multi-scale time-frequency feature fusion method comprises: short-time Fourier transform is calculated to extract the steady-state frequency domain characteristics of the EEG signals; continuous wavelet transform is calculated to extract the short-time dynamic characteristics of the EEG signals; the above characteristics are combined through a feature fusion mechanism to enhance the recognition ability of emotional patterns of different time scales.

7. The method of claim 1, wherein, The nodes, the adjacency matrix and the node characteristics are constructed into graph characteristics in the following manner: the EEG channels are taken as nodes of a graph, each node corresponds to an EEG channel, and the node characteristics comprise time domain characteristics, frequency domain characteristics and wavelet transform characteristics; an adaptive dynamic adjacency matrix is adopted to dynamically adjust the edge weights of the adjacency matrix based on adaptive weighting; the adjacency matrix is optimized in combination with physical distance and correlation information; the adjacency matrix, the node characteristics and the brain network characteristics are integrated to obtain graph structure data.

8. The method of claim 1, wherein, The graph convolution network and the long short-term memory network are combined to form a graph convolution long short-term memory network, and the graph convolution long short-term memory network comprises a graph convolution layer, a long short-term memory network and a multilayer perceptron; wherein the graph convolution layer is used to extract brain network features in the graph features, the long short-term memory network is used to extract time-frequency features, and the multilayer perceptron is used to fuse the brain network features and the time-frequency features and output an emotion recognition result.

9. The method of claim 8, wherein, The graph convolution layer is arranged in three layers, each layer of the graph convolution layer comprises a convolution layer, a linear activation function layer and a random inactivation layer connected in sequence, and the outputs of the three layers of the graph convolution layer are aggregated by a global pooling layer and then input to the long short-term memory network.

10. A deep neural network-based electroencephalogram emotion recognition system, characterized by, The system comprises: A node feature determination module configured to take all electroencephalogram channels as nodes to construct an adaptive dynamic adjacency matrix; and take time series electroencephalogram signals, statistical features of the electroencephalogram signals and / or frequency domain features of the electroencephalogram signals as node features; A graph feature construction module configured to construct the nodes, the adaptive dynamic adjacency matrix and the node features into graph features; A graph feature processing module configured to construct a GNN-LSTM-MSTF model, wherein the GNN-LSTM-MSTF model comprises a graph convolution network, a long short-term memory network and a multi-scale time-frequency feature fusion network; the GNN-LSTM-MSTF model is responsive to input graph features, extracts spatial features of the graph features by using the graph convolution network, extracts time features in combination with the long short-term memory network, and combines the multi-scale time-frequency feature fusion to enhance modeling of instantaneous emotional changes, to obtain time coding representation and time-frequency fusion features of the entire sequence; A brain network feature extraction module configured to extract topological structure information between brain regions as brain network features by using a functional connection network; a feature concatenation module configured to concatenate the brain network features with the output of the GNN-LSTM-MSTF model to form a final feature vector ; A classification output module configured to output an emotion recognition result by using a multilayer perceptron classification model based on the final feature vector, wherein the emotion recognition result is one of alertness, arousal, degree of being dominated, and degree of liking, and combinations thereof.

Images