Emotion recognition method and device based on electroencephalogram signals, equipment and storage medium

Abstract

The present application relates to the technical field of signal processing, and particularly relates to an emotion recognition method, device and equipment based on electroencephalogram signals and a storage medium, wherein the electroencephalogram signals of a user to be recognized are subjected to multi-dimensional feature extraction, the feature information associated with emotion recognition in the electroencephalogram signals is fully considered, the signal feature data of the multiple dimensions extracted is subjected to feature fusion, electroencephalogram signal feature fusion data is obtained, the key feature information in the electroencephalogram information feature fusion data is extracted by combining a graph embedding method and an attention extraction method, and is used for emotion recognition, thereby improving the accuracy of emotion recognition of the electroencephalogram signals.

Description

Technical Field

[0001] This invention relates to the field of signal processing technology, and in particular to a method, apparatus, device, and storage medium for emotion recognition based on electroencephalogram (EEG) signals. Background Technology

[0002] Emotions play a vital role in people's lives, influencing our thinking and judgment. Brain-computer interfaces (BCIs) are a new research area in emotion recognition. Affective BCIs are a type of BCI used to identify or regulate human emotions. Electroencephalography (EEG) is the most widely used physiological signal in affective BCIs. Research in cognitive science and clinical medicine has found that brain electrical activity is closely related to emotion and cognition. For example, the alpha and theta bands in EEG are associated with mental exertion or memory load. Similarly, related clinical studies have found that asymmetry in resting-state prefrontal cortex alpha band EEG activity is associated with the degree of depression. Based on these physiological phenomena and EEG signal characteristics, researchers can use signal processing techniques and deep learning algorithms to analyze EEG signals, extract features related to emotion and cognition, and thus detect a person's internal state.

[0003] Because there are significant differences in how people express emotions, and the EEG data representing different people's emotions also have specificities, graph convolutional neural networks are currently commonly used for emotion recognition from EEG signals. However, traditional graph convolutional neural networks are relatively simple, lack flexibility, and contain a lot of redundant information, making it difficult to perform accurate and efficient emotion recognition from EEG signals. Summary of the Invention

[0004] Based on this, the purpose of this invention is to provide a method, apparatus, device, and storage medium for emotion recognition based on electroencephalogram (EEG) signals. By extracting features from the EEG signals of the user to be identified in multiple dimensions, the invention fully considers the feature information in the EEG signals that is related to emotion recognition. The extracted multi-dimensional signal feature data is then fused to obtain EEG signal feature fusion data. Combined with graph embedding and attention extraction methods, key feature information in the EEG information feature fusion data is extracted for emotion recognition, thereby improving the accuracy of emotion recognition based on EEG signals.

[0005] In a first aspect, embodiments of this application provide an emotion recognition method based on electroencephalogram (EEG) signals, comprising the following steps:

[0006] The system obtains the EEG signals of the user to be identified and a preset emotion recognition model, wherein the emotion recognition model includes a feature extraction module, a feature fusion module, a graph attention module, and a recognition module.

[0007] The EEG signal is input into the feature extraction module, and features are extracted from the EEG signal to obtain EEG signal feature extraction data, wherein the EEG signal feature extraction data includes several types of signal feature data;

[0008] The extracted EEG signal features are input into the feature fusion module, and feature fusion is performed on several types of signal feature data to obtain EEG signal feature fusion data.

[0009] The fused EEG signal data is input into the graph attention module to extract graph attention features and obtain graph attention feature data.

[0010] The graph attention feature data is input into the recognition module for emotion recognition to obtain predicted emotion recognition probability data. Based on the predicted emotion recognition probability data, the emotion polarity corresponding to the predicted emotion recognition probability vector with the largest dimension is obtained, which is used as the emotion recognition result of the user to be identified.

[0011] Secondly, embodiments of this application provide an emotion recognition device based on electroencephalogram (EEG) signals, comprising:

[0012] The data acquisition module is used to acquire the EEG signals of the user to be identified and a preset emotion recognition model, wherein the emotion recognition model includes a feature extraction module, a feature fusion module, a graph attention module, and a recognition module;

[0013] The EEG signal feature extraction module is used to input the EEG signal into the feature extraction module, extract features from the EEG signal, and obtain EEG signal feature extraction data, wherein the EEG signal feature extraction data includes several types of signal feature data;

[0014] The EEG signal feature fusion module is used to input the extracted EEG signal feature data into the feature fusion module, perform feature fusion on several types of signal feature data, and obtain EEG signal feature fusion data.

[0015] The EEG signal graph attention feature extraction module is used to input the EEG signal feature fusion data into the graph attention module to extract graph attention features and obtain graph attention feature data.

[0016] The emotion recognition module is used to input the graph attention feature data into the recognition module for emotion recognition, obtain predicted emotion recognition probability data, and obtain the emotion polarity corresponding to the predicted emotion recognition probability vector with the largest dimension based on the predicted emotion recognition probability data, which is used as the emotion recognition result of the user to be identified.

[0017] Thirdly, embodiments of this application provide a computer device including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the emotion recognition method based on electroencephalogram signals as described in the first aspect.

[0018] Fourthly, embodiments of this application provide a storage medium storing a computer program that, when executed by a processor, implements the steps of the emotion recognition method based on electroencephalogram (EEG) signals as described in the first aspect.

[0019] In this application embodiment, an emotion recognition method, device, equipment, and storage medium based on electroencephalogram (EEG) signals are provided. By extracting features from the EEG signals of the user to be identified in multiple dimensions, the feature information in the EEG signals associated with emotion recognition is fully considered. The extracted multi-dimensional signal feature data is fused to obtain EEG signal feature fusion data. Combined with graph embedding and attention extraction methods, key feature information in the EEG information feature fusion data is extracted for emotion recognition, thereby improving the accuracy of emotion recognition based on EEG signals.

[0020] To better understand and implement this invention, the following detailed description is provided in conjunction with the accompanying drawings. Attached Figure Description

[0021] Figure 1 A flowchart illustrating an emotion recognition method based on electroencephalogram (EEG) signals provided in one embodiment of this application;

[0022] Figure 2 This is a flowchart illustrating step S2 of an emotion recognition method based on electroencephalogram (EEG) signals provided in one embodiment of this application.

[0023] Figure 3 This is a flowchart illustrating step S2 of an emotion recognition method based on electroencephalogram (EEG) signals provided in one embodiment of this application.

[0024] Figure 4 This is a flowchart illustrating step S3 of an emotion recognition method based on electroencephalogram (EEG) signals provided in one embodiment of this application.

[0025] Figure 5 This is a flowchart illustrating step S4 of an emotion recognition method based on electroencephalogram (EEG) signals provided in one embodiment of this application.

[0026] Figure 6 This is a flowchart illustrating step S42 of an emotion recognition method based on electroencephalogram (EEG) signals provided in one embodiment of this application.

[0027] Figure 7This is a flowchart illustrating step S43 of an emotion recognition method based on electroencephalogram (EEG) signals provided in one embodiment of this application.

[0028] Figure 8 A flowchart illustrating step S6 of an emotion recognition method based on electroencephalogram (EEG) signals provided in another embodiment of this application;

[0029] Figure 9 A schematic diagram of the structure of an emotion recognition device based on electroencephalogram (EEG) signals provided in one embodiment of this application;

[0030] Figure 10 This is a schematic diagram of the structure of a computer device provided in one embodiment of this application. Detailed Implementation

[0031] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0032] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0033] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0034] Please see Figure 1 , Figure 1 The flowchart illustrates an emotion recognition method based on electroencephalogram (EEG) signals according to an embodiment of this application. The method includes the following steps:

[0035] S1: Obtain the EEG signals of the user to be identified and the preset emotion recognition model.

[0036] The execution subject of the emotion recognition method based on EEG signals is the recognition device (hereinafter referred to as the recognition device). In an optional embodiment, the recognition device may be a computer device, a server, or a server cluster composed of multiple computer devices.

[0037] In this embodiment, the identification device can be an EEG interface device, which collects the EEG signals of the user to be identified when the user receives external stimuli, such as videos, pictures, and music.

[0038] In an optional embodiment, after the EEG signal is recorded at a sampling rate of 1000 Hz, the recognition device filters it using a bandpass filter of 0-50 Hz and resamples the EEG signal to 200 Hz to accelerate the efficiency of emotion recognition. Additionally, the recognition device can apply a linear dynamic system method to preprocess the EEG signal. This preprocessing includes removing noise and interference such as electrooculogram (EOG) artifacts, electromyogram (EMG) artifacts, electrocardiogram (ECG) artifacts, electrodermal activity interference, and power line interference. Finally, to enhance the features associated with the stimulus material in the EEG signal, the recognition device uses amplitude truncation to limit the amplitude of the EEG signal to within ±40 μV. It iterates through the data of each channel of the EEG signal, setting values ​​with amplitudes exceeding ±40 μV to +40 μV or -40 μV respectively, removing extreme values, reducing the negative impact of noise or artifacts, and improving the signal-to-noise ratio of the EEG signal, thereby more accurately extracting the feature information associated with the stimulus material from the EEG signal.

[0039] The recognition device obtains a preset emotion recognition model, wherein the emotion recognition model includes a feature extraction module, a feature fusion module, a graph attention module, and a recognition module.

[0040] S2: Input the EEG signal into the feature extraction module, perform feature extraction on the EEG signal, and obtain EEG signal feature extraction data.

[0041] Because the EEG signals induced by the stimulating material have high temporal and frequency complexity, in order to capture their multidimensional characteristics and fully extract the emotion-related feature information in the EEG signals, in this embodiment, the recognition device inputs the EEG signals into the feature extraction module to extract features from the EEG signals and obtain EEG signal feature extraction data, wherein the EEG signal feature extraction data includes several types of signal feature data.

[0042] The signal feature data includes differential entropy feature data, which has good discriminative ability for emotion recognition; please refer to [link to relevant documentation]. Figure 2 , Figure 2The flowchart of S2 in the emotion recognition method based on electroencephalogram (EEG) signals provided in one embodiment of this application is as follows: S201 to S202 are detailed below:

[0043] S201: The EEG signal is filtered using a filtering method to obtain EEG signals in several frequency bands. The EEG signals in several frequency bands are then divided to obtain several first EEG unit signals of the EEG signals in several frequency bands.

[0044] In this embodiment, the recognition device uses a filtering method to filter the EEG signal according to a preset filter to obtain EEG signals in several frequency bands, including Delta (1-3Hz), Theta (4-7Hz), Alpha (8-13Hz), Beta (14-30Hz), and Gamma (31-50Hz).

[0045] The recognition device divides the EEG signals into several frequency bands according to a preset first signal length, obtaining several first EEG unit signals of the several frequency bands. The first EEG unit signal includes EEG data from several channels.

[0046] S202: Perform differential entropy feature extraction on the EEG data of several channels of the first EEG unit signal to obtain differential entropy feature matrices of several channels of several channels of the first EEG unit signal in several frequency bands, and construct the differential entropy feature data.

[0047] In this embodiment, the identification device performs differential entropy feature extraction on the EEG data of several channels of the first EEG unit signal to obtain differential entropy feature matrices of several channels of several channels of the first EEG unit signal in several frequency bands, and constructs the differential entropy feature data.

[0048] The signal feature data includes time-domain feature data and frequency-domain feature data, which reflect the time, frequency, and spatial information of the EEG signal; please refer to [link to relevant documentation]. Figure 3 , Figure 3 The flowchart of S2 in the emotion recognition method based on electroencephalogram (EEG) signals provided in one embodiment of this application includes steps S211 to S212, as follows:

[0049] S211: Divide the EEG signal to obtain several second EEG unit signals. Based on the EEG data of several channels in the second EEG unit signals, use wavelet transform to obtain wavelet coefficients of several channels of the several second EEG unit signals. Combine the wavelet coefficients of several channels of the same second EEG unit signal to construct a wavelet coefficient matrix of several second EEG unit signals.

[0050] In this embodiment, the recognition device divides the EEG signal according to a preset second signal length to obtain several second EEG unit signals, wherein the second EEG unit signals include EEG data from several channels. In an optional embodiment, the second signal length is the same as the first signal length.

[0051] The recognition device uses wavelet transform to obtain wavelet coefficients for several channels of the second EEG unit signal based on the EEG data of several channels in the second EEG unit signal. Specifically, the recognition device uses the Daubechies wavelet basis because it has good time and frequency resolution. The recognition device uses the Daubechies wavelet basis to decompose the EEG data of several channels in the second EEG unit signal to level 5 to obtain wavelet coefficients for several channels of the second EEG unit signal. The wavelet coefficients include low-frequency approximation coefficients and high-frequency detail coefficients.

[0052] The recognition device combines the wavelet coefficients of several channels of the same second EEG unit signal to construct several wavelet coefficient matrices of the second EEG unit signal.

[0053] S212: Perform independent component analysis on the wavelet coefficient matrix to obtain independent component matrices of several second EEG unit signals, extract time-domain and frequency-domain features from the independent component matrices to obtain time-domain feature matrices and frequency-domain feature matrices of several second EEG unit signals, and construct the time-domain feature data and frequency-domain feature data respectively.

[0054] In this embodiment, the identification device performs independent component analysis on the wavelet coefficient matrix to obtain independent component matrices of several second EEG unit signals. Specifically, the identification device preprocesses the wavelet coefficient matrix to obtain a preprocessed wavelet coefficient matrix. The preprocessing steps include centering and whitening to remove the mean from the wavelet coefficient matrix and to ensure that the correlation between the variables in the wavelet coefficient matrix is ​​0 and the variance of each variable is 1, thus ensuring that the data conforms to the assumptions of independent component analysis.

[0055] The recognition device employs Fast Independent Component Analysis (Fast-ICA), which effectively separates independent signal sources. It performs independent component analysis on the preprocessed wavelet coefficient matrix to obtain several independent components of the preprocessed wavelet coefficient matrix. These independent components are then combined to obtain several independent component matrices of the second EEG unit signals. Each independent component represents a spatial pattern, reflecting independent activity across several channels of the EEG unit signals.

[0056] The recognition device extracts time-domain and frequency-domain features from the independent component matrix to obtain time-domain and frequency-domain feature matrices of several second EEG unit signals. The time-domain feature matrices of the several second EEG unit signals are combined to construct the time-domain feature data, and the frequency-domain feature matrices of the several second EEG unit signals are combined to construct the frequency-domain feature data.

[0057] S3: Input the extracted EEG signal feature data into the feature fusion module, perform feature fusion on several types of signal feature data, and obtain EEG signal feature fusion data.

[0058] In this embodiment, the recognition device inputs the EEG signal feature extraction data into the feature fusion module, performs feature fusion on several types of signal feature data, obtains EEG signal feature fusion data, effectively extracts multi-dimensional features from EEG signals, and improves the accuracy and reliability of emotion recognition.

[0059] Please see Figure 4 , Figure 4 The flowchart of step S3 in the emotion recognition method based on electroencephalogram (EEG) signals provided in one embodiment of this application is as follows:

[0060] S31: Based on the differential entropy feature data, time domain feature data, frequency domain feature data, and a preset feature fusion algorithm, obtain the EEG signal feature fusion matrix, which serves as the EEG signal feature fusion data.

[0061] The feature fusion algorithm is as follows:

[0062] F DWI =LDA([DE(X) c,b (t))||F TD (X′ c (t))||F FD (X′ c (t))])

[0063] In the formula, F DWI For the EEG signal feature fusion matrix, DE(X)c,b (t) is X c,b The characteristic matrix of the differential entropy of (t), X c,b (t) represents the EEG data of the c-th channel in the t-th first EEG unit signal of the b-th frequency band, F TD (X′ c (t) is X′ c The time-domain characteristic matrix of F(t), FD (X′ c (t) is X′ c The frequency domain characteristic matrix of (t), X′ c (t) represents the EEG data of the c-th channel in the t-th second EEG unit signal, and || is the splicing symbol, which is used to indicate that the feature matrix is ​​spliced ​​on the last dimension.

[0064] In this embodiment, the recognition device, based on the differential entropy feature data, time-domain feature data, and frequency-domain feature data, and a preset feature fusion algorithm, performs a concatenation process on the differential entropy feature matrices of several channels in the differential entropy feature data, the time-domain feature matrices of several channels in the time-domain feature data, and the frequency-domain feature matrices of several channels in the frequency-domain feature data, based on the first EEG unit signal and the second EEG unit signal with the same index, to obtain the concatenated result.

[0065] Furthermore, since emotional changes are relatively gradual processes, while the extracted EEG features often exhibit dramatic changes, the recognition device employs Linear Discriminant Analysis (LDA) to reduce or even eliminate the influence of EEG signals unrelated to emotion. This LDA process reduces the dimensionality of the spliced ​​results to one dimension, retaining the main discriminative features, reducing computational complexity, and improving classification performance. This yields the EEG signal feature fusion matrix, fully utilizing the temporal, frequency, and spatial information of the extracted EEG signals. This overcomes the lack of relevant features in the differential entropy features of the extracted EEG signals, improving the comprehensiveness and accuracy of feature extraction, thereby enhancing the accuracy of emotion recognition from EEG signals.

[0066] S4: Input the fused EEG signal feature data into the graph attention module to extract graph attention features and obtain graph attention feature data.

[0067] In this embodiment, the recognition device inputs the fused data of the EEG signal features into the graph attention module to extract graph attention features and obtain graph attention feature data. The graph attention module includes a convolutional pooling network, a graph organization network, and a graph attention extraction network. The convolutional pooling network includes several sequentially connected convolutional pooling layers.

[0068] Please see Figure 5 , Figure 5 The flowchart of S4 in the emotion recognition method based on electroencephalogram (EEG) signals provided in one embodiment of this application includes steps S41 to S43, as follows:

[0069] S41: The fused EEG signal feature data is used as the input data of the first convolutional pooling layer of the convolutional pooling network, and convolutional pooling processing is performed. The output data of the first convolutional pooling layer is used as the input data of the next convolutional pooling layer. The convolutional pooling processing is repeated to obtain several convolutional pooling feature data output by the convolutional pooling layers.

[0070] The convolutional pooling layer structure is a three-layer network, each consisting of a convolutional layer and a pooling layer, with ReLU as the activation function in the convolutional kernel. The first convolutional layer consists of a 5×5×32 kernel, and the first pooling layer is a 1×4 pooling layer to reduce feature dimensionality. The second convolutional layer consists of a 1×5×64 kernel, and the second pooling layer is a 1×2 pooling layer. The third convolutional layer consists of a 1×5×128 kernel, and the second pooling layer is a 1×2 pooling layer. Each layer progressively extracts deeper feature information. To prevent overfitting, a dropout rate of 0.1 is applied after the convolutional layers.

[0071] In this embodiment, the recognition device uses the fused data of the EEG signal features as the input data of the first convolutional pooling layer of the convolutional pooling network, performs convolutional pooling processing, uses the output data of the first convolutional pooling layer as the input data of the next convolutional pooling layer, and repeats the convolutional pooling processing to obtain several convolutional pooling feature data output by the convolutional pooling layers, so as to further extract the depth spatial information of the EEG signal.

[0072] S42: Convert the convolutional pooling feature data into feature vector matrices to obtain several feature vector matrices corresponding to the convolutional pooling feature data. Input the several feature vector matrices into the graph organization network for graph embedding to obtain several edge set matrices corresponding to the feature vector matrices.

[0073] In this embodiment, the recognition device converts the convolutional pooling feature data into feature vector matrices to obtain several feature vector matrices corresponding to the convolutional pooling feature data. Specifically, the feature vector matrices are two-dimensional matrices used for graph embedding.

[0074] The recognition device inputs several feature vector matrices into the graph organization network for graph embedding to obtain edge set matrices corresponding to several feature vector matrices, wherein the edge set matrix is ​​used to represent the adjacency relationship between each channel of the EEG signal.

[0075] Please see Figure 6 , Figure 6 The flowchart of S42 in the emotion recognition method based on electroencephalogram (EEG) signals provided in one embodiment of this application is as follows: S421 to S422 are detailed below:

[0076] S421: Based on the aforementioned feature vector matrices and a preset weight matrix calculation algorithm, obtain several weight feature matrices.

[0077] The algorithm for calculating the weight feature matrix is ​​as follows:

[0078] W = Tanh(Linear(V))

[0079] In the formula, W is the weight eigenma matrix, V is the eigenvector matrix, Tanh(·) is the hyperbolic tangent function, and Linear(·) is the linear fully connected function.

[0080] In this embodiment, the identification device calculates a number of weight feature matrices based on a number of feature vector matrices and a preset weight matrix calculation algorithm. By processing the feature vector matrices through a linear fully connected function to reduce data computation, and then regularizing the data after processing through the linear fully connected function through a hyperbolic tangent function, a number of weight feature matrices are obtained. The weight matrix includes the weight feature vectors of a number of nodes.

[0081] S422: Based on several weight feature matrices and a preset adjacency matrix calculation algorithm, several adjacency matrices are obtained; based on several adjacency matrices and a preset edge set matrix calculation algorithm, several edge set matrices are obtained.

[0082] In this embodiment, the identification device obtains several adjacency matrices based on several weight feature matrices and a preset adjacency matrix calculation algorithm. Each adjacency matrix includes connection weight vectors between several nodes. The adjacency matrix calculation algorithm is as follows:

[0083]

[0084] In the formula, a ij W is the connection weight vector between the i-th node and the j-th node. i Let W be the weight eigenvector of the i-th node. j Let N be the weight eigenvector of the j-th node, N be the number of nodes, and T be the transpose sign.

[0085] The identification device, based on several adjacency matrices and a preset edge set matrix calculation algorithm, selects the maximum connection weight vector between each node and all other nodes in the adjacency matrix, takes the connection between the two nodes associated with the maximum connection weight vector as an edge, obtains several edges corresponding to several adjacency matrices, and combines several edges corresponding to the same adjacency matrix to obtain several edge set matrices. The edge set matrix calculation algorithm is as follows:

[0086] E = (topk(A)) T

[0087] In the formula, E is the edge set matrix, A is the adjacency matrix, and topk(·) is the index function for finding the maximum and minimum values.

[0088] S43: Input several feature vector matrices and their corresponding edge set matrices into the graph attention extraction network to extract attention features and obtain the graph attention feature data.

[0089] The graph attention extraction network described is a graph neural network that incorporates a self-supervised task and a self-attention mechanism. The self-attention mechanism allows each node to dynamically adjust its weights based on its own characteristics and relationships with other nodes. This enables the network to pay more attention to neighboring nodes associated with the current node when processing EEG signals, while reducing its sensitivity to noise interference. The self-supervised task is used to predict whether an edge exists between two nodes. By displaying and encoding information in the graph neural network, it can better understand the importance of each edge, thereby reducing its sensitivity to noise during the learning process.

[0090] In this embodiment, the recognition device inputs several feature vector matrices and corresponding edge set matrices into the graph attention extraction network to extract attention features and obtain graph attention feature data. The attention extraction network includes several graph attention neural units corresponding to the number of edges in the edge set matrix.

[0091] Please see Figure 7 , Figure 7 The flowchart of S43 in the emotion recognition method based on electroencephalogram (EEG) signals provided in one embodiment of this application is as follows:

[0092] S431: The feature vector matrices and their corresponding edge set matrices are used as input data for several graph attention neural units corresponding to the number of edges in the edge set matrix. According to the preset attention feature calculation algorithm, the graph attention feature matrices output by the graph attention neural units are obtained. The graph attention feature matrices are concatenated to obtain the graph attention feature data.

[0093] The attention feature calculation algorithm is as follows:

[0094]

[0095] In the formula, S l This is the graph attention feature matrix corresponding to the l-th feature vector matrix. Let GAT be the number of edges in the edge set matrix, SoftMax(·) be the normalization exponential function, and GAT be the value of GAT. j (·) represents the graph attention neural unit corresponding to the j-th edge in the edge set matrix.

[0096] In this embodiment, the recognition device uses several feature vector matrices and their corresponding edge set matrices as input data for several graph attention neural units corresponding to the number of edges in the edge set matrix, and obtains graph attention feature matrices output by several graph attention neural units according to a preset attention feature calculation algorithm.

[0097] The recognition device uses a linear connection method to concatenate the last dimension of several graph attention feature matrices to obtain a concatenation result, which is used as the graph attention feature data.

[0098] S5: Input the graph attention feature data into the recognition module for emotion recognition, obtain predicted emotion recognition probability data, and obtain the emotion polarity corresponding to the predicted emotion recognition probability vector with the largest dimension based on the predicted emotion recognition probability data, which is used as the emotion recognition result of the user to be identified.

[0099] The recognition module includes a fully connected layer, which includes several units corresponding to emotional polarity;

[0100] In this embodiment, the recognition device inputs the graph attention feature data into the recognition module for emotion recognition, and obtains the predicted emotion recognition probability vectors output by several units of the fully connected layer corresponding to the emotion polarity. These predicted emotion recognition probability vectors are used as several dimensions to construct the predicted emotion recognition probability data.

[0101] The recognition device obtains the emotion polarity corresponding to the largest-dimensional predicted emotion recognition probability vector based on the predicted emotion recognition probability data, and uses this as the emotion recognition result for the user to be identified. By extracting features from the user's EEG signals in multiple dimensions, fully considering the feature information in the EEG signals associated with emotion recognition, the extracted multi-dimensional signal feature data is fused to obtain EEG signal feature fusion data. Combined with graph embedding and attention extraction methods, key feature information in the EEG information feature fusion data is extracted for emotion recognition, thus improving the accuracy of emotion recognition based on EEG signals.

[0102] In an optional embodiment, step S6 is further included: training the emotion recognition model, see [link to relevant documentation]. Figure 8 , Figure 8 A flowchart illustrating step S6 of an emotion recognition method based on electroencephalogram (EEG) signals, provided in another embodiment of this application, includes steps S61 to S62, as detailed below:

[0103] S61: Obtain the EEG signal set of several sample users, input the EEG signal set of several sample users into the emotion recognition model, and obtain the predicted emotion recognition probability data of several sample users.

[0104] In this embodiment, the recognition device obtains a set of EEG signals from several sample users, inputs the set of EEG signals from several sample users into the emotion recognition model, and obtains predicted emotion recognition probability data for several sample users.

[0105] S62: Obtain real emotion recognition probability data of several sample users, obtain loss values ​​based on real emotion recognition probability vectors of several dimensions in the real emotion recognition probability data of several sample users, predicted emotion recognition probability vectors of several dimensions in the predicted emotion recognition probability data, and a preset loss function, and train the emotion recognition model based on the loss values.

[0106] The loss function is:

[0107]

[0108] In the formula, L is the loss value, I is the number of sample users, k is the number of dimensions, and y ik Let p be the true emotion recognition probability vector of the k-th dimension of the i-th sample user. ik Let be the predicted emotion recognition probability vector of the k-th dimension for the i-th sample user.

[0109] In this embodiment, the recognition device obtains real emotion recognition probability data from several sample users. Based on the real emotion recognition probability vectors of several dimensions in the real emotion recognition probability data of several sample users, the predicted emotion recognition probability vectors of several dimensions in the predicted emotion recognition probability data, and a preset loss function, a loss value is obtained. Based on the loss value, the emotion recognition model is trained, so that the emotion recognition model can reduce the negative impact of noise, learn the emotion-related features in the input EEG signal more effectively, and improve the accuracy and efficiency of emotion recognition.

[0110] Please refer to Figure 9 , Figure 9 This is a schematic diagram of the structure of an emotion recognition device based on electroencephalogram (EEG) signals according to an embodiment of this application. The device can be implemented in whole or in part through software, hardware, or a combination of both. The device 9 includes:

[0111] The data acquisition module 91 is used to acquire the EEG signal of the user to be identified and the preset emotion recognition model, wherein the emotion recognition model includes a feature extraction module, a feature fusion module, a graph attention module and a recognition module;

[0112] The EEG signal feature extraction module 92 is used to input the EEG signal into the feature extraction module, extract features from the EEG signal, and obtain EEG signal feature extraction data, wherein the EEG signal feature extraction data includes several types of signal feature data;

[0113] The EEG signal feature fusion module 93 is used to input the EEG signal feature extraction data into the feature fusion module, perform feature fusion on several types of signal feature data, and obtain EEG signal feature fusion data;

[0114] The EEG signal graph attention feature extraction module 94 is used to input the EEG signal feature fusion data into the graph attention module to extract graph attention features and obtain graph attention feature data.

[0115] The emotion recognition module 95 is used to input the graph attention feature data into the recognition module for emotion recognition, obtain predicted emotion recognition probability data, and obtain the emotion polarity corresponding to the predicted emotion recognition probability vector with the largest dimension based on the predicted emotion recognition probability data, which is used as the emotion recognition result of the user to be identified.

[0116] In this embodiment, a data acquisition module obtains the EEG signal of the user to be identified and a preset emotion recognition model. The emotion recognition model includes a feature extraction module, a feature fusion module, a graph attention module, and a recognition module. The EEG signal feature extraction module inputs the EEG signal into the feature extraction module to extract features from the EEG signal, obtaining EEG signal feature extraction data. The EEG signal feature extraction data includes several types of signal feature data. The EEG signal feature fusion module inputs the EEG signal feature extraction data into the feature fusion module to fuse the several types of signal feature data, obtaining EEG signal feature fusion data. The EEG signal graph attention feature extraction module inputs the EEG signal feature fusion data into the graph attention module to extract graph attention features, obtaining graph attention feature data. The emotion recognition module inputs the graph attention feature data into the recognition module to perform emotion recognition, obtaining predicted emotion recognition probability data. Based on the predicted emotion recognition probability data, the emotion polarity corresponding to the predicted emotion recognition probability vector with the largest dimension is obtained, serving as the emotion recognition result for the user to be identified. By extracting features from the EEG signals of the user to be identified in multiple dimensions, and fully considering the feature information in the EEG signals that is related to emotion recognition, feature fusion is performed on the extracted signal feature data in multiple dimensions to obtain EEG signal feature fusion data. Combined with graph embedding and attention extraction methods, key feature information in the EEG information feature fusion data is extracted for emotion recognition, thereby improving the accuracy of emotion recognition of EEG signals.

[0117] Please refer to Figure 10 , Figure 10 This is a schematic diagram of the structure of a computer device provided in one embodiment of this application. The computer device 10 includes: a processor 101, a memory 102, and a computer program 103 stored in the memory 102 and executable on the processor 101. The computer device can store multiple instructions, which are adapted to be loaded and executed by the processor 101. Figures 1 to 8 The method steps of the illustrated embodiment can be found in the following documentation for detailed execution. Figures 1 to 8 The specific details of the illustrated embodiments will not be elaborated here.

[0118] The processor 101 may include one or more processing cores. The processor 101 connects to various parts of the server using various interfaces and lines, and executes various functions and processes data of the EEG-based emotion recognition device 9 by running or executing instructions, programs, code sets, or instruction sets stored in the memory 102, and by calling data stored in the memory 102. Optionally, the processor 101 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 101 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required to be displayed on the touch screen; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 101 and may be implemented as a separate chip.

[0119] The memory 102 may include random access memory (RAM) or read-only memory. Optionally, the memory 102 may include a non-transitory computer-readable storage medium. The memory 102 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 102 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch instructions), instructions for implementing the various method embodiments described above, etc.; the data storage area may store data involved in the various method embodiments described above, etc. Optionally, the memory 102 may also be at least one storage device located remotely from the aforementioned processor 101.

[0120] This application embodiment also provides a storage medium that can store multiple instructions, which are adapted to be loaded and executed by a processor as described above. Figures 1 to 8 For details of the method steps described in the embodiment, please refer to [link / reference]. Figures 1 to 8 The specific details of the embodiments will not be elaborated here.

[0121] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0122] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0123] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the algorithm. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0124] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / terminal devices and methods can be implemented in other ways. For example, the apparatus / terminal device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0125] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0126] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0127] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms.

[0128] This invention is not limited to the above-described embodiments. If any modifications or variations to this invention do not depart from the spirit and scope of this invention, and if such modifications and variations fall within the scope of the claims and equivalent technologies of this invention, then this invention also intends to include such modifications and variations.

Claims

1. An emotion recognition method based on electroencephalogram (EEG) signals, characterized in that, Includes the following steps: The system obtains the EEG signal of the user to be identified and a preset emotion recognition model. The emotion recognition model includes a feature extraction module, a feature fusion module, a graph attention module, and a recognition module. The graph attention module includes a convolutional pooling network, a graph organization network, and a graph attention extraction network. The convolutional pooling network includes several sequentially connected convolutional pooling layers. The EEG signal is input into the feature extraction module, and features are extracted from the EEG signal to obtain EEG signal feature extraction data, wherein the EEG signal feature extraction data includes several types of signal feature data; The extracted EEG signal features are input into the feature fusion module, and feature fusion is performed on several types of signal feature data to obtain EEG signal feature fusion data. The fused EEG signal feature data is input into the graph attention module. Based on the fused EEG signal feature data, fused EEG signal feature data is constructed. The fused EEG signal feature data is used as the input data of the first convolutional pooling layer of the convolutional pooling network for convolutional pooling processing. The output data of the first convolutional pooling layer is used as the input data of the next convolutional pooling layer. The convolutional pooling processing is repeated to obtain several convolutional pooling feature data output by the convolutional pooling layers. The convolutional pooling feature data is converted into feature vector matrices to obtain several feature vector matrices corresponding to the convolutional pooling feature data. These feature vector matrices are then input into the graph organization network. Based on the feature vector matrices and a preset weight matrix calculation algorithm, several weight feature matrices are obtained. Each weight matrix includes the weight feature vectors of several nodes. The weight feature matrix calculation algorithm is as follows: In the formula, The weight eigenma matrix, The eigenvector matrix, It is the hyperbolic tangent function. It is a linear fully connected function; Based on several weight feature matrices and a preset adjacency matrix calculation algorithm, several adjacency matrices are obtained. Based on several adjacency matrices and a preset edge set matrix calculation algorithm, edge set matrices corresponding to several feature vector matrices are obtained. The adjacency matrix includes connection weight vectors between several nodes. The adjacency matrix calculation algorithm is as follows: In the formula, For the first i The node and the first j The connection weight vector between nodes. For the first i The weight eigenvectors of each node. For the first j The weight eigenvectors of each node. For the number of nodes, T It is the transpose symbol; The algorithm for calculating the edge set matrix is ​​as follows: In the formula, E Let be the edge set matrix. A It is an adjacency matrix. This is an index function for finding the maximum and minimum values; Several feature vector matrices and their corresponding edge set matrices are respectively input into the graph attention extraction network to extract attention features and obtain graph attention feature data; The graph attention feature data is input into the recognition module for emotion recognition to obtain predicted emotion recognition probability data. Based on the predicted emotion recognition probability data, the emotion polarity corresponding to the predicted emotion recognition probability vector with the largest dimension is obtained, which is used as the emotion recognition result of the user to be identified.

2. The emotion recognition method based on electroencephalogram (EEG) signals according to claim 1, characterized in that: The signal feature data includes differential entropy feature data; The step of inputting the EEG signal into the feature extraction module, performing feature extraction on the EEG signal, and obtaining EEG signal feature extraction data includes the following steps: The EEG signal is filtered using a filtering method to obtain EEG signals in several frequency bands. The EEG signals in several frequency bands are then divided to obtain several first EEG unit signals of the EEG signals in several frequency bands. The first EEG unit signal includes EEG data from several channels. Differential entropy features are extracted from the EEG data of several channels of the first EEG unit signal to obtain differential entropy feature matrices of several channels of several channels of the first EEG unit signal in several frequency bands, and the differential entropy feature data is constructed.

3. The emotion recognition method based on electroencephalogram (EEG) signals according to claim 2, characterized in that: The signal feature data includes time-domain feature data and frequency-domain feature data; The step of inputting the EEG signal into the feature extraction module, performing feature extraction on the EEG signal, and obtaining EEG signal feature extraction data includes the following steps: The EEG signal is divided to obtain several second EEG unit signals. Based on the EEG data of several channels in the second EEG unit signals, wavelet transform is used to obtain wavelet coefficients of several channels of several second EEG unit signals. The wavelet coefficients of several channels of the same second EEG unit signal are combined to construct a wavelet coefficient matrix of several second EEG unit signals. Independent component analysis is performed on the wavelet coefficient matrix to obtain independent component matrices of several second EEG unit signals. Time-domain and frequency-domain features are extracted from the independent component matrices to obtain time-domain and frequency-domain feature matrices of several second EEG unit signals. The time-domain feature data and frequency-domain feature data are constructed respectively.

4. The emotion recognition method based on electroencephalogram (EEG) signals according to claim 3, characterized in that, The step of inputting the extracted EEG signal feature data into the feature fusion module, performing feature fusion on several types of signal feature data to obtain EEG signal feature fusion data includes the following steps: Based on the differential entropy feature data, time-domain feature data, and frequency-domain feature data, and a preset feature fusion algorithm, an EEG signal feature fusion matrix is ​​obtained as the EEG signal feature fusion data. The feature fusion algorithm is as follows: In the formula, This is the EEG signal feature fusion matrix. for The differential entropy feature matrix, For the first b The first frequency band t The first EEG unit signal in the first EEG unit c EEG data from multiple channels, for The time-domain feature matrix, for The frequency domain characteristic matrix, For the first t The first of the second EEG unit signals c EEG data from multiple channels, This is a splicing symbol.

5. The emotion recognition method based on electroencephalogram (EEG) signals according to claim 1, characterized in that: The attention extraction network includes several graph attention neural units corresponding to the number of edges in the edge set matrix; The step of inputting several feature vector matrices and their corresponding edge set matrices into the graph attention extraction network for attention feature extraction to obtain the graph attention feature data includes the following steps: Several feature vector matrices and their corresponding edge set matrices are used as input data for several graph attention neural units corresponding to the number of edges in the edge set matrices. According to a preset attention feature calculation algorithm, graph attention feature matrices output by several graph attention neural units are obtained. These graph attention feature matrices are then concatenated to obtain the graph attention feature data. The attention feature calculation algorithm is as follows: In the formula, For the first l The graph attention feature matrix corresponding to each of the aforementioned feature vector matrices. Let be the number of edges in the edge set matrix. For normalized exponential functions, Represented as the edge set matrix of the th j Each edge corresponds to a graph attention neural unit.

6. The emotion recognition method based on electroencephalogram (EEG) signals according to claim 1, characterized in that, It also includes the following steps: Obtain a set of EEG signals from several sample users, input the set of EEG signals from several sample users into the emotion recognition model, and obtain the predicted emotion recognition probability data of several sample users. Obtain real emotion recognition probability data from several sample users. Based on the real emotion recognition probability vectors of several dimensions in the real emotion recognition probability data, the predicted emotion recognition probability vectors of several dimensions in the predicted emotion recognition probability data, and a preset loss function, obtain a loss value. Train the emotion recognition model based on the loss value, wherein the loss function is: In the formula, L The loss value. I The number of sample users, For the number of dimensions, For the first i The first sample user k A probability vector for true emotion recognition in 1 / 2 dimension For the first i The first sample user k A probability vector for predicting emotion recognition in 10 dimensions.

7. An emotion recognition device based on electroencephalogram (EEG) signals, characterized in that, include: The data acquisition module is used to acquire the EEG signals of the user to be identified and a preset emotion recognition model. The emotion recognition model includes a feature extraction module, a feature fusion module, a graph attention module, and a recognition module. The graph attention module includes a convolutional pooling network, a graph organization network, and a graph attention extraction network. The convolutional pooling network includes several sequentially connected convolutional pooling layers. The EEG signal feature extraction module is used to input the EEG signal into the feature extraction module, extract features from the EEG signal, and obtain EEG signal feature extraction data, wherein the EEG signal feature extraction data includes several types of signal feature data; The EEG signal feature fusion module is used to input the extracted EEG signal feature data into the feature fusion module, perform feature fusion on several types of signal feature data, and obtain EEG signal feature fusion data. The EEG signal graph attention feature extraction module is used to input the EEG signal feature fusion data into the graph attention module, construct EEG signal feature fusion data based on the EEG signal feature fusion data, use the EEG signal feature fusion data as input data of the first convolutional pooling layer of the convolutional pooling network, perform convolutional pooling processing, use the output data of the first convolutional pooling layer as input data of the next convolutional pooling layer, repeat the convolutional pooling processing, and obtain several convolutional pooling feature data output by the convolutional pooling layer; The convolutional pooling feature data is converted into feature vector matrices to obtain several feature vector matrices corresponding to the convolutional pooling feature data. These feature vector matrices are then input into the graph organization network. Based on the feature vector matrices and a preset weight matrix calculation algorithm, several weight feature matrices are obtained. Each weight matrix includes the weight feature vectors of several nodes. The weight feature matrix calculation algorithm is as follows: In the formula, The weight eigenma matrix, The eigenvector matrix, It is the hyperbolic tangent function. It is a linear fully connected function; Based on several weight feature matrices and a preset adjacency matrix calculation algorithm, several adjacency matrices are obtained. Based on several adjacency matrices and a preset edge set matrix calculation algorithm, edge set matrices corresponding to several feature vector matrices are obtained. The adjacency matrix includes connection weight vectors between several nodes. The adjacency matrix calculation algorithm is as follows: In the formula, For the first i The node and the first j The connection weight vector between nodes. For the first i The weight eigenvectors of each node. For the first j The weight eigenvectors of each node. For the number of nodes, T It is the transpose symbol; The algorithm for calculating the edge set matrix is ​​as follows: In the formula, E Let be the edge set matrix. A It is an adjacency matrix. This is an index function for finding the maximum and minimum values; Several feature vector matrices and their corresponding edge set matrices are respectively input into the graph attention extraction network to extract attention features and obtain graph attention feature data; The emotion recognition module is used to input the graph attention feature data into the recognition module for emotion recognition, obtain predicted emotion recognition probability data, and obtain the emotion polarity corresponding to the predicted emotion recognition probability vector with the largest dimension based on the predicted emotion recognition probability data, which is used as the emotion recognition result of the user to be identified.

8. A computer device, characterized in that, The method includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the emotion recognition method based on electroencephalogram (EEG) signals as described in any one of claims 1 to 6.

Images