An emotion recognition method based on Siamese network architecture and graph convolution
By using twin network architecture and graph convolution combined with multi-head self-attention mechanism in EEG emotion recognition, the problem of insufficient accuracy of EEG emotion recognition in the existing technology is solved, and higher accuracy and better Model performance.
Patent Information
- Application Number
- CN202111617915.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-22
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2041-12-22
AI Technical Summary
The existing EEG emotion recognition methods are insufficient in improving the accuracy of emotion classification, especially when dealing with complex brain network structures. Too many graph convolution layers will lead to excessive smoothing and affect the accuracy.
Using a method based on twin network architecture and graph convolution, combined with the multi-head self-attention mechanism, adaptively gives data different importance, and improves the performance and generalization capabilities of the model through feature comparison learning of the intermediate layer.
The accuracy of emotion recognition is improved to 94.78%±05.97%, and the performance and generalization ability of the model are improved.
Smart Images

Figure CN114330436B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an emotion recognition method based on a twin network architecture and graph convolution, and belongs to the technical field of electroencephalogram emotion recognition. Background Art
[0002] Emotions play a role in information transmission and behavior regulation in people's daily communication, work and study, and cognitive decision-making. Identifying the right emotions can help people grasp the right information. Emotions are a complex psychological and physiological state that originates from the brain's response to these physiological changes and plays a vital role in our lives. In recent years, more and more research has focused on emotion recognition, not only to create emotional interaction interfaces for machines to perceive human emotions, but also to assess psychological diseases in patients with neurological disorders, such as Parkinson's disease, autism spectrum disorder, schizophrenia, depression, etc.
[0003] There are two main methods for emotion recognition: non-physiological signals and physiological signals. Since some patients cannot express emotions through external physiological characteristics such as facial expressions and body postures, and some people deliberately conceal their emotions, the measurement of physiological signals is often used as an analytical signal source for emotion classification. EEG signals have the advantages of high temporal resolution, non-invasiveness, low acquisition cost and easy acquisition. They are one of the commonly used physiological signals and have been proven to reflect important information about human emotional state.
[0004] EEG data is not regular Euclidean data. For irregular brain network structures, graph convolution can better capture the complex connections between channels. However, adding too many graph convolution layers will lead to over-smoothing and affect the accuracy. It is worth considering how to extract more effective features and improve the accuracy of emotion classification. There are already methods such as convolutional neural network (CNN), recurrent neural network (RNN), support vector machine (SVM) to identify EEG emotions, and they have also obtained very impressive results, but there are still some shortcomings in the accuracy of emotion recognition.
[0005] At present, there is a research method for emotion recognition and classification based on feature extraction of differential entropy of EEG signals combined with LSTM neural network model (publication number CN110897648A), which includes the following steps: (1) extracting 62-channel EEG signals of normal adults; (2) calculating its differential entropy (DE), thereby forming a 62-dimensional time series feature; (3) using the time series feature as the input of the LSTM neural network and training and learning; (4) evaluating the results of network training using the average classification accuracy, standard deviation and F1 value. This method has good effect and can effectively identify and classify three emotions. The above method finds the similarities and differences of three different emotional EEG signals from the characteristics of non-stationarity, nonlinearity, time-frequency domain and complexity of EEG signals, thereby distinguishing the three emotions and helping to assist in the recovery of various diseases. The difference between the present application and the present invention is that the present invention uses the twin network framework to train auxiliary tasks, and while using the model to classify emotions, uses the characteristics of the intermediate layer for comparative learning, thereby improving the performance and generalization ability of the model, and the emotion recognition accuracy is also different. Summary of the invention
[0006] In view of the above shortcomings, the present invention proposes an emotion recognition method based on the twin network architecture and graph convolution, which uses the multi-head self-attention mechanism to adaptively assign different importance to data and extract deeper and more effective information. In addition, in order to further improve the accuracy of the model, the present invention uses the twin network framework to train auxiliary tasks. While using the model for emotion classification, the features of the intermediate layer are used for comparative learning, thereby improving the performance and generalization ability of the model.
[0007] To achieve the above object, the technical solution of the present invention is:
[0008] An emotion recognition method based on Siamese network architecture and graph convolution includes the following steps:
[0009] Dataset acquisition. The EEG signals of 62 designated electrode positions were collected while the subjects watched the movie clips, and the subjects were asked to complete a questionnaire immediately after watching each movie clip to report their emotional response to each movie clip (neutral, negative, positive emotion);
[0010] Data preprocessing and feature extraction. The original EEG signal was downsampled and preprocessed to remove artifacts. The time domain signal was filtered using a Hamming window and Fast Fourier Transform (FFT). Each second of the signal was used as a sliding window to calculate the differential entropy features of 62 channels in 5 frequency bands and normalize them.
[0011] Sample generation: To effectively utilize time information, the 62-channel DE feature of 3s is used as a sample, and the dimension of an input sample is 3*62*5 (time*channel*frequency band).
[0012] Sample pair generation. Traverse each sample generated in step 3 and use it as a reference, record it as input_1, randomly select a sample with the same emotion (recorded as input_2), and form a positive sample pair with input_1. Then randomly select a sample with a different emotion (recorded as input_3), and form a negative sample pair with input_1. That is, in the positive sample pair, input_2 and input_1 are samples with the same emotion, and in the negative sample pair, input_3 and input_1 are samples with different emotions.
[0013] Define the basic model. Take the spatiotemporal graph convolutional neural network model as the basic model. The basic neural network consists of an adaptive graph learning layer and a spatiotemporal convolution module. The adaptive graph learning layer is designed to learn the connectivity of the brain network; the spatiotemporal convolution module consists of a spatiotemporal self-attention mechanism, graph convolution, and ordinary convolution. The spatiotemporal self-attention mechanism captures the dynamics of the spatial and temporal dimensions under different emotional states, graph convolution realizes the aggregation of adjacent nodes, and ordinary convolution is used to extract features in the temporal dimension.
[0014] Define the twin network architecture. Input the two samples in the sample pair, namely input_1 and input_2 or input3, into the same basic model in sequence, and then generate two intermediate features embedding_1 and embedding_2 respectively, and then calculate the distance between embedding_1 and embedding_2. Then use the multi-head self-attention layer to extract deeper features of embedding_1, and output the probability of input_1 belonging to each category after passing through the fully connected layer and softmax layer.
[0015] Define the input and output of the model. The model input is a positive sample pair (input_1 and input_2) or a negative sample pair (input_1 and input_3). The model has two outputs output_1 and output_2, where output_1 is the distance between the intermediate features embedding_1 and embedding_2, and output_2 is the probability that the sample input_1 belongs to each category.
[0016] Define the objective function. The final objective function of the model consists of three loss functions. First, in the adaptive graph learning layer, this layer aims to learn the connectivity relationship of the brain. The loss function is used to constrain the relationship between the distance of the features between channels and the connectivity strength. The greater the feature distance between the two channels, the weaker the connectivity strength. Since the connectivity structure of the brain is not fully connected, the L2 norm regularization term is used to control the sparsity of the learned graph. The second is the contrast loss of the twin network, the purpose of which is to make the distance between the positive sample pairs in step four closer and the distance between the negative sample pairs farther. The third is the cross entropy loss, which is used to measure the error between the predicted value of input_1 and the true sample label in step six.
[0017] Training and testing. Input the sample pairs generated in step 6 into the model for training. After the model is trained, use the samples in the test set as input_1, and obtain input_2 by randomly initializing a tensor with the same dimension as input_1, thus forming the input sample pairs. Use output_2 to calculate the classification accuracy.
[0018] The average classification accuracy is used to evaluate the learning results.
[0019] Compared with the prior art, the present invention has the following beneficial effects:
[0020] The present invention proposes an emotion recognition method based on a twin network architecture and graph convolution, which uses a multi-head self-attention mechanism to adaptively assign different importance to data and extract deeper and more effective information; at the same time, the twin network framework is borrowed to use the features of the middle layer for comparative learning, thereby improving the performance and generalization ability of the model. In addition, compared with other graph convolution methods, the emotion recognition accuracy of the model based on the twin network architecture in the present invention has been improved, with an accuracy of 94.78% ± 05.97%. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0022] Figure 1 A flowchart of an emotion recognition method based on a twin network architecture and graph convolution provided by the present invention;
[0023] Figure 2 A spatiotemporal graph convolutional neural network model for an emotion recognition method based on a twin network architecture and graph convolution provided by the present invention;
[0024] Figure 3 An experimental method for an emotion recognition method based on a twin network architecture and graph convolution provided by the present invention;
[0025] Figure 4 The channel position of the collected EEG model of the emotion recognition method based on the twin network architecture and graph convolution provided by the present invention;
[0026] Figure 5 An experimental comparison chart of an emotion recognition method based on a twin network architecture and graph convolution provided by the present invention. DETAILED DESCRIPTION
[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0028] Reference Figure 1-Figure 4 , an emotion recognition method based on Siamese network architecture and graph convolution, specifically including the following steps:
[0029] Step (1) Dataset acquisition
[0030] Step (1-1) selected 15 movie clips (containing positive, neutral, and negative emotions) from the material library as stimuli used in the experiment;
[0031] Step (1-2) The experimenter watched 15 clips in each experiment. There was a 5-second prompt before each clip, and each clip was presented for 4 minutes. After each clip, the experimenter had a 45-second self-assessment and a 15-second rest after each clip. During the self-assessment stage, the experimenter completed a questionnaire to feedback the emotional response of the experimenter to each movie clip.
[0032] Step (2) Data preprocessing and feature extraction
[0033] Step (2-1) downsamples the original signal to 200 Hz to remove the signal interfered by electrooculography and electromyography;
[0034] Step (2-2) extracts DE features for each channel from 5 frequency bands: δband (1-3Hz), θband (4-7), αband (8-13Hz), βband (14-30Hz), γband (31-50Hz):
[0035] A. Use Hamming window to filter the raw data, perform fast Fourier transform on the data per second, and calculate the differential entropy of the above five frequency bands;
[0036] B. The definition of differential entropy is as follows:
[0037] Suppose the value of the random variable X is X={x1,x2,...,x n}, n≥1, the corresponding probability is P= According to the definition of Shannon information entropy, the amount of information of this non-deterministic system is shown in formula (1):
[0038]
[0039] The state probability pi in the time domain in the above formula is replaced by the frequency domain power spectrum density p^ defined based on the fast Fourier transform, which leads to the definition of differential entropy as shown in formula (2):
[0040]
[0041] in represents the power spectral density;
[0042] Step (2-3) uses z-score to normalize the EEG signal. The normalization formula is as shown in formula (3):
[0043]
[0044] Among them, X is the EEG signal on each channel, is the average value of the EEG signal on each channel, and S is the standard deviation of the EEG signal on each channel.
[0045] Step (3) Sample generation
[0046] In order to effectively utilize the time information, the DE features of 62 channels of 3s are taken as a sample, and the dimension of an input sample is 3*62*5 (time*channel*frequency band).
[0047] Step (4) Sample pair generation
[0048] Traverse each sample generated in step 3 and use it as a reference (referred to as input_1), randomly select a sample with the same emotion (referred to as input_2) to form a positive sample pair with input_1, and then randomly select a sample with a different emotion (referred to as input_3) to form a negative sample pair with input_1, that is, in the positive sample pair, input_2 and input_1 are samples with the same emotion, and in the negative sample pair, input_3 and input_1 are samples with different emotions.
[0049] Step (5) Define the basic model
[0050] The spatiotemporal graph convolutional neural network model is used as the basic model. The basic model consists of an adaptive graph learning layer and a spatiotemporal convolution module. The adaptive graph learning layer is designed to learn the connectivity of the brain network. The spatiotemporal convolution module consists of a spatiotemporal self-attention mechanism, graph convolution, and ordinary convolution. The spatiotemporal self-attention mechanism captures the dynamics of the spatial and temporal dimensions under different emotional states. The graph convolution realizes the aggregation of adjacent nodes, and the ordinary convolution is used to extract features in the temporal dimension.
[0051] Step (5-1) Adaptive graph learning
[0052] A non-negative adjacency matrix is defined based on the channel features, as shown in formula (4):
[0053] A pq =g(x p ,x q ), where p,q∈{1,2,...,N} (4)
[0054] Among them A pq represents the connection relationship between channel p and channel q, that is, the weight of the edge connecting node p and node q, g(x p ,x q ) aims to represent the adjacency matrix A by learning a weight vector w. The definition of A is shown in formula (5):
[0055]
[0056] Step (5-2) Spatiotemporal Self-Attention Mechanism
[0057] A. Calculate temporal self-attention. In time, there is correlation between the states of different time slices. The correlation is different in different situations. The attention mechanism is used to adaptively capture the dynamic correlation between nodes in the time dimension. Transpose the input to get χ h , whose dimension is (62*5*3), and the temporal attention is defined as shown in formula (6) and formula (7):
[0058] T=V T ·σ(((χ h ) T U1)U2(U3χ h )+b T ) (6)
[0059]
[0060] Where T′ i,j Represents the similarity between time i and time j. V T 、U1、U2、U3、b T is the learned parameter and σ is the sigmoid activation function.
[0061] B. Calculate spatial attention. In space, channels at different locations influence each other, and this influence is highly dynamic. The attention mechanism is used to adaptively capture the dynamic correlation between nodes in the spatial dimension. Transpose the input to get χ h , whose dimension is (62*5*3), and the spatial attention is defined as shown in formula (8) and formula (9):
[0062] S=V S ·σ((χ h W1)W2(W3x h ) T +b s ) (8)
[0063]
[0064] where S′ p,q V represents the similarity between channel p and channel q. S , W1, W2, W3, b S is the learned parameter and σ is the sigmoid activation function.
[0065] Step (5-3) Spatial convolution
[0066] Calculate the Laplace matrix L = DA, where A is the adjacency matrix learned in step (4-2), and D is the degree matrix calculated based on A, that is, D is a diagonal matrix with the same dimension as A, where the elements Dii on the diagonal of the D matrix are the sum of the i-th row in A.
[0067] A. Calculate the maximum eigenvalue λ of the Laplacian matrix L max , calculated by formula (10)
[0068]
[0069] where I is the identity matrix.
[0070] B. Recursively calculate the Chebyshev polynomial according to formula (11):
[0071]
[0072] in
[0073] C. Perform graph convolution according to formula (12)
[0074]
[0075] where g θ represents the convolution kernel, *G represents the graph convolution operation, θ kRepresents the Chebyshev coefficient, obtained through learning, and x is the input data.
[0076] Step (5-4) Temporal Convolution
[0077] In this layer, a 2D convolution is performed in the time dimension using a 3*1 convolution kernel with a stride of 1 and a padding of 1 to preserve the input height and width.
[0078] Step (5-5) Residual connection and layer normalization
[0079] In order to alleviate the gradient vanishing problem and help the network train better, a layer of residual network is added and layer normalization is performed.
[0080] Step (6) Define the twin network architecture
[0081] Step (6-1) Obtaining the middle layer features
[0082] The two samples in the sample pair generated in step 5, namely input_1 and input_2 or input3, are input into the same basic model in sequence, and two intermediate features embedding_1 and embedding_2 are generated respectively.
[0083] Step (6-2) Calculate the distance between sample pairs
[0084] Calculate the distance between the two outputs embedding_1 and embedding_2 of step (6-1) as shown in formula (13):
[0085]
[0086] Where emb_1 represents embedding_1, emb_2 represents embedding_2, C represents the number of channels, T represents the time length, F represents the number of features, emb_1 ctf represents the fth row element of the c-channel t feature at embedding_1, emb_2 ctf Represents the f-th row element of the c-channel t feature at embedding_2;
[0087] Step (6-3) Multi-head self-attention layer
[0088] A. Randomly initialize a learnable position matrix P and position encode embedding_1 according to formula (14):
[0089] X embedding =embedding_1+P (14)
[0090] B. Randomly initialize 8 groups (also called 8 heads) of matrices Wq i , W k i , W v i (i=0,1,2,3,4,5,6,7) respectively embedding_1 (here denoted as X embedding ) point multiplication to obtain 8 sets of Q, K, V matrices, as shown in equations (15)-(17):
[0091] Q i =X embedding W q i (15)
[0092] K i =X embedding W k i (16)
[0093] V i =X embedding W v i (17)
[0094] i=0,1,2,3,4,5,6,7
[0095] C. For each group, the matrix calculates the size of the attention weight and divides it by W k The square root of the first dimension of the matrix, that is Then multiply it with V to get the output of the attention layer, and finally get 8 sets of matrices (Z0-Z7), as shown in formula (18):
[0096]
[0097] i=0,1,2,3,4,5,6,7
[0098] D. Concatenate these 8 matrices horizontally (Z0, Z1, ..., Z7), and then randomly initialize a matrix W o , multiply the above two matrices to obtain the matrix Z, as shown in formula (19):
[0099] Z=concatenate(Z0,Z1,Z2,Z3,Z4,Z5,Z6,Z7)·W o (19)
[0100] Step (6-4) Fully connected layer and softmax layer
[0101] A. Flatten the output Z of step (6-3) into a one-dimensional vector;
[0102] B. After a fully connected layer transformation, a vector with a dimension of 16 is obtained;
[0103] C. After another fully connected layer, a vector of dimension 3 is obtained, which is activated using the softmax function to obtain the probability that the sample input_1 belongs to each category.
[0104] Step (7) Define the input and output of the model
[0105] The model input is a positive sample pair (input_1 and input_2) or a negative sample pair (input_1 and input_3). The model has two outputs output_1 and output_2, where output_1 is the distance between the intermediate features embedding_1 and embedding_2, and output_2 is the probability that the sample input_1 belongs to each category.
[0106] Step (8) Define the objective function
[0107] The final objective function of the model is composed of three loss functions. First, in the adaptive graph learning layer, this layer aims to learn the brain connectivity graph, and uses the loss function to constrain the relationship between the distance of channel features and the strength of connectivity. The greater the distance between the features of the two channels, the weaker the connectivity strength. Since the connectivity structure of the brain is not fully connected, the L2 norm regularization term is used to control the sparsity of the learned graph. The second is the contrast loss of the twin network, which aims to make the distance between the positive sample pairs in step four closer and the distance between the negative sample pairs farther. The third is the cross entropy loss, which is used to measure the error between the predicted value of input_1 and the true sample label in step six.
[0108] The final objective function form of the model is shown in formula (20):
[0109] L=L graph_learn +ηL contrastive_loss +L cross_entropy (20)
[0110] Where η is the adjustment parameter between the two loss functions. The larger η is, the greater the proportion of contrast loss is, and vice versa. The three parts that constitute the objective function are shown in equations (21)-(23):
[0111]
[0112] Among them, x p is the characteristic of the p channel, x q is the characteristic of the q channel, A pq is the connection strength between the p channel and the q channel, and λ is the regularization coefficient.
[0113]
[0114] Where d is the Euclidean distance between channels p and q, y is the binary classification label, y=0 means that samples m and n are not from the same emotion, y=1 means that samples m and n are from the same emotion, N is the number of sample pairs in a batch, and margin is a hyperparameter, which indicates the distance separating samples of different emotions.
[0115]
[0116] Among them, y i,r Is the flag of whether sample i belongs to category r, if yes, it is 1, otherwise it is 0, is the probability that the predicted sample i belongs to category r.
[0117] Step (9) Training and testing
[0118] Input the sample pairs generated in step (4) into the model for training. After the model is trained, use the samples in the test set as input_1, and obtain input_2 by randomly initializing a tensor with the same dimension as input_1, thus forming the input sample pairs. Use output_2 to calculate the classification accuracy.
[0119] Step (10) uses the average classification accuracy to evaluate the learning results.
[0120] Step (10-1) uses accuracy to evaluate the model. Accuracy is the ratio of correctly classified samples to the total number of samples. A total of 15 experimenters participated in the experiment. Each experimenter conducted three experiments and watched 15 clips in each experiment, so a total of 45 experiments were conducted. The accuracy calculation formula for the ith experiment is shown in formula (24):
[0121]
[0122] i=1,2,3,...,45
[0123] Among them, TP is the positive sample predicted by the model as positive, TN is the negative sample predicted by the model as negative, FP is the negative sample predicted by the model as positive, and FN is the positive sample predicted by the model as negative.
[0124] The average accuracy of 15 subjects in 2 experiments is shown in formula (25):
[0125]
[0126] The standard deviation of this experiment is shown in formula (26)
[0127]
[0128] The accuracy of the present invention is tested after training and testing, and the results are compared with the existing technologies (SVM, GCNN, DGCNN, BiDANN) as shown in Table 1. In the comparison, two experimental results of each subject are taken for testing; and the average value of all the accuracy data obtained is used to measure the effect of the model.
[0129] from Figure 5 It can be found that the accuracy of the present invention is higher than that of SVM, GCNN, DGCNN and BiDANN methods.
[0130] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions and variations of these embodiments are made without departing from the principles and spirit of the present invention, and still fall within the scope of protection of the present invention.
Claims
1. An emotion recognition method based on a twin network architecture and graph convolution, characterized by: The following steps are involved: Step 1, data set acquisition: collect EEG data of subjects when watching movie clips, and ask subjects to complete a questionnaire immediately after watching each movie clip to report their emotional response to each movie clip. The emotional response includes positive, neutral, and negative. The EEG data is the EEG signal of 62 channels at designated electrode positions collected by the 10-20 international standard lead system; Step 2: Data preprocessing and feature extraction: downsample the original EEG signal, perform artifact removal preprocessing, filter the time domain signal using a Hamming window and perform a fast Fourier transform. Use each second of the signal as a sliding window to calculate the differential entropy features of 62 channels in 5 frequency bands and perform normalization. Step 3: Sample generation: Take the differential entropy features of 62 channels of 3s as a sample, then the dimension of an input sample is time*channel*frequency band; Step 4: Generate sample pairs: Traverse each sample generated in step 3 and use it as a reference, record it as input_1, randomly select a sample with the same emotion, record it as input_2, and form a positive sample pair with input_1, and then randomly select a sample with a different emotion, record it as input_3, and form a negative sample pair with input_1; Step 5: Define the basic model: Use the spatiotemporal graph convolutional neural network model as the basic model, which includes an adaptive graph learning layer and a spatiotemporal convolution module; Step 6. Define the twin network architecture: Input the two samples of the sample pair generated in step 5, input_1 and input_2 or input_3, into the same basic model in turn, generate two intermediate features embedding_1 and embedding_2 respectively, then calculate the distance between embedding_1 and embedding_2, and then use the multi-head self-attention layer to extract deeper features of embedding_1. After passing through the fully connected layer and the softmax layer, the probability of input_1 belonging to each category is output; Step 7. Define the input and output of the model: the model input is a positive sample pair or a negative sample pair; the model has two outputs output_1 and output_2, output_1 is the distance between the intermediate features embedding_1 and embedding_2, and output_2 is the probability that the sample input_1 belongs to each category; Step 8. Define the objective function: The final objective function of the model is composed of three loss functions; Step 9, training and testing: During training, the sample pairs generated in step 4 are input into the basic model for training; during testing, in order to keep the input scale consistent, the samples in the test set are used as input_1, and input_2 is obtained by randomly initializing a tensor with the same dimension as input_1 to form the input sample pairs, and the output output_2 is used to calculate the classification accuracy; Step 10: Use the average classification accuracy to evaluate the learning results.
2. The emotion recognition method based on twin network architecture and graph convolution according to claim 1, characterized in that: The adaptive graph learning layer is designed to learn the connectivity relationship of the brain network; the spatiotemporal convolution module includes a spatiotemporal self-attention mechanism, a graph convolution and an ordinary convolution. The spatiotemporal self-attention mechanism captures the dynamics in the spatial and temporal dimensions under different emotional states, the graph convolution realizes the aggregation of adjacent nodes, and the ordinary convolution is used to extract features in the temporal dimension.
3. The emotion recognition method based on twin network architecture and graph convolution according to claim 2, characterized in that: The three loss functions are: In the adaptive graph learning layer, the loss function is used to constrain the relationship between the distance of the features between channels and the connectivity strength. The greater the distance between the features of two channels, the weaker the connectivity strength. The L2 norm regularization term is used to control the sparsity of the learned graph. The contrast loss of the twin network makes the distance between the positive sample pairs in step 4 closer and the distance between the negative sample pairs farther; Cross entropy loss is used to measure the error between the predicted value of input_1 in step 6 and the actual sample label.
4. The emotion recognition method based on twin network architecture and graph convolution according to claim 1, 2 or 3, characterized in that: The step 1 comprises the following steps: Step (1-1): 15 movie clips were selected from the material library as stimuli used in the experiment, and the movie clips respectively contained movie clips with positive, neutral and negative emotions; Step (1-2): The subjects watched 15 movie clips in each experiment. There was a 5-second prompt before each clip, and each clip was presented for 4 minutes. After each clip, they conducted a 45-second self-assessment and rested for 15 seconds after each clip. During the self-assessment stage, the subjects completed a questionnaire to provide feedback on their emotional response to each movie clip.
5. The emotion recognition method based on twin network architecture and graph convolution according to claim 1, 2 or 3, characterized in that: The step 2 comprises the following steps: Step (2-1): downsample the original signal to 200 Hz to remove the signal interfered by electrooculography and electromyography; Step (2-2): Extract DE features for each channel from 5 frequency bands: δ band(1-3Hz) ,θ band(4-7) , α band(8-13Hz) , β band(14-30Hz) , γ band(31-50Hz) : Then use the Hamming window to filter the original data, perform fast Fourier transform on the data per second, and calculate the differential entropy of the above five frequency bands; The definition of differential entropy is as follows: Suppose the value of the random variable X is X={x1,x2,...,x n }, n≥1, the corresponding probability is P={p1,p2,...,pn}, 0≤pi≤1, According to the definition of Shannon information entropy, the amount of information of this non-deterministic system is shown in formula (1): The state probability p in the time domain in the above formula i Replaced by the frequency domain power spectral density defined based on the fast Fourier transform This leads to the definition of differential entropy as shown in formula (2): in represents the power spectral density; Step (2-3): Use z-score to normalize the EEG signal. The normalization formula is as shown in formula (3): Among them, X is the EEG signal on each channel, is the average value of the EEG signal on each channel, and S is the standard deviation of the EEG signal on each channel.
6. The emotion recognition method based on twin network architecture and graph convolution according to claim 1, 2 or 3, characterized in that: The step five comprises the following steps: Step (5-1) Adaptive graph learning: A non-negative adjacency matrix is defined based on the channel features, as shown in formula (4): A pq =g(x p ,x q ), where p,q∈{1,2,...,N} (4) Among them A pq represents the connection relationship between channel p and channel q, that is, the weight of the edge connecting node p and node q, g(x p ,x q ) aims to represent the adjacency matrix A by learning a weight vector w. The definition of A is shown in formula (5): Step (5-2) Spatiotemporal self-attention mechanism: Calculate the temporal self-attention, use the attention mechanism to adaptively capture the dynamic correlation between nodes in the time dimension, and transpose the input to obtain χ h , whose dimension is (62*5*3), and the temporal attention is defined as shown in formula (6) and formula (7): T=V T ·s(((x h ) T U1)U2(U3x h )+b T ) (6) Where T i, ' j represents the similarity between time i and time j, V T 、U1、U2、U3、b T is the learned parameter, σ is the sigmoid activation function; Then we calculate the spatial attention, use the attention mechanism to adaptively capture the dynamic correlation between nodes in the spatial dimension, and transpose the input to get χ h , whose dimension is (62*5*3), and the spatial attention is defined as shown in formula (8) and formula (9): S=V S ·s((x h W1)W2(W3x h ) T +b s ) (8) where S′ p,q represents the similarity between channel p and channel q, V S , W1, W2, W3, b S is the learned parameter, σ is the sigmoid activation function; Step (5-3) spatial convolution: Calculate the Laplacian matrix L = DA, where A is the adjacency matrix learned in step (4-2), and D is the degree matrix calculated based on A, that is, D is a diagonal matrix with the same dimension as A, where the elements on the diagonal of D matrix D ii is the sum of the i-th row in A, Compute the largest eigenvalue λ of the Laplacian matrix L max , calculated by formula (10) Where II is the identity matrix; According to formula (11), the Chebyshev polynomial is recursively calculated: in Perform graph convolution according to formula (12): where g θ represents the convolution kernel, *G represents the graph convolution operation, θ k represents the Chebyshev coefficient, obtained through learning, and x is the input data; Step (5-4) time convolution: Use a 3*1 convolution kernel to perform 2D convolution in the time dimension, with a stride of 1 and a padding of 1 to preserve the input height and width; Step (5-5) residual connection and layer normalization: Add a layer of residual network and perform layer normalization.
7. The emotion recognition method based on twin network architecture and graph convolution according to claim 1, 2 or 3, characterized in that: The step six comprises the following steps: Step (6-1) obtains the intermediate layer features: Input the two samples in the sample pair generated in step 5, namely input_1 and input_2 or input3, into the same basic model in sequence, and then generate two intermediate features embedding_1 and embedding_2 respectively; Step (6-2) calculates the distance between sample pairs: Calculate the distance between the two outputs embedding_1 and embedding_2 in step 5 as shown in formula (13): Where emb_1 represents embedding_1, emb_2 represents embedding_2, C represents the number of channels, T represents the time length, F represents the number of features, emb_1 ctf represents the fth row element of the c-channel t feature at embedding_1, emb_2 ctf Represents the f-th row element of the c-channel t feature at embedding_2; Step (6-3) Multi-head self-attention layer: Perform position encoding on embedding_1 according to formula (14): X embedding =embedding_1+P (14) Where P is a learnable matrix; Randomly initialize 8 matrices W q ,W k ,W v , and X embedding The dot product yields 8 sets of Q, K, and V matrices, as shown in equations (15)-(17): Q i =X embedding W q i (15) K i =X embedding W k i (16) V i =X embedding W v i (17) i=0,1,2,3,4,5,6,7 Where W q i , W k i , W v i is a learnable matrix; For each group, the attention weight is calculated by the Q and K matrices and divided by W k The square root of the first dimension of the matrix, that is And then multiply it with V to get the output of the attention layer, and finally get 8 sets of matrices (Z0-Z7), as shown in formula (18): These 8 matrices are concatenated horizontally (Z0, Z1, ..., Z7), and then a learnable matrix W is randomly initialized. o , multiply the above two matrices to obtain the matrix Z, as shown in formula (19): Z = concatenate(Z0,Z1,Z2,Z3,Z4,Z5,Z6,Z7)·W o (19) Step (6-4) Fully connected layer and softmax layer: The output Z of step (6-3) is flattened into a one-dimensional vector; after a fully connected layer transformation, a vector of dimension 16 is obtained; after another fully connected layer transformation, a vector of dimension 3 is obtained, which is activated using the softmax function to obtain the probability that the sample input_1 belongs to each category.
8. The emotion recognition method based on twin network architecture and graph convolution according to claim 1, 2 or 3, characterized in that: The step eight comprises the following steps: The final objective function form of the model is shown in formula (20): L=L graph_learn +ηL contrastive_loss +L cross_entropy (20) Where η is the adjustment parameter between the two loss functions. The larger η is, the greater the proportion of contrast loss is, and vice versa. The three parts that constitute the objective function are shown in equations (21)-(23): Among them, x p is the characteristic of the p channel, x q is the characteristic of the q channel, A pq is the connectivity strength between the p channel and the q channel, λ is the regularization coefficient; Where d is the Euclidean distance between channels p and q, y is the binary classification label, y=0 means that samples m and n are not from the same emotion, y=1 means that samples m and n are from the same emotion, N is the number of sample pairs in a batch, and margin is a hyperparameter, which indicates the distance to separate samples of different emotions; Among them, y i,r Is the flag of whether sample i belongs to category r, if yes, it is 1, otherwise it is 0, is the probability that the predicted sample i belongs to category r.
9. The emotion recognition method based on twin network architecture and graph convolution according to claim 1, 2 or 3, characterized in that: The step ten comprises the following steps: The accuracy rate is used to evaluate the model. The accuracy rate is the ratio of correctly classified samples to the total number of samples. A total of 15 subjects participated in the experiment. Each subject did three experiments and watched 15 clips in each experiment, so a total of 45 experiments were conducted. The accuracy rate calculation formula for the ith experiment is shown in formula (24): Then the average accuracy of 2 experiments with 15 subjects is shown in formula (25): The standard deviation of this experiment is shown in formula (26): Among them, TP is the positive sample predicted by the model as positive, TN is the negative sample predicted by the model as negative, FP is the negative sample predicted by the model as positive, and FN is the positive sample predicted by the model as negative.
Citation Information
Patent Citations
Research method for emotion recognition and classification based on combination of feature extraction of differential entropy of electroencephalogram signals and LSTM neural network model
CN110897648A
Facial expression recognition method and device based on expression intensity label distribution
CN110765873A
Human face expression intensity recognition method and system based on hidden variable analysis
CN112686117A
Visual perception emotion recognition method
CN113017630A
KR20210099492A
