Emotion recognition method and device based on long-time electroencephalogram signal, equipment and medium

Abstract

The application relates to an emotion recognition method and device based on long-time electroencephalogram signals, equipment and a medium, wherein the method comprises the following steps: obtaining a subject data set; extracting differential entropy features of each long-time electroencephalogram signal in the subject data set; performing feature extraction on spatial information and frequency information of the differential entropy features through a spatial feature extraction network in a space-time feature extraction network to obtain initial features; extracting time sequence information in the initial features through a time feature extraction network to obtain target features; performing classification processing based on the target features through a classifier and calculating a classification loss, and calculating a domain discrimination loss based on the target features through a domain adversarial network; training an emotion recognition model based on the classification loss and the domain discrimination loss to obtain a target emotion recognition model; and performing emotion recognition on to-be-recognized electroencephalogram signals based on the target emotion recognition model to obtain a target emotion recognition result. The application improves the emotion recognition accuracy of long-time electroencephalogram signals.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to an emotion recognition method and device based on long-time electroencephalogram signals, equipment and medium. BACKGROUND

[0002] Electroencephalography (EEG) is a non-invasive brain electrical physiological activity detection technology. It records the current activity signals of brain neurons through scalp electrodes, and can more directly reflect the changes in emotions compared to non-physiological signals such as facial expressions and speech. In recent years, researchers have been committed to developing various machine learning and deep learning models and applying them to emotion recognition tasks based on electroencephalogram signals. However, most of these methods are based on short-time electroencephalogram signals.

[0003] Specifically, the existing method first segments the original electroencephalogram signal by seconds, and the label corresponding to each segment of the electroencephalogram signal is the label of the original whole electroencephalogram signal. However, in real-world scenarios, some emotional states evolve over a longer period of time, for example, sadness may not be effectively awakened under short-time stimulation. At the same time, the average classification accuracy of sadness is the lowest among various emotional categories, only 18%. In addition, studies have shown that strong emotions often require time accumulation, and long-time electroencephalogram stimulation can induce stronger emotional representation, which will be beneficial to the model for better emotion decoding. Therefore, compared to short-time electroencephalogram signals, long-time electroencephalogram signals reflect emotional states that are more consistent with real-world application scenarios. How to design and develop an emotion recognition model for long-time electroencephalogram signals has become a major challenge in the field of affective computing. Therefore, there is an urgent need for an emotion recognition method for long-time electroencephalogram signals to improve the accuracy of emotion recognition. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide an emotion recognition method, device, equipment and medium based on long-time electroencephalogram signals to improve the accuracy of emotion recognition for long-time electroencephalogram signals.

[0005] To solve the above technical problems, the embodiments of the present application provide an emotion recognition method based on long-time electroencephalogram signals, comprising:

[0006] obtaining a subject data set, wherein the subject data set includes a plurality of long-time electroencephalogram signals;

[0007] extracting the differential entropy features of each long-time electroencephalogram signal in the subject data set;

[0008] input the differential entropy feature into a space-time feature extraction network based on a Transformer network, and perform feature extraction of spatial information and frequency information on the differential entropy feature through a spatial feature extraction network in the space-time feature extraction network to obtain an initial feature;

[0009] extract time sequence information in the initial feature through a time feature extraction network in the space-time feature extraction network to obtain a target feature;

[0010] perform classification processing based on the target feature through a classifier and calculate a classification loss, and calculate a domain discrimination loss based on the target feature through a domain adversarial network;

[0011] train an emotion recognition model based on the classification loss and the domain discrimination loss to obtain a target emotion recognition model;

[0012] obtain a target emotion recognition result by performing emotion recognition on the to-be-recognized electroencephalogram signal based on the target emotion recognition model.

[0013] To solve the above technical problems, an emotion recognition device based on long-time electroencephalogram signals is provided, comprising:

[0014] a subject data acquisition unit configured to acquire a subject data set, wherein the subject data set comprises a plurality of long-time electroencephalogram signals;

[0015] a differential entropy feature extraction unit configured to extract a differential entropy feature of each long-time electroencephalogram signal in the subject data set;

[0016] a spatial feature extraction unit configured to input the differential entropy feature into a space-time feature extraction network based on a Transformer network, and perform feature extraction of spatial information and frequency information on the differential entropy feature through a spatial feature extraction network in the space-time feature extraction network to obtain an initial feature;

[0017] a time sequence feature extraction unit configured to extract time sequence information in the initial feature through a time feature extraction network in the space-time feature extraction network to obtain a target feature;

[0018] a loss calculation unit configured to perform classification processing based on the target feature through a classifier and calculate a classification loss, and calculate a domain discrimination loss based on the target feature through a domain adversarial network;

[0019] a model training unit configured to train an emotion recognition model based on the classification loss and the domain discrimination loss to obtain a target emotion recognition model;

[0020] The recognition result generation unit is configured to acquire a brain electrical signal to be recognized, and perform emotion recognition on the brain electrical signal to be recognized based on the target emotion recognition model to obtain a target emotion recognition result.

[0021] To solve the above technical problems, one technical solution adopted by the present application is to provide a computer device, comprising one or more processors; a memory for storing one or more programs, so that the one or more processors implement the emotion recognition method based on long-time brain electrical signals according to any one of the above.

[0022] To solve the above technical problems, one technical solution adopted by the present application is to provide a computer readable storage medium, the computer readable storage medium has a computer program stored thereon, and the computer program is executed by a processor to implement the emotion recognition method based on long-time brain electrical signals according to any one of the above.

[0023] The embodiment of the present application provides an emotion recognition method, device, equipment and medium based on long-time brain electrical signals. Wherein, the method comprises: acquiring a subject data set, wherein the subject data set comprises a plurality of long-time brain electrical signals; extracting differential entropy features of each long-time brain electrical signal in the subject data set; inputting the differential entropy features into a space-time feature extraction network based on a Transformer network, and extracting spatial information and frequency information features of the differential entropy features through a spatial feature extraction network in the space-time feature extraction network to obtain initial features; extracting time sequence information in the initial features through a time feature extraction network in the space-time feature extraction network to obtain target features; performing classification processing and calculating classification loss based on the target features through a classifier, and calculating domain discrimination loss based on the target features through a domain adversarial network; training an emotion recognition model based on the classification loss and the domain discrimination loss to obtain a target emotion recognition model; acquiring a brain electrical signal to be recognized, and performing emotion recognition on the brain electrical signal to be recognized based on the target emotion recognition model to obtain a target emotion recognition result. The embodiment of the present application extracts spatial information, frequency information and time sequence information in long-time brain electrical signals, and trains an emotion recognition model based on the extracted information, which can fully utilize the time information and spatial information of brain electrical signals, and solves the shortcomings of traditional Transformers. In addition, the embodiment of the present application introduces a domain adversarial network in transfer learning to alleviate the individual differences between different subject brain electrical signals, and improves the emotion recognition accuracy of long-time brain electrical signals. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the scheme in the present application, the drawings needed in the description of the embodiments of the present application will be briefly introduced as follows. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0025] Figure 1 is an implementation flowchart of the emotion recognition method based on long-time electroencephalogram signals provided by the embodiments of the present application;

[0026] Figure 2 is an implementation flowchart of a sub-process of the emotion recognition method based on long-time electroencephalogram signals provided by the embodiments of the present application;

[0027] Figure 3 is an implementation flowchart of a sub-process of the emotion recognition method based on long-time electroencephalogram signals provided by the embodiments of the present application;

[0028] Figure 4 is a schematic diagram of an improved multi-head attention mechanism provided by the embodiments of the present application;

[0029] Figure 5 is an implementation flowchart of a sub-process of the emotion recognition method based on long-time electroencephalogram signals provided by the embodiments of the present application;

[0030] Figure 6 is an implementation flowchart of a sub-process of the emotion recognition method based on long-time electroencephalogram signals provided by the embodiments of the present application;

[0031] Figure 7 is an implementation flowchart of a sub-process of the emotion recognition method based on long-time electroencephalogram signals provided by the embodiments of the present application;

[0032] Figure 8 is an implementation flowchart of a sub-process of the emotion recognition method based on long-time electroencephalogram signals provided by the embodiments of the present application;

[0033] Figure 9 is a schematic diagram of an emotion recognition device based on long-time electroencephalogram signals provided by the embodiments of the present application;

[0034] Figure 10 is a schematic diagram of a computer device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0035] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used in the description herein is for describing particular embodiments only and is not intended to be limiting of the application. Unless otherwise defined, all terms used herein, including technical and scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description herein is for describing particular embodiments only and is not intended to be limiting of the application. The terms "comprises", "comprising", "includes", "including", "has", "having" and their variants are intended to be inclusive and allow for items not listed to be present.

[0036] Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase "in one embodiment" in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily all referring to a common embodiment, or an embodiment that is independent of other embodiments. It will be explicitly understood that the application described herein can be combined with other embodiments in a non- limiting fashion.

[0037] In order to make the technical personnel in the art better understand the scheme of the application, the technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings.

[0038] The application will be described in detail below with reference to the drawings and embodiments.

[0039] It should be noted that the emotion recognition method based on long-time electroencephalogram signals provided by the embodiments of the application is generally executed by a server, and accordingly, the emotion recognition device based on long-time electroencephalogram signals is generally configured in the server.

[0040] Please refer to Figure 1 , Figure 1 An embodiment of the emotion recognition method based on long-time electroencephalogram signals is shown.

[0041] It should be noted that the method of the application is not limited to the order of the flowchart shown Figure 1 The method includes the following steps:

[0042] S1: Obtain a subject data set, wherein the subject data set includes a plurality of long-time electroencephalogram signals.

[0043] Specifically, a subject data set including a plurality of long-time electroencephalogram signals is obtained, and the subject data set is divided into a training set and a test set, which are used to train an emotion recognition model. The emotion recognition model includes a spatial-temporal feature extraction network based on a Transformer network, a domain adversarial network, and a classifier, and the spatial-temporal feature extraction network includes a spatial feature extraction network and a temporal feature extraction network.

[0044] S2: extract differential entropy features of each long-time electroencephalogram signal in the subject data set.

[0045] Specifically, since the input of the spatial feature extraction network is the differential entropy features (DE) of each second of the multi-channel electroencephalogram signal, the differential entropy features of each long-time electroencephalogram signal in the subject data set need to be extracted before training the emotion recognition model.

[0046] Please refer to Figure 2 , Figure 2 An embodiment of step S2 is shown as follows:

[0047] S21: extract a plurality of preset frequency band data corresponding to each channel of each second of each long-time electroencephalogram signal in the subject data set by Fourier transform, to obtain target frequency band data.

[0048] S22: calculate the differential entropy features of each frequency band in the target frequency band data by using a preset formula.

[0049] Specifically, the electroencephalogram signal obtained by the embodiment of the present application is a multi-channel electroencephalogram signal, and 10 frequency band data corresponding to each channel of each second of each long-time electroencephalogram signal in the subject data set are extracted by Fourier transform. The 10 frequency bands are as follows: (1) Theta frequency band: 4-6 Hz; (2) Alpha 1 frequency band: 6-8 Hz; (3) Alpha 2 frequency band: 8-10 Hz; (4) Alpha 3 frequency band: 10-12 Hz; (5) Beta 1 frequency band: 12-16 Hz; (6) Beta 2 frequency band: 16-20 Hz; (7) Beta 3 frequency band: 20-28 Hz; (8) Gamma 1 frequency band: 28-34 Hz; (9) Gamma 2 frequency band: 34-39 Hz; and (10) Gamma 3 frequency band: 39-45 Hz. Then the differential entropy features of each frequency band in the target frequency band data are calculated by using a preset formula.

[0050] The preset formula is as follows:

[0051]

[0052] wherein x is a multi-channel electroencephalogram signal of a specific length and subject to Gaussian distribution N(μ,σ 2 ), μ is the mean of x, σ is the standard deviation of x, and DE is the differential entropy feature.

[0053] S3: input the differential entropy feature into a space-time feature extraction network based on a Transformer network, and perform feature extraction of spatial information and frequency information on the differential entropy feature through a spatial feature extraction network in the space-time feature extraction network to obtain an initial feature.

[0054] Specifically, a conventional Transformer network mainly consists of an encoder and a decoder. The encoder is used to complete a classification task, and the decoder is used to complete a generation task. Therefore, the encoder of the Transformer network is also applicable to an emotion classification task. Meanwhile, since a long-time electroencephalogram signal is different from a one-dimensional speech sequence, it contains rich time information and spatial information. If the conventional Transformer network is directly applied to a time sequence of the long-time electroencephalogram signal, the spatial information contained in the electroencephalogram signal is easily neglected. Therefore, the embodiment of the present application designs a space-time feature extraction network based on a feature-optimized Transformer network, which includes two parts: a spatial feature extraction network and a time feature extraction network. This network fully utilizes the time and spatial information of the long-time electroencephalogram signal by extracting the spatial features of the electroencephalogram signal, and solves the shortcomings of the conventional Transformer.

[0055] Please refer to Figure 3 and Figure 4 , Figure 3 a specific implementation of step S3 is shown, Figure 4 is a schematic diagram of the improved multi-head attention mechanism provided by the embodiment of the present application, which is described in detail as follows:

[0056] S31: input the differential entropy feature into a space-time feature extraction network based on a Transformer network, and perform feature extraction of spatial information and frequency information on the differential entropy feature through a spatial feature extraction network in the space-time feature extraction network to obtain an initial feature.

[0057] Specifically, for the spatial information of the multi-channel electroencephalogram signal, the embodiment of the present application takes different channels of the differential entropy feature at each time point as a spatial information sequence, and for the frequency information of the multi-channel electroencephalogram signal, takes different frequency bands of the differential entropy feature at each time point as a frequency information sequence.

[0058] S32: in the spatial feature extraction network, map the differential entropy feature per second into a Query vector, a Key vector and a Value vector through a multi-head attention mechanism.

[0059] S33: Perform inner product processing and normalization processing on the Query vector and the Key vector corresponding to the spatial information sequence and the frequency information sequence to calculate the similarity between the Query vector and the Key vector of the spatial information sequence and the frequency information sequence respectively, to obtain a target similarity of spatial information and a target similarity of frequency information.

[0060] S34: Perform inner product processing on the target similarity of spatial information and the Value vector corresponding to the spatial information sequence through the even-numbered attention head to extract the attention score corresponding to the spatial information sequence, to obtain spatial information features.

[0061] S35: Perform inner product processing on the target similarity of frequency information and the transpose of the Value vector corresponding to the frequency information sequence through the odd-numbered attention head to extract the attention score corresponding to the frequency information sequence, to obtain frequency information features.

[0062] S36: Generate the initial features based on the spatial information features and the frequency information features.

[0063] Specifically, the embodiments of the present application improve the multi-head attention MHSA of the traditional Transformer network, so that the network can efficiently process the spatial information and the frequency information in the electroencephalogram signal at the same time. In the improved multi-head attention module, part of the attention heads are used for spatial feature extraction, and the other part of the attention heads are used for frequency information extraction. The improved multi-head attention mechanism is as shown in Figure 4 In the multi-head attention MHSA module, for each self-attention head, the differential entropy feature per second is mapped into a Query vector, a Key vector and a Value vector through weight vectors W q , W k and W v . The specific formula is as follows:

[0064]

[0065] wherein X is the spatial information sequence or the frequency information sequence of the input multi-channel electroencephalogram signal, c is the number of channels of the multi-channel electroencephalogram signal, f is the number of frequency bands, n is the sequence length, d k is the dimension of the feature vector, which serves as a normalization factor to ensure the stability of the attention score, q is the Query vector, k is the Key vector, and v is the Value vector.

[0066] The Query vector and the Key vector corresponding to the spatial information sequence and the frequency information sequence are subjected to inner product processing and normalization processing to calculate the similarity between the Query vector and the Key vector of the spatial information sequence and the frequency information sequence respectively, to obtain a target similarity of spatial information and a target similarity of frequency information. The target similarity of spatial information and the Value vector corresponding to the spatial information sequence are subjected to inner product processing through the even-numbered attention head to extract the attention score corresponding to the spatial information sequence, to obtain spatial information features; the target similarity of frequency information and the transpose of the Value vector corresponding to the frequency information sequence are subjected to inner product processing through the odd-numbered attention head to extract the attention score corresponding to the frequency information sequence, to obtain frequency information features. Further, an improved multi-head attention calculation formula is provided:

[0067]

[0068] wherein i represents an odd number, j represents an even number, d k is the dimension of the feature vector. Since the conversion of the Query, Key and Value vectors of the Transformer network is a linear mapping, in the improved multi-head attention module, only the Query, Key and Value vectors of a part of the heads need to be transposed, so as to realize the extraction of the frequency domain information. Finally, by using the improved multi-head attention module, the spatial feature extraction network can fully extract the spatial information and the frequency information of the multi-channel long-time electroencephalogram signal.

[0069] Further, the above steps have obtained the attention score corresponding to each head of the multi-head attention weight, that is, the corresponding spatial information features and frequency information features, so in the embodiment of the present application, the attention scores corresponding to each head of the multi-head attention weight are spliced, and then multiplied by a linear transformation matrix to integrate the attention scores of each head, to obtain the output of the multi-head attention module MHSA. A calculation formula of the multi-head attention module MHSA is provided:

[0070] MHSA(X)=W o ×(head1,head2,…,head i );

[0071] wherein W o is a weight matrix, and head i is the attention score corresponding to different attention heads.

[0072] A feedforward network FFN is connected after the multi-head attention module MHSA. The FFN is a network layer composed of fully connected layers, which performs independent and identical nonlinear mapping on the MHSA output at each position of the time series. In the embodiment of the present application, the output result of the multi-head attention module MHSA is input into the feedforward network FFN for nonlinear mapping to obtain a nonlinear mapping result. The calculation formula is as follows:

[0073] FFN(X)=GeLU(XW1+b1)W2+b2;

[0074] Wherein, X is the input data of the feedforward network FFN; W1 and W2 are two learnable weight matrices for dimension transformation of features; b1 and b2 are two bias vectors, and GeLU is an activation function.

[0075] In the embodiment of the present application, in addition to the multi-head attention MHSA module and the feedforward network FFN, there are also residual connection modules and layer normalization modules between the MHSA and the FFN. The residual connection is used to build a deeper network, and the layer normalization is used to normalize the input data. Finally, the output H of the entire encoder network layer is:

[0076] H'=LN(MHSA(LN(X))+LN(X));

[0077] H=LN(FFN(H')+H');

[0078] The output of the entire encoder network layer is the initial feature H.

[0079] S4: extracting time sequence information in the initial feature through a time feature extraction network in the spatio-temporal feature extraction network to obtain a target feature.

[0080] Please refer to Figure 5 , Figure 5 An embodiment of step S4 is shown as follows:

[0081] S41: performing time restoration processing and dimension reduction processing on the initial feature to obtain a basic feature.

[0082] Please refer to Figure 6 , Figure 6 An embodiment of step S41 is shown as follows:

[0083] S411: restoring the initial feature to the same order of the original time series in the time dimension to obtain a restored feature.

[0084] S412: performing nonlinear mapping on the restored feature through a fully connected network to project the restored feature into a low-dimensional space to obtain the basic feature.

[0085] Specifically, the input of the time feature extraction network is the output of the spatial feature extraction network. Since the spatial feature extraction network outputs spatial features of each second, it is necessary to restore the spatial features of each second in the time dimension to a spatial feature sequence with the same order as the original time sequence. The restored spatial feature sequence is denoted as where t is the length of the time sequence, and h is obtained by merging the number of channels and the number of frequency bands. Since the dimension of h is usually high, before inputting the time feature extraction network, the embodiment of the application maps the high-dimensional features through a fully connected network for nonlinear mapping, projects them into a low-dimensional space, and obtains the basic features.

[0086] S42: A trainable parameter is added in front of the sequence corresponding to the basic feature in a random initialization manner to obtain a target sequence.

[0087] S43: The time sequence information in the target sequence is extracted through the time feature extraction network in the spatio-temporal feature extraction network to obtain a target feature.

[0088] Specifically, to enable the emotion recognition model to perform emotion classification on the extracted features, the embodiment of the application adds a randomly initialized trainable parameter CLS Token at the frontmost position of the sequence corresponding to the basic feature. The trainable parameter CLS Token is used to extract the time sequence information of the entire long EEG signal. Finally, after the time feature extraction network, the CLS Token output by the model can be used as the feature representation of the entire long EEG signal. Finally, the time sequence information in the target sequence is extracted through the time feature extraction network in the spatio-temporal feature extraction network to obtain a target feature.

[0089] S5: The classifier performs classification processing based on the target feature and calculates a classification loss, and the domain adversarial network calculates a domain discrimination loss based on the target feature.

[0090] Please refer to Figure 7 , Figure 7 An embodiment of step S5 is shown as follows:

[0091] S51: The target feature is classified by the classifier to obtain a classification label.

[0092] S52: The cross-entropy function is used as a classification loss function, and the classification label is loss calculated based on the classification loss function to obtain the classification loss.

[0093] Specifically, the target feature is classified by the classifier to obtain a classification label; a cross-entropy function is used as a classification loss function, and the classification label is calculated based on the classification loss function to obtain a classification loss wherein the classification loss is a cross-entropy classification loss used to measure the modeling capability of the emotion recognition task. The cross-entropy classification loss function is defined as:

[0094]

[0095] wherein N is the number of samples, M is the number of categories, y ij represents that the sample i belongs to the category j, p ij represents the probability that the sample i belongs to the category j.

[0096] S53: In the domain adversarial network, the target feature corresponding to the training set is taken as a source domain, and the target feature corresponding to the test set is taken as a target domain.

[0097] S54: The domain adversarial network is used to calculate a domain discrimination loss based on the source domain and the target domain to obtain the domain discrimination loss.

[0098] Specifically, since the electroencephalogram signal has significant individual difference, the electroencephalogram signals of different subjects present different emotional states under the same stimulus. Therefore, the domain adversarial network in the transfer learning is introduced into the emotion recognition model. In the domain adversarial network, the target feature corresponding to the training set is taken as a source domain (source domain), represented by D s , and the target feature corresponding to the test set is taken as a target domain (target domain), represented by D t . At the same time, the feature extractor is defined as F, and the domain discriminator is D. Therefore, the loss function of the domain discriminator is defined as:

[0099]

[0100] wherein x s is the source domain data, and x t is the target domain data. Here, the feature extractor is the spatiotemporal feature extraction network designed in the embodiments of the present application. In the embodiments of the present application, the domain label of the source domain data is defined as d=0, and the domain label of the target domain data is defined as d=1.

[0101] S6: The emotion recognition model is trained based on the classification loss and the domain discrimination loss to obtain a target emotion recognition model.

[0102] Please refer to Figure 8 , Figure 8 a specific implementation of step S6 is shown, which is described in detail as follows:

[0103] S61: Calculate a model total loss based on the classification loss and the domain discrimination loss.

[0104] S62: Train the emotion recognition model based on the model total loss in a gradient back propagation manner to obtain the target emotion recognition model.

[0105] Specifically, in the emotion recognition model training process, the gradient reversal layer R is introduced to realize the adversarial training. In the gradient reversal layer, the calculation formula of the gradient I back propagation is:

[0106]

[0107] In the training process, the model is trained as a whole by jointly optimizing the classification loss and the domain discrimination loss. Among them, the feature extractor F maximizes the classification ability of the features to improve the emotion recognition performance of the model. At the same time, the domain discriminator D reduces the distribution difference between the source domain and the target domain data by optimizing Therefore, the overall training objective function of the emotion recognition model is:

[0108]

[0109] Where λ is a balance hyperparameter to ensure the stability of the domain adversarial training. The training process λ is calculated and updated by the following formula:

[0110]

[0111] Where p is a factor related to the training round, which is calculated by the ratio between the current training round and the maximum training round.

[0112] S7: Obtain the brain electrical signal to be recognized, and perform emotion recognition on the brain electrical signal to be recognized based on the target emotion recognition model to obtain a target emotion recognition result.

[0113] Specifically, the above steps have trained the target emotion recognition model, so when emotion recognition is needed, the brain electrical signal to be recognized is obtained, and emotion recognition is performed on the brain electrical signal to be recognized based on the target emotion recognition model to obtain a target emotion recognition result.

[0114] The embodiment of the application obtains a subject data set, wherein the subject data set includes a plurality of long-time electroencephalogram signals; differential entropy features of each long-time electroencephalogram signal in the subject data set are extracted; the differential entropy features are input into a space-time feature extraction network based on a Transformer network, and spatial information and frequency information of the differential entropy features are extracted by a spatial feature extraction network in the space-time feature extraction network to obtain initial features; time sequence information in the initial features is extracted by a time feature extraction network in the space-time feature extraction network to obtain target features; a classifier performs classification processing based on the target features and calculates a classification loss, and a domain adversarial network calculates a domain discrimination loss based on the target features; an emotion recognition model is trained based on the classification loss and the domain discrimination loss to obtain a target emotion recognition model; a to-be-recognized electroencephalogram signal is obtained, and the target emotion recognition model is used for emotion recognition of the to-be-recognized electroencephalogram signal to obtain a target emotion recognition result. The embodiment of the application extracts spatial information, frequency information and time sequence information in the long-time electroencephalogram signal, and trains an emotion recognition model based on the extracted information, so that the time information and spatial information of the electroencephalogram signal can be fully utilized, and the shortcomings of the traditional Transformer are solved. In addition, the embodiment of the application introduces a domain adversarial network in transfer learning to alleviate the individual differences between different subject electroencephalogram signals, and improves the emotion recognition accuracy of the long-time electroencephalogram signal.

[0115] Please refer to Figure 9 , as an implementation of the method shown in Figure 1 , the application provides an embodiment of an emotion recognition device based on long-time electroencephalogram signals. The device embodiment corresponds to the method embodiment shown in Figure 1 , and the device can be applied to various electronic devices.

[0116] As shown in Figure 9 , the emotion recognition device based on long-time electroencephalogram signals of the embodiment includes a subject data acquisition unit 81, a differential entropy feature extraction unit 82, a spatial feature extraction unit 83, a time sequence feature extraction unit 84, a loss calculation unit 85, a model training unit 86 and an identification result generation unit 87, wherein:

[0117] The subject data acquisition unit 81 is used to obtain a subject data set, wherein the subject data set includes a plurality of long-time electroencephalogram signals;

[0118] The differential entropy feature extraction unit 82 is used to extract differential entropy features of each long-time electroencephalogram signal in the subject data set;

[0119] The spatial feature extraction unit 83 is configured to input the differential entropy feature into a spatial feature extraction network based on a Transformer network, and perform feature extraction of spatial information and frequency information on the differential entropy feature by the spatial feature extraction network in the spatial-temporal feature extraction network to obtain an initial feature.

[0120] The time sequence feature extraction unit 84 is configured to extract time sequence information in the initial feature by a time feature extraction network in the spatial-temporal feature extraction network to obtain a target feature.

[0121] The loss calculation unit 85 is configured to perform classification processing based on the target feature by a classifier and calculate a classification loss, and calculate a domain discrimination loss based on the target feature by a domain adversarial network.

[0122] The model training unit 86 is configured to train an emotion recognition model based on the classification loss and the domain discrimination loss to obtain a target emotion recognition model.

[0123] The recognition result generation unit 87 is configured to obtain a brain electrical signal to be recognized, and perform emotion recognition on the brain electrical signal to be recognized based on the target emotion recognition model to obtain a target emotion recognition result.

[0124] Further, the differential entropy feature extraction unit 82 comprises:

[0125] The target frequency band data generation unit is configured to extract, by a Fourier transform, a plurality of preset frequency band data corresponding to each channel per second of each long-time brain electrical signal in the subject data set to obtain target frequency band data.

[0126] The differential entropy feature calculation unit is configured to calculate a differential entropy feature of each frequency band in the target frequency band data by using a preset formula.

[0127] The preset formula is:

[0128]

[0129] wherein x is a multi-channel brain electrical signal of a specific length and subject to a Gaussian distribution N(mu, sigma), mu is the mean of x, sigma is the standard deviation of x, and DE is the differential entropy feature. 2

[0130] Further, the spatial feature extraction unit 83 comprises:

[0131] ​The information sequence generation unit is configured to input the differential entropy feature into a space-time feature extraction network based on a Transformer network, take different channels of the differential entropy feature at each time point as a spatial information sequence, and take different frequency bands of the differential entropy feature at each time point as a frequency information sequence.

[0132] The feature mapping unit is configured to map the differential entropy feature per second into a Query vector, a Key vector and a Value vector through a multi-head attention mechanism in the spatial feature extraction network.

[0133] The feature inner product unit is configured to perform inner product processing and normalization processing on the Query vector and the Key vector corresponding to the spatial information sequence and the frequency information sequence, to calculate the similarity between the Query vector and the Key vector of the spatial information sequence and the frequency information sequence respectively, to obtain a target similarity of spatial information and a target similarity of frequency information.

[0134] The spatial information extraction unit is configured to perform inner product processing on the target similarity of spatial information and the Value vector corresponding to the spatial information sequence through an even-numbered attention head, to extract an attention score corresponding to the spatial information sequence, and to obtain a spatial information feature.

[0135] The frequency information extraction unit is configured to perform inner product processing on the target similarity of frequency information and the transpose of the Value vector corresponding to the frequency information sequence through an odd-numbered attention head, to extract an attention score corresponding to the frequency information sequence, and to obtain a frequency information feature.

[0136] The feature merging unit is configured to generate the initial feature based on the spatial information feature and the frequency information feature.

[0137] Further, the time sequence feature extraction unit 84 includes:

[0138] The basic feature generation unit is configured to perform time restoration processing and dimension reduction processing on the initial feature, to obtain a basic feature.

[0139] The target sequence generation unit is configured to add a trainable parameter in front of a sequence corresponding to the basic feature in a random initialization manner, to obtain a target sequence.

[0140] The target feature generation unit is configured to extract time sequence information in the target sequence through a time feature extraction network in the space-time feature extraction network, to obtain a target feature.

[0141] Further, the basic feature generation unit includes:

[0142] a feature reduction unit, configured to reduce the initial features in a time dimension to a same order of an original time sequence to obtain reduced features;

[0143] a feature dimension reduction unit, configured to perform nonlinear mapping on the reduced features by a fully connected network to project the reduced features into a low-dimensional space to obtain the basis features.

[0144] Further, the loss calculation unit 85 comprises:

[0145] a classification processing unit, configured to perform classification processing on the target features by the classifier to obtain a classification label;

[0146] a classification loss calculation unit, configured to take a cross-entropy function as a classification loss function and perform loss calculation on the classification label based on the classification loss function to obtain a classification loss.

[0147] a data division unit, configured to take the target features corresponding to the training set as a source domain and take the target features corresponding to a test set as a target domain in the domain adversarial network;

[0148] a domain discrimination loss calculation unit, configured to perform domain discrimination loss calculation based on the source domain and the target domain by the domain adversarial network to obtain a domain discrimination loss.

[0149] Further, the model training unit 86 comprises:

[0150] a model loss calculation unit, configured to calculate a model total loss based on the classification loss and the domain discrimination loss;

[0151] a target emotion recognition model generation unit, configured to train the emotion recognition model based on the model total loss in a gradient back propagation manner to obtain the target emotion recognition model, wherein the emotion recognition model comprises the spatio-temporal feature extraction network, the classifier and the domain discriminator.

[0152] To solve the above technical problems, the embodiment of the present application further provides a computer device. For details, please refer to Figure 10 , Figure 10 The basic structure block diagram of the computer device of the present embodiment is shown in FIG. 1.

[0153] The computer device 9 comprises a memory 91, a processor 92, and a network interface 93 which are communicatively connected by a system bus. It should be noted that the computer device 9 is only shown with three components, i.e., the memory 91, the processor 92, and the network interface 93, but it should be understood that not all the shown components are required to be implemented, and more or less components can be alternatively implemented. Among them, those skilled in the art can understand that the computer device herein is a device capable of automatically performing numerical calculation and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0154] The computer device can be a desktop computer, a notebook computer, a palm computer, a cloud server, or the like. The computer device can interact with a user through a keyboard, a mouse, a remote controller, a touchpad, a voice control device, or the like.

[0155] The memory 91 comprises at least one type of readable storage medium, including a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 91 can be an internal storage unit of the computer device 9, such as a hard disk or a memory of the computer device 9. In other embodiments, the memory 91 can also be an external storage device of the computer device 9, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device 9. Of course, the memory 91 can also include both the internal storage unit and the external storage device of the computer device 9. In the present embodiment, the memory 91 is generally used to store an operating system and various application software installed on the computer device 9, such as program codes of the emotion recognition method based on long-time electroencephalogram signals, etc. In addition, the memory 91 can also be used to temporarily store various data that have been output or will be output.

[0156] The processor 92 may, in some embodiments, be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 92 is generally used to control the overall operation of the computer device 9. In the present embodiment, the processor 92 is configured to run program codes or process data stored in the memory 91, such as program codes of the emotion recognition method based on long-time electroencephalogram signals described above, to implement various embodiments of the emotion recognition method based on long-time electroencephalogram signals.

[0157] The network interface 93 may include a wireless network interface or a wired network interface, and is generally used to establish a communication connection between the computer device 9 and other electronic devices.

[0158] The present application also provides another embodiment, i.e., to provide a computer readable storage medium storing a computer program, which can be executed by at least one processor to make the at least one processor execute the steps of an emotion recognition method based on long-time electroencephalogram signals as described above.

[0159] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be realized by means of software and necessary general hardware platforms, of course, they can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for making a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) execute the methods of various embodiments of the present application.

[0160] Obviously, the above-described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application, and the preferred embodiments of the present application are given in the drawings, but do not limit the patent scope of the present application. The present application can be realized in many different forms, and conversely, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions recorded in the foregoing specific embodiments, or make equivalent replacements to some technical features. Any equivalent structure made by referring to the contents of the specification and drawings, directly or indirectly applied to other related technical fields, is also within the scope of the patent protection of the present application.

Claims

1. An emotion recognition method based on long-term electroencephalogram (EEG) signals, characterized in that, include: Obtain a subject dataset, wherein the subject dataset includes multiple long-term electroencephalogram (EEG) signals; Extract the differential entropy features of each long-term EEG signal from the subject dataset; The differential entropy features are input into a spatiotemporal feature extraction network based on a Transformer network, and the spatial and frequency information features of the differential entropy features are extracted through the spatial feature extraction network in the spatiotemporal feature extraction network to obtain the initial features; The temporal information in the initial features is extracted by the temporal feature extraction network in the spatiotemporal feature extraction network to obtain the target features; The target features are classified using a classifier, and the classification loss is calculated. The target features are also classified using a domain adversarial network, and the domain discrimination loss is calculated using a domain adversarial network. The emotion recognition model is trained based on the classification loss and the domain discrimination loss to obtain the target emotion recognition model. Acquire the EEG signal to be identified, and perform emotion recognition on the EEG signal to be identified based on the target emotion recognition model to obtain the target emotion recognition result; The process involves inputting the differential entropy features into a Transformer-based spatiotemporal feature extraction network, and then using the spatial feature extraction network within that network to extract spatial and frequency information from the differential entropy features to obtain initial features, including: The differential entropy features are input into a spatiotemporal feature extraction network based on a Transformer network. The different channels of the differential entropy features at each time point are taken as a spatial information sequence, and the different frequency bands of the differential entropy features at each time point are taken as a frequency information sequence. In the spatial feature extraction network, the differential entropy features per second are mapped into Query vector, Key vector, and Value vector through a multi-head attention mechanism; The Query vectors and Key vectors corresponding to the spatial information sequence and the frequency information sequence are processed by inner product and normalization to calculate the similarity between the Query vectors and Key vectors of the spatial information sequence and the frequency information sequence, so as to obtain the target similarity of spatial information and the target similarity of frequency information. By using an even-numbered attention head, the target similarity of the spatial information and the value vector corresponding to the spatial information sequence are processed by inner product to extract the attention score corresponding to the spatial information sequence and obtain the spatial information features. By using the attention head with an odd number of terms, the inner product of the target similarity of the frequency information and the transpose of the value vector corresponding to the frequency information sequence is performed to extract the attention score corresponding to the frequency information sequence and obtain the frequency information features. The initial features are generated based on the spatial information features and the frequency information features.

2. The emotion recognition method based on long-term EEG signals according to claim 1, characterized in that, The extraction of differential entropy features of each long-term EEG signal in the subject dataset includes: By using Fourier transform, multiple preset frequency band data corresponding to each channel per second of each long-term EEG signal in the subject dataset are extracted to obtain target frequency band data. The differential entropy characteristics of each frequency band in the target frequency band data are calculated using a preset formula; The preset formula is: Where x is a segment of a specific length that follows a Gaussian distribution N(μ,σ). 2 The multichannel long-term EEG signal is given, where μ is the mean of x, σ is the standard deviation of x, and DE is the differential entropy feature.

3. The emotion recognition method based on long-term EEG signals according to claim 1, characterized in that, The step of extracting temporal information from the initial features through the temporal feature extraction network in the spatiotemporal feature extraction network to obtain the target features includes: The initial features are subjected to time restoration and dimensionality reduction processing to obtain the basic features; The target sequence is obtained by adding trainable parameters to the sequence corresponding to the basic features using random initialization. The temporal information in the target sequence is extracted by the temporal feature extraction network in the spatiotemporal feature extraction network to obtain the target features.

4. The emotion recognition method based on long-term EEG signals according to claim 3, characterized in that, The initial features are subjected to time restoration and dimensionality reduction processing to obtain basic features, including: The initial features are restored to the same order as the original time series in the time dimension to obtain the restored features; The restored features are nonlinearly mapped using a fully connected network to project them into a low-dimensional space, thus obtaining the basic features.

5. The emotion recognition method based on long-term electroencephalogram (EEG) signals according to any one of claims 1 to 4, characterized in that, The subject dataset includes a training set and a test set. The process of classifying the data based on the target features using a classifier and calculating the classification loss, and calculating the domain discriminant loss based on the target features using a domain adversarial network, includes: The target features are classified using the classifier to obtain classification labels; The cross-entropy function is used as the classification loss function, and the classification loss is calculated based on the classification loss function to obtain the classification loss; In the domain adversarial network, the target features corresponding to the training set are used as the source domain, and the target features corresponding to the test set are used as the target domain. The domain discrimination loss is obtained by calculating the domain discrimination loss based on the source domain and the target domain using the domain adversarial network.

6. The emotion recognition method based on long-term EEG signals according to any one of claims 1 to 4, characterized in that, The process of training the emotion recognition model based on the classification loss and the domain discrimination loss to obtain the target emotion recognition model includes: The total loss of the model is calculated based on the classification loss and the domain discrimination loss. The emotion recognition model is trained using gradient backpropagation based on the total loss of the model to obtain the target emotion recognition model, wherein the emotion recognition model includes the spatiotemporal feature extraction network, the classifier, and the domain discriminator.

7. An emotion recognition device based on long-term electroencephalogram (EEG) signals, characterized in that, include: The subject data acquisition unit is used to acquire a subject dataset, wherein the subject dataset includes multiple long-term electroencephalogram (EEG) signals; The differential entropy feature extraction unit is used to extract the differential entropy features of each long-term EEG signal in the subject dataset; The spatial feature extraction unit is used to input the differential entropy features into the spatiotemporal feature extraction network based on the Transformer network, and to extract spatial and frequency information features from the differential entropy features through the spatial feature extraction network in the spatiotemporal feature extraction network to obtain initial features; The temporal feature extraction unit is used to extract temporal information from the initial features through the temporal feature extraction network in the spatiotemporal feature extraction network to obtain the target features; The target features are classified using a classifier, and the classification loss is calculated. The target features are also classified using a domain adversarial network, and the domain discrimination loss is calculated using a domain adversarial network. The model training unit is used to train the emotion recognition model based on the classification loss and the domain discrimination loss to obtain the target emotion recognition model. The recognition result generation unit is used to acquire the EEG signal to be recognized, and perform emotion recognition on the EEG signal to be recognized based on the target emotion recognition model to obtain the target emotion recognition result; The spatial feature extraction unit includes: The information sequence generation unit is used to input the differential entropy feature into a spatiotemporal feature extraction network based on the Transformer network, and to take different channels of the differential entropy feature at each time point as a spatial information sequence, and take different frequency bands of the differential entropy feature at each time point as a frequency information sequence. The feature mapping unit is used in the spatial feature extraction network to map the differential entropy features per second into Query vector, Key vector and Value vector through a multi-head attention mechanism. The feature inner product unit is used to perform inner product processing and normalization processing on the Query vector and Key vector corresponding to the spatial information sequence and the frequency information sequence, so as to calculate the similarity between the Query vector and Key vector of the spatial information sequence and the frequency information sequence respectively, and obtain the target similarity of spatial information and the target similarity of frequency information. The spatial information extraction unit is used to perform inner product processing on the target similarity of the spatial information and the value vector corresponding to the spatial information sequence through the attention head of even-numbered terms, so as to extract the attention score corresponding to the spatial information sequence and obtain the spatial information features. The frequency information extraction unit is used to perform inner product processing on the target similarity of the frequency information and the transpose of the value vector corresponding to the frequency information sequence through the attention head of the odd-numbered terms, so as to extract the attention score corresponding to the frequency information sequence and obtain the frequency information features. The feature merging unit is used to generate the initial feature based on the spatial information feature and the frequency information feature.

8. A computer device, characterized in that, The method includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the emotion recognition method based on long-term EEG signals as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the emotion recognition method based on long-term electroencephalogram (EEG) signals as described in any one of claims 1 to 6.

Images