Emotion recognition method, device and equipment based on electroencephalogram signals and medium

Abstract

The invention relates to the field of artificial intelligence technology and medical treatment, and discloses an emotion recognition method, device and equipment based on electroencephalogram signals and a medium. The method comprises the steps that the electroencephalogram signals in a training data set are input into a feature encoder used for representing emotions and subjected to discretization processing; performing masking processing on the obtained target electroencephalogram feature sequence, and inputting the obtained mask electroencephalogram feature sequence into an electroencephalogram feature reconstruction model to obtain a predicted electroencephalogram feature sequence; constructing a target loss function based on the predicted and target electroencephalogram feature sequence, and training and updating the feature reconstruction model through a back propagation algorithm; and inputting a to-be-classified electroencephalogram signal into the trained feature reconstruction model, and then inputting an obtained electroencephalogram feature sequence into a pre-trained emotion classifier to obtain an emotion classification tag. The problem that in the prior art, the prediction result is not accurate enough when the emotion is predicted through electroencephalogram signals is solved.

Description

Technical Field

[0001] The present invention relates to artificial intelligence technology and medical fields, and in particular to an emotion recognition method, device, equipment and medium based on electroencephalogram signals. Background Art

[0002] In the medical industry, using EEG signals to predict emotions is a cutting-edge and challenging technology. This technology is mainly based on EEG signal acquisition equipment such as electroencephalogram (EEG), which identifies the user's emotional state by analyzing and interpreting the brain's electrical signals. EEG signals are spontaneous electrical signals generated by human brain consciousness activities. The information of brain activity extracted from EEG signals can be used as an indicator of human brain ideology. The thinking activities of the human brain are mainly reflected in the changes in the frequency of EEG signals. Different thinking activities of the human brain are distinguished by different frequency changes.

[0003] Using EEG signals to identify emotions helps to better understand the brain's perception of stimuli, and lays a solid foundation for the future realization of EEG human-computer interface and deeper research on human emotions. Smart wearable devices can identify human emotions based on EEG signals, while providing emotional value, completing deep human-computer interaction, greatly improving user experience, alleviating emotional stress, and reducing the incidence of mental health diseases caused by long-term emotional problems, so that human society can develop harmoniously and stably.

[0004] The process of collecting, cleaning and preprocessing EEG signals is very complicated, time-consuming and labor-intensive, so it is usually difficult to obtain labeled data for EEG signals. The amount of data in existing data sets is not enough to support the training process of large models (such as Transformer-based feature extractors), which makes the existing large models lack sufficient training data, resulting in inaccurate prediction results when predicting emotions through EEG signals. Summary of the invention

[0005] The present invention provides an emotion recognition method, device, computer equipment and medium based on electroencephalogram signals to solve the problem that the prediction results of emotions predicted by electroencephalogram signals in the prior art are not accurate enough.

[0006] In a first aspect, a method for emotion recognition based on electroencephalogram signals is provided, comprising: Obtain a training data set of EEG signals; Inputting the EEG signals in the training data set into an EEG signal feature encoder for representing emotions to obtain an initial EEG feature sequence; performing discretization processing on the initial EEG feature sequence to obtain a target EEG feature sequence; After masking a part of the feature sequence in the target EEG feature sequence, a masked EEG feature sequence is obtained, and the masked EEG feature sequence is input into an EEG signal feature reconstruction model to obtain a predicted EEG feature sequence; a target loss function is constructed based on the predicted EEG feature sequence and the target EEG feature sequence; and the EEG signal feature reconstruction model is trained and updated according to the target loss function to obtain a trained EEG signal feature reconstruction model; The EEG signal to be classified is input into a trained EEG signal feature reconstruction model to obtain an EEG feature sequence to be classified, and the EEG feature sequence to be classified is input into a pre-trained emotion classifier to obtain an emotion classification label.

[0007] In a second aspect, an emotion recognition device based on electroencephalogram signals is provided, comprising: An acquisition module is used to acquire a training data set of EEG signals; A discrete processing module, used for inputting the EEG signals in the training data set into an EEG signal feature encoder for representing emotions to obtain an initial EEG feature sequence; and obtaining a target EEG feature sequence after discretizing the initial EEG feature sequence; A training module is used to mask a part of the feature sequence in the target EEG feature sequence to obtain a masked EEG feature sequence, input the masked EEG feature sequence into an EEG signal feature reconstruction model to obtain a predicted EEG feature sequence; construct a target loss function based on the predicted EEG feature sequence and the target EEG feature sequence; train and update the EEG signal feature reconstruction model according to the target loss function to obtain a trained EEG signal feature reconstruction model; The classification module is used to input the EEG signal to be classified into a trained EEG signal feature reconstruction model to obtain an EEG feature sequence to be classified, and input the EEG feature sequence to be classified into a pre-trained emotion classifier to obtain an emotion classification label.

[0008] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned method for emotion recognition based on EEG signals when executing the computer program.

[0009] In a fourth aspect, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps of the above-mentioned emotion recognition method based on EEG signals are implemented.

[0010] In the scheme implemented by the above-mentioned EEG signal-based emotion recognition method, device, computer equipment and storage medium, a masked EEG feature sequence can be obtained by masking part of the feature sequence in the target EEG feature sequence, and the masked EEG feature sequence can be input into the EEG signal feature reconstruction model to obtain a predicted EEG feature sequence; a target loss function is constructed based on the predicted EEG feature sequence and the target EEG feature sequence; the EEG signal feature reconstruction model is trained and updated through the back propagation algorithm according to the target loss function to obtain a trained EEG signal feature reconstruction model; the EEG signal to be classified is input into the trained EEG signal feature reconstruction model and the pre-trained emotion classifier in turn to obtain an emotion classification label.

[0011] The present invention randomly masks the feature sequences of some EEG signal frames, and allows an EEG signal feature reconstruction model to restore the feature sequences of the masked frames according to context information, constructs a target loss function by predicting the EEG feature sequence and the real target EEG feature sequence, and trains and updates the parameters of the EEG signal feature reconstruction model through the target loss function to obtain a trained EEG signal feature reconstruction model, so that the EEG signal feature reconstruction model learns some information related to the features of the EEG signal itself but irrelevant to the specific task, and uses the key feature information of the EEG signal itself learned by the EEG signal feature reconstruction model to perform emotion classification prediction on the EEG signal, thereby improving the accuracy of the emotion classification prediction results, thereby solving the problem of inaccurate prediction results when predicting emotions through EEG signals in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0013] Figure 1 is a schematic diagram of an application environment of an emotion recognition method based on EEG signals in one embodiment of the present invention; Figure 2 is a schematic diagram of the overall process of an emotion recognition method based on EEG signals in one embodiment of the present invention; Figure 3 is a schematic diagram of a specific flow chart of an emotion recognition method based on EEG signals in one embodiment of the present invention; Figure 4 is a schematic diagram of a structure of an emotion recognition device based on EEG signals in one embodiment of the present invention; Figure 5 It is a structural diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0014] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0015] The speech synthesis method for the insurance business terminal provided by the embodiment of the present invention can be applied in Figure 1 In an application environment, the application environment includes a client and a server, wherein the client can communicate with the server through a network. Specifically, taking the server as an example, the server obtains a training data set of EEG signals; the EEG signals in the training data set are input into an EEG signal feature encoder used to characterize emotions to obtain an initial EEG feature sequence; the initial EEG feature sequence is discretized to obtain a target EEG feature sequence; part of the feature sequence in the target EEG feature sequence is masked to obtain a masked EEG feature sequence, and the masked EEG feature sequence is input into an EEG signal feature reconstruction model to obtain a predicted EEG feature sequence; a target loss function is constructed based on the predicted EEG feature sequence and the target EEG feature sequence; the EEG feature sequence is reconstructed using a back-propagation algorithm according to the target loss function. The EEG signal feature reconstruction model is trained and updated to obtain a trained EEG signal feature reconstruction model; the EEG signal to be classified is input into an EEG signal feature encoder used to characterize emotions to obtain a first EEG feature sequence, the EEG feature sequence is discretized to obtain a second EEG feature sequence, the second EEG feature sequence is input into the trained EEG signal feature reconstruction model to obtain a third EEG feature sequence, the third EEG feature sequence is input into a pre-trained emotion classifier to obtain an emotion classification label, and the emotion classification label is sent to the terminal client; wherein the client is also called the user end, which refers to a program corresponding to the server that provides local services to customers. The client can be installed on, but not limited to, various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server can be implemented as an independent server or a server cluster consisting of multiple servers.

[0016] See also Figure 2 As shown, Figure 2 A flowchart of an emotion recognition method based on EEG signals provided by an embodiment of the present invention includes the following steps: S01: Obtain a training dataset of EEG signals.

[0017] Among them, EEG signals are electrical signals generated when the human brain is active. These signals are detected by electrodes placed on the scalp and converted into graphic or digital form.

[0018] EEG signals are very weak, generally only 10~50μV, and the maximum is only 100μV, with a frequency between 0.1~100Hz or 0.5~100Hz. In addition, as a window to reflect brain activity, EEG signals need to be conducted through several tissues such as cerebrospinal fluid, skull and scalp. The EEG signals, which are not strong to begin with, are weakened layer by layer (especially the weak conductive medium of the skull), and are extremely weak (uV level) when they reach the recording electrode.

[0019] EEG signals are easily affected by other electrophysiological signals, such as motion artifacts caused by eye movement, electromyography, blinking or facial muscle activity, etc. At the same time, EEG signals are also easily interfered by background noise and various artifacts, such as power frequency interference of the mains, electromagnetic interference in space, etc.

[0020] The main device for collecting EEG signals is an electroencephalograph, and its working principle is: electrodes placed on the scalp can detect weak EEG signals, which are then amplified, processed and digitized, and finally recorded through a matching recording system.

[0021] In this embodiment, the prepared emotion-inducing material is provided for the subject to watch. In daily scenarios, as long as the subject has an EEG acquisition device, the EEG signal data of the subject can be collected through the EEG acquisition device. Specifically, the EEG acquisition device can use a wet electrode EEG cap to collect EEG signals. In this way, a large number of EEG signals of the subjects can be obtained in daily scenarios as training data for training emotion recognition models, and then used for emotion recognition in various fields. For example, the daily scene can be that the subject sits quietly in the room, and a display is placed in front of the subject to play an emotion-inducing video for the subject to watch. At this time, the EEG signal is collected based on the EEG acquisition device.

[0022] For step S11, after obtaining the EEG signals of the subjects, considering that these EEG signals originate from daily scenes, in order to further improve the accuracy, it is necessary to pre-process the EEG signals of the subjects by filtering and reducing noise. In this embodiment, the pre-processing includes baseline removal, downsampling, filtering and other processing of the collected EEG data. The details are as follows: 1) Baseline removal aims to eliminate DC offset or baseline drift in EEG signals, which may be caused by factors such as equipment instability, ambient temperature changes, or physiological changes of the subject (such as breathing, heartbeat). This implementation achieves baseline removal by calculating the mean or median of the entire signal and then subtracting this value from each time point. Sometimes, more complex methods such as adaptive filtering or linear prediction can also be used to remove baseline drift.

[0023] 2) Downsampling is to reduce the sampling rate of EEG signals, thereby reducing the amount of data and improving computing efficiency. This is particularly important for real-time processing or resource-constrained environments. In this embodiment, downsampling is achieved by retaining certain time points in the signal (such as taking one point every n points).

[0024] 3) Filtering is to remove noise and interference in the EEG signal, which may come from power lines, muscle activity, eye movement, etc. Filtering helps to improve the signal-to-noise ratio of the signal, making subsequent analysis more accurate. In this embodiment, first, the 50Hz AC power noise in the EEG signal is removed; then, a 1-75Hz bandpass filter is used to remove low-frequency and high-frequency invalid signals in the EEG signal.

[0025] S02: Inputting the EEG signals in the training data set into an EEG signal feature encoder for representing emotions to obtain an initial EEG feature sequence; discretizing the initial EEG feature sequence to obtain a target EEG feature sequence.

[0026] In this embodiment, the EEG signal feature encoder used to characterize emotions is a network model based on multi-layer one-dimensional convolution. Specifically, after a series of preprocessing processes (such as baseline removal, downsampling, filtering, etc.) for the EEG signal, the feature encoder based on the convolutional network encodes the time series of the original EEG signal into a feature sequence (initial EEG feature sequence). In this embodiment, the features of each frame in the original EEG signal are converted into discrete features through the VQ module and used as the target of self-supervised pre-training. The VQ (vector quantization) module is an important component of the SciPy library, and is mainly used in fields such as data clustering, signal processing, and deep learning. The core idea of ​​the VQ module is to convert continuous feature vectors into discrete representations. This is usually achieved through a predefined codebook, which contains multiple discrete vectors (or codewords). During the conversion process, the VQ module calculates the distance between the input feature vector and each codeword in the codebook, and selects the codeword with the smallest distance as the discrete representation of the feature vector.

[0027] Specifically, the VQ module converts the analog signal value of the original EEG signal feature sequence into a digital signal. That is, through the vector quantization method, the continuous signal value is mapped to a discrete codebook to obtain the target EEG feature sequence, thereby realizing the digital processing of the signal.

[0028] S03: After masking part of the feature sequence in the target EEG feature sequence, a masked EEG feature sequence is obtained, and the masked EEG feature sequence is input into an EEG signal feature reconstruction model to obtain a predicted EEG feature sequence; a target loss function is constructed based on the predicted EEG feature sequence and the target EEG feature sequence; the EEG signal feature reconstruction model is trained and updated according to the target loss function to obtain a trained EEG signal feature reconstruction model.

[0029] In this embodiment, a mask training method is used to randomly mask the feature sequences of some continuous EEG signal frames, and let the EEG signal feature reconstruction model restore these masked frames according to context information, so that the model can learn the features of the EEG signals.

[0030] Among them, the masked training method is an important technique in deep learning. It masks part of the information by applying a mask to the input data or model features, thereby forcing the model to learn to reconstruct or understand the masked content.

[0031] The basic principle of masked training is to block part of the information by randomly or selectively applying masks to the input data or features. In this way, the model must learn to reconstruct or infer the masked content from the remaining information during training. This method can increase the generalization ability of the model, enabling it to better handle unseen data.

[0032] Specifically, the above-mentioned target EEG feature sequence is trained using a masking method, and a masked EEG feature sequence is obtained by randomly masking the feature sequences of some continuous EEG signal frames; then the masked EEG feature sequence is input into the EEG signal feature reconstruction model. The EEG signal feature reconstruction model in this embodiment is a method that can generate a context representation (predicted EEG feature sequence) by extracting the features of the EEG signal in the masked EEG feature sequence, and then restore these masked feature sequences according to the context information, so that the EEG signal feature reconstruction model learns the features of the EEG signal. The model learns by contrast learning, using a loss function to shorten the distance between the context representation of the masked position and the discrete features corresponding to the position (target EEG feature sequence), that is, a target loss function is constructed according to the distance between the predicted EEG feature sequence and the target EEG feature sequence; the EEG signal feature reconstruction model is trained and updated through a back propagation algorithm according to the target loss function, so as to obtain a trained EEG signal feature reconstruction model.

[0033] S04: Inputting the EEG signal to be classified into a trained EEG signal feature reconstruction model to obtain an EEG feature sequence to be classified, and inputting the EEG feature sequence to be classified into a pre-trained emotion classifier to obtain an emotion classification label.

[0034] After the EEG signal feature reconstruction model is trained in the above step S03, the trained EEG signal feature reconstruction model can be used for EEG signal emotion recognition. However, it is necessary to connect a classifier for emotion classification behind the trained EEG signal feature reconstruction model, and the parameters of the classifier are also the trained emotion classifier. The EEG signal to be classified in this embodiment is the EEG signal of the target subject who currently needs to perform emotion recognition. It is also necessary to perform baseline removal, downsampling, filtering and other processing on the EEG signal of the target subject in step S01.

[0035] Specifically, the EEG signal to be classified is input into an EEG signal feature encoder used to characterize emotions, the time series of the original EEG signal is encoded into a first EEG feature sequence, the EEG feature sequence is discretized through the VQ module in step S02 to obtain a second EEG feature sequence, the second EEG feature sequence is input into a trained EEG signal feature reconstruction model to perform feature reasoning and extraction of the EEG signal to obtain a third EEG feature sequence, and finally the third EEG feature sequence is input into a pre-trained emotion classifier to obtain an emotion classification label corresponding to the EEG signal of the target subject.

[0036] This application takes the scenario of a smart wearable device that can perform emotion recognition in a medical device as an example to illustrate, and applies the smart wearable device to the above-mentioned steps S01, S02 and S03. Specifically, the smart wearable device includes an EEG acquisition device, and a wet electrode EEG cap can be used to collect EEG signals. The EEG acquisition device is worn on the subject's head to obtain a large amount of EEG signals of the subject as training data for training the emotion recognition model; Figure 3As shown, since the model used in the solution of the present invention has a basic structural design similar to a self-supervised learning framework for speech representation (wav2vec2.0), the wav2vec2.0 model is an audio pre-training model based on self-supervised learning, which learns the deep representation of audio through training on large-scale unlabeled audio data. Compared with traditional supervised learning methods, self-supervised learning can use unlabeled data for pre-training, so that the model can achieve better performance on limited labeled data. The wav2vec2.0-like structure used in the present invention also includes two major parts: a feature encoder (brain wave signal feature encoder) and a context network (brain wave signal feature reconstruction model); the brain wave signal in the training data set is input into the brain wave signal feature encoder used to characterize emotions to obtain an initial brain wave feature sequence; the initial brain wave feature sequence is discretized to obtain a target brain wave feature sequence; a part of the feature sequence in the target brain wave feature sequence is masked to obtain a masked brain wave feature sequence, and the masked brain wave feature sequence is input into the brain wave signal feature reconstruction model to obtain a predicted brain wave feature sequence; a target loss function is constructed based on the predicted brain wave feature sequence and the target brain wave feature sequence; the brain wave signal feature reconstruction model is trained and updated by the back propagation algorithm according to the target loss function to obtain a trained brain wave signal feature reconstruction model. Similarly, the brain wave acquisition device is worn on the head of the target subject to obtain the brain wave signal to be classified; the brain wave signal to be classified is input into the trained brain wave signal feature reconstruction model to obtain the brain wave feature sequence to be classified, and the brain wave feature sequence to be classified is input into the pre-trained emotion classifier to obtain an emotion classification label.

[0037] The solution proposed in the present invention solves the problem of insufficient EEG signal annotated data by using a masked training method, so that when a model is trained with a small amount of data, the model is forced to learn the overall structure of the data in the absence of some information, thereby improving the generalization ability of the model. When performing emotion recognition on EEG signals based on the important data features of the learned brain signals, the accuracy of the emotion classification results can be improved; compared with other EEG signal classification methods that use pre-training, the solution proposed in the present invention can use task-related and task-independent EEG signal data to learn common features, and the model converges faster during the training process.

[0038] In one embodiment, masking a portion of the target EEG feature sequence includes: According to the target EEG feature sequence, a mask sequence having the same number of rows and columns as the target EEG feature sequence is generated; the mask sequence includes mask values ​​and non-mask values.

[0039] The data at the corresponding position in the target feature sequence is reset to zero or replaced with a preset character according to the mask value in the mask sequence.

[0040] The data at the corresponding position in the target feature sequence is retained according to the non-mask value in the mask sequence.

[0041] It is important to understand that random masking refers to randomly selecting a portion of the original data to mask or discard, thereby changing certain characteristics of the data or for specific data processing tasks. In deep learning, this usually means applying a random mask to the input data (such as images, audio, or text) to block or select some specific elements or areas. Random masking specifically includes the following steps: Based on the type and shape of the data, a random mask with the same shape as the data is generated. Each element in the mask can be a binary value (0 or 1), indicating whether the corresponding original data element is masked.

[0042] Apply the generated random mask to the original data. For the elements covered by the mask, you can choose to discard, replace with a specific value (such as zero), or perform other processing.

[0043] The data after random masking will be used for subsequent deep learning model training or signal processing tasks.

[0044] Specifically, based on the above-mentioned target EEG feature sequence, a mask sequence with the same data dimension size as the target EEG feature sequence is generated; wherein the specific numerical values ​​in the mask sequence are divided into mask values ​​and non-mask values; in this embodiment, a mask sequence with the same data dimension size as the target EEG feature sequence is generated by random generation, for example, the target EEG feature sequence is a sequence of 128 rows * 256 columns, and the generated mask sequence must also be a sequence of 128 rows * 256 columns.

[0045] In this embodiment, the mask value in the mask sequence is set to 0, and the non-mask value is set to 1; according to the mask value 0 in the mask sequence, the data at the same position in the target feature sequence is set to zero, or it can be replaced with a special character. According to the non-mask value 1 in the mask sequence, the data at the corresponding position in the target feature sequence is retained. The masking formula corresponding to the above masking method after masking is: in, is the target EEG feature sequence, is the mask sequence, C is the mask EEG feature sequence, and the matrix operator is the Hadamard product, i={1, 2, 3, ..., M}, j={1, 2, 3, ..., N}, M is the number of rows of sequences A, B and C, and N is the number of columns of sequences A, B and C.

[0046] In one embodiment, the training process of the pre-trained emotion classifier includes: Inputting a training data set of EEG signals with real values ​​of emotion classification labels into a trained EEG signal feature reconstruction model to obtain a first predicted EEG feature, and inputting the first predicted EEG feature into the emotion classifier to obtain a predicted value of the emotion classification label; A first loss function is constructed according to the true value and the predicted value of the emotion classification label.

[0047] The first loss function value is minimized by iteratively updating the parameters of the emotion classifier to obtain the optimal parameter setting of the emotion classifier.

[0048] In this embodiment, the K-nearest neighbor (KNN) classifier is selected to perform emotion recognition on different features. Other commonly used emotion classifiers may also be selected, such as neural network, support vector machine and logistic regression, among which the KNN classifier has significant advantages in processing small sample data and is better applied in real-time detection and recognition.

[0049] KNN (K-Nearest Neighbors) is a simple but very effective classification algorithm. Its working principle is: given a test sample, the KNN algorithm finds the K samples (i.e., "neighbors") that are most similar to the sample in the training data set, and predicts the category of the test sample based on the categories of these neighbors.

[0050] The emotion classification labels of this embodiment mainly include seven types: fear, anger, disgust, preference, sadness, happiness and surprise. Specifically, the training data set of EEG signals with the real values ​​of the above emotion classification labels is input into the trained EEG signal feature reconstruction model to obtain the first predicted EEG feature.

[0051] The first predicted EEG feature is input into the KNN classifier, specifically including: First choose the K value: Choose an appropriate K value.

[0052] Calculate the distance between the first predicted EEG feature of the input and all classified training data. Commonly used distance measurement methods include Euclidean distance, Manhattan distance, etc.

[0053] According to the distance sorting, the K training data with the closest distance are selected as neighbors.

[0054] According to the emotion classification categories of the neighbors, the emotion classification category of the first predicted EEG feature is determined by majority voting principle. That is, the emotion classification category with the most occurrences among the K neighbors is selected as the predicted value of the emotion classification label of the first predicted EEG feature.

[0055] Calculate the proportion of training data correctly classified by the classifier to the total training data, that is, the accuracy. If the accuracy does not meet the requirements, you can adjust the K value, distance measurement method and other parameters, and retrain the classifier. In this embodiment, the classifier can also be composed of multiple layers of fully connected layers. At this time, it is necessary to construct a first loss function based on the true value and predicted value of the emotion classification label; minimize the first loss function value by iteratively updating the parameters of the emotion classifier to obtain the optimal parameter setting of the emotion classifier.

[0056] In one embodiment, constructing a target loss function based on the predicted EEG feature sequence and the target EEG feature sequence includes: A distance function is constructed based on the cosine similarity between the predicted EEG feature sequence and the target EEG feature sequence.

[0057] The target loss function is constructed based on the distance function through contrast loss.

[0058] Specifically, a distance function is constructed according to the cosine similarity between the predicted EEG feature sequence and the target EEG feature sequence, and a target loss function is constructed based on the distance function through contrast loss. The specific calculation formula of the target loss function is as follows: Where: y is the label, the label of the positive sample pair is usually 1, indicating that the two samples are similar. The label of the negative sample pair is usually 0, indicating that the two samples are dissimilar.

[0059] D is the cosine similarity between the predicted EEG feature sequence sample and the target EEG feature sequence sample (in the range of [-1, 1]). It is set that when D is greater than the preset value, they are similar, otherwise they are not similar.

[0060] m is a threshold, usually set to a value between 0 and 1 (such as 0.5), which is used to control the minimum distance between positive and negative sample pairs.

[0061] In one embodiment, the EEG signal feature encoder is a feature encoder based on a convolutional network, and the EEG signal feature reconstruction model is a deep learning model based on a self-attention mechanism.

[0062] Specifically, Figure 3As shown, this implementation uses a basic structure design similar to wav2vec2.0, which consists of a data preprocessing step, an improved wav2vec2.0-like structure, and a classifier for a specific task. The design of the wav2vec2.0-like structure is similar to the original wav2vec2.0 model, including two parts: feature encoder (EEG signal feature encoder) and context network (EEG signal feature reconstruction model). The feature encoder is a network based on multi-layer one-dimensional convolution, the context network is composed of multi-layer position encoders and Transformer modules stacked together, and the emotion classifier is composed of multi-layer fully connected layers.

[0063] In one embodiment, the step of inputting the EEG signals in the training data set into an EEG signal feature encoder for representing emotions includes: The EEG signal feature encoder includes a one-dimensional convolution layer, a pooling layer and a fully connected layer.

[0064] The EEG signals in the training data set are divided into a plurality of EEG signal segments of preset lengths.

[0065] Each EEG signal segment is input into the one-dimensional convolution layer, and after convolution processing by the one-dimensional convolution layer, an EEG signal feature map is obtained.

[0066] The EEG signal feature map is input into the pooling layer, and features in the EEG signal feature map are extracted to obtain EEG signal features.

[0067] The EEG signal features are input into the fully connected layer, and the EEG signal features are integrated and processed to obtain the initial EEG feature sequence.

[0068] The EEG signal feature encoder in this embodiment is similar to the feature encoder in the original wav2vec2.0 model. Specifically, the EEG signal feature encoder includes a one-dimensional convolution layer, a pooling layer and a fully connected layer; a plurality of one-dimensional convolution layers are provided to divide the EEG signals in the training data set into a plurality of EEG signal segments of preset lengths; each EEG signal segment is input into the convolution layer, and after multiple convolution processes, an EEG signal feature map is obtained; the EEG signal feature map is then input into the pooling layer, and the features in the EEG signal feature map are extracted to obtain the EEG signal features; the EEG signal features are input into the fully connected layer, and the EEG signal features are integrated and processed to obtain the initial EEG feature sequence.

[0069] In one embodiment, inputting the masked EEG feature sequence into an EEG signal feature reconstruction model comprises: The EEG signal feature reconstruction model includes a first position encoder and a first converter, a second position encoder and a second converter, and a third position encoder and a third converter.

[0070] The masked EEG feature sequence is input into the first position encoder, and the position relationship of the masked EEG feature sequence is extracted to obtain a first EEG feature position sequence.

[0071] The masked EEG feature sequence and the first EEG feature position sequence are input into a first converter, and feature extraction is performed through a self-attention mechanism in the first converter to obtain a first reconstructed EEG feature sequence.

[0072] The first reconstructed EEG feature sequence is input into the second position encoder, and the position relationship of the first reconstructed EEG feature sequence is extracted to obtain a second EEG feature position sequence.

[0073] The first reconstructed EEG feature sequence and the second EEG feature position sequence are input into a second converter, and feature extraction is performed through a self-attention mechanism in the second converter to obtain a second reconstructed EEG feature sequence.

[0074] The second reconstructed EEG feature sequence is input into the third position encoder, and the position relationship of the second reconstructed EEG feature sequence is extracted to obtain a third EEG feature position sequence.

[0075] The second reconstructed EEG feature sequence and the third EEG feature position sequence are input into a third converter, and feature extraction is performed through a self-attention mechanism in the third converter to obtain the predicted EEG feature sequence.

[0076] The EEG signal feature reconstruction model in this embodiment is similar to the context network in the original wav2vec2.0 model. Specifically, Figure 3 As shown, the EEG signal feature reconstruction model is composed of multiple layers of position encoders and Transformer modules stacked together. In this embodiment, the EEG signal feature reconstruction model is set to three layers, and each layer includes a position encoder and a Transformer module.

[0077] Specifically, the above-mentioned masked EEG feature sequence is input into the first layer of the EEG signal feature reconstruction model, and the position relationship of the masked EEG feature sequence is extracted through the position encoder (first position encoder) to obtain the first EEG feature position sequence; the masked EEG feature sequence and the first EEG feature position sequence are sent to the Transformer module (first converter), and feature extraction is performed through the self-attention mechanism in the Transformer module to obtain the first reconstructed EEG feature sequence.

[0078] The first reconstructed EEG feature sequence is input into the second layer of the EEG signal feature reconstruction model, and the positional relationship of the first reconstructed EEG feature sequence is extracted through the position encoder (second position encoder) to obtain the second EEG feature position sequence; the first reconstructed EEG feature sequence and the second EEG feature position sequence are sent to the Transformer module (second converter), and features are extracted through the self-attention mechanism of the Transformer module to obtain the second reconstructed EEG feature sequence.

[0079] The second reconstructed EEG feature sequence is input into the third layer of the EEG signal feature reconstruction model, and the positional relationship of the second reconstructed EEG feature sequence is extracted through the position encoder (the third position encoder) to obtain the third EEG feature position sequence; the second reconstructed EEG feature sequence and the third EEG feature position sequence are sent to the Transformer module (the third converter), and features are extracted through the self-attention mechanism in the Transformer module to obtain the predicted EEG feature sequence.

[0080] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0081] In one embodiment, an emotion recognition device based on EEG signals is provided, and the emotion recognition device based on EEG signals corresponds one-to-one to the emotion recognition method based on EEG signals in the above embodiment. Figure 4 As shown, the emotion recognition device based on EEG signals includes an acquisition module 101, a discrete processing module 102, a training module 103 and a classification module 104. The functional modules are described in detail as follows: The acquisition module 101 is used to acquire a training data set of EEG signals.

[0082] The discrete processing module 102 is used to input the EEG signals in the training data set into an EEG signal feature encoder for representing emotions to obtain an initial EEG feature sequence; and obtain a target EEG feature sequence after discretizing the initial EEG feature sequence.

[0083] The training module 103 is used to mask part of the feature sequence in the target EEG feature sequence to obtain a masked EEG feature sequence, input the masked EEG feature sequence into an EEG signal feature reconstruction model to obtain a predicted EEG feature sequence; construct a target loss function based on the predicted EEG feature sequence and the target EEG feature sequence; train and update the EEG signal feature reconstruction model according to the target loss function to obtain a trained EEG signal feature reconstruction model.

[0084] The classification module 104 is used to input the EEG signal to be classified into a trained EEG signal feature reconstruction model to obtain an EEG feature sequence to be classified, and input the EEG feature sequence to be classified into a pre-trained emotion classifier to obtain an emotion classification label.

[0085] In one embodiment, the training module 103 specifically includes: Mask sequence conversion submodule: used to generate a mask sequence corresponding to the same number of rows and columns as the target EEG feature sequence according to the target EEG feature sequence; the mask sequence includes mask values ​​and non-mask values.

[0086] The sequence data processing submodule is used to reset the data at the corresponding position in the target feature sequence to zero or replace it with a preset character according to the mask value in the mask sequence.

[0087] The sequence output submodule is used to retain the data at the corresponding position in the target feature sequence according to the non-mask value in the mask sequence.

[0088] In one embodiment, the classification module 104 specifically includes: The classification prediction submodule is used to input the training data set of EEG signals with real values ​​of emotion classification labels into the trained EEG signal feature reconstruction model to obtain a first predicted EEG feature, and input the first predicted EEG feature into the emotion classifier to obtain a predicted value of the emotion classification label.

[0089] The classification loss function submodule is used to construct a first loss function according to the true value and the predicted value of the emotion classification label.

[0090] The classification optimization submodule is used to minimize the first loss function value by iteratively updating the parameters of the emotion classifier to obtain the optimal parameter setting of the emotion classifier.

[0091] In one embodiment, the training module 103 specifically includes: The distance construction submodule is used to construct a distance function based on the cosine similarity between the predicted EEG feature sequence and the target EEG feature sequence.

[0092] The target loss function submodule is used to construct the target loss function based on the distance function through contrast loss.

[0093] In one embodiment, the training module 103 specifically includes: The model submodule is used for the EEG signal feature encoder, which is a feature encoder based on a convolutional network, and the EEG signal feature reconstruction model is a deep learning model based on a self-attention mechanism.

[0094] In one embodiment, the training module 103 specifically includes: The feature encoding structure submodule, used for the EEG signal feature encoder, includes a one-dimensional convolutional layer, a pooling layer and a fully connected layer.

[0095] The feature coding segmentation submodule is used to segment the EEG signal in the training data set into a plurality of EEG signal segments of preset lengths.

[0096] The feature coding convolution submodule is used to input each EEG signal segment into the one-dimensional convolution layer, and obtain an EEG signal feature map after convolution processing by the one-dimensional convolution layer.

[0097] The feature encoding pooling submodule is used to input the EEG signal feature map into the pooling layer, extract the features in the EEG signal feature map, and obtain the EEG signal features.

[0098] The feature encoding fully connected submodule is used to input the EEG signal features into the fully connected layer, and obtain the initial EEG feature sequence after integrating the feature processing of the EEG signal features.

[0099] In one embodiment, the training module 103 specifically includes: The feature reconstruction submodule, used for the EEG signal feature reconstruction model, includes a first position encoder and a first converter, a second position encoder and a second converter, and a third position encoder and a third converter.

[0100] The first position encoding submodule is used to input the masked EEG feature sequence into the first position encoder, extract the position relationship of the masked EEG feature sequence, and obtain the first EEG feature position sequence.

[0101] The first feature extraction submodule is used to input the masked EEG feature sequence and the first EEG feature position sequence into the first converter, perform feature extraction through the self-attention mechanism in the first converter, and obtain a first reconstructed EEG feature sequence.

[0102] The second position encoding submodule is used to input the first reconstructed EEG feature sequence into the second position encoder, extract the position relationship of the first reconstructed EEG feature sequence, and obtain the second EEG feature position sequence.

[0103] The second feature extraction submodule is used to input the first reconstructed EEG feature sequence and the second EEG feature position sequence into the second converter, perform feature extraction through the self-attention mechanism in the second converter, and obtain a second reconstructed EEG feature sequence.

[0104] The third position encoding submodule is used to input the second reconstructed EEG feature sequence into the third position encoder, extract the position relationship of the second reconstructed EEG feature sequence, and obtain the third EEG feature position sequence.

[0105] The third feature extraction submodule is used to input the second reconstructed EEG feature sequence and the third EEG feature position sequence into the third converter, perform feature extraction through the self-attention mechanism in the third converter, and obtain the predicted EEG feature sequence.

[0106] The present invention provides an emotion recognition device based on EEG signals. The device first obtains preliminary ranking results of candidate questions through semantic matching, and then proposes a scheme for optimizing a question-answering engine based on entity alignment. The ranking results of candidate questions are sorted again through entity alignment, so that more matching candidate questions are selected. This can effectively avoid the generalization ability defects of the model, greatly and efficiently improve the effect of entity matching, and improve the effect of the question-answering engine.

[0107] For the specific definition of the emotion recognition device based on EEG signals, please refer to the definition of the emotion recognition method based on EEG signals above, which will not be repeated here. Each module in the above-mentioned emotion recognition device based on EEG signals can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0108] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client through a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the server side of an emotion recognition method based on EEG signals.

[0109] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program: Get the training dataset of EEG signals.

[0110] The EEG signals in the training data set are input into an EEG signal feature encoder for representing emotions to obtain an initial EEG feature sequence; and the initial EEG feature sequence is discretized to obtain a target EEG feature sequence.

[0111] After masking part of the feature sequences in the target EEG feature sequence, a masked EEG feature sequence is obtained, and the masked EEG feature sequence is input into an EEG signal feature reconstruction model to obtain a predicted EEG feature sequence; a target loss function is constructed based on the predicted EEG feature sequence and the target EEG feature sequence; the EEG signal feature reconstruction model is trained and updated according to the target loss function to obtain a trained EEG signal feature reconstruction model.

[0112] The EEG signal to be classified is input into a trained EEG signal feature reconstruction model to obtain an EEG feature sequence to be classified, and the EEG feature sequence to be classified is input into a pre-trained emotion classifier to obtain an emotion classification label.

[0113] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: Get the training dataset of EEG signals.

[0114] The EEG signals in the training data set are input into an EEG signal feature encoder for representing emotions to obtain an initial EEG feature sequence; and the initial EEG feature sequence is discretized to obtain a target EEG feature sequence.

[0115] After masking part of the feature sequences in the target EEG feature sequence, a masked EEG feature sequence is obtained, and the masked EEG feature sequence is input into an EEG signal feature reconstruction model to obtain a predicted EEG feature sequence; a target loss function is constructed based on the predicted EEG feature sequence and the target EEG feature sequence; the EEG signal feature reconstruction model is trained and updated according to the target loss function to obtain a trained EEG signal feature reconstruction model.

[0116] The EEG signal to be classified is input into a trained EEG signal feature reconstruction model to obtain an EEG feature sequence to be classified, and the EEG feature sequence to be classified is input into a pre-trained emotion classifier to obtain an emotion classification label.

[0117] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0118] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0119] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0120] The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some of the technical features therein by equivalents; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should be included in the protection scope of the present invention. The non-company software tools or components that appear in the embodiments of this application are merely examples and do not represent actual use.

Claims

1. An emotion recognition method based on EEG signals, characterized in that: include: Obtain a training data set of EEG signals; Inputting the EEG signals in the training data set into an EEG signal feature encoder for representing emotions to obtain an initial EEG feature sequence; performing discretization processing on the initial EEG feature sequence to obtain a target EEG feature sequence; After masking a part of the feature sequence in the target EEG feature sequence, a masked EEG feature sequence is obtained, and the masked EEG feature sequence is input into an EEG signal feature reconstruction model to obtain a predicted EEG feature sequence; a target loss function is constructed based on the predicted EEG feature sequence and the target EEG feature sequence; and the EEG signal feature reconstruction model is trained and updated according to the target loss function to obtain a trained EEG signal feature reconstruction model; The EEG signal to be classified is input into a trained EEG signal feature reconstruction model to obtain an EEG feature sequence to be classified, and the EEG feature sequence to be classified is input into a pre-trained emotion classifier to obtain an emotion classification label.

2. The method for emotion recognition based on EEG signals as claimed in claim 1, characterized in that: The masking of part of the target EEG feature sequence comprises: According to the target EEG feature sequence, a mask sequence having the same number of rows and columns as the target EEG feature sequence is generated; the mask sequence includes mask values ​​and non-mask values; According to the mask value in the mask sequence, the data at the corresponding position in the target feature sequence is set to zero or replaced with a preset character; The data at the corresponding position in the target feature sequence is retained according to the non-mask value in the mask sequence.

3. The method for emotion recognition based on EEG signals as claimed in claim 1, characterized in that: The training process of the pre-trained emotion classifier includes: Inputting a training data set of EEG signals with real values ​​of emotion classification labels into a trained EEG signal feature reconstruction model to obtain a first predicted EEG feature, and inputting the first predicted EEG feature into the emotion classifier to obtain a predicted value of the emotion classification label; Constructing a first loss function according to the true value and the predicted value of the emotion classification label; The first loss function value is minimized by iteratively updating the parameters of the emotion classifier to obtain the optimal parameter setting of the emotion classifier.

4. The method for emotion recognition based on EEG signals as claimed in claim 1, characterized in that: The constructing a target loss function based on the predicted EEG feature sequence and the target EEG feature sequence includes: Constructing a distance function based on the cosine similarity between the predicted EEG feature sequence and the target EEG feature sequence; The target loss function is constructed based on the distance function through contrast loss.

5. The method for emotion recognition based on EEG signals as claimed in claim 1, characterized in that: The EEG signal feature encoder is a feature encoder based on a convolutional network, and the EEG signal feature reconstruction model is a deep learning model based on a self-attention mechanism.

6. The method for emotion recognition based on EEG signals as claimed in claim 5, characterized in that: The step of inputting the EEG signals in the training data set into an EEG signal feature encoder for representing emotions comprises: The EEG signal feature encoder includes a one-dimensional convolution layer, a pooling layer and a fully connected layer; Segmenting the EEG signals in the training data set into a plurality of EEG signal segments of preset lengths; Input each EEG signal segment into the one-dimensional convolution layer, and obtain an EEG signal feature map after convolution processing by the one-dimensional convolution layer; Inputting the EEG signal feature map into the pooling layer, extracting features from the EEG signal feature map, and obtaining EEG signal features; The EEG signal features are input into the fully connected layer, and the EEG signal features are integrated and processed to obtain the initial EEG feature sequence.

7. The method for emotion recognition based on EEG signals as claimed in claim 5, characterized in that: The step of inputting the masked EEG feature sequence into an EEG signal feature reconstruction model comprises: The EEG signal feature reconstruction model includes a first position encoder and a first converter, a second position encoder and a second converter, and a third position encoder and a third converter; Inputting the masked EEG feature sequence into the first position encoder, extracting the position relationship of the masked EEG feature sequence, and obtaining a first EEG feature position sequence; Inputting the masked EEG feature sequence and the first EEG feature position sequence into a first converter, performing feature extraction through a self-attention mechanism in the first converter, and obtaining a first reconstructed EEG feature sequence; Inputting the first reconstructed EEG feature sequence into the second position encoder, extracting the position relationship of the first reconstructed EEG feature sequence, and obtaining a second EEG feature position sequence; Inputting the first reconstructed EEG feature sequence and the second EEG feature position sequence into a second converter, performing feature extraction through a self-attention mechanism in the second converter, and obtaining a second reconstructed EEG feature sequence; Inputting the second reconstructed EEG feature sequence into the third position encoder, extracting the position relationship of the second reconstructed EEG feature sequence, and obtaining a third EEG feature position sequence; The second reconstructed EEG feature sequence and the third EEG feature position sequence are input into a third converter, and feature extraction is performed through a self-attention mechanism in the third converter to obtain the predicted EEG feature sequence.

8. An emotion recognition device based on EEG signals, characterized in that: include: An acquisition module is used to acquire a training data set of EEG signals; A discrete processing module, used for inputting the EEG signals in the training data set into an EEG signal feature encoder for representing emotions to obtain an initial EEG feature sequence; and obtaining a target EEG feature sequence after discretizing the initial EEG feature sequence; A training module is used to mask a part of the feature sequence in the target EEG feature sequence to obtain a masked EEG feature sequence, input the masked EEG feature sequence into an EEG signal feature reconstruction model to obtain a predicted EEG feature sequence; construct a target loss function based on the predicted EEG feature sequence and the target EEG feature sequence; train and update the EEG signal feature reconstruction model according to the target loss function to obtain a trained EEG signal feature reconstruction model; The classification module is used to input the EEG signal to be classified into a trained EEG signal feature reconstruction model to obtain an EEG feature sequence to be classified, and input the EEG feature sequence to be classified into a pre-trained emotion classifier to obtain an emotion classification label.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method for emotion recognition based on EEG signals as claimed in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for emotion recognition based on EEG signals as claimed in any one of claims 1 to 7 are implemented.

Images