A method for enhancing emotional EEG data by integrating spatiotemporal features

Abstract

This invention relates to a method for enhancing emotional EEG data by integrating spatiotemporal features. To address the challenges of existing techniques, EEG data for different emotions is first extracted as differential entropy features in five frequency bands. This data is then segmented, with a set of data taken every second. Each set of data is processed into corresponding temporal data and spatial data distributed according to electrode positions. Finally, a dual encoder is designed and introduced using a VAE-GAN to learn these two data features, which are then fused into latent variables for generating artificial samples. This method comprehensively considers both temporal and spatial distribution features when generating artificial EEG feature samples, resulting in excellent performance in classification tasks.

Description

Technical Field

[0001] The present invention relates to an emotional EEG data enhancement method integrating spatiotemporal features, which is a method for extracting features from EEG signals and then artificially expanding the features, and belongs to the technical field of emotion recognition. Background Art

[0002] Affective computing is a type of computing that measures, analyzes, and influences human outward expressions and physiological reactions. Emotions are physiological changes caused by external stimuli. Currently, the main types of emotional representations used are explicit and implicit. Explicit features include body movements, facial expressions, language, and voice; implicit features include changes in blood pressure, electrocardiograms (ECGs), and electroencephalograms (EEGs). As physiological signals that are not controlled by humans, EEG signals are more capable of reflecting an individual's underlying emotional state. Therefore, emotion recognition based on EEG signals is increasingly being applied to diverse fields, including medical diagnosis, e-learning, and leisure and entertainment.

[0003] However, as research continues to develop, the limitations of emotion recognition based on EEG signals have gradually emerged.

[0004] Traditional analysis methods often require a lot of prior knowledge to obtain the deep features of EEG signals, which consumes a lot of manpower and material resources in the process. Deep learning, as a method that can mine the more essential features of data, has been applied by many researchers in the field of EEG emotion recognition. The above-mentioned deep learning methods require a large amount of labeled data as training samples. However, unlike image recognition datasets, there are few public EEG signal datasets that can be used for emotion recognition, and there are many problems with EEG signal data collection:

[0005] (1) The acquisition equipment is expensive and requires complex preparation.

[0006] (2) The wearing time should not be too long, otherwise it will cause discomfort to the subject and reduce the conductivity of the conductive medium, causing data distortion.

[0007] (3) There will be a lot of noise in the process of collecting EEG signals, which requires relatively complicated data preprocessing. Some data will be discarded during the preprocessing process, and there is not much high-quality data available.

[0008] (4) EEG signals have large individual differences. Different subjects will have certain differences in their responses to the same emotion-inducing material. Even the data from different trials of the same subject will be different. EEG data lack universality.

[0009] (5) Many subjects expect their privacy to be protected and are unwilling to disclose their data.

[0010] Therefore, it is very difficult to obtain a large amount of labeled data for training deep learning models. Generating high-quality artificial samples to expand existing datasets may become an efficient solution.

[0011] Data augmentation can effectively address the problem of data scarcity. It involves transforming real data to generate new, similar-to-real data. However, due to the strong temporal correlation of EEG signals, traditional methods are not suitable for EEG signal augmentation. Compared to traditional methods, deep generative models can learn a representation of the distribution within samples, thereby generating more realistic artificial samples. Summary of the Invention

[0012] To address the current data scarcity problem, this paper simultaneously considers the relative positional relationship between electrodes and the temporal characteristics of EEG signals during the generation of artificial samples. Specifically, the preprocessed differential entropy feature data is first segmented and processed into two forms: temporal features and spatial features of different dimensions. Furthermore, a dual encoder is introduced based on the traditional VAE-GAN, extracting latent variables from the two different forms of features. These two extracted latent variables are then fused and decoded to guide the generation of artificial samples.

[0013] The specific steps of a method for enhancing emotional EEG data by integrating spatiotemporal features are as follows:

[0014] Step 1: Use the device to collect induced EEG signals of different emotions.

[0015] Step 2: Extract the differential entropy features of the original EEG signal and process it into two different forms, which are respectively aimed at the temporal characteristics and spatial distribution characteristics of the EEG signal.

[0016] Step 3: Build a dual encoder module and embed it into the original VAE-GAN.

[0017] Step 4: Separate each emotion and use them to train the corresponding model, and train a different model for each emotion.

[0018] Step 5: Use the trained model to generate artificial samples and use a simple classifier to detect the quality of artificial samples.

[0019] In step 2, the differential entropy data is cut and processed into two required forms. The specific steps are:

[0020] S2.1: Extract the differential entropy features of the processed EEG sequence data.

[0021] S2.2: Arrange the processed differential entropy data by time. Starting from the first second, continuously collect t seconds of data. Every 1 second, collect t seconds of data. To ensure the integrity of the data involved in the model, if the subsequent data is less than t seconds, the first data will be used to supplement it. If the total data length is T seconds, then T segments of t seconds can be obtained.

[0022] S2.3: A piece of data x is retrieved i The size is t×bands×channels. Flatten the data according to time to obtain data of time features of size t×(bands×channels)

[0023] S2.4: Arrange the data of each frequency band at each second according to the two-dimensional coordinates of the 62 electrodes in polar coordinate projection and interpolate to obtain a matrix of size t×bands W×H, that is, the data for spatial features

[0024] In step 3, the dual encoder is embedded in the VAE-GAN. The specific steps are:

[0025] S3.1: Construct an encoder to extract time series features. The encoder is named Encoder I and consists of three convolutional layers and one fully connected layer. The fully connected layer converts the features into latent variables of length 70.

[0026] S3.2: Construct an encoder to extract spatial features. This encoder is named Encoder II and consists of three convolutional layers and one fully connected layer. The fully connected layer converts the features into latent variables of length 30.

[0027] S3.3: The decoder consists of a fully connected layer and two deconvolutional layers.

[0028] S3.4: The discriminator consists of four convolutional layers and one fully connected layer.

[0029] In step 4, several independent models are trained for different emotions. The specific steps are:

[0030] S4.1: Use Encoder I to extract time series information and obtain the latent variable z time .

[0031] S4.1: Use Encoder II to extract spatial distribution information and obtain the latent variable z space .

[0032] S4.2: After concatenating the two latent variables, we get a new latent variable z containing complementary information. concat, with a length of 100, is decoded by the decoder to generate data. The loss of VAE at this time is:

[0033]

[0034] The loss consists of three parts: the first is the prior loss of extracting data from latent variables, the second is the data reconstruction loss, and the third is the original loss of GAN.

[0035] S4.4: Considering the instability of GAN training, this work introduces the principle of WGAN-GP into DEVAE-GAN. The loss function is rewritten as:

[0036]

[0037] The loss of GAN is transformed into the loss of WGAN-GP, and gradient penalty is added to make the training more stable.

[0038] S4.5: Train and save parameters.

[0039] In step 5, the trained model is called to generate artificial samples. The specific steps are as follows:

[0040] S5.1: Call the trained model parameters and load the decoder.

[0041] S5.2: Generate Gaussian noise, call the decoder, and generate artificial samples.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] This method uses differential entropy features over continuous time, repeatedly samples them, and processes them into two distinct features: temporal and spatial features. This approach considers both temporal and spatial correlations between different electrodes. When training a model to generate artificial samples, the generator (i.e., decoder) acquires both the temporal and spatial features extracted by the encoder, enabling the scientific generation of higher-quality artificial samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 This is a flow chart of the emotional EEG data enhancement method integrating spatiotemporal features of the present invention;

[0046] Figure 2This is the coordinate diagram corresponding to the electrodes of the EEG device of the present invention;

[0047] Figure 3 This is a schematic diagram of the data processing proposed by the present invention into two types of data with different focuses;

[0048] Figure 4 This is the network structure diagram of the dual-encoder VAE-GAN proposed in this invention. DETAILED DESCRIPTION

[0049] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0050] The specific steps of a method for enhancing emotional EEG data by integrating spatiotemporal features are as follows:

[0051] Step 1: The data used in this example comes from the public dataset SEED. Videos containing certain emotional qualities are used to induce emotions in the subjects, and the induced EEG signals of different emotions are collected using a 62-channel EEG device using the international 10-20 system.

[0052] Step 2: Extract the differential entropy features of the original EEG signal and process it into two different forms, which respectively highlight the temporal characteristics and spatial distribution characteristics of the EEG signal. Figure 3 shown.

[0053] S2.1: Extract the differential entropy features of the processed EEG sequence data.

[0054] S2.2: Arrange the processed differential entropy data by time and segment it. The EEG signal before segmentation is 3394 × 5 × 62, where 5 represents the five frequency bands of the EEG signal and 62 represents the 62 channels of the EEG device. Starting from the first second, continuously sample 5 seconds of data, and then sample 5-second segments every 1 second, for a total of 3394 5-second segments.

[0055] S2.2: A piece of data x is retrieved i The size is 5×5×62. Flatten the data by time to obtain data of 5×310 for time features.

[0056] S2.4: Arrange the data of each frequency band at each second according to the two-dimensional coordinates of the 62 electrodes in polar coordinate projection and interpolate to obtain 25 16×16 matrices, i.e., the data for spatial features.

[0057] Step 3: Build a dual encoder module and embed it into the original VAE-GAN. The specific diagram is as follows Figure 4 shown.

[0058] S3.1: Construct an encoder to extract time series features. This encoder, named Encoder I, consists of three convolutional layers and one fully connected layer. The dimensions of the three convolutional layers are 16×(1,3), 32×(3,3), and 64×(3,3), respectively. Before passing through the fully connected layer, the data is a 64×1×37 matrix. The fully connected layer converts the features into latent variables of length 70.

[0059] S3.2: Construct an encoder to extract spatial features. This encoder, named Encoder II, consists of three convolutional layers and one fully connected layer. The sizes of the three convolutional layers are 32×(8,8), 32×(6,6), and 64×(4,4), respectively. The fully connected layer converts the features into latent variables of length 30.

[0060] S3.3: The decoder consists of a fully connected layer and two deconvolutional layers, with sizes of 32×(3,3) and 5×(3,4) respectively.

[0061] S3.4: The discriminator consists of four convolutional layers and one fully connected layer. The first two convolutional layers are (3,3) with channels of 16 and 32, respectively. The last two convolutional layers are (1,1) with channels of 16 and 1, respectively.

[0062] Step 4: Separate each emotion and use them to train the corresponding model. A different model is trained for each emotion. The dataset used has three emotions. The training process is as follows: Figure 1 shown.

[0063] Step 5: The Adam optimizer was used in this example, with a learning rate of 0.0001, a batch size of 64, an epoch of 150, and a LeakyReLU activation function. The gradient penalty factor in the loss function was set to λ = 10. Data was normalized before all experiments. Experimental parameters were identical for all compared networks. All methods in this example were implemented in Python, and the deep neural network was implemented in PyTorch.

[0064] Use the trained model to generate 15,000 artificial samples, add 200, 500, 800, 1,000, 2,000, 3,000, 5,000, 8,000, 10,000, and 15,000 samples to the dataset respectively, and use the classifier to detect the classification accuracy.

[0065] Table 1 compares the final results of the present invention with those of other classic methods. Specifically, for each subject, different numbers of artificial samples are added to the original dataset, and the average classification accuracy is obtained by DNN.

[0066] Table 1 Average classification accuracy with different numbers of artificial samples

[0067]

[0068]

[0069] Compared to other generation methods, the dual-encoder VAE-GAN generated data achieved the highest classification accuracy of 97.21%, a five-percentage-point improvement compared to the data without artificial samples. This shows that the present invention can generate high-quality artificial samples.

[0070] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. It is apparent to those skilled in the art that various changes, modifications, substitutions, and variations to these embodiments may be made without departing from the principles and spirit of the present invention, and these changes and modifications still fall within the scope of protection of the present invention.

Claims

1. A method for enhancing emotional EEG data by integrating spatiotemporal features, characterized by: The following steps are involved: Step 1: Use the device to collect induced EEG signals of different emotions; Step 2: Extract the differential entropy features of the original EEG signal and process it into two different forms, one for the temporal characteristics and the other for the spatial distribution characteristics of the EEG signal; Step 3: Build a dual encoder module and embed it into the original VAE-GAN; Step 4: Separate each emotion and use them to train the corresponding model, training a different model for each emotion; Step 5: Generate artificial samples using the trained model and use a simple classifier to detect the quality of artificial samples; The step three specifically includes: S3.1: Construct an encoder to extract temporal features. The temporal feature encoder consists of three convolutional layers and one fully connected layer. The fully connected layer converts the features into latent variables of length 70. S3.2: Construct an encoder to extract spatial features. The encoder consists of three convolutional layers and one fully connected layer. The fully connected layer converts the features into latent variables of length 30. S3.3: The decoder consists of a fully connected layer and two deconvolutional layers; S3.4: The discriminator consists of four convolutional layers and one fully connected layer; The step 4 specifically includes: S4.1: Use the encoder of the time series feature to extract the time series information and obtain the latent variable z time ; S4.2: Use the spatial feature encoder to extract spatial distribution information and obtain the latent variable z space ; S4.3: After concatenating the two latent variables, we get a new latent variable z containing complementary information. concat , with a length of 100, is decoded by the decoder to generate data, and the loss of VAE is: The loss consists of three parts: the first is the prior loss of extracting data from latent variables, the second is the data reconstruction loss, and the third is the original loss of GAN; S4.4: The principle of WGAN-GP is introduced into DE-VAEGAN, and the loss function is rewritten as: Transform the GAN loss into the WGAN-GP loss and add gradient penalty; S4.5: Train and save parameters.

2. The method for enhancing emotional EEG data by integrating spatiotemporal features according to claim 1, characterized in that: The second step specifically includes: S2.1: Extract differential entropy features from processed EEG sequence data; S2.2: Arrange the processed differential entropy data by time. Starting from the first second, continuously take t seconds of data. Take t seconds of fragments every 1 second. If the subsequent fragments are less than t seconds, take the first data to make up for it. If the total data length is T seconds, then T fragments of t seconds can be obtained. S2.3: A piece of data x is retrieved i The size is t×bands×channels. Flatten the data according to time to obtain data of time features of size t×(bands×channels). S2.4: Arrange the data of each frequency band at each second according to the two-dimensional coordinates of the 62 electrodes in polar coordinate projection and interpolate to obtain a matrix of size t×bands W×H, that is, the data for spatial features 3. The method for enhancing emotional EEG data by integrating spatiotemporal features according to claim 1, characterized in that: The step five specifically includes: S5.1: Call the trained model parameters and load the decoder; S5.2: Generate Gaussian noise, call the decoder, and generate artificial samples.

Images