Multi-scale space-time perception electroencephalogram data amplification method and system based on diffusion enhancement
Through a multi-scale spatiotemporal perception method based on diffusion enhancement, the problem of low signal-to-noise ratio and individual differences in EEG signal recognition is solved, high-quality diversified data is generated, the accuracy of emotion recognition and model performance is improved, and the rapid generation and function of the ability of the emotion recognition is achieved. Explained EEG data.
Patent Information
- Application Number
- CN202410965778.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-18
- Publication Date
- 2024-11-12
AI Technical Summary
Existing EEG-based emotion recognition methods face the problems of low signal-to-noise ratio, insufficient recognition accuracy and robustness, especially insufficient consideration of individual differences, and pattern crashes during deep learning model training, resulting in difficult interpretation of generated data.
Using a multi-scale spatiotemporal perception method based on diffusion enhancement, high-quality and diverse emotional EEG data are generated through diffusion model and multi-scale spatiotemporal perception model, combined with Euclidean alignment method and attention mechanism, a diffusion enhancement model and multi-dimensional Scaling spatiotemporal perception models generate enhanced samples and fine-tune them to improve the accuracy of emotion recognition.
It realizes the rapid generation of large-scale interpretable EEG data, improves the accuracy of emotion recognition and model performance, ensures the diversity and quality of generated data, and enhances the robustness and generalization capabilities of the model.
Smart Images

Figure CN118940031A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of EEG emotion recognition, and in particular to an EEG data amplification method and system based on diffusion-enhanced multi-scale spatiotemporal perception. Background Art
[0002] With the rapid development of artificial intelligence and machine learning technologies, emotion recognition technology has become a key technology in the fields of human-computer interaction, mental health monitoring and smart home. Emotion recognition technology aims to identify and understand the emotional state of an individual by analyzing physiological signals or behavioral patterns. Among them, emotion recognition based on electroencephalogram (EEG) has attracted widespread attention because it can directly reflect brain activity.
[0003] EEG signals are non-invasive, real-time and cost-effective physiological signals that can effectively capture changes in brain activity. However, the acquisition and processing of EEG signals face multiple challenges. First, EEG signals are highly non-stationary and have a low signal-to-noise ratio, which makes it difficult to extract stable emotion-related features from the signals. Secondly, there are significant differences in the EEG activity patterns of different individuals, which places higher demands on the generalization ability of emotion recognition models. In addition, existing EEG-based emotion recognition methods often rely on specific frequency bands and electrode channels, lack consideration of global EEG activity, and limit recognition accuracy and robustness.
[0004] Emotion recognition determines an individual's emotional state by analyzing data using specific criteria. In the early days of emotion recognition, experts used handcrafted features such as differential entropy (DE) and power spectral density (PSD) to identify different states, however, these features often miss important information that humans may have difficulty recognizing. With the advent of deep learning, more and more researchers are using end-to-end methods for emotion recognition based on EEG data.
[0005] In recent years, deep learning techniques have been widely used in EEG signal processing to improve the accuracy of emotion recognition. Nevertheless, deep learning models usually require a large amount of labeled data for training, and a large amount of data is needed to improve the robustness and generalization ability of the model. However, the labeling cost of EEG data is very high. In the field of EEG signals, researchers are exploring various methods to improve the quality of EEG signal samples. Some researchers use power spectral density or time-frequency analysis techniques to convert EEG signals into spectrogram images, and apply computer vision techniques such as cropping, splicing and noise addition to improve data quality. There are also methods such as Fourier transform to add information to the spectrum segment to amplify the data. These methods have shown good results in improving the quality of EEG signals and facilitating model training. GAN (Generative Adversarial Network) is becoming increasingly popular in generating data in various fields, including EEG-based emotion recognition. Researchers have conducted a lot of research in this field and developed various GAN-based models. Many researchers have proposed various GAN-based EEG generation strategies to achieve good results, but there are still many problems. One of the main problems is the mode collapse during training, which may make it difficult to interpret the generated data.
[0006] Therefore, there is an urgent need to provide a new type of EEG data amplification method and system based on diffusion-enhanced multi-scale spatiotemporal perception to solve the above problems. Summary of the invention
[0007] The technical problem to be solved by the present invention is to provide an EEG data amplification method and a system based on diffusion-enhanced multi-scale spatiotemporal perception, and propose a strategy for generating EEG data using a diffusion model, in the hope of generating high-quality and diverse emotional EEG data and achieving high-accuracy emotion recognition end-to-end.
[0008] In order to solve the above technical problems, a technical solution adopted by the present invention is to provide an EEG data amplification method based on diffusion-enhanced multi-scale spatiotemporal perception, comprising the following steps:
[0009] S1: Collect the user's EEG data for preprocessing and convert it into a two-dimensional matrix array;
[0010] S2: Use the Euclidean alignment method to align and standardize the data of all users, and divide all the data into training sets and test sets;
[0011] S3: Construct a diffusion enhancement model and use the training set data to train the diffusion enhancement model;
[0012] S4: Build a multi-scale spatiotemporal perception model and use the training set data for training. After completion, fix the parameters of the diffusion enhancement model to obtain a trained diffusion enhancement model.
[0013] S5: Generate enhanced samples through the trained diffusion enhancement model, mix the enhanced samples with the real samples and input them into the multi-scale spatiotemporal perception model for fine-tuning to obtain the trained multi-scale spatiotemporal perception model;
[0014] S6: Input the test set into the trained multi-scale spatiotemporal perception model to obtain the classified sentiment labels.
[0015] In a preferred embodiment of the present invention, in step S1, the international 10-20 system EEG acquisition equipment is used to collect EEG signal data of N users under K emotional stimulations, and the data at the corresponding positions are projected into a two-dimensional matrix through the corresponding electrode positions to obtain standardized data.
[0016] In a preferred embodiment of the present invention, the specific steps of step S3 include:
[0017] The diffusion enhancement model includes a forward training and a backward generation process. During the forward training process, Gaussian noise is gradually added to the original EEG signal to eventually convert it into pure noise. During the backward generation process, the original Gaussian noise is gradually denoised through the learned diffusion enhancement model to obtain data that conforms to the real distribution.
[0018] In a preferred embodiment of the present invention, in step S4, the multi-scale spatiotemporal perception model includes a spatial frequency learning module and a temporal learning module. The spatial frequency learning module uses the preprocessed EEG signal as its input, generates a one-dimensional vector as its output, and obtains the necessary spatial frequency features; the temporal learning module improves the accuracy of emotion classification by effectively processing continuous EEG data to obtain global and local temporal features.
[0019] Furthermore, the spatial frequency learning module sequentially includes a first multi-scale convolution layer, three two-dimensional convolution layers, a residual layer, a parallel separable convolution layer, and a second multi-scale convolution layer.
[0020] Furthermore, the temporal learning module consists of a long short-term memory network LSTM and a temporal separable convolutional network TCN.
[0021] In a preferred embodiment of the present invention, in step S4, the loss function of the stage of inputting the real sample into the multi-scale spatiotemporal perception model is as follows:
[0022]
[0023] Among them, the variable x i represents the true label of the i-th element, and y i represents the predicted class probability assigned by the model, and c represents the number of true samples.
[0024] In a preferred embodiment of the present invention, in step S5, the loss function of the fine-tuning stage is targeted at a specific emotional task, emphasizing the subtle differences and goals of the task, and each iteration requires input of real samples and enhanced samples, and the formula is as follows:
[0025]
[0026] Where N represents the number of enhanced samples, h(X i ) and h(x i ) represent the expected feature vector and the feature vector of the real sample with the same label, respectively, and λ is a hyperparameter that determines the impact of the enhanced sample on the model.
[0027] In order to solve the above technical problems, another technical solution adopted by the present invention is to provide an EEG data amplification system based on diffusion-enhanced multi-scale spatiotemporal perception, comprising:
[0028] The data acquisition and preprocessing module is used to collect the user's EEG data for preprocessing and convert it into a two-dimensional matrix array;
[0029] The data normalization module is used to align and normalize the data of all users using the Euclidean alignment method, and divide all the data into training sets and test sets;
[0030] A data enhancement diffusion model building module is used to build a diffusion enhancement model and train the diffusion enhancement model using training set data;
[0031] The multi-scale spatiotemporal perception model construction module is used to construct a multi-scale spatiotemporal perception model and use the training set data for training. After completion, the parameters of the diffusion enhancement model are fixed to obtain a trained diffusion enhancement model.
[0032] A multi-scale spatiotemporal perception model training module, used to generate enhanced samples through the diffusion enhancement model trained by the data enhancement diffusion model construction module, and mix the enhanced samples with real samples and input them into the multi-scale spatiotemporal perception model constructed by the multi-scale spatiotemporal perception model construction module for fine-tuning to obtain a trained multi-scale spatiotemporal perception model;
[0033] The EEG emotion recognition module is used to input the test set into the trained multi-scale spatiotemporal perception model to obtain classified emotion labels.
[0034] In a preferred embodiment of the present invention, the multi-scale spatiotemporal perception model constructed by the multi-scale spatiotemporal perception model construction module includes a spatial frequency learning module and a temporal learning module. The spatial frequency learning module uses the preprocessed EEG signal as its input, generates a one-dimensional vector as its output, and obtains the necessary spatial frequency features; the temporal learning module improves the accuracy of emotion classification by effectively processing continuous EEG data to obtain global and local temporal features.
[0035] The beneficial effects of the present invention are: compared with the prior art, the present invention can achieve excellent results, and compared with the current method of generating EEG data, it can quickly generate large-scale EEG data, and the generated data is interpretable. Specifically, it is shown in:
[0036] 1) Propose a diffusion enhancement model that integrates diffusion models and self-supervised learning, aiming to synthesize diverse EEG signals suitable for various downstream tasks. It combines residual and attention mechanisms, along with temporal embedding, to ensure the diversity and quality of the generated EEG signals;
[0037] 2) A new multi-scale spatiotemporal perception model, MSTPN, is designed, which uses attention mechanism and multi-scale parallel stream mechanism to emphasize emotion-related features in EEG signals. By studying the impact of different numbers of augmented samples on model performance, a quantitative evaluation method is used to evaluate the quality and diversity of augmented samples. In addition, visualization results of qualitative analysis are proposed. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 It is a flow chart of the EEG data amplification method based on diffusion-enhanced multi-scale spatiotemporal perception of the present invention;
[0039] Figure 2 It is the data map after projection mapping of the DEAP data set;
[0040] Figure 3 is a data map after projection mapping of the DREAMER dataset;
[0041] Figure 4 is a structural diagram of the diffusion enhancement model;
[0042] Figure 5 is a network structure diagram of the multi-scale spatiotemporal perception;
[0043] Figure 6 It is a schematic diagram of the distribution of valence and arousal features extracted from the original EEG signal and enhanced EEG signal on the DEAP dataset;
[0044] Figure 7 It is a box plot of the valence and arousal classification accuracy of the DEAP dataset under different scales of data augmentation;
[0045] Figure 8 It is a structural block diagram of the EEG data amplification system based on diffusion-enhanced multi-scale spatiotemporal perception of the present invention. DETAILED DESCRIPTION
[0046] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present invention.
[0047] See also Figure 1 , the embodiment of the present invention includes:
[0048] A method for amplifying EEG data based on diffusion-enhanced multi-scale spatiotemporal perception includes the following steps:
[0049] S1: Collect the user's EEG data for preprocessing and convert it into a two-dimensional matrix array;
[0050] The EEG data of N users under K emotional stimulation were collected using the international 10-20 system EEG acquisition equipment, and the sampling rate was downsampled to 128Hz. The EEG signal was subjected to a 4Hz-45Hz bandpass filter to remove physiological and power frequency noise. The experimental data were segmented into 1-second blocks using a non-overlapping sliding window, and these segmented EEG signals were used as data samples. Through the corresponding electrode positions, the data at the corresponding positions were projected into a 9*9 two-dimensional matrix to obtain the standardized data Sn=n*128*9*9.
[0051] In order to evaluate the performance of the method proposed in the present invention, this example conducted experiments on two public electroencephalogram (EEG) emotion datasets: the DEAP dataset and the DREAMER dataset. DEAP dataset: contains EEG data of 32 subjects when watching emotional stimulation videos. Each subject watched 40 1-minute video clips and scored on the emotional dimensions, including pleasure, arousal, dominance, and similarity. In this example, a sampling rate of 128 Hz was used and preprocessing was performed, including bandpass filtering (4-45 Hz) and denoising. DREAMER dataset: contains EEG data of 23 subjects when watching emotional stimulation videos. Each subject watched 18 videos and scored on the emotional dimension. In this example, a sampling rate of 128 Hz was used and preprocessing was performed, including bandpass filtering (4-45 Hz) and denoising.
[0052] Project the data into a 9×9 two-dimensional matrix, such as Figure 2 and 3 For each dataset, a corresponding mapping scheme is used.
[0053] S2: Use the Euclidean alignment method to align and standardize the data of all users, and divide all the data into training sets and test sets;
[0054] According to the Euclidean alignment method, all data are aligned in the Euclidean space. Using the ten-fold cross-validation method, the data are randomly divided into 10 different subsets. In each fold, the subsets are further divided into training and test sets, using 80% of the samples for training and 20% for testing.
[0055] S3: The principle of the diffusion model is derived from non-equilibrium thermodynamics. The theory first defines a Markov chain that describes the diffusion process. Then it constructs the required data samples by gradually introducing Gaussian noise to learn the reverse diffusion process. When applied to EEG data, it is necessary to gradually add noise to the actual EEG signal. The diffusion model includes a forward training process and a backward generation process. In the forward training process, Gaussian noise is gradually added to the original EEG signal Sn. Starting from the initial state x0, the state at each moment in the forward process depends only on the previous moment, thus constituting a Markov process. According to this principle, a diffusion enhancement model is constructed, and the diffusion enhancement model is trained using the training set data to obtain enhanced samples.
[0056] In the existing technology, the diffusion model has not been applied in the EEG field. In order to adapt to the generated data in the EEG field, we need to consider the time domain. The input of data needs to take into account the coherence, and the training data should not be disrupted.
[0057] Specifically, the diffusion enhancement model includes forward training and backward generation processes. In the forward training process, Gaussian noise is gradually added to the original EEG signal Sn. Starting from the initial state x0, the state at each moment in the forward process depends only on the previous moment.
[0058]
[0059] q is the forward diffusion process, the noise adding time step is represented by t, ranging from 1 to T, x0 is an initial noise-free image, x T The noise is Gaussian-distributed, βt is the noise rate, and the fixed sequence of t means the proportion of noise to be added at each step t, which is set by linear interpolation between the values 0.0001 and 0.02. t It follows a Gaussian distribution N(0, I). As diffusion occurs, the N(0, I) noise distribution parameters gradually change the variable x over time. t Converted to pure noise. When T reaches a high enough value, it converges to standard Gaussian noise N(0, I).
[0060]
[0061] The initial state x0 is gradually converted into the final state Gaussian noise xt using the linear interpolation parameter β. αt is the signal rate, defined as α+β=1, which means the image ratio of the previous step is retained at each step t. A fixed sequence about t, in which the initial state x0 is gradually transformed into the final state Gaussian noise x using a linear interpolation parameter β t This reverse generation process involves denoising inference. If we want to gradually obtain the distribution q(x t-1 , x t ), the initial distribution x0 can be recovered from the standard Gaussian distribution N(0, I). However, directly solving q(x t-1 , x t ) is challenging. Therefore, we can rely on neural networks to learn this distribution. Typically, the UNet 2D conditional model is used in this scenario.
[0062]
[0063] Through the training process, the model must master the μ in the learning formula θ (x t , t) and σ θ (x t , t) to understand the noise ε θ (x t , t), where μ θ With σ θ It is the predicted noise parameter. During the reverse diffusion process, the model uses maximum likelihood estimation to establish the probability distribution of Markov chain transfer. The main goal of training is to reduce the variance between the actual noise variance and the noise variance θ expected by the model.
[0064]
[0065] Here we use the neural network ∈ 0 (x t ,t) to approximate the noise∈, which is the simplified loss function.
[0066] The reverse generation process is the so-called inference process. Given a Gaussian noise data, through the formula:
[0067]
[0068] P represents the reverse generation process, from state x t Derivation x t-1 , this process is obtained by Markov chain and Bayesian formula, and after predicting the noise ∈, subtracting it will get x t-1 Status data.
[0069] At time T, the final data x0 is obtained through a step-by-step denoising process. However, due to the limited number of electrodes on the EEG acquisition cap, the resulting matrix may be sparse, resulting in noise in the data. This noise is usually caused by the synthesis affecting the area where the electrode is missing. To solve this problem, a special matrix needs to be multiplied with the data. The elements of this matrix come from the initial data, and the value is set to 1 when there is an electrode and to 0 when there is no electrode. These operations enable the acquisition of reasonably enhanced sample data. For noise prediction, the input consists of a two-dimensional matrix array with noise added, and the output is a noise array of the same dimension. Therefore, this task can be classified as an Img2Img problem. Given the similarity of this task to image segmentation, the UNet 2D conditional model proposed in the field of image segmentation is adopted and enhanced in this example. As Figure 4 As shown in the figure, the UNet model consists of two main components: an encoder and a decoder. The encoder progressively compresses the image size, while the decoder progressively reconstructs the original image size. The residual connection retains the information of the previous steps during inference and image reconstruction. Channel adjustment only occurs in the same layer, while image compression occurs between different layers. Each encoder layer doubles the number of channels, reducing the size from 9×9 to 5×5, 3×3, and finally 1×1. The decoder performs the inverse operation. The temporal embedding layer uses trigonometric position encoding, similar to the Transformer, to convert constants into vectors. The attention layer splits the image into tokens along the channel dimension and then reassembles the tokens after the attention is completed.
[0070] S4: Build a multi-scale spatiotemporal perception model and use the training set data for training. After completion, fix the parameters of the diffusion enhancement model to obtain a trained diffusion enhancement model.
[0071] In order to evaluate the effectiveness and diversity of the generated sample data, this paper proposes a novel network architecture, MSTPN, for extracting emotion-related feature information from EEG signals. MSTPN includes two main components: a spatial frequency learning module and a temporal learning module.
[0072] like Figure 5As shown, the spatial frequency learning module is the module responsible for learning spatial frequency patterns, using the processed EEG signal Sn as input and producing a one-dimensional vector as output. This module uses the first multi-scale convolution layer to extract information of different size scales from the EEG signal. In order to minimize the lost data, the number of channels is enlarged and a relatively large amount of valuable information is extracted from the original signal. Subsequently, the signal passes through three two-dimensional convolution layers to reduce the dimensionality of the channels. Each convolution layer halves the number of input feature channels while keeping the size of each signal unchanged. Each layer is followed by an activation layer, which integrates the scaled exponential linear unit (SELU) activation function and the dropout function to improve learning efficiency. Residual connections have been integrated to optimize feature representation and minimize information loss and distortion. In addition, cross-layer connections are implemented to enhance direct information transfer. Studies have shown that different emotions can be identified by local patterns of spatial or temporal features. A parallel separable convolution layer is introduced after the residual layer without changing the dimension to effectively capture these patterns. This separable convolution process involves depthwise convolution and pointwise convolution, allowing the identification of local temporal and spatial correlations. Finally, the Inception module (the second multi-scale convolutional layer) is used again to halve the dimension and obtain the necessary spatial frequency features Sn.
[0073] The temporal learning module aims to improve the accuracy of emotion classification by effectively processing continuous EEG data. The fluctuations of each electrode channel over time are crucial because they contain important hidden insights. To achieve this, global and local temporal features must be obtained. The temporal learning module consists of an attention LSTM and a temporally separable convolutional network (TCN). The long short-term memory network (LSTM) is a sequence network that effectively extracts temporal information through mechanisms such as input, forget, and output gates. TCN mainly adopts one-dimensional convolution and residual connection, including separable convolution and dilated convolution. The residual block utilizes a layer of separable convolution and uses the SELU activation function, while introducing dropout regularization to combat overfitting. The residual connection includes a forward neural network and a skip connection. A 1×1 convolution layer is added to ensure that the tensor scale between input and output is uniform. Multi-scale depth information is obtained by expanding the convolution operation. The introduction of separable convolution reduces the model complexity, while the residual connection transmits both the original feature information and the deep feature information to subsequent operations. The features extracted from the LSTM and TCN are fused and classified using a linear classifier to obtain the final classification result.
[0074] S5: Generate enhanced samples through the trained diffusion enhancement model, mix the enhanced samples with real samples and input them into the multi-scale space-time perception model for fine-tuning to obtain the trained multi-scale space-time perception model; here, the real samples are the training set.
[0075] The process of inputting real samples into the multi-scale spatiotemporal perception model for training is the pre-training stage, in which the model parameters are adjusted using real data. After that, the model parameters of the pre-training stage are fine-tuned using enhanced samples and real samples, which is the fine-tuning stage.
[0076] The loss function is crucial for training the model because it quantifies the difference between the predicted output and the true label. The loss function in the pre-training phase is as follows:
[0077]
[0078] Among them, the variable x i represents the true label of the i-th element, and y i represents the predicted class probability assigned by the model, where c represents the number of real samples. During the pre-training phase, the model only uses real samples, so weight control is considered unnecessary.
[0079] The fine-tuning phase introduces augmented data, which requires a replacement of the loss function. Unlike the loss function used during pre-training, the fine-tuning loss function is tailored for a specific emotional task. It places more emphasis on the nuances and goals of the task. The loss function used during fine-tuning plays a key role in optimizing the model and improving its overall performance. When adding augmented samples, feature vectors above the fully connected layer must be retrieved. N represents the number of augmented samples, and h(X i ) and h(x i ) denote the expected feature vector of the augmented sample with the same label and the feature vector of the real sample, respectively. To achieve this, the mean squared error (MSE) loss function is used to calculate the difference between the feature vectors of the real sample and the augmented sample. This provides useful cumulative feedback and helps in collaborative execution. λ is a hyperparameter that determines the influence of the augmented samples on the model; as λ increases, their influence also increases. The default value of λ is 0.5.
[0080]
[0081] S6: Input the test set into the trained multi-scale spatiotemporal perception model to obtain the classified sentiment labels.
[0082] See also Figure 8 The example of the present invention also provides an EEG data amplification system based on diffusion-enhanced multi-scale spatiotemporal perception, including a data acquisition and preprocessing module, a data standardization module, a data enhancement diffusion model construction module, a multi-scale spatiotemporal perception model construction module, a multi-scale spatiotemporal perception model training module and an EEG data classification module (EEG emotion recognition module).
[0083] The data acquisition and preprocessing module is used to collect the user's EEG data for preprocessing and convert it into a two-dimensional matrix array;
[0084] The data standardization module is used to align and standardize the data of all users using the Euclidean alignment method, and divide all the data into a training set and a test set;
[0085] The data enhancement diffusion model construction module is used to construct a diffusion enhancement model and train the diffusion enhancement model using training set data;
[0086] The multi-scale spatiotemporal perception model construction module is used to construct a multi-scale spatiotemporal perception model, and use the training set data for training. After completion, the parameters of the diffusion enhancement model are fixed to obtain a trained diffusion enhancement model;
[0087] The multi-scale spatiotemporal perception model training module is used to generate enhanced samples through the diffusion enhancement model trained by the data enhancement diffusion model construction module, and mix the enhanced samples with real samples and input them into the multi-scale spatiotemporal perception model constructed by the multi-scale spatiotemporal perception model construction module for fine-tuning to obtain a trained multi-scale spatiotemporal perception model;
[0088] The EEG emotion recognition module is used to input the test set into the trained multi-scale spatiotemporal perception model to obtain classified emotion labels.
[0089] Furthermore, the multi-scale spatiotemporal perception model constructed by the multi-scale spatiotemporal perception model construction module includes a spatial frequency learning module and a temporal learning module. The spatial frequency learning module uses the preprocessed EEG signal as its input, generates a one-dimensional vector as its output, and obtains the necessary spatial frequency features; the temporal learning module improves the accuracy of emotion classification by effectively processing continuous EEG data to obtain global and local temporal features.
[0090] The EEG data amplification system based on diffusion-enhanced multi-scale spatiotemporal perception in this example can execute the EEG data amplification method based on diffusion-enhanced multi-scale spatiotemporal perception provided by the present invention, and can execute any combination of implementation steps of the method example, and has the corresponding functions and beneficial effects of the method.
[0091] The advantages of the method described in the present invention are explained below from three aspects: data accuracy, sample quality and generalization of the verification method.
[0092] In order to verify the effectiveness of the invention, the present invention was experimented on two common datasets, DEAP and DREAMER, where the accuracies of the two datasets are as follows: the model without data enhancement achieved excellent accuracies of 93.03% and 93.51% in the valence and arousal binary classification tasks, respectively, and the accuracy of the four-category classification also reached 90.42%. After adding data enhancement, the performance of the fine-tuned model in the valence and arousal binary classification tasks was improved by 2.02% and 2.22%, respectively, and the accuracy of the four-category classification was also improved to 92.63%.
[0093] In Table 1 below, a 5-fold cross validation is performed using the DEAP dataset, which is different from the method of performing 5-fold cross validation on a single subject and reporting the average performance of all subjects. To this end, the data samples are divided into five different subsets. In each subset, the subsets are further divided into training and test sets, using 80% of the samples for training and 20% for testing. The MSTPN model is first trained 150 times using 80% of the randomly shuffled data, always keeping the parameters unchanged. Then the DEM is trained 70 times using the same training set. After training the DEM, it is switched to the generation mode to generate enhanced samples of the same size as the real samples, and these enhanced samples are used to fine-tune the model 50 times. Finally, the performance of the fine-tuned model is evaluated, and the average of 5 experiments is reported. As shown in Table 1, the study shows that the MSTPN model of the present invention achieves significant accuracy rates of 93.03% and 93.51% in the valence and arousal binary classification tasks, respectively, without the help of data augmentation. In addition, the accuracy of the four-class classification also reached an impressive 90.42%. These findings indicate that the MSTPN model can effectively extract discriminative features from EEG signals for emotion recognition. In order to evaluate the effectiveness of the novel DEM method in generating data, the generated data was combined with real data and the pre-trained model was fine-tuned. The results show that the fine-tuned model showed 2.02% and 2.22% improved performance in the valence and arousal binary classification tasks, respectively. At the same time, the accuracy of the four-class classification was also improved to 92.63%. In addition, a comparative analysis of the method described in the present invention and other GAN-based data enhancement techniques was conducted to verify its effectiveness.
[0094] Table 1
[0095]
[0096] As shown in Table 2 below, DEM-MSTPN shows good results after the fine-tuning process. Although the improvement is not as significant as that of GAN and other specific methods, it can be attributed to the fact that their initial accuracy is lower and the effect is more significant. According to the final results, DEM-MSTPN shows excellent performance.
[0097] Table 2
[0098]
[0099] In order to observe the quality of the enhanced samples, qualitative and quantitative experiments were performed to evaluate the quality of the synthesized EEG signal samples.
[0100] (1) Quantitative experiment: Use the trained multi-scale spatiotemporal perception model as the feature extractor, input the real and enhanced samples under the same conditions, and extract the reciprocal output vector of the second fully connected layer. Then, calculate the sample means μ and μa of the real and enhanced samples, as well as the sample covariance matrices μ and μa. The FSTD calculation formula is as follows:
[0101]
[0102] The calculation results are shown in Table 3 below:
[0103] Table 3
[0104]
[0105] The smaller the score, the higher the degree of sample fitting, indicating that the present invention can well synthesize data similar to the real sample.
[0106] (2) Qualitative experiment: At the same time, we also obtain the features of the fully connected layer and use T-SNE to reduce the dimension. We observe the scatter plots of real samples and generated samples in two-dimensional space. The results are as follows: Figure 6 As shown, blue and red represent different labels, blue represents 1 and red represents 0. The real samples are shown as circles, while the enhanced samples are shown as crosses. When inspecting the scatter plot, it is obvious that the real samples can be easily distinguished and the enhanced samples are very similar to the real samples, indicating the authenticity of the generated EEG signals. This feature space shows a wide distribution range of the generated EEG signals, highlighting the diversity of the generated signals and helping to classify the complex boundaries of EEG signals with different labels.
[0107] Explore the impact of different amounts of augmented data on model performance and seek to determine the optimal ratio of augmented samples to real samples. In the examples of the present invention, data is generated at a ratio of 1:1, 2:1, or 4:1 (augmented samples to real samples), and the performance of the model is evaluated by five-fold cross validation. The box plot of the results is shown in Figure 7 shown.
[0108] To evaluate the effectiveness of the proposed method and assess the generalization of the model, further experiments were conducted on the widely used DREAMER dataset. The model was evaluated using a ten-fold cross-validation technique, where the DREAMER dataset samples were randomly divided into 10 parts. Initially, 80% of the randomly selected data samples were used as training data for MSTPN over 150 epochs. The model parameters of MSTPN remained unchanged while the same training data was input into DEM for model training to create a training model. The DEM generative model was then used for reverse generation to improve the samples. The fine-tuning process of 50 epochs was applied to MSTPN using augmented samples and real samples to obtain classification results. Finally, the classification results were averaged to determine the final results. Table 4 below shows the evaluation of valence and arousal in the DREAMER dataset and is compared with the results of several advanced methods. Without fine-tuning, the MSTPN method achieved an accuracy of 84.63% and 82.94% in valence and arousal, respectively. The method uses a multi-scale attention mechanism for feature extraction. It uses multi-scale concurrent RNN to extract more discriminative features, ultimately improving the accuracy of sentiment classification results. By incorporating enhanced samples for fine-tuning after DEM model training, the classification performance is improved, with accuracies of 85.53% and 84.37% for valence and arousal, respectively. The versatility of this method has been verified by experiments on the DEAP and DREAMER datasets.
[0109] Table 4
[0110]
[0111] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
Claims
1. A method for amplifying EEG data based on diffusion-enhanced multi-scale spatiotemporal perception, characterized in that: The following steps are involved: S1: Collect the user's EEG data for preprocessing and convert it into a two-dimensional matrix array; S2: Use the Euclidean alignment method to align and standardize the data of all users, and divide all the data into training sets and test sets; S3: Construct a diffusion enhancement model and use the training set data to train the diffusion enhancement model; S4: Build a multi-scale spatiotemporal perception model and use the training set data for training. After completion, fix the parameters of the diffusion enhancement model to obtain a trained diffusion enhancement model. S5: Generate enhanced samples through the trained diffusion enhancement model, mix the enhanced samples with the real samples and input them into the multi-scale spatiotemporal perception model for fine-tuning to obtain the trained multi-scale spatiotemporal perception model; S6: Input the test set into the trained multi-scale spatiotemporal perception model to obtain the classified sentiment labels.
2. The method for amplifying EEG data based on diffusion-enhanced multi-scale spatiotemporal perception according to claim 1, characterized in that: In step S1, the international 10-20 system EEG acquisition equipment is used to collect EEG signal data of N users under K emotional stimulations, and the data at the corresponding positions are projected into a two-dimensional matrix through the corresponding electrode positions to obtain standardized data.
3. The method for amplifying EEG data based on diffusion-enhanced multi-scale spatiotemporal perception according to claim 1, characterized in that: The specific steps of step S3 include: The diffusion enhancement model includes a forward training and a backward generation process. During the forward training process, Gaussian noise is gradually added to the original EEG signal to eventually convert it into pure noise. During the backward generation process, the original Gaussian noise is gradually denoised through the learned diffusion enhancement model to obtain data that conforms to the real distribution.
4. The method for amplifying EEG data based on diffusion-enhanced multi-scale spatiotemporal perception according to claim 1, characterized in that: In step S4, the multi-scale spatiotemporal perception model includes a spatial frequency learning module and a temporal learning module, wherein the spatial frequency learning module uses the preprocessed EEG signal as its input, generates a one-dimensional vector as its output, and obtains the necessary spatial frequency features; The temporal learning module improves the accuracy of emotion classification by effectively processing continuous EEG data and obtaining global and local temporal features.
5. The method for amplifying EEG data based on diffusion-enhanced multi-scale spatiotemporal perception according to claim 4, characterized in that: The spatial frequency learning module sequentially includes a first multi-scale convolution layer, three two-dimensional convolution layers, a residual layer, a parallel separable convolution layer, and a second multi-scale convolution layer.
6. The method for amplifying EEG data based on diffusion-enhanced multi-scale spatiotemporal perception according to claim 4, characterized in that: The temporal learning module consists of a long short-term memory network (LSTM) and a temporal separable convolutional network (TCN).
7. The method for amplifying EEG data based on diffusion-enhanced multi-scale spatiotemporal perception according to claim 1, characterized in that: In step S4, the loss function of the multi-scale spatiotemporal perception model stage where the real sample is input is as follows: Among them, the variable x i represents the true label of the i-th element, and y i represents the predicted class probability assigned by the model, and c represents the number of true samples.
8. The method for amplifying EEG data based on diffusion-enhanced multi-scale spatiotemporal perception according to claim 1, characterized in that: In step S5, the loss function of the fine-tuning stage targets a specific emotional task, emphasizing the nuances and goals of the task. Each iteration requires input of real samples and enhanced samples. The formula is as follows: Where N represents the number of enhanced samples, h(X i ) and h(x i ) represent the expected feature vector and the feature vector of the real sample with the same label, respectively, and λ is a hyperparameter that determines the impact of the enhanced sample on the model.
9. A multi-scale spatiotemporal perception EEG data amplification system based on diffusion enhancement, characterized in that: include: The data acquisition and preprocessing module is used to collect the user's EEG data for preprocessing and convert it into a two-dimensional matrix array; The data normalization module is used to align and normalize the data of all users using the Euclidean alignment method, and divide all the data into training sets and test sets; The data enhancement diffusion model construction module is used to construct a diffusion enhancement model, and train the diffusion enhancement model using the training set data to obtain enhanced samples; The multi-scale spatiotemporal perception model construction module is used to construct a multi-scale spatiotemporal perception model and use the training set data for training. After completion, the parameters are fixed. A multi-scale spatiotemporal perception model training module is used to mix enhanced samples and real samples and input them into the multi-scale spatiotemporal perception model constructed by the multi-scale spatiotemporal perception model construction module through the diffusion enhancement model trained by the data enhancement diffusion model construction module to perform fine-tuning to obtain a trained multi-scale spatiotemporal perception model; The EEG emotion recognition module is used to input the test set into the trained multi-scale spatiotemporal perception model to obtain classified emotion labels.
10. The EEG data amplification system based on diffusion-enhanced multi-scale spatiotemporal perception according to claim 9, characterized in that: The multi-scale spatiotemporal perception model constructed by the multi-scale spatiotemporal perception model construction module includes a spatial frequency learning module and a time learning module. The spatial frequency learning module uses the preprocessed EEG signal as its input, generates a one-dimensional vector as its output, and obtains the necessary spatial frequency features; The temporal learning module improves the accuracy of emotion classification by effectively processing continuous EEG data and obtaining global and local temporal features.
Citation Information
Patent Citations
Electroencephalogram signal recognition system and method
CN111973178A
Electroencephalogram signal recognition model training method, recognition method, device and equipment
CN114418026A
Cross-domain electroencephalogram emotion recognition method and system based on space-time feature fusion model
CN114469137A
Headset capable of being used for monitoring and adjusting emotional state
CN117714935A
Cited By
Self-training-based biomedical signal-oriented data amplification method and system
CN119719869A
High-quality video reconstruction method based on electroencephalogram signals
CN119987549A
Pain electroencephalogram data enhancement method based on generative adversarial network and multi-feature fusion
CN120180183A
Electroencephalogram signal amplification method, system and terminal based on Gaussian mixture distribution model
CN120959760A
