EEG emotion recognition method and system based on generative adversarial network and self-supervised learning

Abstract

The application discloses an EEG emotion recognition method and system based on a generative adversarial network and self-supervised learning, comprising: pre-processing and topological mapping of an EEG signal, converting a one-dimensional time series EEG signal into an electroencephalogram spatiotemporal topological signal; performing hard span masking on the electroencephalogram spatiotemporal topological signal to obtain a masked electroencephalogram spatiotemporal topological signal; inputting the masked electroencephalogram spatiotemporal topological signal into a sample generation network to obtain an enhanced electroencephalogram spatiotemporal topological signal; constructing a self-supervised classification network, extracting spatial features by using a convolutional neural network, combining a bidirectional long short-term memory network to model time-dependent features, and realizing emotion classification; using the same processing on a to-be-recognized EEG signal to obtain a to-be-recognized electroencephalogram spatiotemporal topological signal, inputting the to-be-recognized electroencephalogram spatiotemporal topological signal into the trained self-supervised classification network, and outputting an emotion recognition result. The application can effectively improve the extraction capability of the EEG signal spatiotemporal features, enhance the adaptability of the model to time series missing and complex noise interference, and improve the accuracy and robustness of emotion recognition.

Description

Technical Field

[0001] This invention belongs to the field of emotion recognition technology, specifically relating to an EEG emotion recognition method and system based on generative adversarial networks and self-supervised learning. Background Technology

[0002] In the fields of affective computing and human-computer interaction, emotion recognition is a core component for achieving intelligent human-computer collaboration. Its recognition accuracy and robustness directly determine the interactive experience and system performance. Electroencephalography (EEG), as a physiological signal that directly reflects the activity state of neurons in the brain, can directly access the internal neural mechanisms that generate emotions. Compared to external signals such as facial expressions and speech, which are easily affected by subjective faking, EEG has higher reliability in emotion representation and has become one of the core research objects in the field of emotion recognition.

[0003] However, the inherent characteristics of EEG signals and the bottlenecks in data acquisition greatly limit the industrial application of EEG emotion recognition technology. On the one hand, EEG signals have inherent defects such as low signal-to-noise ratio, high data dimensionality, and significant individual differences. Brain activity signals are easily interfered with by physiological noise such as electromyography and electrooculography. At the same time, multi-channel acquisition mode leads to a surge in data dimensionality, and the differences in EEG waveforms and rhythm characteristics among different individuals are significant, posing a challenge to the unified modeling of signals. On the other hand, the acquisition cost of high-quality emotion-annotated EEG data is extremely high. It not only requires professional EEG acquisition equipment and experimental environment, but also requires strict control of the emotion induction conditions of subjects to ensure data validity. This results in existing public datasets generally having problems such as limited scale, class imbalance, and insufficient sample representativeness. This directly leads to overfitting during deep neural network training, and the model's generalization ability is difficult to meet the needs of practical applications.

[0004] Early EEG emotion recognition methods mostly adopted a technical approach of "manual feature extraction + classical classifiers". That is, they relied on the experience of domain experts to extract time-domain (such as mean and variance), frequency-domain (such as power spectral density), wavelet transform and other features from EEG signals, and then combined them with traditional classifiers such as support vector machines and random forests to complete emotion classification. The core drawback of the above approach is the over-reliance on expert experience. Manually designed features cannot adaptively capture the complex spatiotemporal correlation structure contained in EEG signals, and it is difficult to match the dynamic changes in brain activity when emotions are generated. Both recognition accuracy and robustness have bottlenecks.

[0005] With the development of deep learning technology, deep models such as Convolutional Neural Networks (CNNs) and Spatiotemporal Hybrid Networks have been gradually introduced into the field of EEG emotion recognition. To adapt to the requirement of two-dimensional data for convolutional operations, researchers typically map the spatial locations of EEG electrodes to a two-dimensional topological mesh and use two-dimensional convolutional kernels to model the spatial correlation between different electrodes. This improves the automation level of feature extraction to some extent. Meanwhile, to alleviate the small sample size problem, traditional data augmentation techniques such as truncation, adding noise, and time scaling have been attempted to expand the training samples. However, these methods can only perform minor local transformations on the original samples, generating mostly "nearest neighbor samples." Their effect on expanding the overall data distribution and improving sample diversity is weak, and their relevance to the core task of emotion recognition is insufficient, failing to fundamentally solve the data scarcity problem.

[0006] In recent years, deep generative models such as Generative Adversarial Networks (GANs) have been widely used in EEG data augmentation scenarios due to their powerful data distribution fitting capabilities. The core idea of ​​GANs is to train the generator against the discriminator, enabling the generator to gradually learn the distribution characteristics of real EEG data and generate a large number of virtual EEG samples consistent with the distribution of real samples, thereby expanding the training set. Existing research has confirmed that GANs have significant potential in alleviating the problem of small sample sizes in EEG emotion recognition, but in practical applications, they still face four major technical bottlenecks: First, it is difficult to balance the diversity and quality of generated samples, which can easily lead to... The problems include: firstly, the generation of samples suffers from "pattern collapse" (convergence of generated samples) or distortion of emotional semantics; secondly, insufficient modeling of the unique spatiotemporal topology of EEG signals, resulting in generated samples lacking physiological rationality and deviating from the actual EEG activity patterns; thirdly, the integration of the generation process with the downstream emotion classification task is too loose, and the feature distribution of the generated samples does not match the needs of the classifier, resulting in limited improvement in classification performance; and fourthly, insufficient utilization of self-supervised signals, with existing solutions mostly using simple distance terms with fixed weights as feature constraints, failing to fully explore the correlation information between "original samples - enhanced samples - perturbation intensity," and failing to build a systematic self-supervised training mechanism to improve the model's generalization ability.

[0007] To address some of the aforementioned issues, some studies have proposed a fusion scheme of "GAN-based EEG data augmentation and self-supervised classification constraints." A typical implementation involves: using the generator-discriminator structure of a GAN to model the distribution of EEG data, and adding structured perturbations such as time-slice occlusion to the generator input to enhance sample diversity; in the classification stage, a convolutional neural network is used to extract spatial features from the EEG signals after two-dimensional topological mapping, and a temporal modeling network is combined to model the temporal dependencies of the EEG signals; during training, supervised classification loss is used to optimize the classifier performance on real EEG samples, and feature consistency constraints are introduced for the augmented samples to improve the ability of the generated samples to support downstream emotion recognition tasks.

[0008] Although the above optimization schemes have achieved certain technical improvements, they still have significant shortcomings: First, the static time-slice occlusion strategy lacks modeling of the long-term temporal dependence of EEG signals, making it difficult to simulate the natural temporal evolution of EEG signals. Furthermore, it lacks explicit regulation of occlusion intensity based on physiological characteristics, making it difficult to balance the diversity of generated samples with the rationality of emotional semantics. Second, the generation process does not fully incorporate manifold consistency constraints, resulting in insufficient physiological dynamics support for the generated completion signals. The degree of matching between generated samples and real samples in the feature space is insufficient, limiting its effective support for downstream emotion classification tasks. Third, there is a lack of close collaboration between the generation task and the classification task. Existing self-supervised constraints are mostly simple mappings with fixed weights, making it difficult to fully utilize the correlation information between original samples, augmented samples, and perturbation intensity, leading to insufficient model collaborative optimization capabilities. Fourth, after extracting spatial features of EEG signals, existing classification models often fail to adequately model temporal dependencies, making it difficult to fully characterize the dynamic changes of EEG signals between different time segments, thus limiting the accuracy and robustness of emotion recognition in complex environments.

[0009] In summary, existing EEG emotion recognition technology still faces many unresolved issues in areas such as data augmentation, network modeling, and task collaboration. Constructing a data augmentation mechanism that balances sample quality and diversity, achieving collaborative optimization between generative and classification networks, and fully utilizing self-supervised signals to enhance model generalization capabilities have become key breakthroughs for driving EEG emotion recognition technology towards practical applications. Summary of the Invention

[0010] To overcome the shortcomings of existing technologies, this invention provides an EEG emotion recognition method based on generative adversarial networks and self-supervised learning. This EEG emotion recognition method can improve the diversity of sample distribution while ensuring the physiological rationality of generated samples. Furthermore, it enhances the joint modeling ability of spatial and temporal dependent features of EEG signals by using a classifier backbone that combines convolutional neural networks and bidirectional long short-term memory networks, thereby improving the accuracy, stability, and robustness of emotion recognition in complex noisy environments.

[0011] The second objective of this invention is to provide an EEG emotion recognition system based on generative adversarial networks and self-supervised learning.

[0012] The technical solution of the present invention to solve the above-mentioned technical problems is:

[0013] An EEG emotion recognition method based on generative adversarial networks and self-supervised learning includes the following steps:

[0014] Step S1: Acquire EEG signals, perform preset data preprocessing and topology mapping operations on the EEG signals, and convert the one-dimensional time-series EEG signals into original EEG spatiotemporal topology signals that integrate time dimension and spatial electrode distribution information.

[0015] Step S2: The continuous temporal region of the original EEG spatiotemporal topology signal is masked using a hard span masking strategy to obtain a masked EEG spatiotemporal topology signal with continuous temporal loss characteristics.

[0016] Step S3: Construct a generative adversarial network (GAN) sample generation network containing a generator and a discriminator. Input the masked EEG spatiotemporal topology signal into the generator, and obtain the reconstructed EEG spatiotemporal topology signal through manifold reconstruction transformation. The discriminator performs a realism judgment between the reconstructed EEG spatiotemporal topology signal and the original EEG spatiotemporal topology signal, and constructs an adversarial loss function based on the judgment result. At the same time, combined with manifold consistency constraints, the network parameters of the generator and discriminator are iteratively optimized through gradient backpropagation until the preset convergence condition is met. The output is an enhanced EEG spatiotemporal topology signal that matches the spatiotemporal topology features of the original EEG spatiotemporal topology signal and has physiological rationality.

[0017] Step S4: Construct a self-supervised classification network, which uses a classifier backbone combining a convolutional neural network and a bidirectional long short-term memory network; input the original EEG spatiotemporal topology signal and the enhanced EEG spatiotemporal topology signal as joint training samples into the self-supervised classification network; construct a global total loss function, which includes the adversarial loss of the sample generation network, the supervised classification loss of the self-supervised classification network, and the self-supervised feature consistency loss, and balance the contribution of each loss term in the global optimization by a preset weight coefficient; with the goal of minimizing the global total loss function, the network parameters of the sample generation network and the self-supervised classification network are synchronously and iteratively updated using a gradient backpropagation mechanism until the preset convergence condition is met, and the trained self-supervised classification network is obtained;

[0018] Step S5: Obtain the EEG signal to be identified. Using the same data preprocessing procedure and topology mapping operation as in Step S1, convert the EEG signal to be identified into the spatiotemporal topological signal of the EEG to be identified. Input the spatiotemporal topological signal of the EEG to be identified into the trained self-supervised classification network. The convolutional neural network extracts spatial features, and the bidirectional long short-term memory network models the temporal dependency relationship to output the final emotion recognition result.

[0019] Preferably, in step S1, the original EEG signal is subjected to baseline correction, rereference, and band filtering in sequence to obtain a standardized multi-channel EEG time series signal. Based on the spatial distribution relationship of electrode positions in the international 10-20 electrode system, a three-dimensional coordinate projection algorithm is used to map the spatial position information of each channel of the standardized multi-channel EEG time series signal to a preset two-dimensional planar grid. Then, the time series signal of each channel is spatially allocated according to the mapped grid position to obtain an EEG spatiotemporal topological signal that has time dimension, spatial row dimension, and spatial column dimension. This EEG spatiotemporal topological signal is a three-dimensional data set with a corresponding three-dimensional length.

[0020] Preferably, in step S2; a physiological feature-guided hard span masking strategy is used to mask the original EEG spatiotemporal topological signal to obtain a masked EEG spatiotemporal topological signal with continuous temporal missing features, wherein the steps of the hard span masking strategy are as follows:

[0021] A physiological feature-guided random matrix is ​​generated that is completely consistent with the dimension of the original EEG spatiotemporal topology signal. The elements of the physiological feature-guided random matrix range from 0 to 1, and the weight distribution of each frequency band corresponds to the instantaneous energy density of the original EEG spatiotemporal topology signal. That is, the weight distribution accurately reflects the difference in instantaneous energy density of each frequency band of the original EEG spatiotemporal topology signal.

[0022] The mask span parameter and enhancement factor are sampled from a preset parameter range respectively; the mask span parameter ranges from 0 to a preset maximum span value and is used to define the length of continuous occlusion in the time domain; the enhancement factor ranges from a preset minimum enhancement value to a preset maximum enhancement value and is used to adjust the occlusion intensity of non-span regions.

[0023] A masking start position is randomly selected in the time domain dimension of the original EEG spatiotemporal topology signal. The start position must satisfy the condition that the continuous masking length from the masking start position does not exceed the total time domain length of the original EEG spatiotemporal topology signal, ensuring that the masking range does not exceed the signal boundary. Starting from the masking start position, all topology elements within the continuous mask span parameter step size are set to 0 to generate a binary span mask, which is used to simulate long-range time domain signal missing scenarios.

[0024] For non-span regions in the original EEG spatiotemporal topology signal that are not covered by the span mask, the physiological features are used to guide the random matrix and the enhancement factor to generate a discrete mask with a value range of 0 to 1. This discrete mask is used to achieve discontinuous random masking of non-span regions, and the masking process conforms to the distribution of physiological features of the original signal.

[0025] Each element of the original EEG spatiotemporal topology signal is multiplied element-by-element by the span mask and the discrete mask to obtain the masked EEG spatiotemporal topology signal after joint masking.

[0026] Preferably, in step S3, the generator adopts a UNet-like hierarchical spatiotemporal fusion structure and introduces manifold-consistent skip connections. The generator includes an encoding module, a decoding module, and an output module.

[0027] The encoding module captures the local physiological activity patterns of the input masked EEG spatiotemporal topology signal through multi-scale two-dimensional convolution operations, and simultaneously performs downsampling processing to obtain high-dimensional spatiotemporal features.

[0028] The decoding module gradually upsamples the high-dimensional spatiotemporal features output by the encoding module through deconvolution operations to restore them to the same spatial resolution as the occluded EEG spatiotemporal topology signal; at the same time, it uses the manifold consistency jump connection to force the geometric alignment of the corresponding features of the encoding module and the decoding module in the feature manifold space; and it synchronously fuses the low-dimensional detail features of the corresponding level of the encoding module with the high-dimensional semantic features of the decoding module to achieve multi-scale feature complementary fusion.

[0029] The output module is used to perform feature mapping processing on the preliminary generation result output by the decoding module to obtain a candidate reconstruction signal with the same dimension as the original EEG spatiotemporal topology signal. The candidate reconstruction signal is then multiplied element-wise with a preset electrode channel prior mask to meet the physical constraints of the EEG signal. The electrode channel prior mask is a two-dimensional binary matrix that perfectly matches the spatial dimension of the EEG signal. Its elements are only 0 or 1. When the element at the corresponding position of the mask is 1, the corresponding position value of the candidate reconstruction signal is retained. When the element at the corresponding position of the mask is 0, the corresponding position value of the candidate reconstruction signal is forcibly set to 0. Finally, the reconstructed EEG spatiotemporal topology signal is output.

[0030] Preferably, in step S3, the discriminator adopts a dynamic co-occurrence spatiotemporal convolutional network structure, which mainly includes a shallow feature extraction layer, a middle feature optimization layer, and a deep feature aggregation layer, wherein,

[0031] The shallow feature extraction layer uses multi-scale two-dimensional convolutional kernels and the SELU activation function to capture local spatiotemporal features of input samples containing the original EEG spatiotemporal topological signal and the reconstructed EEG spatiotemporal topological signal output by the generator. By covering spatiotemporal correlation information of different ranges through multi-scale convolutional kernels, the detailed features and basic spatiotemporal patterns of the signal are initially extracted.

[0032] The mid-layer feature optimization layer uses depthwise separable convolution, which reduces the number of network parameters and computational complexity while maintaining feature expressive power. At the same time, it separates the coupling relationship of spatiotemporal features and strengthens the independent feature modeling of the temporal dynamic changes and spatial distribution differences of EEG signals.

[0033] The deep feature aggregation layer adopts an Inception-like multi-branch convolutional structure, extracting features in parallel through multiple convolutional branches with different receptive fields, aggregating deep spatiotemporal semantic features at different scales, and finally mapping the aggregated feature vector to a realism score with a value range of [0,1] through a fully connected layer, outputting the realism evaluation result of the input sample.

[0034] Preferably, in step S3, the training steps of the sample generation network are as follows:

[0035] A generator loss function combining manifold consistency constraints is adopted. By minimizing this loss function, the generator is guided to accurately reconstruct the masked EEG spatiotemporal topology signal. This makes the generated reconstructed EEG spatiotemporal topology signal continuously approach the truth score of the original EEG spatiotemporal topology signal in the truth score output by the discriminator. Ultimately, the distribution error between the reconstructed EEG spatiotemporal topology signal and the original EEG spatiotemporal topology signal is lower than the preset error threshold.

[0036] A Wasserstein loss function with gradient penalty is used as the discriminator loss function. The discriminator loss function includes a real sample discrimination score term, a generated sample discrimination score term, and a gradient penalty constraint term. The gradient penalty constraint term is calculated using intermediate samples obtained by linear interpolation of the original EEG spatiotemporal topological signal and the reconstructed EEG spatiotemporal topological signal output by the generator using random coefficients. By calculating the squared deviation of the gradient L2 norm of this intermediate sample after processing by the discriminator from a fixed value of 1, the gradient of the discriminator is forced to satisfy the Lipschitz continuity constraint condition, ensuring the training stability and realism evaluation accuracy of the discriminator.

[0037] The network parameters of the generator are fixed, and the network parameters of the discriminator are iteratively updated using the backpropagation algorithm by minimizing the loss function of the discriminator. After the discriminator's ability to distinguish between the original EEG spatiotemporal topological signal and the reconstructed EEG spatiotemporal topological signal reaches a preset level and the fluctuation of the loss function value is lower than a set threshold, the updated network parameters of the discriminator are fixed, and the network parameters of the generator are iteratively updated using the backpropagation algorithm by minimizing the loss function of the generator. The above alternating optimization process is repeated until the sample generation network meets the preset training convergence condition, and finally the trained sample generation network is obtained.

[0038] Preferably, in step S4, the self-supervised classification network includes an emotion classifier and a feature extraction module, wherein,

[0039] The emotion classifier employs a classifier backbone combining a convolutional neural network and a bidirectional long short-term memory network. The convolutional neural network is used to extract spatial features from the input EEG spatiotemporal topology signal, and the bidirectional long short-term memory network is used to model the temporal series dependencies of the spatial features. A fully connected layer is set after the bidirectional long short-term memory network, and combined with the Softmax activation function, the output is the probability of each emotion category corresponding to the input EEG spatiotemporal topology signal.

[0040] The last fully connected layer and the Softmax activation function in the emotion classifier are removed to obtain the feature extraction module. The feature extraction module is used to receive EEG spatiotemporal topological signals and enhanced EEG spatiotemporal topological signals, and output a fixed-dimensional feature vector representing the spatiotemporal features of the input signal.

[0041] Preferably, the training steps of the self-supervised classification network are as follows:

[0042] Based on the emotion labels of the original EEG spatiotemporal topology signals, a supervised classification loss is calculated to constrain the emotion discrimination accuracy of the emotion classifier.

[0043] Based on the consistency constraint between the feature vectors obtained by the feature extraction module of the original EEG spatiotemporal topology signal and the enhanced EEG spatiotemporal topology signal, the self-supervised feature consistency loss is calculated to constrain the emotion classifier to learn robust feature representations that are insensitive to temporal missing data and local perturbations.

[0044] The supervised classification loss and the self-supervised feature consistency loss are fused to obtain the total loss of the self-supervised classification network. The total loss is then used as the optimization objective, and the gradient backpropagation algorithm is employed to update the network parameters of the convolutional neural network, bidirectional long short-term memory network, and fully connected layer in the emotion classifier.

[0045] A global total loss function is constructed, which includes the adversarial loss of the sample generation network and the total loss of the self-supervised classification network, and the contribution of each loss term in the global optimization is balanced by a preset weight coefficient.

[0046] By minimizing the global total loss function, the network parameters of the sample generation network and the self-supervised classification network are simultaneously and iteratively optimized until the preset training convergence condition is met, thus obtaining the trained self-supervised classification network.

[0047] Preferably, in step S5, the step of obtaining the emotion recognition result is as follows:

[0048] The spatiotemporal topological signal of the EEG to be identified, after data preprocessing and topological mapping, is input into the trained self-supervised classification network;

[0049] The convolutional neural network in the emotion classifier extracts spatial features from the spatiotemporal topological signal of the EEG to be identified. The convolutional neural network extracts and compresses the local spatial correlation information in the spatiotemporal topological signal of the EEG step by step through multi-layer convolution, normalization, activation and pooling operations to obtain a spatial feature sequence that represents the spatial distribution characteristics of different time segments.

[0050] The spatial feature sequence is input into a bidirectional long short-term memory network. The dependency relationship between time segments before and after the EEG signal is jointly modeled by forward temporal modeling and reverse temporal modeling. The temporal evolution features in the spatiotemporal topological signal of the EEG to be identified are extracted to obtain a spatiotemporal joint feature representation that integrates spatial distribution information and temporal dependency information.

[0051] The spatiotemporal joint feature representation is input into the fully connected layer for feature mapping, and the predicted probability of each emotion category corresponding to the spatiotemporal topological signal of the EEG to be identified is output by combining the Softmax function.

[0052] The emotion category with the highest predicted probability is selected as the final emotion recognition result.

[0053] An EEG emotion recognition system based on generative adversarial networks and self-supervised learning includes:

[0054] Data processing module: used to perform preset data preprocessing and topology mapping operations on the acquired EEG signals, converting one-dimensional time-series EEG signals into raw EEG spatiotemporal topology signals that integrate time dimension information and spatial electrode distribution information;

[0055] Temporal masking module: Used to mask the continuous temporal region of the original EEG spatiotemporal topology signal output by the data processing module using a hard span masking strategy, and generate a masked EEG spatiotemporal topology signal with continuous temporal loss characteristics.

[0056] The sample generation module includes a generator and a discriminator. The generator receives the masked EEG spatiotemporal topology signal output by the temporal masking module, and outputs a reconstructed EEG spatiotemporal topology signal after manifold reconstruction transformation. The discriminator performs authenticity discrimination between the reconstructed EEG spatiotemporal topology signal and the original EEG spatiotemporal topology signal output by the data processing module, and constructs an adversarial loss function based on the discrimination result. Simultaneously, combined with manifold consistency constraints, the network parameters of the generator and discriminator are iteratively optimized through gradient backpropagation until a preset convergence condition is met, and an enhanced EEG spatiotemporal topology signal that matches the spatiotemporal topology features of the original EEG spatiotemporal topology signal and has physiological rationality is output.

[0057] Self-supervised classification module: Used to receive the original EEG spatiotemporal topological signal output by the data processing module and the enhanced EEG spatiotemporal topological signal output by the sample generation module as joint training samples, extract spatial features using a convolutional neural network, and model time series dependencies using a bidirectional long short-term memory network to output emotion classification results;

[0058] Joint training module: used to construct the global total loss function, which includes the adversarial loss of the sample generation module, the supervised classification loss of the self-supervised classification module, and the self-supervised feature consistency loss. The contribution weight of each loss term in the model optimization is balanced by preset weight coefficients. With the goal of minimizing the global total loss function, the network parameters of the sample generation module and the self-supervised classification module are synchronously and iteratively updated using the gradient backpropagation mechanism until the preset convergence condition is met, and the trained self-supervised classification network is obtained.

[0059] Emotion Recognition Module: Used to acquire the EEG signal to be recognized, call the preset process and algorithm of the data processing module to convert the EEG signal to be recognized into the spatiotemporal topological signal of the EEG to be recognized; input the spatiotemporal topological signal of the EEG to be recognized into the trained self-supervised classification module, extract the spatial feature sequence by the convolutional neural network, and model the temporal dependency by the bidirectional long short-term memory network, and output the final emotion recognition result.

[0060] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0061] 1. This invention achieves deep collaboration between generation and classification tasks through a fully integrated neurodynamic manifold design. Based on a unified two-dimensional topological mapping constructed in the preprocessing stage, the Sample Generation Network (SGN) introduces manifold consistency constraints. This not only makes the augmented samples statistically approximate real EEG signals, but also forces them to strictly follow the inherent neurodynamic manifold laws of EEG signals in the low-dimensional embedding space. This ensures a high degree of isomorphism between the generated samples and the classification feature space, significantly reducing the adaptation cost of migrating augmented samples to classification tasks and providing direct, efficient, and effective feature support for downstream emotion recognition.

[0062] 2. This invention employs a physiologically-guided hard span masking strategy, combining mask span and enhancement factor to precisely characterize the depth of perturbation, achieving efficient mining of long-range temporal dependencies. On one hand, the generator is forced to reconstruct physiological patterns in missing continuous time spans, effectively simulating real temporal loss scenarios in EEG signals. This significantly improves sample diversity while strengthening the model's ability to capture long-range temporal dependencies, solving the problem of insufficient modeling of temporal correlations in EEG signals. On the other hand, through dual constraints of channel masking and WGAN-GP, the physical rationality and physiological semantic consistency of the generated sample's spatiotemporal structure are strictly guaranteed, avoiding sample distortion caused by excessive pursuit of diversity. Based on the adjustable parameters of mask span and enhancement factor, a precise balance is established between diversity and physiological consistency, ensuring that the generated samples possess sufficient perturbation while conforming to the physiological characteristics of EEG signals, providing high-quality data support for model training in small-sample scenarios.

[0063] 3. This invention uses the original EEG spatiotemporal topological signal and the enhanced EEG spatiotemporal topological signal as joint training samples, and introduces the joint constraints of supervised classification loss and self-supervised feature consistency loss. This enables the classification network to learn robust feature representations that are insensitive to temporal missing data and local perturbations while using labeled samples for emotion discrimination learning. This improves the model's generalization ability and cross-sample adaptation ability under small sample conditions.

[0064] 4. This invention employs a classifier backbone that combines a convolutional neural network and a bidirectional long short-term memory network. The convolutional neural network can effectively extract local spatial correlation features from the spatiotemporal topological signals of EEG, while the bidirectional long short-term memory network can simultaneously model the dependency relationship between time segments of EEG signals, thereby achieving a joint representation of spatial and temporal features. This improves the ability to recognize complex emotional states and enhances the accuracy and robustness of the model in complex noisy environments.

[0065] 5. This invention obtains high-quality augmented samples with physiological rationality by performing hard span occlusion and manifold constraint adversarial training; improves the model's generalization ability through the joint constraint of supervised classification loss and self-supervised feature consistency loss; and significantly improves the accuracy, stability and robustness of EEG emotion recognition by using convolutional neural networks and bidirectional long short-term memory networks to extract spatial features and model temporal dependencies of EEG signals. Attached Figure Description

[0066] Figure 1 This is a flowchart illustrating the EEG emotion recognition method based on generative adversarial networks and self-supervised learning of the present invention.

[0067] Figure 2 This is a schematic diagram of how EEG signals are mapped from a one-dimensional channel sequence to a two-dimensional spatiotemporal topological grid in this invention.

[0068] Figure 3 This is an overall block diagram of the EEG emotion recognition system based on generative adversarial networks and self-supervised learning according to the present invention.

[0069] Figure 4 This is a schematic diagram comparing the hard span occlusion strategy and the random occlusion strategy in this invention.

[0070] Figure 5 This is a schematic diagram of the structure and training process of the sample generation network in this invention.

[0071] Figure 6 This is a schematic diagram of the training process of the self-supervised classification network in this invention.

[0072] Figure 7 This is a schematic diagram of the emotion recognition process based on convolutional neural networks and bidirectional long short-term memory networks in this invention. Detailed Implementation

[0073] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0074] Example 1

[0075] See Figures 1-7 The EEG emotion recognition method based on generative adversarial networks and self-supervised learning of the present invention includes the following steps:

[0076] Step S1: Acquire EEG signals, perform preset data preprocessing and topological mapping operations on the EEG signals, and convert the one-dimensional time-series EEG signals into raw EEG spatiotemporal topological signals that integrate time dimension and spatial electrode distribution information, specifically:

[0077] Step S101: The original EEG signal is subjected to baseline correction, rereference and band filtering in sequence to obtain a standardized multi-channel EEG time series signal;

[0078] Step S102: Based on the preprocessed standardized multi-channel EEG time-series signal, according to the spatial distribution relationship of electrode positions in the international 10-20 electrode system, a three-dimensional coordinate projection algorithm is used to map the spatial position information of each channel in the standardized multi-channel EEG time-series signal to a preset two-dimensional planar grid. Then, the time-series signal of each channel is spatially allocated according to the mapped grid position to obtain an EEG spatiotemporal topological signal with time dimension, spatial row dimension, and spatial column dimension. The data scale of this EEG spatiotemporal topological signal corresponds to a three-dimensional data set with time dimension length, spatial row dimension length, and spatial column dimension length, i.e., the EEG spatiotemporal topological signal. For a three-dimensional tensor:

[0079] ;

[0080] In the formula: For time steps; The size of the two-dimensional grid is obtained by mapping the physical coordinates of the electrodes;

[0081] Step S2: To introduce controllable diversity simulating long-range missing data, a physiologically guided hard-span masking strategy is used to mask the original EEG spatiotemporal topological signal, resulting in a masked EEG spatiotemporal topological signal with continuous temporal missing features. The specific steps are as follows:

[0082] Step S201: Generate spatiotemporal topological signals related to the original EEG signals Physiological characteristics with completely identical dimensions guide random matrices The physiological characteristics guide the random matrix. The element values ​​range from 0 to 1, and the weight distribution of each frequency band is related to the original EEG spatiotemporal topology signal. The instantaneous energy density is in a corresponding relationship;

[0083] Step S202: Sample the mask span parameter from the preset parameter range respectively. With enhancing factors The mask span parameter The value ranges from 0 to the preset maximum span value. ,Right now The enhancement factor is used to define the length of continuous occlusion in the temporal domain. The value range is the preset minimum enhancement value. Up to the preset maximum enhancement value ,Right now Used to adjust the shading intensity of non-spanning areas;

[0084] Step S203: In the original EEG spatiotemporal topological signal Randomly select the occlusion start position in the time domain dimension The starting position must satisfy the condition that, starting from the occlusion start position, the continuous occlusion length does not exceed the original EEG spatiotemporal topology signal. The total time domain length, i.e. Ensure that the masking range does not exceed the signal boundary; starting from the masking start position. Starting from the first element, all topological elements within the continuous mask span parameter step size are forcibly set to 0 to generate a binary span mask. This is used to simulate scenarios where long-range time-domain signals are missing.

[0085] Step S204: Targeting the original EEG spatiotemporal topological signals The middle was not masked by the span. The non-spanning region covered guides the physiological characteristics into a random matrix. With enhancing factors Perform the operation to generate a discrete mask with values ​​ranging from 0 to 1. This enables discontinuous random masking of non-spanning regions, and the masking process conforms to the original spatiotemporal topological signals of the EEG. Distribution of physiological characteristics;

[0086] Step S205: Convert the original EEG spatiotemporal topology signal Each element in the range is associated with the span mask. Discrete mask By performing element-wise multiplication operations sequentially, the combined masked EEG spatiotemporal topological signal is obtained. ,Right now:

[0087]

[0088] In the formula: For Hadamard products. The span parameter represents the duration of continuous temporal occlusion. This represents the intensity adjustment parameter for random occlusion in non-span areas.

[0089] Construct a training set, which contains a total of A labeled EEG sample, denoted as ,in, These are one-hot hashtags for sentiment categories; the generator is denoted as... The discriminator is denoted as The classifier is denoted as And its feature extraction module is denoted as .

[0090] Step S3: Construct a generative adversarial network (GAN) sample generation network containing a generator and a discriminator. Input the masked EEG spatiotemporal topology signal into the generator, and obtain the reconstructed EEG spatiotemporal topology signal through manifold reconstruction transformation. The discriminator performs a realism judgment between the reconstructed EEG spatiotemporal topology signal and the original EEG spatiotemporal topology signal, and constructs an adversarial loss function based on the judgment result. At the same time, combined with manifold consistency constraints, the network parameters of the generator and discriminator are iteratively optimized through gradient backpropagation until the preset convergence condition is met. The output is an enhanced EEG spatiotemporal topology signal that matches the spatiotemporal topology features of the original EEG spatiotemporal topology signal and has physiological rationality.

[0091] In this embodiment, the Sample Generation Network (SGN) adopts a generative adversarial network architecture, combining a physiologically-guided hard span masking strategy with WGAN-GP adversarial training to generate high-fidelity augmented samples with a distribution that follows a neurodynamic manifold and possesses controlled diversity. Specifically, its structure and training process correspond to... Figure 4 As shown, where,

[0092] The generator adopts a UNet-like hierarchical spatiotemporal fusion structure and introduces manifold-consistent skip connections. The generator includes an encoding module (encoder), a decoding module (decoder), and an output module.

[0093] The encoding module captures the spatiotemporal topological signals of the input masked EEG signals through multi-scale two-dimensional convolution operations. The local physiological activity patterns are analyzed, and downsampling is performed simultaneously to obtain high-dimensional spatiotemporal features;

[0094] The decoding module uses deconvolution operations to progressively upsample the high-dimensional spatiotemporal features output by the encoding module to restore them to the spatiotemporal topology of the masked EEG signal. The same spatial resolution; at the same time, by utilizing the manifold consistency jump connection, the geometric alignment of the corresponding features of the encoding module and the decoding module is forced in the feature manifold space; the low-dimensional detail features of the corresponding level of the encoding module and the high-dimensional semantic features of the decoding module are fused synchronously to achieve multi-scale feature complementary fusion;

[0095] The output module is used to perform feature mapping processing on the preliminary generation results output by the decoding module to obtain the original EEG spatiotemporal topological signal. Candidate reconstruction signals with consistent dimensions are compared with a preset electrode channel prior mask. A position-wise element-wise multiplication operation is performed to satisfy the physical constraints of the EEG signal, wherein the electrode channel prior mask... A two-dimensional binary matrix that perfectly matches the spatial dimension of EEG signals is used, with elements taking only 0 or 1 values. When the element at the corresponding position of the mask is 1, the corresponding position value of the candidate reconstructed signal is retained; when the element at the corresponding position of the mask is 0, the corresponding position value of the candidate reconstructed signal is forcibly set to 0. The final output is a reconstructed EEG spatiotemporal topology signal, which serves as a candidate enhanced EEG spatiotemporal topology signal. ;

[0096] ;

[0097] The core mathematical expression of a generator;

[0098] ;

[0099] The discriminator employs a Dynamic Co-occurrence Spatiotemporal Convolutional Network (STNet) structure to estimate the "realism" of input samples. Its main components include shallow feature extraction layers, mid-level feature optimization layers, and deep feature aggregation layers.

[0100] The shallow feature extraction layer uses multi-scale two-dimensional convolutional kernels and the SELU activation function to capture local spatiotemporal features of input samples containing the original EEG spatiotemporal topological signal and the reconstructed EEG spatiotemporal topological signal output by the generator. By covering spatiotemporal correlation information of different ranges through multi-scale convolutional kernels, the detailed features and basic spatiotemporal patterns of the signal are initially extracted.

[0101] The mid-layer feature optimization layer uses depthwise separable convolution, which reduces the number of network parameters and computational complexity while maintaining feature expressive power. At the same time, it separates the coupling relationship of spatiotemporal features and strengthens the independent feature modeling of the temporal dynamic changes and spatial distribution differences of EEG signals.

[0102] The deep feature aggregation layer adopts an Inception-like multi-branch convolutional structure, extracting features in parallel through multiple convolutional branches with different receptive fields, aggregating deep spatiotemporal semantic features at different scales, and finally mapping the aggregated feature vector to a realism score with a value range of [0,1] through a fully connected layer, outputting the realism evaluation result of the input sample;

[0103] Based on the structure of the generator and discriminator described above, the training steps of the sample generation network are as follows:

[0104] Step S301: A generator loss function combined with manifold consistency constraints is adopted. By minimizing the generator loss function, the generator is guided to accurately reconstruct the masked EEG spatiotemporal topology signal. This causes the generated reconstructed EEG spatiotemporal topology signal to continuously approach the authenticity score of the original EEG spatiotemporal topology signal in the authenticity score output by the discriminator. Ultimately, the distribution error between the reconstructed EEG spatiotemporal topology signal and the original EEG spatiotemporal topology signal is lower than the preset error threshold.

[0105] To ensure the generation of enhanced EEG spatiotemporal topological signals In accordance with the laws of neurodynamics, this invention employs a generator loss that incorporates manifold consistency constraints. :

[0106] ;

[0107] In the formula Constraints enhance the geometric consistency between EEG spatiotemporal topological signals and the original EEG spatiotemporal topological signals in low-dimensional embedded manifolds.

[0108] Step S302: The Wasserstein loss function with gradient penalty is used as the discriminator loss function. The discriminator loss function includes a real sample discrimination score term, a generated sample discrimination score term, and a gradient penalty constraint term. The gradient penalty constraint term is calculated using intermediate samples obtained by linear interpolation of the original EEG spatiotemporal topological signal and the reconstructed EEG spatiotemporal topological signal output by the generator using random coefficients. By calculating the squared deviation of the gradient L2 norm of the intermediate sample after processing by the discriminator from the fixed value 1, the gradient of the discriminator is forced to satisfy the Lipschitz continuity constraint condition to ensure the training stability and realism evaluation accuracy of the discriminator.

[0109] For the discriminator, this invention introduces a gradient penalty term to force the discriminator to satisfy Lipschitz continuity. The discriminator loss function is defined as:

[0110] ;

[0111]

[0112] In the formula: From the original spatiotemporal topological signals of EEG With enhanced spatiotemporal topological signals of brainwaves Intermediate samples obtained by linear interpolation using random coefficients.

[0113] Step S303: Fix the network parameters of the generator, and iteratively update the network parameters of the discriminator by minimizing the discriminator loss function using the backpropagation algorithm; after the discriminator's ability to distinguish between the original EEG spatiotemporal topological signal and the reconstructed EEG spatiotemporal topological signal reaches a preset level and the fluctuation range of the loss function value is lower than a set threshold, fix the updated network parameters of the discriminator, and iteratively update the network parameters of the generator by minimizing the generator loss function using the backpropagation algorithm; repeat the above alternating optimization process until the sample generation network meets the preset training convergence condition, and finally obtain the trained sample generation network;

[0114] During the training process described above, the discriminator loss function is minimized. Improve the discriminator's ability to assess realism, i.e., its ability to distinguish between real and fake samples, while minimizing the generator's loss function. The generator is used to complete the occluded region, thus "deceiving" the discriminator. The two work in an adversarial game, alternately optimizing each other to ultimately output highly discriminative and realistically distributed enhanced EEG samples (i.e., enhanced spatiotemporal topological signals of brainwaves). The training process is as follows: Figure 4 As shown; the above training objective can be written as:

[0115] ;

[0116] In the formula This ensures that the generated enhanced EEG spatiotemporal topological signals satisfy manifold alignment constraints in the low-dimensional feature space. Through pre-training, the sample generation network is able to generate enhanced samples with physiological rationality and controlled diversity, laying the foundation for semantic learning of the self-supervised classification network.

[0117] Step S4: Construct a self-supervised classification network, which uses a classifier backbone combining a convolutional neural network and a bidirectional long short-term memory network; input the original EEG spatiotemporal topology signal and the enhanced EEG spatiotemporal topology signal as joint training samples into the self-supervised classification network; construct a global total loss function, which includes the adversarial loss of the sample generation network, the supervised classification loss of the self-supervised classification network, and the self-supervised feature consistency loss, and balance the contribution of each loss term in the global optimization by a preset weight coefficient; with the goal of minimizing the global total loss function, the network parameters of the sample generation network and the self-supervised classification network are synchronously and iteratively updated using a gradient backpropagation mechanism until the preset convergence condition is met, and the trained self-supervised classification network is obtained;

[0118] In this embodiment, a self-supervised classification network is used to train an emotion classifier in the joint representation space of the original EEG spatiotemporal topological signal and the enhanced EEG spatiotemporal topological signal. Its structure corresponds to the training process. Figure 6 As shown. The self-supervised classification network includes an emotion classifier and a feature extraction module, wherein the emotion classifier adopts a classifier backbone that combines a convolutional neural network and a bidirectional long short-term memory network.

[0119] The convolutional neural network is used to extract spatial features from the input EEG spatiotemporal topology signal, and the bidirectional long short-term memory network is used to model the temporal series dependencies of the spatial features. A fully connected layer is placed after the bidirectional long short-term memory network, and a softmax activation function is applied to output the probabilities of each emotion category corresponding to the input EEG spatiotemporal topology signal. The feature extraction module is obtained by removing the last fully connected layer and the softmax activation function from the emotion classifier. The feature extraction module is used to receive EEG spatiotemporal topology signals and enhanced EEG spatiotemporal topology signals, and output a fixed-dimensional feature vector characterizing the spatiotemporal features of the input signal.

[0120] Among them, the feature extraction module for:

[0121] ;

[0122] Based on the structure of the self-supervised classification network described above, the training steps of the self-supervised classification network are as follows:

[0123] Step S401: Convert the original EEG spatiotemporal topology signal With enhanced spatiotemporal topological signals of brainwaves The feature extraction modules of the emotion classifier are respectively input into the function. In the process, the corresponding spatiotemporal feature vectors are extracted. The feature extraction module is obtained by removing the last fully connected layer and the Softmax activation function from the emotion classifier. The emotion classifier uses a classifier backbone combining a convolutional neural network and a bidirectional long short-term memory network. The convolutional neural network is used to extract the spatial features of the EEG spatiotemporal topological signal, and the bidirectional long short-term memory network is used to model the temporal series dependencies of the spatial features. Based on the emotion labels of the original EEG spatiotemporal topological signal, a supervised classification loss is calculated to constrain the emotion discrimination accuracy of the emotion classifier. Based on the consistency constraints between the feature vectors obtained by the feature extraction module from the original EEG spatiotemporal topological signal and the enhanced EEG spatiotemporal topological signal, a self-supervised feature consistency loss is calculated to constrain the emotion classifier to learn robust feature representations that are insensitive to temporal missing data and local perturbations.

[0124] For each labeled real sample ( The emotion classifier outputs a category probability vector. = The supervised classification loss uses cross-entropy loss, defined as:

[0125]

[0126] In the formula, This represents the number of samples in the current mini-batch. For the number of emotion categories, For the sample In the Real labels on the class, For the emotion classifier to test samples Belongs to the The predicted probability of a class.

[0127] For each real sample The sample generation network generates corresponding augmented samples. The feature vectors are extracted by the feature extraction module respectively. and The self-supervised feature consistency loss is defined as:

[0128]

[0129] The self-supervised feature consistency loss constrains the consistency of the representations of the original samples and the augmented samples in the feature space, enabling the emotion classifier to maintain a consistent response to the core emotional representations of the original EEG samples when training with augmented samples.

[0130] Step S402: The supervised classification loss and the self-supervised feature consistency loss are fused to obtain the total loss of the self-supervised classification network. Using this total loss as the optimization objective, the gradient backpropagation algorithm is used to update the network parameters of the convolutional neural network, bidirectional long short-term memory network and fully connected layer in the emotion classifier, so that the emotion classifier can improve the emotion discrimination ability and enhance the feature robustness of masked and imputed samples and locally perturbed samples.

[0131] The total loss of a self-supervised classification network is defined as follows:

[0132]

[0133] In the formula, is the weight coefficient of the self-supervised feature consistency loss, used to adjust the relative contributions of supervised classification loss and feature consistency loss in the training of the self-supervised classification network.

[0134] Step S403: Construct a global total loss function, which includes the adversarial loss of the sample generation network and the total loss of the self-supervised classification network. The contribution of each loss term in the global optimization is balanced by preset weight coefficients, so that the sample generation network can generate high-quality enhanced samples while the self-supervised classification network can make full use of the enhanced samples to improve the emotion recognition performance.

[0135] The global total loss function is defined as follows:

[0136]

[0137] In the formula, For the adversarial loss of the sample generation network, and These are the weight coefficients of supervised classification loss and self-supervised feature consistency loss in global joint training, respectively.

[0138] Step S404: By minimizing the global total loss function, the network parameters of the sample generation network and the self-supervised classification network are simultaneously iteratively optimized; wherein, the sample generation network is used to continuously generate enhanced EEG spatiotemporal topological signals that match the original EEG spatiotemporal topological signals in terms of spatiotemporal distribution and have physiological rationality, and the self-supervised classification network is used to jointly train using the original EEG spatiotemporal topological signals and the enhanced EEG spatiotemporal topological signals; when the value of the global total loss function tends to stabilize and meets the preset training convergence condition, the training is stopped, and the trained self-supervised classification network is obtained.

[0139] Step S5: Obtain the EEG signal to be identified, and use the same data preprocessing procedure and topology mapping operation as in Step S1 to convert the EEG signal to be identified into a spatiotemporal topological signal of brainwaves to be identified; input the spatiotemporal topological signal of brainwaves to be identified into the trained self-supervised classification network, and output the final emotion recognition result, specifically as follows:

[0140] Step S501: The spatiotemporal topological signal of the EEG to be identified after data preprocessing and topological mapping. The input to the convolutional neural network in the emotion classifier is processed by the convolutional neural network through multiple layers of convolution, normalization, activation, and pooling operations to extract and compress local spatial correlation information from the spatiotemporal topological signal of EEG step by step, resulting in a spatial feature sequence representing the spatial distribution characteristics of different time segments. ;

[0141] Step S502: The spatial feature sequence The input is a bidirectional long short-term memory network. Forward and backward temporal modeling are used to jointly model the dependencies between different time segments of the EEG signal, extracting temporal evolution features from the spatiotemporal topological signal of the EEG to be identified, thus obtaining a spatiotemporal joint feature representation that integrates spatial distribution information and temporal dependency information. .

[0142] The output of the bidirectional long short-term memory network can be expressed as:

[0143]

[0144]

[0145] The final moment or the hidden state after temporal aggregation is used as the spatiotemporal joint feature representation. ;

[0146] Step S503: Represent the spatiotemporal joint features The input is processed by a fully connected layer for feature mapping, and the output, combined with the Softmax function, provides the predicted probability of each emotion category corresponding to the spatiotemporal topological signal of the EEG to be identified.

[0147]

[0148] in, The The component indicates that the sample belongs to the first... The posterior probabilities of each emotion category are calculated; the emotion category with the highest predicted probability is selected as the final emotion recognition result.

[0149] Through the above steps, the original EEG signal is first preprocessed and topologically mapped to obtain the EEG spatiotemporal topological signal. Then, a physiological feature-guided hard span masking strategy is used to continuously mask the EEG spatiotemporal topological signal in a temporal sequence. An enhanced EEG spatiotemporal topological signal that matches the original EEG spatiotemporal topological signal in terms of spatiotemporal distribution is generated through a sample generation network. Further, a self-supervised classification network combining a convolutional neural network and a bidirectional long short-term memory network is used to jointly train the original and enhanced EEG spatiotemporal topological signals. The convolutional neural network is used to extract the spatial features of the EEG signal, and the bidirectional long short-term memory network is used to model the temporal dependence of the EEG signal. Finally, the EEG spatiotemporal topological signal to be identified is input into the trained self-supervised classification network, and the corresponding emotion recognition result is output.

[0150] Example 2

[0151] This embodiment provides an EEG emotion recognition system for implementing the EEG emotion recognition method based on generative adversarial networks and self-supervised learning described in Embodiment 1, comprising:

[0152] Data processing module: used to perform preset data preprocessing and topology mapping operations on the acquired EEG signals, converting one-dimensional time-series EEG signals into raw EEG spatiotemporal topology signals that integrate time dimension information and spatial electrode distribution information;

[0153] Temporal masking module: Used to mask the continuous temporal region of the original EEG spatiotemporal topology signal output by the data processing module using a hard span masking strategy, and generate a masked EEG spatiotemporal topology signal with continuous temporal loss characteristics.

[0154] The sample generation module includes a generator and a discriminator. The generator receives the masked EEG spatiotemporal topology signal output by the temporal masking module, and outputs a reconstructed EEG spatiotemporal topology signal after manifold reconstruction transformation. The discriminator performs authenticity discrimination between the reconstructed EEG spatiotemporal topology signal and the original EEG spatiotemporal topology signal output by the data processing module, and constructs an adversarial loss function based on the discrimination result. Simultaneously, combined with manifold consistency constraints, the network parameters of the generator and discriminator are iteratively optimized through gradient backpropagation until a preset convergence condition is met, and an enhanced EEG spatiotemporal topology signal that matches the spatiotemporal topology features of the original EEG spatiotemporal topology signal and has physiological rationality is output.

[0155] Self-supervised classification module: Used to receive the original EEG spatiotemporal topological signal output by the data processing module and the enhanced EEG spatiotemporal topological signal output by the sample generation module as joint training samples, extract spatial features using a convolutional neural network, and model time series dependencies using a bidirectional long short-term memory network to output emotion classification results;

[0156] Joint training module: used to construct the global total loss function, which includes the adversarial loss of the sample generation module, the supervised classification loss of the self-supervised classification module, and the self-supervised feature consistency loss. The contribution weight of each loss term in the model optimization is balanced by preset weight coefficients. With the goal of minimizing the global total loss function, the network parameters of the sample generation module and the self-supervised classification module are synchronously and iteratively updated using the gradient backpropagation mechanism until the preset convergence condition is met, and the trained self-supervised classification network is obtained.

[0157] Emotion Recognition Module: Used to acquire the EEG signal to be recognized, call the preset process and algorithm of the data processing module to convert the EEG signal to be recognized into the spatiotemporal topological signal of the EEG to be recognized; input the spatiotemporal topological signal of the EEG to be recognized into the trained self-supervised classification module, extract the spatial feature sequence by the convolutional neural network, and model the temporal dependency by the bidirectional long short-term memory network, and output the final emotion recognition result.

[0158] The above are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above content. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. An EEG emotion recognition method based on generative adversarial networks and self-supervised learning, characterized in that, Includes the following steps: Step S1: Acquire EEG signals, perform preset data preprocessing and topology mapping operations on the EEG signals, and convert the one-dimensional time-series EEG signals into original EEG spatiotemporal topology signals that integrate time dimension and spatial electrode distribution information. Step S2: The continuous temporal region of the original EEG spatiotemporal topology signal is masked using a hard span masking strategy to obtain a masked EEG spatiotemporal topology signal with continuous temporal loss characteristics. Step S3: Construct a generative adversarial network (GAN) sample generation network containing a generator and a discriminator. Input the masked EEG spatiotemporal topology signal into the generator, and obtain the reconstructed EEG spatiotemporal topology signal through manifold reconstruction transformation. The discriminator performs a realism judgment between the reconstructed EEG spatiotemporal topology signal and the original EEG spatiotemporal topology signal, and constructs an adversarial loss function based on the judgment result. At the same time, combined with manifold consistency constraints, the network parameters of the generator and discriminator are iteratively optimized through gradient backpropagation until the preset convergence condition is met. The output is an enhanced EEG spatiotemporal topology signal that matches the spatiotemporal topology features of the original EEG spatiotemporal topology signal and has physiological rationality. Step S4: Construct a self-supervised classification network, which uses a classifier backbone combining a convolutional neural network and a bidirectional long short-term memory network; input the original EEG spatiotemporal topology signal and the enhanced EEG spatiotemporal topology signal as joint training samples into the self-supervised classification network; construct a global total loss function, which includes the adversarial loss of the sample generation network, the supervised classification loss of the self-supervised classification network, and the self-supervised feature consistency loss, and balance the contribution of each loss term in the global optimization by a preset weight coefficient; with the goal of minimizing the global total loss function, the network parameters of the sample generation network and the self-supervised classification network are synchronously and iteratively updated using a gradient backpropagation mechanism until the preset convergence condition is met, and the trained self-supervised classification network is obtained; Step S5: Obtain the EEG signal to be identified. Using the same data preprocessing procedure and topology mapping operation as in Step S1, convert the EEG signal to be identified into the spatiotemporal topological signal of the EEG to be identified. Input the spatiotemporal topological signal of the EEG to be identified into the trained self-supervised classification network. The convolutional neural network extracts spatial features, and the bidirectional long short-term memory network models the temporal dependency relationship to output the final emotion recognition result.

2. The EEG emotion recognition method based on generative adversarial networks and self-supervised learning according to claim 1, characterized in that, In step S1, the original EEG signal is subjected to baseline correction, rereference, and band filtering in sequence to obtain a standardized multi-channel EEG time series signal. Based on the spatial distribution relationship of electrode positions in the international 10-20 electrode system, a three-dimensional coordinate projection algorithm is used to map the spatial position information of each channel of the standardized multi-channel EEG time series signal to a preset two-dimensional planar grid. Then, the time series signal of each channel is spatially allocated according to the mapped grid position to obtain an EEG spatiotemporal topology signal with time dimension, spatial row dimension, and spatial column dimension. This EEG spatiotemporal topology signal is a three-dimensional data set with a corresponding three-dimensional length.

3. The EEG emotion recognition method based on generative adversarial networks and self-supervised learning according to claim 1, characterized in that, In step S2, a physiological feature-guided hard span masking strategy is used to mask the original EEG spatiotemporal topological signal, resulting in a masked EEG spatiotemporal topological signal with continuous temporal loss features. The steps of the hard span masking strategy are as follows: A physiological feature-guided random matrix is ​​generated that is completely consistent with the dimension of the original EEG spatiotemporal topology signal. The elements of the physiological feature-guided random matrix range from 0 to 1, and the weight distribution of each frequency band corresponds to the instantaneous energy density of the original EEG spatiotemporal topology signal. That is, the weight distribution accurately reflects the difference in instantaneous energy density of each frequency band of the original EEG spatiotemporal topology signal. The mask span parameter and enhancement factor are sampled from a preset parameter range respectively; the mask span parameter ranges from 0 to a preset maximum span value and is used to define the length of continuous occlusion in the time domain; the enhancement factor ranges from a preset minimum enhancement value to a preset maximum enhancement value and is used to adjust the occlusion intensity of non-span regions. A masking start position is randomly selected in the time domain dimension of the original EEG spatiotemporal topology signal. The start position must satisfy the condition that the continuous masking length from the masking start position does not exceed the total time domain length of the original EEG spatiotemporal topology signal, ensuring that the masking range does not exceed the signal boundary. Starting from the masking start position, all topology elements within the continuous mask span parameter step size are set to 0 to generate a binary span mask, which is used to simulate long-range time domain signal missing scenarios. For non-span regions in the original EEG spatiotemporal topology signal that are not covered by the span mask, the physiological features are used to guide the random matrix and the enhancement factor to generate a discrete mask with a value range of 0 to 1. This discrete mask is used to achieve discontinuous random masking of non-span regions, and the masking process conforms to the distribution of physiological features of the original signal. Each element of the original EEG spatiotemporal topology signal is multiplied element-by-element by the span mask and the discrete mask to obtain the masked EEG spatiotemporal topology signal after joint masking.

4. The EEG emotion recognition method based on generative adversarial networks and self-supervised learning according to claim 1, characterized in that, In step S3, the generator adopts a UNet-like hierarchical spatiotemporal fusion structure and introduces manifold-consistent skip connections. The generator includes an encoding module, a decoding module, and an output module. The encoding module captures the local physiological activity patterns of the input masked EEG spatiotemporal topology signal through multi-scale two-dimensional convolution operations, and simultaneously performs downsampling processing to obtain high-dimensional spatiotemporal features. The decoding module gradually upsamples the high-dimensional spatiotemporal features output by the encoding module through deconvolution operations to restore them to the same spatial resolution as the occluded EEG spatiotemporal topology signal; at the same time, it uses the manifold consistency jump connection to force the geometric alignment of the corresponding features of the encoding module and the decoding module in the feature manifold space; and it synchronously fuses the low-dimensional detail features of the corresponding level of the encoding module with the high-dimensional semantic features of the decoding module to achieve multi-scale feature complementary fusion. The output module is used to perform feature mapping processing on the preliminary generation result output by the decoding module to obtain a candidate reconstruction signal with the same dimension as the original EEG spatiotemporal topology signal. The candidate reconstruction signal is then multiplied element-wise with a preset electrode channel prior mask to meet the physical constraints of the EEG signal. The electrode channel prior mask is a two-dimensional binary matrix that perfectly matches the spatial dimension of the EEG signal. Its elements are only 0 or 1. When the element at the corresponding position of the mask is 1, the corresponding position value of the candidate reconstruction signal is retained. When the element at the corresponding position of the mask is 0, the corresponding position value of the candidate reconstruction signal is forcibly set to 0. Finally, the reconstructed EEG spatiotemporal topology signal is output.

5. The EEG emotion recognition method based on generative adversarial networks and self-supervised learning according to claim 4, characterized in that, In step S3, the discriminator adopts a dynamic co-occurrence spatiotemporal convolutional network structure, which mainly includes a shallow feature extraction layer, a mid-level feature optimization layer, and a deep feature aggregation layer. The shallow feature extraction layer uses multi-scale two-dimensional convolutional kernels and the SELU activation function to capture local spatiotemporal features of input samples containing the original EEG spatiotemporal topological signal and the reconstructed EEG spatiotemporal topological signal output by the generator. By covering spatiotemporal correlation information of different ranges through multi-scale convolutional kernels, the detailed features and basic spatiotemporal patterns of the signal are initially extracted. The mid-layer feature optimization layer uses depthwise separable convolution, which reduces the number of network parameters and computational complexity while maintaining feature expressive power. At the same time, it separates the coupling relationship of spatiotemporal features and strengthens the independent feature modeling of the temporal dynamic changes and spatial distribution differences of EEG signals. The deep feature aggregation layer adopts an Inception-like multi-branch convolutional structure, extracting features in parallel through multiple convolutional branches with different receptive fields, aggregating deep spatiotemporal semantic features at different scales, and finally mapping the aggregated feature vector to a realism score with a value range of [0,1] through a fully connected layer, outputting the realism evaluation result of the input sample.

6. The EEG emotion recognition method based on generative adversarial networks and self-supervised learning according to claim 5, characterized in that, In step S3, the training steps of the sample generation network are as follows: A generator loss function combining manifold consistency constraints is adopted. By minimizing this loss function, the generator is guided to accurately reconstruct the masked EEG spatiotemporal topology signal. This makes the generated reconstructed EEG spatiotemporal topology signal continuously approach the truth score of the original EEG spatiotemporal topology signal in the truth score output by the discriminator. Ultimately, the distribution error between the reconstructed EEG spatiotemporal topology signal and the original EEG spatiotemporal topology signal is lower than the preset error threshold. A Wasserstein loss function with gradient penalty is used as the discriminator loss function. The discriminator loss function includes a real sample discrimination score term, a generated sample discrimination score term, and a gradient penalty constraint term. The gradient penalty constraint term is calculated using intermediate samples obtained by linear interpolation of the original EEG spatiotemporal topological signal and the reconstructed EEG spatiotemporal topological signal output by the generator using random coefficients. By calculating the squared deviation of the gradient L2 norm of this intermediate sample after processing by the discriminator from a fixed value of 1, the gradient of the discriminator is forced to satisfy the Lipschitz continuity constraint condition, ensuring the training stability and realism evaluation accuracy of the discriminator. The network parameters of the generator are fixed, and the network parameters of the discriminator are iteratively updated using the backpropagation algorithm by minimizing the loss function of the discriminator. After the discriminator's ability to distinguish between the original EEG spatiotemporal topological signal and the reconstructed EEG spatiotemporal topological signal reaches a preset level and the fluctuation of the loss function value is lower than a set threshold, the updated network parameters of the discriminator are fixed, and the network parameters of the generator are iteratively updated using the backpropagation algorithm by minimizing the loss function of the generator. The above alternating optimization process is repeated until the sample generation network meets the preset training convergence condition, and finally the trained sample generation network is obtained.

7. The EEG emotion recognition method based on generative adversarial networks and self-supervised learning according to claim 6, characterized in that, In step S4, the self-supervised classification network includes an emotion classifier and a feature extraction module, wherein, The emotion classifier employs a classifier backbone combining a convolutional neural network and a bidirectional long short-term memory network. The convolutional neural network is used to extract spatial features from the input EEG spatiotemporal topology signal, and the bidirectional long short-term memory network is used to model the temporal series dependencies of the spatial features. A fully connected layer is set after the bidirectional long short-term memory network, and combined with the Softmax activation function, the output is the probability of each emotion category corresponding to the input EEG spatiotemporal topology signal. The last fully connected layer and the Softmax activation function in the emotion classifier are removed to obtain the feature extraction module. The feature extraction module is used to receive EEG spatiotemporal topological signals and enhanced EEG spatiotemporal topological signals, and output a fixed-dimensional feature vector representing the spatiotemporal features of the input signal.

8. The EEG emotion recognition method based on generative adversarial networks and self-supervised learning according to claim 7, characterized in that, The training steps for the self-supervised classification network are as follows: Based on the emotion labels of the original EEG spatiotemporal topology signals, a supervised classification loss is calculated to constrain the emotion discrimination accuracy of the emotion classifier. Based on the consistency constraint between the feature vectors obtained by the feature extraction module of the original EEG spatiotemporal topology signal and the enhanced EEG spatiotemporal topology signal, the self-supervised feature consistency loss is calculated to constrain the emotion classifier to learn robust feature representations that are insensitive to temporal missing data and local perturbations. The supervised classification loss and the self-supervised feature consistency loss are fused to obtain the total loss of the self-supervised classification network. The total loss is then used as the optimization objective, and the gradient backpropagation algorithm is employed to update the network parameters of the convolutional neural network, bidirectional long short-term memory network, and fully connected layer in the emotion classifier. A global total loss function is constructed, which includes the adversarial loss of the sample generation network, the supervised classification loss of the self-supervised classification network, and the self-supervised feature consistency loss. The contribution of each loss term in the global optimization is balanced by a preset weight coefficient. By minimizing the global total loss function, the network parameters of the sample generation network and the self-supervised classification network are simultaneously and iteratively optimized until the preset training convergence condition is met, thus obtaining the trained self-supervised classification network.

9. The EEG emotion recognition method based on generative adversarial networks and self-supervised learning according to claim 8, characterized in that, In step S5, the steps for obtaining the emotion recognition result are as follows: The spatiotemporal topological signal of the EEG to be identified, after data preprocessing and topological mapping, is input into the trained self-supervised classification network; The convolutional neural network in the emotion classifier extracts spatial features from the spatiotemporal topological signal of the EEG to be identified. The convolutional neural network extracts and compresses the local spatial correlation information in the spatiotemporal topological signal of the EEG step by step through multi-layer convolution, normalization, activation and pooling operations to obtain a spatial feature sequence that represents the spatial distribution characteristics of different time segments. The spatial feature sequence is input into a bidirectional long short-term memory network. The dependency relationship between time segments before and after the EEG signal is jointly modeled by forward temporal modeling and reverse temporal modeling. The temporal evolution features in the spatiotemporal topological signal of the EEG to be identified are extracted to obtain a spatiotemporal joint feature representation that integrates spatial distribution information and temporal dependency information. The spatiotemporal joint feature representation is input into the fully connected layer for feature mapping, and the predicted probability of each emotion category corresponding to the spatiotemporal topological signal of the EEG to be identified is output by combining the Softmax function. The emotion category with the highest predicted probability is selected as the final emotion recognition result.

10. An EEG emotion recognition system based on generative adversarial networks and self-supervised learning, characterized in that, include: Data processing module: used to perform preset data preprocessing and topology mapping operations on the acquired EEG signals, converting one-dimensional time-series EEG signals into raw EEG spatiotemporal topology signals that integrate time dimension information and spatial electrode distribution information; Temporal masking module: Used to mask the continuous temporal region of the original EEG spatiotemporal topology signal output by the data processing module using a hard span masking strategy, and generate a masked EEG spatiotemporal topology signal with continuous temporal loss characteristics. The sample generation module includes a generator and a discriminator. The generator receives the masked EEG spatiotemporal topology signal output by the temporal masking module, and outputs a reconstructed EEG spatiotemporal topology signal after manifold reconstruction transformation. The discriminator quantifies and evaluates the spatiotemporal similarity of the reconstructed EEG spatiotemporal topology signal with the original EEG spatiotemporal topology signal output by the data processing module, and constructs an adversarial loss function based on the evaluation results. The network parameters of the generator and discriminator are iteratively optimized through gradient backpropagation until a preset convergence condition is met, and an enhanced EEG spatiotemporal topology signal that matches the spatiotemporal topology features of the original EEG spatiotemporal topology signal and has physiological rationality is output. Self-supervised classification module: Used to receive the original EEG spatiotemporal topological signal output by the data processing module and the enhanced EEG spatiotemporal topological signal output by the sample generation module as joint training samples, extract spatial features using a convolutional neural network, and model time series dependencies using a bidirectional long short-term memory network to output emotion classification results; Joint training module: used to construct a global total loss function, which includes the adversarial loss of the sample generation module, the supervised classification loss of the self-supervised classification module, and the self-supervised feature consistency loss. The contribution of each loss term in the global optimization is balanced by preset weight coefficients. With the goal of minimizing the global total loss function, the network parameters of the sample generation module and the self-supervised classification module are synchronously and iteratively updated using a gradient backpropagation mechanism until the preset convergence condition is met, thus obtaining the trained self-supervised classification network. Emotion Recognition Module: Used to acquire the EEG signal to be recognized, call the preset process and algorithm of the data processing module to convert the EEG signal to be recognized into the spatiotemporal topological signal of the EEG to be recognized; input the spatiotemporal topological signal of the EEG to be recognized into the trained self-supervised classification module, extract the spatial feature sequence by the convolutional neural network, and model the temporal dependency by the bidirectional long short-term memory network, and output the final emotion recognition result.

Images