EEG emotion recognition system and method based on multi-scale spatio-temporal graph convolution and contrastive learning

Abstract

The present application relates to the field of deep learning and emotion recognition, and discloses an EEG emotion recognition system and method based on multi-scale spatio-temporal graph convolution and contrast learning, which comprises the following steps: preprocessing the electroencephalogram (EEG) signal; constructing positive and negative sample pairs based on emotion categories and subject identities; extracting the hidden space representation of the EEG signal by using a stacked autoencoder, dividing the EEG signal into multiple functional brain regions to obtain the dimension-reduced brain-derived signals; constructing a non-directional graph structure of the brain-derived signals; constructing a multi-scale spatio-temporal graph convolution network as a feature extraction network, inputting the non-directional graph structure, dynamically learning and updating the adjacency matrix, expressing the features of the brain-derived signals in the non-directional graph structure, and extracting emotion-related features; and based on the multi-scale spatio-temporal graph convolution and contrast learning framework, building and training an EEG emotion recognition model to eliminate individual differences. The present application makes full use of the multi-dimensional information of the EEG signal, and significantly improves the accuracy, robustness and generalization performance of emotion recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of deep learning and emotion recognition, and particularly relates to an EEG emotion recognition system and method based on multi-scale spatio-temporal graph convolution and contrastive learning. BACKGROUND

[0002] Emotions not only deeply affect human cognitive and behavioral activities, but also are important factors of mental health. In recent years, with the development of deep learning technology, emotion recognition algorithms based on physiological signals have made significant progress, especially electroencephalogram (EEG) signals have attracted widespread attention due to their close relationship with emotions.

[0003] Currently, many EEG emotion recognition algorithms mainly use one-dimensional convolutional neural networks (CNN) to process single-dimensional data. These methods are effective to some extent, but they usually ignore the spatial and frequency information in EEG signals. In addition, these algorithms lack the ability to model the complex relationships between different brain regions, limiting the accuracy and comprehensiveness of emotion recognition. In addition, the individual differences commonly existing in physiological signals will lead to poor performance and low accuracy when general discriminant methods are applied to new subjects, and how to eliminate individual differences has become a key problem.

[0004] In view of the above challenges, it is of great significance to develop an EEG emotion recognition system and method based on multi-scale spatio-temporal graph convolution and contrastive learning. SUMMARY

[0005] In order to overcome the shortcomings and deficiencies of the prior art, the present application provides an EEG emotion recognition system and method based on multi-scale spatio-temporal graph convolution and contrastive learning.

[0006] The emotion recognition method of the present application adopts the following technical scheme: an EEG emotion recognition method based on multi-scale spatio-temporal graph convolution and contrastive learning, comprising the following steps:

[0007] S1, collecting the electroencephalogram signals of the subject, and preprocessing the electroencephalogram signals to obtain sample data that truly reflects the emotion processing process of the subject; constructing positive and negative sample pairs based on emotion categories and subject identities for contrastive learning; constructing a leave-one-subject cross-subject dataset;

[0008] S2, using a stacked autoencoder to extract the hidden space representation of the electroencephalogram signals, dividing the electroencephalogram signals into multiple functional brain regions based on medical prior information to obtain dimensionally reduced brain-derived signals;

[0009] S3, constructing an undirected graph structure of the brain-derived signals;

[0010] S4, a multi-scale spatio-temporal graph convolution network is constructed as a feature extraction network, the undirected graph structure is input into the feature extraction network, the adjacency matrix is dynamically learned and updated, the features of the brain source signals in the undirected graph structure are expressed, and emotion-related features are extracted;

[0011] S5, based on the multi-scale spatio-temporal graph convolution and the contrast learning framework, an EEG emotion recognition model is built and trained to eliminate individual differences.

[0012] The emotion recognition system of the present application adopts the following technical scheme: an EEG emotion recognition system based on multi-scale spatio-temporal graph convolution and contrast learning, comprising the following modules:

[0013] A data set construction module is used to collect the EEG signals of the subjects, and the EEG signals are preprocessed to obtain sample data that truly reflects the emotional processing process of the subjects; a cross-subject data set of leave-one-subject-out is constructed;

[0014] A brain source signal acquisition module uses a stacked autoencoder to extract the hidden space representation of the EEG signals, divides the EEG signals into multiple functional brain regions based on medical prior information, and obtains the reduced brain source signals;

[0015] An undirected graph construction module is used to construct the undirected graph structure of the brain source signals;

[0016] A convolution network construction module is used to construct a multi-scale spatio-temporal graph convolution network as a feature extraction network, input the undirected graph structure into the feature extraction network, dynamically learn and update the adjacency matrix, express the features of the brain source signals in the undirected graph structure, and extract emotion-related features;

[0017] A recognition model building and training module is used to build and train an EEG emotion recognition model based on the multi-scale spatio-temporal graph convolution and the contrast learning framework, so as to eliminate individual differences.

[0018] Compared with the prior art, the present application achieves the following technical effects:

[0019] The spatial features in the EEG signals are effectively captured through the graph convolution network, the autoencoder can perform feature compression and dimensionality reduction processing, the contrast learning is added to eliminate the differences between different individuals, and the generalization performance of the model is improved; the multi-dimensional information of the EEG signals is fully utilized, the accuracy, robustness and generalization performance of the emotion recognition are significantly improved, and the present application has a wide application prospect. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 A flowchart of the multi-scale spatio-temporal graph convolution in the embodiments of the present application;

[0021] Figure 2 An algorithm model schematic diagram in the embodiments of the present application. DETAILED DESCRIPTION

[0022] In order to make the purposes, technical solutions and advantages of the present application more obvious, the example embodiments according to the present application will be described in detail below with reference to the drawings. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the example embodiments described herein; the illustrative embodiments and their descriptions are only used to explain the present application, and not as a limitation of the present application.

[0023] In addition, in the present specification and drawings, steps and elements with substantially the same or similar functions are denoted by the same or similar reference signs, and repeated descriptions of these steps and elements will be omitted.

[0024] EMBODIMENT

[0025] The present embodiment proposes an EEG emotion recognition method based on multi-scale spatio-temporal graph convolution and contrastive learning. The implementation process is based on a deep learning framework of stack autoencoder and spatio-temporal graph convolution, which is used to extract high-quality brain source signals and hidden features from EEG data. First, the stack autoencoder is used to reduce the dimension of the original electrode channel, and extract the key brain source information from the redundant signal. Then, in the feature extraction stage, the spatial structure and temporal structure of the EEG data are modeled: in the spatial dimension, an undirected graph is constructed and an attention mechanism-based graph convolution network (GCN) is introduced to capture the spatial correlation between electrodes; in the time dimension, a multi-scale convolution network (MSCN) is used to extract dynamic features at different time resolutions. In the overall framework, contrastive learning is introduced to reduce the influence of individual differences and enhance the generalization ability of emotion classification. In the contrastive learning strategy, the same emotion data pairs of different subjects are selected as positive pairs, and the different emotion data pairs of the same subject are selected as negative pairs. This contrastive objective is used to eliminate individual differences and model emotion-related features more accurately.

[0026] Specifically, as shown in Figure 1 、 Figure 2 The EEG emotion recognition method based on multi-scale spatio-temporal graph convolution and contrastive learning of the present embodiment includes the following steps:

[0027] S1, collecting the EEG signals of the subjects and preprocessing the EEG signals to obtain sample data that truly reflects the emotional processing process of the subjects; constructing positive and negative sample pairs based on emotion categories and subject identities for contrastive learning; constructing a cross-subject dataset with one subject left out.

[0028] The present embodiment is based on the public EEG emotion dataset "SJTU Emotion EEG Dataset". The dataset induces emotional responses through video media and collects EEG signals during the process of subjects watching different emotion-triggering videos. Specifically, the video media is divided into three categories according to the induced emotion categories: positive emotion, neutral emotion and negative emotion, and the collected EEG signals are labeled with the corresponding emotion categories. The dataset contains EEG signals of 15 subjects.

[0029] In the present embodiment, the data division uses a 1-second time window without overlap. Since the sampling rate of the data is 200Hz, the time series length of each data is 200, and each data contains 62 channels corresponding to 62 EEG acquisition electrodes. The experiment is based on the cross-subject Leave-One-Subject-Out (LOSO) method, that is, each time the sample of one subject is used as the test set, and the samples of all other subjects are used as the training set, and each subject's data is used as the test set in turn to construct a cross-subject dataset.

[0030] The preprocessing includes filtering, denoising, etc. For subsequent contrastive learning, the collected EEG signals are divided into positive pairs and negative pairs during preprocessing. The positive pairs here are the same emotions of different subjects, and the negative pairs are different emotions of the same subject. Specifically: for the SEED dataset, there are 15 subjects, and for each emotion (positive, neutral, negative), data pairs of different subjects are selected, such as: the positive emotion of subject 1 and the positive emotion of subject 2, which constitute a positive pair; the positive emotion of subject 1 and the positive emotion of subject 3, which constitute a positive pair, etc. Similarly, the positive emotion of subject 1 and the neutral emotion of subject 1 constitute a negative pair; the neutral emotion of subject 2 and the negative emotion of subject 2 constitute a negative pair. Since the number of such permutations and combinations is large, in order to reduce the computational complexity while ensuring the diversity of training data and the generalization ability of the model, the present embodiment uses a random sampling strategy to randomly select a part of samples from all positive pairs and negative pairs for training. Through this sampling method, the computational load can be effectively reduced and the training efficiency can be improved on the basis of retaining sufficient data diversity.

[0031] S2, using a stacked autoencoder to extract the hidden space representation of the EEG signal, dividing the EEG signal into multiple functional brain regions based on medical prior information, and obtaining the dimensionality-reduced brain source signal.

[0032] In this step, to extract the hidden space feature representation of the electroencephalogram signal, a stacked autoencoder (SAE) is used, and the electroencephalogram signal is divided into 12 functional brain regions based on medical prior information to generate the reduced dimension brain source signal features. Specifically, the division of the 12 brain regions is based on physiological and medical prior information, by dividing the brain regions into left and right hemispheres, and combining horizontal planes (anterior, lateral, posterior) and vertical planes (inferior, superior) to form the electrode recording position based on the international 10-20 system. In addition, the influence of different numbers of source signal channels (such as 6 channels and 7 channels) is also studied, but the experimental results show that the brain region division effect is best using 12 channels, which is one of the important reasons why 12 brain regions are selected as the basis for functional division in this embodiment.

[0033] The stacked autoencoder SAE is used to reduce the dimension of the multi-channel brain source signal to reconstruct the brain source signal and extract key features. SAE maps high-dimensional input data to low-dimensional hidden space while preserving the core information of the brain source signal through unsupervised learning. Specifically, the encoder part of the stacked autoencoder compresses the high-dimensional brain source signal into low-dimensional hidden features, and the decoder part reconstructs the brain source signal. By minimizing the error between the input signal and the reconstructed brain source signal, it is ensured that the low-dimensional hidden features can truly and comprehensively represent the brain source signal. These features provide a high-quality basis for subsequent feature extraction and analysis.

[0034] Before the stacked autoencoder SAE reduces the dimension of the multi-channel brain source signal, the SAE is first trained. In the training process of SAE, a layer-by-layer unsupervised pre-training strategy is used to learn deep feature representation. Taking a network with n→m→k structure as an example, the training process is divided into two steps: first, train an n→m→n autoencoder to learn the feature mapping from n to m; then, train an m→k→m autoencoder to further extract the feature representation from m to k. Finally, these features are stacked step by step to build a complete deep network structure n→m→k. This layer-by-layer optimization process is similar to building a building, which gradually builds a complete model based on a stable foundation.

[0035] The training process of the stacked autoencoder SAE is as follows:

[0036] First layer autoencoder training: first, the original brain source signal x(k) is input into the first sparse autoencoder, and the network learns the first-order feature representation h (1) (k) of the input data. This step is equivalent to learning the mapping from the original brain source signal to the low-dimensional feature space.

[0037] Second layer autoencoder training: Next, the first-order feature representation h (1) (k) obtained above is passed into a second sparse autoencoder as new input, which is trained to learn a second-order feature representation h (2) (k). This step further extracts higher-level features of the data, gradually reducing the dimensionality of the data.

[0038] Classifier training: After obtaining the second-order feature representation h (2) (k), a Softmax classifier can be trained to map the high-level features to specific emotion classification labels, taking these features as input. In this way, the network can eventually perform emotion recognition based on the reduced features. The entire process ensures that each layer effectively extracts different levels of features in the brain source signals through layer-by-layer training.

[0039] To ensure the performance of the model, each sparse autoencoder is also optimized by reconstruction loss. Specifically, the reconstruction loss measures the difference between the sparse autoencoder output and the input, defined as:

[0040]

[0041] where x is the brain source signal reconstructed by the autoencoder. By minimizing the reconstruction loss, it is ensured that the reduced feature representation still retains the key information of the original brain source signal. Finally, this layer-by-layer training forms a stacked autoencoder network containing two hidden layers (reducing 62 channels to 12 channels respectively).

[0042] S3, construct an undirected graph structure for brain source signals.

[0043] Brain source signals are naturally suitable for corresponding to an undirected graph structure, where each brain source channel can be regarded as a node in the undirected graph, and the relationship between channels is mapped to the edge features of the undirected graph. Therefore, the brain source signals obtained by dimensionality reduction from the stacked autoencoder are converted into data representation of the undirected graph structure. Specifically, the edge features of the undirected graph are defined by constructing an adjacency matrix, and the weight of the edge is measured by the phase-locked value between nodes, that is, by calculating the instantaneous phase relationship between two signals to characterize the dependence and correlation between channels.

[0044] S31, for 12-dimensional EEG channel brain source signals, convert them into data representation of the undirected graph structure.

[0045] Specifically, first, standardize the brain source signal of each channel to reduce its range to [-1, 1]. The standardization formula is:

[0046]

[0047] where x is the original brain source signal, μ is the mean of the brain source signal, and σ is the standard deviation of the brain source signal. The normalized signal values will serve as the features of each node in the undirected graph. Each channel corresponds to a node in the undirected graph, with a total of 12 nodes. Specifically, these normalized brain source signals serve as input node features in the graph convolution operation and will be used to construct the initial representation of the undirected graph. In subsequent graph convolution processes, these node features are optimized through propagation and updating mechanisms in the graph convolution network. The graph convolution network updates the feature values of each node by combining the adjacency relationships between nodes (i.e., the edges of the undirected graph), thereby gradually extracting more complex and high-level spatial features.

[0048] S32, in order to define the connection relationship between channels, the phase locking value (PLV) is used to measure the phase synchronization between the brain source signals of two EEG channels.

[0049] In this embodiment, the calculation formula of the phase locking value PLV is:

[0050]

[0051] where φ i (t) is the instantaneous phase of the i-th channel at time t, φ j (t) is the instantaneous phase of the j-th channel at time t, and T is the length of the time series. The closer the phase locking value PLV is to 1, the stronger the synchronization of the two channels in phase, and the closer the connection relationship. When constructing the undirected graph structure, the PLV value will be used as the weight of the edge to represent the mutual association between channels.

[0052] In the undirected graph modeling of EEG multi-channel signals, PLV is used as the weight of the edge in the undirected graph. This method can effectively capture the phase synchronization between brain source signal channels and reveal the patterns of coordinated work of different brain regions. EEG is a multi-channel time series signal, and each channel records the electrical activity from different brain regions. Due to the complex functional interaction between different brain regions, studying the phase synchronization between channels helps to describe the functional connectivity of brain regions. Specifically, the value of PLV is between 0 and 1, with 1 indicating that the phases of the two signals are completely synchronized, and 0 indicating that the phases of the signals are completely unrelated. Compared with directly using signal amplitude, PLV focuses on the relative phase change between signals, which makes it more stable in describing the relationship between brain regions under different subjects and experimental conditions. In addition, PLV is robust to noise and is not affected by amplitude changes, making it suitable for processing frequency band information in EEG signals. Therefore, using PLV as an edge feature can represent the phase synchronization between each pair of EEG channels as a weight, and construct an undirected graph structure reflecting the functional connectivity of the brain.

[0053] S33, a preset threshold τ is introduced to sparsify the phase locking value matrix A to construct a sparse adjacency matrix of the phase locking value.

[0054] To construct a sparse adjacency matrix and avoid excessive noise or weakly related connections in the undirected graph structure from interfering with the modeling effect, the embodiment introduces a thresholding strategy. Specifically, after calculating the PLV value, the PLV value less than the set threshold is directly set to 0, and only the strongly related connection is retained. This sparsification process not only reduces the computational complexity, but also highlights the more significant functional connections between brain regions. The specific PLV matrix is A ∈ R N×N , where N represents the number of EEG channels, and the element A ij in the matrix represents the PLV value between the i th and j th channels. To construct a sparse adjacency matrix, a preset threshold τ is introduced to sparsify the PLV matrix A, defined as follows:

[0055]

[0056] , where is the element of the sparse adjacency matrix. Through this thresholding operation, weakly related edges with PLV less than τ are removed, and only strongly related connections are retained.

[0057] Correspondingly, the sparse adjacency matrix can be represented as:

[0058] A sas = A ⊙ M,

[0059] where M is a mask matrix defined as:

[0060]

[0061] By constructing the undirected graph structure of the EEG through the adjacency sparse matrix of the PLV, the static functional connection characteristics between brain regions can be captured, and the dynamic connection changes under different emotional states can be explored. This undirected graph representation provides rich and stable structured information for subsequent emotion classification tasks based on spatiotemporal features.

[0062] S4, a multi-scale spatiotemporal graph convolution network is constructed as a feature extraction network, the undirected graph structure is input into the feature extraction network, the adjacency matrix is dynamically learned and updated, the features of the brain source signals in the undirected graph structure are expressed, and emotion-related features are extracted.

[0063] The multi-scale spatio-temporal graph convolutional network includes two main modules: a spatial feature extraction module and a temporal feature extraction module. The spatial feature extraction module uses multiple different scale convolution kernels to extract information from the nodes in the undirected graph, and unifies and combines the extracted features to form a complete spatial representation. The temporal feature extraction module uses a self-attention mechanism to dynamically focus on important time points, thereby extracting key features in the time series. Through the cooperative work of the two modules, a comprehensive feature representation is formed to support subsequent emotion recognition tasks.

[0064] For the spatial feature extraction module, the embodiment proposes a global graph convolutional attention module (GGCAM), which integrates a multi-head self-attention mechanism and a graph convolutional network (GCN), aiming to effectively aggregate local and global node feature information. The GGCAM first calculates the relationship matrix between nodes through the multi-head self-attention mechanism which contains 4 self-attention heads, each of which independently captures the node dependency relationship in different channel feature dimensions. The self-attention mechanism not only enhances the information flow between nodes, but also avoids the influence of a single feature dimension on global relationships. Then, a learnable weight vector is used to integrate the multi-head output into a global attention relationship matrix A global = wG. Since a dense attention matrix can easily cause over-smoothing problems, the embodiment preserves the top 20% connections in the adjacency matrix to ensure the sparsity of the matrix, thereby maintaining the sparsity of the undirected graph structure while aggregating global information.

[0065] The global attention matrix A global is obtained, and the embodiment further calculates its corresponding Laplacian matrix and introduces it into the graph convolutional network (GCN) for feature propagation. Through graph convolution operations, the global node features are aggregated layer by layer, and the specific formula is:

[0066]

[0067] where O (l) represents the input node features of the lth layer, O (l+1) is the output node features of the l+1th layer, W (l) is a learnable weight matrix, and σ is a nonlinear activation function. The initial node features O (0) are the original node features X meso . Through the introduction of the GGCAM module, the embodiment realizes the effective fusion of local information and global information, and enhances the emotion recognition ability of the model based on global feature learning.

[0068] For the time feature extraction module, the embodiment adopts a multi-scale convolution time feature extraction mechanism, aiming to capture key signal features at different time scales. In order to adapt to time signals of different frequency components, the embodiment designs three different convolution kernel scales, corresponding to 1 times, 0.5 times and 0.25 times of the signal sampling rate. This multi-scale design can ensure that features of different time lengths are extracted, so as to balance the short-time and long-time dependencies, effectively improving the richness and robustness of feature representation.

[0069] Specifically, let the input signal be X time ∈R N×C×T , where N is the number of samples, C is the number of channels, and T is the time step. The embodiment designs three convolution operations corresponding to different convolution kernel lengths K1, K2 and K3, whose sizes are 1 times, 0.5 times and 0.25 times of the original EEG signal sampling rate. The convolution operation formula is:

[0070] X conv1 = Conv1D(X time , W1, K1, stride = 1)

[0071] X conv2 = Conv1D(X time , W2, K2, stride = 1)

[0072] X conv3 = Conv1D(X time , W3, K3, stride = 1)

[0073] Where W1, W2, W3 are the learnable parameters of the corresponding convolution kernel, and K1, K2, K3 are the sizes of the convolution kernel. Through these different scale convolution operations, it is ensured that the signal features at different time windows can be effectively captured. Then, the embodiment concatenates the output features of multi-scale convolution to form a comprehensive time feature representation:

[0074] X multi = concat(X conv1 , X conv2 , X conv3 )

[0075] The concatenated time feature X multi contains feature representations at different time lengths, thereby improving the model's ability to perceive signals at different time scales. Through this multi-scale convolution time feature extraction mechanism, the embodiment can effectively extract multi-level time sequence features on complex time series data, providing a strong feature basis for subsequent classification and recognition tasks.

[0076] S5, based on multi-scale spatio-temporal graph convolution and contrastive learning framework, build and train EEG emotion recognition model to eliminate individual differences.

[0077] In view of the problems of low spatial resolution of EEG signal and obvious individual differences, the embodiment builds a contrastive learning module. In this framework, the feature extractor adopts a share weights strategy. Whether the input is a positive sample pair (the same emotion data of different subjects) or a negative sample pair (different emotion data of the same subject), the input EEG signal passes through the same feature extractor. This weight sharing mechanism ensures that the features extracted between different subjects are consistent, thereby enhancing the alignment effect in contrastive learning. The shared feature extractor consists of two modules: a temporal feature extraction module and a spatial feature extraction module based on graph convolution. The temporal feature extraction module captures temporal information at different temporal resolutions through multi-scale convolution, and the spatial feature extraction module extracts spatial features of EEG signals through a graph convolution network. The feature extractor combining the two modules can extract local and global temporal and spatial features from different scales, and then comprehensively represent the emotional features in the EEG signal.

[0078] The training process of the EEG emotion recognition model includes the following steps:

[0079] S51, first, the original EEG data is divided according to the subjects and emotion categories. For each emotion category, select the same emotion data of different subjects as a positive sample pair, and select the different emotion data of the same subject as a negative sample pair.

[0080] S52, then, the positive and negative sample pairs are input into the feature extraction network with shared weights to extract high-dimensional feature representations. In order to ensure the consistency of feature extraction, all samples are processed through the same network structure and parameters. The similarity between the extracted high-dimensional feature vectors is measured by cosine similarity (Cosine Similarity), and the calculation formula is as follows:

[0081]

[0082] where z i and z j are the two vectors in the sample pair.

[0083] S53, then, use the contrastive loss function to maximize the similarity between the positive sample pairs and minimize the similarity between the negative sample pairs. The core goal of the loss function is to adjust the feature space so that samples of the same emotion are clustered together, while samples of different emotions are far apart from each other. The specific contrastive loss function is:

[0084]

[0085] where τ is a temperature parameter, is an indicator function to ensure k≠i. This loss function optimizes the feature space by maximizing the similarity of positive pairs and minimizing the similarity of negative pairs, to achieve the generalization of emotion classification.

[0086] Specifically, for each input pair (positive pair or negative pair), the two input samples are respectively passed through the same feature extractor to generate feature representations and Due to the weight sharing of the feature extractor, the extracted features have good comparability, which is conducive to optimizing the distance between similar samples in contrast learning. The embodiment introduces a projection head to further optimize the comparability of the features. The projection head maps the features to a low-dimensional space, which facilitates fine optimization of the contrast loss on the feature distance between different samples. The projection head usually adopts a multi-layer perceptron (MLP) structure, and the specific form is:

[0087] h(z) = W2σ(W1z),

[0088] where W1and W2are learnable weight matrices, and σ is a nonlinear activation function. The role of the projection head is to project the original feature z to a subspace that is more suitable for calculating the contrast loss, so as to improve the effect of contrast learning. In contrast learning, the representations after the feature extractor and the projection head are denoted as and These projected features will be used to calculate the contrast loss. The specific loss function form is NT-Xent (Normalized Temperature-scaled CrossEntropy Loss), which is a commonly used loss function in contrast learning, used to maximize the similarity between positive pairs and minimize the similarity between negative pairs.

[0089] The formula is as follows:

[0090]

[0091] where and are the representations after the feature extractor and the projection head. represents the cosine similarity between two features. τ is a temperature parameter used to control the smoothness of the distribution. is an indicator function to ensure that the denominator does not contain the same sample. N represents the batch size, i.e. the number of sample pairs in each batch. Through this contrast loss, the embodiment can significantly enhance the alignment effect of the same emotion between different subjects, eliminate the interference brought by individual differences, and thus improve the generalization ability of emotion classification.

[0092] Based on the same inventive concept, the embodiment also provides an EEG emotion recognition system based on multi-scale spatio-temporal graph convolution and contrastive learning, comprising the following modules:

[0093] A data set construction module is configured to collect electroencephalogram signals of a subject, and pre-process the electroencephalogram signals to obtain sample data that truly reflects the emotional processing process of the subject; construct positive and negative sample pairs based on emotion categories and subject identities for contrastive learning; and construct a cross-subject data set of a leave-one-subject-out.

[0094] A brain source signal acquisition module is configured to extract hidden space representation of the electroencephalogram signals by using a stacked autoencoder, divide the electroencephalogram signals into multiple functional brain regions based on medical prior information, and obtain the brain source signals after dimension reduction.

[0095] An undirected graph construction module is configured to construct an undirected graph structure of the brain source signals.

[0096] A convolution network construction module is configured to construct a multi-scale spatio-temporal graph convolution network as a feature extraction network, input the undirected graph structure into the feature extraction network, dynamically learn and update an adjacency matrix, express the features of the brain source signals in the undirected graph structure, and extract emotion-related features.

[0097] A recognition model building and training module is configured to build and train an EEG emotion recognition model based on a multi-scale spatio-temporal graph convolution and a contrastive learning framework, so as to eliminate individual differences.

[0098] It should be noted that each of the above modules is used to implement the corresponding steps of the EEG emotion recognition method in the embodiment. For example, the brain source signal acquisition module uses a stacked autoencoder SAE to reduce the dimension of the multi-channel brain source signals, so as to reconstruct the brain source signals and extract key features; the stacked autoencoder maps high-dimensional input data to a low-dimensional hidden space while retaining the core information of the brain source signals through unsupervised learning.

[0099] For another example, the undirected graph construction module regards each brain source signal channel as a node in the undirected graph, and maps the relationship between the signals of the channels to the edge features of the undirected graph, and converts the brain source signals obtained after dimension reduction by the stacked autoencoder into data representation of the undirected graph structure.

[0100] Other more detailed implementation processes can be referred to the descriptions in steps S1-S5.

[0101] The above embodiment is a preferred implementation of the present application, but the implementation of the present application is not limited by the above embodiment, and any change, modification, substitution, combination, simplification made without departing from the spirit and principle of the present application should be an equivalent replacement method, which is included in the protection scope of the present application.

Claims

1. An EEG emotion recognition method based on multi-scale spatio-temporal graph convolution and contrastive learning, characterized in that, The method comprises the following steps: S1, collecting the brain electrical signals of a subject, and preprocessing the brain electrical signals to obtain sample data that truly reflects the emotional processing process of the subject; based on the emotional categories and the identities of the subjects, a positive and negative sample pair is constructed for contrastive learning; A cross-subject dataset of a leave-one-subject-out is constructed; S2, a stack autoencoder is used to extract the hidden space representation of the brain electrical signals, and the brain electrical signals are divided into multiple functional brain regions based on medical prior information to obtain the dimension-reduced brain source signals; Step S2 uses a stack autoencoder (SAE) to reduce the dimension of the multi-channel brain source signals to reconstruct the brain source signals and extract key features; The stack autoencoder maps high-dimensional input data to a low-dimensional hidden space while retaining the core information of the brain source signals through unsupervised learning; Before step S2 reduces the dimension of the multi-channel brain source signals using the stack autoencoder (SAE), the SAE is trained, including the steps of: The first layer auto-encoder training, the original brain source signal x(k) into the first sparse auto-encoder, training network learns the first order feature representation h of input data (1) (k); Second layer autoencoder training, the trained first order feature representation h (1) (k) as new input, pass into the second sparse autoencoder, train it to learn the second order feature representation h (2) (k); Classifier training, to represent h (2) (k) As input, train a Softmax classifier to map high-level features to specific emotion classification labels; Each sparse autoencoder is also optimized by a reconstruction loss, which is used to measure the difference between the output of the sparse autoencoder and the input, and is defined as: wherein, is the brain-derived signal reconstructed by the autoencoder; by minimizing the reconstruction loss, it is ensured that the reduced-dimensional feature representation still retains the key information of the original brain-derived signal; Finally, a stack autoencoder network containing two hidden layers is formed through layer-by-layer training; S3, a non-directional graph structure of the brain source signals is constructed; S4, a multi-scale spatio-temporal graph convolution network is constructed as a feature extraction network, the non-directional graph structure is input into the feature extraction network, and the adjacency matrix is dynamically learned and updated to express the features of the brain source signals in the non-directional graph structure and extract emotion-related features; S5, based on the multi-scale spatio-temporal graph convolution and the contrastive learning framework, an EEG emotion recognition model is built and trained to eliminate individual differences.

2. The EEG emotion recognition method of claim 1, wherein, Step S3 regards each brain source signal channel as a node in the non-directional graph, and the relationship between the channels is mapped as the edge feature of the non-directional graph. The brain source signals obtained by dimension reduction from the stack autoencoder are converted into data representation of the non-directional graph structure. 3.The EEG emotion recognition method of claim 2, wherein, Step S3 defines the edge feature of the non-directional graph by constructing an adjacency matrix, and the weight of the edge is measured by the phase locking value (PLV) between the nodes, specifically including: S31, the multi-dimensional EEG channel brain source signals are converted into data representation of the non-directional graph structure; First, the brain source signals of each channel are standardized, and the standardized brain source signals are used as the input node features in the graph convolution operation to construct the initial representation of the non-directional graph. In the graph convolution process, the node features are optimized through the propagation and update mechanism in the graph convolution network. The graph convolution network updates the feature values of each node by combining the edge of the non-directional graph, thereby gradually extracting more complex and high-level spatial features; S32, the phase locking value (PLV) is used to measure the phase synchronization between two EEG channel brain source signals. When constructing the non-directional graph structure, the phase locking value is used as the weight of the edge to represent the mutual association between the channels; S33, a preset threshold τ is introduced to perform sparse processing on the phase locking value matrix A to construct a sparse adjacency matrix of the phase locking value. The non-directional graph structure of the EEG is constructed through the sparse adjacency matrix of the phase locking value to capture the static functional connection characteristics between brain regions and mine the dynamic connection changes under different emotional states. 4.The EEG emotion recognition method of claim 1, wherein, The multi-scale spatio-temporal graph convolution network constructed in step S4 includes a spatial feature extraction module and a temporal feature extraction module; The spatial feature extraction module uses multiple convolution kernels of different scales to extract information from the nodes in the undirected graph, and combines the extracted features to form a complete spatial representation; The temporal feature extraction module uses a self-attention mechanism to dynamically focus on important time points, thereby extracting key features in the time series.

5. The EEG emotion recognition method of claim 1, wherein, The training process of the EEG emotion recognition model in step S5 includes the following steps: First, the original EEG data is divided according to the subjects and emotion categories; for each emotion category, the same emotion data of different subjects is selected as a positive sample pair, and the different emotion data of the same subject is selected as a negative sample pair; Then, the positive and negative sample pairs are input into the feature extraction network with shared weights to extract high-dimensional feature representations; the similarity between the extracted high-dimensional feature vectors is measured by cosine similarity; Next, using a contrastive loss function, the similarity between positive sample pairs is maximized and the similarity between negative sample pairs is minimized; the feature space is adjusted by the loss function so that samples of the same emotion are clustered together and samples of different emotions are far apart from each other.

6. An EEG emotion recognition system based on multi-scale spatio-temporal graph convolution and contrastive learning, characterized in that, It includes the following modules: The data set construction module is used to collect the EEG signals of the subjects and preprocess the EEG signals to obtain sample data that truly reflects the emotional processing process of the subjects; based on the emotion categories and subject identities, positive and negative sample pairs are constructed for contrastive learning; A leave-one-subject cross-subject data set is constructed; The brain source signal acquisition module uses a stacked autoencoder to extract the hidden space representation of the EEG signal, divides the EEG signal into multiple functional brain regions based on medical prior information, and obtains the reduced brain source signal; The undirected graph construction module is used to construct the undirected graph structure of the brain source signal; The convolution network construction module is used to construct a multi-scale spatio-temporal graph convolution network as a feature extraction network, input the undirected graph structure into the feature extraction network, dynamically learn and update the adjacency matrix, and express the features of the brain source signal in the undirected graph structure to extract emotion-related features; The recognition model building and training module is used to build and train an EEG emotion recognition model based on the multi-scale spatio-temporal graph convolution and contrastive learning framework to eliminate individual differences; The brain source signal acquisition module uses a stacked autoencoder SAE to reduce the dimensionality of the multi-channel brain source signal to reconstruct the brain source signal and extract key features; the stacked autoencoder maps high-dimensional input data to a low-dimensional hidden space while preserving the core information of the brain source signal through unsupervised learning; Before the stacked autoencoder SAE reduces the dimensionality of the multi-channel brain source signal, the SAE is first trained, including the following steps: The first layer auto-encoder training, the original brain source signal x(k) is transmitted into the first sparse auto-encoder, the network learns the first order feature representation h of the input data (1) (k); Second layer autoencoder training, the trained first order feature representation h (1) (k) as new input, pass into the second sparse autoencoder, train it to learn the second order feature representation h (2) (k); Classifier training, second-order feature representation h (2) (k) As input, train a Softmax classifier to map high-level features to specific emotion class labels; Each sparse autoencoder is also optimized by reconstruction loss, which is used to measure the difference between the output of the sparse autoencoder and the input, and is defined as: wherein, is the brain-derived signal reconstructed by the autoencoder; by minimizing the reconstruction loss, it is ensured that the reduced-dimensional feature representation still retains the key information of the original brain-derived signal; Finally, a stacked autoencoder network containing two hidden layers is formed through layer-by-layer training.

7. The EEG emotion recognition system of claim 6, wherein, The undirected graph construction module takes each brain source signal channel as a node in an undirected graph, and the relationship between the channels is mapped as an edge feature of the undirected graph. The brain source signal obtained by dimension reduction from the stack autoencoder is converted into a data representation of the undirected graph structure.

Images