Multi-task self-supervised graph neural network emotion recognition method and system based on time-space-frequency fusion

Abstract

The invention belongs to the technical field of artificial intelligence and brain-computer interfaces, and discloses a multi-task self-supervision graph neural network emotion recognition method and system based on time-space-frequency fusion, and the method comprises the steps: firstly carrying out the graph modeling of an EEG signal from a time domain, a space domain and a frequency domain, then designing three types of self-supervision tasks, namely, a time sequence puzzle, a space puzzle and a frequency puzzle, the EEG signals are partitioned and rearranged according to time, space and frequency, and the model is guided to autonomously learn key structural features and potential modes in the EEG signals by predicting the original sequence. And meanwhile, a comparative learning task is introduced to realize semantic consistency constraint under different feature perspectives, so that the discrimination and robustness of feature expression are further improved. And the network adopts a dynamic weight distribution mechanism to carry out joint optimization on multi-task loss. The method can effectively improve the capturing capability of the spatial-temporal dynamic characteristics of the electroencephalogram signals, enhances the generalization of the model and the robustness of noise labels, and shows high accuracy in emotion recognition.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and brain-computer interface technology, specifically relating to a multi-task self-supervised graph neural network emotion recognition method and system based on time-space-frequency fusion. Background Technology

[0002] Brain-computer interface (BCI) technology originated in the 1970s and 80s. Early systems were primarily used in the medical field to provide assistance to seriously ill patients with neurological or muscular dysfunctions, such as brain-controlled wheelchairs, text input tools, prostheses, and robotic arms. With further research, the applications of BCIs have expanded: in medical rehabilitation, it enables more precise intervention for neurological diseases; in education, student attention monitoring headbands with real-time feedback capabilities have been commercialized, and brainwave modulation technology can also improve ADHD in children; the healthcare industry has launched sleep monitoring and intervention devices based on brainwave data; in home entertainment, brainwave interaction systems provide new control methods for virtual reality (VR) and augmented reality (AR), enhancing the immersive experience; in special operations scenarios, brainwave devices can monitor the emotions and fatigue status of personnel in real time, providing timely warnings of high-risk operations; in the military field, countries like the United States are exploring the use of this technology to improve individual soldier combat capabilities. Due to cost and portability considerations, most current mainstream BCI systems adopt non-invasive brainwave acquisition methods.

[0003] In recent years, with the rapid development of brain science and artificial intelligence technologies, the study of electroencephalography (EEG) has gradually become an important foundation for human-computer interaction, mental health monitoring, and intelligent control systems. EEG signals can directly reflect the electrophysiological characteristics of neuronal activity in the brain, offering advantages such as high temporal resolution and non-invasiveness. However, due to the inherent noise, significant individual differences, and complex high-dimensional nonlinear characteristics of EEG signals, traditional feature extraction and classification methods often struggle to achieve high accuracy and robustness in EEG decoding.

[0004] Existing research on EEG emotion recognition largely relies on shallow machine learning or single-domain feature analysis methods. For example, it extracts statistical features, power spectral density, or wavelet energy features only in the time or frequency domains, and then combines these with classifiers such as Support Vector Machines (SVM) and Random Forests (RF) for recognition. While these methods are simple in structure and have low computational cost, they are susceptible to noise interference and feature loss when dealing with complex emotional states and cross-subject scenarios, exhibiting poor generalization ability. Furthermore, shallow feature engineering depends on human experience and struggles to fully exploit the high-dimensional semantic features and potential spatiotemporal dependencies of EEG signals.

[0005] In recent years, deep learning has demonstrated significant advantages in EEG signal analysis by directly extracting emotional features from EEG signals using convolutional neural networks (CNNs), recurrent neural networks (RNNs / LSTMs), or fully connected networks. With the development of graph convolutional neural networks (GNNs), emotion recognition research based on EEG topology has gradually emerged. These networks can model the spatial connections between EEG channels through graph structures, thereby capturing functional connectivity patterns between brain regions. However, existing GNN models mostly only model in the spatial domain, neglecting the dynamic evolution of EEG signals in the time domain and the energy distribution characteristics in the frequency domain, resulting in incomplete feature representation. Furthermore, due to insufficient labeled samples, supervised learning is prone to overfitting in EEG emotion recognition, limiting the model's generalization ability.

[0006] To address the aforementioned issues, this invention proposes a multi-task self-supervised graph neural network emotion recognition method based on time-space-frequency fusion. Summary of the Invention

[0007] To address the problems of insufficient feature representation, inadequate labeled samples, and poor generalization ability in existing EEG decoding technologies, this invention provides a multi-task self-supervised graph neural network emotion recognition method and system based on time-space-frequency fusion. By jointly optimizing temporal, spatial, and frequency features through multi-task learning, it enhances the ability to capture the spatiotemporal dynamic characteristics of EEG signals and improves the model's generalization and robustness to noise labels.

[0008] To achieve the above objectives, the present invention provides the following solution: A multi-task self-supervised graph neural network emotion recognition method based on spatiotemporal-frequency fusion, the method comprising: S1: Acquire raw EEG signals and preprocess them to obtain standardized time-series EEG data; S2: Based on the obtained standardized time-series EEG data, the graph modeling module is used to generate standard graph structure data by multi-domain graph generation. S3: Based on standard graph structure data, construct a multi-task self-supervised learning task and generate corresponding self-supervised samples; S4: Based on multi-task self-supervised learning tasks and self-supervised samples, a graph neural network and a classification head are constructed for feature extraction and classification. At the same time, a temporal attention and dynamic weight allocation mechanism are combined to focus on key segments, so as to achieve effective feature extraction and deep fusion. S5: Based on the fused features, a dynamic weight allocation mechanism is used to jointly optimize the multi-task loss, and finally output the emotion recognition result.

[0009] Preferably, multi-domain graph generation includes designing graph node and edge relationships from the time, space, and frequency domains of standardized temporal EEG data, specifically: In the time domain, the EEG signal is divided into several continuous segments along the time axis. Each time segment is used as a graph node, and the node features are the average energy or differential entropy features of each channel in different frequency bands within the segment. The initial adjacency matrix is ​​constructed based on the cosine similarity between the features of the time segments and dynamically updated by combining the time decay factor to generate a time-varying adjacency matrix to characterize the temporal evolution characteristics of emotions. In the spatial domain, EEG channels are defined as graph nodes, and node features are the multi-band energy features of the corresponding channels. The adjacency matrix is ​​jointly constructed by the three-dimensional spatial distance matrix between electrodes and the phase-locked value functional connection matrix. The two types of weights are adaptively fused through the attention mechanism to generate a time-varying adjacency matrix to reflect the dynamic topological relationship between brain regions. In the frequency domain, the signals of each channel are divided into graph nodes according to several frequency bands. The node features are the energy values ​​of the corresponding frequency bands. The adjacency matrix is ​​constructed based on the energy correlation between frequency bands, thereby realizing the structured modeling and dynamic correlation learning of frequency information. In the time domain, the features of each node are represented as follows: , ; in Indicates time segment upper frequency band The average energy; Represents the time segment of the c-th channel. upper frequency band The power or differential entropy, C Indicates the total number of channels; The initial adjacency matrix is ​​defined as: ; in and They represent the first The first time segment and the first The node features of each time segment are used to calculate the correlation between nodes through cosine similarity, and then the initial neighbor matrix is ​​obtained. A time distance factor is introduced to dynamically update the neighbor matrix, achieving a balanced modeling of long-term dependencies and short-term changes; specifically, time segments are defined. and The time distance is ,in For time segments The corresponding time point, For time segments At the corresponding time point, the elements of the dynamic adjacency matrix are updated as follows: ; in >0 represents the time constant; In the spatial domain, the adjacency matrix adaptively fuses the distance matrix and PLV values ​​through an attention mechanism to achieve adaptive fusion of topological and functional information, generating a spatial domain adjacency matrix that updates over time. ; ; ; in Represents a node ij The distance matrix between them express Represents a node ij The PLV functional connection matrix between them The adjacency matrix of the spatial domain graph represents the first... The final connection weight of a node; and Indicates the first and the Three-dimensional coordinate vector of each EEG channel , Represents the total number of sampling points in a time segment. and Indicates the first and the Each channel at time The instantaneous phase, The activation function, usually sigmoid, maps the weighted result to the interval [0, 1]. This is a distance normalization constant used to control the decay rate. w 1 and w 2 The learnable fusion weights control the contribution of geometric distance and functional connectivity to the final adjacency, respectively. In the frequency domain, the node characteristic is the average energy of the corresponding frequency band across all channels. C represents the total number of channels. The correlation between different frequency bands is calculated using the Pearson correlation coefficient or mutual information. ; Introducing spectral energy difference weights: ; in It is the dynamic connection strength considering the energy difference. E b Indicates the first b Energy of each frequency band This represents the energy of the b'-th frequency band. By controlling the impact of energy differences in different frequency bands on connection strength, structured modeling and dynamic correlation learning of frequency information can be achieved.

[0010] Preferably, the constructed multi-task self-supervised tasks include spatial mosaicking, frequency mosaicking, temporal mosaicking, and contrastive learning tasks, specifically: The spatial jigsaw puzzle task is used to divide brainwave channels into several blocks according to brain regions in order to learn spatial topological relationships. The frequency mosaic task is used to divide the features of each channel into a corresponding number of blocks according to several frequency bands, and to uncover the inherent patterns of the frequency bands by shuffling and prediction. The temporal jigsaw puzzle task is used to divide EEG signals into several consecutive segments along the time axis in order to predict the dynamic evolution pattern of emotions in the original sequence. The method for generating the corresponding self-supervised samples is as follows: Based on spatial jigsaw puzzle tasks, frequency jigsaw puzzle tasks, and temporal jigsaw puzzle tasks, EEG data is simultaneously augmented by spatial segmentation of brain regions, frequency segmentation, and temporal segment perturbation to generate positive and negative sample pairs for self-supervised learning. Positive samples are augmented versions of the same EEG sequence after slight temporal / spatial / frequency perturbation, while negative samples are different emotional sequences, scrambled temporal segments, or cross-subject data. The contrastive learning task aims to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs by mapping augmented samples to a common feature space through a projection head.

[0011] Preferably, a graph neural network is designed based on Chebyshev polynomials. The expression for the Chebyshev polynomial is: ; ; in, It is the activation function, and X is the input data. β k These are the parameters learned during network training. T k (·) is a Chebyshev polynomial of order K, where K = 2. λ max It is the Laplace matrix L The largest eigenvalue, This represents the normalized Laplace matrix. I Represents the identity matrix.

[0012] This invention provides a multi-task self-supervised graph neural network emotion recognition system based on spatiotemporal-frequency fusion. The system is used to implement the aforementioned method and includes: an acquisition and preprocessing module, a graph construction module, a feature extraction and classification module, and a recognition module. The acquisition and preprocessing module is used to acquire raw EEG signals and preprocess them to obtain standardized time-series EEG data. The graph construction module is used to generate standard graph structure data by multi-domain graph generation based on the obtained standardized time-series EEG data using the graph modeling module; and to construct a multi-task self-supervised learning task and generate corresponding self-supervised samples based on the standard graph structure data. The feature extraction and classification module is used to construct a graph neural network and a classification head for feature extraction and classification based on multi-task self-supervised learning tasks and self-supervised samples. At the same time, it combines temporal attention and dynamic weight allocation mechanisms to focus on key segments, thereby achieving effective feature extraction and deep fusion. The recognition module is used to jointly optimize the multi-task loss based on the fused features using a dynamic weight allocation mechanism, and finally output the emotion recognition result.

[0013] Preferably, multi-domain graph generation includes designing graph node and edge relationships from the time, space, and frequency domains of standardized temporal EEG data, specifically: In the time domain, the EEG signal is divided into several continuous segments along the time axis. Each time segment is used as a graph node, and the node features are the average energy or differential entropy features of each channel in different frequency bands within the segment. The initial adjacency matrix is ​​constructed based on the cosine similarity between the features of the time segments and dynamically updated by combining the time decay factor to generate a time-varying adjacency matrix to characterize the temporal evolution characteristics of emotions. In the spatial domain, EEG channels are defined as graph nodes, and node features are the multi-band energy features of the corresponding channels. The adjacency matrix is ​​jointly constructed by the three-dimensional spatial distance matrix between electrodes and the phase-locked value functional connection matrix. The two types of weights are adaptively fused through the attention mechanism to generate a time-varying adjacency matrix to reflect the dynamic topological relationship between brain regions. In the frequency domain, the signals of each channel are divided into graph nodes according to several frequency bands. The node features are the energy values ​​of the corresponding frequency bands. The adjacency matrix is ​​constructed based on the energy correlation between frequency bands, thereby realizing the structured modeling and dynamic correlation learning of frequency information. In the time domain, the features of each node are represented as follows: , ; in Indicates time segment upper frequency band The average energy; Indicates the first Each channel time segment upper frequency band The power or differential entropy, Indicates the total number of channels; The initial adjacency matrix is ​​defined as: ; in and They represent the first The first time segment and the first The node features of each time segment are used to calculate the correlation between nodes through cosine similarity, and then the initial neighbor matrix is ​​obtained. A time distance factor is introduced to dynamically update the neighbor matrix, achieving a balanced modeling of long-term dependencies and short-term changes; specifically, time segments are defined. and The time distance is ,in For time segments The corresponding time point, For time segments At the corresponding time point, the elements of the dynamic adjacency matrix are updated as follows: ; in >0 represents the time constant; In the spatial domain, the adjacency matrix adaptively fuses the distance matrix and PLV values ​​through an attention mechanism to achieve adaptive fusion of topological and functional information, generating a spatial domain adjacency matrix that updates over time. ; ; ; in Represents a node ij The distance matrix between them express Represents a node ij The PLV functional connection matrix between them The adjacency matrix of the spatial domain graph represents the first... The final connection weight of a node; and Indicates the first and the Three-dimensional coordinate vector of each EEG channel , Represents the total number of sampling points in a time segment. and Indicates the first and the Each channel at time The instantaneous phase, The activation function, usually sigmoid, maps the weighted result to the interval [0, 1]. This is a distance normalization constant used to control the decay rate. w 1 and w 2The learnable fusion weights control the contribution of geometric distance and functional connectivity to the final adjacency, respectively. In the frequency domain, the node characteristic is the average energy of the corresponding frequency band across all channels. C represents the total number of channels. The correlation between different frequency bands is calculated using the Pearson correlation coefficient or mutual information. ; Introducing spectral energy difference weights: ; in It is the dynamic connection strength considering the energy difference. E b Indicates the first b Energy of each frequency band This represents the energy of the b'-th frequency band. By controlling the impact of energy differences in different frequency bands on connection strength, structured modeling and dynamic correlation learning of frequency information can be achieved.

[0014] Preferably, the constructed multi-task self-supervised tasks include spatial mosaicking, frequency mosaicking, temporal mosaicking, and contrastive learning tasks, specifically: The spatial jigsaw puzzle task is used to divide brainwave channels into several blocks according to brain regions in order to learn spatial topological relationships. The frequency mosaic task is used to divide the features of each channel into a corresponding number of blocks according to several frequency bands, and to uncover the inherent patterns of the frequency bands by shuffling and prediction. The temporal jigsaw puzzle task is used to divide EEG signals into several consecutive segments along the time axis in order to predict the dynamic evolution pattern of emotions in the original sequence. The method for generating the corresponding self-supervised samples is as follows: Based on spatial jigsaw puzzle tasks, frequency jigsaw puzzle tasks, and temporal jigsaw puzzle tasks, EEG data is simultaneously augmented by spatial segmentation of brain regions, frequency segmentation, and temporal segment perturbation to generate positive and negative sample pairs for self-supervised learning. Positive samples are augmented versions of the same EEG sequence after slight temporal / spatial / frequency perturbation, while negative samples are different emotional sequences, scrambled temporal segments, or cross-subject data. The contrastive learning task aims to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs by mapping augmented samples to a common feature space through a projection head.

[0015] Preferably, a graph neural network is designed based on Chebyshev polynomials. The expression for the Chebyshev polynomial is: ; ; in, It is the activation function, and X is the input data. β k These are the parameters learned during network training. T k (·) is a Chebyshev polynomial of order K, where K = 2. λ max It is the Laplace matrix L The largest eigenvalue, This represents the normalized Laplace matrix. I Represents the identity matrix.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. Multi-domain fusion modeling improves the completeness of feature representation. This invention performs multi-domain graph structure modeling of EEG signals from three dimensions: time, space, and frequency. It can simultaneously capture the temporal dynamics of EEG signals, brain region topological connectivity, and spectral energy distribution characteristics, effectively overcoming the problem of incomplete features in traditional single-domain analysis.

[0017] 2. Multi-task self-supervised learning enhances model generalization ability. By introducing self-supervised tasks such as temporal mosaicking, spatial mosaicking, frequency mosaicking, and contrastive learning in addition to the main task (emotion recognition), cross-task feature sharing and knowledge transfer are achieved, reducing dependence on large-scale labeled data and significantly improving the model's robustness and generalization performance in cross-subject and noisy environments.

[0018] 3. Dynamic Graph Neural Networks for Time-Varying Modeling of EEG Topology. This invention employs a dynamic graph convolutional structure that integrates spatial distance and functional connectivity, combined with an attention mechanism to achieve adaptive updating of the adjacency matrix. This dynamically reflects changes in functional coupling relationships between brain regions, thereby more accurately capturing the instantaneous changes in emotional states. Attached Figure Description

[0019] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a system framework diagram of the multi-task self-supervised graph neural network emotion recognition method based on time-space-frequency fusion provided by the present invention; Figure 2 A schematic diagram illustrating the steps of the multi-task self-supervised graph neural network emotion recognition method based on time-space-frequency fusion provided by the present invention; Figure 3 This is a schematic diagram of a time-series jigsaw puzzle task construction method provided in an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating a spatial jigsaw puzzle construction method provided in an embodiment of the present invention; Figure 5 This is a schematic diagram illustrating the frequency mosaic task construction method provided in an embodiment of the present invention; Figure 6 This is a schematic diagram illustrating the method for constructing a comparative learning task according to an embodiment of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] Example 1 To address the shortcomings of existing EEG decoding technologies, such as insufficient temporal modeling, weak generalization ability, and sensitivity to noise labels, this invention provides a multi-task self-supervised graphical neural network (SVM) emotion recognition method based on temporal-spatial-frequency fusion. A multi-task SVM model is proposed for EEG emotion classification, particularly by combining temporal, spatial, and frequency domains to enhance the model's feature extraction capabilities and generalization performance. The main steps include: S1: After acquiring raw electroencephalogram (EEG) signals, perform relevant preprocessing operations to obtain standardized time-series EEG data.

[0024] S2: Based on the obtained standardized time-series EEG data, the graph modeling module is used to generate standard graph structure data by multi-domain graph generation. S3: Design multi-task self-supervised learning tasks, including designing temporal jigsaw puzzles, spatial jigsaw puzzles, frequency jigsaw puzzles, and contrastive learning tasks, and generate corresponding self-supervised samples; S4: Design a graph neural network and a subsequent classification head for feature extraction and classification. At the same time, combine temporal attention and dynamic weight allocation mechanisms to focus on key segments, thereby achieving effective feature extraction and deep fusion.

[0025] S5: Train and test the graph neural network based on the public dataset, evaluate the model performance through indicators such as emotion classification accuracy, and verify the generalization ability and noise robustness of the designed neural network.

[0026] Furthermore, in step S1, the relevant preprocessing includes filtering the original data, data normalization, segmentation and removal of baseline and power frequency, and preliminary feature extraction. The filtering process involves multi-stage filtering of the EEG signal: First, a notch filter is used to suppress power frequency interference (50 Hz or 60 Hz) and its harmonic components, reducing the impact of power supply noise on the signal; then, a band-pass filter is used to further limit the effective frequency range. The high-pass filter removes DC offset, slow drift caused by poor electrode contact, and low-frequency trends caused by breathing and movement, while the low-pass filter suppresses amplifier noise, electromyography (EMG) interference, and high-frequency power frequency harmonics or electromagnetic noise. This significantly improves the signal-to-noise ratio while preserving the main EEG activity frequency bands (e.g., 0.5–45 Hz, where δ: 0.5–4 Hz, θ: 4–8 Hz, α: 8–13 Hz, β: 13–30 Hz).

[0027] ; The above represents the raw EEG signals. After notch filtering, the result is... Based on this, a bandpass filter is then applied to obtain... ,in t This represents a time-point variable used to describe the changes in EEG signals over time and to determine the signal values ​​at different moments. Notch(·) Indicates notch filtering. f 0 It is usually set to 50 Hz or 60 Hz; Bandpass(·) Indicates bandpass filtering, where f l It is usually set to 40-45Hz. f h It is usually set to 0.1-1.0Hz.

[0028] Then process the signal Channel adaptive weighted normalization (CAN) is performed to obtain normalized EEG signals. .

[0029] ; in, i This is a channel index used to distinguish different EEG acquisition channels. Indicates the first i The signal after channel adaptive weighted normalization processing of each channel signal w i Indicates the first i The learning weights of each channel, Indicates the first i The signals from each channel, after being bandpass filtered, are the input for CAN processing. Indicates the first i The average value of the signals in each channel. Indicates the first i The standard deviation of the channel signal.

[0030] In particular, to enhance cross-subject consistency, this invention introduces Channel-wise Adaptive Normalization (CAN) as shown in the above equation, based on traditional Z-score normalization. That is, learning weights are introduced into each channel. w i This approach maintains signal scale uniformity while enhancing the contribution of highly correlated channels and suppressing noise channels, thereby improving the robustness of model feature extraction.

[0031] Furthermore, in step S2, the multi-domain graph generation includes processing the obtained standardized temporal EEG data. ( C (Each channel has undergone preprocessing such as bandpass / normalization.) The relationship between nodes and edges in the design graph is based on the time, space, and frequency domains. Specifically: In the time domain, the EEG signal is divided into several continuous segments along the time axis. Each time segment is used as a graph node. The node features are the mean energy or differential entropy features of each channel in different frequency bands (δ, θ, α, β, γ) within the segment. The initial adjacency matrix is ​​constructed based on the cosine similarity between the features of the time segments and dynamically updated by combining the time decay factor to generate a time-varying adjacency matrix to characterize the temporal evolution characteristics of emotions. Specifically, the features of each node can be represented as follows: , ; in Indicates time segment upper frequency band The average energy; Indicates the first Each channel time segment upper frequency band The power or differential entropy, Indicates the total number of channels; Initial adjacency matrix The definition is as follows: ; in and They represent the first The first time segment and the first The node features of each time segment are used to calculate the correlation between nodes through cosine similarity, and then the initial neighbor matrix is ​​obtained. Specifically, to reflect the temporal decay pattern, a time distance factor is introduced to dynamically update the neighbor matrix, thereby achieving a balanced modeling of long-term dependencies and short-term changes. Specifically, time segments are defined. and The time distance is ,in For time segments The corresponding time point, For time segments At the corresponding time point, the elements of the dynamic adjacency matrix are updated as follows: ; in It is a time constant (which controls the influence of adjacent time periods on the decay rate).

[0032] In the spatial domain, EEG channels are defined as graph nodes, and node features are the multi-band energy characteristics of the corresponding channels (similar to the time domain definition above, node features can be represented as...). ,in They represent the first c A brainwave channel in (Energy characteristics of frequency bands), the adjacency matrix is ​​jointly constructed by the three-dimensional spatial distance matrix between electrodes and the phase-locked value (PLV) functional connectivity matrix. The two types of weights are adaptively fused through the attention mechanism to generate a time-varying adjacency matrix to reflect the dynamic topological relationship between brain regions; Specifically, the adjacency matrix adaptively fuses the distance matrix and PLV values ​​through an attention mechanism to achieve adaptive fusion of topological and functional information, generating a spatial domain adjacency matrix that can be updated over time. ; ; ; in Represents a node ij The distance matrix between them express Represents a node ij The PLV functional connection matrix between them The adjacency matrix of the spatial domain graph represents the first... The final connection weight of the node. Specifically, and Indicates the first and the Three-dimensional coordinate vector of each EEG channel (electrode) , Represents the total number of sampling points in a time segment. and Indicates the first and the Each channel at time The instantaneous phase (usually obtained through Hilbert transform or wavelet transform). The activation function, usually sigmoid, maps the weighted result to the interval [0, 1]. This is a distance normalization constant used to control the decay rate (scale parameter). w 1 and w 2 The learnable fusion weights control the contribution of geometric distance and functional connectivity to the final adjacency, respectively. In the frequency domain, each channel signal is divided into graph nodes according to five frequency bands: δ, θ, α, β, and γ. The node features are the energy values ​​of the corresponding frequency bands. The adjacency matrix is ​​constructed based on the energy correlation between frequency bands, thereby realizing the structured modeling and dynamic correlation learning of frequency information. Specifically, the node characteristic is the average energy of the corresponding frequency band across all channels. (C represents the total number of channels). The correlation between different frequency bands can be calculated using the Pearson correlation coefficient or mutual information. ; In addition, to model the dynamic dependencies between frequency bands, a spectral energy difference weight is introduced: ; in It is the dynamic connection strength considering the energy difference. E b Indicates the first b Energy of each frequency band This represents the energy of the b'th frequency band ( ), By controlling the impact of energy differences in different frequency bands on connection strength, structured modeling and dynamic correlation learning of frequency information can be achieved.

[0033] Furthermore, in step S3, the constructed self-supervised task includes spatial mosaicking, frequency mosaicking, temporal mosaicking, and contrastive learning tasks, specifically: The spatial jigsaw puzzle task divides the brainwave channels into 10 blocks according to brain regions to learn spatial topological associations. And randomly shuffle to obtain the corresponding sequence. By learning the topological relationships between brain regions through block-level spatial rearrangement, the ability to perceive cross-regional emotional patterns can be improved. The frequency mosaicking task divides the features of each channel into 5 blocks according to the five frequency bands of δ, θ, α, β, and γ. By disrupting and predicting, the inherent patterns of frequency bands can be uncovered; The time-series jigsaw puzzle task divides the EEG signal into T (e.g., 10) consecutive segments along the time axis. By shuffling and rearranging the order, dynamic evolutionary patterns of emotions can be captured to predict the original sequence. The contrastive learning method, based on the above three tasks, simultaneously performs brain region spatial segmentation, frequency band segmentation, and time sequence segment perturbation (such as short time shift and local segment preservation) on EEG data to generate positive and negative sample pairs for self-supervised learning. In this embodiment, the positive samples of the contrastive learning method are obtained by applying spatial perturbation to the same electroencephalogram (EEG) sequence. Frequency disturbance Time-series disturbances Generates through a combination of operations, i.e. ,in Spatial structure disturbance is achieved by randomly blocking or replacing parts of the channels. Frequency domain perturbation is achieved by scaling or randomly masking amplitudes in frequency bands such as δ, θ, α, β, and γ. Temporal perturbations are achieved by shuffling time segments or pruning time windows, thereby enhancing the model's robustness to multi-domain perturbations. Negative samples From different emotion samples Disordered time sequence fragment samples and cross-subject data samples Composition, that is To increase the discriminative power of contrastive learning.

[0034] The self-supervised learning method of contrastive learning described above maps augmented samples to a common feature space through a projection head, aiming to maximize the similarity of positive sample pairs (preserving temporal continuity and sentiment consistency) and minimize the similarity of negative sample pairs (corresponding to the loss). L p ); ; in , , These are the original EEG samples. x Positive samples With negative samples via feature encoder and projector head The feature representation obtained after processing, Represents the feature encoder. For projection head, For cosine similarity, Temperature parameter (controls the smoothness of similarity distribution). The multimodal perturbation consistency weight is obtained by weighting consistency measures in spatial, frequency, and temporal dimensions; Furthermore, in step S4, the loss function used by the dynamic graph convolutional neural network is the cross-entropy loss function optimization, combined with L2 regularization to improve generalization ability. ; in, For cross-entropy loss, N For the sample size, y i For unique hot encoding of tags, For classification head or projection head, Graph convolutional neural networks, X i For input data.

[0035] Furthermore, in step S5, the main parameters for training the dynamic graph convolutional neural network are designed as follows: The Adam optimizer has a batch size of 100; the initial learning rate is 0.001 and the weight decay rate is 8e-5; fine-tuning will be performed according to specific circumstances during the experiment.

[0036] Mean Accuracy (ACC) and Standard Deviation (STD) are used as evaluation criteria for all EEG sentiment datasets.

[0037] Furthermore, in the graph neural network, the absence of overfitting is used as the standard for good training of the graph neural network.

[0038] Example 2 Using the multi-task self-supervised graph neural network emotion recognition method based on spatiotemporal-frequency fusion described in this invention, EEG emotion recognition is performed, mainly including: Figure 2 The steps shown are as follows: S1: After acquiring raw electroencephalogram (EEG) signals, perform relevant preprocessing operations to obtain standardized time-series EEG data.

[0039] The process of collecting EEG emotion data involves the subject wearing an electrode cap and watching an emotion-evoking video. During the viewing of the video, the subject's EEG emotion data is collected through the electrodes on the electrode cap.

[0040] This invention samples publicly available datasets, including the SEED (three emotions: happiness, sadness, and neutral) and SEED-IV (four emotions: happiness, sadness, fear, and neutral) EEG emotion datasets from Shanghai Jiao Tong University, and the MPED (seven emotions: joy, fun, anger, fear, disgust, sadness, and neutral) EEG emotion dataset from Southeast University. All three datasets were acquired using a 62-channel acquisition device with an internationally accepted 10-20 lead electrode distribution. Data acquisition involved 15, 15, and 30 subjects respectively, and the EEG emotion data was segmented into 1-second segments. The data sampling frequency was 1000 Hz, and differential entropy (DE) or short-time Fourier transform (STFT) features were extracted, dividing the data into five subbands: δ (1-3 Hz), θ (4-7 Hz), α (8-13 Hz), β (14-30 Hz), and γ (31-50 Hz).

[0041] The relevant preprocessing includes filtering the raw data, normalizing the data, segmenting and removing the baseline, removing the power frequency, and preliminarily extracting features. S2: Based on the obtained standardized temporal EEG data, the graph modeling module is used to generate a multi-domain graph to obtain standard graph structure data; In the time domain, the EEG signal is divided into several continuous segments along the time axis. Each time segment is used as a graph node. The node features are the mean energy or differential entropy features of each channel in different frequency bands (δ, θ, α, β, γ) within the segment. The initial adjacency matrix is ​​constructed based on the cosine similarity between the features of the time segments and dynamically updated by combining the time decay factor to generate a time-varying adjacency matrix to characterize the temporal evolution characteristics of emotions. The EEG signal is divided into T consecutive segments (e.g., each segment is 1 second) along the time axis. Each time segment is a graph node, and the node feature is the average energy of the five frequency bands δ / θ / α / β / γ of the 62 channels within that segment (or the differential entropy DE feature), forming a feature matrix X∈R. Tx5 (T represents the number of time segments, and 5 represents the number of frequency bands); The initial adjacency matrix is ​​formed by calculating the cosine similarity between the feature vectors of any two segments, which measures the degree of correlation between their emotional states (e.g., adjacent segments have high similarity when emotions are stable, and similarity drops sharply when emotions change abruptly), thus forming the initial adjacency matrix A∈R. TxT ; In the spatial domain, EEG channels are defined as graph nodes, and node features are the multi-band energy features of the corresponding channels. The adjacency matrix is ​​jointly constructed by the three-dimensional spatial distance matrix between electrodes and the phase-locked value (PLV) functional connectivity matrix. The two types of weights are adaptively fused through the attention mechanism to generate a time-varying adjacency matrix to reflect the dynamic topological relationships between brain regions. In the frequency domain, each channel signal is divided into graph nodes according to five frequency bands: δ, θ, α, β, and γ. The node features are the energy values ​​of the corresponding frequency bands. The adjacency matrix is ​​constructed based on the energy correlation between frequency bands, thereby realizing the structured modeling and dynamic correlation learning of frequency information. Specifically, the differential entropy (DE) or short-time Fourier transform (STFT) characteristics of the 62-channel EEG signal (divided into five frequency bands: δ / θ / α / β / γ) are mapped to a graph node feature matrix. Each node corresponds to an electrode in a specific brain region. Dynamic adjacency matrix. The construction integrates dual relationships: 1) The spatial distance matrix based on the three-dimensional coordinates of the electrodes (transformed by Gaussian kernel function) preserves the prior knowledge of brain anatomy; 2) Capturing the dynamic synchronicity of neural activity using a functional connectivity matrix based on phase-locked value (PLV). A time-varying adjacency matrix is ​​generated by adaptively integrating spatial distance and PLV weights through an attention mechanism. A Associativity matrix D Constructing the Laplace operator L(t) = D - A(t) ; S3: Design multi-task self-supervised learning tasks, including designing temporal jigsaw puzzles, spatial jigsaw puzzles, frequency jigsaw puzzles, and contrastive learning tasks, and generate corresponding self-supervised samples; The time-series jigsaw puzzle task, such as Figure 3 As shown, the EEG signal is divided into T consecutive segments (e.g., each segment is 1 second long) along the time axis. Each segment contains the energy characteristics of all channels in the five frequency bands δ / θ / α / β / γ and the associated signals. After shuffling the time segment order, pseudo-labels are assigned, transforming the task into a time sequence prediction task. By learning the correct time sequence, the model automatically captures the dynamic evolution of emotions over time (e.g., happy emotions are usually accompanied by a temporal pattern of γ wave energy first increasing and then decreasing), enhancing the temporal discriminative power of the features.

[0042] The spatial jigsaw puzzle task, such as Figure 4 As shown, firstly, based on the international 10-20 system, the 62-channel EEG is divided into 10 brain regions (such as the prefrontal cortex, central region, parietal lobe, etc.). Then, the EEG emotion data is obtained by randomly shuffling the blocks according to different brain regions. Subsequently, the order of the blocks is randomly shuffled, and the original permutation index is used as a pseudo-label to transform it into a permutation recognition task. This process forces the graph convolutional neural network to learn the spatial semantic associations between brain regions (such as the cooperative patterns between the prefrontal cortex and the limbic system), and captures emotion-related spatial topological features by reconstructing the correct permutation order.

[0043] The frequency mosaic task, such as Figure 5As shown, the features of each channel are divided into five blocks according to five frequency bands: δ, θ, α, β, and γ. After shuffling the order of the frequency band blocks, pseudo-labels are assigned, transforming the task into a frequency band arrangement prediction task. By learning the correct frequency band order, the model automatically selects frequency components that are key to emotion classification (such as the increase in θ wave energy during sadness), thereby enhancing the frequency discriminative power of the features.

[0044] The contrastive learning method, based on the three tasks mentioned above, simultaneously performs data augmentation on EEG data by dividing the brain region spatially into blocks, frequency bands, and perturbing time-series segments (such as short-time shifts and local segment preservation). Figure 6 As shown, positive and negative sample pairs are generated for self-supervised learning, forcing EEG data to align semantically in terms of emotion. This further normalizes the model's feature space, enabling it to learn the intrinsic representation of EEG emotion signals and improve the accuracy of EEG emotion recognition.

[0045] The positive samples mentioned above are enhanced versions of the same EEG sequence after slight temporal / spatial / frequency perturbations, while the negative samples are different emotional sequences, fragments with disordered temporal sequences, or cross-subject data. The self-supervised learning method of contrastive learning described above maps enhanced samples to a common feature space through a projection head, with the goal of maximizing the similarity of positive sample pairs (preserving temporal continuity and emotional consistency) and minimizing the similarity of negative sample pairs (corresponding to loss Lp). The above four tasks work together on the full dataset, combined with maximum and minimum value normalization (mapping features to the [0, 1] interval to accelerate optimization), enabling the model to simultaneously capture the spatial topological patterns, frequency-specific patterns, temporal correlations, and semantic consistency of EEG signals, effectively enhancing the ability to discriminate emotional features.

[0046] S4: Design a graph neural network and a subsequent classification head for feature extraction and classification. At the same time, combine temporal attention and dynamic weight allocation mechanisms to focus on key segments, thereby achieving effective feature extraction and deep fusion.

[0047] Based on the design of a graph convolutional neural network using Chebyshev polynomials, the expression for the Chebyshev polynomial is: ; ; Where σ(·) is the activation function, X It is the input data. β k These are the parameters learned during network training. T k (·) is a Chebyshev polynomial of order K, where K = 2; This represents the normalized Laplace matrix. IRepresents the identity matrix. λ max It is the Laplace matrix L The largest eigenvalue. For each task, the input data has been manually extracted features, therefore the input feature dimension is . The output feature dimension is In this way, the output feature dimension is higher than the input feature dimension, with richer high-dimensional feature information. High-dimensional feature information has a high degree of abstraction and contains richer semantic information.

[0048] The feature extraction of the graph neural network is shared by a main task (emotion recognition) and four self-supervised auxiliary tasks (temporal mosaic, spatial mosaic, frequency mosaic, and contrastive learning), which realizes knowledge transfer and feature fusion among multiple tasks.

[0049] The four types of auxiliary task inputs, through a multi-task collaborative mechanism, enable the main task to integrate the complementarity of spatial proximity information, frequency feature correlation, and temporal association in emotion recognition, thereby improving the recognition accuracy. At the same time, the multi-task iterative learning process forms knowledge complementarity, further enhancing the model's ability to discriminate EEG emotion signals.

[0050] The classification head constructs a multi-task-specific classification architecture through fully connected layers, achieving fine-grained mapping of the feature space. Specifically: The spatial jigsaw puzzle task outputs a spatial classification head with a dimension of 128, which predicts 128 spatial category labels to achieve fine-grained modeling of the spatial topological information of EEG signals; the frequency jigsaw puzzle task outputs a frequency classification head with a dimension of 120, which predicts 120 frequency category labels (such as energy distribution in the δ / θ / α / β / γ bands) to enhance feature extraction of emotionally sensitive frequency bands; the temporal jigsaw puzzle task predicts 128 temporal category labels. In particular, the contrastive learning task uses cosine similarity to measure the similarity of sample pairs through the projector head. By maximizing the similarity of positive sample pairs of the same emotion (such as the feature pair of prefrontal β wave and skin conductance response in the anxious state), it forces EEG features to align in the emotional semantic space, which significantly improves the discriminative ability of the features. Furthermore, the main emotion recognition task sets the output dimension based on the number of emotion categories in the SEED, SEED-IV, and MPED datasets to generate the final predicted emotion label. Through the knowledge-sharing mechanism of the feature extraction module, the emotion classification head integrates four key pieces of information when generating the predicted label: ① the brain region functional connectivity patterns captured by the spatial classification head; ② the frequency band energy change patterns extracted by the frequency classification head; ③ the multi-semantic associations constructed by the contrastive learning task; and ④ the temporal evolution characteristics of the temporal classification head. This cross-task knowledge transfer strategy allows the predicted emotion label to naturally include four layers of information: brain spatial topology, frequency characteristics, temporal characteristics, and fusion, significantly improving the accuracy of EEG emotion recognition and the model's generalization ability to multimodal data.

[0051] S5: Train and test the graph neural network based on the public dataset, evaluate the model performance through indicators such as emotion classification accuracy, and verify the generalization ability and noise robustness of the designed neural network.

[0052] For the public datasets SEED, SEED-IV, and MPED, the training and test sets were divided into training and test sets in ratios of 9:6, 16:8, and 21:7, respectively, to conduct subject dependency experiments. This division facilitates comparison with other results. At the same time, the leave-one-out cross-validation strategy was used to conduct subject independent experiments. The loss function used in the above dynamic graph convolutional neural network is the cross-entropy loss function optimization, combined with L2 regularization to improve generalization ability; ; in, For cross-entropy loss, N For the sample size, y i One-hot encoding for tags, For classification head or projection head, Graph convolutional neural networks, X i For input data.

[0053] The loss function of the entire network is a weighted sum of the losses from multiple tasks: ; in, Representing the i Each weighting coefficient Representing the i Each task loss value, of which i It includes one main task (emotion recognition) and four self-supervised auxiliary tasks (temporal jigsaw puzzle, spatial jigsaw puzzle, frequency jigsaw puzzle, and contrastive learning). The Adam optimizer was used during network training with a batch size of 100; the initial learning rate was 0.001 and the weight decay rate was 8e-5; mean accuracy (ACC) and standard deviation (STD) were used as evaluation metrics for all EEG sentiment datasets.

[0054] During training, a dynamic monitoring and adaptive adjustment mechanism is adopted: after completing a training cycle (traversing all training samples), the model performance is evaluated using the test set every 10 iterations, and the accuracy of the training set and the test set are continuously compared; when the difference between the two exceeds 20%, it indicates that the model is overfitting; if the difference in accuracy remains within a reasonable range, a graph neural network model with strong generalization ability and high recognition accuracy can be obtained after 1000 iterations of training.

[0055] When overfitting occurs, the system will automatically trigger a dual optimization strategy. On the one hand, it will reduce the learning rate by using mechanisms such as cosine annealing to reduce the model's fit to the training data. On the other hand, it will randomly resample 100 samples from the training set to construct a new batch, thereby enhancing the model's generalization ability through data distribution perturbation. If the accuracy of the test does not meet the required precision, the learning rate needs to be adjusted, and the training repeated to retrain the graph convolutional neural network until the required precision is achieved.

[0056] This invention addresses the shortcomings of existing EEG decoding technologies, such as insufficient temporal modeling, weak generalization ability, and sensitivity to noise labels. It proposes a multi-task self-supervised learning framework that jointly optimizes temporal, spatial, and frequency features. Specifically, it first performs graph modeling of EEG signals in the time, spatial, and frequency domains. Subsequently, it designs temporal mosaicking, spatial mosaicking, and frequency mosaicking... Figure 3 This self-supervised task involves segmenting and rearranging EEG signals according to time, space, and frequency. By predicting the original sequence order, the model is guided to autonomously learn key structural features and latent patterns in the EEG signals. Simultaneously, a contrastive learning task is introduced to achieve semantic consistency constraints from different feature perspectives, further improving the discriminativeness and robustness of feature representation. The network employs a dynamic weight allocation mechanism to jointly optimize the multi-task loss. This method effectively enhances the ability to capture the spatiotemporal dynamic characteristics of EEG signals, strengthens the model's generalization ability and robustness to noise labels, and demonstrates high accuracy in emotion recognition, particularly suitable for scenarios involving multiple subjects and noise labels.

[0057] Example 3 like Figure 1 As shown, the present invention provides a multi-task self-supervised graph neural network emotion recognition system based on time-space-frequency fusion. The system is used to implement the method described in Embodiment 1. The system includes: an acquisition and preprocessing module, a graph construction module, a feature extraction and classification module, and a recognition module. The acquisition and preprocessing module is used to acquire raw EEG signals and preprocess them to obtain standardized time-series EEG data. The graph construction module is used to generate standard graph structure data by multi-domain graph generation based on the obtained standardized time-series EEG data using the graph modeling module; based on the standard graph structure data, a multi-task self-supervised learning task is constructed and corresponding self-supervised samples are generated; specifically: multi-domain graph structure modeling and self-supervised sample generation based on EEG data are used as input to the subsequent feature extraction module, providing a structured representation basis for subsequent feature extraction and emotion recognition. The feature extraction and classification module is used to construct a graph neural network and classification head for feature extraction and classification based on multi-task self-supervised learning tasks and self-supervised samples. It also combines temporal attention and dynamic weight allocation mechanisms to focus on key segments, achieving effective feature extraction and deep fusion. Specifically: the feature extraction module mainly uses a Chebyshev multinomial-based graph convolutional network for feature extraction, which can efficiently capture the spatial topology and dynamic characteristics of EEG signals; the classification module designs dedicated classification heads for different tasks, including similarity measurement of comparative information, temporal classification heads, spatial classification heads, frequency classification heads, and emotion classification heads corresponding to the main task of emotion recognition, accurately adapting to the feature learning needs of the corresponding learning tasks and ensuring that features of different dimensions are targeted for mining. The recognition module is used to jointly optimize the multi-task loss based on the fused features using a dynamic weight allocation mechanism, and finally output the emotion recognition result.

[0058] In this embodiment, multi-domain graph generation includes designing graph node and edge relationships from the time, space, and frequency domains of standardized temporal EEG data, specifically: In the time domain, the EEG signal is divided into several continuous segments along the time axis. Each time segment is used as a graph node, and the node features are the average energy or differential entropy features of each channel in different frequency bands within the segment. The initial adjacency matrix is ​​constructed based on the cosine similarity between the features of the time segments and dynamically updated by combining the time decay factor to generate a time-varying adjacency matrix to characterize the temporal evolution characteristics of emotions. In the spatial domain, EEG channels are defined as graph nodes, and node features are the multi-band energy features of the corresponding channels. The adjacency matrix is ​​jointly constructed by the three-dimensional spatial distance matrix between electrodes and the phase-locked value functional connection matrix. The two types of weights are adaptively fused through the attention mechanism to generate a time-varying adjacency matrix to reflect the dynamic topological relationship between brain regions. In the frequency domain, the signals of each channel are divided into graph nodes according to several frequency bands. The node features are the energy values ​​of the corresponding frequency bands. The adjacency matrix is ​​constructed based on the energy correlation between frequency bands, thereby realizing the structured modeling and dynamic correlation learning of frequency information. In the time domain, the features of each node are represented as follows: , ; in Indicates time segment upper frequency band The average energy; Represents the time segment of the c-th channel. upper frequency band The power or differential entropy, C Indicates the total number of channels; The initial adjacency matrix is ​​defined as: ; in and They represent the first The first time segment and the first The node features of each time segment are used to calculate the correlation between nodes through cosine similarity, and then the initial neighbor matrix is ​​obtained. A time distance factor is introduced to dynamically update the neighbor matrix, achieving a balanced modeling of long-term dependencies and short-term changes; specifically, time segments are defined. and The time distance is ,in For time segments The corresponding time point, For time segments At the corresponding time point, the elements of the dynamic adjacency matrix are updated as follows: ; in >0 represents the time constant; In the spatial domain, the adjacency matrix adaptively fuses the distance matrix and PLV values ​​through an attention mechanism to achieve adaptive fusion of topological and functional information, generating a spatial domain adjacency matrix that updates over time. ; ; ; in Represents a node ij The distance matrix between them express Represents a node ij The PLV functional connection matrix between them The adjacency matrix of the spatial domain graph represents the first... The final connection weight of a node; and Indicates the first and the Three-dimensional coordinate vector of each EEG channel ,T Represents the total number of sampling points in a time segment. and Indicates the first and the Each channel at time The instantaneous phase, The activation function, usually sigmoid, maps the weighted result to the interval [0, 1]. This is a distance normalization constant used to control the decay rate. w 1 and w 2 The learnable fusion weights control the contribution of geometric distance and functional connectivity to the final adjacency, respectively. In the frequency domain, the node characteristic is the average energy of the corresponding frequency band across all channels. C represents the total number of channels. The correlation between different frequency bands is calculated using the Pearson correlation coefficient or mutual information. ; Introducing spectral energy difference weights: ; in It is the dynamic connection strength considering the energy difference. E b Indicates the first b Energy of each frequency band This represents the b'th frequency band ( ), By controlling the impact of energy differences in different frequency bands on connection strength, structured modeling and dynamic correlation learning of frequency information can be achieved.

[0059] In this embodiment, the constructed multi-task self-supervised task includes spatial mosaicking, frequency mosaicking, temporal mosaicking, and contrastive learning tasks, specifically: The spatial jigsaw puzzle task is used to divide brainwave channels into several blocks according to brain regions in order to learn spatial topological relationships. The frequency mosaic task is used to divide the features of each channel into a corresponding number of blocks according to several frequency bands, and to uncover the inherent patterns of the frequency bands by shuffling and prediction. The temporal jigsaw puzzle task is used to divide EEG signals into several consecutive segments along the time axis in order to predict the dynamic evolution pattern of emotions in the original sequence. The method for generating corresponding self-supervised samples is as follows: self-supervised sample generation involves reconstructing data from the temporal, spatial, and frequency domains, as well as constructing positive and negative sample pairs for the contrastive learning task; specifically: Based on spatial jigsaw puzzle tasks, frequency jigsaw puzzle tasks, and temporal jigsaw puzzle tasks, EEG data is simultaneously augmented by spatial segmentation of brain regions, frequency segmentation, and temporal segment perturbation to generate positive and negative sample pairs for self-supervised learning. Positive samples are augmented versions of the same EEG sequence after slight temporal / spatial / frequency perturbation, while negative samples are different emotional sequences, scrambled temporal segments, or cross-subject data. The contrastive learning task aims to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs by mapping augmented samples to a common feature space through a projection head.

[0060] In this embodiment, the feature extraction module serves as the core component and is shared by one main task (emotion recognition) and four self-supervised auxiliary tasks (temporal mosaic, spatial mosaic, frequency mosaic, and contrastive learning), thereby enabling knowledge transfer and feature fusion among multiple tasks.

[0061] In this embodiment, a graph neural network is designed based on Chebyshev polynomials. The expression for the Chebyshev polynomial is: ; ; in, It is the activation function, and X is the input data. β k These are the parameters learned during network training. T k (·) is a Chebyshev polynomial of order K, where K = 2. This represents the normalized Laplace matrix. I Represents the identity matrix. λ max It is the Laplace matrix L The largest eigenvalue.

[0062] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A multi-task self-supervised graph neural network emotion recognition method based on spatiotemporal-frequency fusion, characterized in that, The method includes: S1: Acquire raw EEG signals and preprocess them to obtain standardized time-series EEG data; S2: Based on the obtained standardized time-series EEG data, the graph modeling module is used to generate standard graph structure data by multi-domain graph generation. S3: Based on standard graph structure data, construct a multi-task self-supervised learning task and generate corresponding self-supervised samples; S4: Based on multi-task self-supervised learning tasks and self-supervised samples, a graph neural network and a classification head are constructed for feature extraction and classification. At the same time, a temporal attention and dynamic weight allocation mechanism are combined to focus on key segments, so as to achieve effective feature extraction and deep fusion. S5: Based on the fused features, a dynamic weight allocation mechanism is used to jointly optimize the multi-task loss, and finally output the emotion recognition result.

2. The method according to claim 1, characterized in that, Multi-domain graph generation involves designing graph node and edge relationships from the time, space, and frequency domains of standardized temporal EEG data, specifically: In the time domain, the EEG signal is divided into several continuous segments along the time axis. Each time segment is used as a graph node, and the node features are the average energy or differential entropy features of each channel in different frequency bands within that segment. The initial adjacency matrix is ​​constructed based on the cosine similarity between features of time segments and dynamically updated by combining a time decay factor to generate a time-varying adjacency matrix to characterize the temporal evolution characteristics of emotions. In the spatial domain, EEG channels are defined as graph nodes, and node features are the multi-band energy features of the corresponding channels. The adjacency matrix is ​​jointly constructed by the three-dimensional spatial distance matrix between electrodes and the phase-locked value functional connection matrix. The two types of weights are adaptively fused through the attention mechanism to generate a time-varying adjacency matrix to reflect the dynamic topological relationship between brain regions. In the frequency domain, the signals of each channel are divided into graph nodes according to several frequency bands. The node features are the energy values ​​of the corresponding frequency bands. The adjacency matrix is ​​constructed based on the energy correlation between frequency bands, thereby realizing the structured modeling and dynamic correlation learning of frequency information. In the time domain, the features of each node are represented as follows: , ； in Indicates time segment upper frequency band The average energy; Indicates the first Each channel time segment upper frequency band The power or differential entropy, Indicates the total number of channels; The initial adjacency matrix is ​​defined as: ； in and They represent the first The first time segment and the first The node features of each time segment are used to calculate the correlation between nodes through cosine similarity, thereby obtaining the initial neighbor matrix; A time distance factor is introduced to dynamically update the neighbor matrix, achieving a balanced modeling of long-term dependencies and short-term changes; specifically, time segments are defined. and The time distance is ,in For time segments The corresponding time point, For time segments At the corresponding time point, the elements of the dynamic adjacency matrix are updated as follows: ； in It is a time constant; In the spatial domain, the adjacency matrix adaptively fuses the distance matrix and PLV values ​​through an attention mechanism to achieve adaptive fusion of topological and functional information, generating a spatial domain adjacency matrix that updates over time. ； ； ； in Represents a node The distance matrix between them express Represents a node The PLV functional connection matrix between them The adjacency matrix of the spatial domain graph represents the first... The final connection weight of a node; and Indicates the first and the Three-dimensional coordinate vector of each EEG channel , Represents the total number of sampling points in a time segment. and Indicates the first and the Each channel at time The instantaneous phase, The activation function, usually sigmoid, maps the weighted result to the interval [0, 1]. This is a distance normalization constant used to control the decay rate. and The learnable fusion weights control the contribution of geometric distance and functional connectivity to the final adjacency, respectively. In the frequency domain, the node characteristic is the average energy of the corresponding frequency band across all channels. C represents the total number of channels. The correlation between different frequency bands is calculated using the Pearson correlation coefficient or mutual information. ； Introducing spectral energy difference weights: ； in It is the dynamic connection strength considering the energy difference. E b Indicates the first b Energy of each frequency band This represents the energy of the b'-th frequency band. By controlling the impact of energy differences in different frequency bands on connection strength, structured modeling and dynamic correlation learning of frequency information can be achieved.

3. The method according to claim 1, characterized in that, The constructed multi-task self-supervised tasks include spatial mosaicking, frequency mosaicking, temporal mosaicking, and contrastive learning tasks, specifically: The spatial jigsaw puzzle task is used to divide brainwave channels into several blocks according to brain regions in order to learn spatial topological relationships. The frequency mosaic task is used to divide the features of each channel into a corresponding number of blocks according to several frequency bands, and to uncover the inherent patterns of the frequency bands by shuffling and prediction. The temporal jigsaw puzzle task is used to divide EEG signals into several consecutive segments along the time axis in order to predict the dynamic evolution pattern of emotions in the original sequence. The method for generating the corresponding self-supervised samples is as follows: Based on spatial jigsaw puzzle tasks, frequency jigsaw puzzle tasks, and temporal jigsaw puzzle tasks, EEG data is simultaneously augmented by spatial segmentation of brain regions, frequency segmentation, and temporal segment perturbation to generate positive and negative sample pairs for self-supervised learning. Positive samples are augmented versions of the same EEG sequence after slight temporal / spatial / frequency perturbation, while negative samples are different emotional sequences, scrambled temporal segments, or cross-subject data. The contrastive learning task aims to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs by mapping augmented samples to a common feature space through a projection head.

4. The method according to claim 1, characterized in that, Based on the design of graph neural networks using Chebyshev polynomials, the expression for the Chebyshev polynomial is: ； ； in, It is the activation function, and X is the input data. β k These are the parameters learned during network training. T k (·) is a Chebyshev polynomial of order K, where K = 2. λ max It is the Laplace matrix L The largest eigenvalue, This represents the normalized Laplace matrix. I Represents the identity matrix.

5. A multi-task self-supervised graph neural network emotion recognition system based on spatiotemporal-frequency fusion, the system being used to implement the method described in any one of claims 1-4, characterized in that, The system includes: an acquisition and preprocessing module, a graph construction module, a feature extraction and classification module, and a recognition module; The acquisition and preprocessing module is used to acquire raw EEG signals and preprocess them to obtain standardized time-series EEG data. The graph construction module is used to generate standard graph structure data by multi-domain graph generation based on the obtained standardized time-series EEG data using the graph modeling module; and to construct a multi-task self-supervised learning task and generate corresponding self-supervised samples based on the standard graph structure data. The feature extraction and classification module is used to construct a graph neural network and a classification head for feature extraction and classification based on multi-task self-supervised learning tasks and self-supervised samples. At the same time, it combines temporal attention and dynamic weight allocation mechanisms to focus on key segments, thereby achieving effective feature extraction and deep fusion. The recognition module is used to jointly optimize the multi-task loss based on the fused features using a dynamic weight allocation mechanism, and finally output the emotion recognition result.

6. The system according to claim 5, characterized in that, Multi-domain graph generation involves designing graph node and edge relationships from the time, space, and frequency domains of standardized temporal EEG data, specifically: In the time domain, the EEG signal is divided into several continuous segments along the time axis. Each time segment is used as a graph node, and the node features are the average energy or differential entropy features of each channel in different frequency bands within that segment. The initial adjacency matrix is ​​constructed based on the cosine similarity between features of time segments and dynamically updated by combining a time decay factor to generate a time-varying adjacency matrix to characterize the temporal evolution characteristics of emotions. In the spatial domain, EEG channels are defined as graph nodes, and node features are the multi-band energy features of the corresponding channels. The adjacency matrix is ​​jointly constructed by the three-dimensional spatial distance matrix between electrodes and the phase-locked value functional connection matrix. The two types of weights are adaptively fused through the attention mechanism to generate a time-varying adjacency matrix to reflect the dynamic topological relationship between brain regions. In the frequency domain, the signals of each channel are divided into graph nodes according to several frequency bands. The node features are the energy values ​​of the corresponding frequency bands. The adjacency matrix is ​​constructed based on the energy correlation between frequency bands, thereby realizing the structured modeling and dynamic correlation learning of frequency information. In the time domain, the features of each node are represented as follows: , ； in Indicates time segment upper frequency band The average energy; Represents the time segment of the c-th channel. upper frequency band The power or differential entropy, C Indicates the total number of channels; The initial adjacency matrix is ​​defined as: ； in and They represent the first The first time segment and the first The node features of each time segment are used to calculate the correlation between nodes through cosine similarity, thereby obtaining the initial neighbor matrix; A time distance factor is introduced to dynamically update the neighbor matrix, achieving a balanced modeling of long-term dependencies and short-term changes; specifically, time segments are defined. and The time distance is ,in For time segments The corresponding time point, For time segments At the corresponding time point, the elements of the dynamic adjacency matrix are updated as follows: ； in >0 represents the time constant; In the spatial domain, the adjacency matrix adaptively fuses the distance matrix and PLV values ​​through an attention mechanism to achieve adaptive fusion of topological and functional information, generating a spatial domain adjacency matrix that updates over time. ； ； ； in Represents a node ij The distance matrix between them express Represents a node ij The PLV functional connection matrix between them The adjacency matrix of the spatial domain graph represents the first... The final connection weight of a node; and Indicates the first and the Three-dimensional coordinate vector of each EEG channel , Represents the total number of sampling points in a time segment. and Indicates the first and the Each channel at time The instantaneous phase, The activation function, usually sigmoid, maps the weighted result to the interval [0, 1]. This is a distance normalization constant used to control the decay rate. w 1 and w 2 The learnable fusion weights control the contribution of geometric distance and functional connectivity to the final adjacency, respectively. In the frequency domain, the node characteristic is the average energy of the corresponding frequency band across all channels. C represents the total number of channels. The correlation between different frequency bands is calculated using the Pearson correlation coefficient or mutual information. ； Introducing spectral energy difference weights: ； in It is the dynamic connection strength considering the energy difference. E b Indicates the first b Energy of each frequency band This represents the energy of the b'-th frequency band. By controlling the impact of energy differences in different frequency bands on connection strength, structured modeling and dynamic correlation learning of frequency information can be achieved.

7. The system according to claim 5, characterized in that, The constructed multi-task self-supervised tasks include spatial mosaicking, frequency mosaicking, temporal mosaicking, and contrastive learning tasks, specifically: The spatial jigsaw puzzle task is used to divide brainwave channels into several blocks according to brain regions in order to learn spatial topological relationships. The frequency mosaic task is used to divide the features of each channel into a corresponding number of blocks according to several frequency bands, and to uncover the inherent patterns of the frequency bands by shuffling and prediction. The temporal jigsaw puzzle task is used to divide EEG signals into several consecutive segments along the time axis in order to predict the dynamic evolution pattern of emotions in the original sequence. The method for generating the corresponding self-supervised samples is as follows: Based on spatial jigsaw puzzle tasks, frequency jigsaw puzzle tasks, and temporal jigsaw puzzle tasks, EEG data is simultaneously augmented by spatial segmentation of brain regions, frequency segmentation, and temporal segment perturbation to generate positive and negative sample pairs for self-supervised learning. Positive samples are augmented versions of the same EEG sequence after slight temporal / spatial / frequency perturbation, while negative samples are different emotional sequences, scrambled temporal segments, or cross-subject data. The contrastive learning task aims to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs by mapping augmented samples to a common feature space through a projection head.

8. The system according to claim 5, characterized in that, Based on the design of graph neural networks using Chebyshev polynomials, the expression for the Chebyshev polynomial is: ； ； in, It is the activation function, and X is the input data. β k These are the parameters learned during network training. T k (·) is a Chebyshev polynomial of order K, where K = 2. λ max It is the Laplace matrix L The largest eigenvalue, This represents the normalized Laplace matrix. I Represents the identity matrix.

Images