Electroencephalogram emotion recognition method based on spatiotemporal mask contrast learning and electronic device

Abstract

The application discloses an electroencephalogram emotion recognition method based on space-time mask contrast learning and electronic equipment, when electroencephalogram emotion recognition is performed, first, electroencephalogram EEG signals in multiple electroencephalogram emotion data sets are subjected to channel configuration standardization preprocessing to obtain standardized EEG signals; then, through a double-flow encoder, the standardized EEG signals are subjected to space feature extraction and time feature extraction, and the extracted space flow representation and time flow representation are fused to obtain global space-time representation; then, a classification head module is used to predict the global space-time representation to obtain a predicted emotion category. The double-flow encoder constructed by the application is a model obtained after space-time mask contrast pre-training, can flexibly process heterogeneous electroencephalogram data input, and can automatically adapt without being redesigned regardless of changes in the number of channels and the length of time series, thereby improving the cross-data set generalization capability. Moreover, the double-flow encoder comprises two independent processing branches of space flow and time flow, the two flows are optimized in respective independent feature spaces, mutual interference and compromise of two types of heterogeneous dependence in unified modeling are avoided, the complex space-time dynamics related to emotions can be more fully and accurately modeled, and more robust deep features can be extracted.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a brainwave emotion recognition method and electronic device based on spatiotemporal mask contrast learning. Background Technology

[0002] Emotions are fundamental psychological and physiological responses in humans, playing a crucial role in cognition, decision-making, and social interaction. Accurate identification of emotional states is essential for building intelligent and human-centered artificial intelligence systems, with broad application prospects in fields such as healthcare, education, and human-computer interaction. With the advancement of artificial intelligence technology, emotion recognition has become a research hotspot in affective computing and human-computer interaction. Among various emotion recognition technologies, electroencephalography (EEG), as a non-invasive technique, exhibits unique advantages. Unlike external behavioral signals such as facial expressions, speech, and body movements, which are easily controlled or masked, EEG can directly capture the neural electrical activity of the brain during emotional processing, thus providing a more objective assessment of emotional states, less susceptible to the influence of the subject's subjective will. Simultaneously, EEG possesses millisecond-level high temporal resolution, enabling real-time tracking of dynamic changes in emotional states, which is of significant value in revealing the mechanisms of emotion generation, development, and regulation.

[0003] In recent years, the rapid development of deep learning technology has driven significant progress in the field of EEG emotion recognition. Among related technologies, neural networks such as convolutional neural networks (CNNs) are used to directly learn temporal or spatial feature representations from raw EEG signals, avoiding the tedious manual feature design of traditional methods, thus significantly improving the efficiency and quality of feature extraction. Recurrent neural networks (RNNs) and their variants, especially Long Short-Term Memory networks (LSTMs) and gated recurrent units (GRUs), with their recursive structures and gating mechanisms, can effectively capture long-term dependencies in EEG signals, helping to model the dynamic evolution of emotions over time.

[0004] While deep learning has significantly improved the performance of EEG emotion recognition, these supervised learning-based methods generally face the problem of relying on large amounts of labeled data. EEG data acquisition requires specialized equipment, controlled experimental environments, and significant time investment. Furthermore, emotion labeling depends on subject-specific reports or expert evaluations, introducing a degree of uncertainty, making the acquisition of high-quality labeled data extremely costly. In small sample sizes, deep models are prone to overfitting the training data, resulting in poor generalization ability. A more prominent issue is the significant heterogeneity among publicly available datasets due to differences in EEG acquisition equipment, electrode configurations, and durations among different research institutions. This makes it difficult to directly transfer models trained on one dataset to other datasets. In other words, most existing EEG emotion recognition models are designed for single datasets, generally lacking cross-dataset generalization ability, leading to high costs and long deployment cycles in practical applications. For example, a model trained on 64-channel data is difficult to directly apply to 32-channel or 128-channel acquisition systems because the input dimension mismatch causes the model to malfunction. Additionally, existing EEG emotion recognition methods often uniformly model the spatiotemporal features of emotions, resulting in insufficient feature robustness. Summary of the Invention

[0005] To address the aforementioned problems in existing technologies, this invention provides a brainwave emotion recognition method based on spatiotemporal mask contrast learning. The technical problem to be solved by this invention is achieved through the following technical solution: According to a first aspect of the present invention, a brainwave emotion recognition method based on spatiotemporal mask contrast learning is provided, the method comprising: Channel configuration standardization preprocessing was performed on EEG signals from multiple EEG emotion datasets to obtain standardized EEG signals. A dual-stream encoder is used to extract spatial and temporal features from standardized EEG signals, and the extracted spatial and temporal representations are fused to obtain a global spatiotemporal representation. This dual-stream encoder is based on a model obtained after pre-training using spatiotemporal mask comparison. The global spatiotemporal representation is predicted using the classification head module to obtain the predicted sentiment category.

[0006] In one embodiment of the present invention, the dual-stream encoder includes an input embedding layer, a four-dimensional spatiotemporal position coding layer, a spatial flow sparse attention module, a temporal flow sparse attention module, and a spatiotemporal feature fusion layer. The dual-stream encoder extracts spatial and temporal features from the standardized EEG signal, and fuses the extracted spatial and temporal representations to obtain a global spatiotemporal representation, including: The input embedding layer performs time segmentation and local feature extraction on the standardized EEG signal, and outputs segment embeddings. Using the four-dimensional spatiotemporal location coding layer, spatial and temporal location information is added to the segment embedding, and location-aware segment embedding is output. The spatial flow sparse attention module extracts spatial features from the embedded position-aware segments, models the spatial dependencies between different electrodes at the same time, and outputs a spatial flow representation; the temporal flow sparse attention module extracts temporal features from the embedded position-aware segments, models the temporal dependencies of the same electrode at different times, and outputs a temporal flow representation. The spatiotemporal feature fusion layer integrates the spatial flow representation and the temporal flow representation to output a global spatiotemporal representation.

[0007] In one embodiment of the present invention, the step of performing time segmentation and local feature extraction on the standardized EEG signal through the input embedding layer, and outputting segment embedding, includes: The standardized EEG signal is divided along the time dimension by the input embedding layer using a fixed-length non-overlapping sliding window to obtain several time segments. For each time segment, a multi-layer one-dimensional convolutional encoder is used to extract local temporal features and output segment embeddings.

[0008] In one embodiment of the present invention, the step of using the four-dimensional spatiotemporal location coding layer to add spatial and temporal location information to the segment embedding and outputting a location-aware segment embedding includes: The four-dimensional spatiotemporal position encoding layer is used to generate spatial position information based on the three-dimensional Cartesian coordinates of each channel electrode on the scalp, and to generate temporal position information based on the one-dimensional time index of each segment. The spatial location information and the temporal location information are concatenated to obtain four-dimensional spatiotemporal location information; and the four-dimensional spatiotemporal location information is mapped to the feature dimension through a fully connected layer to obtain the location embedding. The location embedding and the fragment embedding are added element by element to obtain the location-aware fragment embedding.

[0009] In one embodiment of the present invention, the step of extracting spatial features from the position-aware segment embedding using the spatial flow sparse attention module, modeling the spatial dependency between different electrodes at the same time, and outputting a spatial flow representation; and extracting temporal features from the position-aware segment embedding using the temporal flow sparse attention module, modeling the temporal dependency of the same electrode at different times, and outputting a temporal flow representation, includes: The spatial flow sparse attention module is used to reassemble the embedded position-aware segments into multiple independent sequences along the time dimension. Each sequence is input into a spatial flow Transformer encoder for parallel processing and output. The output sequences are then reassembled back into the original spatiotemporal structure to obtain a spatial flow representation. The spatial flow Transformer encoder employs a spatial sparse attention mechanism, which ensures that each channel only focuses on the adjacent channels of its own time slice. The temporal sparse attention module is used to reassemble the position-aware segments into multiple independent sequences along the channel dimension. Each sequence is input into a time-flow Transformer encoder for parallel processing and output. The output sequence is then reassembled back into the original spatiotemporal structure to obtain a time-flow representation. The time-flow Transformer encoder employs a time-sparse attention mechanism, which ensures that each time step only focuses on the adjacent time steps of its channel.

[0010] In one embodiment of the present invention, the spatiotemporal feature fusion layer integrates the spatial flow representation and the temporal flow representation to output a global spatiotemporal representation, including: The spatial flow representation and the temporal flow representation are spliced ​​together along the feature dimension using the spatiotemporal feature fusion layer, and then projected back to the original feature dimension through a fully connected layer linear transformation to obtain the global spatiotemporal representation.

[0011] In one embodiment of the present invention, the pre-training of the dual-stream encoder includes: Acquire training data samples, which include multiple EEG emotion datasets. These multiple EEG emotion datasets differ in several aspects, including channel configuration, duration, and subject population. The EEG signals in the training data samples are preprocessed with channel configuration standardization to obtain standardized EEG signal samples. The standardized EEG signal sample is randomly masked, and the randomly masked EEG signal sample is input into the dual-stream encoder to obtain the first sample global spatiotemporal representation. Using a pre-defined reconstruction head module, the original signal value of each masked location in the global spatiotemporal representation of the first sample is predicted, and the EEG mask reconstruction loss is calculated at the masked location using mean square error. The standardized EEG signal samples are mirrored and then input into the momentum encoder to obtain the sample spatial flow representation, the sample temporal flow representation, and the second sample global spatiotemporal representation. Based on the sample spatial flow representation, the sample temporal flow representation, and the second sample global spatiotemporal representation, comparative learning is performed and a multi-view comparative loss is output. The EEG mask reconstruction loss and the multi-view contrast loss are weighted and combined to obtain the total model loss. Based on the total model loss, the gradient descent method is used to iteratively optimize the parameters of the dual-stream encoder to obtain the trained dual-stream encoder.

[0012] In one embodiment of the present invention, the step of performing contrastive learning and outputting a multi-view contrastive loss based on the sample spatial flow representation, the sample temporal flow representation, and the sample second global spatiotemporal representation includes: Attention pooling is applied to the sample space flow representation in the channel dimension, and averaging is performed in the time dimension to obtain the sample space vector. Attention pooling is applied to the sample time flow representation in the time dimension, and averaging is performed in the channel dimension to obtain the sample time vector; The second global spatiotemporal representation is subjected to attention pooling in both the channel and time dimensions to obtain the second global vector. The first global vector of each sample in the training data sample is used as the query vector, and the sample time vector, sample space vector and second global vector are used as the key vector of the positive sample. The sample time vector, sample space vector and second global vector of other samples in the training data sample are used as the key vector of the negative sample. Comparative learning is performed to output multi-view contrast loss. The first global vector is the feature vector obtained by performing channel-dimensional and time-dimensional attention pooling on the first global spatiotemporal representation.

[0013] In one embodiment of the present invention, the method is applicable to a preset EEG emotion recognition model, the EEG emotion recognition model including a two-stream encoder and a classification head module, and after pre-training the two-stream encoder, further comprising: Collect and label EEG emotion data for the target scene, and fine-tune the EEG emotion recognition model for task adaptability to obtain an EEG emotion recognition model for a specific scene. According to a second aspect of the present invention, an electronic device is provided, the device comprising: One or more processors; A computer-readable medium configured to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the EEG emotion recognition method based on spatiotemporal mask contrast learning as described in any one of the first aspects.

[0014] According to a third aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, characterized in that, when the program is executed by a processor, it implements the EEG emotion recognition method based on spatiotemporal mask contrast learning as described in any one of the first aspects.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: The EEG emotion recognition method and electronic device based on spatiotemporal mask contrast learning provided in this invention first performs channel configuration standardization preprocessing on EEG signals from multiple EEG emotion datasets to obtain standardized EEG signals. Then, a dual-stream encoder extracts spatial and temporal features from the standardized EEG signals and fuses the extracted spatial and temporal representations to obtain a global spatiotemporal representation. The dual-stream encoder is a model obtained after pre-training based on spatiotemporal mask contrast. Next, a classification head module predicts the global spatiotemporal representation to obtain the predicted emotion category. The dual-stream encoder constructed in this invention, based on a model obtained after pre-training based on spatiotemporal mask contrast, can flexibly handle heterogeneous EEG data input, i.e., it can handle "multiple EEG emotion datasets." Regardless of changes in the number of channels or the length of the time series, the model can automatically adapt without redesign, improving cross-dataset generalization ability and reducing the deployment cycle and cost in new scenarios. Furthermore, the dual-stream encoder of this invention comprises two independent processing branches: a spatial stream and a temporal stream. The spatial stream extracts spatial features from standardized EEG signals, enabling the modeling of spatial dependencies between different electrodes (brain regions) at the same time, capturing brain region functional connectivity patterns, and learning emotion-related distributed brain network activation features. The temporal stream extracts temporal features from standardized EEG signals, enabling the modeling of temporal dependencies of the same electrode at different times, capturing the dynamic evolution of neural activity, learning the temporal process characteristics of emotions, and the changing trajectories of neural oscillations at different frequency bands. The two streams are optimized within their respective independent feature spaces, avoiding mutual interference and compromise between the two types of heterogeneous dependencies in unified modeling. This allows for a more comprehensive and accurate modeling of complex spatiotemporal dynamics related to emotions, extracting more robust deep features. Therefore, even when facing complex challenges in real-world applications such as noise interference, individual differences, and device variations, this invention maintains stable and reliable emotion recognition performance, providing feasibility and reliability assurance for the practical application of EEG emotion recognition technology.

[0016] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0017] Figure 1 A flowchart illustrating the steps of an EEG emotion recognition method based on spatiotemporal mask contrast learning, provided in an embodiment of the present invention; Figure 2 A schematic diagram of a standardized 60-channel EEG template provided in an embodiment of the present invention; Figure 3 A schematic diagram of the structure of a two-stream decoupled spatiotemporal sparse attention encoder model; Figure 4This is a schematic diagram of the dual-stream encoder's workflow; Figure 5 This is a schematic diagram of the EEG sentiment recognition method based on spatiotemporal mask comparison pre-training. Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0019] Example 1 Reference Figure 1 The diagram illustrates a flowchart of a brainwave emotion recognition method based on spatiotemporal mask contrast learning according to Embodiment 1 of the present invention, including the following steps: Step 101: Perform channel configuration standardization preprocessing on the EEG signals from multiple EEG emotion datasets to obtain standardized EEG signals.

[0020] The standardized channel configuration preprocessing described above can involve constructing a standardized channel template based on the international 10-20 system according to the target application scenario. This standardized channel template contains a preset number of standard electrode channels. For each dataset, alignment to this standard template is achieved by discarding extra channels and interpolating missing channels. Discarding extra channels means deleting electrode channels not in the standard template, and interpolating missing channels means estimating the signal at the location of missing electrodes using spherical spline interpolation. After this processing, all datasets are unified into a standardized channel data format, ensuring consistency in channel configuration across datasets from different sources.

[0021] For example, the aforementioned EEG emotion datasets may include the SEED, SEED-IV, SEED-SD, EmoEEG-MC, SEED-FRA, and SEED-GER datasets. To address the differences in channel configurations across different datasets, a standardized 60-channel template based on the international 10-20 system can be constructed. The detailed layout of this channel template is as follows... Figure 2 As shown. For each dataset, alignment to the template is achieved by discarding extra channels and interpolating missing channels. For missing standard electrode locations, a spherical spline interpolation method is used to model the scalp as a sphere, and the missing electrode locations are then interpolated. The potential estimate is a weighted combination of existing channel signals, which can be calculated using the following formula:

[0022] in Indicates the number of available electrodes. It is an electrode In time The measured potential, The interpolation weights are determined by the proximity of the electrode space; this allows all datasets to be standardized into a 60-channel data format, resulting in a standardized EEG signal.

[0023] Step 102: Using a dual-stream encoder, spatial and temporal features are extracted from the standardized EEG signal, and the extracted spatial and temporal features are fused to obtain a global spatiotemporal representation.

[0024] The aforementioned dual-stream encoder is a model obtained after pre-training based on spatiotemporal mask contrast. It can include two independent processing branches: spatial stream and temporal stream. The spatial stream extracts spatial features from the standardized EEG signal, which can model the spatial dependencies between different electrodes (brain regions) at the same time, capture brain region functional connectivity patterns, learn emotion-related distributed brain network activation features, and obtain spatial stream representations. The temporal stream extracts temporal features from the standardized EEG signal, which can model the temporal dependencies of the same electrode at different times, capture the dynamic evolution of neural activity, learn the temporal process features of emotion and the changing trajectories of neural oscillations in different frequency bands, and obtain temporal stream representations.

[0025] Specifically, refer to Figure 3 , Figure 3 This is a schematic diagram of the structure of a two-stream decoupled spatiotemporal sparse attention encoder model (hereinafter referred to as the two-stream encoder). The two-stream encoder includes an input embedding layer, a four-dimensional spatiotemporal position encoding layer, a spatial flow sparse attention module, a temporal flow sparse attention module, and a spatiotemporal feature fusion layer. After training the two-stream encoder, spatial and temporal features are extracted from the standardized EEG signal, and the extracted spatial and temporal representations are fused to obtain a global spatiotemporal representation. Figure 4 The steps shown are to be followed, refer to [reference]. Figure 4 Here is a schematic diagram of the dual-stream encoder workflow: Step 1021: Through the input embedding layer, the standardized EEG signal is divided into time segments and local features are extracted, and the segment embedding is output.

[0026] Specifically, the standardized EEG signal can be divided into several time segments along the time dimension using a fixed-length non-overlapping sliding window through an input embedding layer. Then, a multi-layer one-dimensional convolutional encoder is used to extract local temporal features for each time segment, and the segment embedding is output. The one-dimensional convolutional encoder sequentially passes the input standardized EEG signal through a one-dimensional convolutional layer, group normalization, and the GELU activation function before outputting the signal.

[0027] For example, let the standardized EEG sample be... ,in Number of electrode channels (in this embodiment) =60), The number of time points. When the input embedding layer performs time segmentation and local feature extraction on the standardized EEG signal, it first uses a length of [number] along the time dimension. The non-overlapping sliding window is used to divide the L time points of each channel into... Each time segment contains [number] time segments. The algorithm iterates through consecutive time points. Next, a convolutional segment encoder is used to extract the local temporal features of each segment. This encoder consists of three stacked one-dimensional convolutional encoders, each containing a one-dimensional convolutional layer, a group normalization layer, and a GELU activation function layer. After processing by the convolutional segment encoder, the output segment embedding is obtained. , where D represents the feature dimension.

[0028] Step 1022: Using a four-dimensional spatiotemporal location coding layer, add spatial and temporal location information to the fragment embedding and output the location-aware fragment embedding.

[0029] Specifically, a four-dimensional spatiotemporal position encoding layer can be used to generate spatial position information based on the three-dimensional Cartesian coordinates of each channel electrode on the scalp, and temporal position information based on the one-dimensional time index of each segment. Then, the spatial position information and temporal position information are concatenated to obtain four-dimensional spatiotemporal position information, and the four-dimensional spatiotemporal position information is mapped to the feature dimension through a fully connected layer to obtain the position embedding. Then, the obtained position embedding is added element by element to the above segment embedding to obtain the position-aware segment embedding.

[0030] For example, we can first construct a four-dimensional spacetime location tensor. For the first The first channel The location information of each segment The three-dimensional Cartesian coordinates of this channel electrode in a normalized 10-20 system With the time index of this segment splicing together, that is After normalizing the four-dimensional spatiotemporal location tensor, it is projected onto the feature dimension through a two-layer fully connected network. To obtain the four-dimensional position embedding Finally, the four-dimensional position embedding and the fragment embedding are added element-wise to obtain the position-aware fragment embedding. .

[0031] Step 1023: Using the spatial flow sparse attention module, spatial features are extracted from the position-aware segment embeddings to model the spatial dependencies between different electrodes (brain regions) at the same time and output spatial flow representations; and using the temporal flow sparse attention module, temporal features are extracted from the position-aware segment embeddings to model the temporal dependencies of the same electrode at different times and output temporal flow representations.

[0032] Specifically, a spatial flow sparse attention module can be used to reassemble the embedding of position-aware segments along the time dimension into multiple independent sequences, where each sequence represents the features of all channels at the same time. These sequences are then fed into a spatial flow Transformer encoder for parallel processing and output, and the output sequences are reassembled back into the original spatiotemporal structure to obtain the spatial flow representation. Similarly, a temporal flow sparse attention module can be used to reassemble the embedding of position-aware segments along the channel dimension into multiple independent sequences, where each sequence represents the features of the same channel at all times. These sequences are then fed into a temporal flow Transformer encoder for parallel processing and output, and the output sequences are reassembled back into the original spatiotemporal structure to obtain the temporal flow representation. The spatial flow Transformer encoder described above can employ a spatial sparse attention mechanism, ensuring that each channel only focuses on the adjacent channels of its own time slice; similarly, the temporal flow Transformer encoder can also employ a temporal sparse attention mechanism, ensuring that each time step only focuses on the adjacent time steps of its own channel.

[0033] For example, when extracting spatial features from position-aware fragment embeddings using a spatial flow sparse attention module, the position-aware fragment embeddings can first be reassembled along the time dimension. Where B is the batch size, the T time steps of the B samples are expanded separately, and each time step forms an independent sequence composed of C channel features. Then, this... The spatial stream Transformer encoder processes a sequence of input spatial streams. Layer Transformer blocks, each layer containing a multi-head self-attention mechanism and a feedforward neural network, for the th layer... Layer, Input (Initial time) After layer normalization, it is linearly projected into query (Q), key (K), and value (V). It has... The multi-head self-attention mechanism of a large number can be calculated as follows:

[0034] in It is the output projection matrix, for each attention head. The calculation is as follows:

[0035] in The dimension of the key is represented. A spatially sparse attention pattern is adopted in the spatial flow, which reduces computational complexity by using structured sparse connections so that each channel only focuses on its spatially neighboring channels in the same time step.

[0036] The output of multi-head self-attention is added to the input via a residual connection:

[0037] A feedforward network is then applied, consisting of two linear transformations and a nonlinear activation (such as GELU). This produces the... The final output of the layer:

[0038] go through After processing by the Transformer layer, the output is reconstructed back to the original spatiotemporal structure to obtain a spatial flow representation. .

[0039] When extracting temporal features from position-aware fragment embeddings using a temporal sparse attention module to model the temporal dependency of the same electrode at different times, the position-aware fragment embeddings are first reassembled along the channel dimension. The C channels of the B samples are expanded separately, with each channel forming an independent sequence consisting of T time-step features. These B·C sequences are then input in parallel into a temporal-stream Transformer encoder for processing. The temporal-stream Transformer encoder contains... The Transformer layer, with the same structure as the spatial flow Transformer, includes a multi-head self-attention mechanism and a feedforward neural network. In the temporal flow, a temporally sparse attention pattern is employed, using structured sparse connections to ensure that each time step only focuses on its temporally neighboring time steps within the same channel. The computation process for the temporal flow is the same as that for the spatial flow, only the sequence dimension being processed differs. After processing by the Transformer layer, the output is reassembled back into the original spatiotemporal structure to obtain a temporal flow representation. .

[0040] Step 1024: Integrate spatial flow representation and temporal flow representation using the spatiotemporal feature fusion layer to output a global spatiotemporal representation.

[0041] Specifically, a spatiotemporal feature fusion layer can be used to concatenate spatial flow representations and temporal flow representations along the feature dimension, and then project them back to the original feature dimension through a fully connected layer linear transformation to obtain a global spatiotemporal representation, which serves as the final output of the dual-stream encoder.

[0042] For example, spatial flow can be characterized by the following expression. and time flow representation By concatenating along the feature dimensions and projecting the concatenated representation back to the original feature dimensions through a linear transformation, a fused global spatiotemporal representation is obtained:

[0043] in and Represents learnable weights and biases. Global spatiotemporal representation. As the final output of the dual-stream encoder, it is used for subsequent tasks.

[0044] Step 103: Use the classification head module to predict the global spatiotemporal representation and obtain the predicted sentiment category.

[0045] The classification head module can be composed of three fully connected layers, specifically consisting of a first fully connected layer, a first ELU activation function layer, a first Dropout layer, a second fully connected layer, a second ELU activation function layer, a second Dropout layer, and a third fully connected layer connected sequentially.

[0046] The first fully connected layer flattens the input features from C×T×D dimensions and maps them to the first hidden dimension. After ELU activation and Dropout regularization, the second fully connected layer maps the first hidden dimension to the second hidden dimension. After ELU activation and Dropout again, the third fully connected layer maps the second hidden dimension to the number of target sentiment categories.

[0047] The EEG emotion recognition method based on spatiotemporal mask contrast learning provided in this invention first performs channel configuration standardization preprocessing on EEG signals from multiple EEG emotion datasets to obtain standardized EEG signals. Then, a dual-stream encoder extracts spatial and temporal features from the standardized EEG signals and fuses the extracted spatial and temporal representations to obtain a global spatiotemporal representation. The dual-stream encoder is a model obtained through spatiotemporal mask contrast pre-training. Finally, a classification head module predicts the global spatiotemporal representation to obtain the predicted emotion category. The dual-stream encoder constructed in this invention, based on a model obtained through spatiotemporal mask contrast pre-training, can flexibly handle heterogeneous EEG data input, i.e., it can handle "multiple EEG emotion datasets." Regardless of changes in the number of channels or the length of the time series, the model can automatically adapt without redesign, improving cross-dataset generalization ability and reducing the deployment cycle and cost in new scenarios. Furthermore, the dual-stream encoder of this invention comprises two independent processing branches: a spatial stream and a temporal stream. The spatial stream extracts spatial features from standardized EEG signals, enabling the modeling of spatial dependencies between different electrodes (brain regions) at the same time, capturing brain region functional connectivity patterns, and learning emotion-related distributed brain network activation features. The temporal stream extracts temporal features from standardized EEG signals, enabling the modeling of temporal dependencies of the same electrode at different times, capturing the dynamic evolution of neural activity, learning the temporal process characteristics of emotions, and the changing trajectories of neural oscillations at different frequency bands. The two streams are optimized within their respective independent feature spaces, avoiding mutual interference and compromise between the two types of heterogeneous dependencies in unified modeling. This allows for a more comprehensive and accurate modeling of complex spatiotemporal dynamics related to emotions, extracting more robust deep features. Therefore, even when facing complex challenges in real-world applications such as noise interference, individual differences, and device variations, this invention maintains stable and reliable emotion recognition performance, providing feasibility and reliability assurance for the practical application of EEG emotion recognition technology.

[0048] Example 2 The pre-training of the dual-stream encoder in this invention will be described in detail below, with reference to... Figure 5 This is a schematic diagram of the EEG sentiment recognition method based on spatiotemporal mask comparison pre-training.

[0049] In this embodiment, the pre-training of the two-stream encoder includes: acquiring training data samples; and then using the training data samples to perform spatiotemporal mask comparison pre-training on the two-stream encoder.

[0050] The training data samples include multiple EEG emotion datasets, which differ in several aspects, including channel configuration, duration, and subject population. It should be noted that the datasets used for pre-training are not limited to the six datasets (SEED, SEED-IV, SEED-SD, EmoEEG-MC, SEED-FRA, and SEED-GER) used in this example. Any EEG emotion dataset can be used, such as publicly available datasets like DEAP, DREAMER, AMIGOS, MAHNOB-HCI, SEED-V, SEED-VII, and FACED, or privately collected EEG emotion datasets. The number of pre-training datasets is not limited to six; fewer (e.g., 3-5) or more (e.g., 10 or more) datasets can be used for pre-training. The choice of dataset number can be flexibly adjusted according to computing resources and application requirements.

[0051] Specifically, when using training data samples to perform spatiotemporal mask contrast pre-training on the two-stream encoder, the reference... Figure 5 This can be achieved through the following steps: First, the EEG signals in the training data samples are preprocessed by channel configuration standardization to obtain standardized EEG signal samples.

[0052] In this embodiment, the number of channels in the standardized channel template is not limited to 60 channels. Different sizes of templates can be selected according to the application scenario, such as a 32-channel standard template (suitable for portable devices), a 64-channel standard template, or a 128-channel standard template (suitable for high-density EEG). The standardized channel template is not limited to the international 10-20 system; it can also be constructed based on the 10-10 system (higher density), the 10-5 system (highest density), or other standard electrode positioning systems.

[0053] Secondly, the standardized EEG signal samples are randomly masked, and the randomly masked EEG signal samples are input into a dual-stream encoder to obtain the first sample global spatiotemporal representation.

[0054] The random masking process involves independently masking the input EEG signal at each time step along the time dimension. Channels within that time step are randomly selected based on a preset masking probability, and their locations are replaced with learnable mask markers. The signal processed by the random masking is then input into a dual-stream encoder to obtain the first sample's global spatiotemporal representation.

[0055] Next, using the preset reconstruction head module, the original signal value of each masked location in the global spatiotemporal representation of the first sample is predicted, and the EEG mask reconstruction loss is calculated using mean square error at the masked location.

[0056] A reconstruction head module consisting of fully connected layers is set at the top of the model. The global spatiotemporal representation of the first sample is input into the reconstruction head to predict the original signal value at each masked location. The mean squared error is used to calculate the reconstruction loss at the masked location. At the same time, channel-dimensional attention pooling and time-dimensional attention pooling operations are applied sequentially to the global spatiotemporal representation of the first sample to obtain the first global vector ( Figure 5 (represented by the first global representation), which is subsequently used as the query vector in contrastive learning.

[0057] Then, the standardized EEG signal samples are mirrored and the mirrored EEG signal samples are input into the momentum encoder to obtain the sample spatial flow representation, the sample temporal flow representation, and the second sample global spatiotemporal representation.

[0058] The mirror masking process involves inverting a random mask, so that signal segments visible in the random mask are masked in the mirror mask, and segments masked in the random mask become visible in the mirror mask. This ensures that the information contained in the two masked views is completely complementary and covers the original signal entirely. A momentum encoder is constructed, sharing the same architecture as the aforementioned two-stream encoder model. Its parameters are updated using an exponential moving average of the model parameters to maintain smooth parameter changes and stable feature extraction. The mirror-masked EEG signal samples are input into the momentum encoder. The spatial flow module of the momentum encoder outputs a spatial flow representation of the samples, the temporal flow module outputs a temporal flow representation of the samples, and the fusion layer outputs a second global spatiotemporal representation of the samples. The spatial flow representation, temporal flow representation, and second global spatiotemporal representation of the samples output by the momentum encoder are compressed into a fixed-dimensional feature vector through a pooling operation.

[0059] Next, based on the sample spatial flow representation, sample temporal flow representation, and second sample global spatiotemporal representation, comparative learning is performed and a multi-view comparative loss is output.

[0060] Specifically, attention pooling is applied to the sample space flow representation in the channel dimension, and averaging is performed in the time dimension to obtain the sample space vector. Figure 5 The second spatial representation is used to represent the sample time flow; attention pooling is applied to the time dimension of the sample time flow representation, and averaging is performed in the channel dimension to obtain the sample time vector. Figure 5The second global vector is obtained by performing channel- and time-dimensional attention pooling on the second sample's global spatiotemporal representation. Then, the first global vector of each sample in the training data is used as the query vector, and the sample time vector, sample space vector, and second global vector are used as the key vectors for positive samples. The sample time vector, sample space vector, and second global vector of other samples in the training data are used as the key vectors for negative samples. Comparative learning is then performed to output the multi-view contrastive loss. The multi-view contrastive loss is then defined as the query vector, and the sum of the contrastive losses with the spatial key vector, temporal key vector, and global key vector is calculated. Each contrastive loss can be calculated using the InfoNCE loss function.

[0061] Finally, the EEG mask reconstruction loss and the multi-view contrast loss are weighted and combined to obtain the total model loss. Based on this total model loss, the gradient descent method is used to iteratively optimize the parameters of the two-stream encoder to obtain the trained two-stream encoder.

[0062] The pre-training process described above combines two training strategies: EEG masking reconstruction and multi-view contrastive learning. The EEG masking reconstruction loss and multi-view contrastive learning loss are output separately. These two losses are weighted and combined to obtain the total loss. Gradient descent is used to iteratively optimize the model parameters, resulting in a pre-trained model. The EEG masking reconstruction task randomly masks standardized EEG signal samples, training the model to reconstruct the masked EEG signal segments, enabling the model to learn local spatiotemporal features. Multi-view contrastive learning mirrors standardized EEG signal samples, using contrastive learning to constrain the consistency of the global representation of the same sample in the random mask view with the temporal, spatial, and global representations in the mirror mask view, enabling the model to extract multi-view collaborative complementary features. In other words, the EEG masking reconstruction task uses randomly masked input signals to train the model to reconstruct the masked segments, learning local spatiotemporal patterns; the multi-view contrastive learning constructs complementary views through a mirror masking strategy and constrains the consistency of the representation of the same sample under different views, learning global discriminative features. The two tasks output reconstruction and contrastive losses respectively, which are weighted and combined to obtain the total loss for joint optimization.

[0063] For example, when performing random masking on standardized EEG signal samples in an EEG masking reconstruction task, a time-dimensional channel mask can be constructed. The mask is sampled independently along the time dimension. For each time step, the mask probability is calculated according to a Bernoulli distribution. Randomly select the channels that need to be masked from C channels. The set Ω of the masked positions is defined as:

[0064] in Indicates the first The first channel The segment was masked. The visible segments are represented by a mask. All masked locations are replaced with learnable mask markers. These mask markers are combined with the visible segments and input into a two-stream encoder, outputting a first-sample global spatiotemporal representation. A reconstruction head module, consisting of fully connected layers, is placed at the top of the model. Using the first-sample global spatiotemporal representation as input, it predicts the original signal value at each masked location. The reconstruction head outputs the predicted signal. The mean square error between the predicted and true signals is calculated only at the masked locations. The EEG masking reconstruction loss is defined as:

[0065] Simultaneously, to generate the query vector for contrastive learning, the global spatiotemporal representation of the first sample is subjected to dimensionality reduction pooling. First, attention pooling is applied along the channel dimension, weighting and aggregating C channels using learnable attention weights. Then, attention pooling is applied along the time dimension, weighting and aggregating T time steps to obtain the global query vector (i.e., the first global vector). .

[0066] Multi-view contrastive learning constructs a mirrored view that is completely complementary to the information in the random masked view through a mirror masking strategy. The random mask... Reverse element by element to generate a mirror mask This ensures that segments visible in a random mask are masked in a mirror mask, and segments masked in a random mask are visible in a mirror mask. A mirror mask view is represented as:

[0067] Build a momentum encoder that shares the exact same architecture as the dual-stream encoder. The parameters of the momentum encoder are... Through the main encoder parameters The exponential moving average is updated slowly, and the update rules are as follows: ,in This refers to the momentum coefficient. The mirror-masked EEG signal sample is input into the momentum encoder, where representations are extracted from the spatial flow, temporal flow, and fusion layer, respectively. Then, three types of key vectors are generated using different pooling strategies. Spatial flow representation... Attention pooling is applied in the channel dimension, followed by average pooling in the time dimension to obtain the spatial key (sample space vector). Representation of time flow Attention pooling is applied in the time dimension, followed by average pooling in the channel dimension to obtain the time key (sample time vector). Global spatiotemporal representation of the second sample Attention pooling is applied sequentially along the channel and time dimensions to obtain the global key (the second global vector). The multi-view contrast loss is defined as the sum of the contrast losses between the query vector and the three types of key vectors.

[0068] Each comparison term is calculated using the InfoNCE loss function. For each sample in the batch, its query vector and its corresponding key vector are taken as positive sample pairs, and the query vector and the key vectors of other samples in the batch are taken as negative sample pairs. The InfoNCE loss is defined as:

[0069] in Indicates batch size, Indicates temperature parameter, This represents the query vector for the i-th sample (i.e., the first global vector). ), This represents the key vector corresponding to the i-th sample (depending on the specific comparison terms, it can correspond to the sample space vector). Sample time vector Or the second global vector ).

[0070] The spatiotemporal mask contrast pre-training method combines the EEG mask reconstruction loss and multi-view contrast loss into a total loss through weighted combination. The gradient descent method is used to iteratively optimize the model parameters. It can be jointly trained on the aforementioned six EEG emotion datasets (SEED, SEED-IV, SEED-SD, EmoEEG-MC, SEED-FRA, and SEED-GER). After multiple training epochs, a pre-trained dual-stream encoder model is obtained.

[0071] In this embodiment, the EEG emotion recognition model includes a dual-stream encoder and a classification head module. That is, by connecting the output of the pre-trained dual-stream encoder obtained above to the classification head module, the EEG emotion recognition model is formed.

[0072] The aforementioned classification head module can be composed of three fully connected network layers. Specifically, the first fully connected layer flattens the input features from C×T×D dimensions and maps them to the first hidden dimension. After ELU activation and Dropout regularization, the second fully connected layer maps the first hidden dimension to the second hidden dimension. After ELU activation and Dropout again, the third fully connected layer maps the second hidden dimension to the target sentiment category number.

[0073] refer to Figure 5 In the upper right corner, after pre-training the dual-stream encoder, labeled EEG emotion data can be collected for the target scene, and the EEG emotion recognition model can be fine-tuned for task adaptability to obtain an EEG emotion recognition model for a specific scene.

[0074] This embodiment selects the SEED-VII and FACED datasets as target application scenarios for fine-tuning and testing, verifying the model's cross-scenario adaptability. For the SEED-VII dataset, a subject-dependent experimental setup was used. The data was divided into 281,679 1-second samples. For each session of 20 trials, the first 10 trials were used for training, the middle 5 for validation, and the last 5 for testing. For the FACED dataset, a subject-independent experimental setup was used. The data was divided into 10,332 10-second samples. A cross-subject protocol was adopted, assigning subjects 1-80 to the training set, subjects 81-100 to the validation set, and subjects 101-123 to the test set. For each labeled EEG emotion sample in the training set, the EEG emotion recognition model was input to generate an emotion category prediction probability. The model is optimized using the cross-entropy loss function, which is defined as:

[0075] in The total number of training samples, Indicates the number of emotion categories. The true labels are encoded using one-hot encoding. Gradient descent is used to update the parameters of the EEG emotion recognition model. During fine-tuning, the model performance is evaluated on the validation set after each training epoch. The accuracy on the validation set is recorded, and the checkpoint with the highest accuracy is selected as the final EEG emotion recognition model. Validation set selection avoids overfitting on the test set and ensures the model's generalization ability. In this embodiment, fine-tuning is performed for 100 epochs on the SEED-VII and FACED datasets, with a batch size of 64 for both.

[0076] In this embodiment, a finely tuned EEG emotion recognition model tailored to a specific scenario is used to evaluate emotion recognition performance on a test set: The EEG emotion recognition model is obtained by inputting the EEG emotion signals to be identified from the test set and outputting the predicted emotion category. The model's recognition performance on the SEED-VII and FACED datasets is calculated using a balanced accuracy and kappa coefficient as evaluation metrics. Experimental results show that the method of this invention achieves excellent emotion recognition performance on the test sets of both target scenarios, and significantly improves the recognition accuracy compared to other existing methods.

[0077] Table 1. Emotion Recognition Performance of the Invention

[0078] It is easy to understand that after fine-tuning the EEG emotion recognition model for task adaptability, a more targeted EEG emotion recognition model for specific scenarios can be obtained, making the model's prediction ability more accurate.

[0079] This invention designs a dual-stream decoupled spatiotemporal sparse attention encoder model, employing a decoupled design that processes spatial and temporal streams in parallel. It models inter-channel spatial dependencies and temporal series dependencies separately, complemented by a four-dimensional spatiotemporal position encoding layer, and finally integrates spatiotemporal features through a fusion layer. This invention proposes a spatiotemporal mask contrastive pre-training strategy, particularly extending contrastive learning to multiple spatiotemporal perspectives. A random mask view is used to generate a global query vector via the main encoder, while a mirrored mask view is used to extract representations from the spatial stream, temporal stream, and fusion layer via a momentum encoder. Spatial key vectors, temporal key vectors, and global key vectors are generated through different pooling methods, achieving contrastive learning of global queries and multi-perspective keys, rather than the traditional single global contrastive approach.

[0080] This invention pre-trains on multiple heterogeneous EEG emotion datasets. Channel configurations across these datasets are standardized and unified. After pre-training on multiple datasets in a unified format, a classification head is added to a small amount of labeled data related to the target scene for fine-tuning, enabling rapid adaptation.

[0081] The present invention also has the following beneficial effects: 1. Constructing a universal foundational model for EEG emotion to reduce deployment costs in new scenarios. This invention standardizes channel configuration across datasets from different sources using a channel configuration standardization method based on the international 10-20 system. Self-supervised pre-training is then performed on multiple heterogeneous EEG emotion datasets, enabling the model to learn consistent, universal emotion representations across datasets. When deployed in new application scenarios, only a small amount of labeled data is needed for fine-tuning to achieve good performance, eliminating the need for retraining from scratch or redesigning the model architecture, significantly shortening the deployment cycle and reducing data acquisition costs.

[0082] 2. Achieving multi-level feature learning. This invention enables the model to learn local spatiotemporal patterns and global discriminative features simultaneously by jointly optimizing mask reconstruction loss and multi-view contrast loss. Compared with existing single-objective learning methods, this invention achieves multi-level feature learning from local to global and from reconstruction to discrimination, improving the modeling ability and recognition accuracy of complex emotional patterns.

[0083] 3. Enhanced Discriminativity and Robustness of Emotion Representation. This invention employs a dual-stream decoupled spatiotemporal coding architecture, where the spatial and temporal streams independently model inter-channel spatial and temporal dependencies, avoiding feature compromise issues caused by unified modeling in existing technologies. Through a spatiotemporal sparse attention mechanism, each channel focuses only on its neighboring channels and neighboring time steps, eliminating cross-dimensional spurious dependencies and reducing computational complexity. Combined with a mirror mask multi-view contrastive learning strategy, key vectors are extracted from the spatial stream, temporal stream, and fusion layer and compared with the global query vector, constraining the consistency of representation of the same sample across the temporal, spatial, and global perspectives, enabling the model to learn multi-view collaborative and complementary discriminative features. Compared to existing single-global contrastive learning methods, this invention can more comprehensively capture the spatiotemporal features of emotion, maintaining stable recognition performance even in complex scenarios such as noise interference, individual differences, and device variations.

[0084] Example 3 This invention also provides an electronic device, such as... Figure 6 As shown, it includes a processor 301, a communication interface 302, a memory 303, and a communication bus 304, wherein the processor 301, the communication interface 302, and the memory 303 communicate with each other through the communication bus 304. Memory 303 is used to store computer programs; When processor 301 executes program 305 stored in memory 303, it performs the following steps: Channel configuration standardization preprocessing is performed on EEG signals from multiple EEG emotion datasets to obtain standardized EEG signals. Spatial and temporal features are extracted from the standardized EEG signals using a two-stream encoder, and the extracted spatial and temporal representations are fused to obtain a global spatiotemporal representation. This two-stream encoder is a model obtained after pre-training based on spatiotemporal mask contrast. The global spatiotemporal representation is then used to predict the predicted emotion category using a classification head module.

[0085] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0086] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0087] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0088] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0089] The method provided in this invention can be applied to electronic devices. Specifically, the electronic device can be a desktop computer, a portable computer, a smart mobile terminal, a server, etc. No limitation is made herein; any electronic device that can implement this invention falls within the protection scope of this invention.

[0090] For the electronic device / storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and relevant details can be found in the description of the method embodiments.

[0091] It should be noted that the electronic device and storage medium in the embodiments of the present invention are respectively electronic devices and storage media that apply the above-mentioned EEG emotion recognition method based on spatiotemporal mask comparison learning. Therefore, all embodiments of the above-mentioned EEG emotion recognition method based on spatiotemporal mask comparison learning are applicable to the electronic device and storage medium, and can achieve the same or similar beneficial effects.

[0092] The terminal device provided by the embodiments of the present invention can display proper nouns and / or fixed phrases for users to select, thereby reducing user input time and improving user experience.

[0093] This terminal device exists in various forms, including but not limited to: (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and are primarily designed to provide voice and data communication. These terminals include smartphones (e.g., iPhones), multimedia phones, feature phones, and low-end phones.

[0094] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, such as the iPad.

[0095] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes audio and video players (such as iPods), handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0096] (4) Other electronic devices with data interaction functions.

[0097] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0098] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0099] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0100] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus (devices), or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects, all of which are collectively referred to herein as "modules" or "systems." Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The computer program may be stored / distributed in a suitable medium, provided with or as part of other hardware, or may take other distribution forms, such as via the Internet or other wired or wireless telecommunications systems.

[0101] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0102] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0103] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0104] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A brainwave emotion recognition method based on spatiotemporal mask contrast learning, characterized in that, The method includes: Channel configuration standardization preprocessing was performed on EEG signals from multiple EEG emotion datasets to obtain standardized EEG signals. The standardized EEG signal is subjected to spatial and temporal feature extraction using a dual-stream encoder. The extracted spatial and temporal features are then fused to obtain a global spatiotemporal representation. The dual-stream encoder is a model obtained after pre-training based on spatiotemporal mask comparison. The global spatiotemporal representation is predicted using the classification head module to obtain the predicted sentiment category.

2. The method according to claim 1, characterized in that, The dual-stream encoder includes an input embedding layer, a four-dimensional spatiotemporal position coding layer, a spatial flow sparse attention module, a temporal flow sparse attention module, and a spatiotemporal feature fusion layer. The dual-stream encoder extracts spatial and temporal features from the standardized EEG signal, and fuses the extracted spatial and temporal representations to obtain a global spatiotemporal representation, including: The input embedding layer performs time segmentation and local feature extraction on the standardized EEG signal, and outputs segment embeddings. Using the four-dimensional spatiotemporal location coding layer, spatial and temporal location information is added to the segment embedding, and location-aware segment embedding is output. The spatial flow sparse attention module extracts spatial features from the embedded position-aware segments, models the spatial dependencies between different electrodes at the same time, and outputs a spatial flow representation; the temporal flow sparse attention module extracts temporal features from the embedded position-aware segments, models the temporal dependencies of the same electrode at different times, and outputs a temporal flow representation. The spatiotemporal feature fusion layer integrates the spatial flow representation and the temporal flow representation to output a global spatiotemporal representation.

3. The method according to claim 2, characterized in that, The process of segmenting the standardized EEG signal into time segments and extracting local features through the input embedding layer, and outputting segment embeddings, includes: The standardized EEG signal is divided along the time dimension by the input embedding layer using a fixed-length non-overlapping sliding window to obtain several time segments. For each time segment, a multi-layer one-dimensional convolutional encoder is used to extract local temporal features and output segment embeddings.

4. The method according to claim 2, characterized in that, The process of adding spatial and temporal location information to the segment embedding using the four-dimensional spatiotemporal location encoding layer and outputting a location-aware segment embedding includes: The four-dimensional spatiotemporal position encoding layer is used to generate spatial position information based on the three-dimensional Cartesian coordinates of each channel electrode on the scalp, and to generate temporal position information based on the one-dimensional time index of each segment. The spatial location information and the temporal location information are concatenated to obtain four-dimensional spatiotemporal location information; and the four-dimensional spatiotemporal location information is mapped to the feature dimension through a fully connected layer to obtain the location embedding. The location embedding and the fragment embedding are added element by element to obtain the location-aware fragment embedding.

5. The method according to claim 2, characterized in that, The process involves extracting spatial features from the position-aware segment embedding using the spatial flow sparse attention module, modeling the spatial dependencies between different electrodes at the same time, and outputting a spatial flow representation; and extracting temporal features from the position-aware segment embedding using the temporal flow sparse attention module, modeling the temporal dependencies of the same electrode at different times, and outputting a temporal flow representation, including: The spatial flow sparse attention module is used to reassemble the embedded position-aware segments into multiple independent sequences along the time dimension. Each sequence is input into a spatial flow Transformer encoder for parallel processing and output. The output sequences are then reassembled back into the original spatiotemporal structure to obtain a spatial flow representation. The spatial flow Transformer encoder employs a spatial sparse attention mechanism, which ensures that each channel only focuses on the adjacent channels of its own time slice. The temporal sparse attention module is used to reassemble the position-aware segments into multiple independent sequences along the channel dimension. Each sequence is input into a time-flow Transformer encoder for parallel processing and output. The output sequence is then reassembled back into the original spatiotemporal structure to obtain a time-flow representation. The time-flow Transformer encoder employs a time-sparse attention mechanism, which ensures that each time step only focuses on the adjacent time steps of its channel.

6. The method according to claim 2, characterized in that, The spatiotemporal feature fusion layer integrates the spatial flow representation and the temporal flow representation to output a global spatiotemporal representation, including: The spatial flow representation and the temporal flow representation are spliced ​​together along the feature dimension using the spatiotemporal feature fusion layer, and then projected back to the original feature dimension through a fully connected layer linear transformation to obtain the global spatiotemporal representation.

7. The method according to claim 1, characterized in that, Pre-training of the dual-stream encoder includes: Acquire training data samples, which include multiple EEG emotion datasets. These multiple EEG emotion datasets differ in several aspects, including channel configuration, duration, and subject population. The EEG signals in the training data samples are preprocessed with channel configuration standardization to obtain standardized EEG signal samples; The standardized EEG signal sample is randomly masked, and the randomly masked EEG signal sample is input into the dual-stream encoder to obtain the first sample global spatiotemporal representation. Using a pre-defined reconstruction head module, the original signal value of each masked location in the global spatiotemporal representation of the first sample is predicted, and the EEG mask reconstruction loss is calculated at the masked location using mean square error. The standardized EEG signal samples are mirrored and then input into the momentum encoder to obtain the sample spatial flow representation, the sample temporal flow representation, and the second sample global spatiotemporal representation. Based on the sample spatial flow representation, the sample temporal flow representation, and the second sample global spatiotemporal representation, comparative learning is performed and a multi-view comparative loss is output. The EEG mask reconstruction loss and the multi-view contrast loss are weighted and combined to obtain the total model loss. Based on the total model loss, the gradient descent method is used to iteratively optimize the parameters of the dual-stream encoder to obtain the trained dual-stream encoder.

8. The method according to claim 7, characterized in that, The process of performing contrastive learning and outputting a multi-view contrastive loss based on the sample spatial flow representation, the sample temporal flow representation, and the sample second global spatiotemporal representation includes: Attention pooling is applied to the sample space flow representation in the channel dimension, and averaging is performed in the time dimension to obtain the sample space vector. Attention pooling is applied to the sample time flow representation in the time dimension, and averaging is performed in the channel dimension to obtain the sample time vector; The second global spatiotemporal representation is subjected to attention pooling in both the channel and time dimensions to obtain the second global vector. The first global vector of each sample in the training data sample is used as the query vector, and the sample time vector, sample space vector and second global vector are used as the key vector of the positive sample. The sample time vector, sample space vector and second global vector of other samples in the training data sample are used as the key vector of the negative sample. Comparative learning is performed to output multi-view contrast loss. The first global vector is the feature vector obtained by performing channel-dimensional and time-dimensional attention pooling on the first global spatiotemporal representation.

9. The method according to claim 7, characterized in that, The method is applicable to a pre-defined EEG emotion recognition model, which includes a two-stream encoder and a classification head module. After pre-training the two-stream encoder, the method further includes: Collect and label EEG emotion data for the target scene, and fine-tune the EEG emotion recognition model for task adaptability to obtain an EEG emotion recognition model for a specific scene.

10. An electronic device, characterized in that, The device includes: One or more processors; A computer-readable medium configured to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the EEG emotion recognition method based on spatiotemporal mask contrast learning as described in any one of claims 1-9.

Images