Weakly supervised eeg emotion recognition method and system based on spatiotemporal multiple instance learning

Abstract

This invention discloses a weakly supervised EEG emotion recognition method and system based on spatiotemporal multi-instance learning. The method includes the following steps: preprocessing the acquired EEG signal to obtain a differential entropy feature matrix; passing the matrix through a Time-Inception module to obtain a spatial instance sequence; passing the matrix through a Space-Inception module to obtain a temporal instance sequence; passing the temporal and spatial instance sequences through corresponding time-aware MIL pooling branches and space-aware MIL pooling branches to obtain temporal and spatial packet features, respectively; inputting these features into a linear classifier to obtain the emotion category probability distributions for the temporal and spatial branches; training an emotion category prediction model based on a total loss function; and outputting the final emotion category prediction result. This invention can simultaneously model the temporal dynamics and spatial topological features of EEG signals, effectively mitigating the noise problem caused by coarse labels and improving the accuracy and robustness of cross-subject emotion recognition.

Description

Technical Field

[0001] This invention relates to the field of emotion recognition technology, specifically to a weakly supervised EEG emotion recognition method and system based on spatiotemporal multi-instance learning. Background Technology

[0002] Emotion recognition based on electroencephalography (EEG) has broad application prospects in fields such as mental health monitoring and human-computer interaction due to its advantages such as strong signal objectivity and difficulty in spoofing. However, EEG signals are characterized by high dimensionality, non-stationarity, and low signal-to-noise ratio, and emotion-related neural activity is transient and region-specific, appearing only in specific time windows and brain regions.

[0003] Most existing EEG emotion recognition methods employ a fully supervised learning paradigm, uniformly assigning coarse trial labels to all time segments and channels. This approach implicitly assumes that emotional responses are uniform in time and space, which contradicts neurophysiological realities and introduces significant label noise, resulting in poor model generalization ability.

[0004] Multiple instance learning (MIL) provides an effective approach to address the aforementioned weak supervision problem. It treats each trial as a bag and time segments or channels as instances, learning instance importance weights using only bag-level labels. However, most existing EEG-MIL methods are single-branch temporal MIL frameworks, treating each time segment as a spatially uniform whole and ignoring the spatial heterogeneity between brain regions. This makes it impossible to simultaneously capture the dynamic temporal evolution of emotions and spatial co-occurrences. Furthermore, the permutation-invariant pooling in existing multiple instance learning (MIL) methods destroys the spatiotemporal location information of instances, further limiting model performance. Summary of the Invention

[0005] To overcome the defects and shortcomings of existing technologies, this invention provides a weakly supervised EEG emotion recognition method and system based on spatiotemporal multi-instance learning. This invention can simultaneously model the temporal dynamics and spatial topological features of EEG signals, effectively alleviate the noise problem caused by coarse labels, improve the accuracy and robustness of cross-subject emotion recognition, and provide interpretable spatiotemporal attention visualization.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] This invention provides a weakly supervised EEG emotion recognition method based on spatiotemporal multi-instance learning, comprising the following steps:

[0008] EEG signals are acquired and preprocessed to obtain the differential entropy feature matrix;

[0009] Construct a dual-branch instance feature extractor, including a Time-Inception module and a Space-Inception module, and input the differential entropy feature matrix into the Time-Inception module and the Space-Inception module respectively;

[0010] The Time-Inception module extracts multi-scale temporal features based on multi-scale convolution and organizes these multi-scale temporal features into a spatial instance sequence according to EEG channels.

[0011] The Space-Inception module performs global convolution and hemispherical convolution to extract the spatial relationship between the whole brain and the left and right hemispheres. It extracts spatial dimension features through temporal patch embedding and depthwise separable convolution, and organizes the spatial dimension features into a temporal instance sequence according to the time window.

[0012] Construct a time-aware MIL pooling branch and a space-aware MIL pooling branch. The time instance sequence is processed by the time-aware MIL pooling branch to obtain time packet features, and the space instance sequence is processed by the space-aware MIL pooling branch to obtain space packet features.

[0013] Input the temporal and spatial features into a linear classifier to obtain the probability distribution of sentiment categories in the temporal and spatial branches. Construct the total loss function and train the sentiment category prediction model.

[0014] The final emotion category prediction result is output based on the emotion category prediction model.

[0015] As a preferred technical solution, acquiring EEG signals and performing data preprocessing specifically includes:

[0016] The EEG signal is bandpass filtered, the filtered signal is resampled, the resampled signal is divided into multiple non-overlapping time windows, and the differential entropy features of different frequency bands are extracted for each time window to obtain the differential entropy feature matrix.

[0017] As a preferred technical solution, the Time-Inception module extracts multi-scale temporal features from the differential entropy feature matrix based on multi-scale convolution, specifically including:

[0018] The Time-Inception module adjusts the number of channels through convolution and extracts multi-scale temporal features through four parallel branches. The four parallel branches use one-dimensional convolution with different kernel lengths to extract multi-scale temporal features. The fourth branch also performs max pooling before the one-dimensional convolution.

[0019] The outputs of the four parallel branches are element-wise summed and fused, and then an intermediate feature representation of size T×N is generated through convolution, batch normalization and activation function, where N is the number of EEG channels and T is the number of time windows.

[0020] The intermediate features represent the spatial indices of N EEG channels. The intermediate features are organized into an N×T spatial instance sequence according to the EEG channels. Each spatial instance corresponds to one EEG channel. The instance features include the multi-scale temporal features of the corresponding EEG channel over T time windows.

[0021] As a preferred technical solution, the Space-Inception module performs global convolution and hemispherical convolution to extract the spatial relationships of the whole brain and the left and right hemispheres. Spatial dimensional features are then extracted through temporal patch embedding and depthwise separable convolution, specifically including:

[0022] The Space-Inception module rearranges the EEG channels according to the left and right hemispheres, dividing them into a global branch and a hemisphere branch.

[0023] The global branch extracts whole-brain spatial features based on one-dimensional convolution, while the hemispherical branch uses two one-dimensional convolutions to extract left-brain and right-brain features respectively. The outputs of the global branch and the hemispherical branch are concatenated along the feature channel direction and batch normalized to obtain an intermediate feature representation that includes the spatial relationship between the whole brain and the left and right hemispheres.

[0024] The intermediate feature representation is input into the temporal patch embedding layer and reorganized into a patch sequence arranged by time window. Based on depthwise convolution, pointwise convolution, batch normalization and activation function, a T×N temporal instance sequence is obtained, where N is the number of EEG channels and T is the number of time windows. Each temporal instance corresponds to a time window, and the instance features include the spatial dimension features corresponding to the time window.

[0025] As a preferred technical solution, the time instance sequence is processed through a time-aware MIL pooling branch to obtain time packet features, specifically including:

[0026] The category labels are concatenated with the time instance sequence to obtain the input sequence;

[0027] The input sequence is convolved using Mexican hat wavelets with different parameters. The convolution results are then summed and fused element by element to obtain a time position code. The time position code is then added element by element to the input sequence to obtain a feature sequence that incorporates time position information.

[0028] The long-range dependencies between time instances are modeled based on feature sequences that integrate time and location information, and the output corresponding to the category label is extracted as the final time packet feature.

[0029] As a preferred technical solution, the spatial instance sequence is processed through the spatially aware MIL pooling branch to obtain spatial packet features, specifically including:

[0030] The category labels are concatenated with the spatial instance sequence to obtain the input sequence;

[0031] Add spatial location coding based on the 3D distribution of EEG electrodes, extract the coding information corresponding to the electrode positions in the spatial location coding, and add it element by element to the input sequence to obtain a feature sequence that fuses spatial location information;

[0032] Modeling the correlation between spatial instances based on feature sequences that integrate spatial location information;

[0033] The output corresponding to the category label is extracted as the final spatial bag feature.

[0034] As a preferred technical solution, the total loss function is constructed and expressed as:

[0035] ;

[0036] ;

[0037] in, Represents the total loss function. The cross-entropy supervised loss represents the time branch. The cross-entropy supervised loss represents the spatial branch. For KL divergence calculation, Indicates the dynamic annealing factor. This represents the cross-branch consistency constraint loss.

[0038] As a preferred technical solution, the final emotion category prediction result is output based on the emotion category prediction model, specifically including:

[0039] For the EEG signal to be identified, the arithmetic mean of the probability distribution of the emotion category in the time branch and spatial branch of the emotion category prediction model is calculated for each category, and the category with the highest average probability is determined as the final emotion category.

[0040] As a preferred technical solution, the output of the final emotion category prediction result also includes a spatiotemporal attention importance map generation step, specifically including:

[0041] Calculate the self-attention scores of the category labels and all time instances in the time-aware MIL pooling branch, and obtain the importance weight of each time window by softmax normalization to form a time importance graph;

[0042] Calculate the self-attention scores of the category labels and all spatial instances in the spatially aware MIL pooling branch, and obtain the importance weight of each EEG channel after softmax normalization to form a spatial importance map.

[0043] This invention also provides a weakly supervised EEG emotion recognition system based on spatiotemporal multi-instance learning, used to implement the above-mentioned weakly supervised EEG emotion recognition method based on spatiotemporal multi-instance learning, including: a data preprocessing module, a dual-branch instance feature extractor construction module, a spatiotemporal dual-branch multi-instance learning pooling architecture construction module, a model training module, and an emotion category prediction module;

[0044] The data preprocessing module is used to preprocess the acquired EEG signal to obtain the differential entropy feature matrix;

[0045] The dual-branch instance feature extractor construction module is used to construct a dual-branch instance feature extractor, including a Time-Inception module and a Space-Inception module, and the differential entropy feature matrix is ​​input into the Time-Inception module and the Space-Inception module respectively;

[0046] The Time-Inception module extracts multi-scale temporal features based on multi-scale convolution and organizes these multi-scale temporal features into a spatial instance sequence according to EEG channels.

[0047] The Space-Inception module performs global convolution and hemispherical convolution to extract the spatial relationship between the whole brain and the left and right hemispheres. It extracts spatial dimension features through temporal patch embedding and depthwise separable convolution, and organizes the spatial dimension features into a temporal instance sequence according to the time window.

[0048] The spatiotemporal dual-branch multi-instance learning pooling architecture construction module is used to construct the time-aware MIL pooling branch and the space-aware MIL pooling branch. The time instance sequence is processed by the time-aware MIL pooling branch to obtain time packet features, and the space instance sequence is processed by the space-aware MIL pooling branch to obtain space packet features.

[0049] The model training module is used to train an emotion category prediction model. It inputs the time packet features and spatial packet features into a linear classifier to obtain the emotion category probability distribution of the time branch and the spatial branch, constructs the total loss function, and trains the emotion category prediction model.

[0050] The emotion category prediction module is used to output the final emotion category prediction result based on the emotion category prediction model.

[0051] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0052] (1) This invention solves the technical problems caused by the uniform assignment of trial coarse labels to all time windows and EEG channels, such as label granularity mismatch, destruction of instance position order by traditional permutation invariant pooling, and neglect of brain region spatial heterogeneity by single-time MIL, through a collaborative technical solution of spatiotemporal instance decoupling extraction, dual-branch position-aware aggregation, cross-branch consistency joint training, and probabilistic fusion decision. This enables the model to locate key time windows and key EEG channels related to emotion discrimination under the condition of using only trial labels, and form mutually constrained emotion category prediction results.

[0053] (2) This invention is not a simple parallel arrangement of time branches and spatial branches, but rather organizes the differential entropy feature matrix of the same EEG trial into time instance sequences and spatial instance sequences respectively. It uses time-aware MIL pooling branches to select key time windows and spatial-aware MIL pooling branches to select key EEG channels. Through cross-branch consistency constraints and probabilistic fusion, the two branches form a joint discrimination result with mutual constraints while retaining complementary information, thereby alleviating the problems of single time MIL ignoring spatial heterogeneity and single spatial modeling ignoring time dynamics.

[0054] (3) In view of the fact that time instances and spatial instances have a definite order, this invention sets up learnable wavelet time position codes and spatial position codes based on the three-dimensional distribution of EEG electrodes, respectively, and then performs self-attention aggregation after fusing the position codes with the corresponding instance sequences, thereby preserving the time window order and electrode topology relationship during the MIL pooling process and reducing the loss of position information caused by permutation invariant aggregation.

[0055] (4) The present invention adopts a hybrid training mechanism, which uses the cross-entropy supervision loss of each of the two branches and the bidirectional KL divergence between the output probability distributions of the two branches to form the training objective. The dynamic annealing coefficient is used to adjust the strength of the consistency constraint, so that the time branch and the spatial branch gradually align the class discrimination distribution while learning the temporal dynamics and spatial topological features respectively, thereby reducing the risk of overfitting of a single branch and improving the stability of cross-subject prediction.

[0056] (5) This invention extracts the attention scores of category tokens to time instances and space instances from the time-aware MIL pooling branch and the space-aware MIL pooling branch respectively, forming a time importance map and a space importance map; by displaying the spatial importance distribution of the same EEG trial in each time window in chronological order, the key time windows and key brain regions that affect the judgment of emotion categories can be located, providing a traceable spatiotemporal visualization basis for the prediction results.

[0057] (6) The present invention achieves higher average recognition accuracy than the listed comparison methods on the binary classification tasks of arousal and pleasure in the DEAP dataset and the tri-class classification task in the SEED dataset. At the same time, it achieves lower cross-subject result fluctuation, indicating that the present invention can improve the accuracy and stability of weakly supervised cross-subject EEG emotion recognition. Attached Figure Description

[0058] Figure 1 This is a flowchart illustrating the weakly supervised EEG emotion recognition method based on spatiotemporal multi-instance learning according to the present invention.

[0059] Figure 2 This is a schematic diagram of the overall implementation architecture of the weakly supervised EEG emotion recognition method based on spatiotemporal multi-instance learning of the present invention.

[0060] Figure 3 This is a schematic diagram of wavelet time-position encoding of the present invention;

[0061] Figure 4 This is a schematic diagram of the spatial location encoding based on the 3D distribution of EEG electrodes according to the present invention;

[0062] Figure 5 This is a joint spatial attention diagram of low-arousal, high-pleasure EEG trials in the DEAP dataset of this invention at different time windows. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0064] Example 1

[0065] like Figure 1 As shown, this embodiment provides a weakly supervised EEG emotion recognition method based on spatiotemporal multi-instance learning, including the following steps:

[0066] S1: Obtain the raw EEG signal and preprocess it to obtain the differential entropy feature matrix, specifically including:

[0067] S11: Perform a 1.0-50Hz bandpass filter on the original EEG signal to remove noise and artifacts;

[0068] S12: Resample the filtered signal to 200Hz to reduce computational complexity;

[0069] S13: Divide the resampled signal into a 1-second non-overlapping time window;

[0070] S14: Extract differential entropy features for each time window from five frequency bands: delta (1-4Hz), theta (4-8Hz), alpha (8-13Hz), beta (13-30Hz), and gamma (30-50Hz), forming a differential entropy feature matrix of size F×N×T, where F is the number of frequency bands, N is the number of EEG channels, and T is the number of time windows;

[0071] Specifically, this embodiment takes the DEAP dataset as an example. This dataset contains 32 channels of EEG signals recorded when 32 subjects watched 40 one-minute music videos. The sampling rate is 512Hz. First, the original EEG signal is bandpass filtered from 1.0 to 50Hz to remove 50Hz power frequency interference and high-frequency noise. Then, the signal is resampled to 200Hz. Next, each 60-second trial is divided into 60 one-second non-overlapping time windows. Finally, differential entropy features of five frequency bands are extracted for each time window to form a differential entropy feature matrix of size 5×32×60.

[0072] S2: Construct a dual-branch instance feature extractor, including a Time-Inception module and a Space-Inception module. The differential entropy feature matrix serves as the input to both the Time-Inception and Space-Inception modules. The Time-Inception module performs multi-scale convolution along the time axis to extract features in the time dimension and organizes the extracted features into a spatial instance sequence according to the EEG channel, which serves as the input to the subsequent spatially aware MIL pooling branch. The Space-Inception module performs global convolution and hemispherical convolution along the electrode channel axis to extract the spatial relationship between the whole brain and the left and right hemispheres. After processing with temporal patch embedding and depthwise separable convolution, it extracts features in the spatial dimension and organizes the extracted features into a temporal instance sequence according to the time window, which serves as the input to the subsequent time-aware MIL pooling branch.

[0073] like Figure 2 As shown, the Time-Inception module uses multi-scale one-dimensional convolutional kernels to process the differential entropy feature matrix simultaneously. The kernel lengths are 10, 20, and 40, respectively. Combined with parallel max pooling operations, it outputs a feature representation that fuses multi-scale time patterns.

[0074] Specifically, the Time-Inception module receives a differential entropy feature matrix of size 5×32×60. First, it adjusts the number of channels through 1×1 convolution, and then divides it into four parallel branches: three branches use one-dimensional convolution with kernel lengths of 10, 20, and 40 to extract multi-scale temporal features, respectively; the fourth branch uses max pooling with a window size of 3 to enhance robustness; the outputs of the four branches are element-wise summed and fused, and then generated as an intermediate feature representation of size T×N through 1×1 convolution, batch normalization, and ReLU activation; preferably, T=60 and N=32.

[0075] In this embodiment, the intermediate feature output by the Time-Inception module represents the spatial index of N EEG channels, which are organized into an N×T spatial instance sequence according to the EEG channels. Each spatial instance corresponds to one EEG channel, and the instance features include the multi-scale temporal features of the EEG channel over T time windows. The spatial instance sequence is then input into the spatially aware MIL pooling branch.

[0076] like Figure 2 As shown, the differential entropy feature matrix is ​​input into the Space-Inception module, and EEG channel rearrangement, global convolution and hemispherical convolution are processed in parallel, feature concatenation, batch normalization, time patch embedding, depthwise convolution, pointwise convolution, batch normalization and ReLU activation are performed in sequence to obtain a time instance sequence organized by time window, and the time instance sequence is input into the time-aware MIL pooling branch.

[0077] Specifically, the Space-Inception module receives the differential entropy feature matrix, first rearranges the EEG channels according to the left and right hemispheres, and then divides them into two parallel branches: the global branch uses a one-dimensional convolution with a size of 32 to extract the spatial features of the whole brain, and the hemisphere branch uses two one-dimensional convolutions with a size of 16 and a stride of 16 to extract the features of the left and right brains respectively. The outputs of the two branches are concatenated along the feature channel direction and batch normalized to obtain an intermediate feature representation that includes the spatial relationship between the whole brain and the left and right hemispheres.

[0078] The intermediate feature representation is input into the temporal patch embedding layer and reorganized into a patch sequence arranged by time window. Then, the depthwise convolution extracts the internal features of each patch sequence separately. The pointwise convolution uses 1×1 convolution to fuse different feature channels. The two together form a depthwise separable convolution. Batch normalization and ReLU activation are performed after the convolution. After the above processing, a T×N temporal instance sequence is obtained, where each temporal instance corresponds to a time window. The instance features include the EEG spatial features corresponding to the time window. The temporal instance sequence is input into the time-aware MIL pooling branch.

[0079] in, Figure 2In this context, N represents the number of EEG channels, N / 2 represents the number of channels corresponding to a single hemisphere, (1, 10), (1, 20), and (1, 40) represent the three 2D convolution kernel sizes set along the time axis, (1, 3) represents the max pooling window size, and (1, 1) represents the pointwise convolution kernel size. "Transformer block × 2" means setting two Transformer blocks consecutively, the category token is a learnable vector used to aggregate bag-level information, and "pointwise convolution" means pointwise convolution, which uses 1×1 convolution to linearly fuse different feature channels.

[0080] In this embodiment, the Time-Inception module is a temporal multi-scale Inception instance feature extraction module, and the Space-Inception module is a spatial multi-scale Inception instance feature extraction module. Both are instance feature extraction modules targeting the differential entropy feature matrix.

[0081] S3: Construct a spatiotemporal dual-branch multi-instance learning pooling architecture, including a time-aware MIL pooling branch and a space-aware MIL pooling branch; process the time instance sequence through the time-aware MIL pooling branch to generate time packet features, and process the spatial instance sequence through the space-aware MIL pooling branch to generate spatial packet features.

[0082] In this embodiment, the spatiotemporal dual-branch multi-instance learning pooling architecture is not simply a parallel arrangement of two classification branches. During training, the sentiment label of the same trial level supervises the temporal branch and the spatial branch respectively. Bidirectional KL divergence further constrains the two branches to output similar class probability distributions for the same EEG trial. Dynamic annealing coefficient is used to avoid excessively strong consistency constraints in the early stages of training inhibiting each branch from learning its own specific features. During inference, the class probability distributions of the two branches are fused to determine the final class. Thus, the brain region topological evidence provided by the spatial branch constrains the temporal branch through consistency training, and the dynamic rhythmic evidence provided by the temporal branch also constrains the spatial branch, thereby forming a technical closed loop from complementary instance extraction, joint optimization to fusion decision.

[0083] In this embodiment, the time-aware MIL pooling branch is used to model the temporal dynamic correlation of EEG signals, and the processing steps are as follows:

[0084] S311: Initialize the learnable category token and concatenate it with the time instance sequence;

[0085] Specifically, the category token (i.e., category label) has a size of 1×N, where N is the number of EEG channels. In this embodiment, N=32 is preferred. The category token is concatenated with a time instance sequence of size T×N to obtain an input sequence of size (T+1)×N. Preferably, T=60, corresponding to 60 1-second time windows.

[0086] S312: Add learnable wavelet time-position coding to capture the time-frequency information of EEG signals;

[0087] like Figure 3 As shown, the input sequence is processed by convolution using three Mexican hat wavelets with different parameters. The three convolution results are summed and fused element by element to obtain the time position code. The time position code is added to the input sequence element by element to obtain the feature sequence with fused time position information.

[0088] in, Figure 3 In Let X represent a learnable category token used to aggregate packet-level information, where X is a sequence of T time instances, belonging to a T×N dimensional real space, T is the number of time windows, and N is the feature dimension of each time instance. Placed at the beginning of X's sequence, this becomes the input sequence fed into the time-aware MIL pooling branch. Since the category token does not correspond to a specific time position, the time position encoding is first calculated by... Separating from the input sequence, the remaining time instance sequence is denoted as... , The dimensions are T×N; Figure 2 In this context, Ψ1, Ψ2, and Ψ3 represent three parallel learnable Mexican hat wavelet kernel groups, with the subscript j indicating the wavelet branch number, where j takes the values ​​1, 2, or 3; Φ(X̃,Ψ j ) indicates the use of the j-th wavelet kernel pair The branch output obtained after performing the convolution transformation still has a size of T×N. Summing Φ(X̃,Ψ1), Φ(X̃,Ψ2), and Φ(X̃,Ψ3) element-wise yields the result... Time position codes of the same size; then compare this time position code with... Add element by element, and combine the previously separated elements. Repositioned at the beginning of the sequence, forming a position-enhanced time instance sequence of the input Transformer block.

[0089] S313: By modeling the correlation between temporal instances using two layers of Nyströmformer Transformer blocks, the computational complexity of self-attention is reduced from O(n²) to O(n);

[0090] Specifically, the feature sequences that incorporate temporal and location information are input into two layers of Nyströmformer Transformer blocks to model long-range dependencies between temporal instances;

[0091] Each Nyströmformer Transformer block contains a multi-head self-attention layer and a feedforward neural network. The multi-head self-attention layer is used to compute the correlation between instances at different times, and the feedforward neural network is used to perform non-linear transformations on the features.

[0092] S314: Extract the category token as the final time packet feature;

[0093] Specifically, the first element (i.e. the output corresponding to the category token) in the output sequence of the two-layer Nyströmformer Transformer blocks is extracted as the final time packet feature, and the size of the time packet feature is 1×N.

[0094] In this embodiment, the spatially aware MIL pooling branch is used to model the spatial topological correlation of EEG signals, and the processing steps are as follows:

[0095] S321: Initialize the learnable category token and concatenate it with the spatial instance sequence;

[0096] Specifically, the size of the category token is 1×T, where T is the number of time windows. In this embodiment, T=60 is preferred. The category token is concatenated with the spatial instance sequence of size N×T to obtain the input sequence of size (N+1)×T. Preferably, N=32.

[0097] S322: Add spatial location coding based on the 3D distribution of EEG electrodes to preserve the spatial topology information of the electrodes;

[0098] like Figure 4 As shown, the 32 EEG electrodes are first converted into a 9×9 2D matrix according to the spatial distribution of the international 10-20 system. Zero-filling is performed on the positions without electrodes. Then, it is expanded into a 9×9×T 3D representation. A 3×3 convolution kernel is used to convolve the 3D representation to generate spatial position codes. The encoding information corresponding to the 32 actual electrode positions in the spatial position codes is extracted and added element by element to the input sequence to obtain a feature sequence that fuses spatial position information.

[0099] in, Figure 4In this context, H and W represent the number of rows and columns of the two-dimensional electrode layout matrix, respectively; T represents the number of time windows; and N represents the actual number of EEG channels. Conv indicates zero-padding; H×W×T indicates performing convolution to generate spatial location codes; Identity indicates identity connections that preserve the original features; Fusion indicates adding the spatial location codes to the original features element-wise; 3D Restoration indicates restoring the two-dimensional layout according to the three-dimensional electrode distribution and extending it along the time axis; and 1DRestoration indicates extracting only N actual electrode positions and restoring them as a sequence of instances arranged by channels.

[0100] S323: Modeling the correlation between spatial instances using two layers of Nyströmformer Transformer blocks;

[0101] In this embodiment, the feature sequence fused with spatial location information is input into two layers of Nyströmformer blocks to model the spatial correlation between channels in different brain regions.

[0102] S324: Extract the category token as the final spatial packet feature;

[0103] In this embodiment, the first element (i.e. the output corresponding to the category token) in the output sequence of the two-layer Nyströmformer blocks is extracted as the final spatial bag feature, and the size of the spatial bag feature is 1×T;

[0104] S4: Employ a hybrid training mechanism for end-to-end model training, combining supervised loss and cross-branch feature fusion loss to optimize model parameters, specifically including:

[0105] S41: Input the temporal and spatial features into a linear classifier to obtain the probability distributions of sentiment categories for the temporal and spatial branches, and calculate the cross-entropy supervised loss for the temporal branch. Cross-entropy supervision loss of spatial branches ;

[0106] S42: Calculate the bidirectional KL divergence between the output probability distributions of the time branch and the spatial branch as the cross-branch consistency constraint loss. This promotes the two branches to learn complementary and consistent feature representations:

[0107] ;

[0108] In the formula, and These are the output probability distributions for the time branch and the spatial branch, respectively. For KL divergence calculation;

[0109] S43: Use dynamic annealing coefficient Balancing monitoring loss and fusion loss, among which The total loss function is: (The number of training steps is given by the function.)

[0110] ;

[0111] In the formula, The training steps decrease cosine-decreasing:

[0112] ;

[0113] In the formula, The initial maximum weight, This represents the total number of training steps.

[0114] In this embodiment, an initial maximum weight is set. Total training steps The Adam optimizer was used to optimize the model parameters, and the learning rate was set to [value missing]. The batch size is 32, and 300 batches are trained until the model converges;

[0115] S5: Input the EEG signal to be identified into the trained model, output the final emotion category prediction result, and generate the corresponding spatiotemporal attention importance map, specifically including:

[0116] S51: Time Attention Weights: Calculate the self-attention scores of category tokens and all time instances in the time-aware MIL pooling branch, and obtain the importance weights of each time window through softmax normalization to form a time importance graph;

[0117] S52: Spatial Attention Weights: Calculate the self-attention scores of category tokens and all spatial instances in the spatially aware MIL pooling branch, and obtain the importance weights of each EEG channel through softmax normalization to form a spatial importance graph.

[0118] In this embodiment, the temporal importance map provides relative importance along T time windows, while the spatial importance map provides relative importance along N EEG channels. Both map charactersize the temporal dynamic evidence and spatial topological evidence for the same EEG trial, respectively. By arranging the spatial importance map sequentially according to the corresponding time windows and combining it with the key windows indicated by the temporal importance map for joint observation, it is possible to simultaneously determine "when" and "which brain regions" contribute significantly to the prediction of the current emotion category, thus simultaneously capturing the dynamic temporal evolution and spatial coordination patterns of emotion-related neural activity.

[0119] In this embodiment, for the EEG signal to be identified, the model outputs the prediction results of the temporal and spatial branches, and takes the average of the two as the final emotion category. Specifically, the arithmetic mean of the probability distributions of the emotion categories output by the two branches is calculated for each category, and the category with the highest average probability is determined as the final emotion category. Simultaneously, the self-attention score of the category token and all instances is calculated, such as... Figure 5 As shown, a spatiotemporal attention heatmap was generated, where red represents high attention weight and blue represents low attention weight. The figure shows the attention distribution of a subject in a low arousal and high pleasure state, and it can be seen that the model focuses on the central brain region and the right occipital-temporal lobe region.

[0120] Specifically, let the class probability distributions of the time branch and spatial branch outputs be pt and ps, respectively. Then, the fused probability distribution p = (pt + ps) / 2, and the final predicted class is the class corresponding to the largest probability component in p. For the DEAP dataset, arousal and pleasure are classified as low / high, respectively, so the output classes are low arousal or high arousal, and low pleasure or high pleasure, respectively. For the SEED dataset, the output classes are negative, neutral, or positive. Figure 5 The samples correspond to low-arousal, high-pleasure trials in the DEAP dataset.

[0121] Combination Figure 5 As shown, this diagram presents a combined representation of temporal and spatial importance, rather than simply adding the two weights together to form a new attention weight. The 5s, 10s, up to 60s values ​​above each column indicate the displayed time window. The color of each scalp topography map represents the relative spatial importance of each EEG channel within that time window. The Arousal row corresponds to the arousal classification head, and the Valence row corresponds to the pleasure classification head. Thus, the central region channels CP1, CP2, and FC1 exhibit higher relative weights in most time windows, corresponding to the primary spatial evidence in low arousal discrimination. The right occipital lobe channels O2 and PO4, and the right temporal lobe channels T8 and CP6, exhibit higher relative weights in some time windows, corresponding to the primary spatiotemporal evidence in high pleasure discrimination. The color bars display the normalized relative visual values, with red indicating relatively high and blue indicating relatively low. These values ​​are used to explain the relative contribution of different time-channel positions within the same classification head and are not considered as the probability of the emotion category.

[0122] To verify the effectiveness of this invention, leave-one-out cross-validation experiments were conducted on the DEAP and SEED datasets, and the results are shown in Tables 1 and 2.

[0123] Table 1. Emotion recognition results across participants using the DEAP dataset.

[0124]

[0125] Table 2. Emotion recognition results across participants in the SEED dataset.

[0126]

[0127] Experimental results show that the method of this invention significantly outperforms traditional methods and existing single-branch MIL methods on both datasets, especially achieving an accuracy improvement of over 18% on the DEAP dataset, demonstrating the effectiveness of the spatiotemporal dual-branch architecture. Furthermore, the standard deviation of the method of this invention is significantly lower than that of the comparative methods, indicating stronger robustness across different subject scenarios.

[0128] To verify the effects of the spatiotemporal bi-branch MIL pooling structure, dedicated positional encoding, and cross-branch consistency training, three ablation experiments were conducted on the DEAP dataset using leave-one-out cross-validation. All experiments used the same data partitioning and evaluation metrics; within each experiment, all training conditions remained consistent except for the structure or parameter being examined. In the table, ACC represents the average classification accuracy, F1 represents the average F1 score, and the values ​​after "±" represent the standard deviation of test results among different participants.

[0129] The first set of experiments only changed the MIL pooling structure to compare the use of no learnable MIL pooling, general MIL pooling, single temporal or spatially aware MIL pooling, and spatiotemporal dual-branch aware MIL pooling. The results are shown in Table 3 below:

[0130] Table 3 Ablation experimental results of different MIL pooling structures

[0131]

[0132] As shown in Table 3, spatiotemporal-aware MIL pooling outperforms the no-MIL pooling, general-purpose MIL pooling, and the two single-branch pooling methods in both arousal and pleasure tasks, achieving higher ACC and F1 scores. Specifically, the arousal ACC of spatiotemporal-aware MIL pooling is 75.00%, higher than the 71.56% of time-aware MIL pooling and the 71.88% of space-aware MIL pooling; its pleasure ACC is 72.11%, higher than the 67.81% and 69.14% of the two single-branch pooling methods. This set of experiments directly demonstrates that, under the same experimental conditions, simultaneously modeling temporal and spatial instances yields better classification results than modeling only one instance dimension.

[0133] The second set of experiments, which varied the position coding configuration in different MIL branches, was used to verify the effects of wavelet temporal position coding (TPE) and electrode distribution-based spatial position coding (SPE). The results are shown in Table 4 below.

[0134] Table 4 Ablation experimental results with different coding configurations at different locations

[0135]

[0136] As shown in Table 4, in the time-aware branch, the arousal and pleasure ACC using TPE were 68.67% and 67.81%, respectively, higher than the 68.28% and 66.09% without location encoding. In the spatial-aware branch, the corresponding ACC using SPE were 69.30% and 69.14%, respectively, higher than the 67.73% and 66.56% without location encoding. In the spatiotemporal-aware dual-branch, using both TPE and SPE resulted in ACCs of 75.00% and 72.11%, respectively. This set of experiments is used to demonstrate that preserving temporal order and electrode spatial topology information has an effect on instance aggregation in the corresponding MIL branches, and is not considered as evidence of cross-branch consistency training effectiveness on its own.

[0137] The third group of experiments kept the complete spatiotemporal bi-branch structure unchanged, and only changed the initial maximum weight λmax of the dynamic annealing coefficient of the cross-branch consistency loss. The results are shown in Table 5 below. When λmax is 0, the cross-branch consistency loss does not participate in the optimization of model parameters.

[0138] Table 5 Ablation experiment results with different cross-branch consistency loss weights

[0139]

[0140] As shown in Table 5, when λmax is adjusted from 0 to a non-zero value, the overall ACC and F1 of the complete model improve under the listed settings; when λmax is 1.0, the arousal and pleasure ACC reach 75.00% and 72.11%, respectively. This set of experiments provides experimental evidence that cross-branch consistency loss participating in joint training can improve the discrimination results of the current embodiment. Combining Tables 3 to 5, the dual-branch pooling structure, dedicated spatiotemporal location encoding, and cross-branch consistency loss each have corresponding ablation evidence, therefore, it is no longer necessary to use a single set of location encoding data to infer the effects of the above three technical links simultaneously.

[0141] Example 2

[0142] This embodiment provides a weakly supervised EEG emotion recognition system based on spatiotemporal multi-instance learning, which is used to implement the weakly supervised EEG emotion recognition method based on spatiotemporal multi-instance learning in Embodiment 1. It includes: a data preprocessing module, a dual-branch instance feature extractor construction module, a spatiotemporal dual-branch multi-instance learning pooling architecture construction module, a model training module, and an emotion category prediction module.

[0143] In this embodiment, the data preprocessing module is used to preprocess the acquired EEG signal to obtain the differential entropy feature matrix;

[0144] In this embodiment, the dual-branch instance feature extractor construction module is used to construct a dual-branch instance feature extractor, including a Time-Inception module and a Space-Inception module, and the differential entropy feature matrix is ​​input into the Time-Inception module and the Space-Inception module respectively;

[0145] The Time-Inception module extracts multi-scale temporal features based on multi-scale convolution and organizes these multi-scale temporal features into a spatial instance sequence according to EEG channels.

[0146] The Space-Inception module performs global convolution and hemispherical convolution to extract the spatial relationship between the whole brain and the left and right hemispheres. It extracts spatial dimension features through temporal patch embedding and depthwise separable convolution, and organizes the spatial dimension features into a temporal instance sequence according to the time window.

[0147] In this embodiment, the spatiotemporal dual-branch multi-instance learning pooling architecture building module is used to construct the time-aware MIL pooling branch and the space-aware MIL pooling branch. The time instance sequence is processed by the time-aware MIL pooling branch to obtain time packet features, and the space instance sequence is processed by the space-aware MIL pooling branch to obtain space packet features.

[0148] In this embodiment, the model training module is used to train an emotion category prediction model. The time packet features and spatial packet features are input into a linear classifier to obtain the probability distribution of emotion categories in the time branch and spatial branch, and a total loss function is constructed to train the emotion category prediction model.

[0149] In this embodiment, the emotion category prediction module is used to output the final emotion category prediction result based on the emotion category prediction model.

[0150] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A weakly supervised EEG emotion recognition method based on spatiotemporal multi-instance learning, characterized in that, Includes the following steps: EEG signals are acquired and preprocessed to obtain the differential entropy feature matrix; Construct a dual-branch instance feature extractor, including a Time-Inception module and a Space-Inception module, and input the differential entropy feature matrix into the Time-Inception module and the Space-Inception module respectively; The Time-Inception module extracts multi-scale temporal features based on multi-scale convolution and organizes these multi-scale temporal features into a spatial instance sequence according to EEG channels. The Space-Inception module performs global convolution and hemispherical convolution to extract the spatial relationship between the whole brain and the left and right hemispheres. It extracts spatial dimension features through temporal patch embedding and depthwise separable convolution, and organizes the spatial dimension features into a temporal instance sequence according to the time window. Construct a time-aware MIL pooling branch and a space-aware MIL pooling branch. The time instance sequence is processed by the time-aware MIL pooling branch to obtain time packet features, and the space instance sequence is processed by the space-aware MIL pooling branch to obtain space packet features. Input the temporal and spatial features into a linear classifier to obtain the probability distribution of sentiment categories in the temporal and spatial branches. Construct the total loss function and train the sentiment category prediction model. The final emotion category prediction result is output based on the emotion category prediction model.

2. The weakly supervised EEG emotion recognition method based on spatiotemporal multi-instance learning according to claim 1, characterized in that, Acquiring EEG signals and performing data preprocessing specifically includes: The EEG signal is bandpass filtered, the filtered signal is resampled, the resampled signal is divided into multiple non-overlapping time windows, and the differential entropy features of different frequency bands are extracted for each time window to obtain the differential entropy feature matrix.

3. The weakly supervised EEG emotion recognition method based on spatiotemporal multi-instance learning according to claim 1, characterized in that, The Time-Inception module extracts multi-scale temporal features from the differential entropy feature matrix based on multi-scale convolution, specifically including: The Time-Inception module adjusts the number of channels through convolution and extracts multi-scale temporal features through four parallel branches. The four parallel branches use one-dimensional convolution with different kernel lengths to extract multi-scale temporal features. The fourth branch also performs max pooling before the one-dimensional convolution. The outputs of the four parallel branches are element-wise summed and fused, and then an intermediate feature representation of size T×N is generated through convolution, batch normalization and activation function, where N is the number of EEG channels and T is the number of time windows. The intermediate features represent the spatial indices of N EEG channels. The intermediate features are organized into an N×T spatial instance sequence according to the EEG channels. Each spatial instance corresponds to one EEG channel. The instance features include the multi-scale temporal features of the corresponding EEG channel over T time windows.

4. The weakly supervised EEG emotion recognition method based on spatiotemporal multi-instance learning according to claim 1, characterized in that, The Space-Inception module performs global and hemispherical convolutions to extract the spatial relationships between the whole brain and the left and right hemispheres. Spatial dimensional features are extracted through temporal patching embedding and depthwise separable convolution, specifically including: The Space-Inception module rearranges the EEG channels according to the left and right hemispheres, dividing them into a global branch and a hemisphere branch. The global branch extracts whole-brain spatial features based on one-dimensional convolution, while the hemispherical branch uses two one-dimensional convolutions to extract left-brain and right-brain features respectively. The outputs of the global branch and the hemispherical branch are concatenated along the feature channel direction and batch normalized to obtain an intermediate feature representation that includes the spatial relationship between the whole brain and the left and right hemispheres. The intermediate feature representation is input into the temporal patch embedding layer and reorganized into a patch sequence arranged by time window. Based on depthwise convolution, pointwise convolution, batch normalization and activation function, a T×N temporal instance sequence is obtained, where N is the number of EEG channels and T is the number of time windows. Each temporal instance corresponds to a time window, and the instance features include the spatial dimension features corresponding to the time window.

5. The weakly supervised EEG emotion recognition method based on spatiotemporal multi-instance learning according to claim 1, characterized in that, The temporal instance sequence is processed through the time-aware MIL pooling branch to obtain temporal packet features, specifically including: The category labels are concatenated with the time instance sequence to obtain the input sequence; The input sequence is convolved using Mexican hat wavelets with different parameters. The convolution results are then summed and fused element by element to obtain a time position code. The time position code is then added element by element to the input sequence to obtain a feature sequence that incorporates time position information. The long-range dependencies between time instances are modeled based on feature sequences that integrate time and location information, and the output corresponding to the category label is extracted as the final time packet feature.

6. The weakly supervised EEG emotion recognition method based on spatiotemporal multi-instance learning according to claim 1, characterized in that, Spatial instance sequences are processed through the spatially aware MIL pooling branch to obtain spatial packet features, which specifically include: The category labels are concatenated with the spatial instance sequence to obtain the input sequence; Add spatial location coding based on the 3D distribution of EEG electrodes, extract the coding information corresponding to the electrode positions in the spatial location coding, and add it element by element to the input sequence to obtain a feature sequence that fuses spatial location information; Modeling the correlation between spatial instances based on feature sequences that integrate spatial location information; The output corresponding to the category label is extracted as the final spatial bag feature.

7. The weakly supervised EEG emotion recognition method based on spatiotemporal multi-instance learning according to claim 1, characterized in that, Construct the total loss function, expressed as: ； ； in, Represents the total loss function. The cross-entropy supervised loss represents the time branch. The cross-entropy supervised loss represents the spatial branch. For KL divergence calculation, Indicates the dynamic annealing factor. This represents the cross-branch consistency constraint loss.

8. The weakly supervised EEG emotion recognition method based on spatiotemporal multi-instance learning according to claim 1, characterized in that, The final emotion category prediction result is output based on the emotion category prediction model, specifically including: For the EEG signal to be identified, the arithmetic mean of the probability distribution of the emotion category in the time branch and spatial branch of the emotion category prediction model is calculated for each category, and the category with the highest average probability is determined as the final emotion category.

9. The weakly supervised EEG emotion recognition method based on spatiotemporal multi-instance learning according to claim 1, characterized in that, The final emotion category prediction result also includes a spatiotemporal attention importance map generation step, which specifically includes: Calculate the self-attention scores of the category labels and all time instances in the time-aware MIL pooling branch, and obtain the importance weight of each time window by softmax normalization to form a time importance graph; Calculate the self-attention scores of the category labels and all spatial instances in the spatially aware MIL pooling branch, and obtain the importance weight of each EEG channel after softmax normalization to form a spatial importance map.

10. A weakly supervised EEG emotion recognition system based on spatiotemporal multi-instance learning, characterized in that, The method for implementing the weakly supervised EEG emotion recognition method based on spatiotemporal multi-instance learning as described in any one of claims 1-9 includes: a data preprocessing module, a dual-branch instance feature extractor construction module, a spatiotemporal dual-branch multi-instance learning pooling architecture construction module, a model training module, and an emotion category prediction module. The data preprocessing module is used to preprocess the acquired EEG signal to obtain the differential entropy feature matrix; The dual-branch instance feature extractor construction module is used to construct a dual-branch instance feature extractor, including a Time-Inception module and a Space-Inception module, and the differential entropy feature matrix is ​​input into the Time-Inception module and the Space-Inception module respectively; The Time-Inception module extracts multi-scale temporal features based on multi-scale convolution and organizes these multi-scale temporal features into a spatial instance sequence according to EEG channels. The Space-Inception module performs global convolution and hemispherical convolution to extract the spatial relationship between the whole brain and the left and right hemispheres. It extracts spatial dimension features through temporal patch embedding and depthwise separable convolution, and organizes the spatial dimension features into a temporal instance sequence according to the time window. The spatiotemporal dual-branch multi-instance learning pooling architecture construction module is used to construct the time-aware MIL pooling branch and the space-aware MIL pooling branch. The time instance sequence is processed by the time-aware MIL pooling branch to obtain time packet features, and the space instance sequence is processed by the space-aware MIL pooling branch to obtain space packet features. The model training module is used to train an emotion category prediction model. It inputs the time packet features and spatial packet features into a linear classifier to obtain the emotion category probability distribution of the time branch and the spatial branch, constructs the total loss function, and trains the emotion category prediction model. The emotion category prediction module is used to output the final emotion category prediction result based on the emotion category prediction model.

Images