A multi-scale network-based emotion recognition method for EEG emotion recognition

Abstract

The application belongs to the technical field of electroencephalogram signal processing and analysis, and particularly relates to a multi-scale network-based emotion recognition method for EEG emotion recognition, comprising the following steps: EEG signal acquisition and preprocessing, original signals are segmented according to non-overlapping time windows, each window corresponds to a signal segment, and then the signal segments are stacked into a matrix; a mask alignment module is used for processing and aligning data, so as to improve the clarity of data and the reliability of analysis; a PTSwin-Net model, the model is composed of parallel Temporal Swin blocks, and finally feature extraction and fusion are performed through 3D convolution; a dynamic linear layer is used for a dynamic linear layer for classification, so that the model can capture complex patterns and make accurate predictions; emotion recognition, in each stage, the samples of each subject are sequentially taken as a test set, and the samples of the remaining subjects are combined into a training set. The PTSwin-Net model of the application realizes multi-scale space-time feature extraction of EEG signals through a parallel Temporal Swin block architecture.

Description

Technical Field

[0001] This invention belongs to the field of EEG signal processing and analysis technology, specifically relating to an emotion recognition method based on multi-scale networks for EEG emotion recognition. Background Technology

[0002] Emotion recognition plays a crucial role in healthcare and human-computer interaction, with the potential to significantly improve experiences in areas such as mental health monitoring, market analysis, education, and training. With advancements in artificial intelligence, emotion recognition models have made significant progress in understanding and interpreting human emotions. These electroencephalography (EEG)-based models provide important support for individuals who struggle to express emotions. EEG stands out for its ability to directly capture brain activity, making it particularly effective in applications requiring realistic and accurate emotion detection. EEG-based emotion recognition offers a powerful solution for advancing emotion recognition in key areas.

[0003] Most research focuses on within-subject emotion recognition, which requires extensive data collection, typically lasting 30 to 60 minutes, to learn individual-relevant EEG representations. This time-consuming process limits practical applications, highlighting the need for methods with cross-subject generalization capabilities. Significant inter-individual EEG variability poses a challenge to cross-subject recognition. For example, the SEED dataset shows an accuracy drop from 96% for within-subject recognition to 58% for cross-subject recognition using a general classifier. This drop stems from individual differences in EEG-based emotion representations. However, psychology and neuroscience support consistent emotion representation across subjects, and stable neural patterns have been identified for emotion representation, suggesting the potential of cross-subject algorithms. EEG signals, known for their high temporal resolution, are increasingly being integrated into deep learning models, particularly in Natural Language Processing (NLP). Previous EEG emotion recognition research focused on frequency domain analysis, neglecting temporal and spatial information, which limits its sensitivity to non-stationary signals and understanding of EEG structure, highlighting the necessity of spatiotemporal fusion. The fusion of spatiotemporal information leverages various deep learning models for feature extraction. In spatiotemporal information models for EEG sentiment recognition, STILN uses CNNs and Bi-LSTMs, but struggles with long-range dependencies. Parallel Transformer and 3D-CNN methods integrate spatiotemporal features, but lack flexibility in modeling global dependencies. These challenges highlight the advantages of Transformer models in effectively learning global and complex spatiotemporal relationships in EEG data. In spatiotemporal Transformer-based EEG sentiment recognition, the Swin Transformer fuses spatial and temporal information through its attention mechanism, using four-dimensional artificial EEG features to reduce noise interference. Similarly, Bi-ViTNet integrates spatial and temporal representations through bi-branch linear embeddings and a Transformer encoder to extract spatiotemporal features. However, these methods fail to capture the unique spatiotemporal dependencies of EEG signals, which differ significantly from visual or temporal data, limiting their effectiveness in EEG applications. On the other hand, stacking layers in deep learning models can improve performance. As the model deepens, the learned representations become more biased towards higher-level features. Combining low-level and high-level representations has become a promising approach to improve emotion detection performance. Although many recent studies have achieved significant accuracy, most existing EEG-based emotion recognition methods rely on fixed-duration time windows for instance segmentation. Therefore, it is necessary to provide a novel emotion classification method to address these issues. Summary of the Invention

[0004] To address the technical problem that existing EEG-based emotion recognition methods rely on fixed-duration time windows for instance segmentation, this invention provides a multi-scale network-based emotion recognition method for EEG emotion recognition.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0006] A multi-scale network-based emotion recognition method for EEG emotion recognition includes the following steps:

[0007] S1. EEG signal acquisition and preprocessing: The raw signal is divided into non-overlapping time windows, each window corresponds to a signal segment, and then these signal segments are stacked into a matrix.

[0008] S2, Mask Alignment Module, is used to process and align data, thereby improving data clarity and the reliability of analysis;

[0009] The S3 and PTSwin-Net models consist of parallel Temporal Swing blocks, and feature extraction and fusion are finally performed through 3D convolution.

[0010] S4, Dynamic Linear Layer: A dynamic linear layer used for classification, enabling the model to capture complex patterns and make accurate predictions.

[0011] S5. Emotion recognition: In each stage, the samples of each subject are used as the test set, while the samples of the remaining subjects are merged into the training set.

[0012] The data standards used in S1 include: having complete scale information.

[0013] The mask alignment method in S2 is as follows:

[0014] The EEG signal was initially represented as Where C represents the number of EEG electrodes and T represents the total length of the signal; a discrete, non-overlapping window of length W is used to divide the signal into blocks, each block... Corresponding to by An index is a segment; then these blocks are stacked together to form a matrix. in Indicates the total number of segments;

[0015] A mask alignment module was introduced to align datasets by utilizing infinite values ​​in data processing; initially, infinite values ​​are used for padding to align segment lengths, thus ensuring data consistency and alignment; then, masking techniques are used to filter out the infinite values.

[0016] To align the length of the segments, an infinite value is applied as padding; the padding is represented as:

[0017] Z padded =concatenate(Z,padding)

[0018] Padding is an array filled with infinite values ​​to ensure that the data reaches the maximum segment length and maintains consistency and alignment.

[0019] The PTSwin-Net model parameters in S3 are set as follows: the input data size is Samples×C×W×1, where Samples is the number of samples, C is the number of channels, and W is the width; the patch size is 1×4×16, and after applying 3D convolution operations, the data is converted into... Each token has 96-dimensional features; a linear embedding layer maps the features of each token to an arbitrary dimension D; the kernel size and stride are set to the patch size to ensure uniform segmentation and processing of the input data; this operation extracts spatiotemporal features from each patch while reducing spatial and temporal resolution.

[0020] The method for feature extraction and fusion using 3D convolution in S3 is as follows:

[0021] The features extracted from the two Temporal Swin modules at different depths are as follows: and Where D1 and D2 represent the number of channels in the hidden layer of the two parallel time Swin modules, D1 corresponds to Temporal Swin module 1, and D2 corresponds to Temporal Swin module 3; Samples, and Let F1 and F2 represent the time frame, height, and width, respectively; the two features are concatenated along the channel dimension of the hidden layer; the following equation represents the concatenation of the two feature matrices F1 and F2 to obtain the combined feature matrix F. concat :

[0022]

[0023] By applying a 1×1×1 convolutional kernel K to the concatenated feature map F concat The fused output features F are obtained. out :

[0024] F out =3DConv(F concat ,K)

[0025] By maintaining a 1×1×1 kernel size, we ensure that the convolution effectively integrates channel information while preserving temporal dependencies.

[0026] The prediction method for the dynamic linear layer in S4 is as follows:

[0027] The fused features are then flattened and fed into a dynamic linear layer; the output feature F out First, it is flattened into a 1D vector, and then processed through a dynamic linear layer; this process is represented as:

[0028] F flat =Flatten(F out )

[0029] After being flattened, the input is transformed through a dynamic linear layer:

[0030] Y = W dynamic ·F flat +b dynamic

[0031] Among them, F flat W is the flattened eigenvector. dynamic It is a dynamically changing weight matrix, b dynamic It is a dynamically changing bias vector;

[0032] By applying a dynamic linear layer after flattening the input, the model gains the flexibility to adapt to different feature sizes, enabling it to effectively handle inputs of different shapes and learn from complex data patterns. Next, the Softmax function is applied to calculate the predicted probability of each class, allowing the model to identify the class with the highest probability as the final classification result. Cross-entropy loss is used as the loss function of PTSwin-Net to minimize the difference between the predicted label and the true label, promoting accurate classification.

[0033] In S5, emotion recognition is evaluated using the average accuracy of leave-one-out cross-validation.

[0034] Compared with the prior art, the beneficial effects of this invention are:

[0035] The PTSwin-Net model of this invention achieves multi-scale spatiotemporal feature extraction from EEG signals through a parallel Temporal Swin block architecture. The shallow modules capture local temporal features (such as transient signals corresponding to micro-expressions), while the deep modules extract global contextual features (such as persistent emotional states), and cross-dimensional feature fusion is achieved through 3D convolution. Furthermore, this invention employs a learnable weight matrix W_dynamic and a bias vector b_dynamic to dynamically adjust the classification boundary through end-to-end training. Combined with the intelligent padding technique of the mask alignment module, this invention can adaptively handle signal length variations caused by different sampling frequencies and individual differences. Attached Figure Description

[0036] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0037] The structures, proportions, sizes, etc. illustrated in this specification are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed herein, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0038] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention;

[0039] Figure 2 This is a schematic diagram illustrating the process of emotion data preprocessing in an embodiment of the present invention;

[0040] Figure 3 This is a schematic diagram illustrating the principle of the mask alignment module in an embodiment of the present invention;

[0041] Figure 4 This is a schematic diagram of the Temporal Swin block in an embodiment of the present invention;

[0042] Figure 5 This is a schematic diagram of the PTSwin-Net model principle in an embodiment of the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. These descriptions are only for further illustrating the features and advantages of the present invention, and not for limiting the claims of the present invention. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0044] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0045] This invention provides a multi-scale network-based emotion recognition method for EEG emotion recognition, such as... Figure 1As shown, it includes the following steps:

[0046] Step S1, as follows Figure 2 As shown, EEG signal acquisition and preprocessing: The EEG signal is initially represented as... Where C represents the number of EEG electrodes (channels), and T represents the total length of the signal. A discrete, non-overlapping window of length W is applied to divide the signal into blocks, each block... Corresponding to by An indexed segment. These blocks are then stacked together to form a matrix. in Indicates the total number of segments.

[0047] In step S2, a mask alignment module is introduced to align the dataset by utilizing infinite values ​​during data processing. Initially, infinite values ​​are used for padding to align segment lengths, thus ensuring data consistency and alignment. Then, a masking technique is used to filter out the mask.

[0048] To align the segment lengths, an infinite value is applied as padding. Padding can be represented as:

[0049] Z padded =concatenate(Z,padding)

[0050] Padding is an array filled with infinite values ​​to ensure that the data reaches the maximum segment length and maintains consistency and alignment.

[0051] Step S3, PTSwin-Net model. The model consists of parallel Temporal Swin blocks, and finally features are extracted and fused through 3D convolution.

[0052] Step S4, Dynamic Linear Layer. The dynamic linear layer used for classification enables the model to capture complex patterns and make accurate predictions.

[0053] Step S5, emotion recognition. In each stage, the samples of each subject are used as the test set, while the samples of the remaining subjects are merged into the training set.

[0054] Specifically, such as Figure 3 As shown in step S2, mask alignment is introduced to align the dataset by utilizing infinite values ​​during data processing. Initially, infinite values ​​are used for padding to align segment lengths, thereby ensuring data consistency and alignment. Then, masking techniques are used to filter out the mask.

[0055] To align the segment lengths, an infinite value is applied as padding. Padding can be represented as:

[0056] Z padded=concatenate(Z,padding)

[0057] Padding is an array filled with infinite values ​​to ensure that the data reaches the maximum segment length and maintains consistency and alignment.

[0058] Specifically, such as Figure 4 As shown in step S3, the size of the input data is Samples.

[0059] ×C×W×1, where Samples is the number of samples, C is the number of channels, and W is the width. The patch size is 1×4×16. After applying the 3D convolution operation, the data is transformed into… Each token has 96-dimensional features. A linear embedding layer maps the features of each token to an arbitrary dimension D. The kernel size and stride are both set to the patch size to ensure uniform segmentation and processing of the input data. This operation extracts spatiotemporal features from each patch while reducing both spatial and temporal resolution.

[0060] The Temporal Swin block, based on the Transformer architecture, aims to capture the global dependencies of EEG signals. Multi-head self-attention (MSA) plays a crucial role in its core components. MSA generates different attention distributions for the input using multiple attention heads, allowing information to be retrieved from different subspaces. Its basic principle is to map input features to multiple subspaces and compute attention vectors within each subspace using a dot product operation. Finally, these attention vectors from different subspaces are combined and mapped back to the original space to produce the final attention vector. The specific implementation of the MSA mechanism is shown below:

[0061]

[0062] Among them: headh i Let N represent the nth attention head, where N is the nth attention head. h This indicates the total number of heads of attention. Furthermore, Let d represent a linear mapping matrix, where d model Indicates the dimension of the input embedding. (Terminology) These correspond to the query, key, and value in the second header, respectively. The weight matrix is ​​defined as follows: Where d q ,d k ,d vThese represent the dimensions of the query, key, and value, respectively. In the Temporal Swin block, the standard Temporal Swin block is introduced to extract semantic features from three dimensions. Specifically, the Temporal Swin block consists of two consecutive components: one is Three-Dimensional Window Multi-Head Self-Attention (3DW-MSA), and the other is Three-Dimensional Shifted Window Multi-Head Self-Attention (3DSW-MSA). Before performing the computation of 3DW-MSA and 3DSW-MSA, the input sequence is first subjected to layer normalization. The results of 3DW-MSA and 3DSW-MSA are then subjected to layer normalization through residual connections before being input into the multilayer perceptron. The formulas for 3DW-MSA and 3DSW-MSA are as follows:

[0063]

[0064] variable and z l These represent the output characteristics of the 3D(S)W-MSA module and the FFN module, respectively. The terms 3DW-MSA and 3DSW-MSA refer to multi-head self-attention mechanisms based on 3D windows, using regular and moving window partitioning configurations, respectively.

[0065] Given a sequence of 3D tokens and a specified 3D window size, these windows are arranged to partition the input non-overlappingly. In other words, the input tokens are divided into distinct, non-overlapping 3D windows.

[0066] Because a multi-head self-attention mechanism is applied within each non-overlapping 3D window, the 3D moving window mechanism lacks connections across different windows. This limitation can restrict the representational power of the architecture. To address this issue, this embodiment extends the 2D moving window mechanism in the Swing Transformer to a 3D context, thereby introducing connections between windows while maintaining the efficient computation of self-attention based on non-overlapping windows. Given a total of T... ′ ×H ′ ×W ′ The input is given by a parameter, where each 3D window is P×M×M in size. For two consecutive layers, the self-attention module of the first layer uses a conventional window partitioning strategy. This yields... A non-overlapping 3D window. In contrast, the self-attention module of the second layer offsets the window partitioning configuration along the time, height, and width axes. Each of the tokens is compared to the self-attention module in the previous layer.

[0067] To highlight the contributions of different positions, this embodiment uses relative positional deviation when calculating self-attention, and rewrites the self-attention formula as follows:

[0068]

[0069] In the formula, B is the weight of the corresponding position.

[0070] An improved parallel temporal Swin module is proposed, which constructs temporal Swin modules of varying depths in parallel at different stages. This design allows for the simultaneous capture of feature information across multiple levels and scales. Specifically, the traditional temporal Swin module is divided into two independent stages, each configured with a different depth to extract rich temporal and spatial features.

[0071] In Temporal Swin module 1, the architecture begins with a shallow structure designed to capture the dynamic dependencies between spatial and temporal features. This stage alternates between 3D window multi-head self-attention and 3D offset window multi-head self-attention layers, with a depth of 1. Within this shallow layer, the attention module focuses on different regions of the input data, enabling the extraction of dynamic relationships between spatial and temporal features. The primary objective of this stage is to capture the evolution and interaction of these features over time.

[0072] In Temporal Swin module 3, the architecture is deepened, allowing for more detailed extraction of spatial patterns and dynamic relationships. This stage continues to alternate between 3DW-MSA and 3DSW-MSA layers, but the depth is increased to 3. In this deeper layer, the attention module operates over a wider range, enabling the capture of more complex and nuanced dynamic features. The increased depth helps extract richer representations of evolutionary patterns in the data, providing a more comprehensive understanding of dynamic dependencies in both space and time. Features extracted at different depths complement each other, forming an enhanced feature set and improving overall performance.

[0073] Specifically, in step S4, the overall process of implementing the training is as follows: Figure 5 As shown, to achieve the fusion of multi-scale features extracted at different depths using the TemporalSwin module, a 3D convolution-based feature fusion module was designed. This module aims to integrate two parallel TemporalSwin modules to capture the dynamic temporal dependencies extracted at different depths.

[0074] The features extracted from the two Temporal Swin modules at different depths are as follows:

[0075] and Where D1 and D2 represent the number of channels in the hidden layer of the two parallel time Swin modules, D1 corresponds to Temporal Swin module 1, and D2 corresponds to Temporal Swin module 3. Samples and Let F1 and F2 represent the time frame, height, and width, respectively. The two features are concatenated along the channel dimension of the hidden layer. The following equation represents the concatenation of the two feature matrices F1 and F2 to obtain the combined feature matrix F. concat :

[0076]

[0077] By applying a 1×1×1 convolutional kernel K to the concatenated feature map F concat In this embodiment, the fused output feature F is obtained. out :

[0078] F out =3DConv(F concat ,K)

[0079] By maintaining a 1×1×1 kernel size, we can ensure that the convolution effectively integrates channel information while preserving temporal dependencies.

[0080] The fused features are then flattened and fed into a Dynamic Linear (D-Linear) layer. The output feature F... out First, it is flattened into a 1D vector, and then processed through a dynamic linear layer. This process can be represented as:

[0081] F flat =Flatten(F out )

[0082] After being flattened, the input is transformed through a dynamic linear layer:

[0083] Y = W dynamic ·F flat +b dynamic

[0084] Among them, F flat W is the flattened eigenvector. dynamic It is a dynamically changing weight matrix, b dynamic It is a dynamically changing bias vector.

[0085] By applying a dynamic linear layer after flattening the input, the model gains the flexibility to adapt to different feature sizes, enabling it to effectively handle inputs of varying shapes and learn from complex data patterns. Next, a Softmax function is applied to compute the predicted probability for each class, allowing the model to identify the class with the highest probability as the final classification result. To optimize performance, cross-entropy loss is used as the loss function for PTSwin-Net, minimizing the difference between the predicted and true labels and promoting accurate classification.

[0086] This invention provides PTSwin-Net, an EEG-based emotion recognition model that utilizes a parallel-temporal Swin-Net structure for dimensionality transformation. Experimental results show that the model achieves classification accuracies of 93.92% and 86.76% on the SEED and SEED-IV datasets, respectively, demonstrating the effectiveness and innovation of this method in emotion recognition. Evaluations on the SEED and SEED-IV datasets show that PTSwin-Net outperforms current state-of-the-art methods, exhibiting stronger performance in emotion recognition. This model incorporates key modules such as data alignment, dynamic feature extraction, and classification, enabling it to adapt to individual differences and enhance its generalization ability, demonstrating its potential application in healthcare, psychological monitoring, and human-computer interaction through precise temporal and spatial EEG signal analysis.

[0087] The above description only illustrates the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention, and all such changes should be included within the protection scope of the present invention.

Claims

1. A multi-scale network-based emotion recognition method for EEG emotion recognition, characterized in that, Includes the following steps: S1. EEG signal acquisition and preprocessing: The raw signal is divided into non-overlapping time windows, each window corresponds to a signal segment, and then these signal segments are stacked into a matrix. S2, Mask Alignment Module, is used to process and align data, thereby improving data clarity and the reliability of analysis; The EEG signal was initially represented as ,in Indicates the number of EEG electrodes. Represents the total length of the signal; apply a length of Discrete, non-overlapping windows are used to divide the signal into blocks, each block Corresponding to by An index is a segment; then these blocks are stacked together to form a matrix. ,in Indicates the total number of segments; The mask alignment module was introduced to align datasets by utilizing infinite values ​​in data processing; initially, infinite values ​​were used for padding to align segment lengths, thereby ensuring data consistency and alignment. Then, use masking techniques to filter out infinite values; To align the length of the segments, an infinite value is applied as padding; the padding is represented as: Here, padding is an array filled with infinite values ​​to ensure that the data reaches the maximum segment length and maintains consistency and alignment; The S3 and PTSwin-Net models consist of parallel Temporal Swing blocks, and feature extraction and fusion are finally performed through 3D convolution. The PTSwin-Net model parameters are set as follows: the input data size is... ,in It is the sample size. It is the number of channels. It is the width; the patch size is... After applying 3D convolution operations, the data is transformed into... Each token has 96-dimensional features; a linear embedding layer maps the features of each token to arbitrary dimensions. The kernel size and stride are both set to the patch size to ensure uniform segmentation and processing of the input data; this operation extracts spatiotemporal features from each patch while reducing spatial and temporal resolution. S4, Dynamic Linear Layer: A dynamic linear layer used for classification, enabling the model to capture complex patterns and make accurate predictions. S5. Emotion recognition: In each stage, the samples of each subject are used as the test set, while the samples of the remaining subjects are merged into the training set.

2. The emotion recognition method based on multi-scale networks for EEG emotion recognition according to claim 1, characterized in that: The data standards used in S1 include: having complete scale information.

3. The multi-scale network-based emotion recognition method for EEG emotion recognition according to claim 1, characterized in that, The method for feature extraction and fusion using 3D convolution in S3 is as follows: The features extracted from the two Temporal Swin modules at different depths are as follows: and ,in and This represents the number of channels in the hidden layer of the two parallel-time Swin modules. Corresponding to Temporal Swin module 1, Corresponds to Temporal Swin module 3; , and Let these represent the time frame, height, and width, respectively; the two features are concatenated along the channel dimension of the hidden layer; the following equation represents the two feature matrices. and The concatenation of these features yields the combined feature matrix. : By convolution kernel Applied to the stitched feature map The fused output features are obtained. : By maintaining The size of the convolution kernel ensures that the convolution effectively integrates channel information while preserving temporal dependencies.

4. The multi-scale network-based emotion recognition method for EEG emotion recognition according to claim 1, characterized in that, The prediction method for the dynamic linear layer in S4 is as follows: The fused features are then flattened and fed into a dynamic linear layer; the output features are... First, it is flattened into a 1D vector, and then processed through a dynamic linear layer; this process is represented as: After being flattened, the input is transformed through a dynamic linear layer: in, These are the flattened feature vectors. It is a dynamically changing weight matrix. It is a dynamically changing bias vector; By applying a dynamic linear layer after flattening the input, the model gains the flexibility to adapt to different feature sizes, enabling it to effectively handle inputs of different shapes and learn from complex data patterns. Next, the Softmax function is applied to calculate the predicted probability of each class, allowing the model to identify the class with the highest probability as the final classification result. Cross-entropy loss is used as the loss function of PTSwin-Net to minimize the difference between the predicted label and the true label, promoting accurate classification.

5. The multi-scale network-based emotion recognition method for EEG emotion recognition according to claim 1, characterized in that, In S5, emotion recognition is evaluated using the average accuracy of leave-one-out cross-validation.

Images