CNN (Convolutional Neural Network) brain electrical emotion recognition method in combination with attention mechanism and Swin Transform

Abstract

The invention relates to a CNN (Convolutional Neural Network) brain electrical emotion recognition method in combination with an attention mechanism and a Swin Transform. Comprising the following steps: S1, performing fast Fourier transform on obtained electroencephalogram emotion data; s2, dividing the processed data into three paths, inputting the three paths of data into a convolution module, and obtaining features of different levels of time-frequency information; s3, respectively inputting the three paths of data into an attention mechanism module, and focusing on local time-frequency characteristics; s4, performing feature fusion on the obtained three paths of data; s5, inputting the data into a Swin Transform so as to capture an advanced time frequency characteristic; and S6, finally, the obtained features are remodeled and sent to a classifier for emotion recognition. The method solves the problems that a traditional electroencephalogram emotion recognition method is low in accuracy and poor in feature representativeness, and the high-performance classification level can be achieved only by analyzing electroencephalogram time-frequency signals.

Description

Technical Field

[0001] The present invention belongs to the field of electroencephalogram signal processing and pattern recognition, and specifically relates to a CNN electroencephalogram emotion recognition method combining an attention mechanism and Swin Transformer. Background Art

[0002] The statements in this part merely provide background technical information related to the present disclosure, and these statements may constitute prior art. In the process of implementing the present invention, the inventors found that at least the following problems exist in the prior art.

[0003] Electroencephalogram (EEG), as a physiological signal directly reflecting brain activities, has been widely applied in the field of emotion recognition in recent years. Usually, when facing long-term EEG records, professional doctors only visually analyze the waveform diagrams of multi-channel EEG signals, so the recognition of EEG features is time-consuming and inefficient. Emotion recognition is an important part of emotion computing, aiming to identify the emotional states of individuals from various physiological signals. Most previous studies focused on non-physiological factors such as facial expressions, speech signals, and body postures, but their reliability is difficult to guarantee. The non-invasive physiological electrical signals based on EEG signals are spontaneous and subjectively uncontrollable, and can more accurately reflect the true emotional states. Due to the characteristics of high temporal resolution and close correlation with brain activities, EEG signals have become the preferred signal source for emotion recognition research. By analyzing different frequency bands and spatial patterns in EEG signals, effective emotion features can be extracted to provide support for emotion classification.

[0004] However, EEG signals have strong individual differences and are interfered by factors such as noise and artifacts, resulting in certain challenges for the emotion classification task. Traditional EEG emotion recognition methods mostly rely on manual feature extraction, such as power spectral density, waveform, etc. Researchers have also made corresponding studies. For example, in 2009, Petrantonakis et al. extracted EEG signals in the frontal region for emotion recognition research, using a method of hybrid filtering and higher-order cross to classify six emotion categories; Lin et al. used the short-time Fourier transform to convert the time-domain EEG signals into the frequency domain, and used the energy spectra of five frequency bands, namely δ, θ, α, β, and γ, as the EEG features of emotions; Atkinson et al. used wavelet signals to extract features from EEG signals, and then used an improved radial basis function as a classifier to predict emotions. However, these methods have significant limitations: manual features (such as power spectral density, time-frequency energy) can only reflect the local statistical characteristics of EEG signals and often cannot fully capture the high-dimensional information and complex spatio-temporal relationships of EEG signals.

[0005] With the development of deep learning in recent years, traditional emotion recognition methods (such as methods based on behavioral and physiological signals) often suffer from problems such as insufficient accuracy and poor real-time performance, which provides a new opportunity for the research on emotion recognition from electroencephalogram (EEG) signals based on deep learning. Compared with the manually extracted features in traditional machine learning, deep learning models use an end-to-end approach to extract features and classify EEG data, reducing the error caused by human participation; therefore, the features extracted by deep learning models are more accurate and the classification effect is better.

[0006] In order to accurately express the hidden emotional states in EEG, researchers have tried to extract EEG features related to emotions from multiple source domains such as the time domain, frequency domain, time-frequency domain, and spatial domain. However, compared with the relatively small number of effective EEG samples, the high-dimensional EEG features composed of multi-source domain features bring many difficulties to the real-time application research of emotion-based brain-computer interfaces. For example, problems such as classifier overfitting, high computational cost, and poor real-time performance. Therefore, in the research on emotion recognition based on emotion-based brain-computer interfaces, when the types and numbers of the extracted EEG features are relatively large, the feature selection step is an important step. The feature selection step can eliminate the noise features with interference information in the high-dimensional EEG features and select a subset of EEG features that are discriminative for the emotion recognition task, thereby reducing the dimension of the EEG features. In short, the feature selection step can reduce the computational cost of the emotion classifier and improve the performance of emotion recognition based on EEG.

[0007] The commonly used feature selection models are mainly the following three types: Filter, such as the Ant Colony Optimization (ACO) algorithm; Wrapper, such as the Genetic Algorithm (GA); Embedded, such as the Robust Feature Selection (RFS) algorithm.

[0008] However, each feature selection method has its own advantages and disadvantages, such as redundant information and overfitting, high computational complexity, and the trade-off between time and frequency resolution. Just like the patent with the application number 202410778246.5 and the patent name "A Classification Method for EEG Emotion Recognition Based on Multi-Scale Spatiotemporal Feature Extraction Using CNN and Transformer", since the extracted feature information is in the time-frequency domain and spatio-temporal domain, weight allocation is required to balance the two types of feature information so that different features can complement each other. But this also makes such patents more complex.

[0009] Due to the high time resolution of EEG, whether it is possible to accurately perform emotion recognition of EEG signals by only analyzing the time-frequency signals of EEG, thereby reducing the disadvantages brought by the feature selection step, is the problem to be solved by the present invention. Summary of the Invention

[0010] In view of the above problems, the purpose of the present invention is to solve a part of the problems in the prior art, or at least alleviate these problems.

[0011] A CNN electroencephalogram (EEG) emotion recognition method combining an attention mechanism and Swin Transformer includes the following steps:

[0012] Perform a fast Fourier transform on the obtained EEG emotion data to obtain an EEG time-frequency signal;

[0013] Divide the EEG time-frequency signal into three paths and input it into a convolutional module to obtain three paths of feature data with different levels of time-frequency information;

[0014] Input the three paths of feature data into the attention mechanism module respectively, focus on the local time-frequency features, and obtain three paths of output data after the local self-attention layer;

[0015] Perform feature fusion on the obtained three paths of output data to obtain the fused data;

[0016] Input the fused data into the Swin Transformer module to capture high-level time-frequency features;

[0017] Reshape the obtained high-level time-frequency features and send them into a classifier for emotion recognition.

[0018] Furthermore, the specific formula for performing a fast Fourier transform on the obtained EEG emotion data is as follows:

[0019]

[0020] Wherein, E[k] is the DFT of the even terms x[2m], x[2m] represents downsampling the discrete-time sequence x[n], that is, only retaining the samples with even indices (n = 0, 2, 4,...), DFT represents the discrete Fourier transform, n is the time index of the discrete-time sequence x[n], and the value range is n = 0, 1,..., N - 1, and N is the length of the discrete sequence x[n] (total number of sampling points); O[k] is the DFT of the odd terms x[2m + 1], and the rotation factor is the key weight.

[0021] The convolutional module includes three convolutional branches, and the convolutional kernel sizes of each branch are the same; the first convolutional branch from top to bottom includes a convolutional structure stacked in three layers (convolutional layer, batch normalization layer, and activation layer); the second convolutional branch from top to bottom includes a convolutional layer, a max-pooling layer, and a convolutional structure stacked in three layers (convolutional layer, batch normalization layer, and activation layer); the third convolutional branch from top to bottom includes a convolutional layer, a max-pooling layer, a convolutional layer, a max-pooling layer, and a convolutional structure stacked in three layers (convolutional layer, batch normalization layer, and activation layer). Figure 1The steps inside the dashed line in this part are performed three times.

[0022] The attention mechanism module is a local self-attention mechanism module, and the specific definition is as follows

[0023] Q = query_cinv(x), K = key_conv(x), V = value_conv(x)

[0024] Among them, query_conv, key_conv, and value_conv are convolutional layers, and x is the input; Q, K, and V respectively represent Query, Key, and Value.

[0025] Furthermore, input the three-way feature data into the attention mechanism module respectively, focus on the local time-frequency features, and obtain the three-way output data after the local self-attention layer, including the following steps:

[0026] Calculate Query, Key, and Value: Calculate the input Query, Key, and Value through a 3x3 convolutional layer respectively, and the formula is as follows:

[0027] Q = query_cinv(x), K = key_conv(x), V = value_conv(x)

[0028] In the formula, query_conv, key_conv, and value_conv are convolutional layers, and x is the input;

[0029] Calculate the attention score: Calculate the attention score by performing matrix multiplication on the query (Query) and the key (Key);

[0030] A = Q · K T

[0031] Among them, A is the attention score matrix, with a shape of (B, C, H×W); B represents Batch, that is, the batch size, C represents Channel, that is, the number of channels, and H×W represents Height×Width, that is, height × width;

[0032] After obtaining the attention score matrix, apply the Softmax operation along the last dimension to normalize the scores:

[0033] A softmax = Softmax(A, dim = -1)

[0034] Use the calculated attention scores to perform weighted aggregation on the value tensor V to obtain the aggregated output:

[0035] O = A softmax · V​​

[0036] Among them, O is the output tensor after applying attention;

[0037] Reshaping and residual connection: Reshape the output tensor O to the same spatial dimension as the original input feature map, and then apply the residual connection to add the input X and the output O to obtain the output data:

[0038] Output = O + X

[0039] Among them, the input X refers to the EEG time-frequency signal X[k].

[0040] The Swin Transformer module includes a Patch Embedding layer (embedding layer), a Layer Normalization layer (normalization layer), a Window Attention layer (window attention layer), and an MLP layer (multi-layer perceptron); remove the classification layer of the MLP layer so that the data can be input into the next module for classification.

[0041] Furthermore, the Window Attention layer includes a self-attention mechanism, a sliding window mechanism, and a window shift mechanism; the core formula of the self-attention mechanism is as follows:

[0042]

[0043] Among them, Q, K, and V represent Query, Key, and Value respectively, is the dimension of the key;

[0044] In Swin Transformer, the self-attention calculation is restricted within each window; assuming the window size is M×M, the self-attention calculation within each window can be expressed as:

[0045]

[0046] Among them, Q window , V window are the query, key, and value within the window respectively;

[0047] The window shift mechanism is that every other layer, the position of the window is shifted, and the shift amount is (M 2 , M 2 ).

[0048] Furthermore, input the fused data into the Swin Transformer module to capture high-level time-frequency features, including the following steps:

[0049] The fused data is subjected to Patch Embedding through a convolutional layer, which divides the image into multiple non-overlapping patches and maps each patch to a high-dimensional vector; a patch refers to dividing the input image into multiple non-overlapping local small blocks, and each small block is a local area (for example, a square of 16×16 pixels).

[0050] Then the data is normalized through the LN layer.

[0051] After normalization, the data enters the Window Attention layer, and its main formula is as follows:

[0052]

[0053] Among them, Q, K, and V represent Query, Key, and Value respectively. is the dimension of the key.

[0054] Finally, the data enters the LN layer again and then enters the MLP layer.

[0055] The CNN electroencephalogram emotion recognition method combining the attention mechanism and the Swin Transformer also includes using accuracy for evaluation, and its formula is:

[0056]

[0057] In the formula: TP is the true positive example, that is, the number of samples that are actually positive examples and are correctly predicted; TN is the true negative example, that is, the number of samples that are actually negative examples and are correctly predicted; FP is the false positive example, that is, the number of samples that are actually negative examples and are wrongly predicted; FN is the false negative example, that is, the number of samples that are actually positive examples and are wrongly predicted.

[0058] A computer-readable storage medium stores a computer program thereon, and when the computer program is executed by a processor, it implements the steps of the CNN electroencephalogram emotion recognition method combining the attention mechanism and the Swin Transformer.

[0059] The present invention has the following beneficial effects:

[0060] 1. The present invention proposes a local attention mechanism, which can provide focus on local structures, accelerate the transmission of local information, help the model efficiently capture local detailed information, and enhance the expression ability of local features;

[0061] 2. The present invention applies the variant Swin Transformer model of the Transformer model to the field of electroencephalogram emotion recognition, providing a new idea for the development of electroencephalogram emotion recognition tasks;

[0062] The present invention proposes a high-performance EEG emotion recognition model that only needs to analyze EEG time-frequency signals. This model combines a CNN network optimized by an attention mechanism with a Swin Transformer network. The CNN is used to extract local features of the EEG signal, the attention mechanism is used to capture local detailed information, and then the Swin Transformer module is used to further capture long-range dependencies and global context information in the signal, and finally classification is performed to improve the accuracy of emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 is a framework diagram of a CNN EEG emotion recognition method combining an attention mechanism and a Swin Transformer provided by the present invention;

[0064] Figure 2 is a framework diagram of the attention convolutional network of the feature preliminary extraction module of the present invention;

[0065] Figure 3 is a detailed framework diagram of the attention mechanism of the present invention;

[0066] Figure 4 is a framework diagram of the Swin Transformer of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0067] The following further describes the present invention with reference to the accompanying drawings. The embodiments of the present invention are only used to illustrate the present invention and do not limit the present invention. Without departing from the technical idea of the present invention, various substitutions and changes made according to ordinary technical knowledge and customary means in the art shall be included within the scope of the present invention.

[0068] To solve the above technical problems, as Figure 1 shown, the applicant invented a CNN EEG emotion recognition method combining an attention mechanism and a Swin Transformer, including the following steps:

[0069] S1, performing a fast Fourier transform on the obtained EEG emotion data;

[0070] S2, dividing the processed data into three paths and inputting them into a convolutional module to obtain features of different levels of time-frequency information;

[0071] S3, then inputting the three paths of data into the attention mechanism module respectively to focus on local time-frequency features;

[0072] S4, performing feature fusion on the obtained three paths of data;

[0073] S5. Input the data into the Swin Transformer to capture high-level time-frequency features. The Swin Transformer (Sliding Window Transformer) is a hierarchical vision model based on the Transformer architecture, designed specifically for images and temporal signals (such as time-frequency diagrams).

[0074] S6. Finally, reshape the obtained features and send them into the classifier for emotion recognition.

[0075] This method solves the problems of low accuracy and poor feature representativeness in traditional EEG emotion recognition methods, and achieves a high-performance classification level by only analyzing EEG time-frequency signals.

[0076] The specific formula for the normalization in step S1 is as follows:

[0077]

[0078]

[0079] where E[k] is the DFT of the even terms x[2m], x[2m] represents downsampling the discrete-time sequence x[n], that is, only retaining the samples with even indices (n = 0, 2, 4,...), DFT represents the discrete Fourier transform, n is the time index of the discrete-time sequence x[n], with the value range n = 0, 1,..., N - 1, and N is the length (total number of sampling points) of the discrete sequence x[n]; O[k] is the DFT of the odd terms x[2m + 1], and the rotation factor is the key weight.

[0080] The 3 local attention convolution modules in steps S2 and S3, as Figure 1 shown, mainly include a convolution module and an attention module. The specific performance of the three-way data is as Figure 1 shown, which is: the first-way data from top to bottom undergoes calculations of a convolutional layer - batch normalization layer - activation layer - local attention layer three times; the second-way data first passes through a convolutional layer - max pooling layer, and then undergoes calculations of a convolutional layer - batch normalization layer - activation layer - local attention layer three times; the third-way data first passes through a convolutional layer - max pooling layer - convolutional layer - max pooling layer, and then undergoes calculations of a convolutional layer - batch normalization layer - activation layer - local attention layer three times. Among them, Figure 1 the steps inside the dotted line in this part indicate three times of execution.

[0081] where the convolution module in step S2 contains three convolutional branches in parallel. The convolutional kernel sizes of each branch are the same, and the data dimension changes after one calculation. This is to obtain features at different levels in the data. The convolution module includes the following steps:

[0082] A convolutional layer with a 3x3 convolutional kernel;

[0083] After passing through a max-pooling layer once, the model complexity and computational volume can be reduced, and more effective information can be obtained;

[0084] Pass through a convolutional layer with a 3x3 convolutional kernel again;

[0085] Pass through a batch normalization layer. The batch normalization layer is a technique for accelerating neural network training, aiming to solve the problems of vanishing gradients and exploding gradients in deep network training;

[0086] Then the data passes through a ReLU layer. ReLU mainly increases the non-linear representation ability of the neural network by performing non-linear transformations on the input values.

[0087] In step S3, the three-way data obtained is respectively fed into the attention mechanism module. Based on the self-attention mechanism, this patent proposes a local self-attention mechanism module, named the LAT module. It uses convolution to simulate the effect of a local window and calculates the attention through a local convolutional window, thereby focusing on local time-frequency features.

[0088] The main steps of the attention module are as follows:

[0089] Calculate Query, Key, and Value: The traditional attention mechanism obtains Query, Key, and Value through a linear calculation, while the LAT mechanism in this paper calculates the input Query, Key, and Value respectively through a 3x3 convolutional layer.

[0090] The LAT mechanism formula is as follows:

[0091] Q = query_cinv(x), K = key_conv(x), V = value_conv(x)

[0092] In the formula, query_conv, key_conv, and value_conv are convolutional layers, and x is the input. [[ID=�1]]

[0093] Calculate the attention score: Calculate the attention score by performing matrix multiplication on the query (Query) and the key (Key). Specifically, reshape the query and key tensors into matrices and calculate their dot product:

[0094] A = Q · K T

[0095] Where A is the attention score matrix, with a shape of (B, C, H×W).

[0096] After obtaining the original attention score, apply the Softmax operation along the last dimension to normalize the score:

[0097] A = Softmax(A, dim=-1)

[0098] This can ensure that the sum of the attention scores at each spatial position is 1, guaranteeing that each position can attend to other positions in the local window.

[0099] The calculated attention scores will be used to perform weighted aggregation on the value tensor (Value). That is, multiply the attention score matrix A softmax by the value tensor V to obtain the aggregated output:

[0100] O = A softmax · V

[0101] where O is the output tensor after applying attention.

[0102] Reshaping and residual connection: The output tensor O is reshaped to have the same spatial dimensions as the original input feature map. Then, a residual connection is applied by adding the input X and the output O, so that the model can learn the features modulated by the attention mechanism and the original features. This residual connection ensures that the network can both retain the original information and fuse the attention-weighted features.

[0103] S4. Feature fusion is performed on the three-way data obtained. The specific steps are as follows: The three-way features are fused by concatenation.

[0104] S5. The data is input into the Swin Transformer to capture high-level time-frequency features. The Swin Transformer module reduces the computational complexity by introducing a sliding window mechanism, which can extract local features and retain the global modeling ability of the Transformer.

[0105] The main structure includes: Patch Embedding layer, Layer Normalization layer, Window Attention layer, MLP layer. The data is calculated in the order of Patch Embedding layer - Layer Normalization layer - Window Attention layer - Layer Normalization layer - MLP layer. The functions of each layer will be introduced below.

[0106] Patch Embedding Layer (Embedding Layer): First, the input image undergoes Patch Embedding through a convolutional layer, which divides the image into multiple non-overlapping patches and maps each patch to a high-dimensional vector. Patch Embedding is a crucial step in converting image data into a form suitable for input to the ST model. By slicing the image and generating embedding representations for each patch, it significantly improves the model's performance in image tasks.

[0107] Layer Normalization Layer (Normalization Layer): Then the data passes through the LN layer. LN is a key module for normalizing each input layer. It helps improve the training stability of the model, avoid the problems of vanishing gradients and exploding gradients, and accelerate the training process. The operation of LN is to normalize the input of each layer so that the mean of each input is 0 and the variance is 1. Specifically, assume the input of a certain layer is the vector x i , whose original mean is μ and standard deviation is σ. LN will perform the following normalization:

[0108]

[0109] where β is the offset and γ is the scaling factor.

[0110] Window Attention Layer (Window Attention Layer): After normalization, the data enters the Window Attention layer. The Window Attention layer includes self-attention mechanism, sliding window mechanism, and window offset mechanism.

[0111] The core formula of the self-attention mechanism is as follows:

[0112]

[0113] where Q, K, and V represent Query, Key, and Value respectively, is the dimensional ability of the key.

[0114] In the Swin Transformer, the self-attention calculation is restricted within each window. Assuming the window size is M×M, the self-attention calculation within each window can be expressed as:

[0115]

[0116] where Q window , V window are the query, key, and value within the window respectively.

[0117] To capture information between different windows, the ST module introduces a window offset mechanism. Specifically, every other layer, the position of the window is offset by (M 2 , M 2 ). In this way, information between different windows can be interacted through the offset windows. Inside each window, self-attention is calculated.

[0118] Window Attention enhances the model's global information exchange ability within local windows and gradually expands the receptive field through window offset, ultimately capturing long-range dependencies.

[0119] MLP layer (Multi-Layer Perceptron): Finally, the data enters the MLP layer after entering the LN layer again. MLP is a simple and effective neural network structure, especially suitable for processing structured data or data with extracted features. It constructs complex non-linear mappings by stacking multiple fully connected layers and activation functions. To classify more accurately, the classification layer of the MLP is removed in this patent, and the data is input to the next module for classification.

[0120] S6. Finally, the obtained features are reshaped and sent into the classifier for emotion recognition. The data inputs from three channels are fused through the fully connected layer, and the probability prediction of the motor imagery task is output through the Softmax function.

[0121] The formula for evaluation using accuracy is:

[0122]

[0123] In the formula: TP is the true positive, that is, the number of samples that are actually positive and are correctly predicted; TN is the true negative, that is, the number of samples that are actually negative and are correctly predicted; FP is the false positive, that is, the number of samples that are actually negative but are wrongly predicted; FN is the false negative, that is, the number of samples that are actually positive but are wrongly predicted.

[0124] The main innovations of the present invention are concentrated in three parts: S2, S3, and S5 (referring to the introduction of the attention mechanism, the hierarchical structure of the convolutional module, and the introduction of the SWIN Transformer). By reasonably applying the attention convolutional module and the Swin Transformer module, the problems of low accuracy and neglect of local time-frequency features in EEG signals in traditional methods are effectively solved, and the average classification accuracy of EEG signal emotion recognition is improved. This method not only solves the problems of low accuracy and poor feature representativeness in traditional EEG emotion recognition methods, but also can achieve a high-performance classification level by only analyzing the EEG time-frequency signals. The proposed model uses the DEAP dataset as the experimental data source. DEAP (Dataset for Emotion Analysis using Physiologicalsignals) is a multimodal dataset for emotion analysis, aiming to study the relationship between human emotions and physiological signals. It provides rich data support for emotion recognition research by recording the physiological signals and subjective emotion scores of participants while watching music videos. The proposed model achieved an average classification accuracy of 96.38% on the DEAP dataset, showing a certain improvement compared with the existing benchmark models.

[0125] A computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the CNN EEG emotion recognition method combining the attention mechanism and the Swin Transformer are implemented.

[0126] The method illustrated in the above embodiments can be specifically implemented by a computer chip or an entity, or by a product with a certain function.

[0127] It should also be noted that the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, commodity or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of another identical element in the process, method, commodity or device including the element.

[0128] The above embodiments should be understood as being only for illustrative purposes of the present invention and not for limiting the protection scope of the present invention. After reading the content recorded in the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.

Claims

1. A CNN electroencephalogram emotion recognition method combining an attention mechanism and Swin Transformer, characterized in that It includes the following steps: Perform fast Fourier transform on the obtained EEG emotion data to obtain EEG time-frequency signals; Divide the EEG time-frequency signals into three paths and input them into the convolution module to obtain three paths of feature data with different levels of time-frequency information; Input the three paths of feature data into the attention mechanism module respectively, focus on the local time-frequency features, and obtain three paths of output data after the local self-attention layer; Perform feature fusion on the obtained three paths of output data to obtain the fused data; Input the fused data into the Swin Transformer module to capture high-level time-frequency features; Reshape the obtained high-level time-frequency features and send them into the classifier for emotion recognition.

2. The CNN electroencephalogram emotion recognition method combining an attention mechanism and a Swin Transformer according to claim 1, characterized in that, The specific formula for performing fast Fourier transform on the obtained EEG emotion data is as follows: Among them, E[k] is the DFT of the even terms x[2m], where x[2m] represents downsampling the discrete-time sequence x[n], that is, only retaining the samples with even indices (n = 0, 2, 4, …), DFT represents the discrete Fourier transform, n is the time index of the discrete-time sequence x[n], with the value range n = 0, 1, …, N - 1, and N is the length of the discrete sequence x[n] (total number of sampling points); O[k] is the DFT of the odd terms x[2m + 1], and the rotation factor is the key weight.

3. The CNN electroencephalogram emotion recognition method combining an attention mechanism and Swin Transformer according to claim 1, wherein The convolution module includes three convolutional branches, and the convolutional kernel sizes of each branch are the same; specifically: The first convolutional branch from top to bottom includes a convolutional structure stacked in three layers (convolutional layer, batch normalization layer, and activation layer); the second convolutional branch from top to bottom includes a convolutional layer, a max pooling layer, and a convolutional structure stacked in three layers (convolutional layer, batch normalization layer, and activation layer); the third convolutional branch from top to bottom includes a convolutional layer, a max pooling layer, a convolutional layer, a max pooling layer, and a convolutional structure stacked in three layers (convolutional layer, batch normalization layer, and activation layer).

4. The CNN electroencephalogram emotion recognition method combining an attention mechanism and a Swin Transformer according to claim 1 or 3, characterized in that The attention mechanism module is a local self-attention mechanism module, and the specific definition is as follows Q = query_cinv(x), K = key_conv(x), V = value_conv(x) Where query_conv, key_conv, and value_conv are convolutional layers, and x is the input; Q, K, and V represent Query, Key, and Value respectively.

5. The CNN-based EEG emotion recognition method combining the attention mechanism and Swin Transformer according to claim 4, wherein Input the three paths of feature data into the attention mechanism module respectively, focus on the local time-frequency features, and obtain three paths of output data after the local self-attention layer, including the following steps: Calculate Query, Key, and Value: Calculate the input Query, Key, and Value through a 3x3 convolutional layer respectively, and the formula is as follows: Q = query_cinv(x), K = key_conv(x), V = value_conv(x) In the formula, query_conv, key_conv, and value_conv are convolutional layers, and x is the input; Calculate the attention score: Calculate the attention score by performing matrix multiplication on the query (Query) and the key (Key); A = Q · K T Where A is the attention score matrix, with a shape of (B, C, H×W); B represents Batch, that is, the batch size, C represents Channel, that is, the number of channels, and H×W represents Height×Width, that is, height × width; After obtaining the attention score matrix, apply the Softmax operation along the last dimension to normalize the scores: A softmax = Softmax(A, dim=-1) Use the calculated attention scores to perform weighted aggregation on the value tensor V to obtain the aggregated output: O = A softmax ·V Where O is the output tensor after applying attention; Reshaping and residual connection: Reshape the output tensor O to the same spatial dimension as the original input feature map, and then apply the residual connection by adding the input X to the output O to obtain the output data: Output = O + X where the input X refers to the EEG time-frequency signal X[k].

6. The CNN-based EEG emotion recognition method combining the attention mechanism and Swin Transformer according to claim 1, characterized in that The Swin Transformer module includes a Patch Embedding layer (embedding layer), a Layer Normalization layer (normalization layer), a Window Attention layer (window attention layer), and an MLP layer (multi-layer perceptron); remove the classification layer of the MLP layer so that the data can be input into the next module for classification.

7. The CNN-based EEG emotion recognition method combining an attention mechanism and Swin Transformer according to claim 6, wherein The Window Attention layer includes a self-attention mechanism, a sliding window mechanism, and a window shift mechanism; the core formula of the self-attention mechanism is as follows: Among them, Q, K, and V represent Query, Key, and Value respectively, is the dimension of the key; In Swin Transformer, the self-attention calculation is restricted within each window; assuming the window size is M×M, the self-attention calculation within each window can be expressed as: Among them, Q window , V window are the query, key, and value within the window respectively. The window offset mechanism is that the position of the window is offset every other layer, and the offset amount is (M 2 , M 2 ).

8. The CNN electroencephalogram emotion recognition method combining an attention mechanism and Swin Transformer according to claim 6, characterized in that, Input the fused data into the Swin Transformer module to capture high-level time-frequency features, including the following steps: Perform Patch Embedding on the fused data through a convolutional layer, divide the image into multiple non-overlapping patches, and map each patch to a high-dimensional vector; Then the data is normalized through the LN layer; After normalization, the data enters the Window Attention layer, and its main formula is: where Q, K, and V represent Query, Key, and Value respectively, is the dimension of the key; Finally, the data enters the MLP layer after entering the LN layer again.

9. The CNN electroencephalogram emotion recognition method combining an attention mechanism and Swin Transformer according to claim 1, wherein It also includes using accuracy for evaluation, and its formula is: In the formula: TP is the true positive, that is, the number of samples that are actually positive and are correctly predicted; TN is the true negative, that is, the number of samples that are actually negative and are correctly predicted; FP is the false positive, that is, the number of samples that are actually negative but are wrongly predicted; FN is the false negative, that is, the number of samples that are actually positive but are wrongly predicted.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the CNN EEG emotion recognition method combining the attention mechanism and Swin Transformer described in any one of claims 1 to 9.

Images