Emotion recognition method of hierarchical Mama model based on electroencephalogram signals

Abstract

The invention discloses an electroencephalogram signal-based hierarchical Mama model emotion recognition method, which creatively adopts a local-to-global hierarchical processing mode to perform emotion recognition, integrates a bidirectional patch division mechanism on the basis of a Mama network, and remarkably improves the ability of understanding complex spatial distribution characteristics. According to the network, deep analysis is carried out on a channel level and a local area level through a fine-grained information extraction strategy. Specifically, the method comprises the following steps: firstly exploring a relationship among multiple frequency bands by adopting a channel multi-layer perceptron; then, realizing multi-scale brain information efficient learning in a specific emotional state in combination with a dynamic convolution and selective kernel network technology; furthermore, global spatial information is captured in different brain connection modes through an innovative bidirectional patch segmentation method.

Description

Technical Field

[0001] The invention belongs to the field of EEG emotion recognition. Background Art

[0002] EEG emotion recognition is at the forefront of an interdisciplinary field that merges neuroscience, psychology, and computer science. Its goal is to use EEG signals to analyze and decipher an individual's emotional state. With advances in artificial intelligence and biomedical engineering, EEG emotion recognition has become a promising technology in healthcare, human-computer interaction, and psychological research. Neurological studies have established a strong connection between human emotions and specific regions of the cerebral cortex, such as the amygdala, frontal lobe, and parietal lobe. However, the involvement of these different brain regions has not been fully captured in current EEG emotion recognition techniques. In addition, different brain regions work together, highlighting the importance of extracting the complex relationships between these regions from a global perspective. Therefore, a hierarchical framework that exploits information from brain regions across multiple scales is crucial. This approach requires learning local-to-global spatial information, which can enable a deeper understanding of human emotional states and enhance the application of EEG emotion recognition.

[0003] Differential entropy is an indicator that quantifies the amount of information in a continuous random variable and is widely used to reflect the frequency domain characteristics of EEG signals. In addition, many models under the hierarchical network framework are based on differential entropy features and have achieved good performance. However, there are still several challenges in extracting local and global information from EEG signals:

[0004] First, it is crucial to explore how the model learns the features of differential entropy more comprehensively. Differential entropy is usually calculated within artificially defined frequency bands, such as the delta band (1-4Hz), theta band (4-8Hz), alpha band (8-13Hz), beta band (13-30Hz), and gamma band (30-50Hz). Although differential entropy has advantages in terms of accuracy and stability in emotion recognition compared to other frequency domain features, there is a lack of in-depth research on the interdependence features between these frequency bands. This oversight may lead to the loss of valuable information, thereby limiting the performance of emotion recognition models.

[0005] Second, the problem of how to naturally learn local information in specific brain regions and adaptively respond to different emotional states remains unsolved. Current hierarchical models often rely on researchers to manually divide the brain into multiple regions to learn local information [1,2,3]. This approach is inherently subjective and destroys the intrinsic connectivity between electrodes, thus limiting the effectiveness of the model. In addition, it is crucial to explore different brain patterns that appear in different emotional states, which is an aspect that is often overlooked in current research.

[0006] Finally, despite significant progress in various fields, transformer-based models are insufficient to accurately capture global spatial information in 2D EEG data. Known for its innovative self-attention mechanism and powerful parallel processing, the Transformer has made significant progress, especially in the Vit network that divides the data into patches. This approach effectively bridges the gap between transformers and classification tasks in 2D data (such as images) and has been successfully applied in EEG-based emotion recognition. However, when it comes to 2D EEG data, the traditional single-block segmentation method and the parallel structure of the transformer have difficulty in sensitively capturing the complex positional relationships between electrodes in 2D space. This limitation weakens the model's ability to effectively extract and represent global spatial information. Summary of the invention

[0007] The present invention provides an emotion recognition method based on a hierarchical Mamba model of electroencephalogram signals to improve the emotion recognition capability.

[0008] The present invention adopts a gradual structure from local to global on the basis of deeply mining the frequency band information in each electrode, and introduces a bidirectional patch division mechanism when extracting global spatial information to enhance its understanding of spatial distribution. Specifically, in order to fully mine the frequency band information, the present invention uses differential entropy features and a multi-layer perceptron (MLP) model that can adaptively combine multi-band information. Secondly, in order to enable the model to flexibly learn local information and adjust parameters according to different emotional states, the present invention combines dynamic convolution and selective kernel network technology to overcome the problem of artificially dividing regions while improving the model's emotion recognition ability. In recent years, models based on state space theory have received widespread attention. Mamba introduces an input-dependent parameter adjustment mechanism, which enables it to effectively model sequence data while maintaining linear time complexity. In addition, due to Mamba's cyclic structure, it is able to integrate position information into its modeling framework. Therefore, in order to enable the model to deeply understand the spatial distribution of the brain to effectively extract the global spatial features of EEG, the present invention adopts a horizontal and vertical patch division mechanism and combines the Mamba model to extract global information.

[0009] Based on the above, the present invention is divided into three levels, which systematically extracts fine-grained information of brain activity, takes into account both local and global feature extraction, and effectively improves the emotion recognition ability of the model.

[0010] The creativity of the present invention is mainly reflected in:

[0011] (1) MLP is creatively used as the initial encoding layer to capture the information between different frequency bands within the electrical channel.

[0012] (2) Combine dynamic convolution and selective kernel networks to learn local spatial information of different scales around each electrode under specific emotional states.

[0013] (3) A bidirectional patch segmentation strategy is proposed, combined with the Mamba model, to deeply learn the global spatial features of the brain. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a model diagram of the emotion recognition method based on the hierarchical Mamba model of EEG signals of the present invention.

[0015] Figure 2 This is a schematic diagram of the spatial mapping of the present invention.

[0016] Figure 3 This is a schematic diagram of the Fuse and Select operations in the present invention.

[0017] Figure 4 This is a schematic diagram of the Mamba module of the present invention. DETAILED DESCRIPTION

[0018] The specific implementation modes of the present invention are further described in detail below with reference to the accompanying drawings and examples.

[0019] Figure 1 It is a model diagram of the emotion recognition method of the hierarchical Mamba model based on EEG signals of the present invention, which includes the following steps:

[0020] Step 1: Differential entropy feature extraction

[0021] This step uses a 1-second non-overlapping window to capture the given EEG signal and extract the differential entropy feature. The following is the differential entropy feature calculation formula:

[0022] X=DE(t)

[0023] =-∫g(t)log g(t)dt (1) Where DE represents the differential entropy formula, t represents the EEG signal, and g(t) represents the probability density function of t. Because the EEG signal obeys the Gaussian distribution, the above formula can be expressed as:

[0024]

[0025] Among them, π is the ratio of the circumference of a circle to the diameter of a circle, e is a natural constant, μ and σ are 2 represents the mean and variance of the EEG signal t.

[0026] Step 2: Input data into the model

[0027] Step 2.1: Channel-level encoding module

[0028] In this step, the differential entropy features extracted from the EEG signal are fed into the channel-level encoding module as input. The channel-level encoding module learns the relationship between frequency bands within a single electrode and provides rich potential representations for the extraction of spatial information. Therefore, during the encoding process, each channel is independent of each other to ensure that spatial information is not integrated. Figure 1 As shown, the channel-level coding module consists of an MLP and a batch normalization layer. The MLP contains two fully connected layers (with a ReLU activation function between them), the input dimension of the MLP is 5, the output dimension is 20, and the intermediate layer dimension is 10. The present invention defines the process of the channel-level coding process as:

[0029] X c =CFE(X) (3)

[0030] Where CFE represents the channel-level coding module, X c ∈R C×D represents the output of the channel-level encoding module, and D is the output dimension of the MLP. Given an input X∈R C×F , where C represents the number of EEG signal channels and F represents the number of frequency bands. After encoding, the dimension F of each channel of the EEG signal is finally mapped to D.

[0031] In order to learn the spatial topological structure of the EEG and maintain the relative relationship between the positions of the EEG electrodes on the head, the present invention performs spatial projection on the EEG features. Figure 2 As shown, based on their relative positions, X c ∈R C×D Mapped to X c ∈R H×W×D , where C is the number of EEG channels, H and W represent the number of rows and columns of the two-dimensional matrix converted from the number of channels C. Based on the relative positions of the EEG channels, H and W are both set to 9 in the present invention, and the remaining positions in the two-dimensional matrix are supplemented with 0. c Dimension adjustment, becomes X p ∈R D×H×W , where D is the output dimension of MLP, H and W are the number of rows and columns after mapping the EEG channels into a two-dimensional matrix.

[0032] Step 2.2: Local Spatial Feature Extractor

[0033] The brain consists of different brain regions, each with its own function. When a person experiences emotions, each region of the brain will respond accordingly. In order to adapt to different brain regions and understand their responses when producing different emotions, the present invention designs a local space feature extractor to learn the features of the local brain space. Two convolution kernels of different sizes are used to extract the local spatial features of the brain. pDynamic convolution is performed, and the sizes of the two convolution kernels are 3 and 5 respectively. After obtaining the results of the two dynamic convolutions, they are aggregated through Fuse and Select to obtain the output of the local space feature extractor. The following is a detailed introduction to the local space feature extractor:

[0034] This section includes three steps: Split, Fuse, Select.

[0035] In Split, convolution kernels of size 3×3 and 5×5 are used to filter X p Perform dynamic convolution to obtain and After dynamic convolution, the channel dimension of the two changes from D to L, where L is twice D, that is, 40. Figure 1 As shown in the local spatial feature extractor in the present invention, the present invention sets K groups of parameters that share the same kernel size and input / output dimensions for the convolution kernel of size 3×3, and uses the attention weight Aggregate them, where represents the weight of the kth group of dynamic convolution when the convolution kernel size is 3×3, and K is the number of emotion categories. p Global average pooling is performed to compress global spatial information, and then two fully connected layers are used (with ReLU activation function between them), and softmax is used to calculate the weight of each group of convolution kernel parameters. Finally, K groups of convolutions are obtained by weight aggregation. For the convolution kernel of size 5×5, K groups of parameters sharing the same kernel size and input / output dimensions are set, and the attention weights are Aggregate them, where represents the weight of the kth group of dynamic convolution when the convolution kernel size is 5×5, and K is the number of emotion categories. p Global average pooling is performed to compress global spatial information, and then two fully connected layers are used (with ReLU activation function between them), and softmax is used to calculate the weight of each group of convolution kernel parameters. Finally, K groups of convolutions are obtained by weight aggregation.

[0036] Fuse first and Add and fuse into U, and then generate the feature vector S∈R through global average pooling L , where L is the channel dimension, R represents the real number domain, and S is the result of global average pooling of a two-dimensional matrix with H rows and W columns. This vector counts the feature information of each channel and is further processed by the fully connected layer to generate a vector Z∈RL .

[0037] In the Select step, two learnable parameter matrices are randomly initialized and After operating them with Z respectively, we can get and Weight and Finally and Multiply each weight and add them together to get the output V∈R of the local spatial feature extractor L×H×W . Figure 3 This is a schematic diagram of Fuse and Select.

[0038] Step 2.3: Global Space Feature Extractor

[0039] The present invention explores different connection modes between electrodes through different patch division methods. This step consists of two branches, which use two different patch division methods, namely horizontal patch and vertical patch. For the result V∈R L×H×W , the horizontal patch is divided into 9 patches according to the H dimension, and the vertical patch is divided into 9 patches according to the W dimension. L×H×W Convert to sequence M H ∈R J×(P×L) Corresponding to the lateral branch, M V ∈R J×(P×L) Corresponding to the vertical branch, where L represents the number of channels, P is the size of the patch in each channel, and J is the length of the sequence. Then, M is transformed into H ∈R J×(P×L) Convert to M H ∈R J×G , M V ∈R J×(P×L) Convert to M V ∈R J×G , and add the position code E H ∈R J ×G and E V ∈R J×G :

[0040] M H =M H Q H +E H (4)

[0041] M V =M V Q V +EV (5)

[0042] Where Q H ,Q V ∈R (P×L)×G is a learnable matrix corresponding to the horizontal patch and the vertical patch respectively, and G is set to 450. Next, M H and M V Send to Mamba H and Mamba V . Mamba H and Mamba V Each contains 3 layers of Mamba blocks, Figure 4 The following is a detailed description of the Mamba block in the horizontal patch branch. H After LayerNorm and linear mapping, two branches are generated:

[0043] Mx=Linear1(LN(M H )) (6)

[0044] Mz=Linear2(LN(M H )) (7)

[0045] Where LN represents LayerNorm, Linear1 and Linear2 represent two linear mappings with different parameters. Is to M H After normalization, it is obtained by Linear1 linear mapping. Is to M H After normalization, it is obtained by Linear2 linear mapping, J is the length of the sequence, D in is the dimension after mapping, D in Set to 540. Mx enters SSM after one-dimensional convolution and SiLU activation function:

[0046] Mxc=SiLU(Conv1d(Mx)) (8)

[0047] Mxo=SSM(Mxc) (9)

[0048] Conv1d represents one-dimensional convolution. In the present invention, the convolution kernel size of the one-dimensional convolution is set to 4, the step size is 1, the padding is 3, and SSM represents the state space model. At the same time, Mz is activated by SiLU function to generate Mzo:

[0049] Mzo=SiLU(Mz) (10)

[0050] The SSM-based model is inspired by the state space expression. The system maps input to output through hidden states. The present invention uses A to represent the system matrix, B to represent the control matrix, and I to represent the output matrix. For the SSM in formula 9, it uses hidden states Map Mxc to Mxo, D in Same as the second dimension of Mxc, N represents the state dimension, and in the present invention, N is 8.

[0051] The following is a detailed introduction to SSM:

[0052]

[0053] Linear B 、Linear I 、Linear Δ Denotes three linear mappings with different parameters used to generate B, I, and Δ, B∈R J×N It is done by linear mapping Mxc B We get, I∈R J×N It is done by linear mapping Mxc I Got it, It is done by linear mapping Mxc Δ Got it, is a randomly initialized learnable parameter matrix, EX_RE is a function defined in the present invention for expanding tensors of different dimensions to the same dimension, EX_RE A 、EX_RE B 、EX_RE Δ are different forms acting on A, B, and Δ respectively. They are the results obtained after A, B, and Δ are expanded. is A′ converted by the time scale parameter Δ′, is obtained by transforming B′ through the time scale parameter Δ′, represents the parameter matrix of the hidden state h at the jth step, express In the parameter matrix of step j, ⊙ represents the multiplication of corresponding elements, express The parameter matrix at step j, I j ∈R N represents the j-th row vector of I, Represents the j-th element of Mxo.

[0054] As described above, First, we transform it into B∈R through linear mapping with different parameters. J×N , I∈R J×N, Then through the function EX_RE A 、EX_RE B 、EX_RE Δ Map A, B, Δ to the same dimension Δ′ converts A′ into Convert B′ to Similarly, through the function EX_RE, Mxc is changed from becomes And through the hidden state Calculate Mxo.

[0055] Finally, Mxo and Mzo are multiplied and passed through a linear layer to be combined with M H After adding, we get the result of the horizontal patch branch:

[0056] Mo H =M H +Linear3(Mxo·Mzo) (12)

[0057] Linear3 represents a linear mapping, M H ∈R J×G is the input of the Mamba block, Mo H ∈R J×G is the output of the Mamba block, G size is 450, Mamba H It contains three layers of Mamba blocks, and the output of the last layer of Mamba blocks is the final result.

[0058] The following is a detailed introduction to vertical patch branches. V After LayerNorm and linear mapping, two branches are generated:

[0059] Mx=Linear1(LN(M V )) (13)

[0060] Mz=Linear2(LN(M V )) (14)

[0061] Where LN represents LayerNorm, Linear1 and Linear2 represent two linear mappings with different parameters. Is to M H After normalization, it is obtained by Linear1 linear mapping. Is to M H After normalization, it is obtained by Linear2 linear mapping, J is the length of the sequence, D in is the dimension after mapping. Mx enters SSM after one-dimensional convolution and SiLU activation function:

[0062] Mxc=SiLU(Conv1d(Mx)) (15)

[0063] Mxo=SSM(Mxc) (16)

[0064] Conv1d represents one-dimensional convolution. In the present invention, the convolution kernel size of the one-dimensional convolution is set to 4, the step size is 1, the padding is 3, and SSM represents the state space model. At the same time, Mz is activated by SiLU function to generate Mzo:

[0065] Mzo=SiLU(Mz) (17)

[0066] For the SSM in Formula 16, it is through the hidden state Map Mxc to Mxo, D in Same as the second dimension of Mxc, N represents the state dimension.

[0067] The following is a detailed introduction to SSM:

[0068]

[0069] Linear B 、Linear I 、Linear Δ Denotes three linear mappings with different parameters used to generate B, I, and Δ, B∈R J×N It is done by linear mapping Mxc B We get, I∈R J×N It is done by linear mapping Mxc I Got it, It is done by linear mapping Mxc Δ Got it, is a randomly initialized learnable parameter matrix, EX_RE is a function defined in the present invention for expanding tensors of different dimensions to the same dimension, EX_RE A 、EX_RE B 、EX_RE Δ are different forms acting on A, B, and Δ respectively. They are the results obtained after A, B, and Δ are expanded. is A′ converted by the time scale parameter Δ′, is obtained by transforming B′ through the time scale parameter Δ′, represents the parameter matrix of the hidden state h at the jth step, express In the parameter matrix of step j, ⊙ represents the multiplication of corresponding elements, express The parameter matrix at step j, I j ∈R N represents the j-th row vector of I, Represents the j-th element of Mxo.

[0070] As described above, First, we transform it into B∈R through linear mapping with different parameters. J×N , I∈R J×N , Then through the function EX_RE A 、EX_RE B 、EX_RE Δ Map A, B, Δ to the same dimension Δ′ converts A′ into Convert B′ to Similarly, through the function EX_RE, Mxc is changed from becomes And through the hidden state Calculate Mxo.

[0071] Finally, Mxo and Mzo are multiplied and passed through a linear layer to be combined with M V After adding, we get the result of the vertical patch branch:

[0072] Mo V =M V +Linear3(Mxo·Mzo) (19)

[0073] Linear3 represents a linear mapping, M V ∈R J×G is the input of the Mamba block, Mo V ∈R J×G is the output of the Mamba block, G size is 450, Mamba V It contains three layers of Mamba blocks, and the output of the last layer of Mamba blocks is the final result.

[0074] The horizontal patch and the vertical patch go through the same number of Mamba blocks but with different parameters. After going through 3 layers of Mamba blocks, the final output Mo of the two branches is H and Mo V Perform global average pooling and add:

[0075] MO=GAP(Mo H )+GAP(Mo V ) (20)

[0076] Where GAP represents global average pooling, Mo H∈R J×G is the output of the horizontal patch branch, Mo V ∈R J×G is the output of the vertical patch branch, and the obtained MO∈R G It is fed into the classifier to obtain the prediction result.

[0077] Step 3: Classification prediction

[0078] The loss function of classification is cross entropy. In this step, the final result MO∈R obtained in step 2 is G (R represents the real number domain), and is sent to the linear layer to obtain the predicted probability of the category p∈R M , and then the loss is obtained by calculating the cross entropy loss function, and then the model parameters are updated through back propagation.

[0079] Here is how the cross entropy loss function is calculated:

[0080]

[0081] Where M is the number of categories, y c is 0 or 1 (if the sample category is equal to c, it takes 1, otherwise it takes 0), p c is the probability that the sample belongs to category c, and Loss is the loss value of the sample.

[0082] Step 4: Train the model

[0083] The present invention sets the learning rate to 1e-4 and the epoch to 50. Repeat steps 2, 3 to 50 to obtain the final accuracy.

[0084] The present invention is verified on the SEED and SEED-IV datasets through a 5-fold cross-validation experiment. The experiment confirms that the proposed model achieves accuracies of 99.07% and 96.29% on the SEED and SEED-IV datasets, respectively, demonstrating its superiority in emotion recognition.

[0085] References

[0086] [1] Y.Li, W.Zheng, L.Wang, Y.Zong, and Z.Cui, "From regional to globalbrain: Anovel hierarchical spatial-temporal neural network model for eegemotion recognition," IEEE Transactions on Affective Computing, vol.13, no.2, pp.568–578, 2019.

[0087] [2]M.Ye,C.P.Chen,and T.Zhang,“Hierarchical dynamic graphconvolutional network with interpretability for eeg-based emotionrecognition,”IEEE Transactions on Neural Networks and Learning Systems,2022.

[0088] [3]D.-K.Jeong,H.-G.Kim,and J.-Y.Kim,“Emotion recognition usinghierarchical spatiotemporal electroencephalograminformation from local toglobal brain regions,”Bioengineering,vol.10,no.9,p.1040,2023.

Claims

1. An emotion recognition method based on a hierarchical Mamba model of EEG signals, characterized in that include: Step 1: Capture the EEG signal using non-overlapping windows and extract the differential entropy features of 5 frequency bands; Step 2: Multilayer Perceptron (MLP) is used as a channel-level encoder to extract the frequency band relationship within each electrode. During the encoding process, each electrode is independent of each other to ensure that no spatial information is integrated; Step 3: Project the encoded representation into space according to the relative relationship of the head electrode positions, and map the channels into a two-dimensional matrix: R C →R H×W , where C is the number of channels of the EEG signal, H and W are the number of rows and columns converted into a two-dimensional matrix; Step 4: Use convolution kernels of different sizes to dynamically convolve and aggregate the data to extract local spatial information of EEG; Step 5: For the extraction of global spatial information of EEG signals, the horizontal and vertical patch division methods are combined to explore different connection patterns of the brain, and the information of the horizontal and vertical branches is extracted through the Mamba network for emotion recognition.

2. The emotion recognition method based on the hierarchical Mamba model of EEG signals according to claim 1 is characterized in that: The specific operation of extracting differential entropy in step 1 is: X=DE(t)=-∫g(t)logg(t)dt (1) Where DE represents the differential entropy formula, t represents the EEG signal, and g(t) represents the probability density function of t; because the EEG signal obeys the Gaussian distribution, the above formula can be expressed as: Among them, π is the ratio of the circumference of a circle to the diameter of a circle, e is a natural constant, μ and σ are 2 represents the mean and variance of the EEG signal t.

3. The emotion recognition method based on the hierarchical Mamba model of EEG signals according to claim 1 is characterized in that: The channel-level encoder in step 2 consists of an MLP and a batch normalization layer; The MLP consists of two fully connected layers with a ReLU activation function between them; the process is defined as: X c =CFE(X) (3) Where CFE represents the channel level encoder, X c ∈R C×D Represents the output of CFE.

4. The emotion recognition method based on the hierarchical Mamba model of EEG signals according to claim 3 is characterized in that: Step 3 X c ∈R C×D is transposed to X c ∈R D×C Then the second dimension C is spatially projected into H×W according to the relative relationship of the head electrode positions, that is, X p ∈R D×H×W , X p is the result after conversion.

5. The emotion recognition method based on the hierarchical Mamba model of EEG signals according to claim 4 is characterized in that: Step 4 consists of three parts: Split, Fuse, and Select. In Split, two convolution kernels of different sizes are used, one with a size of 3×3 and the other with a size of 5×5. p Dynamic convolution is performed, in which each convolution kernel performs K convolutions, each time with different convolution parameters, and then K weights are assigned through the attention mechanism to aggregate the results of the K convolutions; Fuse combines the results of the dynamic convolution of two convolution kernels of different sizes; Select calculates the weight of each dynamic convolution using the results obtained by Fuse to obtain the final result.

6. The emotion recognition method based on the hierarchical Mamba model of EEG signals according to claim 1, characterized in that: The method of dividing the patch horizontally and vertically in step 5 is as follows: For the output V∈R obtained in step 4 L×H×W , and convert it into M according to the horizontal patch and vertical patch methods. H ∈R J×(P×L) , M V ∈R J×(P×L) , where P×L is the size of each patch, and J is the length of the sequence obtained after the conversion. After horizontal and vertical division, two branches will be generated, each of which will learn the global spatial information of EEG through the Mamba model, and finally aggregate the results of the two branches.

Images