An electroencephalogram emotion recognition method based on double gate chebyshev connection

Abstract

The application discloses an electroencephalogram emotion recognition method based on double-gate Chebyshev connection, and comprises the following steps: data preprocessing; adopting an improved graph feature extraction module to extract features of graph-structured electroencephalogram signal information; and adopting a dense connection deep network to learn features. The method is improved in the mode of neighbor information flow, combines a Chebyshev-like module and a double-gate unit to effectively extract different-order neighbor information features and global features, sends the extracted features into a classifier, and finally obtains the classification result of emotion. The two sub-modules of the Chebyshev-like connection module of the application have targeted improvement for different data sets; the double-gate unit introduced in the application controls the Chebyshev-like connection and the information flow of each order respectively, and effectively adjusts the information redundancy and noise generated by the information flow.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of emotion recognition, and relates to an electroencephalogram emotion recognition method based on double-gate Chebyshev connection. BACKGROUND

[0002] Electroencephalogram (EEG) is an electrical signal that records the activity of neurons in the brain, which is collected and recorded through electrodes on the scalp. It has a high temporal resolution, which can capture the subtle differences in human emotional changes. Therefore, electroencephalogram is widely used in emotion recognition research and application, aiming to determine the emotional state of people, such as happiness, sadness, anger, fear, etc., by analyzing the characteristics of these signals.

[0003] Different methods can be used to extract the characteristics of electroencephalogram signals, among which the most commonly used is frequency domain analysis. Frequency domain analysis decomposes electroencephalogram signals into different frequency components, each of which represents a brain rhythm. Electroencephalogram rhythms are divided into five sub-bands: δ (1-4 Hz), θ (4-8 Hz), α (8-12 Hz), β (13-30 Hz), γ (31-45 Hz). These five sub-bands are closely related to human psychological activities, for example, α rhythm is related to relaxation and closed eyes, β rhythm is related to attention and thinking, and γ rhythm is related to cognition and memory. Therefore, by analyzing the frequency domain characteristics of electroencephalogram signals, the emotional information of humans can be obtained. In addition to frequency domain analysis, other methods can also be used to extract the characteristics of electroencephalogram signals, such as time domain analysis, time-frequency domain analysis, multi-electrode characteristics or connectivity characteristics, etc. Time domain analysis directly uses the waveform characteristics of electroencephalogram signals, such as peak value, mean value, variance, etc. Time-frequency domain analysis combines time domain and frequency domain information, such as short-time Fourier transform, wavelet transform, etc. Multi-electrode characteristics or connectivity characteristics consider the relationship between electroencephalogram signals at different electrode positions or regions, such as coherence, phase locking value, etc.

[0004] After extracting the characteristics of electroencephalogram signals, a computational model for emotion recognition can be established to classify emotional information. Computational models can use traditional machine learning methods, such as support vector machines, decision trees, random forests, etc., or modern deep learning methods, such as convolutional neural networks, recurrent neural networks, attention mechanisms, etc. The selection and optimization of computational models need to be carried out according to different data sets and tasks.

[0005] Graph neural networks aim to represent the vertices in a graph as low-dimensional vectors by preserving the network topology and node content information of the graph, so as to be processed using simple machine learning algorithms (e.g., support vector machine classification). It can effectively process graph structure data such as electroencephalogram signals and is widely used in the field of electroencephalogram signal emotion recognition. Current graph neural networks mainly include graph convolutional network (GCN), graph attention network, graph autoencoder, graph generative network and graph spatial-temporal network.

[0006] Graph convolutional network has gradually become a main method in the field of electroencephalogram signal emotion recognition. Through a graph feature extractor, non-Euclidean structure data is extracted for features, making the electroencephalogram features originally unable to be learned by a deep network become learnable. GCN converts spectral domain signals to spatial domain through Fourier transform, and according to time sequence, respectively evolves into spectral GCN, Chebyshev GCN and GCN.

[0007] Electroencephalogram (EEG) is a physiological signal reflecting human emotional changes, which can be collected and recorded through scalp electrodes. To extract effective emotional features, graph convolutional network (GCN) is often used to model the spatial structure and frequency domain features of EEG. This network is a deep learning model based on graph structure, which can perform convolution operations on node features and adjacency relationships to learn high-order representations of nodes. The EEG emotion recognition method based on graph convolutional network generally includes the following four steps: 1. Induction and EEG recording synchronization: Select appropriate stimuli (such as pictures, videos, music) to induce different emotional states, and record the EEG signals of the subjects. Video stimuli can most effectively trigger fast and accurate emotional responses. 2. EEG signal preprocessing: Due to the small amplitude of EEG signals (only 10-100 μV), they are easily disturbed by various noises (such as ECG, EMG, EOG, and other physiological signals, as well as slight body movements, power line interference, and baseline drift). The signal needs to be preprocessed to remove artifacts. Common methods include filtering, downsampling, baseline correction, artifact removal, and segmentation. 3. EEG signal feature extraction: Convert EEG signals into a graph structure, which is generally based on the spatial location and frequency domain features of EEG channels. After decomposing the signal into different frequency components, calculate the energy or entropy of each channel at each frequency component as the node feature. Then, according to the correlation or synchronicity between channels, construct an adjacency matrix as the edge of the graph, and common indicators include Pearson correlation coefficient, mutual information, coherence, and phase locking value. 4. Emotion recognition classifier construction: Based on the extracted EEG features, establish an emotion classifier, and different deep network structures have a significant impact on signal classification accuracy. Variants of graph convolutional network such as dynamic graph convolutional network (DGCNN) and graph convolutional block (GCB) can dynamically adjust the graph structure and features according to the time-varying and frequency-varying nature of EEG signals, improving the performance of emotion recognition.

[0008] The EEG emotion recognition method based on graph convolutional network effectively utilizes the spatial structure and frequency domain features of EEG signals, and can improve the accuracy and robustness of emotion recognition. However, there are still some challenges, such as the selection of appropriate graph construction method, the processing of multi-modal signal heterogeneity and correlation, etc. With the development of graph neural network, the EEG emotion recognition method based on graph convolutional network is expected to achieve better results. The recognition method based on graph neural network (GNN) has achieved good results in EEG emotion recognition. The graph convolutional network (GCN) is used to convert the original graph structure data into regular structured data, and then a deep network is used to extract higher level features, which has good EEG signal feature classification performance. Dynamic graph convolutional network (DGCNN) learns the adjacency relationship between EEG channels by using a dynamically updated network, extracts features from non-Euclidean structured data, and enables the deep network to learn EEG signal features. The width graph convolutional network (GCB-net) innovatively combines the width network and the width idea with GCN, realizes efficient and fast graph feature extraction, and effectively combines the advantages of deep network and width network; the residual width graph convolutional network (Residual GCB-net) uses the residual module to break through the network degradation problem caused by the deep depth of the original deep network.

[0009] Graph convolutional networks usually focus more on low-order neighbor information, and the extracted features often do not have globality. Emotional stimuli of EEG often have multiple brain function regions and multiple nodes responding, and existing graph convolutional networks are difficult to solve global emotion feature extraction. Graph convolutional networks usually cannot balance the performance on datasets with different node numbers. Even using the same model, the accuracy often has a large difference on datasets with different node numbers. Graph convolutional networks usually cannot balance the importance of low-order neighbor information and high-order neighbor information, and unreasonable information output often produces a lot of redundant information and noise.

[0010] A graph convolutional brain electrical emotion recognition method combined with width learning (1.T.Zhang, X.Wang, X.Xu and C.L.P.Chen, "GCB-Net:Graph Convolutional Broad Network and Its Application in Emotion Recognition," in IEEE Transactions on Affective Computing, vol.13, no.1, pp.379-388, 1Jan.-March 2022, doi:10.1109 / TAFFC.2019.2937768.), the accuracy of the technology will decrease significantly when the number of layers of the deep network is increased, and the global features cannot be effectively extracted when the number of layers is low. The present application can effectively solve the network degradation problem and obtain better global feature extraction effect using a densely connected network.

[0011] An improved method of graph width convolutional network using residual learning (2.Q.Li, T.Zhang, C.L.P.Chen, K.Yi and L.Chen, "Residual GCB-Net:Residual Graph Convolutional Broad Network on Emotion Recognition," in IEEE Transactions on Cognitive and Developmental Systems, vol.15, no.4, pp.1673-1685, Dec.2023, doi:10.1109 / TCDS.2022.3147839.), using residual connection effectively solves the degradation problem of deep network, however, this method still cannot solve the problem that the feature extraction of low-order neighbor information and high-order neighbor information of graph convolutional network cannot be balanced. The present application effectively controls the flow of low-order information to high-order information by setting a Chebyshev gating network, and effectively solves the problem that local features and global features cannot be considered by setting a gating to adjust the propagation of important information. SUMMARY

[0012] In order to solve the problems existing in the prior art, the application provides a network connection mode of a quasi-Chebyshev connection, by introducing a residual learning idea and a Chebyshev polynomial, low-order information is effectively transmitted to high-order information, so that feature extraction has globality. The two sub-modules of the quasi-Chebyshev connection module of the application have targeted improvement for different data sets; the introduction of the double-gated unit of the application controls the information flow of the quasi-Chebyshev connection and each order respectively, and effectively adjusts the information redundancy and noise generated by the information flow.

[0013] The application is implemented by at least one of the following technical solutions.

[0014] A brain electrical emotion recognition method based on a double-gated Chebyshev connection, comprising the following steps:

[0015] Collecting and preprocessing graph-structured electroencephalogram data;

[0016] Extracting features from the graph-structured electroencephalogram signal information by using a graph feature extraction module;

[0017] Learning the features by using a densely connected deep network, further extracting the features learned by the graph feature extraction module, sending the extracted features into a classifier, and finally obtaining a classification result of emotions.

[0018] Further, the preprocessing includes removing artifacts, electrooculogram and electrocardiogram interference of the collected electroencephalogram data by using an EEGLAB tool in MATLAB, and dividing the features into different frequencies of δ, θ, α, β and γ according to the frequency.

[0019] Further, the graph-structured electroencephalogram signal information includes data in a SEED data set.

[0020] Further, the graph feature extraction module includes a quasi-Chebyshev connection module (CL) and a double-gated quasi-Chebyshev connection module (DGC-Link).

[0021] Further, the CL module includes a first sub-module and a second sub-module, and the two sub-modules are respectively used for feature extraction of data sets with different node numbers.

[0022] Further, the first sub-module is suitable for data sets with a node number less than or equal to 32, and the second sub-module is suitable for data sets with a node number greater than 32.

[0023] Further, the two sub-modules adopt a connection mode of a Chebyshev polynomial, and on this basis, a residual connection is introduced, when there are more than two orders, the neighbor information of the previous order is input into the next order by a direct connection mode, the first sub-module adopts a summation mode in the multi-order neighbor information aggregation stage, and the second sub-module directly outputs the last-order neighbor information in this stage.

[0024] Further, the first sub-module is represented as:

[0025]

[0026] where y α is the output of the first sub-module, Ω is an activation function, K is the maximum order of graph convolution, is an improved Chebyshev connection function, is an original Chebyshev connection function, x is a graph-structured electroencephalogram signal, A k is a k-order intra-corresponding inter-node adjacency matrix, W k is a k-order corresponding weight, is a different-order adjacency matrix.

[0027] Further, the second sub-module is represented as:

[0028]

[0029] where y β is the output of the second sub-module.

[0030] Further, the DGC-Link introduces a double-gating unit on the basis of the CL module to improve the flow of information. The first gating unit includes a GRU model, and the second gating unit is represented as a forget gate, which is used to control the importance of each order of neighbor information in the final summation. The calculation formula of the forget gate is as follows:

[0031]

[0032] where is the output of the forget gate, is an activation function, is a weight of the forget gate, is the result of each order of graph convolution;

[0033] The DGC-Link module is represented as:

[0034]

[0035] where y DGC-Link is the output of the DGC-Link, Ω is an activation function, K is the maximum order of graph convolution, x is a graph-structured electroencephalogram signal, A k is a k-order intra-corresponding inter-node adjacency matrix, W k is a k-order corresponding weight, is remembered information, is forgotten information.

[0036] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0037] The present application realizes the effective transmission of low-order neighbor information to high-order neighbor information. The Chebyshev connection method proposed by the present application effectively solves the problem of losing low-order neighbor information when extracting high-order features, so that the model can consider both the local importance of low-order neighbor information and the global nature of high-order neighbor features when extracting graph features, and higher accuracy can be achieved in practical applications.

[0038] The two Chebyshev connection submodules of the present application can achieve good performance on datasets with different numbers of nodes. By setting different output modes, the submodules effectively solve the problem that the same method cannot achieve the best effect in different application scenarios. When using a brain cap with a small number of electrodes for data collection and analysis (such as fatigue monitoring), the alpha module can achieve better classification accuracy; in application scenarios with a large number of electrodes (such as hospitals), the beta module can achieve higher classification accuracy in the scenario by extracting global features.

[0039] The double-gating unit of the present application effectively adjusts the importance of neighbor information of different orders by controlling the flow and output of neighbor information of different orders, reducing the negative impact of noise and redundant information on the classification result. The double-gating unit of the present application can control the flow of information of each order of the Chebyshev connection, so that irrelevant and redundant information is prevented from flowing to higher orders, so that the model can better balance low-order information and high-order information; by controlling the output of neighbor information of each order, irrelevant information and redundant information can be more effectively avoided, and the impact of noise and irrelevant information can be reduced as much as possible. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 A schematic diagram of a brain electrical emotion recognition method based on double-gate Chebyshev connection according to an embodiment of the present application;

[0041] Figure 2 A flowchart of a brain electrical emotion recognition method based on double-gate Chebyshev connection according to an embodiment of the present application;

[0042] Figure 3 A feature extraction module structure diagram according to an embodiment of the present application;

[0043] Figure 4 A gating unit structure diagram according to an embodiment of the present application;

[0044] Figure 5 A dense connection deep network structure diagram according to an embodiment of the present application. DETAILED DESCRIPTION

[0045] The invention is further described below with reference to the accompanying drawings and specific embodiments.

[0046] The present invention proposes an EEG emotion recognition method based on a dual-gate Chebyshev connection. The method is improved on the graph convolution module. The graph convolution module includes a Chebyshev-like connection module and a dual-gated Chebyshev-like connection module. The Chebyshev-like connection module is used to improve the transmission of low-order neighbor information to high-order neighbor information. The two submodules of the Chebyshev-like connection module are used for classification tasks on data sets with different numbers of nodes. The dual-gated Chebyshev-like connection module is used to regulate the information flow from low-order neighbor information to high-order neighbor information and the output of each order of neighbor information.

[0047] like Figure 1 、 Figure 2 The present embodiment provides an EEG emotion recognition method based on a dual-gate Chebyshev connection, comprising the following steps:

[0048] Step 1: Data preprocessing: Use the EEGLAB tool in MATLAB to remove artifacts, oculoculograms, and electrocardiograms (ECGs). The data is then classified into δ, θ, α, β, and γ frequency features based on the frequency range (0-100 Hz). Some datasets also provide preprocessed features (e.g., the SEED dataset).

[0049] Step 2: Use the graph feature extraction module to extract features from the graph-structured EEG signal information, as follows:

[0050] EEG signal information is used as input to the graph feature extraction module, which first extracts regularized features. The graph feature extraction module primarily consists of a Chebyshev-like link module (CL) and a dual-gated Chebyshev-like link module (DGC-Link). The CL module contains two submodules, each of which can be used for feature extraction for specific tasks.

[0051] As an embodiment, the first submodule is applicable to a data set with a smaller number of nodes (less than or equal to 32), and the second submodule is applicable to a data set with a larger number of nodes (greater than 32).

[0052] The CL module comprises a first submodule and a second submodule, the first submodule is an alpha module, and the second submodule is a beta module. Both submodules adopt a connection mode of Chebyshev polynomials, and residual connection is introduced on this basis. When the order is two or more, the neighbor information of the previous order is input to the next order through direct connection, and when the order is high, the model can also retain the low-order neighbor information. The alpha module adopts a summation mode in the multi-order neighbor information aggregation stage, and the beta module directly outputs the last-order neighbor information in this stage. For a dataset with a small number of nodes (less than or equal to 32), the former ensures the importance of low-order neighbor information in the classification task, and the latter often achieves better classification effect on a dataset with a large number of nodes (more than 32) due to better global information performance.

[0053] The alpha module can be represented as:

[0054]

[0055]

[0056] wherein y α is the output of the first submodule, Ω is an activation function, which is a sigmoid function in this case, and K is the maximum order of graph convolution, is an improved Chebyshev connection function is an original Chebyshev connection function, x is an input, A k is a node adjacency matrix corresponding to the k-order, W k is a weight corresponding to the k-order, is a neighbor matrix of different orders.

[0057] The beta module can be represented as:

[0058]

[0059] wherein y β is the output of the second submodule.

[0060] The DGC-Link module is as shown in the right part of FIG. 2, and the DGC-Link introduces a double-gating unit to improve the flow of information on the basis of the CL module. The specific implementation of the double-gating unit is as shown in FIG. 3. Figure 3 Figure 4

[0061] The first gating unit 1 comprises the idea of a gated recurrent unit model, and realizes the memory of effective information and the discard of redundant information and noise through resetting and updating units.

[0062] ​​The update unit adjusts the amount of previous stage hidden layer information that should be retained in the current stage hidden layer output. When the update gate is close to 1, it can effectively copy the hidden layer information of the previous stage to the current stage, thereby promoting long-distance dependence learning from low stages to high stages. In general, the reset unit can more effectively process emotional features with short distance dependence. In contrast, the update unit can effectively extract emotional features with long distance dependence or more global features.

[0063] The second gating unit 2 is represented as a forget gate, which is used to control the importance of each stage neighbor information in the final summation. The calculation formula of the forget gate is as follows:

[0064]

[0065] wherein is the output of the forget gate, is an activation function, here the ELu function, is the weight of the forget gate, is the result of each stage graph convolution.

[0066] Finally, the DGC-Link can be represented as:

[0067]

[0068] wherein y DGC-Link is the output of the DGC-Link, is the information of the memory, is the information of the forget.

[0069] Both the alpha module and the beta module are graph feature extraction modules for electroencephalogram signal graph feature extraction. The first gating unit and the second gating unit are structures for controlling information flow in the graph feature extraction module, the first gating unit is used to control the information transmission from the kth stage electroencephalogram node to the k+1th stage, and the second gating unit is used to control the output of each stage information.

[0070] Step three, using a densely connected deep network to learn the features. This method improves the neighbor information flow, combines the Chebyshev connection module and the double gating unit to effectively extract different stage neighbor information features and global features.

[0071] The dense connection deep network (Dense-net) further extracts the features learned by the graph feature extraction module to extract deep-level features. Through dense connection, the Dense-net can propagate the gradient to the earlier network layer more quickly. This alleviates the problem of gradient disappearance, making the Dense-net easier to train and maintaining better gradient stability in the deep network. The dense network has stronger model expression ability and can capture the details and diversity of the input data. The dense connection enables each layer to directly access the features of the previous layer, thereby more fully utilizing the information. Due to the liquidity of gradient and parameter reuse, the DenseNet often converges faster in the initial stage of training.

[0072] The method extracts the non-Euclidean structure electroencephalogram data into brick structure data through the graph feature extraction module, uses the dense connection network to learn high-level features, and finally uses the full connection layer as a classifier to realize the emotion recognition of the electroencephalogram signal. For different classification tasks, only the output of the classifier needs to be adjusted, and the double gate Chebyshev connection in the application does not need to be changed.

[0073] As an embodiment, tests are performed on the SEED, DREAMER and MPED electroencephalogram signal data sets. The SEED data set selects 15 subjects (8 females and 7 males), and makes them watch 3 groups (positive, general and negative) of films 3 times within a week, and each group has 5 different film segments. In this way, the corresponding emotions of the subjects are aroused, and the features are recorded. The SEED data set records 5 groups of electroencephalogram features in different frequencies (δ, θ, α, β, γ"bands"), which are power spectral density (PSD), differential entropy (DE), difference causality (DCAU), difference asymmetry (DASM) and rational asymmetry (RASM). The DREAMER data set selects 23 subjects (9 females and 14 males), adopts a similar method, and records the power spectral density (PSD) features (δ, θ, α) of 9 emotions (happy, pleasant, excited, surprised, calm, disgusted, fearful, sad and angry) of the subjects in 18 films. The MPED is a data set specially designed for multi-modal physiological emotion recognition, which is collected from 23 subjects. It contains four modes of physiological signals: electrocardiogram (ECG), electromyogram (EMG), galvanic skin response (GSR) and respiration (RSP). After the subjects watch 28 carefully selected movie segments, electroencephalogram data is collected using a 62-electrode electroencephalogram cap. These movie segments have undergone strict psychological evaluation before being included in the study.

[0074] The embodiments are classified on the corresponding features on SEED, DREAMER and MPED datasets. In SEED dataset, 96.43% accuracy can be achieved on all-band features of DE. In DREAMER dataset, 98.58%, 98.01% and 97.62% are achieved on arousal, dominance and valence features of DGC-Link. In MPED dataset, 78.48% accuracy can be achieved on three-classification task, and 44.93% accuracy can be achieved on seven-classification task.

[0075] The specific embodiments described herein are merely illustrative of the principles of this application. Numerous modifications and adaptations will be readily apparent to those skilled in the art of this application that do not depart from the spirit of the scope of the claimed application as defined by the following claims.

Claims

1. A method for electroencephalogram emotion recognition based on double-gate Chebyshev connection, characterized in that, The method comprises the following steps: Collect and pre-process the graph-structured electroencephalogram data; Extract features from the graph-structured electroencephalogram signal information by using a graph feature extraction module; the graph feature extraction module comprises a Chebyshev connection module CL and a double-gated Chebyshev connection module DGC-Link; the CL module comprises a first submodule and a second submodule, and the two submodules are respectively used for feature extraction of data sets with different node numbers; The first submodule is represented as: where y α is the output of the first sub-module, Ω is the activation function, and K is the maximum order of graph convolution, is the improved Chebyshev connection function, is the original Chebyshev connection function, x is the graph structured electroencephalogram signal, A k is the k-order inner corresponding inter-node adjacency matrix, W k is the k-order corresponding weight, is the adjacency matrix of different orders; the second sub-module is where y β is expressed as the output of the second submodule; Learn the features by using a densely connected deep network, further extract the features learned by the graph feature extraction module, send the extracted features into a classifier, and finally obtain the classification result of the emotion.

2. The electroencephalogram emotion recognition method based on double-gate Chebyshev connection according to claim 1, characterized in that: The pre-processing comprises removing artifacts, electrooculogram and electrocardiogram interference of the collected electroencephalogram data by using an EEGLAB tool in MATLAB, and dividing the features into different frequency characteristics according to frequencies, such as δ, θ, α, β and γ.

3. The electroencephalogram emotion recognition method based on double-gate Chebyshev connection according to claim 1, characterized in that: The graph-structured electroencephalogram signal information comprises data in a SEED data set.

4. The electroencephalogram emotion recognition method based on double-gate Chebyshev connection according to claim 1, characterized in that: The first submodule is suitable for data sets with node numbers less than or equal to 32, and the second submodule is suitable for data sets with node numbers greater than 32.

5. The electroencephalogram emotion recognition method based on double-gate Chebyshev connection according to claim 1, characterized in that: Both the two submodules adopt a connection mode of Chebyshev polynomials, and residual connections are introduced on this basis; when the order is greater than two, the neighbor information of the previous order is input into the next order by a direct connection mode; the first submodule adopts a summation mode in the multi-order neighbor information aggregation stage, and the second submodule directly outputs the last-order neighbor information in this stage.

6. The electroencephalogram emotion recognition method based on double-gate Chebyshev connection according to claim 1, characterized in that: The DGC-Link introduces a double-gated unit on the basis of the CL module to improve the flow mode of information; the first gated unit comprises a GRU model, and the second gated unit is represented as a forgetting gate and is used for controlling the importance of each order of neighbor information in the final summation; the calculation formula of the forgetting gate is as follows: wherein is an output represented as a forget gate, is represented as an activation function, is a weight represented as a forget gate, is represented as a result of each order graph convolution; The DGC-Link module is represented as: where y DGC-Link is the output of DGC-Link, Ω is the activation function, K is the maximum order of graph convolution, x is the graph structured electroencephalogram signal, A k is the adjacency matrix between nodes corresponding to the k-th order, W k is the weight corresponding to the k-th order, is the information of memory, is the information of forgetting.

Images