Electroencephalogram emotion recognition method based on multi-branch graph convolutional network

Abstract

The application discloses an electroencephalogram emotion recognition method based on a multi-branch graph convolution network. The graph signal feature data modeled by the application restores the spatial and functional connection of the electroencephalogram data itself, retains more emotional information by using multi-dimensional features, and is used for accurately classifying and recognizing the emotional state of an individual. In combination with an attention mechanism, different sample data are well distinguished in terms of emotional information richness, and then the interference of redundant data on the classification performance is greatly reduced, so that the application has good classification effect and generalizability; a multi-branch graph convolution model is adopted, the model simultaneously considers the physical connection and the correlation connection among channels, and an attention mechanism based on the correlation connection is used to assist in optimizing an adjacency matrix based on the physical connection, so that the problem of single information of the adjacency matrix in the current ordinary graph convolution model is solved. The application uses an online brain-computer interface system, changes the phenomenon that the offline electroencephalogram data is analyzed in the past, and improves the real-time performance of the electroencephalogram data analysis.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of signal processing, relates to the field of brain-computer interface, and relates to an electroencephalogram emotion recognition method based on a multi-branch graph convolution network, and more particularly to an online electroencephalogram emotion feature recognition brain-computer interface system based on a spatio-temporal graph convolution neural network. BACKGROUND

[0002] At present, electroencephalogram has made many important discoveries in the field of emotion recognition. Among numerous physiological signals, electroencephalogram can directly reflect the real-time activity state of the brain because it is directly taken from the brain signal, and has non-deceptiveness, so emotion recognition based on electroencephalogram signals has practical significance. Effective interpretation of the emotional state of individuals can solve various implementation problems in work and life, for example, doctors monitor the emotional state of patients in real time to adjust the treatment plan in time, understand the satisfaction of customers to products, detect the fatigue condition of drivers during driving, and the like. At the same time, due to the wide application of brain-computer interface (BCI), emotion recognition based on electroencephalogram signals will greatly help to improve the user experience in human-computer interaction.

[0003] Electroencephalogram is widely used in the application research of emotion recognition because of its high time resolution, real-time performance, and more direct response to the emotional state of the human brain. Early research on brain information processing mechanism is mainly based on the establishment of various electroencephalogram channels of the whole brain. However, with the deepening of research, although the brain is divided into many brain regions, the brain region related to a specific emotion will always show an active state under the corresponding emotion. Therefore, when studying a specific emotion, the specific brain region becomes our focus, and the use of the characteristics of different reactions of specific brain regions to different emotions has become a hot spot of attention all over the world.

[0004] The electroencephalogram emotion recognition method mainly includes traditional machine learning methods and deep learning methods, and in recent years, deep neural networks have been proven to be superior to traditional machine learning methods in the application fields of face recognition, image recognition, speech recognition, and the like. Among them, the convolutional neural network has been widely used in computer vision, natural language processing, and other spatial continuous data. Neuroscience has proved that adjacent brain regions play a very important role in brain function activities. Traditional machine learning methods ignore the spatial characteristics of electroencephalogram signals, and only simply analogize electroencephalogram signals to image signals or sound signals in Euclidean space, resulting in that these methods fail to well solve the important problem of modeling the relationship between electrodes in electroencephalogram signals.

[0005] With the rise of graph convolutional network, the electroencephalogram emotion recognition method based on graph convolutional neural network is also proposed. The graph convolutional network is proposed to solve the limitation of traditional discrete convolution in solving non-Euclidean structure data. In order to effectively extract spatial features on non-Euclidean data for machine learning, the graph convolutional network has become the focus of research. The graph convolutional neural network input contains the original electroencephalogram data and the graph structure constructed by modeling the electrode. According to the connection relationship between nodes in the graph structure, the information transmission between nodes is realized. However, the existing technology still cannot solve the technical problems:

[0006] (1) Low stability; the influence of the spatial discreteness of emotional electroencephalogram signal and the discontinuity in time dimension is not fully considered.

[0007] (2) Low recognition accuracy; the spatial information retained by low-dimensional features is limited, thereby reducing the classification accuracy. (3) Low interpretability; neuroscience has proved that adjacent brain areas play a very important role in brain function activity. The traditional machine learning method ignores the spatial characteristics of electroencephalogram signal, and only analogs the electroencephalogram signal to the image signal or sound signal in Euclidean domain, which leads to that these methods cannot well solve the important problem of modeling the relationship between electrodes in electroencephalogram signal. At the same time, the information of edge features in the current emotion classification method based on graph convolutional network is too single, which reduces the classification accuracy.

[0008] (4) Low real-time performance; the existing electroencephalogram emotion classification method mostly uses offline data to analyze the offline data, which separates real-time data from the algorithm, resulting in that the research in the field of brain-computer interface mostly stays in the experimental stage. However, the actual application mostly needs real-time analysis of electroencephalogram signal to give analysis results, therefore, the present application is dedicated to researching the BCI system with strong real-time performance, and improving the real-time performance of brain intention decoding. SUMMARY

[0009] The purpose of the present application is to propose an electroencephalogram emotion recognition method based on multi-branch graph convolutional network, to solve the above-mentioned problems that have not been solved at present, to apply the graph convolutional neural network to the field of processing spatial discrete unstructured data such as electroencephalogram signal, and to realize the real-time performance of electroencephalogram signal data analysis, thereby improving the accuracy of electroencephalogram emotion recognition.

[0010] To achieve the above object, the specific idea of the present application is: a multi-branch graph convolution model is proposed, which simultaneously considers the physical connection and correlation connection between channels, uses a channel correlation-based attention mechanism to solve the problem of single information of the adjacency matrix in the current ordinary graph convolution model, and combines a space-time attention mechanism to obtain the representation of real emotional features, so as to filter redundant data and assist in emotional classification. Specifically, the present application provides an electroencephalogram emotion recognition method based on a space-time graph convolutional neural network and an online electroencephalogram emotion classification system based on the method.

[0011] In a first aspect, an electroencephalogram emotion recognition method based on a multi-branch graph convolutional network is provided, which comprises the following steps:

[0012] S1: Obtain emotional electroencephalogram data and pre-process the emotional electroencephalogram data;

[0013] S101: Filter the emotional electroencephalogram data using a 1-50Hz band-pass filter, and then divide it into 4 frequency bands, namely theta (4-8Hz), alpha (8-14Hz), beta (14-31Hz), and gamma (31-50Hz);

[0014] S102: Slice the above different frequency band electroencephalogram data and perform a sliding window operation with a fixed length s;

[0015] S103: Obtain node features;

[0016] According to the international 10-20 system electrode position, the features of the sliced electroencephalogram data are extracted, the extracted feature data is converted from 1-dimensional data to 2-dimensional plane format, and 4 frequency band 2-dimensional planes of each electroencephalogram sample are obtained, and 3-dimensional electroencephalogram feature data is obtained as node features by integrating the 2-dimensional planes of the 4 frequency bands;

[0017] S104: Obtain edge features A1 based on physical distance;

[0018] The present application adopts a multi-branch graph convolutional network, one branch of which obtains edge features A1 with richer emotional information through the fusion of point-by-point multiplication of 2 identical adjacency matrices based on physical distance:

[0019]

[0020] wherein A d represents an adjacency matrix based on physical connection, represents point-by-point multiplication, and A d is calculated as follows:

[0021]

[0022] Where τ represents the preset threshold, θ represents the fixed parameter, and dist(i,j) represents the distance between the i-th node and the j-th node;

[0023] S105: Obtain edge features considering correlation connections A2:

[0024] Edge features A2 are obtained by optimizing the physical distance-based adjacency matrix through a relevance-based attention mechanism:

[0025]

[0026] Where X is the attention matrix representing the feature differences between different channels obtained after learning based on the adaptive network layer;

[0027] S2: Construct a multi-branch graph convolutional network;

[0028] The multi-branch graph convolutional network includes a first branch and a second branch in parallel;

[0029] The first branch and the second branch have the same structure, both including an input layer, a channel attention module, and a spatiotemporal graph convolutional neural network;

[0030] The first branch is used to extract emotional information based on physical distance from the EEG; its input layer is used to receive node features and edge features A1 to form feature map G1;

[0031] The second branch is used to extract emotional information optimized by the attention mechanism based on relevance connections of EEG; its input layer is used to receive node features and edge features A2 to form feature map G2;

[0032] The channel attention module takes into account that different regions have different effects on emotion classification, and this effect is dynamic and changes during the emotion generation process. To dynamically and automatically extract spatial attention, this invention utilizes a spatial attention mechanism, defined as follows:

[0033] Attn s mn =V s ·σ((X s Z1)Z2(Z3X s )+C s Equation (4)

[0034] Attn s mn =SoftMax(Attn) s m Equation (5)

[0035] Attn s Represents the spatial attention matrix; X s ∈RN×W×T is the corresponding spatial layer input; Attn s mn ∈R W×W represents the importance of the edge composed of channel m and channel n; σ represents the sigmoid activation function; SoftMax represents the matrix normalization operation; V s ∈R T×T , C s ∈R T×T , Z1∈R W×1 ,Z2∈R W×N ,Z3∈R W×1 is the parameter to be learned, N is the number of EEG channels, T is the context length of the sample to be classified, and W is the feature dimension;

[0036] Attn s is the spatial attention matrix, which is dynamically calculated by the input of the current layer of the graph convolution layer and dynamically updated in the subsequent graph convolution block. If the ith electrode pair contributes more to the classification task, the value of the attention coefficient corresponding to the electrode pair is larger, and vice versa.

[0037] The spatio-temporal graph convolutional neural network comprises a time attention block, a graph convolution block, and a decision fusion layer.

[0038] ①The time attention module takes into account that there is correlation between adjacent emotional segments in the time dimension, and the correlation is different in different cases. Therefore, the time attention mechanism is used to capture the dynamic time information in the brain emotional network. It is defined as follows:

[0039] Attn t =V t ·σ((X t M1)MZ2(M3X t )+C t ) Equation (6)

[0040] Attn t mn =SoftMax(Attn t m ) Equation (7)

[0041] wherein Attn t is the time attention matrix; X t ∈R N×W×T is the corresponding time step input; Attn t mn ∈R W×W represents the correlation between segments, that is, the contribution of the nth time step segment information to the classification of the mth time step sample; the SoftMax function represents the normalization operation; σ represents the sigmoid activation function; Vt ∈R T×T , C t ∈R T×T , M1∈R W×1 ,M2∈R W×N ,M3∈R W×1 are parameters to be learned, N is the number of EEG channels, T is the context length of the sample to be classified, and W is the feature dimension;

[0042] Attn t is a time attention matrix, and the input of different graph convolution layers is adjusted by the time attention matrix to pay attention to time data with richer emotional information.

[0043] ② The graph convolution block considers the difference in information contained in the adjacency matrix constructed in different ways, and in the application, an adjacency matrix A d based on physical connection and an attention matrix X based on correlation connection are established at the same time, in order to more intuitively fuse the information of the two matrices, the application adopts a branch to multiply A d point by point and to use X to assist in optimizing A d , the adjacency matrix obtained after optimization considers both physical information and correlation information, which can provide the model with more comprehensive and rich relationships between channels, helping the model to extract more effective and emotion-related features. The graph convolution block receives the feature map G after the channel attention weight is assigned by the channel attention module, and the Laplacian matrix of G can be obtained according to formula (8).

[0044] L=D-A Equation (8)

[0045] where D is the degree matrix of the channel, that is, D is a diagonal matrix with elements a ij , a ij representing the degree of connection between channel i and channel j; A is the adjacency matrix used by the graph convolution block, and each element in the matrix A represents the degree of connection between different channels; the normalized Laplacian matrix is calculated according to formula (9):

[0046]

[0047] where E is an identity matrix;

[0048] The Chebyshev polynomial term is calculated according to formula (10):

[0049]

[0050] where θ k is the k-order Chebyshev polynomial coefficient, and T k () is the k-order Chebyshev polynomial, which is used to replace the filter.

[0051] After the graph convolution calculation, an output feature map G is obtained s , and the specific calculation manner is as follows:

[0052] Gs=σ(g(L)G) Formula (11)

[0053] wherein G represents the feature map G1 or G2, σ represents a graph convolution parameter, G s represents the feature map G1 or G2 after fusion after the graph convolution;

[0054] ③ The decision fusion layer fuses the output of the first branch and the output of the second branch through a fusion function to obtain a feature map G s ;

[0055] S3: constructing a prediction classifier;

[0056] The prediction classifier comprises one full connection layer and one SoftMax layer, and accepts the output G s of the graph convolution block to further predict a classification result;

[0057] As a preferred, in the training process, the spatio-temporal graph convolutional neural network adopts a back propagation algorithm and a cross-entropy loss function to optimize network parameters, and adopts a Relu activation function to reduce the mutual dependence relationship between parameters.

[0058] As a preferred, in the training process, the loss function of the multi-branch graph convolutional network joint prediction classifier is:

[0059]

[0060] wherein H is the number of samples, R y is the number of classifications, is a predicted classification result, and y is a true label.

[0061] In a second aspect, the present application provides a multi-dimensional electroencephalogram emotion recognition system, characterized in comprising

[0062] An electroencephalogram signal acquisition module is configured to acquire EEG signals of a user;

[0063] A data preprocessing module is configured to preprocess the acquired EEG signals to obtain feature maps G1 and G2;

[0064] An electroencephalogram emotion recognition module is configured to use the trained and verified multi-branch graph convolutional network to recognize emotions of the feature map G to obtain a feature map G s , and input G s into a prediction classifier to obtain a classification result;

[0065] A result visualization module is configured to display the classification result output by the EEG emotion recognition module in real time.

[0066] In a third aspect, the present application provides a computer readable storage medium having stored thereon a computer program which, when executed in a computer, causes the computer to perform the method.

[0067] In a fourth aspect, the present application provides a computing device comprising a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement the method. Compared with the prior art, the present application has the following beneficial effects:

[0068] (1) Method: The graph signal feature data modeled in the present application restores the spatial and functional connection of the data itself, and retains more emotional feature information by using multi-dimensional features, which can be used for accurate classification and recognition of individual emotional states. Meanwhile, the present application combines the attention mechanism to well distinguish different sample data in terms of emotional information richness, thereby greatly reducing the interference of redundant data on the classification performance, and has good classification effect and generalization ability.

[0069] (2) Model: The present application considers the difference of information contained in the adjacency matrix constructed in different ways, and proposes a multi-branch graph convolution model. The difference of the present application from the previous model lies in that it considers the difference of information between different adjacency matrices, and optimizes the adjacency matrix based on the physical distance connection by increasing the sub-module to utilize the attention mechanism based on the correlation connection. The optimized adjacency matrix contains more emotional information, and the matrix can provide more comprehensive and rich relationships between channels for the model, which helps the model to extract more effective and emotion-related features.

[0070] (3) Practicality: The present application uses an online brain-computer interface system, which changes the previous phenomenon of analyzing offline data, improves the real-time performance of electroencephalogram data analysis, and thus makes the analysis results of electroencephalogram data be applied in time and better.

[0071] (4) Scalability: The present application adopts a module independent development method, and each module has good independence, so that the function of any module can be improved and expanded. For example, the attention module can use other ways to calculate the attention index, without affecting the normal function of other modules. BRIEF DESCRIPTION OF DRAWINGS

[0072] Figure 1 A schematic diagram of the steps of the electroencephalogram emotion feature recognition method based on the multi-branch graph convolution network in the online system of the present application is provided.

[0073] Figure 2A framework diagram of a multi-branch graph convolutional network.

[0074] Figure 3 A brain-computer interface system interface (UI) diagram. DETAILED DESCRIPTION

[0075] In order to make the implementation process of the present application clearer, the following will be described in detail in conjunction with the accompanying drawings.

[0076] As Figure 1 The present application provides a method for electroencephalogram emotion recognition based on a spatio-temporal graph convolutional neural network, which comprises the following steps:

[0077] S1: obtaining emotional electroencephalogram data and pre-processing the emotional electroencephalogram data;

[0078] S101: the obtained emotional electroencephalogram data can be experimental data of a subject collected by oneself, can also be an existing data set, and can also be a new data set composed of the two. The present application uses the public electroencephalogram emotion data set DEAP data set and the public SEED emotional electroencephalogram data set of Shanghai Jiaotong University as examples, wherein the DEAP data set collects electroencephalogram data of 32 (16 males and 16 females) healthy participants. The participants in the experiment are in a healthy state in body and mind, and the electroencephalogram signals are collected by a 32-lead (see electrode distribution diagram) electrode cap according to the "10-20" international lead standard. The volunteers are required to watch 40 one-minute long videos, and the EEG signals of the subjects are collected at a sampling frequency of 512 Hz. All subjects are required to mark the size of Valence (valence), Arousal (arousal), and Dominance (dominance) of the video watched according to the size from 1 to 9 after watching the video. In the SEED data set, a total of 15 subjects (7 males and 8 females) participated in the experiment, and in each experiment, 15 movie clips were played to evoke the emotions corresponding to the videos. The ESI NeuroScan System with 62 channels was used to record the electroencephalogram signals in the SEED data set experiment, and the sampling frequency was 1000 Hz. The 32-channel DEAP electroencephalogram emotion data and the 62-channel SEED electroencephalogram emotion data are filtered by a band-pass filter respectively, and the data is divided into data of four frequency bands of θ (4-8 Hz), α (8-14 Hz), β (14-31 Hz), and γ (31-50 Hz);

[0079] S102: In order to increase more sample data, we perform sliding window operation on the electroencephalogram data obtained from the previous two data sets, and the window size is 3s. After completing the slicing operation of the data, we obtain a large amount of sample data, and we divide the data set, taking the data of 28 subjects in the DEAP data set as the training set, and taking the data of the remaining 4 subjects as the test set. Taking the data of 12 subjects in the SEED data set as the training set, and taking the data of the remaining 3 subjects as the test set;

[0080] S103: According to the electrode position of international 10-20 system, the feature extraction is performed on the electroencephalogram data after slicing, the extracted feature data is converted from 1-dimensional data to 2-dimensional plane format, and 4 frequency band two-dimensional planes of each electroencephalogram sample are obtained. The 2-dimensional planes of 4 frequency bands are integrated to obtain 3-dimensional electroencephalogram feature data as node features; S104: Obtain edge feature A1 based on physical distance;

[0081] The edge feature A1 based on physical distance is obtained by point-by-point multiplication fusion of two same adjacency matrices based on physical distance, so that the edge feature A1 based on physical distance is obtained;

[0082] S105: Obtain edge feature A2 considering correlation connection;

[0083] The edge feature A2 is obtained by optimizing the adjacency matrix based on physical distance through the attention mechanism based on correlation connection;

[0084] S2: Construct a multi-branch graph convolution network, such as Figure 2 ;

[0085] The multi-branch graph convolution network includes a parallel first branch and a second branch;

[0086] The first branch and the second branch have the same structure, and each includes an input layer, a channel attention module and a spatio-temporal graph convolutional neural network;

[0087] The first branch is used for extracting the emotional information of electroencephalogram based on physical distance; the input layer is used for receiving node features and edge features A1 to form a feature map G1;

[0088] The second branch is used for extracting the emotional information of electroencephalogram based on the attention mechanism based on correlation connection; the input layer is used for receiving node features and edge features A2 to form a feature map G2;

[0089] When the brain is stimulated by external stimuli, the activation levels of different regions of the brain are different, resulting in different influences of different brain regions on emotion recognition, and different regions correspond to different electrode channels, so it is necessary to consider the importance of different electrode channels. The spatial attention module is to model the correlation between channels, and is used to enhance the feature propagation between more important edges in the subsequent graph convolution operation, that is, to pay more attention to the channels that contribute more to the emotion classification task. Define a graph model G=(V, E), V represents a vertex, and E represents an edge in the graph. Each electroencephalogram channel corresponds to a vertex in the graph. The adjacency matrix A of the brain graph is determined according to the Euclidean distance between nodes. In the present application, the size of the corresponding adjacency matrix for the SEED dataset is 62x62, and the size of the corresponding adjacency matrix for the DEAP dataset is 32x32.

[0090] The channel attention module extracts a spatial attention matrix using a spatial attention mechanism;

[0091] The spatio-temporal graph convolutional neural network comprises a temporal attention block, a graph convolution block and a decision fusion layer.

[0092] ①The temporal attention module considers that there is correlation between adjacent emotion segments in the time dimension, and the correlation is different in different cases. Therefore, the temporal attention mechanism is used to capture dynamic time information in the brain emotion network.

[0093] ②The graph convolution block considers the difference in information contained in the adjacency matrix constructed in different ways. In the present application, an adjacency matrix A d based on physical connection and an attention matrix X based on correlation connection are established at the same time. In order to more intuitively fuse the information of the two matrices, the present application adopts a branch to multiply A d point by point and to use X to assist in optimizing A d . The adjacency matrix obtained after optimization considers both physical information and correlation information, which can provide the model with more comprehensive and rich relationships between channels, helping the model to extract more effective and emotion-related features.

[0094] ③The decision fusion layer fuses the output of the first branch and the output of the second branch through a fusion function to obtain a feature map G s .

[0095] S3: Construct a prediction classifier;

[0096] The prediction classifier comprises a fully connected layer and a SoftMax layer, and accepts the output G s of the graph convolution block to predict the classification result;

[0097] Graph convolutional neural network is a deep learning network combining CNN and graph theory, which has the advantage of processing spatially discrete data (non-Euclidean data). The graph signal obtained after feature extraction is used as the input of the network, and the graph features are extracted from the input data by using the characteristics of the graph convolutional network. By mining the graph domain features and time domain information, the performance of EEG emotion recognition is improved, which provides a guarantee for the subsequent prediction and classification recognition research. As known from the above, the basic algorithm steps for training the proposed model are as follows:

[0098] 1. Initialize model parameters: learning rate p, Chebyshev polynomial order k, and iteration number e.

[0099] 2. Input: sample set, label label, input feature segment number T.

[0100] 3. The output is the trained model parameters.

[0101] 4. Calculate the adjacency matrix A based on the Euclidean distance d and the attention matrix X based on the correlation connection, and based on the interaction of A d and X, get the multi-branch adjacency matrix A1 and A2, and A1 and A2 are collectively referred to as A below. The elements of matrix A are regularized using the Relu activation function.

[0102] 5. Calculate the normalized Laplacian matrix L according to the formula A and D∈R N×N , where D is a diagonal matrix and E is a unit matrix.

[0103] 6. Calculate the Chebyshev polynomial term according to the formula , where θ k is the Chebyshev polynomial coefficient, and T k () is the calculation method of Chebyshev polynomial.

[0104] 7. Calculate

[0105] 8. Regularize the result using the Relu function.

[0106] 9. The output of the graph convolution block is input to the prediction classification block after being processed by the time attention block.

[0107] 10. Calculate the output result of the fully connected layer.

[0108] Use to calculate the loss function, and update the multi-branch graph convolution network joint prediction classifier model parameters.

[0109] The number of training times is set to 100-300, the loss function is set to a cross-entropy loss function, Adam is selected as an optimization algorithm, the initial learning rate is set to 0.001, and learning rate decay is added. The learning rate is linearly decayed with the increase of the number of iterations, so as to help the model update at a smaller amplitude in the later training. The training set divided in step S1 is used to train the constructed neural network, and the test set divided in step S1 is used to test the neural network in training. If overfitting occurs, adjust the learning rate and retrain the neural network until overfitting no longer occurs. In this way, the parameters of the neural network can be adjusted in time, and a preliminarily trained spatio-temporal graph convolutional neural network can be obtained faster and better. For example Figure 2 The multi-branch graph convolutional network includes two branches, and the sub-modules of feature fusion and decision fusion added in the model can provide more comprehensive and rich relationships between channels for the model, which helps the model to extract more effective and emotion-related features.

[0110] In the multi-branch graph convolutional model, the full connection layer makes a prediction classification on the output of the graph convolutional layer to obtain a classification result.

[0111] An online brain-computer interface system for implementing electroencephalogram emotion recognition based on a spatio-temporal graph convolutional neural network, comprising:

[0112] An electroencephalogram signal acquisition module for acquiring EEG signals of a user;

[0113] A data preprocessing module for preprocessing the acquired EEG signals to obtain feature maps G1 and G2;

[0114] An electroencephalogram emotion recognition module for recognizing emotions of the feature map G using the trained and verified multi-branch graph convolutional network, and inputting the output G s into a prediction classifier to obtain a classification result;

[0115] A result visualization module for displaying the classification result output by the electroencephalogram emotion recognition module in real time, as shown in Figure 3 .

[0116] For those skilled in the art, some improvements and refinements can be made without departing from the principles of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. An electroencephalogram emotion recognition method based on a multi-branch graph convolution network, characterized in that The method comprises the following steps: S1: Obtain emotional electroencephalogram data and pre-process the emotional electroencephalogram data; S101: Filter the emotional electroencephalogram data by using a 1-50 Hz band-pass filter, and then divide the emotional electroencephalogram data into a theta frequency band, an alpha frequency band, a beta frequency band and a gamma frequency band; S102: Slice the above-mentioned electroencephalogram data of different frequency bands, and perform a sliding window operation with a fixed length s; S103: Obtain node features; According to the electrode position of the international 10-20 system, the features of the sliced electroencephalogram data are extracted, the extracted feature data is converted from 1-dimensional data to 2-dimensional plane format, and 4 frequency band 2-dimensional planes of each electroencephalogram sample are obtained, and 3-dimensional electroencephalogram feature data is obtained as node features by integrating the 2-dimensional planes of the 4 frequency bands; S104: Obtain edge features A1 based on physical distance; The edge features A1 are obtained by point-by-point multiplication of two same adjacency matrices based on physical distance, so that the edge features A1 are more rich in emotional information; wherein denotes point-wise multiplication; A d denotes the adjacency matrix based on physical connections, which is calculated as follows: where τ represents a preset threshold, θ a denotes a fixed parameter, and dist(x, y) represents the distance between the xth node and the yth node. S105: Obtain edge features A2 considering correlation connection; The edge features A2 are obtained by optimizing the adjacency matrix based on physical distance through a correlation connection-based attention mechanism: Wherein X is an attention matrix representing the feature difference between different channels after learning based on an adaptive network layer; S2: Construct a multi-branch graph convolutional network; The multi-branch graph convolutional network comprises a first branch and a second branch in parallel; The first branch and the second branch have the same structure and each comprises an input layer, a channel attention module and a spatio-temporal graph convolutional neural network; The first branch is used for extracting emotional information of electroencephalogram based on physical distance, and the input layer thereof is used for receiving node features and edge features A1 to form a feature map G1; The second branch is used for extracting emotional information of electroencephalogram optimized by a correlation connection-based attention mechanism, and the input layer thereof is used for receiving node features and edge features A2 to form a feature map G2; The channel attention module extracts a spatial attention matrix by using a spatial attention mechanism, and the spatial attention matrix is defined as follows: Attn s = V s • σ1((X s Z1)Z2(Z3X s ) + C s ) Equation (4) Attn s mn = SoftMax(Attn s m ) Equation (5) where Attn s denotes the spatial attention matrix; X s ∈R N×W×T is the corresponding spatial layer input; Attn s mn ∈R W×W denotes the importance of the edge composed of channel m and channel n; σ1 denotes the sigmoid activation function; SoftMax denotes the matrix normalization operation; V s ∈R T ×T , C s ∈R T×T , Z1∈R W×1 , Z2∈R W×N , Z3∈R W×1 is the parameter to be learned, N is the number of EEG channels, T is the context length of the sample to be classified, and W is the feature dimension; The spatio-temporal graph convolutional neural network comprises a time attention block, a graph convolution block and a decision fusion layer; ①The time attention block considers that there is correlation between adjacent emotional segments in the time dimension, and the correlation is different in different cases; therefore, a time attention mechanism is used to capture dynamic time information in the brain emotional network, and the time attention mechanism is defined as follows: Attn t = V t • σ1((X t M1)M2(M3X t ) + C t ) Equation (6) Attn t ef = SoftMax(Attn t e ) Equation (7) where Attn t is the time attention matrix; X t ∈R N×W×T is the input at the corresponding time step; Attn t ef ∈R W×W represents the correlation between the segments, that is, the contribution of the segment information at the fth time step to the classification of the sample at the eth time step; the SoftMax function represents the normalization operation; σ1represents the sigmoid activation function; V t ∈R T×T , C t ∈R T×T , M1∈R W×1 , M2∈R W×N , M3∈R W×1 are parameters to be learned, N is the number of EEG channels, T is the context length of the sample to be classified, and W is the feature dimension; The graph convolution block adopts a multi-layer Gcn stack; the graph convolution block receives the feature map G output after the channel attention weight is given by the channel attention module attn G attn The Laplacian matrix of G can be obtained according to formula (8); L = D - A Equation (8) Where D is the degree matrix of the channels, that is, D is a matrix with elements a cd a diagonal matrix cd The degree of connection between channel c and channel d is represented; A is the adjacency matrix used by the graph convolution block, and each element in matrix A represents the degree of connection between different channels; the normalized Laplacian matrix is ​​calculated according to formula (9): Wherein E is a unit matrix; Calculate the Chebyshev polynomial term according to formula (10): where θ k are the coefficients of the k-th order Chebyshev polynomial, T k () is the k-th order Chebyshev polynomial, which is used to replace the role of the filter; After the graph convolution calculation, the output feature map G is obtained s The specific calculation method is as follows: G s = σ(g(L)G) Equation (11) Wherein G represents the feature map G1 or G2, and σ represents a graph convolution parameter; ③ the decision fusion layer fuses the output of the first branch and the output of the second branch through a fusion function to obtain a feature map G s ; S3: Construct a prediction classifier; The prediction classifier includes a full connection layer and a SoftMax layer, accepts the output G of the decision fusion layer s , and further predicts a classification result.

2. The method of claim 1, wherein The time-frequency features are extracted in step S103.

3. The method of claim 1, wherein In the training process, the spatio-temporal graph convolutional neural network optimizes the network parameters by using a back propagation algorithm and a cross-entropy loss function, and reduces the mutual dependence between parameters by using a Relu activation function.

4. The method of claim 1, wherein In the training process, the loss function of the multi-branch graph convolutional network combined with the prediction classifier is as follows: where H is the number of samples, R y is the number of classes, is the predicted class result, y is the true label.

5. A multi-dimensional electroencephalogram emotion recognition system for implementing the method of any one of claims 1-4, characterized in that Including: An electroencephalogram signal acquisition module is configured to acquire EEG signals of a user; A data pre-processing module is configured to pre-process the acquired EEG signals to obtain feature maps G1 and G2; The electroencephalogram emotion recognition module adopts a multi-branch graph convolution network trained and verified to perform emotion recognition on the feature map G to obtain the feature map G s , and input G s to a prediction classifier to obtain a classification result. A result visualization module is configured to display the classification result output by the EEG emotion recognition module in real time. 6.A computer readable storage medium having stored thereon a computer program which, when executed in a computer, causes the computer to perform the method of any one of claims 1-4. 7.A computing device comprising a memory and a processor, the memory having stored therein executable code, the processor executing the executable code to implement the method of any one of claims 1-4.

Images