EEG emotion recognition method based on EEG emotion recognition network

Abstract

The present application discloses an EEG emotion recognition method based on an EEG emotion recognition network, which relates to the field of artificial intelligence. The EEG emotion recognition network includes a temporal feature extraction network and a spatial feature extraction network. The method comprises: obtaining EEG signals within a first time period, wherein the EEG signals include local EEG signals corresponding to different brain functional areas; using the temporal feature extraction network to perform feature extraction on EEG signal segments at different times within the first time period to obtain temporal features of the EEG signals, wherein the temporal features include the fluctuation state of the EEG signals within different time periods; using the spatial feature extraction network to perform feature extraction on the local EEG signals to obtain spatial features of the EEG signals, wherein the spatial features include the fluctuation state of local EEG signals and global EEG signals within different brain functional areas; performing emotion recognition based on the temporal features and spatial features to obtain the emotion category to which the EEG signals belong. This method can improve the accuracy of emotion recognition.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to an EEG emotion recognition method based on an EEG emotion recognition network. Background Art

[0002] Emotions play a crucial role in human-computer interaction. Their recognition enables computer systems to provide more humane and intelligent services. Through emotion recognition technology, computers can classify human emotions into different categories. Emotion recognition can be achieved by identifying both external manifestations and internal signals. External manifestations can manifest as behavioral signals such as facial expressions, verbal expressions, and body posture. However, these non-physiological signals often suffer from noise and camouflage. In contrast, internal physiological signals can directly reflect an individual's emotional state. These signals include multi-channel recordings such as electroencephalography (EEG) and electromyography (EMG), encompassing information from both the central and autonomic nervous systems. Among these signals, EEG offers the advantages of being non-invasive, easy to operate, and readily available for collection. Consequently, a growing number of studies are focusing on using EEG (electroencephalography) for emotion recognition.

[0003] In recent years, deep learning-related methods have been increasingly used in the field of EEG emotion recognition. Among them, models based on convolutional neural networks and recurrent temporal networks have been used in EEG emotion recognition, and related models based on these two models have also been gradually proposed. However, these methods can only extract features from a single channel and have certain limitations in processing multi-channel features. This is because the distribution between channels is irregular, and these networks can only process structured data, which affects the recognition of EEG signals. Recently, graph neural networks have been introduced to the field of emotion recognition and are used to process unstructured data to better understand the relationships between spatial channels. Although recent research has begun to use multiple channels to construct a graph relationship to extract spatial features, most studies only use the global graph structure, ignoring local information, resulting in low accuracy in EEG emotion recognition. Summary of the Invention

[0004] In response to the above problems and technical needs, the applicant has proposed an EEG emotion recognition method based on an EEG emotion recognition network. The technical solution of this application is as follows:

[0005] On the one hand, a method for EEG emotion recognition based on an EEG emotion recognition network is provided, comprising the following steps:

[0006] Acquiring an EEG signal within a first period of time, wherein the EEG signal at least includes local EEG signals corresponding to different brain functional areas;

[0007] Using the time feature extraction network to extract time features from EEG signal segments at different times within the first time, to obtain time features of the EEG signal, wherein the time features include fluctuation states of the EEG signal within different time periods;

[0008] Performing spatial feature extraction on the local EEG signal using the spatial feature extraction network to obtain spatial features of the EEG signal, wherein the spatial features include fluctuation states of local EEG signals in different brain functional areas and global EEG signals;

[0009] Emotion recognition is performed based on the temporal feature and the spatial feature to obtain the emotion category to which the EEG signal belongs.

[0010] Its further technical solution is:

[0011] The step of obtaining the time characteristic of the EEG signal includes:

[0012] Using the time feature extraction network to extract time features from each of the EEG signal segments to obtain a time segment feature set, wherein the time segment feature set includes time segment features of each of the signal segments at different moments;

[0013] fusing the time period features contained in the time period feature set to obtain the time features of the EEG signal;

[0014] The step of obtaining the spatial features of the EEG signal includes:

[0015] Performing spatial feature extraction on each of the local EEG signals using the spatial feature extraction network to obtain a local feature set, wherein the local feature set includes local features of the local EEG signals corresponding to each brain functional area;

[0016] The local features included in the local feature set are fused to obtain the spatial features of the EEG signal.

[0017] The temporal feature extraction network includes a bidirectional short-term memory network BiLSTM;

[0018] The step of extracting time features from each of the signal segments using the time feature extraction network to obtain a time segment feature set includes:

[0019] Dividing the EEG signal within the first time into segments to obtain EEG signal segments at different moments;

[0020] Extracting initial features of each of the EEG signal segments, and inputting each of the initial features into the BiLSTM network for feature extraction to obtain bidirectional output features corresponding to each of the EEG signal segments;

[0021] Based on the bidirectional output feature, a time segment feature of the EEG signal segment is determined.

[0022] The fusing of the time period features included in the time period feature set to obtain the time features of the EEG signal includes:

[0023] Determining an attention coefficient of an EEG signal segment corresponding to the time period feature based on a correlation between the time period feature and a time feature sequence, wherein the time feature sequence includes the time period feature corresponding to each EEG signal segment;

[0024] Based on the attention coefficient of each of the EEG signal segments, the features of each of the time periods are fused to obtain the time features of the EEG signal.

[0025] The spatial feature extraction network includes a graph attention network;

[0026] The utilizing the spatial feature extraction network to extract spatial features from each of the local EEG signals comprises:

[0027] Divide the brain regions into different functional areas;

[0028] Inputting the local EEG signal of the brain functional area into the graph attention network for feature extraction to obtain a node feature set corresponding to the brain functional area, wherein the node feature set includes node features corresponding to the EEG signal of each node, and the node EEG signal is the EEG signal corresponding to a single node in the brain functional area;

[0029] The node features in the node feature set are aggregated to obtain the local features corresponding to the brain functional areas.

[0030] The aggregating the node features in the node feature set to obtain the local features corresponding to the brain functional areas includes:

[0031] The feature aggregation of each node feature is performed through an aggregation function to obtain the local feature corresponding to the brain functional area, and the aggregation function is one of an average value function, a maximum value function or a minimum value function.

[0032] The fusing of the local features included in the local feature set to obtain the spatial features of the EEG signal includes:

[0033] Each of the local features is input into the graph attention network for feature extraction to obtain the spatial features of the EEG signal.

[0034] The performing emotion recognition based on the temporal feature and the spatial feature to obtain the emotion category to which the EEG signal belongs includes:

[0035] Fusing the temporal feature with the spatial feature to obtain the spatiotemporal feature of the EEG signal;

[0036] Inputting the spatiotemporal features into a classifier for emotion recognition, and obtaining the probability that the spatiotemporal features belong to different emotion categories;

[0037] The emotion category corresponding to the maximum probability value is determined as the emotion category to which the EEG signal belongs.

[0038] The EEG emotion recognition network also includes a spatiotemporal feature extraction network;

[0039] The step of fusing the temporal feature with the spatial feature to obtain the spatiotemporal feature of the EEG signal includes:

[0040] Inputting the temporal features and the spatial features into the spatiotemporal feature extraction network to perform feature extraction, thereby obtaining local spatiotemporal features corresponding to each brain functional area;

[0041] Determining the attention coefficient of the brain functional area corresponding to the local spatiotemporal feature based on the correlation between the local spatiotemporal feature and the spatiotemporal feature sequence, wherein the spatiotemporal feature sequence includes the local spatiotemporal features corresponding to each of the brain functional areas;

[0042] Based on the attention coefficient corresponding to each of the brain functional areas, each of the local spatiotemporal features is fused to obtain the spatiotemporal features of the EEG signal.

[0043] The EEG emotion recognition network is trained based on sample EEG signals, the sample EEG signals are annotated with real emotion labels, the sample EEG signals are signals within the second time period, and the sample EEG signals are composed of sample local EEG signals corresponding to different brain functional areas. The method further includes:

[0044] Constructing an initial EEG emotion recognition network, wherein the initial EEG emotion recognition network includes an initial time feature extraction network and an initial spatial feature extraction network;

[0045] Using the initial time feature extraction network, extracting time features from sample signal segments at different moments within the second time to obtain sample time features of the sample EEG signals;

[0046] Performing spatial feature extraction on the sample local EEG signal using the initial spatial feature extraction network to obtain sample spatial features of the sample EEG signal;

[0047] Performing emotion recognition based on the sample time feature and the sample space feature to obtain a predicted emotion category to which the EEG signal belongs;

[0048] A classification loss is determined based on the predicted emotion category and the true emotion category, and when the classification loss indicates that the training state reaches a training end condition, the training of the initial EEG emotion recognition network is stopped to obtain the EEG emotion recognition network.

[0049] The beneficial technical effects of this application are:

[0050] In the present application, a brain wave emotion recognition network is provided, which includes a time feature extraction network and a spatial feature extraction network. For the brain wave signal to be recognized, the time feature extraction network can extract the signal features of the brain wave signal segments at different times, thereby capturing the emotional intensity at different times to better obtain the emotional fluctuation information; and the spatial feature extraction network can extract the signal features corresponding to the local brain wave signals of different brain functional areas and the signal features corresponding to the global brain wave signals, and can obtain the emotional activity corresponding to different brain functional areas. That is, the brain wave emotion recognition network provided in the embodiment of the present application can perform emotion recognition based on the emotional fluctuation at different times and the emotional activity in different brain functional areas, which can improve the accuracy of emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a flowchart of an EEG emotion recognition method based on an EEG emotion recognition network provided by an exemplary embodiment of the present application;

[0052] Figure 2 is a flowchart of an EEG emotion recognition method based on an EEG emotion recognition network provided by another exemplary embodiment of the present application;

[0053] Figure 3 is a schematic diagram of a BiLSTM network provided by an exemplary embodiment of the present application;

[0054] Figure 4 This is a schematic diagram of the regional division of brain functional areas provided by an exemplary embodiment of the present application;

[0055] Figure 5 This is a schematic diagram of the structure of an EEG emotion recognition network provided by an exemplary embodiment of the present application;

[0056] Figure 6 This is a flowchart of an EEG emotion recognition network training method provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0057] The specific implementation of this application will be further described below with reference to the accompanying drawings.

[0058] In an embodiment of the present application, an EEG emotion recognition network is proposed for identifying the emotion category to which an EEG signal belongs. The EEG emotion recognition network includes a temporal feature extraction network for extracting signal features of the EEG signal in the temporal dimension, and a spatial feature extraction network for extracting signal features of the EEG signal in the spatial dimension. The recognition process is illustrated below using an exemplary embodiment.

[0059] Please refer to Figure 1 , which shows a flowchart of an EEG emotion recognition method based on an EEG emotion recognition network provided by an exemplary embodiment of the present application, which can be executed by a computer device. The method includes the following steps:

[0060] Step 101: Acquire an EEG signal within a first period of time, where the EEG signal at least includes local EEG signals corresponding to different brain functional areas.

[0061] In one possible embodiment, an EEG device is used to collect EEG signals. During the EEG signal collection process, electrodes are placed on the scalp for collection. The electrodes include an active electrode and a reference electrode. The signal collected by the reference electrode serves as the baseline signal for the EEG signal. The final collected EEG signal is the potential difference between the signal collected by the active electrode and the signal collected by the reference electrode.

[0062] During the collection process, at least one reference electrode must be placed on the subject's head to serve as a reference signal for the EEG signal. Optionally, the earlobe or nose can be used as the placement area for the reference electrode.

[0063] For the collected EEG signals, the EEG signals within the first time are the EEG signals collected within the sampling time. And the signals collected in the same brain functional area constitute the local EEG signals corresponding to the brain functional area. After obtaining the EEG signals, the EEG signals can be first matrix-converted and used as the input of the EEG emotion recognition network. In one possible implementation, the EEG signals can be converted into an n×L matrix for input into the EEG emotion recognition network for emotion recognition, where n is the number of channels, that is, the number of electrodes used when collecting EEG signals, and L is obtained according to the sampling time and sampling frequency when collecting EEG signals, L=sampling frequency×sampling time.

[0064] In another possible embodiment, the EEG signal may be a signal simulated and generated by a computer device. The EEG signal includes simulated local EEG signals corresponding to different EEG functional areas, and in this manner, the first time is the signal generation time corresponding to the simulated EEG signal generation process. This embodiment is not limited to this.

[0065] Step 102 : Using a time feature extraction network, extract time features from EEG signal segments at different moments within the first time period to obtain time features of the EEG signals.

[0066] Among them, the time feature includes the fluctuation state of the EEG signal in different time periods.

[0067] The intensity of emotions is constantly changing, and the intensity of emotions at different times is different. If only EEG signals within a certain period of time are used for emotion recognition, the emotional state within the entire time cannot be recognized. Therefore, in an embodiment of the present application, a time feature extraction network is used to extract features from EEG signal segments at different times in the entire EEG signal, and the signal features of the entire EEG signal in the time dimension can be obtained, capturing the fluctuation state of the EEG signal within the first time, thereby capturing the changes in emotions. It should be noted that in the process of feature extraction of EEG signal segments within a certain period of time, feature extraction is performed on all local EEG signals corresponding to all brain functional areas within the period, that is, feature extraction is performed on the entire EEG signal within the period.

[0068] In a possible implementation, time features may be extracted for signal segments at different moments, and then the extracted features may be fused to obtain the time features of the EEG signal over the entire period of time.

[0069] Step 103: Use a spatial feature extraction network to extract spatial features of the local EEG signal to obtain the spatial features of the EEG signal.

[0070] Among them, the spatial features include the fluctuation state of local EEG signals in different brain functional areas and global EEG signals. The global EEG signal is the EEG signal corresponding to the overall brain functional area.

[0071] During the emotional fluctuation process, multiple functional areas of the brain begin to become active, but the contribution of different functional areas to emotional activity is not the same. Based on the role of different functional areas in emotional activity, and combined with the fluctuation state of EEG signals in each functional area, it is more conducive to identifying the emotions represented by EEG signals. In the process of acquiring EEG signals, they can be acquired from several nodes in the brain functional area, and the signals acquired at the nodes belonging to the same brain functional area constitute the local EEG signals of the brain functional area. In the process of simulating the generation of EEG signals, local EEG signals corresponding to different brain functional areas will also be generated. In the embodiment of the present application, a spatial feature extraction network is used to extract features of the local EEG signals corresponding to different brain functional areas to obtain the spatial features of the EEG signals. It should be noted that in the process of extracting features from local EEG signals, feature extraction is performed on the entire local EEG signal within the first time.

[0072] In a possible implementation, spatial features are extracted from local EEG signals of different brain functional areas respectively, and then the extracted features are fused to obtain spatial features of the EEG signals.

[0073] Spatial features include not only the spatial features of each local EEG signal, but also the spatial features of the global EEG signal. Based on the signal characteristics of the local EEG signal and the global EEG signal, spatial features can be selectively selected for use in the emotion recognition process. The role of different brain functional areas in emotional activity can be integrated into the emotion recognition process, which can improve the accuracy of emotion recognition.

[0074] Step 104 : Emotion recognition is performed based on the temporal and spatial features to obtain the emotion category to which the EEG signal belongs.

[0075] In an embodiment of the present application, the time features and spatial features of the EEG signal are extracted from the time dimension and the space dimension respectively. When emotion recognition is performed based on the time features and the space features, it is possible to simultaneously perform recognition based on the emotional states in different time periods and the emotional states of different brain functional areas to obtain emotion categories.

[0076] In a possible implementation, the temporal features and the spatial features may be fused to obtain the spatiotemporal features of the EEG signal, and the spatiotemporal features are then used to perform emotion recognition to obtain the emotion category to which the EEG signal belongs.

[0077] In the present application, a brain wave emotion recognition network is provided, which includes a time feature extraction network and a spatial feature extraction network. For the brain wave signal to be recognized, the time feature extraction network can extract the signal features of the brain wave signal segments at different times, thereby capturing the emotional intensity at different times to better obtain the emotional fluctuation information; and the spatial feature extraction network can extract the signal features of the local brain wave signal corresponding to different brain functional areas and the signal features corresponding to the global brain wave signal, and can obtain the emotional activity corresponding to different brain functional areas. That is, the brain wave emotion recognition network provided in the embodiment of the present application can perform emotion recognition based on the emotional fluctuation at different times and the emotional activity in different brain functional areas, which can improve the accuracy of emotion recognition.

[0078] Please refer to Figure 2 , which shows a flowchart of an EEG emotion recognition method based on an EEG emotion recognition network provided by another exemplary embodiment of the present application, which can be executed by a computer device. The method includes the following steps:

[0079] Step 201: Acquire the EEG signal within the first time period.

[0080] Step 202: Use a time feature extraction network to extract time features from each EEG signal segment to obtain a time segment feature set.

[0081] The time segment feature set includes the time segment features of each EEG signal segment at different times.

[0082] In order to extract rich temporal features, in one possible implementation, the temporal feature extraction network may use a bidirectional long short-term memory network (BiLSTM). BiLSTM can use the preceding and following time sequence information for calculation, and therefore, can extract more information in the time dimension. In one possible implementation, the EEG signal can be divided into N time segments, and the EEG signal segment of each time segment is used as a unit of BiLSTM, so as to perform feature extraction on the EEG signal segment within each time segment. The temporal feature extraction process may include steps 202a-202c:

[0083] Step 202a: segment the EEG signal within the first time period to obtain EEG signal segments at different moments.

[0084] First, the EEG signal within the first time period can be divided into N equal-length, non-overlapping time periods. In one possible implementation, the time period length T of each time period can be pre-set, and the EEG signal within the first time period can be segmented based on the time period length T to obtain N EEG signal segments, where the first time = T × N. In another possible implementation, the number of segmented time periods N can be pre-set, and the EEG signal within the first time period can be segmented based on the number N to obtain EEG signal segments of each length T, where the first time = T × N.

[0085] Optionally, the EEG signal dimension is n×L, where n is the number of electrodes and L is the number of EEG signals, then the following relationship exists:

[0086] L=f s ×N×T

[0087] Where N is the number of EEG signal segments, and T is the length of the time period corresponding to each EEG signal segment. s is the sampling frequency of the EEG signal.

[0088] Step 202b: extracting initial features of each EEG signal segment, and inputting each initial feature into a BiLSTM network for feature extraction to obtain bidirectional output features corresponding to each EEG signal segment.

[0089] In the process of emotion recognition, the EEG signals that represent emotions are mainly concentrated in the range of 0-50Hz, including the Delta (0-4Hz), Theta (4-7Hz), Alpha (8-13Hz), Beta (14-30Hz) and Gamma (31-50Hz) frequency bands. These frequency bands have obvious synchronization and reaction during emotion recognition. Therefore, the EEG signals in this frequency band can be used for emotion recognition. Normally, EEG signals in the frequency bands of θ (4-7hz), α (8-13hz), β (14-30hz) and γ (>30hz) can be used. After obtaining the EEG signals of each frequency band, they need to be preprocessed, that is, the initial features are extracted and used as the input of the EEG emotion recognition network. In one possible implementation, the differential entropy (DE) feature of the EEG signal is used as the initial feature. For a segment with an expectation of μ and a variance of σ 2 , the EEG signal obeys Gaussian distribution, and its DE feature extraction method is as follows:

[0090]

[0091] Where e is a constant.

[0092] In one possible implementation, DE features of the target frequency bands (including θ (4-7 Hz), α (8-13 Hz), β (14-30 Hz), and γ (>30 Hz)) in each EEG signal segment can be extracted to obtain the input feature sequence, that is, the input feature sequence of the BiLSTM network is: Among them, x i represents the initial features extracted from the i-th EEG signal segment, k = n × T × d is the number of features input into BiLSTM, and d is the total number of frequency bands of the target frequency band extracted during the DE feature extraction process.

[0093] In the above method, after the EEG signal is divided into segments, the DE features of each EEG signal segment are extracted separately. In another possible implementation, the features of the entire EEG signal can be extracted first to obtain the DE features of the entire EEG signal, and then the segments are divided according to time to obtain the DE features of each EEG signal segment. This embodiment does not limit the specific division process.

[0094] The feature sequence is After input into the BiLSTM network, the BiLSTM network will extract time features.

[0095] In the feature extraction process, BiLSTM includes input gate i t 、Forget Gate t and output gate o t , based on the input gate i t 、Forget Gate tand output gate o t , the BiLSTM unit can learn how much information to store or discard, and t and the state h generated in the previous step t-1 , and temporary cell states Update hidden state C in time t , specifically:

[0096] i t =σ(W i ·[h t-1 ,x t ]+b i )

[0097]

[0098] f t =σ(W f ·[h t-1 ,x t ]+b f )

[0099] o t =σ(W o ·[h t-1 ,x t ]+b o )

[0100]

[0101] h t =o t tanh(C t )

[0102] Among them, σ is the sigmoid activation function, W is the weight matrix (including the weight matrix W of the input gate i , the weight matrix W of the temporary cell state C , the weight matrix W of the forget gate f And the weight matrix o of the output gate t , b is the bias matrix (including the bias matrix b of the input gate i , the deviation matrix b of the temporary cell state C , the bias matrix b of the forget gate f And the bias matrix b of the output gate o ), h t-1 is the state generated in step t-1, x t Represents the initial features corresponding to the t-th signal segment, and the value range of t is [1, N].

[0103] like Figure 3As shown in the figure, the BiLSTM network contains two parallel LSTM layers in forward and backward propagation. In the process of extracting signal segment features, bidirectional output features can be obtained, including forward output features and reverse output features:

[0104]

[0105]

[0106] in, is the forward output feature (forward hidden state), For the reverse output feature (reverse hidden state), LSTM f Indicates the forward propagation direction of BiLSTM, LSTM b Indicates the back propagation direction of BiLSTM.

[0107] Step 202c: determining the time period characteristics of the EEG signal segment based on the bidirectional output characteristics.

[0108] After obtaining the bidirectional output features of the EEG signal segment, the forward output features and the reverse output features can be connected to obtain the time segment features of the EEG signal segment.

[0109]

[0110] Among them, H t ∈R N×k′ That is, it is the time segment feature of the t-th EEG signal segment, and k′ is the number of features output by the BiLSTM network.

[0111] In this process, the time segment features of each EEG signal segment can be obtained, that is, the time segment feature set can be obtained.

[0112] Step 203: The time period features included in the time period feature set are integrated to obtain the time features of the EEG signal.

[0113] After obtaining the time period feature set, the features of each time period contained therein can be fused to obtain the time features of the overall EEG signal. The fusion process includes steps 203a-203b, which are as follows:

[0114] Step 203a, determining the attention coefficient of the EEG signal segment corresponding to the time period feature based on the correlation between the time period feature and the time feature sequence, wherein the time feature sequence includes the time period feature corresponding to each EEG signal segment;

[0115] Emotional intensity changes over time. To accurately capture this characteristic of emotional intensity over time and fully utilize all the hidden state information extracted by BiLSTM, this embodiment introduces an attention module to enhance the representation of key signal segments. The attention module assigns different weights to each EEG signal segment, which represents the EEG signal segment's contribution to emotion recognition. In this embodiment, the weight corresponding to each EEG signal segment is represented by the attention coefficient.

[0116] For the extracted time feature sequence H t =[x′1,x′2,...,x′ N ], which contains the time segment features corresponding to each EEG signal segment, x′ N That is, it represents the time period feature corresponding to the Nth EEG signal segment.

[0117] The correlation between the time segment feature of an EEG signal segment and the time feature sequence indicates the degree of correlation between the time segment feature and the time feature sequence. The higher the correlation, the higher the weight corresponding to the time segment feature. In one possible implementation, the attention coefficient of the EEG signal segment can be determined by using the correlation between the time segment feature corresponding to the EEG signal segment and the time feature sequence. The calculation method is as follows:

[0118]

[0119] Among them, q is the query vector, which is the time feature sequence H t Obtained through linear transformation, it is used to characterize the time feature sequence. Among them, z∈[1,N], s(x′ i ,q) represents the attention scoring function. In the embodiment of the present application, the attention scoring function adopts the dot product model, and the calculation method of the attention coefficient can be expressed as:

[0120]

[0121] Where x′ i T q represents the correlation between the time segment feature and the time feature sequence corresponding to the i-th EEG signal segment, α i Represents the attention coefficient corresponding to the i-th EEG signal segment, i∈[1,N].

[0122] Step 203b: Based on the attention coefficient of each EEG signal segment, the features of each time period are fused to obtain the time features of the EEG signal.

[0123] After obtaining the attention coefficient of each EEG signal segment through the above method, the time segment features corresponding to each EEG signal segment can be fused by weighted summation to obtain the time feature of the final EEG signal. The time feature can be expressed as:

[0124]

[0125] in, That is, the time feature, att represents the attention function, x′ i Represents the time segment characteristics of the i-th EEG signal segment.

[0126] like Figure 3 As shown, the time segment features can be fused through the attention layer to obtain the final time feature H′ t .

[0127] Step 204: Use a spatial feature extraction network to extract spatial features from each local EEG signal to obtain a local feature set.

[0128] The local feature set contains the local features of the local EEG signals corresponding to each brain functional area.

[0129] In order to extract the features of local EEG signals in different brain functional areas, in this embodiment, Graph Attention Networks (GAT) is used as a spatial feature extraction network to perform local feature extraction. Taking the collection of EEG signals through electrodes as an example, according to the collection position of each electrode in the collected EEG signal, an EEG map corresponding to each brain functional area can be generated, wherein each electrode (each channel) is regarded as a node of the EEG map. The local features of each brain functional area are extracted by the graph attention network to obtain a local feature set. Among them, the process of local feature extraction may include steps 204a-204c:

[0130] Step 204a, dividing the brain area into different brain functional areas, each of which contains a number of nodes for acquiring EEG signals.

[0131] First, the brain regions can be divided into regions. The regional division method can be determined according to actual needs. For example, Figure 4As shown in the figure, five different division methods are shown. Division method a divides the brain area into 9 different functional areas, division method b divides the brain area into 11 different functional areas, division method c divides the brain area into 14 different functional areas, division method d divides the brain area into 17 different functional areas, and division method e divides the brain area into 3 different functional areas. Each functional area contains several nodes that can acquire EEG signals. Taking electrode acquisition as an example, each functional area contains several electrode acquisition points, which are nodes.

[0132] In one possible implementation, the brain functional areas are divided into r regions as follows:

[0133]

[0134] …

[0135]

[0136] Among them, v r That is, it represents the rth brain functional area, n k represents the number of nodes in the kth brain functional area (such as the number of electrodes on the brain functional area), k∈[1,r], n1+n2+…+n r =n.

[0137] In step 204b, the local EEG signals of the brain functional areas are input into the graph attention network for feature extraction to obtain a node feature set corresponding to the brain functional areas, wherein the node feature set contains node features corresponding to the EEG signals of each node, and the node EEG signal is the EEG signal corresponding to a single node in the brain functional area.

[0138] In one possible implementation, the local EEG signals corresponding to each brain functional area can be represented as a block matrix and input into the GAT network. The block matrix is:

[0139] X local =[v1,v2,...,v r ]

[0140] By inputting the block matrix into the graph attention network and performing feature extraction on each column in the block matrix, we can extract features from the node EEG signals of different nodes within each brain functional area. Taking electrode acquisition as an example, a node EEG signal refers to the EEG signal collected by a single electrode.

[0141] After the graph attention network extracts local features, we can get:

[0142]

[0143] …

[0144]

[0145] Among them, G local () represents the output obtained after input into the GAT network, It is the node feature corresponding to the node EEG signal of the jth node in the i-th brain functional area after passing through the GAT network (taking electrode collection as an example, it is the node EEG signal collected by the j-th electrode), and f1 represents the output feature.

[0146] That is, after the block matrix corresponding to any brain functional area is input into the GAT network, the node features of the node brain signals corresponding to each node in the brain functional area can be obtained to form the node feature set corresponding to the brain functional area.

[0147] The local characteristics of all brain functional areas can be expressed as:

[0148]

[0149] Among them, G′ local Contains characteristics corresponding to each functional area of the brain.

[0150] Step 204c: Aggregate the node features in the node feature set to obtain local features corresponding to the brain functional areas.

[0151] In the above process, after the local EEG signals of the brain functional area are input into the GAT network, the node feature set of the brain functional area is obtained. In this embodiment, for a brain functional area, the node features in the corresponding node feature set can be aggregated to obtain the local features corresponding to the brain functional area.

[0152] Optionally, the features of each node are aggregated using an aggregation function to obtain local features corresponding to the brain functional area. The aggregation function is one of an average function, a maximum function, or a minimum function. Schematically, the average function is used as the aggregation function to aggregate the node features corresponding to each brain functional area to obtain the local features of each brain functional area and form a local feature set. This is shown below:

[0153]

[0154]

[0155] Among them, AG represents the average function, p is the node index in each brain functional area, and the value of p depends on the number of nodes in the brain functional area (such as the number of electrodes), n r is the number of nodes in the rth brain functional area, That is the local feature corresponding to the rth brain functional area.

[0156] Step 205: Fusing the local features contained in the local feature set to obtain the spatial features of the EEG signal.

[0157] Optionally, each local feature is input into the graph attention network for feature extraction to obtain the spatial features of the EEG signal, that is, to obtain the spatial features corresponding to the global EEG signal.

[0158] After obtaining the local features corresponding to each brain functional area, in this embodiment, global spatial features are also learned to learn the complex relationship between brain functional areas. In one possible implementation, the global structure of EEG signals in the brain area is captured using the GAT network, which can be expressed as:

[0159]

[0160] in, is the local feature obtained after the local signal in the i-th brain functional area is extracted by the GAT network feature, i∈[1,r], for example, f2 represents the output feature, G global (G′ AG ) is the spatial feature of the EEG signal output by the graph attention network.

[0161] Step 206: Fusing the temporal features with the spatial features to obtain the temporal and spatial features of the EEG signal.

[0162] In this embodiment, when emotion recognition is performed based on temporal features and spatial features, the temporal features and spatial features are first fused to obtain spatiotemporal features. In one possible implementation, the EEG emotion recognition network further includes a spatiotemporal feature extraction network for the fusion process of temporal features and spatial features. Optionally, the spatiotemporal feature extraction network may also be a BiLSTM network, that is, a BiLSTM network may be used to fuse features to obtain a deep spatiotemporal representation. The process may include steps 206a-206c:

[0163] In step 206a, the temporal features and the spatial features are input into a spatiotemporal feature extraction network for feature extraction to obtain local spatiotemporal features corresponding to each brain functional area.

[0164] In this embodiment, the spatiotemporal features of the EEG signal are obtained by fusing the temporal features with the spatial features through a spatiotemporal feature extraction network. In one possible implementation, the spatiotemporal feature extraction network may be a BiLSTM network.

[0165] First, the temporal and spatial features are input into BiLSTM for feature extraction. During the feature extraction process, the forward and reverse outputs of BiLSTM are obtained. The forward and reverse outputs are then concatenated to obtain the local spatiotemporal features corresponding to each brain functional area. This is shown in the following formula:

[0166] Z st =[H′ t ,G global (G′ AG )]

[0167]

[0168] in, As the input of the BiLSTM network, by concatenating the time feature H′ t With spatial feature G global (G′ AG ) is obtained. After feature extraction by BiLSTM network, is the forward output of BiLSTM, It is the reverse output of BiLSTM, and the local spatiotemporal features are obtained by splicing

[0169] Step 206b: Determine the attention coefficient of the brain functional area corresponding to the local spatiotemporal feature based on the correlation between the local spatiotemporal feature and the spatiotemporal feature sequence, where the spatiotemporal feature sequence includes the local spatiotemporal features corresponding to each brain functional area.

[0170] By extracting the local spatiotemporal features corresponding to each brain functional area in the EEG functional area, we can obtain them, forming a spatiotemporal feature sequence. In this embodiment, the attention module assigns weights to the local spatiotemporal features corresponding to different brain functional areas. The weights are related to the contribution of each brain functional area to emotion.

[0171] For each local spatiotemporal feature, the corresponding attention coefficient can be calculated by using its correlation with the spatiotemporal feature sequence, which can be used for weighted fusion.

[0172] Among them, the method of calculating the attention coefficient can refer to the calculation of the attention coefficient corresponding to each EEG signal segment in the above embodiment, which will not be repeated here.

[0173] Step 206c: Based on the attention coefficient corresponding to each brain functional area, each local spatiotemporal feature is fused to obtain the spatiotemporal features of the EEG signal.

[0174] The attention coefficients corresponding to each brain functional area are used as weights to perform weighted fusion on each local spatiotemporal feature to obtain the spatiotemporal features of the EEG signal. This is shown below:

[0175] Z = att(Z′ st ,q st )

[0176] Among them, q st It is the representation of the spatial-temporal feature sequence after linear transformation. That is, the spatiotemporal features obtained by fusion, r is the number of regions, and k″ is the number of features of the output features of BiLSTM.

[0177] Step 207: Input the spatiotemporal features into a classifier for emotion recognition to obtain the probability that the EEG signal belongs to different emotion categories.

[0178] Optionally, a softmax classifier is used for emotion recognition. The spatiotemporal feature Z can be input into a linear layer and then into a softmax function for emotion recognition, that is:

[0179] Z output =Φ softmax (WF dropout (ψ flatten (Z))+b)

[0180] Among them, Φ softmax Represents the softmax function, W, b are weights and biases. Z output Contains the probabilities of spatiotemporal features belonging to different emotion categories. For example, it contains the probabilities of belonging to the corresponding categories of happiness, sadness, fear, and terror.

[0181] Step 208: Determine the emotion category corresponding to the maximum probability value as the emotion category to which the EEG signal belongs.

[0182] After obtaining the probabilities of belonging to each emotion category, the emotion category corresponding to the maximum probability value can be determined as the emotion category to which the EEG signal belongs.

[0183] Alternatively, in another possible implementation, the emotion category corresponding to the EEG signal may include two different emotions, and the emotion category corresponding to the first m probability values may be determined as the emotion category of the EEG signal. Schematically, m may be 2.

[0184] like Figure 5As shown, the EEG signal 501 at different times is segmented to obtain different EEG signal segments, and the features 502 of the EEG signal segments are input into the time feature extraction network (including BiLSTM network and attention module) 503. The BiLSTM network first extracts the time features to obtain the time period features of each EEG signal segment. After that, the attention module (Attention) fuses the features of each time period to obtain the time features of the EEG signal. The local EEG signals 504 corresponding to different brain functional areas are input into the spatial feature extraction network (GAT network) 505. First, each local EEG signal is input into the GAT network respectively to perform spatial feature extraction on each local EEG signal to obtain the local features 506 corresponding to each brain functional area. The local features 506 are then input into the GAT network for global feature extraction to obtain spatial features. Afterwards, the temporal features and spatial features are input into the spatiotemporal feature extraction network (including the BiLSTM network and the Attention layer) 507 for fusion to obtain the spatiotemporal features of the EEG signal. The spatiotemporal features are input into the classifier 508 for emotion recognition to obtain the emotion category of the EEG signal.

[0185] In this embodiment, the BiLSTM network is used to extract the time features of signal segments in different time periods to capture the emotional changes at different moments, and the graph attention network is used to capture the spatial features of different brain functional areas to capture the emotional changes in different brain functional areas. Emotion recognition is performed based on the emotional changes at different moments and the emotional changes in different brain functional areas, which can improve the accuracy of emotion recognition.

[0186] In the above embodiment, the method for emotion recognition based on the EEG emotion recognition network is illustrated. The following will describe the training process of the EEG emotion recognition network.

[0187] Among them, the EEG emotion recognition network is trained based on sample EEG signals, and the sample EEG signals are marked with real emotion labels. The sample EEG signals are signals within the second time, and the sample EEG signals are composed of sample local EEG signals corresponding to different brain functional areas. In one possible embodiment, videos or music with specific labels can be played to the subjects. During this process, the EEG signals of the subjects are collected, and during the collection process, the length and number of each trial need to be determined, and there is an interval between each trial so that the subjects can rest. Schematically, the collection time of each trial is from several seconds to tens of seconds, and the specific time can be determined according to the emotion recognition task. For example, when the playback segment is long, the collection time of the trial is also longer. Finally, the collected EEG signal is used as a sample EEG signal, the sampling time is the second time, and the label corresponding to the playback segment is used as the real emotion label for the training process of the EEG emotion recognition network. In another possible embodiment, EEG signals corresponding to different emotion labels can be simulated and generated by computer equipment and used as sample EEG signals, which is not limited in this embodiment. Figure 6 As shown in the figure, the training process of the EEG emotion recognition network includes the following steps:

[0188] Step 601: construct an initial EEG emotion recognition network, which includes an initial time feature extraction network and an initial spatial feature extraction network.

[0189] An initial temporal feature extraction network is constructed based on the BiLSTM network structure, and an initial spatial feature extraction network is constructed based on the GAT network. During the construction process, the initial network parameters, including weights and biases, can be configured. The feature dimensions during training also need to be configured, which can be set based on the actual recognition task. For example, in this application, k′, f1, f2, and k″ can be fixed to 256, 150, 256, and 64, respectively.

[0190] Step 602 : Using the initial time feature extraction network, time features are extracted from the sample signal segments at different moments in the second time, to obtain sample time features of the sample EEG signals.

[0191] In one possible implementation, the sample EEG signal within the second time period can be segmented to obtain different sample signal segments, each representing a signal segment at a different moment. Subsequently, the different sample signal segments can be input into a BiLSTM network for feature extraction to obtain sample time segment features for each sample signal segment. The features of each sample time segment are then fused to obtain the sample time features of the sample EEG signal.

[0192] Among them, the process of using the initial time feature extraction network to extract the time features of the sample signal segments in the sample EEG signal to obtain the sample time features of the sample EEG signal can refer to the process of using the time feature extraction network to extract the time features of the signal segments in the EEG signal to obtain the time features of the EEG signal in the above embodiment, which will not be repeated in this embodiment.

[0193] Step 603: Use the initial spatial feature extraction network to extract spatial features of the sample local EEG signal to obtain sample spatial features of the sample EEG signal.

[0194] In one possible implementation, the sample EEG signal can be partitioned to obtain sample local signals corresponding to different EEG functional areas. Afterwards, the different sample local signals can be input into the GAT network for feature extraction to obtain sample local features of different sample local signals. The sample local features can then be fused to obtain sample spatial features of the sample EEG signal.

[0195] The process of using the initial spatial feature extraction network to extract spatial features of local signals of different brain functional areas in the sample EEG signals to obtain the sample spatial features can refer to the process of using the spatial feature extraction network to extract spatial features of local EEG signals to obtain the spatial features of EEG signals in the above embodiment, which will not be repeated in this embodiment.

[0196] Step 604 : Emotion recognition is performed based on the sample time feature and the sample space feature to obtain the predicted emotion category to which the EEG signal belongs.

[0197] In one possible implementation, the sample temporal features and the sample spatial features can be input into a spatiotemporal feature extraction network for fusion to obtain the sample spatiotemporal features. The sample spatiotemporal features are then input into a classifier for classification to obtain the predicted emotion category of the EEG signal. This process can be referred to the process of performing emotion recognition based on spatiotemporal and spatial features to obtain the EEG signal emotion category in the above embodiment, and will not be repeated in this embodiment.

[0198] Step 605 , determining the classification loss based on the predicted emotion category and the actual emotion category, and stopping the training of the initial EEG emotion recognition network when the classification loss indicates that the training state reaches the training end condition, thereby obtaining the EEG emotion recognition network.

[0199] In this embodiment, the sample EEG signal has a true emotion label, which is the true emotion category. In one possible implementation, a classification loss can be determined based on the difference between the predicted emotion category and the true emotion category, thereby determining the training status of the network based on the classification loss.

[0200] Optionally, the cross-entropy loss function can be used to measure the gap between the predicted emotion category and the true emotion category. The method is as follows:

[0201]

[0202] Among them, L cls represents the classification loss, Θ is all the parameters in the EEG emotion recognition network, and y i Indicates the true emotion category, Indicates the predicted emotion category, and K indicates the total number of samples selected during one round of training.

[0203] During the training process, the loss function is determined to have reached a convergence state based on the classification loss. If it has reached a convergence state, it is determined that the training state meets the training end condition, and the training process of the initial EEG emotion recognition network is stopped to obtain the EEG emotion recognition network for recognizing EEG signals. If the classification loss has not converged, the next round of training is required. At this time, the network parameters of the EEG emotion recognition network (including the network parameters of the time feature extraction network, the spatial feature extraction network, and the spatiotemporal feature extraction network) need to be updated, and then the updated EEG emotion recognition network is used to re-extract the time features and spatial features of the sample EEG signal. The classification loss is then recalculated based on the updated time features and spatial features to perform the next round of judgment. In one possible embodiment, the network parameters are updated using a backpropagation algorithm to gradually reduce the classification loss. In one possible embodiment, when the classification loss fluctuation value is less than a preset value, it is determined that the classification loss has converged and the network training process is terminated. Among them, the fluctuation value of the classification loss can be the difference between the classification losses after two adjacent training sessions. The size of the preset value can generally be selected and determined based on the actual application scenario, etc., so as to meet the actual application requirements. Of course, the existing technical methods can also be used to judge the convergence state of the classification loss, specifically based on whether the convergence state of the classification loss can be satisfied to judge the training state of the EEG emotion recognition network.

[0204] And in a possible implementation, two stages of training are performed respectively, the first stage is the pre-training stage, and the second stage is the fine-tuning stage. During the training process, the data set is divided into a training set, a validation set, and a test set. In the first stage, an early stopping strategy is adopted to train the model with the training set, and the accuracy of the validation set is monitored. When the accuracy of the validation set no longer changes, it is determined that the early stopping condition is met and the second stage of training is entered. In the second stage, the model parameters saved in the first stage are loaded, and training is continued on the validation set and the training set. When the classification loss value corresponding to the validation set is less than the classification loss value corresponding to the training set in the first stage, the training can be stopped and the network parameters of the EEG emotion recognition network obtained by training are saved. Schematically, the epoch of the first stage is set to 200, and the second stage is set to 20. The learning rate is set to 1e-3 in the first stage and 1e-4 in the second stage. The dropout is set to 0.5, and Adam optimization is used for optimization during the training process. Through the two-stage training method, the network overfitting can be prevented, thereby better optimizing the network performance and improving the training effect of the EEG emotion recognition network.

[0205] The above description is only a preferred embodiment of the present application, and the present application is not limited to the above embodiments. It is understood that other improvements and variations directly derived or imagined by those skilled in the art without departing from the spirit and concept of the present application should be considered to be included in the scope of protection of the present application.

Claims

1. A method for EEG emotion recognition based on an EEG emotion recognition network, characterized in that: The EEG emotion recognition network includes a temporal feature extraction network and a spatial feature extraction network. The method includes: Acquiring an EEG signal within a first period of time, wherein the EEG signal at least includes local EEG signals corresponding to different brain functional areas; Using the time feature extraction network to extract time features from EEG signal segments at different times within the first time, to obtain time features of the EEG signal, wherein the time features include fluctuation states of the EEG signal within different time periods; Performing spatial feature extraction on the local EEG signal using the spatial feature extraction network to obtain spatial features of the EEG signal, wherein the spatial features include fluctuation states of local EEG signals in different brain functional areas and global EEG signals; Performing emotion recognition based on the temporal feature and the spatial feature to obtain the emotion category to which the EEG signal belongs; The EEG emotion recognition network also includes a spatiotemporal feature extraction network; The temporal feature and the spatial feature are fused to obtain the spatiotemporal feature of the EEG signal, including: Inputting the temporal features and the spatial features into the spatiotemporal feature extraction network to perform feature extraction, thereby obtaining local spatiotemporal features corresponding to each brain functional area; Determining the attention coefficient of the brain functional area corresponding to the local spatiotemporal feature based on the correlation between the local spatiotemporal feature and the spatiotemporal feature sequence, wherein the spatiotemporal feature sequence includes the local spatiotemporal features corresponding to each of the brain functional areas; Based on the attention coefficient corresponding to each of the brain functional areas, each of the local spatiotemporal features is fused to obtain the spatiotemporal features of the EEG signal.

2. The method according to claim 1, characterized in that The obtaining of the temporal characteristics of the EEG signal includes: Using the time feature extraction network to extract time features from each of the EEG signal segments to obtain a time segment feature set, wherein the time segment feature set includes time segment features of each of the EEG signal segments at different moments; fusing the time period features contained in the time period feature set to obtain the time features of the EEG signal; The obtaining of the spatial features of the EEG signal includes: Performing spatial feature extraction on each of the local EEG signals using the spatial feature extraction network to obtain a local feature set, wherein the local feature set includes local features of the local EEG signals corresponding to each brain functional area; The local features included in the local feature set are fused to obtain the spatial features of the EEG signal.

3. The method according to claim 2, characterized in that The temporal feature extraction network includes a bidirectional short-term memory network BiLSTM; The extracting time features of each of the EEG signal segments using the time feature extraction network includes: Dividing the EEG signal within the first time into segments to obtain EEG signal segments at different moments; Extracting initial features of each of the EEG signal segments, and inputting each of the initial features into the BiLSTM network for feature extraction to obtain bidirectional output features corresponding to each of the EEG signal segments; Based on the bidirectional output feature, a time segment feature of the EEG signal segment is determined.

4. The method according to claim 2, characterized in that The fusing of the time period features included in the time period feature set to obtain the time features of the EEG signal includes: Determining an attention coefficient of an EEG signal segment corresponding to the time period feature based on a correlation between the time period feature and a time feature sequence, wherein the time feature sequence includes the time period feature corresponding to each EEG signal segment; Based on the attention coefficient of each of the EEG signal segments, the features of each of the time periods are fused to obtain the time features of the EEG signal.

5. The method according to claim 2, characterized in that The spatial feature extraction network includes a graph attention network; The utilizing the spatial feature extraction network to extract spatial features from each of the local EEG signals comprises: Dividing the brain region into different brain functional areas, each of which contains a number of nodes for acquiring EEG signals; Inputting the local EEG signal of the brain functional area into the graph attention network for feature extraction to obtain a node feature set corresponding to the brain functional area, wherein the node feature set includes node features corresponding to the EEG signal of each node, and the node EEG signal is the EEG signal corresponding to a single node in the brain functional area; The node features in the node feature set are aggregated to obtain the local features corresponding to the brain functional areas.

6. The method according to claim 5, characterized in that The aggregating the node features in the node feature set to obtain the local features corresponding to the brain functional areas includes: The feature aggregation of each node feature is performed through an aggregation function to obtain the local feature corresponding to the brain functional area, and the aggregation function is one of an average value function, a maximum value function or a minimum value function.

7. The method according to claim 5, characterized in that The fusing of the local features included in the local feature set to obtain the spatial features of the EEG signal includes: Each of the local features is input into the graph attention network for feature extraction to obtain the spatial features of the EEG signal.

8. The method according to any one of claims 1 to 7, characterized in that: The performing emotion recognition based on the temporal feature and the spatial feature to obtain the emotion category to which the EEG signal belongs includes: Fusing the temporal feature with the spatial feature to obtain the spatiotemporal feature of the EEG signal; Inputting the spatiotemporal features into a classifier for emotion recognition to obtain the probability that the EEG signal belongs to different emotion categories; The emotion category corresponding to the maximum probability value is determined as the emotion category to which the EEG signal belongs.

9. The method according to any one of claims 1 to 7, characterized in that: The EEG emotion recognition network is trained based on sample EEG signals, the sample EEG signals are annotated with real emotion labels, the sample EEG signals are signals within the second time period, and the sample EEG signals are composed of sample local EEG signals corresponding to different brain functional areas. The method further includes: Constructing an initial EEG emotion recognition network, wherein the initial EEG emotion recognition network includes an initial time feature extraction network and an initial spatial feature extraction network; Using the initial time feature extraction network, extracting time features from sample signal segments at different moments within the second time to obtain sample time features of the sample EEG signals; Performing spatial feature extraction on the sample local EEG signal using the initial spatial feature extraction network to obtain sample spatial features of the sample EEG signal; Performing emotion recognition based on the sample time feature and the sample space feature to obtain a predicted emotion category to which the EEG signal belongs; A classification loss is determined based on the predicted emotion category and the true emotion label, and when the classification loss indicates that the training state reaches a training end condition, the training of the initial EEG emotion recognition network is stopped to obtain the EEG emotion recognition network.

Images