A method and device for EEG signal emotion recognition based on adaptive fusion network

Abstract

The present invention relates to the field of emotion recognition technology, and more particularly to a method and device for EEG signal emotion recognition based on an adaptive fusion network. The method comprises: preprocessing the EEG signal, inputting EEG signals of different frequency bands into different feature extraction submodules, obtaining output features of each feature extraction submodule via a first long short-term memory converter and a first adaptive graph convolutional network; inputting the output features of all feature extraction submodules into a feature fusion module, obtaining fused features via a second long short-term memory converter and a second adaptive graph convolutional network; and identifying emotion categories based on the fused features. Based on the long short-term memory converter and the adaptive graph convolutional network, the present invention can consider individual differences in EEG signals and adaptively obtain frequency band relationship information, thereby improving the accuracy and versatility of EEG emotion recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of emotion recognition, and in particular to a method and device for recognizing emotions using electroencephalogram (EEG) signals based on an adaptive fusion network. Background Art

[0002] The task of emotion recognition is to use machines to capture the emotional state of humans. This is a research hotspot in the fields of human-computer interaction and artificial intelligence. Emotion recognition refers to the identification of an individual's emotional state based on physiological signals (such as heart rate, electrooculogram, or electroencephalogram) or non-physiological signals (such as facial expressions, body movements, or sounds). It has broad application prospects, such as in driving safety, personalized recommendations, and the treatment of emotional disorders. Since physiological signals are not easily controlled by humans, they can show the true state of emotions, so emotion recognition through physiological signals is more reliable. Electroencephalogram (EEG), as a physiological signal triggered by brain neuron activity, is often used for emotion recognition. EEG-based emotion recognition has a high accuracy rate, and it is relatively simple and convenient to collect EEG signals using non-invasive, wearable devices.

[0003] In existing research, Zhong et al. proposed a regularized graph neural network (RGNN) based on EEG signals in the paper "EEG-based emotion recognition using regularized graph neural networks". This network constructs channel relationships by combining spatial relationships with important global connections. However, it does not pay attention to the temporal information within the electrode channels. The constructed channel relationships cannot adapt to changes in the subjects, and the relationship between frequency bands is not represented. Summary of the Invention

[0004] To this end, the technical problem to be solved by the present invention is to overcome the problem that the EEG signal emotion recognition model in the prior art does not pay attention to the timing information within the electrode channel and cannot adapt to changes in the subject.

[0005] To solve the above technical problems, the present invention provides an EEG signal emotion recognition method based on an adaptive fusion network, comprising:

[0006] Constructing an EEG signal emotion recognition model, including a data preprocessing module, a feature extraction module, a feature fusion module, and a classification module; the feature extraction module includes multiple feature extraction submodules corresponding to the number of EEG signal frequency bands, each feature extraction submodule includes a first long short-term memory converter and a first adaptive graph convolutional neural network with the same structure; the feature fusion module includes a second long short-term memory converter and a second adaptive graph convolutional neural network;

[0007] Input the EEG signal into the data preprocessing module to obtain EEG signals in multiple frequency bands;

[0008] The EEG signals of each frequency band are input into the first long short-term memory converter of the corresponding feature extraction submodule. The output features thereof are processed by residual and layer normalization and used as the input features of the first adaptive graph convolutional neural network of the feature extraction submodule. The first adaptive graph convolutional network of the feature extraction submodule generates an adaptive graph based on the input features, and obtains the output features of the feature extraction submodule according to the adaptive graph.

[0009] The output features of all feature extraction submodules are input into the feature fusion module, and then pass through the second long short-term memory converter and the second adaptive graph convolutional neural network in sequence to output the fused features;

[0010] The fused features are input into the classification module to obtain the emotion category.

[0011] Preferably, the step of inputting the EEG signal into a data preprocessing module to obtain EEG signals in multiple frequency bands includes:

[0012] The EEG signal X j ∈R M×N×5 Input to the data preprocessing module, where M is the number of electrodes, N is the sequence length of the EEG signal, and the EEG signal X j Includes 5 frequency bands: S j,1 ,S j,2 ,…S j,5 , where S j,n ∈R M×N Represents the EEG signal of the nth frequency band, n = 1, 2, ..., 5; the 5 frequency bands are δ, θ, α, β and γ bands respectively.

[0013] Preferably, the first long short-term memory converter and the second long short-term memory converter are both variants of the Transformer model, and long short-term memory is used to replace the linear mapping in the encoder of the Transformer model to obtain the first long short-term memory converter and the second long short-term memory converter.

[0014] Preferably, the EEG signals of each frequency band are input to the first long short-term memory converter of the corresponding feature extraction submodule, and the output features thereof are processed by residual and layer normalization and used as the input features of the first adaptive graph convolutional neural network of the feature extraction submodule. The formula is:

[0015]

[0016] Among them, S j,n is the jth EEG signal X j The EEG signal of the nth frequency band, is the output feature of the first long short-term memory converter of the nth feature extraction submodule, LN(·) is the layer normalization operation, Input features of the first adaptive graph convolutional neural network of the nth feature extraction submodule.

[0017] Preferably, the first adaptive graph convolutional network of the feature extraction submodule generates an adaptive graph based on the input features, and obtains the output features of the feature extraction submodule according to the adaptive graph, and the formula is:

[0018]

[0019] in, is the EEG signal S j,n Adaptive graph, Softmax(·) is the Softmax activation function, D is The degree matrix of , I is the identity matrix, is the input feature of the first adaptive graph convolutional neural network of the nth feature extraction submodule, W n is the parameter matrix of the first adaptive graph convolutional network, S' j,n Output features of the nth feature extraction submodule.

[0020] Preferably, the EEG signal S j,n Adaptive graph The formula is:

[0021]

[0022] in, represents the Hadamard product, is the time domain adaptive adjacency matrix, Z n is a randomly initialized learnable matrix for time domain information, is the position-based spatial adjacency matrix, is a randomly initialized learnable matrix representing spatial information;

[0023] Time-domain adaptive adjacency matrix The element in row k and column q of is defined as:

[0024]

[0025] Among them, ||·|| is the 2-norm, and are the input features of the first adaptive graph convolutional neural network. The kth and qth rows of ;

[0026] Position-based spatial adjacency matrix The elements of are defined as:

[0027]

[0028] Among them, k1 and q1 represent the k1th channel and q1th channel respectively. is a hyperparameter, is the physical distance between the k1th channel and the q1th channel, M is the EEG signal S j,n The number of electrodes.

[0029] Preferably, after the first adaptive graph convolutional network of the feature extraction submodule generates an adaptive graph based on the input features and obtains the output features of the feature extraction submodule according to the adaptive graph, it also includes: using a top-k graph pooling operation to retain the weights of important electrode channel sampling points in the output features of the feature extraction submodule, and assigning the remaining weights to 0.

[0030] Preferably, the output features of all feature extraction submodules are inputted into the feature fusion module, sequentially passed through the second long short-term memory converter and the second adaptive graph convolutional neural network, and the fusion features are outputted, including:

[0031] The output features of all feature extraction submodules are X' j =[S' j,1 ,S' j,2 ,…S' j,5 ], where S' j,n is the output feature of the nth feature extraction submodule, n = 1, 2, ..., 5;

[0032] The output features X' of all feature extraction submodules j Input feature fusion module, use flatten operation to transform X' j Reshape into a two-dimensional matrix F j , as the input feature of the second long short-term memory converter;

[0033] The output characteristics of the second long short-term memory converter are After residual and layer normalization processing, the input features of the second adaptive graph convolutional neural network are obtained, and the formula is:

[0034]

[0035] Among them, LN(·) is the layer normalization operation, is the input feature of the second adaptive graph convolutional neural network;

[0036] The second adaptive graph convolutional network is based on the input features Generate adaptive graph The formula is:

[0037]

[0038] in, represents the Hadamard product, is the adaptive adjacency matrix, Z' is the learning matrix;

[0039] The adaptive adjacency matrix The elements of are defined as:

[0040]

[0041] Among them, ||·|| is the 2-norm, and are the input features of the second adaptive graph convolutional neural network respectively. The k2th and q2th rows of ;

[0042] Based on adaptive graph Get the output feature F' of the feature extraction submodule j , the formula is:

[0043]

[0044] in, Softmax(·) is the Softmax activation function, for degree matrix, I is the identity matrix, W is the parameter matrix, F' j It is the fusion feature output by the feature fusion module.

[0045] Preferably, the fusion features are input into a classification module to obtain the emotion category, and the formula is:

[0046] c X =argmax(Softmax(Flatten(F′ j )))

[0047] Among them, c Xis the emotion category, Softmax(·) is the Softmax activation function, Flatten(·) is the flatten operation, F' j The fusion feature of .

[0048] The present invention also provides an EEG signal emotion recognition device based on an adaptive fusion network, comprising:

[0049] A model construction unit, for constructing an EEG signal emotion recognition model, comprising a data preprocessing module, a feature extraction module, a feature fusion module, and a classification module; the feature extraction module comprises a plurality of feature extraction submodules corresponding to the number of EEG signal frequency bands, each feature extraction submodule comprising a first long short-term memory converter and a first adaptive graph convolutional neural network with the same structure; the feature fusion module comprises a second long short-term memory converter and a second adaptive graph convolutional neural network;

[0050] A preprocessing unit, used to input the EEG signal into the data preprocessing module to obtain EEG signals in multiple frequency bands;

[0051] A feature extraction unit is configured to input the EEG signals of each frequency band into the first long short-term memory converter of the corresponding feature extraction submodule, and the output features thereof are processed by residual and layer normalization, and then used as the input features of the first adaptive graph convolutional neural network of the feature extraction submodule; the first adaptive graph convolutional network of the feature extraction submodule generates an adaptive graph based on the input features, and obtains the output features of the feature extraction submodule based on the adaptive graph;

[0052] A feature fusion unit is used to input the output features of all feature extraction submodules into the feature fusion module, pass them through the second long short-term memory converter and the second adaptive graph convolutional neural network in sequence, and output fused features;

[0053] The classification unit is used to input the fusion features into the classification module to obtain the emotion category.

[0054] The above technical solution of the present invention has the following beneficial effects compared with the prior art:

[0055] The present invention discloses an EEG emotion recognition method based on an adaptive fusion network, which uses a long short-term memory converter and an adaptive graph convolutional neural network for feature extraction and feature fusion. The long short-term memory converter can focus on important information within the EEG signal channel, thereby extracting the long-term dependencies of the EEG signal sequence; the adaptive graph convolutional neural network calculates the similarity of signals between channels, and the adaptive graph constructed by the adaptive graph changes adaptively with the changes in the input signals of different subjects, flexibly reflecting the relationship between channels. In the feature extraction process, each feature extraction submodule extracts the features of EEG signals in different frequency bands to reflect the time-varying electrode channel relationship of individual differences. The features extracted by each feature extraction submodule are then fused through a feature fusion module to extract the relationship between frequency bands. Therefore, the present invention can take into account individual differences in EEG signals and adaptively obtain the relationship information of frequency bands, thereby improving the accuracy and versatility of EEG emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:

[0057] Figure 1 This is a flow chart of a method for EEG signal emotion recognition based on an adaptive fusion network according to the present invention;

[0058] Figure 2 is a structural diagram of an EEG signal emotion recognition model according to an embodiment of the present invention;

[0059] Figure 3 The embodiment of the present invention is a schematic diagram of selected electrode channels. DETAILED DESCRIPTION

[0060] The present invention will be further described below with reference to the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention.

[0061] Reference Figure 1 As shown, the present invention provides an EEG signal emotion recognition method based on an adaptive fusion network, which performs emotion recognition on the input EEG signal by constructing an EEG signal emotion recognition model. The structure of the EEG emotion recognition model is referenced Figure 2 As shown in FIG, it includes a data preprocessing module, a feature extraction module, a feature fusion module and a classification module.

[0062] This embodiment conducts experiments on the target counting dataset SEED dataset, which contains 3 emotion categories (positive, neutral, and negative) and 15 subjects. The SEED dataset selects 15 movie clips to form emotional stimuli. In the process of collecting the dataset, each subject watched all 15 movie clips once a week for three weeks. Their corresponding EEG signals were recorded. Since the duration of EEG signals is inconsistent, the shorter signals are zero-padded and aligned. Finally, the EEG signal sampling tensor of the subject is 62 channels × 5 frequency bands × 265 sampling points. Each subject has a total of 45 EEG signals. In addition, the SEED dataset uses a 62-channel power imaging neuroscan system device to collect EEG signals with a sampling rate of 1000Hz. In the dataset, all collected EEG signals were downsampled to 200 Hz and then filtered through a 1-50 Hz bandpass filter, dividing the EEG signals into five frequency bands: delta (1-3 Hz), theta (4-7 Hz), alpha (8-13 Hz), beta (14-30 Hz), and gamma (31-50 Hz). The EEG signals in these five frequency bands were further processed using differential entropy and linear dynamic systems, and normalized using z-score normalization.

[0063] The collected EEG signal X j ∈R M×N×5 The data preprocessing module is input to the EEG emotion recognition model, where M is the number of electrodes and N is the sequence length of the EEG signal. j Includes 5 frequency bands: S j,1 ,S j,2 ,…S j,5 , where S j,n ∈R M ×N , n=1,2,…,5, represents the EEG signal of the nth frequency band. The five frequency bands are δ band, θ band, α band, β band and γ band.

[0064] The feature extraction module includes multiple feature extraction submodules corresponding to the number of EEG signal frequency bands. In this embodiment, there are five feature extraction submodules. The EEG signals of the five frequency bands are input into the corresponding feature extraction submodules. Each feature extraction submodule includes a first long short-term memory converter and a first adaptive graph convolutional neural network with the same structure.

[0065] The Long Short Term Memory Transformer (LSTMTrans) is a variant of the Transformer. Considering that the linear mapping in the Transformer is prone to overfitting problems and cannot consider the contextual information of the data, while the LSTM can extract the long-term dependencies of the sequence, the present invention replaces the linear mapping in the Transformer encoder with LSTM to obtain a long short-term memory transformer. In this embodiment, the number of sub-encoders in the long short-term memory transformer of the feature extraction module is set to 2, and the number of heads in the attention mechanism is set to 6. The Modified Adaptive Graph Convolution Network (MAGCN) is used to represent the graph structure of the channel relationship of the EEG signal in each frequency band to extract information in the time domain and the spatial domain.

[0066] Since the structure of each feature extraction submodule is exactly the same, taking the nth frequency band in the jth EEG signal as an example, the EEG signal S j,n The input to the nth feature extraction submodule is used as an example for introduction.

[0067] The EEG signal S j,n Input to the first long short-term memory converter of the nth feature extraction submodule to obtain the output feature of the first long short-term memory converter of the nth feature extraction submodule Output feature Perform residual and layer normalization processing to obtain the input features of the adaptive graph convolutional neural network of the nth feature extraction submodule The formula is:

[0068]

[0069] Among them, S j,n is the jth EEG signal X j The EEG signal of the nth frequency band, is the output feature of the first long short-term memory converter of the nth feature extraction submodule, LN(·) is the layer normalization operation, Input features of the first adaptive graph convolutional neural network of the nth feature extraction submodule.

[0070] Input features Input to the first adaptive graph convolutional neural network of the nth feature extraction submodule, the first adaptive graph convolutional network is based on the input features Generate adaptive graph The formula is:

[0071]

[0072] in, represents the Hadamard product, is the time domain adaptive adjacency matrix, Z n ∈R M×N is a randomly initialized learnable matrix for time domain information, is the position-based spatial adjacency matrix, is a randomly initialized learnable matrix representing spatial information.

[0073] Among them, the time domain adaptive adjacency matrix The element in row k and column q of is defined as:

[0074]

[0075] Among them, ||·|| is the 2-norm, and are the input features of the first adaptive graph convolutional neural network. The kth and qth rows of .

[0076] Position-based spatial adjacency matrix The elements of are defined as:

[0077]

[0078] Where k1 and q1 represent the k1th channel and q1th channel respectively; is a hyperparameter, which is set to 2 in this embodiment; is the physical distance between the k1th channel and the q1th channel.

[0079] In this embodiment, the adjacency value between 11 pairs of electrode channels is set to 1. The selected 11 pairs of important electrode channels are as follows: Figure 3 shown.

[0080] The first adaptive graph convolutional network of the nth feature extraction submodule is based on the adaptive graph Get the output feature S' of the nth feature extraction submodule j,n , the formula is:

[0081]

[0082] in, is the EEG signal S j,n Adaptive graph, Softmax(·) is the Softmax activation function, D is The degree matrix of , I is the identity matrix, is the input feature of the first adaptive graph convolutional neural network of the nth feature extraction submodule, Wn is the parameter matrix of the first adaptive graph convolutional network, S' j,n Output features of the nth feature extraction submodule.

[0083] Finally, the top-k graph pooling operation is used to remove the output feature S' of the nth feature extraction submodule j,n To remove the redundant information in the graph, the top-k graph pooling operation retains the weights of the most important multiple electrode channel sampling points, and assigns the remaining weights to 0.

[0084] EEG SignalX j The output feature obtained after the feature extraction module is X' j =[S' j,1 ,S' j,2 ,…S' j,5 ]∈R M×N'×5 , where S' j,n ∈R M×N ' is the output feature of the nth feature extraction submodule, N' is the number of neurons in the first adaptive graph convolutional neural network, and in this embodiment, N' is set to 32.

[0085] The output feature of the feature extraction module is X' j The input is fed into a feature fusion module, which is also built on a long short-term memory (LSTM) transformer and an adaptive graph convolutional neural network, and includes a second LSTM transformer and a second adaptive graph convolutional neural network. In this embodiment, the number of sub-encoders in the second LSTM transformer is set to 2, and the number of heads in the attention mechanism is set to 6.

[0086] In the feature fusion module, the output features of the feature extraction module are first flattened to X' j Reshape into a two-dimensional matrix F j ∈R (M×N′)×5 , and then input to the second long short-term memory converter. The output characteristics of the second long short-term memory converter are Output features After residual and layer normalization processing, the input features of the second adaptive graph convolutional neural network are obtained The formula is:

[0087]

[0088] Among them, LN(·) is the layer normalization operation, is the input feature of the second adaptive graph convolutional neural network.

[0089] The second adaptive graph convolutional network is based on the input features Generate adaptive graph The formula is:

[0090]

[0091] in, represents the Hadamard product, is the adaptive adjacency matrix, Z'∈R (M×N′)×5 For learning matrix.

[0092] The adaptive adjacency matrix The elements of are defined as:

[0093]

[0094] Among them, ||·|| is the 2-norm, and are the input features of the second adaptive graph convolutional neural network respectively. The k2th and q2th rows of

[0095] Based on adaptive graph Get the fusion feature F' output by the feature fusion module j , the formula is:

[0096]

[0097] in, Softmax(·) is the Softmax activation function, for degree matrix, I is the identity matrix, W is the parameter matrix, F' j ∈R N″×5 is the fusion feature output by the feature fusion module, and N″ is the number of neurons in the second adaptive graph convolutional neural network. In this embodiment, N″ is set to 64.

[0098] The fusion feature F' output by the feature fusion module j Input the classification module to obtain the emotion category. The formula is:

[0099] c X =argmax(Softmax(Flatten(F′ j )))

[0100] Among them, c X is the emotion category, Softmax(·) is the Softmax activation function, Flatten(·) is the flatten operation, F' j It is the fusion feature output by the feature fusion module.

[0101] In this example, the EEG emotion recognition model was trained with a dropout rate of 0.1, an Adam optimizer, and a cross-entropy loss function. Early stopping was used, with patience, epoch number, and learning rate set to 20, 100, and 0.001, respectively.

[0102] To illustrate the effectiveness of the present invention, this example compares the experimental results with those of the prior art RGNN network model. The comparison results are shown in Table 1. The present invention and the RGNN network model were compared for emotion classification on the same dataset, using the leave-one-subject-out (LOSO) validation method, which can be used to verify the model's performance under cross-subject conditions. Using the average accuracy and corresponding standard deviation as metrics for network performance, the results in Table 1 show that the network model proposed in the present invention performs better than the RGNN model for emotion recognition in all five frequency bands and across all frequency bands.

[0103] Table 1. Comparison of target counting results between the present invention and RGNN on the SEED dataset

[0104] method Delta band Theta band α band Beta band Gamma band All frequency bands The present invention 84.29 / 4.67 90.21 / 3.22 89.04 / 2.96 85.92 / 4.49 83.11 / 5.22 93.93 / 2.76 RGNN 64.88 / 6.87 60.69 / 5.79 60.84 / 7.57 74.96 / 8.94 77.50 / 8.10 85.30 / 6.72

[0105] In summary, the present invention proposes a method for EEG emotion recognition based on an adaptive fusion network, which uses a long short-term memory converter and an adaptive graph convolutional neural network for feature extraction and feature fusion. The long short-term memory converter can focus on important information within the EEG signal channel, and then extract the long-term dependency of the EEG signal sequence; the adaptive graph convolutional neural network calculates the similarity of signals between channels, and the adaptive graph constructed will adaptively change with the change of input signals of different subjects, flexibly reflecting the relationship between channels. In the process of feature extraction, each feature extraction submodule extracts the features of EEG signals in different frequency bands to reflect the time-varying electrode channel relationship of individual differences. The features extracted by each feature extraction submodule are then fused through a feature fusion module to extract the relationship between frequency bands. Therefore, the present invention can take into account individual differences in EEG signals and adaptively obtain relationship information of frequency bands, thereby improving the accuracy and versatility of EEG emotion recognition.

[0106] Based on the method for EEG signal emotion recognition based on an adaptive fusion network, this embodiment further provides an EEG signal emotion recognition device based on an adaptive fusion network, comprising:

[0107] A model construction unit, for constructing an EEG signal emotion recognition model, comprising a data preprocessing module, a feature extraction module, a feature fusion module, and a classification module; the feature extraction module comprises a plurality of feature extraction submodules corresponding to the number of EEG signal frequency bands, each feature extraction submodule comprising a first long short-term memory converter and a first adaptive graph convolutional neural network with the same structure; the feature fusion module comprises a second long short-term memory converter and a second adaptive graph convolutional neural network;

[0108] A preprocessing unit, used to input the EEG signal into the data preprocessing module to obtain EEG signals in multiple frequency bands;

[0109] A feature extraction unit is configured to input the EEG signals of each frequency band into the first long short-term memory converter of the corresponding feature extraction submodule, and the output features thereof are processed by residual and layer normalization, and then used as the input features of the first adaptive graph convolutional neural network of the feature extraction submodule; the first adaptive graph convolutional network of the feature extraction submodule generates an adaptive graph based on the input features, and obtains the output features of the feature extraction submodule based on the adaptive graph;

[0110] A feature fusion unit is used to input the output features of all feature extraction submodules into the feature fusion module, pass them through the second long short-term memory converter and the second adaptive graph convolutional neural network in sequence, and output fused features;

[0111] The classification unit is used to input the fusion features into the classification module to obtain the emotion category.

[0112] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0113] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0114] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0115] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0116] Obviously, the above embodiments are merely examples for clarity of explanation and are not intended to limit the implementation methods. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all implementation methods here. Obvious variations or modifications arising therefrom remain within the scope of protection of the present invention.

Claims

1. A method for EEG emotion recognition based on an adaptive fusion network, characterized in that: include: Construct an EEG signal emotion recognition model, including a data preprocessing module, a feature extraction module, a feature fusion module, and a classification module; The feature extraction module includes a plurality of feature extraction submodules corresponding to the number of EEG signal frequency bands, each of which includes a first long short-term memory converter and a first adaptive graph convolutional neural network with the same structure; the feature fusion module includes a second long short-term memory converter and a second adaptive graph convolutional neural network; the first long short-term memory converter and the second long short-term memory converter are both variants of the Transformer model, and the linear mapping in the encoder of the Transformer model is replaced by long short-term memory to obtain the first long short-term memory converter and the second long short-term memory converter; Input the EEG signal into the data preprocessing module to obtain EEG signals in multiple frequency bands; The EEG signals of each frequency band are input into the first long short-term memory converter of the corresponding feature extraction submodule. After the output features are processed by residual and layer normalization, they are used as the input features of the first adaptive graph convolutional neural network of the feature extraction submodule. The first adaptive graph convolutional network of the feature extraction submodule generates an adaptive graph based on the input features, and obtains the output features of the feature extraction submodule according to the adaptive graph; The output features of all feature extraction submodules are input into the feature fusion module, and then pass through the second long short-term memory converter and the second adaptive graph convolutional neural network in sequence to output the fused features; The fused features are input into the classification module to obtain the emotion category.

2. The method for EEG emotion recognition based on an adaptive fusion network according to claim 1, characterized in that: The EEG signal is input into the data preprocessing module to obtain EEG signals of multiple frequency bands, including: EEG signals Input to the data preprocessing module, where is the number of electrodes, is the sequence length of the EEG signal, Includes 5 frequency bands: ,in Indicates the EEG signals in frequency bands, ; The 5 frequency bands are 、 、 、 and frequency band.

3. The method for EEG emotion recognition based on an adaptive fusion network according to claim 1, characterized in that: The EEG signals of each frequency band are input to the first long short-term memory converter of the feature extraction submodule. After the output features are processed by residual and layer normalization, they are used as the input features of the first adaptive graph convolutional neural network of the feature extraction submodule. The formula is: ； in, is the jth EEG signal No. EEG signals in frequency bands, is the output feature of the first long short-term memory converter of the nth feature extraction submodule, is the layer normalization operation, Input features of the first adaptive graph convolutional neural network of the nth feature extraction submodule.

4. The method for EEG emotion recognition based on an adaptive fusion network according to claim 1, characterized in that: The first adaptive graph convolutional network of the feature extraction submodule generates an adaptive graph based on the input features, and obtains the output features of the feature extraction submodule according to the adaptive graph. The formula is: ； in, , EEG The adaptive graph, is the Softmax activation function, for The degree matrix of is the identity matrix, is the input feature of the first adaptive graph convolutional neural network of the nth feature extraction submodule, is the parameter matrix of the first adaptive graph convolutional network, Output features of the nth feature extraction submodule.

5. The method for EEG emotion recognition based on an adaptive fusion network according to claim 4, characterized in that: EEG signals Adaptive graph The formula is: ； in, represents the Hadamard product, is the time-domain adaptive adjacency matrix, is a randomly initialized learnable matrix for time domain information, is the position-based spatial adjacency matrix, is a randomly initialized learnable matrix representing spatial information; Time-domain adaptive adjacency matrix The element in row k and column q of is defined as: ； in, is the 2-norm, and are the input features of the first adaptive graph convolutional neural network. The kth and qth rows of ; Position-based spatial adjacency matrix The elements of are defined as: ； in, and Respectively represent Channel and channels, is a hyperparameter, For the Channel and The physical distance between channels, EEG The number of electrodes.

6. The method for EEG emotion recognition based on an adaptive fusion network according to claim 1, characterized in that: The first adaptive graph convolutional network of the feature extraction submodule generates an adaptive graph based on the input features, and after obtaining the output features of the feature extraction submodule according to the adaptive graph, it also includes: using a top-k graph pooling operation to retain the weights of important electrode channel sampling points in the output features of the feature extraction submodule, and assigning the remaining weights to 0.

7. The method for EEG emotion recognition based on an adaptive fusion network according to claim 1, characterized in that: The output features of all feature extraction submodules are input to the feature fusion module, and sequentially pass through the second long short-term memory converter and the second adaptive graph convolutional neural network to output the fusion features, including: The output features of all feature extraction submodules are ,in is the output feature of the nth feature extraction submodule, ; The output features of all feature extraction submodules Input feature fusion module, use flatten operation to Reshape into a 2D matrix , as the input feature of the second long short-term memory converter; The output characteristics of the second long short-term memory converter are , After residual and layer normalization processing, the input features of the second adaptive graph convolutional neural network are obtained, and the formula is: ； in, is the layer normalization operation, is the input feature of the second adaptive graph convolutional neural network; The second adaptive graph convolutional network is based on the input features Generate adaptive graph , the formula is: ； in, represents the Hadamard product, is the adaptive adjacency matrix, For learning matrix; The adaptive adjacency matrix The elements of are defined as: ； in, is the 2-norm, and are the input features of the second adaptive graph convolutional neural network respectively. No. Row and OK; Based on adaptive graph Get the output features of the feature extraction submodule , the formula is: ； in, , is the Softmax activation function, for The degree matrix of is the identity matrix, is the parameter matrix, It is the fusion feature output by the feature fusion module.

8. The method for EEG emotion recognition based on an adaptive fusion network according to claim 1, characterized in that: The fusion features are input into the classification module to obtain the emotion category. The formula is: ； in, is the emotion category, is the Softmax activation function, For the flatten operation, The fusion feature of .

9. An EEG signal emotion recognition device based on an adaptive fusion network, characterized in that: include: A model building unit, used to build an EEG signal emotion recognition model, including a data preprocessing module, a feature extraction module, a feature fusion module, and a classification module; The feature extraction module includes a plurality of feature extraction submodules corresponding to the number of EEG signal frequency bands, each of which includes a first long short-term memory converter and a first adaptive graph convolutional neural network with the same structure; the feature fusion module includes a second long short-term memory converter and a second adaptive graph convolutional neural network; the first long short-term memory converter and the second long short-term memory converter are both variants of the Transformer model, and the linear mapping in the encoder of the Transformer model is replaced by long short-term memory to obtain the first long short-term memory converter and the second long short-term memory converter; A preprocessing unit, used to input the EEG signal into the data preprocessing module to obtain EEG signals in multiple frequency bands; The feature extraction unit is used to input the EEG signals of each frequency band into the first long short-term memory converter of the corresponding feature extraction submodule, and its output features are processed by residual and layer normalization as the input features of the first adaptive graph convolutional neural network of the feature extraction submodule; The first adaptive graph convolutional network of the feature extraction submodule generates an adaptive graph based on the input features, and obtains the output features of the feature extraction submodule according to the adaptive graph; A feature fusion unit is used to input the output features of all feature extraction submodules into the feature fusion module, pass them through the second long short-term memory converter and the second adaptive graph convolutional neural network in sequence, and output fused features; The classification unit is used to input the fusion features into the classification module to obtain the emotion category.

Images