EEG emotion recognition method based on multi-scale imaging and multi-modal feature fusion

Abstract

The present invention discloses a multi-modal feature fusion EEG emotion recognition method based on multi-scale imaging, which combines multi-scale and time series imaging algorithms, realizes emotion recognition by converting EEG signals into images. Compared with the traditional emotion recognition method based on EEG signals, it can not only preserve the spatial information of EEG signals, but also use multi-scale algorithms to reduce the amount of calculation and find potential EEG signal patterns. At the same time, it encodes high-dimensional information into images so that the images contain rich information, makes full use of the advantages of machine vision, uses a 2DCNN model to extract high-dimensional features of images, and obtains better emotion classification results through different multi-modal feature fusion methods.

Description

Technical Field

[0001] The present invention relates to the field of physiological signal processing, and in particular to a multi-modal feature fusion electroencephalogram emotion recognition method based on multi-scale imaging. Background Art

[0002] Emotion is a complex psychological and physiological state that affects people's cognition, behavior and interpersonal communication. According to cognitive and neurophysiological theories, emotions, which play an important role in human brain activity, can be detected in the brain's electroencephalogram (EEG) signals. Therefore, using EEG signals can effectively identify emotions.

[0003] Traditional emotion recognition methods based on EEG signals mainly use 1DCNN (Chinese meaning: one-dimensional convolution) technology to extract EEG signal features and train classifiers to achieve emotion recognition. This traditional emotion recognition method only focuses on time domain or frequency domain information, resulting in serious loss of EEG spatial information and limited classification performance. It also requires a lot of effort to find emotion-related signal features from the original EEG signal and build corresponding associations, resulting in time-consuming feature calculations and very limited generalization capabilities. In recent years, deep learning has flourished in various fields, providing more possibilities for building emotion classification models. Summary of the invention

[0004] The purpose of the present invention is to provide a multimodal feature fusion EEG emotion recognition method based on multi-scale imaging, aiming to solve the problems in traditional emotion recognition methods such as the need to construct emotion-related features, time-consuming feature calculation, poor model generalization ability, serious loss of EEG spatial information, and limited classification performance.

[0005] To achieve the above purpose, the present invention adopts the following technical scheme: Due to the success of convolutional neural networks in image classification, time series classification using time series coding imaging also shows high performance. The present invention combines multi-scale and time series imaging algorithms, and realizes emotion recognition by converting EEG signals into images. Compared with the traditional emotion recognition method based on EEG signals, it can not only preserve the spatial information of EEG signals, but also use multi-scale algorithms to reduce the amount of calculation, find potential EEG signal patterns, and encode high-dimensional information into images at the same time, so that the images contain rich information, make full use of the advantages of machine vision, use 2DCNN (Chinese meaning: two-dimensional convolution) model to extract high-dimensional features of images, and obtain better emotion classification results through different multimodal feature fusion methods.

[0006] The multi-modal feature fusion EEG emotion recognition method based on multi-scale imaging of the present invention comprises the following steps:

[0007] S1, use python code to remove the baseline of the original EEG signal to obtain the first EEG signal;

[0008] S2, performing multi-scale processing on the first EEG signal to obtain a second EEG signal;

[0009] S3, converting the second EEG signal into an image using a time series imaging algorithm to obtain N image data sets;

[0010] S4, performing data enhancement on the N image data sets to construct N samples and label sets;

[0011] S5, the N samples and label sets are respectively passed through the ResNet model and the DNN-01 model to obtain N first feature vectors;

[0012] S6, N first eigenvectors are combined to form three second eigenvectors;

[0013] S7, the three second feature vectors are respectively passed through the DNN-02 model to form a multi-modal feature fusion EEG emotion classification model;

[0014] S8, using the ten-fold cross-validation method to randomly divide the N samples and label sets in step S4 into M parts, of which M-1 parts are used as training data and the others are used as test data, to train the EEG emotion classification model with multimodal feature fusion to obtain the EEG emotion classification recognition model.

[0015] Furthermore, the baseline removal includes the following: dividing the baseline signal and the experimental signal in the original EEG signal into K segments and I segments of length L respectively, and subtracting the average value of all baseline signal segments from each experimental signal segment.

[0016] Furthermore, the mathematical definition of the multi-scale processing is:

[0017] ,

[0018] in, is the time scale set, L is the length of the original EEG signal, is the signal value of the original EEG signal at time i, is the second EEG signal, and j is the index of the second EEG signal.

[0019] Furthermore, the data enhancement is Mixup.

[0020] Furthermore, the three second eigenvectors are: a second eigenvector formed by adding N first eigenvectors; a second eigenvector formed by the maximum values ​​of the same positions of N first eigenvectors; and a second eigenvector formed by weighted combination of N first eigenvectors using a fully connected layer.

[0021] Furthermore, the weights in the weighted combination are parameters of the N first feature vectors after training through a fully connected layer.

[0022] Furthermore, the loss calculation formula of the multimodal feature fusion EEG emotion classification model is as follows:

[0023] ,

[0024] in, and are preset parameters, Li is the loss of the N samples and label sets through the ResNet model and the DNN-01 model, integer; Lcom is the loss of the three second feature vectors passed by the DNN-02 model respectively.

[0025] The advantage of the present invention is that it combines multi-scale and time series imaging algorithms, and realizes emotion recognition by converting EEG signals into images. Compared with the traditional emotion recognition method based on EEG signals, it can not only preserve the spatial information of EEG signals, but also use multi-scale algorithms to reduce the amount of calculation and find potential EEG signal patterns. At the same time, it encodes high-dimensional information into the image so that the image contains rich information, makes full use of the advantages of machine vision, uses a 2DCNN model to extract high-dimensional features of the image, and obtains better emotion classification results through different multimodal feature fusion methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a flow chart of the method of the present invention.

[0027] Figure 2 32 electrode position diagrams in the method embodiment of the present invention.

[0028] Figure 3 It is a schematic diagram of spatial information after the EEG signal is converted into an image in an embodiment of the method described in the present invention.

[0029] Figure 4 Schematic diagram of the ResNet network structure in the method embodiment of the present invention. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] like Figure 1 As shown, the multi-modal feature fusion EEG emotion recognition method based on multi-scale imaging of the present invention comprises the following steps:

[0032] S1, use python code to remove the baseline of the original EEG signal to obtain the first EEG signal;

[0033] This example downloads an EEG signal dataset from the public DEAP as raw data. In the DEAP database, 32 participants participated in the experiment. Each participant was asked to watch 40 one-minute music videos. The EEG signals were recorded from 32 electrodes according to the international 10-20 system. The electrode positions are as follows: Figure 2 As shown. After watching each video, participants rated valence, arousal, dominance, and liking on a continuous scale between 1 and 9. The data recorded for each participant included 40 segments of electroencephalogram (EEG) data and corresponding labels. Each segment of EEG data contained a 60-second experimental signal and a 3-second baseline signal in a relaxed state.

[0034] Since human EEG signals are unstable and easily affected by slight changes in the surrounding environment, and the EEG signals generated by emotional stimulation are also affected by the emotional state before receiving the stimulation to a certain extent. Therefore, removing the baseline can achieve better classification results. In the present invention, the baseline signal and the experimental signal are divided into K segments and I segments of length L, respectively, and then the EEG signal is processed by removing the baseline signal (i.e., the EEG signal in a relaxed state) using python code. The method is to subtract the average value of all baseline signal segments from each experimental signal segment. The mathematical expression is:

[0035] ,

[0036] ,

[0037] in, is the average value of all baseline signal segments; is the i-th baseline signal segment; is the i-th experimental signal segment; is the i-th experimental signal segment after removing the baseline;

[0038] S2, performing multi-scale processing on the first EEG signal to obtain a second EEG signal;

[0039] Since the potential EEG patterns of EEG signals are unknown and the related time scale is also unknown, using multi-scale processing of EEG signals can not only reduce the amount of data, but also enable machines to learn different scale patterns, thereby improving classification performance. The mathematical definition of multi-scale processing is:

[0040] ,

[0041] in, To set the time scale, QUOTE is the length of the original EEG signal, is the signal value of the original EEG signal at the i-th moment, is the second EEG signal, and j is the index of the second EEG signal.

[0042] when hour, That is the signal value of the original EEG signal.

[0043] when hour, That is, it is a coarse-grained time series formed by the average value of the original EEG signal at every two consecutive moments. hour, , hour, , hour, ,……

[0044] when hour, That is, it is a coarse-grained time series formed by the average value of the original EEG signal at every three consecutive moments. hour, , hour, , hour, ,……

[0045] The coarse-grained time series is the second EEG signal.

[0046] S3, converting the second EEG signal into an image using a time series imaging algorithm to obtain N image data sets;

[0047] This step converts the second EEG signal into an image, which can utilize the information in the original EEG signal and encode high-dimensional information into the image, so that the image contains rich information and can fully utilize the advantages of current machine vision to obtain better emotion classification results. The present invention uses a time series imaging algorithm (abbreviated as MDF in English) to convert the second EEG signal.

[0048] First, from a coarse-grained time series , Take n values ​​as a basic time series unit, denoted as An integer; and set the value interval d and the starting index s of the value.

[0049] When d=1, it means that from a coarse-grained time series Take n values ​​in succession as a basic time series unit, then

[0050]

[0051] When n=2, ;

[0052] When n=3, ;

[0053] when , which means that the time series after a coarse-graining Take n values ​​as a basic time series unit, then

[0054]

[0055] When n=2, ;

[0056] When n=3, ;

[0057] Right now ;

[0058] After that, the difference between the basic units is calculated at different intervals d to obtain a new time series, which is recorded as , the specific steps are shown in the following formula:

[0059] ;

[0060] in, ;

[0061] because, The lengths are different, so a new sequence needs to be constructed , the formula is:

[0062] ;

[0063] in, ;

[0064] Then, construct the MDF matrix, the formula is expressed as:

[0065] ;

[0066] When the value of n is determined, the corresponding That is elements, which can generate Channel data, among which, The matrix of channels can be defined as:

[0067] ;

[0068] in, .

[0069] To fill The elements with value 0 in the matrix, each channel of the MDF image is defined as:

[0070]

[0071] in represents the Hadamard product. yes Matrix rotation After the matrix, To prevent and Overlap when added,

[0072]

[0073] At the same time, after using the above-mentioned time series imaging algorithm to convert each channel data of the second EEG signal into an image, the images must be spliced ​​into a large image according to the physical positions of the corresponding channels to retain the spatial information of the EEG signal as much as possible, such as Figure 3 The letters in the figure represent the 32 electrode positions according to the international 10-20 system of electrode placement.

[0074] S5, performing data enhancement on the N image data sets to construct N samples and label sets;

[0075] Mixup is a simple and effective data augmentation method that constructs new training samples and labels in a linear interpolation manner. This application uses Mixup to significantly enhance the generalization ability of the network architecture with very little computational overhead. The specific process of this step is expressed in the following formula:

[0076] ;

[0077] ;

[0078] in, and is sample data in the image dataset; and For and The corresponding label. For an obedient The parameters of the distribution, ,in .

[0079] S6, the N samples and label sets are respectively passed through the ResNet model and the DNN-01 model to obtain N first feature vectors;

[0080] The classic ResNet model is used as the feature extraction network. Compared with the traditional CNN (convolutional neural network) network, it has two differences: (1) the use of residual structure can build an ultra-deep network structure, and (2) the use of batch normalization layer (full spelling in English is Batch Normalization), which solves two problems existing in traditional convolutional networks: (1) gradient disappearance or gradient explosion, and (2) degradation problem.

[0081] The ResNet18 network is mainly composed of input layer, convolution layer, Batch Normalization layer, activation function, pooling layer, residual structure, fully connected layer, and exponential normalization layer (Softmax). The specific structure is as follows Figure 4 shown.

[0082] In the embodiment of the present application, when the MDF algorithm is used to transform the second EEG signal, n is taken as 2, 3, and 4 respectively, and three different image data sets are obtained. Considering that the information contained in the images transformed by different algorithms is different, in order to make full use of all the image data, the present invention sends the three different image data sets to the ResNet18 model and the DNN-01 model respectively, extracts different image high-level features, and obtains three first feature vectors, which are recorded as , , .

[0083] The network structure of the DNN-01 model is shown in Table 1 below. In the table, FL represents the fully connected layer, RELU represents the linear rectification function, Dr represents random dropout, and the numbers in the Output column represent the dimensions of the output features of this layer.

[0084] Table 1 DNN-01 network structure

[0085]

[0086] Among them, the meaning of RELU in the supplementary table and the meaning of the numbers in the Outpu column

[0087] Although the three different image data sets all need to go through the same ResNet18 model and DNN-01 model, the parameters after model training are different due to the different image data.

[0088] S6, the three first eigenvectors are combined to form three second eigenvectors;

[0089] like Figure 4 As shown in , the present application combines three first feature vectors into three second feature vectors through a feature combiner (Comber in English) module. The specific combination method is:

[0090] The first combination method is denoted as SUM. , , The three first eigenvectors are added together to get a new vector , the formula is:

[0091] ;

[0092] The second combination method is recorded as MAX, taking out , , The maximum values ​​of the three first eigenvectors at the same position are used to form a new vector , the formula is:

[0093]

[0094] The third combination method is denoted as FC, assuming , , The three first eigenvectors have a linear relationship, and then a fully connected layer is used to weightedly combine the three first eigenvectors to obtain a new vector , the formula is:

[0095] ;

[0096] in yes , , Parameters of the three first eigenvectors after training with the fully connected layer.

[0097] S7, the three second feature vectors formed after combination are sent to the DNN-02 model to form a multi-modal feature fusion EEG emotion classification model; the DNN-02 network structure is shown in Table 2.

[0098] Table 2 DNN-02 network structure

[0099] ;

[0100] S8, using the ten-fold cross-validation method to randomly divide the N samples and label sets in step S4 into 10 parts, 9 of which are used as training data and the remaining 1 is used as test data, to train the EEG emotion classification model with multimodal feature fusion to obtain the EEG emotion classification recognition model.

[0101] In this embodiment, the EEG signal of each experimenter is transformed using MDF, and n is set to 2, 3, and 4, obtaining three different image data sets with 2400 images. The image data set is randomly divided into 10 parts using the ten-fold cross-validation method, and 9 of them are used as training data and 1 as test data in turn. Afterwards, the images of the training set are sent to the EEG emotion classification model with multimodal feature fusion obtained in step S7 above for training. The Adam optimizer is selected, and the learning rate is set to 0.0001, the loss function is set to the cross entropy loss function, and the batch size is set to 32 to obtain the EEG emotion classification recognition model.

[0102] In this embodiment, a total of 4 loss functions are defined: , , , To train the classification network, , , It is used to optimize the loss of three different image datasets when training with a single ResNet18 and DNN-01 network structure. Used to optimize the loss of the second feature vector through the DNN-02 model. Optimize the loss of the entire network structure, The formula is as follows:

[0103] ;

[0104] in, and It is a preset parameter that focuses on learning specific features or combinations. In this embodiment, it is set to 1 / 3 and 1.0 to make the contribution of each image data during training equal.

Claims

1. A multi-modal feature fusion EEG emotion recognition method based on multi-scale imaging, characterized by: The following steps are involved: S1, use python code to remove the baseline of the original EEG signal to obtain the first EEG signal; S2, performing multi-scale processing on the first EEG signal to obtain a second EEG signal; S3, converting the second EEG signal into an image using a time series imaging algorithm to obtain N image data sets; Take n values ​​from the second EEG signal as a basic time series unit, denoted as integer; and set the interval d and the starting index s of the value; find the difference between the basic units at different intervals d to get a new time series, recorded as , construct the MDF matrix, convert each channel data of the second EEG signal into an image, and then splice the images into a large image according to the physical positions of the corresponding channels; S4, performing data enhancement on the N image data sets to construct N samples and label sets; S5, the N samples and label sets are respectively passed through the ResNet model and the DNN-01 model to obtain N first feature vectors; S6, N first eigenvectors are combined to form three second eigenvectors; S7, the three second feature vectors are respectively passed through the DNN-02 model to form a multi-modal feature fusion EEG emotion classification model; S8, using the ten-fold cross-validation method to randomly divide the N samples and label sets in step S4 into ten parts, nine of which are used as training data and the remaining one is used as test data, to train the EEG emotion classification model with multimodal feature fusion to obtain the EEG emotion classification recognition model.

2. The multi-modal feature fusion EEG emotion recognition method based on multi-scale imaging according to claim 1 is characterized in that: The baseline removal includes the following: dividing the baseline signal and the experimental signal in the original EEG signal into K segments and I segments of length L respectively, and subtracting the average value of all baseline signal segments from each experimental signal segment.

3. The multi-modal feature fusion EEG emotion recognition method based on multi-scale imaging according to claim 1 is characterized in that: The mathematical definition of the multi-scale process is: , in, is the time scale set, L is the length of the original EEG signal, is the signal value of the original EEG signal at time i, is the second EEG signal, and j is the index of the second EEG signal.

4. The multi-modal feature fusion EEG emotion recognition method based on multi-scale imaging according to claim 1 is characterized in that: The data enhancement is Mixup.

5. The multi-modal feature fusion EEG emotion recognition method based on multi-scale imaging according to claim 1 is characterized in that: The three second eigenvectors are: a second eigenvector formed by adding N first eigenvectors; a second eigenvector formed by the maximum values ​​of the same positions of N first eigenvectors; and a second eigenvector formed by weighted combination of N first eigenvectors using a fully connected layer.

6. The multi-modal feature fusion EEG emotion recognition method based on multi-scale imaging according to claim 5 is characterized in that: The weights in the weighted combination are parameters of the N first feature vectors after training through a fully connected layer.

7. The multi-modal feature fusion EEG emotion recognition method based on multi-scale imaging according to claim 1 is characterized by: The loss calculation formula of the multimodal feature fusion EEG emotion classification model is as follows: ， in, and are preset parameters, Li is the loss of the N samples and label sets through the ResNet model and the DNN-01 model, integer; L com The three second feature vectors are respectively passed through the loss of the DNN-02 model.

Images