An emotion recognition method and system based on EEG signals of a small number of channels

Abstract

The present invention belongs to the field of brain-computer interface, and specifically relates to an emotion recognition method and system based on a small number of EEG channels, including obtaining EEG signals from four channels of a user's scalp, and preprocessing the EEG signals; converting the preprocessed EEG signals into basic signals, decomposing and reconstructing the basic signals to obtain a series of component signals; inputting the smoothed features into a trained classification model to detect user emotions; the present invention uses EEG signals from fewer channels to detect user emotions, and the user only needs to mobilize emotions according to actual scenarios. The PC will analyze the collected EEG signals from the user's cerebral cortex, and through preprocessing and feature extraction, the final features will be input into a classifier for classification, which can reflect the user's emotion category at that time.

Description

Technical Field

[0001] The present invention belongs to the field of brain-computer interface, and specifically relates to an emotion recognition method and system based on a small number of EEG channels. Background Art

[0002] Emotions play an important role in daily life. They include positive emotions, negative emotions, alertness, fatigue and other psychological states, which affect a person's physiological state. Researchers in different fields use facial expressions, voice signals, text messages, electrocardiograms and electroencephalograms to analyze people's psychological states. The study found that electroencephalogram signals are least affected by different personal characteristics. However, there is noise in the channel of EEG signals, and the signal-to-noise ratio is low, which limits the analysis and processing of EEG signals.

[0003] With the rapid development of computer, wireless communication and intelligent computing technology, researchers are constantly developing intelligent wearable devices to improve people's lives. However, there are currently few wearable devices based on electroencephalogram (EEG) emotion recognition that have entered the market. The reason lies in the limitations of EEG signal acquisition technology. The EEG signals collected by head-mounted EEG caps have high noise. At the same time, the multi-channel data acquisition method and emotion recognition algorithm currently used in the laboratory are relatively complex and difficult to apply to wearable devices.

[0004] In recent years, many researchers have focused on improving the classification accuracy of emotion recognition based on EEG signals when there are more channels. These algorithms are difficult to apply in wearable devices. At the same time, the EEG data of all channels of the cerebral cortex contain relevant information about various brain regions, but for specific tasks, using all channels is not conducive to the application of algorithms in wearable devices to the mass market. Therefore, using fewer channels to collect real-time data and complete classification tasks can greatly improve the practicality of the algorithm on wearable devices. Summary of the invention

[0005] To solve the above problems, the present invention proposes an emotion recognition method and system based on a small number of EEG channels.

[0006] In a first aspect, the present invention proposes an emotion recognition method based on a small number of EEG channels, comprising:

[0007] S1. Obtain EEG signals from four channels of the user's scalp and pre-process the EEG signals;

[0008] S2. converting the preprocessed EEG signal into a basic signal, decomposing and reconstructing the basic signal to obtain a series of component signals;

[0009] S3. extracting differential entropy of the base signal and the component signal as features, and smoothing the features;

[0010] S4. Input the smoothed features into the trained classification model to detect user emotions.

[0011] Furthermore, the four channels of the user's scalp are FP1 and FP2 channels located in the frontal lobe and T7 and T8 channels located in the temporal lobe.

[0012] Furthermore, the EEG signals of FP1, FP2, T7, and T8 channels were preprocessed, and the potential difference between the channel electrodes was used to construct the basic signal, that is, the four preprocessed EEG signals were subtracted from each other to obtain 6 basic signals, namely FP1-T7, FP1-T8, FP1-FP2, FP2-T7, FP2-T8, and T7-T8.

[0013] Furthermore, the process of preprocessing the EEG signal is as follows:

[0014] Downsample the EEG signal to obtain a 200Hz EEG signal;

[0015] The 200 Hz EEG signal is filtered using a Butterworth filter to obtain the filtered EEG signal.

[0016] Furthermore, the basic signal is decomposed and reconstructed by using intrinsic time scale decomposition, discrete wavelet transform, variational mode decomposition, and phase space reconstruction to expand the existing modes, where:

[0017] The intrinsic time scale decomposition uses affine linear transformation to process the basic signal to obtain PRC1 rotation component signal and PRC2 rotation component signal;

[0018] Discrete wavelet transform extracts component signals of four physiological EEG bands, namely Gamma, Beta, Alpha and Theta, from the basic signal by scaling and shifting the mother wavelet function.

[0019] Variational mode decomposition uses a parallel method to extract intrinsic mode functions from the basic signal, and obtains four component signals: VMD1, VMD2, VMD3, and VMD4;

[0020] Phase space reconstruction reconstructs the basic signal to obtain the 3D-PSR-ED component signal.

[0021] Furthermore, a linear dynamic system is used to filter out the components in the features that are not related to the emotional state and complete feature smoothing. The linear dynamic system is expressed as:

[0022] x t =z t +w t ;

[0023] z t =Az t-1 +v t ;

[0024] where x t represents the observed variable, z t represents the hidden sentiment variable, A represents the transfer matrix, and w t It means the mean Gaussian noise with variance Q, v t It means the mean Gaussian noise with variance R.

[0025] Based on the first aspect, the present invention proposes an emotion recognition system based on a small number of channel EEG signals, comprising:

[0026] The EEG signal collection module is used to collect the EEG signals generated by the FP1, FP2, T7, and T84 channels of the scalp during the target user's emotion detection;

[0027] The signal preprocessing module is used to process the 4-channel EEG signals obtained by the EEG signal collection module and convert them into basic signals;

[0028] A signal decomposition and reconstruction module is used to decompose and reconstruct the basic signal output by the signal preprocessing module to obtain multiple component signals;

[0029] A feature extraction module is used to extract the features of the basic signal and the component signal and perform smoothing on the features;

[0030] The classification module is used to input the smoothed features into the trained classification model to detect user emotions.

[0031] Beneficial effects of the present invention:

[0032] The present invention proposes an emotion recognition method and system based on a small number of EEG channels, which detects user emotions by collecting EEG signals from a small number of channels. The user only needs to mobilize emotions according to the actual scenario. The PC will analyze the collected EEG signals of the user's cerebral cortex, and through preprocessing and feature extraction, the final features are input into the classifier for classification, which can reflect the user's emotion category at that time. The present invention only uses data from 4 channels, which reduces the difficulty of data collection, reduces the amount of data processing, and provides convenience for wearable simple devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a flow chart of the emotion recognition method based on a small number of EEG channels of the present invention. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] In order to improve the accuracy of emotion recognition and enhance the practicability of the algorithm on wearable devices, the present invention proposes an emotion recognition method and system based on a small number of EEG channels.

[0036] In one embodiment, an emotion recognition method based on a small number of EEG channels, such as Figure 1 As shown, including:

[0037] S1. Obtain EEG signals from four channels of the user's scalp and pre-process the EEG signals;

[0038] S2. converting the preprocessed EEG signal into a basic signal, decomposing and reconstructing the basic signal to obtain a series of component signals;

[0039] S3. extracting differential entropy of the base signal and the component signal as features, and smoothing the features;

[0040] S4. Input the smoothed features into the trained classification model to detect user emotions.

[0041] Preferably, in order to accurately identify the user's emotion category and take into account the need for convenient design of portable devices, this embodiment selects the FP1 and FP2 channels located in the frontal lobe and the channels located in the temporal lobe T7 and T8. The locations of these four channels provide convenience for designing wearable products such as headbands, headphones and glasses. The EEG signals generated by these four channels of the mobile phone, in order to retain the spatial asymmetry of the EEG signals in the brain area as much as possible, use the potential difference between the channel electrodes to construct 6 original basic signals, which are: FP1-T7, FP1-T8, FP1-FP2, FP2-T7, FP2-T8, T7-T8, each original basic signal is sliced ​​according to a time period of 5 seconds, and the slices of the 6 original basic signals within a time period are used as a group of basic signals for subsequent processing and feature extraction.

[0042] Preferably, in this embodiment, the process of preprocessing the EEG signal before converting it into a basic signal is as follows:

[0043] Downsample the EEG signal to obtain a 200Hz EEG signal;

[0044] The Butterworth filter was used to filter the 200 Hz EEG signal, and the EEG signal after filtering in the 0-49 Hz frequency band was intercepted.

[0045] In one embodiment, the basic signal is decomposed and reconstructed using intrinsic time scale decomposition, discrete wavelet transform, variational mode decomposition, and phase space reconstruction.

[0046] Specifically, the intrinsic time scale decomposition, also known as the ITD method, can adaptively process nonlinear and non-stationary signals, strictly limit the endpoint effect to the endpoint, and do not propagate inward to pollute the entire data segment, thus overcoming the problem that the empirical mode decomposition method (EMD) propagates the endpoint effect inward to pollute the entire data segment. The ITD method processes the signal in the form of affine linear transformation, decomposing the signal into several proper rotation components (proper rotation components, PRC) and monotonic trend components that do not interfere with each other in the frequency domain. After decomposition, better instantaneous characteristics of the signal can be obtained. Since emotion recognition is mainly related to the high-frequency signals of the human brain, in this embodiment, the first two proper rotation components PRC1 and PRC2 are mainly selected as the main proper rotation components.

[0047] Specifically, Discrete Wavelet Transformation (DWT) discretizes the scale and translation of the basic wavelet, decomposes it into a series of functions called wavelets by scaling and shifting the mother wavelet, and forms a decomposition set of wavelet coefficients. In this embodiment, a 5-level DWT with a third-order Daubechies (db3) wavelet function is used to extract five physiological EEG bands, namely Gamma (31-48Hz), Beta (14-30), Alpha (8-13Hz), Theta (4-7Hz), and Delta (0.5-3Hz), and then the Delta wave is removed because its appearance is often related to human sleep and plays a very small role in the emotion recognition task.

[0048] Specifically, variational mode decomposition (VMD) has a rigorous mathematical model and a solid theoretical basis, and can overcome the shortcomings of frequency aliasing in EMD and local mean decomposition (LMD), and has a better noise filtering effect. VMD does not use a recursive method, but a parallel method to extract intrinsic mode functions from the signal. The number K of decomposition components in VMD needs to be predefined. If K is too small, it is not conducive to obtaining more modes, but if K is too large, the so-called over-segmentation will increase the computational load. Fortunately, a good feature of VMD is that when the signal is over-segmented, the redundant mode is basically a noise component. Considering the performance and the need for denoising, K is set to 4 in this embodiment, that is, the first four intrinsic mode components VMD1, VMD2, VMD3, and VMD4 are selected as the main modes.

[0049] Specifically, since EEG signals have a strong continuity in time, it is sometimes necessary to search for patterns in time series and high-dimensional transformations of time series. Although the phase space reconstruction method uses the value of a variable at different times to form the phase space, the change of a variable in the dynamic system is naturally related to the interaction between this variable and other variables of the system, that is, the change of this variable over time implies the dynamic law of the entire system. Therefore, the trajectory of the reconstructed phase space also reflects the evolution law of the system state. Phase space reconstruction technology has two key parameters: the embedding dimension d and the delay time τ. In this embodiment, 3D-PSR with d=3 and τ=1 is set to obtain the 3D-PSR-ED component signal.

[0050] In one embodiment, the differential entropy of the basic signal and the component signal is extracted as a feature, and the feature is smoothed, that is, the differential entropy is extracted as a feature for the basic signal, the signal after phase space reconstruction (3D-PSR-ED), the intrinsic rotation amount (PRC1, PRC2) decomposed by ITD, the rhythm (Gamma, Beta, Alpha, Theta) after DWT decomposition, and the component (VMD1, VMD2, VMD3, VMD4) after VMD decomposition. After the above processing, 1x12 features can be extracted from each basic signal, and finally 6x12 features are obtained for 6 basic signals. For the extracted features, the linear dynamic system (LDS) method is applied to the time series to filter out the components that are not related to the emotional state, thereby completing feature smoothing.

[0051] Specifically, the linear dynamic system is expressed as:

[0052] x t =z t +w t ;

[0053] z t =Az t-1 +v t ;

[0054] where x t represents the observed variable, z t represents the hidden sentiment variable, A represents the transfer matrix, and w t It means the mean Gaussian noise with variance Q, v t It means the mean Gaussian noise with variance R.

[0055] On this basis, the present invention proposes an emotion recognition system based on a small number of channel EEG signals, comprising:

[0056] The EEG signal collection module is used to collect the EEG signals generated by the FP1, FP2, T7, and T8 channels of the scalp during emotion detection of the target user;

[0057] The signal preprocessing module is used to process the 4-channel EEG signals obtained by the EEG signal collection module and convert them into basic signals;

[0058] A signal decomposition and reconstruction module is used to decompose and reconstruct the basic signal output by the signal preprocessing module to obtain multiple component signals;

[0059] A feature extraction module is used to extract the features of the basic signal and the component signal and perform smoothing on the features;

[0060] The classification module is used to input the smoothed features into the trained classification model to detect user emotions.

[0061] Specifically, a simple CNN structure similar to LeNet-5 is used in the classification module. Compared with LeNet5, the CNN model used in this embodiment reduces the number of layers and convolution kernels to achieve the purpose of reducing model parameters, making the model more suitable for small sample classification tasks based on artificial features. The specific structure of the model is shown in Table 1. The CNN model includes three convolution layers, two pooling layers, and a Linear layer. The convolution kernels of the convolution layer and the pooling layer are both 2. The input data passes through two convolution layers in turn and then performs a maximum pooling operation. The data after the maximum pooling is convolved and pooled again, and the result of the second pooling is input into the Linear layer, and then emotion recognition is performed through Softmax.

[0062] Table 1 Architecture of the CNN model of the present invention

[0063]

[0064] In one embodiment, a public emotional EEG dataset named SEED is used for experiments. The dataset uses movie clips as emotional induction materials, including three categories: positive emotions, neutral emotions, and negative emotions. In each group of experiments, participants watch 15 videos, 5 of each type of emotion. The SEED dataset contains 62 channels (electrodes) of EEG emotional signals recorded by 15 participants (7 males and 8 females). In this embodiment, an experimental paradigm similar to the SEED dataset is adopted, and data from 10 participants are collected in the laboratory. Since the method of the present invention only uses 4 channels, only 14 channels are collected during data collection, namely: FP1, FP2, F7, F8, F3, F4, FC5, FC6, T7, T8, P7, P8, O1, O2. In this embodiment, Type1 and Type2 modes are used for experiments, and Accuracy1 and Accuracy2 are used to record the experimental results.

[0065] Type 1: The EEG signals generated by the cerebral cortex of each participant when they watched 15 videos were collected, and the EEG signals were divided into 5-second trails. All trails were extracted, and all trails were randomly shuffled first. 20% of the data from each type of emotion sample was extracted each time as the test set, and the rest was used as the training set to complete a 5-fold cross-validation experiment.

[0066] Type2: Each time, each participant is selected to watch 3 videos (one positive, one negative, and one neutral), and the EEG signals collected and all the trails extracted are used as the test set. The EEG signals collected when watching the remaining videos are used as the training set to complete a 5-fold cross-validation experiment.

[0067] In the Type 1 experiment, the method achieved an average classification accuracy of 98.67% and 99.97% on the SEED dataset and the dataset of the present invention, respectively. The Type 2 experiment ensured that the training set and the test set were from completely different time periods, which is closer to the actual usage scenario. An average accuracy of 90.77% was obtained on the SEED data, and an average accuracy of 91.38% was obtained on the dataset of the present invention.

[0068] The experimental results show that the best performance can be obtained in Type 1 mode. However, since a person's emotional state has a certain continuity in time, it will not change significantly in a very short period of time. The overall disruption of all data in the Type 1 experiment will cause the trails extracted from a continuous period of time to be distributed in both the training set and the test set, resulting in data leakage. This is also an important reason why the results of both the SEED dataset and the dataset of the present invention in the Type 1 experiment are significantly better than those of Type 2. In addition, differences between participants are also an important reason for differences in experimental results, which may be due to subjective personal will, growth environment and physiological state, or may be due to objective experimental environment.

[0069] In order to verify the effectiveness of the proposed method, experiments were conducted on the 62-channel SEED dataset and the 14-channel dataset of the proposed method according to the classic EEG signal emotion recognition method. PSD features, DE features and normalized first-order differences (1 ST OD), input into SVM and conduct Type 1 and Type 2 experiments respectively, and compare the present invention with the classical method (the classical method usually only uses discrete wavelet transform for signal decomposition) on the same features and classifiers, and obtain the results of PSD features, DE features and applying the DE features to SVM and the CNN model of this paper after LDS feature smoothing. The results are shown in Table 2 and Table 3:

[0070] Table 2 Performance comparison between using 4 channels and traditional 62 channels on SEED dataset

[0071]

[0072] Table 3 Performance comparison between the use of 4 channels and the traditional 14 channels on the dataset of the present invention

[0073]

[0074] As shown in Table 1 and Table 2, although only 4 channels of EEG signal data were collected in the method of this paper, the average accuracy of the present invention is better than that of the 62-channel SEED dataset under the same experimental mode, features and classifiers. On the dataset of the present invention, the method of the present invention is better than the traditional method when the feature is 1stOD, and the performance gap between the method of the present invention and the multi-channel traditional method is also very small when the feature is PSD or DE.

[0075] In addition, under DE features, the results of comparing the use of SVM and CNN after LDS feature smoothing show that a huge performance improvement can be achieved through the improvement of feature smoothing and classification methods.

[0076] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An emotion recognition method based on EEG signals of a small number of channels, characterized in that: include: S1. Obtain EEG signals from four channels of the user's scalp and pre-process the EEG signals; The four channels of the user's scalp are the FP1 and FP2 channels located in the frontal lobe and the T7 and T8 channels located in the temporal lobe; S2. converting the preprocessed EEG signal into a basic signal, decomposing and reconstructing the basic signal to obtain a series of component signals; The EEG signals of FP1, FP2, T7, and T8 channels were preprocessed, and the potential difference between the channel electrodes was used to construct the basic signal, that is, the four preprocessed EEG signals were subtracted from each other to obtain 6 basic signals, namely FP1-T7, FP1-T8, FP1-FP2, FP2-T7, FP2-T8, and T7-T8; S3. extracting differential entropy of the base signal and the component signal as features, and smoothing the features; S4. Input the smoothed features into the trained classification model to detect user emotions.

2. The emotion recognition method based on EEG signals of a small number of channels according to claim 1, characterized in that: The process of preprocessing EEG signals is as follows: Downsample the EEG signal to obtain a 200Hz EEG signal; The 200 Hz EEG signal is filtered using a Butterworth filter to obtain the filtered EEG signal.

3. The emotion recognition method based on EEG signals of a small number of channels according to claim 1, characterized in that: The basic signal is decomposed and reconstructed by using intrinsic time scale decomposition, discrete wavelet transform, variational mode decomposition and phase space reconstruction respectively, where: The intrinsic time scale decomposition uses affine linear transformation to process the basic signal to obtain PRC1 rotation component signal and PRC2 rotation component signal; Discrete wavelet transform is used to extract component signals of four physiological EEG bands, namely Gamma, Beta, Alpha and Theta, from the basic signal. Variational mode decomposition uses a parallel method to extract intrinsic mode functions from the basic signal, and obtains four component signals: VMD1, VMD2, VMD3, and VMD4; Phase space reconstruction reconstructs the basic signal to obtain the 3D-PSR-ED component signal.

4. The emotion recognition method based on EEG signals of a small number of channels according to claim 1, characterized in that: A linear dynamic system is used to filter out the components in the features that are not related to the emotional state and complete feature smoothing. The linear dynamic system is expressed as: x t =z t +w t ; z t =Az t-1 +v t ; where x t represents the observed variable, z t represents the hidden sentiment variable, A represents the transfer matrix, and w t It means the mean Gaussian noise with variance Q, v t It means the mean Gaussian noise with variance R.

5. An emotion recognition system based on EEG signals of a small number of channels, characterized in that: include: The EEG signal collection module is used to collect the EEG signals generated by the FP1, FP2, T7, and T8 channels of the scalp during emotion detection of the target user; The signal preprocessing module is used to process the 4-channel EEG signals obtained by the EEG signal collection module and convert them into basic signals; A signal decomposition and reconstruction module is used to decompose and reconstruct the basic signal output by the signal preprocessing module to obtain multiple component signals; A feature extraction module is used to extract the features of the basic signal and the component signal and perform smoothing on the features; The classification module is used to input the smoothed features into the trained classification model to detect user emotions.

Images