An electroencephalogram emotion recognition method and system based on multi-feature fusion and CLSTN

Abstract

The application discloses an electroencephalogram emotion recognition method based on multi-feature fusion and CLSTN, which comprises the following steps: (1) collecting electroencephalogram data of emotions; (2) preprocessing the data; (3) multi-feature fusion; and (4) CLSTN model building and training. The application further provides an electroencephalogram emotion recognition method based on multi-feature fusion and CLSTN, which comprises the following modules in sequence: (1) an electroencephalogram data collection module of emotions; (2) a data preprocessing module; (3) a multi-feature fusion module; (4) a CLSTN model building and training module; and (5) an emotion recognition module. The application effectively improves the classification accuracy of electroencephalogram emotion recognition and can better recognize emotions.

Description

Technical Field

[0001] This invention relates to the fields of brain-computer interaction and emotion recognition, and specifically to a brainwave emotion recognition method and system based on multi-feature fusion and CLSTN. Background Technology

[0002] Emotion is a complex mental state closely related to daily life. Accurate emotion recognition plays a crucial role in various fields, including psychology, medicine, and education. Previous emotion recognition research typically relied on facial expressions, tone of voice, body language, or simple physiological signals. However, many researchers today are using brain-computer interface (BCI) technology to improve emotion recognition performance. This is because brain signals are difficult to forge and allow for direct measurement of neuronal activity, making the data more authentic and reliable, and enabling better identification of genuine human emotions.

[0003] In EEG emotion recognition, manual feature extraction is required, followed by classification and recognition using machine learning or deep learning methods. Previous studies typically considered only single features and simple models, leading to the loss of other information and resulting in poor recognition accuracy. The EEG emotion recognition method based on multi-feature fusion and CLSTN can fuse multiple features with a two-dimensional matrix, and simultaneously extract useful information from multiple aspects through convolutional and time-series modules, thereby significantly improving the accuracy of emotion recognition. Summary of the Invention

[0004] The present invention aims to overcome the above-mentioned shortcomings of the prior art and provide a brainwave emotion recognition method and system based on multi-feature fusion and CLSTN.

[0005] By collecting users' emotional EEG signals after emotional induction, the collected signals are preprocessed with power frequency filtering, bandpass filtering, downsampling, rereference, and independent component analysis. Power spectral density, differential entropy, and wavelet packet decomposition are extracted from the preprocessed data. The features are fused with a two-dimensional matrix mapping the brain electrode locations to obtain the final features. By combining convolution and long and short-term time series modules and incorporating an attention mechanism to adjust parameters, the features are identified to obtain emotion classification.

[0006] This invention provides a brainwave emotion recognition method based on multi-feature fusion and CLSTN, which mainly includes the following steps:

[0007] (1) Collect emotional EEG data; users watch video clips with emotions, and the corresponding labels are recorded according to three different emotional states. Based on the electrode positions of the international standard 10-20 system, the EEG data of their emotion-related channels are collected.

[0008] (2) Preprocessing data; The recorded EEG signals were preprocessed by removing power frequency interference through notch filter, removing noise through 0.5-45Hz bandpass filter, reducing data volume and computational load through downsampling, removing electrooculography and electromyography noise through rereferenced average electrode potential, and removing electrooculography and electromyography noise through independent component analysis.

[0009] (3) Multi-feature fusion: The power spectral density, differential entropy and wavelet packet decomposition of the preprocessed EEG signal are extracted and fused to obtain the final features, which are then input into the CLSTN model.

[0010] (4) CLSTN model construction and training: The extracted features are learned by combining convolution and long and short-term time series modules, the parameters are adjusted by attention mechanism, the SOFTMAX function is used for classification, and the Adam algorithm is used for optimization to build and train the CLSTN model for EEG emotion recognition.

[0011] This invention also provides a system for EEG emotion recognition based on multi-feature fusion and CLSTN, comprising the following modules:

[0012] (1) Emotional EEG data acquisition module, used to acquire and save emotional EEG data;

[0013] (2) Data preprocessing module, which preprocesses the collected data for subsequent feature extraction and fusion;

[0014] (3) Multi-feature fusion module, which extracts features from the preprocessed data for subsequent model input;

[0015] (4) CLSTN model building and training module: The model is built by the convolution module and the long and short-term time series module. The extracted features are used to train and save the model for subsequent emotion recognition.

[0016] (5) Emotion recognition module, used for the recognition and display of EEG emotions.

[0017] The present invention also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the EEG emotion recognition method of the present invention based on multi-feature fusion and CLSTN.

[0018] The present invention also provides a computing device, including a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement the method of the present invention.

[0019] The advantages of this invention are that by preprocessing the acquired signals to remove noise from the original signals, extracting various commonly used features and fusing them with a two-dimensional matrix of the mapped electrode positions to obtain time-domain, frequency-domain, and spatial-domain information, further capturing frequency-domain and spatial information through a convolution module, and capturing the relationship between time variables and time-domain information through a long-short-term time series module, the classification accuracy of EEG emotion recognition is effectively improved, and emotions can be better identified. Attached Figure Description

[0020] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the method of the present invention.

[0022] Figure 2 This is a schematic diagram of feature extraction and fusion according to the present invention.

[0023] Figure 3 This is a schematic diagram of the model framework of the present invention.

[0024] Figure 4 This is a flowchart of the system of the present invention. Detailed Implementation

[0025] The following will describe in detail, with reference to the accompanying drawings and embodiments, a method and system for EEG emotion recognition based on multi-feature fusion and CLSTN according to the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them, and should not be construed as limiting the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0026] Example 1

[0027] See Figure 1 The present invention provides a flowchart of an EEG emotion recognition method based on multi-feature fusion and CLSTN, and the steps therein are described in detail:

[0028] (1) Collect emotional EEG data;

[0029] Users wear an EEG acquisition device and watch video clips with emotions on a monitor. The device records corresponding labels based on three different emotional states: 0 for neutral, 1 for positive emotions, and 2 for negative emotions. EEG data from 18 channels (Fp1, Fpz, Fp2, F7, F3, Fz, F4, F8, T7, C3, Cz, C4, T8, P7, P3, Pz, P4, and P8) are collected according to the international standard 10-20 system electrode positions.

[0030] (2) Preprocess the data;

[0031] The acquired EEG signals were preprocessed by removing power frequency interference through a notch filter, removing noise through a bandpass filter of 0.5-45Hz, reducing data volume and computational load through downsampling, averaging electrode potentials through rereference, and removing electrooculography and electromyography noise through independent component analysis.

[0032] (3) Multi-feature fusion;

[0033] Feature extraction and fusion are performed on the preprocessed signal. For example... Figure 2 As shown, the signal is divided into equal, non-overlapping segments with a time window of 0.5 seconds, increasing the data volume and defining time-domain information. Five frequency bands—delta, theta, alpha, beta, and gamma—are obtained by filtering the signal at 0.5–4.5 Hz, 4–8 Hz, 8–13 Hz, 13–30 Hz, and 30–42 Hz. The power spectral density is calculated using the Discrete Fourier Transform method, with the following formula:

[0034] PSD={∑[abs(DFT i )] 2} / (EndPos-StartPos) (1)

[0035] Where DFTi represents the signal value corresponding to the i-th point obtained through the Discrete Fourier Transform, and EndPos and StartPos represent the end and start points of a frequency band, respectively. The power spectral density of the signal obtained after the Discrete Fourier Transform is obtained by taking the square of its magnitude. Then, it is averaged over all points in the entire frequency band to obtain the average power spectral density of the frequency band as a feature.

[0036] The formula for calculating the differential entropy feature through frequency bands is as follows:

[0037]

[0038] Where x represents continuous information, p(x) represents the probability density function, and [a, b] represents the range of values.

[0039] The wavelet packet decomposition features are calculated using frequency bands, and the calculation formula is as follows:

[0040]

[0041] Where C n,k k represents the wavelet decomposition level. th Approximate components; D j,k k represents the decomposition level th Details: ψ(t) represents the wavelet function.

[0042] Three features were extracted from 18 channels and 5 frequency bands respectively, resulting in three 18*5 features. By fusing the data from each channel of these three features, a fused feature of 18*15 was obtained. Based on the international standard 10-20 system electrode locations, the electrode locations associated with each brain region were mapped into a 9*9 two-dimensional matrix to obtain spatial information. By fusing the fused features corresponding to each electrode channel with the two-dimensional matrix, a final feature of 9*9*15 dimensions was obtained.

[0043] (4) CLSTN Model Construction and Training

[0044] By combining convolutional neural networks with long short-term time series modules (CLSTN), a model for EEG emotion recognition is built and trained. For example... Figure 3 As shown, features from six time windows are sequentially input into six convolutional modules to extract frequency and spatial domain information. These six outputs are then combined sequentially to preserve temporal information. The outputs are then input into a long-short-term time series module to extract temporal information, and finally classified using the SOFTMAX activation function. Each convolutional module contains five different convolutional layers, one max-pooling layer, one flattening layer, and one fully connected layer. The number of filters in the five convolutional layers are 64, 64, 128, 256, and 64, respectively. The kernel size is 3 in the first four layers and 1 in the last layer. Rectified linear units and dropout are applied after each convolutional layer to prevent overfitting. After all convolutional layers, features of dimension 64*9*9 are obtained. These features are input into a max-pooling layer with a kernel size of 2 to compress and remove redundant information, reducing computational cost. The features are then flattened and input into a fully connected layer of size 512, resulting in a total of 6*512 outputs. These outputs are combined in the input order and input into the long-short-term time series module to further obtain temporal information. The long and short-term time series module consists of convolutional layers, recurrent layers, recurrent-skip layers, temporal attention layers, and autoregressive layers. The convolutional layers have 128 filters and a kernel size of 1, used to extract the relationship between features and variables, which is then provided to the recurrent and recurrent-skip layers to capture long-term dependencies in the sequence and address the vanishing gradient problem. The calculations for each recurrent unit at time t are as follows:

[0045]

[0046] Where ⊙ represents bitwise multiplication, σ represents the sigmoid activation function, and x t Let represent the input of this layer at time t, and p represent the number of hidden units skipped. In the recurrent layer, p is 1, indicating that no units are skipped. Rectified linear units are used as the activation function of the hidden layer to output the hidden state at each time interval. The hyperparameters of the recurrent-skip layer are adjusted through an attention mechanism, and their calculation formula is as follows:

[0047]

[0048] Where, α t H represents the attention weight at time t. t R Here, represents the hidden features in the recurrent layer, and AttnScore represents the similarity function used to calculate attention. The outputs of the recurrent and recurrent-skip layers are combined and input into the fully connected layer. The output of the fully connected layer is then combined with the output of the autoregressive layer, and the final classification result is obtained through the SOFTMAX activation function. During model training, cross-entropy is used to calculate the loss, and its formula is as follows:

[0049]

[0050] Where x represents the input sample, y represents the expected final output, a represents the actual output of the neuron, and n represents the total number of input samples. The Adam optimization algorithm is used to optimize the model to effectively handle a large number of parameters and solve non-convex optimization problems. The learning rate, epochs, and batch size are set to 0.001, 100, and 128, respectively, for training the model.

[0051] Example 2

[0052] like Figure 4 The diagram shown illustrates the basic logical structure of an EEG emotion recognition system based on multi-feature fusion and CLSTN, as provided in this invention. For ease of explanation, only the parts relevant to the embodiments of this invention are shown. The functional modules / units in this system can be hardware modules / units or software modules / units, and mainly include:

[0053] (1) Emotional EEG data acquisition module, used to acquire and save emotional EEG data;

[0054] (2) Data preprocessing module, which preprocesses the collected data for subsequent feature extraction and fusion;

[0055] (3) Multi-feature fusion module, which extracts features from the preprocessed data for subsequent model input;

[0056] (4) CLSTN model building and training module: The model is built by the convolution module and the long and short-term time series module. The extracted features are used to train and save the model for subsequent emotion recognition.

[0057] (5) Emotion recognition module, used for the recognition and display of EEG emotions.

[0058] Example 3

[0059] A computer-readable storage medium is characterized in that it stores a program thereon, which, when executed by a processor, implements the EEG emotion recognition method based on multi-feature fusion and CLSTN as described in Embodiment 1.

[0060] Example 4

[0061] A computing device includes a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method described in Embodiment 1.

[0062] In the embodiments of this invention, the modules can be integrated into one unit, deployed separately, or further divided into multiple sub-modules. The modules can be distributed within the system of the embodiments as described, or they can be modified to reside in one or more systems different from those of the embodiments of this invention.

[0063] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, embodiments of the present invention are not limited to any particular combination of hardware and software.

[0064] The above-disclosed embodiments are merely specific examples of the present invention, but the present invention is not limited thereto. Those skilled in the art can make various modifications and variations to the present invention without departing from its spirit and scope. Obviously, all such modifications and variations should fall within the protection scope claimed by the present invention.

Claims

1. An electroencephalogram emotion recognition method based on multi-feature fusion and CLSTN, characterized in that, The method comprises the following steps: (1) collecting emotional electroencephalogram data; The user watches a video clip with emotions, records the corresponding labels according to three different emotional states, and collects the electroencephalogram data of the emotional related channels according to the international standard 10-20 system electrode position; (2) preprocessing data; The recorded electroencephalogram signal is preprocessed, the power frequency interference is removed through a notch filter, the noise is removed through a 0.5-45Hz band-pass filter, the data amount and calculation amount are reduced through downsampling, the average electrode potential is removed through re-reference, and the electrooculogram and electromyogram noise are removed through independent component analysis; (3) multi-feature fusion; The power spectrum density, differential entropy and wavelet packet decomposition features of the preprocessed electroencephalogram signal are extracted and fused to obtain the final features, which are input into the CLSTN model; (4) building and training the CLSTN model; The extracted features are learned through the combination of convolution and long short-term time sequence modules, the parameters are adjusted through the attention mechanism, the classification is performed through the SOFTMAX function, the optimization is performed through the Adam algorithm, and the CLSTN model for electroencephalogram emotion recognition is built and trained; specifically comprising: The CLSTN model for electroencephalogram emotion recognition is built and trained through the combination of convolution and long short-term time sequence modules; the 6 features are input into 6 convolution modules to extract frequency domain and spatial domain information, the 6 outputs obtained in sequence are combined, the time domain information is extracted through input into the long short-term time sequence module, and the final classification is performed through the SOFTMAX activation function; the convolution module comprises five different convolution layers, a maximum pooling layer, a flattening layer and a fully connected layer, the long short-term time sequence module comprises a convolution layer, a recurrent layer, a recurrent-skip layer, a time attention layer and an autoregressive layer, the hyperparameters of the recurrent-skip layer are adjusted through the attention mechanism, the outputs of the recurrent layer and the recurrent-skip layer are combined and input into the fully connected layer, the output of the fully connected layer is combined with the output of the autoregressive layer, the final classification result is obtained through the SOFTMAX activation function, the cross entropy is used as the loss calculation, the Adam optimization algorithm is used, and the learning rate, epoch and batch size are set to 0.001, 100 and 128 respectively to train the model. 2.The EEG emotion recognition method based on multi-feature fusion and CLSTN according to claim 1, characterized in that, The power spectrum density is calculated by using the discrete Fourier method, and the calculation formula is: PSD = {∑ [abs(DFT i )] 2} / (EndPos - StartPos) (1) where DFT i represents the signal value corresponding to the ith point obtained by the discrete Fourier transform, EndPos and StartPos represent the end point position and the start point position of a frequency band respectively; the square of the modulus of the signal obtained after the discrete Fourier transform is taken to obtain the power spectrum density of the frequency, and the average of all points in the entire frequency band is taken to finally obtain the average power spectrum density of the frequency band as the feature; The differential entropy feature is calculated by frequency band, and the calculation formula is: Wherein, x represents continuous information, p(x) represents the probability density function, and [a, b] represents the value interval; The wavelet packet decomposition feature is calculated by frequency band, and the calculation formula is: where C n,k k denotes the decomposition level of the wavelet th approximation component; D j,k k denotes the decomposition level th detail part; ψ(t) denotes the wavelet function. 3.The EEG emotion recognition method based on multi-feature fusion and CLSTN according to claim 1, characterized in that, In step (3) of multi-feature fusion, the multi-feature extraction and fusion comprises: The signal is divided into equal non-overlapping signal segments by a time window of 0.5 seconds, the data quantity is increased, and the time domain information is divided; the signal is filtered according to 0.5-4.5 Hz, 4-8 Hz, 8-13 Hz, 13-30 Hz and 30-42 Hz to obtain five frequency bands of delta, theta, alpha, beta and gamma; the power spectral density is calculated by using the discrete Fourier method, the differential entropy features and the wavelet packet decomposition features are calculated by the frequency bands, and the spatial information is obtained by mapping the electrode positions related to each brain region to a two-dimensional matrix; the three features corresponding to each electrode are fused and placed in the two-dimensional matrix to obtain the final features. 4.The EEG emotion recognition method based on multi-feature fusion and CLSTN according to claim 1, characterized in that, The long short-term time sequence module is: each cycle unit at time t is calculated as: where, ⊙ represents bitwise multiplication operation, σ represents sigmoid activation function, x t represents the input of the layer at time t, p represents the number of skipped hidden units, p is 1 in the recurrent layer, which means that the unit is not skipped; at the same time, the rectified linear unit is used as the hidden layer activation function to output the hidden state of each time period, and the hyperparameters of the recurrent-skip layer are adjusted through the attention mechanism, and the calculation formula is: where, α t denotes the attention weight at time t, denotes the hidden features in the recurrent layer, AttnScore denotes the similarity function for calculating attention, the output of the recurrent layer and the recurrent-skip layer are combined and input to the fully connected layer, the output of the fully connected layer is combined with the output of the autoregressive layer, and the final classification result is obtained through the SOFTMAX activation function; in the model training stage, the cross entropy is used to calculate the loss, and the calculation formula is: Wherein, x represents the input sample, y represents the final expected output, a represents the actual neuron output, and n represents the total number of input samples.

5. A system for electroencephalogram emotion recognition based on multi-feature fusion and CLSTN, characterized in that It comprises: An emotional electroencephalogram data acquisition module for acquiring and saving emotional electroencephalogram data; A data preprocessing module for preprocessing the collected data for subsequent feature extraction and fusion; A multi-feature fusion module for extracting features from the preprocessed data for subsequent model input; A CLSTN model building and training module for building a model through a convolution module and a long short-term time sequence module, training and saving the model using the extracted features, and subsequent emotional recognition; specifically comprising: By combining the convolution and long short-term time sequence modules, a CLSTN model for electroencephalogram emotion recognition is built and trained; by inputting the six features into the six convolution modules to extract frequency domain and spatial domain information, the six outputs obtained by sequential combination are input into the long short-term time sequence module to extract time domain information, and the final classification is performed through the SOFTMAX activation function; the convolution module comprises five different convolution layers, a maximum pooling layer, a flattening layer and a fully connected layer, the long short-term time sequence module comprises a convolution layer, a cycle layer, a cycle-skip layer, a time attention layer and an autoregressive layer, the hyperparameters of the cycle-skip layer are adjusted through the attention mechanism, the outputs of the cycle layer and the cycle-skip layer are combined and input into the fully connected layer, the output of the fully connected layer is combined with the output of the autoregressive layer, the final classification result is obtained through the SOFTMAX activation function, the cross-entropy is used as the loss calculation, the Adam optimization algorithm is used, and the learning rate, epoch and batch size are set to 0.001, 100 and 128 respectively to train the model; An emotional recognition module for recognizing electroencephalogram emotions and displaying the results.

6. A computer readable storage medium characterized by, A program is stored thereon, which, when executed by a processor, implements the electroencephalogram emotion recognition method based on multi-feature fusion and CLSTN of any one of claims 1-4.

7. A computing device comprising a memory and a processor, wherein, The memory stores executable code, and the processor executes the executable code to implement the method of any one of claims 1-4.

Images