EEG emotion recognition method and system

Abstract

The present invention provides a method and system for EEG emotion recognition, comprising: determining EEG signals of M channels of an EEG emotion entity to be identified; each channel corresponds to an EEG signal measurement location; inputting the EEG signals of the M channels into a pre-trained EEG emotion recognition network model to identify the corresponding EEG emotion; using a fourth-order Butterworth bandpass filter to filter the M channels of EEG signals into five sub-bands, and selecting EEG signals of C channels with high correlation with EEG emotions; using a multi-channel parallel convolutional neural network to extract combined sequence features corresponding to channel features and time features of the pre-processed EEG signals in different frequency bands; using an attention network to fuse the combined sequence features of different frequency bands to obtain fused features; using a feature extraction network to extract deep features of the fused features; and using a classification network to classify the deep features to identify the corresponding EEG emotion. The present invention has a high accuracy rate in EEG emotion recognition.

Description

Technical Field

[0001] The present invention belongs to the field of EEG emotion recognition, and more specifically, relates to an EEG emotion recognition method and system. Background Art

[0002] Emotions can reflect people's true psychological reactions to things, so they have broad applications in medical care, education and other related fields. At present, research related to emotion recognition can be roughly divided into two types: one is based on a variety of manifestations of external behavior (voice, text, images, etc.), and obtains the emotional state of the subject in a non-contact manner. The other is based on neurophysiological state, that is, the collection of multiple physiological signals (electrocardiogram (ECG), electroencephalogram (EEG), heart rate (PPG)). However, there are limitations in studying emotions based on external behavior, and it is more objective to focus on the way that neurophysiological state represents emotions. Therefore, the present invention uses EEG physiological signals to further study emotion recognition.

[0003] Many researchers have created their own EEG signal datasets to study the six basic human emotions, such as the SEED dataset and the DEAP dataset. However, few have focused on emotions during learning. In recent years, research on learning emotions has generally focused on various manifestations of outward behavior. SharmaP et al. studied student engagement in online learning scenarios by combining eye, head, and facial muscle movements. However, in real-life learning scenarios, students' normal emotions are often characterized by small amplitude and short duration of facial muscles, making it difficult to capture facial expressions. Furthermore, facial expressions can be disguised, making it difficult to truly reflect student emotions through facial expression research.

[0004] In summary, although the existing EEG signal emotion classification technology has achieved good recognition results, the following problems still exist: (1) The exploration of multiple channel combination recognition in frequency bands fails to combine the characteristics of the five sub-bands well; (2) Exploring the correlation between frequency bands and comprehensive full-channel research is an important research trend. However, not all brain areas of EEG signals contain effective emotional information, and this method fails to focus on capturing important emotional channels. Summary of the Invention

[0005] In view of the defects of the existing technology, the purpose of the present invention is to provide an EEG emotion recognition method and system, aiming to solve the problem that the existing EEG emotion recognition method fails to focus on capturing important emotion channels and cannot combine different sub-band features, resulting in low EEG emotion recognition accuracy.

[0006] To achieve the above objectives, in a first aspect, the present invention provides an EEG emotion recognition method, comprising the following steps:

[0007] Determine M channels of EEG signals of the EEG emotion entity to be identified; each channel corresponds to an EEG signal measurement position;

[0008] The EEG signals of the M channels are input into a pre-trained EEG emotion recognition network model to identify the corresponding EEG emotions; the EEG emotion recognition network model includes: a fourth-order Butterworth bandpass filter, a multi-channel parallel convolutional neural network, an attention network, a feature extraction network and a classification network; the fourth-order Butterworth bandpass filter is used to filter the EEG signals of the M channels into five sub-bands, and select the EEG signals of C channels with high correlation with EEG emotions from the EEG signals of the M channels, and use the five sub-bands corresponding to the EEG signals of the C channels as the is a preprocessed EEG signal; the multi-channel parallel convolutional neural network is used to extract the combined sequence features corresponding to the channel features and time features of the preprocessed EEG signal in different frequency bands; the attention network is used to fuse the combined sequence features of different frequency bands to obtain fused features; the feature extraction network is used to extract the deep features of the fused features; the classification network is used to classify the deep features to identify the corresponding EEG emotions; the five sub-bands correspond to δ waves, θ waves, α waves, β waves and γ waves respectively; M and C are both positive integers, and C is less than or equal to M.

[0009] In an optional example, the C channels are located in the temporal lobe of the person being measured.

[0010] In an optional example, the multi-channel parallel convolutional neural network is used to extract the combined sequence features of the preprocessed EEG signal in different frequency bands, specifically:

[0011] The EEG sequence X f C Input multi-channel parallel convolutional neural network to extract the combined sequence features containing channel features and time features of different frequency bands to obtain features

[0012]

[0013]

[0014] in, is the combined sequence feature of the multi-channel parallel convolutional neural network output in the f frequency band under the C channel combination; F C Represents the set of 5 sub-band features extracted by the multi-channel parallel convolutional neural network under the combination of C channels; ReLU represents the nonlinear excitation function, which increases the nonlinear relationship between network layers; h*g represents the input EEG sequence X f C Perform convolution; Represents the EEG signal of frequency f under the combination of C channels.

[0015] In an optional example, the attention network is used to fuse the combined sequence features of different frequency bands to obtain fused features, specifically:

[0016] From F c The features of n frequency bands are selected and put into the attention network to fuse the combined sequence features of the selected n frequency bands and output the feature F' after fusion of channel, time series and frequency band:

[0017] Weight k =Sigmoid(q T Mult(Select(F C ) ×n ))

[0018] F'=Mult(Select(F C ) ×n )*Weight k

[0019] Among them, Select means from F C n frequency band combinations are selected from the set; Mult is the multiplication of the frequency band combinations; Sigmoid is the activation function, which maps the output to between 0 and 1 and is used as the threshold value of the output weight; q T Similarity calculation; Weight k is the self-attention weight of the selected n frequency bands; F' is the fused feature output by the attention network; n is less than or equal to 5 and is a positive integer.

[0020] In an optional example, the method further includes the following steps:

[0021] A library of video materials containing three different learning emotions is predetermined; the three learning emotions are: engaged learning emotion, neutral learning emotion, or bored learning emotion;

[0022] Determine a set of to-be-trained EEG signals corresponding to the video material library; the to-be-trained EEG signal set includes: EEG signals corresponding to engaged learning emotions, EEG signals corresponding to neutral learning emotions, and EEG signals corresponding to bored learning emotions; the engaged learning emotion EEG signals are acquired by the learner watching video materials that trigger engaged learning emotions, the neutral learning emotions are acquired by the learner watching video materials that trigger neutral learning emotions, and the bored learning emotions are acquired by the learner watching video materials that trigger bored learning emotions;

[0023] The EEG signal set to be trained is used to train the EEG emotion recognition network model to obtain a trained EEG emotion recognition network model.

[0024] In a second aspect, the present invention provides an EEG emotion recognition system, comprising:

[0025] An EEG signal determination unit, configured to determine EEG signals of M channels of an EEG emotion entity to be identified; each channel corresponds to an EEG signal measurement position;

[0026] The EEG emotion recognition unit is used to input the EEG signals of the M channels into a pre-trained EEG emotion recognition network model to identify the corresponding EEG emotions; the EEG emotion recognition network model includes: a fourth-order Butterworth bandpass filter, a multi-channel parallel convolutional neural network, an attention network, a feature extraction network and a classification network; the fourth-order Butterworth bandpass filter is used to filter the EEG signals of the M channels into five sub-bands, and select the EEG signals of C channels with high correlation with EEG emotions from the EEG signals of the M channels, and the five sub-bands corresponding to the EEG signals of the C channels are selected. sub-bands as preprocessed EEG signals; the multi-channel parallel convolutional neural network is used to extract the combined sequence features corresponding to the channel features and time features of the preprocessed EEG signals in different frequency bands; the attention network is used to fuse the combined sequence features of different frequency bands to obtain fused features; the feature extraction network is used to extract the deep features of the fused features; the classification network is used to classify the deep features to identify the corresponding EEG emotions; the five sub-bands correspond to δ waves, θ waves, α waves, β waves and γ waves respectively; M and C are both positive integers, and C is less than or equal to M.

[0027] In an optional example, the C channels are located in the temporal lobe of the person being measured.

[0028] In an optional example, the multi-channel parallel convolutional neural network is used to extract the combined sequence features of the preprocessed EEG signal in different frequency bands, specifically:

[0029] The EEG sequence X f C Input multi-channel parallel convolutional neural network to extract the combined sequence features containing channel features and time features of different frequency bands to obtain features

[0030]

[0031]

[0032] in, is the combined sequence feature of the multi-channel parallel convolutional neural network output in the f frequency band under the C channel combination; F C Represents the set of 5 sub-band features extracted by the multi-channel parallel convolutional neural network under the combination of C channels; ReLU represents the nonlinear excitation function, which increases the nonlinear relationship between network layers; h*g represents the input EEG sequence Xf C Perform convolution; Represents the EEG signal of frequency f under the combination of C channels.

[0033] In an optional example, the attention network is used to fuse the combined sequence features of different frequency bands to obtain fused features, specifically:

[0034] From F C The features of n frequency bands are selected and put into the attention network to fuse the combined sequence features of the selected n frequency bands and output the feature F' after fusion of channel, time series and frequency band:

[0035] Weight k =Sigmoid(q T Mult(Select(F C ) ×n ))

[0036] F'=Mult(Select(F C ) ×n )*Weight k

[0037] Among them, Select means from F C n frequency band combinations are selected from the set; Mult is the multiplication of the frequency band combinations; Sigmoid is the activation function, which maps the output to between 0 and 1 and is used as the threshold value of the output weight; q T Similarity calculation; Weight k is the self-attention weight of the selected n frequency bands; F' is the fused feature output by the attention network; n is less than or equal to 5 and is a positive integer.

[0038] In an optional example, the system further includes:

[0039] A model training unit is used to predetermine a library of video materials containing three types of video materials that trigger different learning emotions; the three learning emotions are: engaged learning emotion, neutral learning emotion or bored learning emotion; determine a set of to-be-trained EEG signals corresponding to the video material library; the set of to-be-trained EEG signals includes: EEG signals corresponding to engaged learning emotion, EEG signals corresponding to neutral learning emotion and EEG signals corresponding to bored learning emotion; the engaged learning emotion EEG signals are acquired by learners watching video materials that trigger engaged learning emotion, the neutral learning emotion is acquired by learners watching video materials that trigger neutral learning emotion, and the bored learning emotion is acquired by learners watching video materials that trigger bored learning emotion; and the EEG signal set to be trained is used to train an EEG emotion recognition network model to obtain a trained EEG emotion recognition network model.

[0040] In general, the above technical solutions conceived by the present invention have the following beneficial effects compared with the prior art:

[0041] This invention provides an EEG emotion recognition method and system. This method uses learning video clips as emotional stimuli, establishes a library of 28 videos representing three different learning emotions, and uses a 32-channel EEG measurement device to collect EEG signals from learners while they watch the learning videos. The EEG signals are filtered for noise and artifacts, then fed into a multi-channel parallel convolutional neural network to extract channel and temporal features. This is then fed into an attention network to extract inter-band combination features, and finally into a classification network to classify and identify the learner's three emotions. This method embeds an attention module into the convolutional neural network, effectively improving the accuracy of EEG signal emotion recognition.

[0042] The present invention provides an EEG emotion recognition method and system. By focusing on the relationship between frequency bands, channels and time series features, an EEG emotion classification network based on attention fusion (ECN-AF) of multi-channel frequency band features is proposed. Experiments on the SEED dataset confirm that ECN-AF has better classification accuracy than the baseline model. An induction experiment of an online learning scenario was designed. An LE-EEG dataset including three learning emotions, boredom, engagement and neutrality, was established. EEG signals of 45 subjects were collected. The cross-dataset verification of ECN-AF proves its good robustness. It not only performs well on the public data SEED, but also has obvious advantages on the self-built LE-EEG dataset. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a flow chart of the EEG emotion recognition method provided by an embodiment of the present invention;

[0044] Figure 2 This is a technical flow chart of the EEG emotion recognition method combining multi-channel frequency band feature attention fusion provided by an embodiment of the present invention;

[0045] Figure 3 A statistical graph of subjective ratings of learning video clips provided by an embodiment of the present invention;

[0046] Figure 4 A visual display diagram of the statistical analysis of subjective scores of learning video clips provided by an embodiment of the present invention;

[0047] Figure 5 This is a diagram of the overall model structure provided by an embodiment of the present invention;

[0048] Figure 6 This is a graph of the validation set accuracy during the training process of the three models provided in the embodiment of the present invention based on the LE-EEG dataset.

[0049] Figure 7 This is an architecture diagram of the EEG emotion recognition system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0051] The present invention conducts research and builds a self-built EEG learner emotion dataset for learning emotion classification. D'MeLEO points out that although the six basic emotions are common in life, most of them do not exist in learning time of up to 30 minutes to 2 hours. Therefore, six learning emotions are defined, including boredom, engagement, confusion, frustration, happiness, and surprise, and they are arranged in ascending order according to their persistence on the time scale: (joy = surprise) < (confusion = frustration) < (boredom = engagement\concentration). The present invention considers the time scale and the probability of emotion occurrence, and studies two emotions with longer time scales, namely boredom and engagement. A neutral emotional state is added, for a total of three learning emotional states. The present invention attempts to complete learning emotion classification based on EEG signals. When the teaching content is knowledge that students are interested in, can deeply understand and accept, learners will use smiles, nodding, etc. to express their positive learning emotions. This invention describes the learner's emotional state at this time as engaged emotions; when the teaching content is knowledge that students are not interested in or cannot understand, learners will use looking around, sighing, etc. to express their negative learning emotions. This invention describes the learner's emotional state at this time as boredom; and most of the time, learners are in an understandable neutral state, which we define as neutral emotions.

[0052] Currently, deep learning is being used to extract EEG signal features for emotion classification, primarily focusing on two approaches: exploring the effectiveness of multiple channel combinations for emotion recognition within frequency bands, and exploring the effectiveness of integrating full-channel features using frequency band correlations. For this purpose, support vector machines (SVMs), k-NNs, convolutional neural networks (CNNs), and graph neural networks are commonly used. Graph neural networks use graphs to model multi-channel EEG features and learn the intrinsic relationships between different EEG channels. For exploring frequency band correlations for emotion recognition, four-dimensional convolutional recurrent neural networks, transferable attention neural networks, and three-dimensional attention dense networks are commonly used. A four-dimensional convolutional recurrent neural network converts all EEG channels into a two-dimensional image, then overlays all sub-bands to convert the features into three dimensions. A 2D CNN is used to extract channel and frequency band features, and an LSTM is used to extract temporal features. The transferable attention neural network uses two directed RNN modules to extract features across the entire brain region, and a global attention layer fuses these features to highlight key brain regions for emotion classification. The three-dimensional attention dense network uses an attention module to combine important spatial and temporal features across five sub-bands for emotion classification.

[0053] The present invention provides an EEG emotion recognition method and system that combines multi-channel frequency band feature attention fusion. The method comprises: selecting knowledge point video clips and establishing a video material library that triggers learners to produce three different learning emotions; using a 32-channel EEG cap to collect EEG signals while learners watch the learning videos; preliminarily filtering the EEG signals to remove noise and artifact interference; splitting the continuously collected EEG signals into multiple original segments by windowing, filtering each segment into five sub-bands using a fourth-order Butterworth bandpass filter; normalizing the EEG signals of the five sub-bands and performing channel selection to obtain pre-processed EEG signals; inputting the pre-processed EEG signals into a multi-channel parallel convolutional neural network to extract channel and time features to obtain five sub-band features, then inputting the five sub-band features into an attention network to extract inter-band combined features; and inputting the inter-band combined features into a classification network to classify and recognize the learner's three emotions. The present invention embeds the attention module into the convolutional neural network, improving the accuracy of learning EEG emotion recognition.

[0054] In response to the shortcomings of the existing technology, the purpose of the present invention is to provide an EEG emotion recognition method and system that combines multi-channel frequency band feature attention fusion, aiming to focus on the learner's emotions during the learning process, considering specific channels and partial frequency bands, and using the impact of the fusion of attention networks on emotion classification.

[0055] To achieve the above objectives, the present invention provides an EEG emotion recognition method combining multi-channel frequency band feature attention fusion, comprising the following steps:

[0056] Select video clips of knowledge points and build a video material library that triggers learners to have three different learning emotions;

[0057] A 32-channel EEG cap collects EEG signals while learners watch learning videos;

[0058] The EEG signal is initially filtered to remove noise and artifact interference;

[0059] The continuously collected EEG signal is split into multiple sub-segments by windowing, and the signal is filtered into five sub-bands using a fourth-order Butterworth bandpass filter. The EEG signals of the five sub-bands are normalized and channel selected to obtain the preprocessed EEG signal.

[0060] The preprocessed EEG signal is input into a multi-channel parallel convolutional neural network to extract channel and time features, and obtain five sub-band features;

[0061] The five sub-band features are input into the attention network to extract the combined features between the bands;

[0062] The inter-band combined features are input into the classification network to classify and identify the three emotions of the learner.

[0063] As described above, we selected video clips of knowledge points and established a video material library that triggers learners to generate three different learning emotions; specifically:

[0064] Course clips were selected from the Chinese University MOOC website and Bilibili website based on whether the course evaluations contained positive or negative words such as engagement, concentration, and boredom. 49 computer science graduate students were recruited to participate in the evaluation, and a library of video materials was obtained to trigger three different learning emotions in learners.

[0065] As mentioned above, the 32-channel EEG cap is used to collect EEG signals while the learners are watching the learning video. Specifically:

[0066] The 32-channel EEG measurement device EPOC Flex Saline Sensor Kit was used to collect EEG data during the learning process. The reference electrodes were the left and right earlobe electrodes of the subjects, and the sampling frequency was 128 Hz.

[0067] The EEG signal is initially filtered to remove noise and artifacts as described above, specifically:

[0068] MATLAB R2020b and the eeglab toolbox, ICLab, and adjust plug-in were used to perform preprocessing on the EEG signals, including bandpass filtering and automatic artifact processing. After using the automatic toolkit to process the artifacts, some bad data were manually deleted by visual inspection, and finally relatively clean EEG data were obtained.

[0069] The pre-processed EEG signal is input into a multi-channel parallel convolutional neural network to extract channel and time features, and obtain five sub-band features; specifically:

[0070] The features extracted by the multi-channel parallel convolutional neural network are sub-band channel features extracted after the pre-processed EEG data passes through two layers of convolutional neural networks, an average pooling layer, and a normalization layer;

[0071] The sub-band features described above are input into the attention network to extract the inter-band combination features. Specifically, the attention network calculates the attention weights, assigns the weights to the band features, obtains the band attention feature vector, and outputs a new attention feature.

[0072] Figure 1 is a flow chart of the EEG emotion recognition method provided by an embodiment of the present invention; Figure 1 As shown, the following steps are included:

[0073] S101, determining M channels of EEG signals of an EEG emotion entity to be identified; each channel corresponds to an EEG signal measurement position;

[0074] S102, inputting the EEG signals of the M channels into a pre-trained EEG emotion recognition network model to identify the corresponding EEG emotions; the EEG emotion recognition network model includes: a fourth-order Butterworth bandpass filter, a multi-channel parallel convolutional neural network, an attention network, a feature extraction network and a classification network; the fourth-order Butterworth bandpass filter is used to filter the EEG signals of the M channels into five sub-bands, and select the EEG signals of C channels with high correlation with EEG emotions from the EEG signals of the M channels, and the five sub-bands corresponding to the EEG signals of the C channels are selected. The multi-channel parallel convolutional neural network is used to extract the combined sequence features corresponding to the channel features and time features of the preprocessed EEG signal in different frequency bands; the attention network is used to fuse the combined sequence features of different frequency bands to obtain fused features; the feature extraction network is used to extract the deep features of the fused features; the classification network is used to classify the deep features to identify the corresponding EEG emotions; the five sub-bands correspond to δ waves, θ waves, α waves, β waves and γ waves respectively; M and C are both positive integers, and C is less than or equal to M.

[0075] In an optional example, the C channels are located in the temporal lobe of the person being measured.

[0076] In an optional example, the method further includes the following steps:

[0077] A library of video materials containing three different learning emotions is predetermined; the three learning emotions are: engaged learning emotion, neutral learning emotion, or bored learning emotion;

[0078] The EEG signals of learners were collected when they watched video materials that triggered different types of learning emotions. The EEG emotions corresponding to watching video materials that triggered engaged learning emotions were engaged learning emotions, the EEG emotions corresponding to watching video materials that triggered neutral learning emotions were neutral learning emotions, and the EEG emotions corresponding to watching video materials that triggered boredom learning emotions were boredom learning emotions.

[0079] The collected EEG signals are used to train the EEG emotion recognition network model to obtain a trained EEG emotion recognition network model.

[0080] Specifically, the present invention provides a method for EEG emotion recognition based on multi-channel frequency band feature attention fusion. The following is a detailed technical solution: Figure 2 : is a technical flow chart of the EEG emotion recognition method based on multi-channel frequency band feature attention fusion provided by an embodiment of the present invention; Figure 2 As shown, the following steps are included:

[0081] S1. Select video clips of key learning points and build a video library of three different learning emotions. The video clips of key learning points were sourced from the Chinese University MOOC website and Bilibili. Course clips were selected based on whether they contained positive or negative words such as engagement, focus, and boredom in course reviews. Videos were initially screened based on the educational background of computer science graduate students.

[0082] Based on the above approach, 50 course clips were collected, of which a preliminary determination was made that the video material library contained three different learning emotions: 18 videos that could trigger learners' engaged learning emotions, 17 videos that could trigger learners' neutral learning emotions, and 15 videos that could trigger learners' boredom learning emotions.

[0083] The 50 collected course segments were subjectively evaluated. 49 graduate students were recruited to participate in the evaluation. Each graduate student was given a "Self-Assessment of Learning Status" questionnaire. The questions in the questionnaire used a 5-point scale. A total of 44 valid questionnaires were collected.

[0084] The data from 44 valid questionnaires were imported into the statistical software SPSS27.0, and the data were statistically analyzed using descriptive statistics, reliability analysis, and variance analysis. Figure 3 This is a subjective scoring statistical graph provided in an embodiment of the present invention. Figure 3 The 28 video clips with high consistency in the subjective ratings given by the subjects are displayed. Figure 4This is a visualization of the subjective scoring statistical analysis provided in an embodiment of the present invention, showing the dispersion of the five-point ratings and the average score of the emotional segments from 44 valid questionnaires. A video library containing three different learning emotions was obtained: 14 videos designed to trigger learners' engaged learning emotions, 6 videos designed to trigger learners' neutral learning emotions, and 8 videos designed to trigger learners' boredom.

[0085] S2. A 32-channel EEG cap collects EEG signals while learners watch learning videos. Seven videos designed to trigger learners' emotional engagement, six videos designed to trigger learners' neutral learning emotions, and seven videos designed to trigger learners' boredom were selected from the video material library obtained in S1 for EEG acquisition experiments. The present invention recruited 47 computer science graduate students to participate in the EEG experiment. The data was collected using EEG equipment produced by EMOTIV, using the EPOC Flex Saline Sensor Kit as the hardware device and EmotivPRO v2.0 as the software device. Wireless transmission communication was used, and the subjects applied conductive gel to achieve 1 hour and 30 minutes of raw EEG signal measurement of learning emotions.

[0086] S3. Initial EEG signal filtering was performed to remove noise and artifacts, generating an input dataset. The raw EEG signals collected in S2 were imported into MATLAB R2020b and processed using the eeglab toolbox, ICLab, and the adjust plugin. The channels were first located and a 49-51 Hz notch filter was used to remove power frequency interference. After 0.1 Hz high-pass filtering and 40 Hz low-pass filtering, independent principal component analysis (ICA) was performed to remove blinking and muscle tension components, and bad segments were visually removed. Finally, relatively clean EEG data was obtained, referred to as the Learner Emotion (LE-EEG) dataset, which contains EEG signals for three learning emotions: boredom, engagement, and neutrality.

[0087] S4. Figure 5 : is a diagram showing the overall model structure of the EEG emotion recognition network combined with multi-channel frequency band feature attention fusion (ECN-AF) provided by an embodiment of the present invention; Figure 5 As shown, the overall model consists of three main modules:

[0088] 1) Module 1: Frequency Band Division and Channel Selection Module. In this module, the acquired EEG signal is first divided into raw segments using a sliding window with a window size of 10 seconds and a step size of 2 seconds. Secondly, the raw segments are filtered through a fourth-order Butterworth bandpass filter to extract five different frequency bands. Signal normalization is performed on each of the five frequency bands to obtain a single-band signal. Finally, the EEG sequence is generated based on the channel combination method to be input into the neural network.

[0089] 2) Module 2: Frequency Band Attention Feature Extraction Module. This module consists of a multi-channel convolutional parallel convolutional neural network and an attention network. First, the EEG sequence output by Module 1 is fed into the multi-channel convolutional parallel convolutional neural network, which extracts channel and time series features from different frequency bands. Second, the features extracted from different frequency bands are further fed into the attention network, which fuses the channel and time series features from different frequency bands and outputs a combined feature of channels, time series, and frequency bands.

[0090] 3) Module 3: Feature Deep Fusion and Classification Module. In this module, the combined features output by Module 2 are first used as input to the Feature Deep Fusion Module in Module 3. The deep network extracts the combined features output by Module 2 and outputs deep features. Subsequently, the deep features are input to the classification module to produce the final classification results.

[0091] The ECN-AF model was used to verify the emotion classification accuracy of the SEED dataset (a public EEG dataset) and the self-constructed LE-EEG dataset described in S3. The SEED dataset contains 62 channels of EEG signals from 15 subjects, with a sampling rate of 200 Hz. The self-constructed LE-EEG dataset described in S3 contains 32 channels of EEG signals from 45 subjects, with a sampling rate of 128 Hz.

[0092] In module 1, all EEG data in the SEED dataset and the LE-EEG dataset described in S3 are first split into multiple original segments with a window size of W, as follows:

[0093] W=T*C

[0094] Where W is the window size; T is the window length; C is the number of channels.

[0095] First, all EEG data in the SEED dataset and the LE-EEG dataset described in S3 are split into multiple original segments with a window size of W. The original segment W in the SEED and LE-EEG datasets is 2000*62 and 1280*32, respectively. All data are segmented using a sliding window with a window length T of 10 seconds and a step size of 2 seconds.

[0096] S={W1,W2,W3,…W i ,…W n-1 ,W n}

[0097] S represents a subject divided into a set of n original segments; W irepresents the i-th original segment; n represents the total number of samples for a subject; therefore, in the SEED dataset, the sample size S for each subject is 4896, and the total sample size for all 15 subjects is 73,440. In the LE-EEG dataset, the sample size S for each subject ranges from 1082 to 1650, and the total sample size for all 45 subjects is 60,376.

[0098] Secondly, all the original segments in S were filtered using a fourth-order Butterworth bandpass filter to filter the signals into five sub-bands: delta (1–4 Hz), theta (4–7 Hz), alpha (8–13 Hz), beta (13–30 Hz), and gamma (31–50 Hz).

[0099]

[0100]

[0101] Among them, N f is the order of the filter, that is, N f =4; W is the frequency; is the normalized cutoff frequency; f1~f2 is the passband range of the bandpass filter; H(S) is the sub-band signal after filtering by the fourth-order Butterworth bandpass filter; w is the frequency range of the sub-band.

[0102] Again, the EEG signal H(S) filtered by the bandpass filter is used as the input of the normalization layer. After normalization, the EEG data is guaranteed to have the same measurement scale, and a single-band signal is obtained.

[0103]

[0104] Among them, S f is the result of EEG signal normalization; f is one of the five sub-frequency bands; STD is the standard deviation; AVG is the average value.

[0105] Finally, the normalized EEG signal S f Channel selection is performed, and finally the EEG sequence input into the neural network is obtained. Previous studies have found that the combination of frequency channels can improve recognition performance.

[0106] The SEED dataset uses the international 10-20 system for 62 channel electrode placement, while the self-constructed LE-EEG dataset uses the same 10-20 system for 32 channel electrode placement. Electrode placement is indicated by letters and numbers: "Fp" for frontal electrodes, "F" for frontal electrodes, "AF" for frontal electrodes, "T" for temporal electrodes, "O" for occipital electrodes, "P" for parietal electrodes, "C" for central electrodes, "Z" for left and right central electrodes, "FC" for frontal-central electrodes, "FT" for frontal-temporal electrodes, "CP" for central-parietal electrodes, "PO" for parietal-occipital electrodes, and "TP" for temporal-occipital electrodes. The numbers represent the electrode sequence, with odd numbers indicating the left side of the brain and even numbers indicating the right side.

[0107] For example, Zheng et al. used a combination of six channels, "FT7", "FT8", "T7", "T8", "TP7" and "TP8", for emotion classification. In addition, Zheng et al. designed four different electrode placement patterns based on the peak characteristics of weight distribution and the asymmetry of emotion processing, and finally adopted "FT7", "T7", "TP7", "P7", "C5", "CP5", "FT8", "T8", "TP8", "P8", "C6", and "CP6". Compared with the full-channel prediction, the 12-channel combination achieved the best result of 86.65% classification accuracy. This confirms that using fewer channel combinations can achieve better experimental results than full-channel recognition. Combining the above research, we get the following settings:

[0108]

[0109] Among them, X f C is the EEG signal of frequency f under the Cth channel combination; C is the channel combination method; in the SEED dataset, C1 and C2 are taken as C1 = {"FT7", "FT8", "T7", "T8", "TP7", "TP8"} and C2 = {"FT7", "T7", "TP7", "P7", "C5", "CP5", "FT8", "T8", "TP8", "P8", "C6", "CP6"}, respectively.

[0110] In the SEED dataset, C1 and C2 are set to C1 = {"FT7", "FT8", "T7", "T8", "TP7", "TP8"} and C2 = {"FT7", "T7", "TP7", "P7", "C5", "CP5", "FT8", "T8", "TP8", "P8", "C6", "CP6"}, respectively. These channels are located in the temporal lobe, which is consistent with research on the distribution of emotion brain regions. In the SEED dataset, the C2 channel combination was ultimately selected for basic emotion EEG emotion recognition in this paper. Considering the discrepancy in the number of available EEG channels between the LE-EEG dataset and the SEED dataset (64 and 32 channels, respectively), and to verify the robustness of the algorithm, C3 was preset to be as consistent as possible with C2, thereby enabling cross-dataset validation. In the LE-EEG dataset, the present invention finally selects the C3 channel combination to learn emotional EEG emotion recognition.

[0111] In module 2, the EEG sequence X output by module 1 is first converted to f C Input multi-channel parallel convolutional neural network, which extracts the characteristics of channels and time series of different frequency bands and obtains the characteristics

[0112]

[0113]

[0114] in, is the feature output by the convolutional network in the f-band under the C-th channel combination; F C Represents the set of 5 sub-band features extracted by the convolutional network under the C-th channel combination.

[0115] Secondly, from F C Select the features of n frequency bands An attention network is put into it, which fuses the channel and time series features of the selected n frequency bands and outputs a combined feature F' containing the channel, time series and frequency band. Figure 5 is an internal structure diagram of the attention module provided by an embodiment of the present invention; Figure 5 As shown, the features enter the attention module and the feature association is calculated to obtain the self-attention weight Weight k , the obtained attention weight is used to weight the input feature and output a new attention feature vector F'. The present invention converts the feature F C As input to the attention network.

[0116] Weightk =Sigmoid(q T Mult(Select(F C ) ×n ))

[0117] F'=Mult(Select(F C ) ×n )*Weight k

[0118] Among them, Weight k is the attention weight; Select means C n frequency band combinations are selected from 5 sub-bands. In the present invention, three sub-bands are finally selected for both data sets, namely δ, β, and γ, for EEG emotion recognition. Mult is the multiplication of the frequency band combination; q T is the similarity calculation; F' is the combined feature output by the attention network.

[0119] The attention network calculates the feature weights of emotion recognition through the attention mechanism, assigns weights above the threshold to the focus area of ​​emotion recognition, and assigns weights below the threshold to the irrelevant area, so as to associate the feature information between frequency bands, eliminate irrelevant interference features, and obtain the combined feature F'.

[0120] In Module 3, Module 3 consists of a feature depth fusion module and a classification module. First, the combined feature F' output by Module 2 is used as the input to the feature depth fusion module. The feature depth fusion module is composed of a two-layer convolutional network, a pooling layer, and a normalization layer. The combined feature F' passes through these layers in turn and outputs a deep feature. Subsequently, the deep feature is input into the classification module, which is composed of a two-layer convolutional network, a global pooling layer, and a fully connected layer. The deep feature passes through these layers in turn and outputs the final classification result.

[0121] After completing the above steps, emotion classification from EEG signals was achieved. During training, the cross-entropy loss function was used, optimized using the stochastic gradient descent algorithm. Sigmoid was used as the activation function, the model learning rate was set to 0.001, and the weight decay was set to 0.0001. The learning rate was dynamically adjusted during the training process, and the optimization function was set to Adam optimization, ultimately achieving optimal results. The experiment used average accuracy (ACC) and standard deviation (STD) as evaluation metrics for emotion recognition. A higher accuracy value indicates better recognition performance.

[0122] Specifically, to verify the effectiveness of the attention network, we compared three fusion band methods: feature addition fusion, feature multiplication fusion, and attention weight fusion, which are represented as Add, Mult, and Attention in Table 1. The accuracy comparison of the attention fusion method of the present invention and other fusion methods for emotion recognition on the SEED dataset is shown in Table 1:

[0123] Table 1

[0124]

[0125] Experiments in this paper found that, first, the proposed attention network has better overall performance for the model on more frequency band combinations; however, more frequency band combinations do not guarantee higher performance in emotion classification. For example, compared with the sub-band combinations shown in other rows of Table 1, in the case of the sub-bands (δ, α, β, γ) shown in the last row of Table 1, (i) the performance of the model using the Add fusion mode decreases (see the second and fifth columns of the last row of Table 1), but remains relatively stable; (ii) the performance of the model using the Mult or Attention fusion mode (see the third and sixth columns or the fourth and seventh columns of the last row of Table 1) is severely degraded. The reasons for this may include: during model training, the fusion method of Mult and Attention causes the model training parameters to grow exponentially, resulting in severe overfitting caused by overtraining of the model.

[0126] Second, we can see that the best performance achieved by C2 (see columns 5-7 of Table 1) is always higher than that achieved by C1 (see columns 2-4 of Table 2). To illustrate, let's take the sub-band (δ, γ) as an example. From row 4 of Table 1, we can see that: (i) with respect to C1, the attention fusion method achieves the best performance of 95.63%; (ii) with respect to C2, the attention fusion method again achieves the best performance of 95.70%, that is, C2 achieves an accuracy improvement of 0.07% compared to C1.

[0127] Third, regarding C2, the best two performances were achieved using the attention fusion method for the sub-bands (α, β, γ) and (δ, β, γ), at 96.02% and 96.45%, respectively (see the second and third rows of the last column of Table 1). Taking the sub-band (δ, β, γ) as an example, the attention fusion method achieved 0.67% and 0.30% higher accuracy compared to Add and Mult. This suggests that using attention fusion can improve classification performance because the attention weights assign more important features.

[0128] Specifically, the accuracy comparison of emotion recognition on the SEED dataset using the method of the present invention and the methods proposed by other researchers is shown in Table 2:

[0129] Table 2

[0130]

[0131] Based on the above experiments, we used the δ, β, and γ frequency bands and an attention fusion approach to conduct the comparison. Our model was compared with a baseline model on the SEED dataset. Table 2 lists the results. Compared to the best baseline model (see the "RGNN" row in Table 2), our model achieved a 2.21% improvement in performance.

[0132] Specifically, the accuracy comparison of emotion recognition between the method of the present invention and other methods on the self-built LE-EEG dataset is shown in Table 3:

[0133] On the LE-EEG dataset, considering the inconsistency in the number of available EEG channels between the LE-EEG and SEED datasets (64 and 32 channels, respectively), and to verify the robustness of the algorithm, we preset C3 = {"T7", "P7", "CP5", "T8", "P8", "CP6"} to be as consistent as possible with C2, thus enabling cross-dataset validation. In the LE-EEG dataset, the C3 channel combination was ultimately selected for learning emotion EEG recognition in this paper.

[0134] For validation on the LE-EEG dataset, we selected two baseline models, 4D_CRNN and SOGNN, which can be replicated using shared code, for comparison. Table 3 lists the comparisons with the baseline models. Compared to these two baseline models, our model improved performance by 27.32% and 20.42% (see the third column of the "4D_CRNN", "SOGNN", and "Presentation (C3)" rows in Table 3), respectively), confirming the robustness of our network across different datasets. Figure 7 Figure 2 is a graph showing the validation set accuracy of three different models provided by the present invention during training based on the LE-EEG dataset. The present invention (ECN-AF model) produces better performance.

[0135] Table 3

[0136]

[0137] As shown in Tables 2 and 3, the EEG emotion recognition method based on multi-channel frequency band feature attention fusion (ECN-AF) constructed in the present invention has better accuracy than the current mainstream methods in emotion recognition on the SEED dataset and the self-built LE-EEG dataset.

[0138] Figure 7 This is a diagram of the EEG emotion recognition system architecture provided by an embodiment of the present invention. Figure 7 Shown, including:

[0139] The EEG signal determination unit 710 is used to determine the EEG signals of M channels of the EEG emotion entity to be identified; each channel corresponds to an EEG signal measurement position;

[0140] The EEG emotion recognition unit 720 is used to input the EEG signals of the M channels into a pre-trained EEG emotion recognition network model to identify the corresponding EEG emotions; the EEG emotion recognition network model includes: a fourth-order Butterworth bandpass filter, a multi-channel parallel convolutional neural network, an attention network, a feature extraction network and a classification network; the fourth-order Butterworth bandpass filter is used to filter the EEG signals of the M channels into five sub-bands, and select the EEG signals of C channels with high correlation with EEG emotions from the EEG signals of the M channels, and classify the EEG signals of the C channels into five sub-bands. The five sub-bands are used as preprocessed EEG signals; the multi-channel parallel convolutional neural network is used to extract the combined sequence features corresponding to the channel features and time features of the preprocessed EEG signals in different frequency bands; the attention network is used to fuse the combined sequence features of different frequency bands to obtain fused features; the feature extraction network is used to extract the deep features of the fused features; the classification network is used to classify the deep features to identify the corresponding EEG emotions; the five sub-bands correspond to δ waves, θ waves, α waves, β waves and γ waves respectively; M and C are both positive integers, and C is less than or equal to M.

[0141] The model training unit 730 is used to predetermine a library of video materials containing three types of video materials that trigger different learning emotions; the three learning emotions are: engaged learning emotion, neutral learning emotion or bored learning emotion; collect EEG signals of learners when watching video materials that trigger different types of learning emotions; the EEG emotion corresponding to watching video materials that trigger engaged learning emotions is engaged learning emotion, the EEG emotion corresponding to watching video materials that trigger neutral learning emotions is neutral learning emotion, and the EEG emotion corresponding to watching video materials that trigger bored learning emotions is bored learning emotion; and use the collected EEG signals to train the EEG emotion recognition network model to obtain a trained EEG emotion recognition network model.

[0142] It should be noted that Figure 7 The detailed functional implementation of each unit can be found in the introduction of the aforementioned method embodiment, which will not be repeated here.

[0143] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for EEG emotion recognition, characterized in that: The steps include: Determine M channels of EEG signals of the EEG emotion entity to be identified; each channel corresponds to an EEG signal measurement position; Inputting the M channels of EEG signals into a pre-trained EEG emotion recognition network model to identify corresponding EEG emotions; The EEG emotion recognition network model includes: a fourth-order Butterworth bandpass filter, a multi-channel parallel convolutional neural network, an attention network, a feature extraction network and a classification network; the fourth-order Butterworth bandpass filter is used to filter the EEG signals of the M channels into five sub-bands, and select the EEG signals of C channels with high correlation with EEG emotions from the EEG signals of the M channels, and use the five sub-bands corresponding to the EEG signals of the C channels as the preprocessed EEG signals; the multi-channel parallel convolutional neural network is used to extract the combined sequence features corresponding to the channel features and time features of the preprocessed EEG signals in different frequency bands; the attention network is used to fuse the combined sequence features of different frequency bands to obtain the fused features; the feature extraction network is used to extract the deep features of the fused features; the classification network is used to classify the deep features to identify the corresponding EEG emotions; the five sub-bands correspond to δ waves, θ waves, α waves, β waves and γ waves respectively; M and C are both positive integers, and C is less than or equal to M.

2. The EEG emotion recognition method according to claim 1, characterized in that: The C channels are located in the temporal lobe of the person being measured.

3. The EEG emotion recognition method according to claim 1, characterized in that: The multi-channel parallel convolutional neural network is used to extract the combined sequence features of the pre-processed EEG signals in different frequency bands, specifically: The EEG sequence X f C Input multi-channel parallel convolutional neural network to extract the combined sequence features containing channel features and time features of different frequency bands to obtain features in, is the combined sequence feature of the multi-channel parallel convolutional neural network output in the f frequency band under the C channel combination; F C Represents the set of 5 sub-band features extracted by the multi-channel parallel convolutional neural network under the combination of C channels; ReLU represents the nonlinear excitation function, which increases the nonlinear relationship between network layers; h*g represents the input EEG sequence X f C Perform convolution; Represents the EEG signal of frequency f under the combination of C channels.

4. The EEG emotion recognition method according to claim 3, characterized in that: The attention network is used to fuse the combined sequence features of different frequency bands to obtain the fused features, specifically: From F C The features of n frequency bands are selected and put into the attention network to fuse the combined sequence features of the selected n frequency bands and output the feature F' after fusion of channel, time series and frequency band: Weight k =Sigmoid(q T Mult(Select(F C ) ×n )) F’=Mult(Select(F C ) ×n )*Weight k Among them, Select means from F C n frequency band combinations are selected from the set; Mult is the multiplication of the frequency band combinations; Sigmoid is the activation function, which maps the output to between 0 and 1 and is used as the threshold value of the output weight; q T Similarity calculation; Weight k is the self-attention weight of the selected n frequency bands; F' is the fused feature output by the attention network; n is less than or equal to 5 and is a positive integer.

5. The EEG emotion recognition method according to any one of claims 1 to 4, characterized in that: The following steps are also included: A library of video materials containing three different learning emotions is predetermined; the three learning emotions are: engaged learning emotion, neutral learning emotion, or bored learning emotion; Determining a set of EEG signals to be trained corresponding to the video material library; The EEG signal set to be trained includes: EEG signals corresponding to engaged learning emotions, EEG signals corresponding to neutral learning emotions, and EEG signals corresponding to bored learning emotions; the EEG signals of engaged learning emotions are obtained by learners watching video materials that trigger engaged learning emotions, the neutral learning emotions are obtained by learners watching video materials that trigger neutral learning emotions, and the bored learning emotions are obtained by learners watching video materials that trigger bored learning emotions; The EEG signal set to be trained is used to train the EEG emotion recognition network model to obtain a trained EEG emotion recognition network model.

6. An EEG emotion recognition system, characterized in that: include: an EEG signal determination unit, configured to determine EEG signals of M channels of an EEG emotion entity to be identified; Each channel corresponds to an EEG signal measurement location; An EEG emotion recognition unit, configured to input the EEG signals of the M channels into a pre-trained EEG emotion recognition network model to recognize corresponding EEG emotions; The EEG emotion recognition network model includes: a fourth-order Butterworth bandpass filter, a multi-channel parallel convolutional neural network, an attention network, a feature extraction network and a classification network; the fourth-order Butterworth bandpass filter is used to filter the EEG signals of the M channels into five sub-bands, and select the EEG signals of C channels with high correlation with EEG emotions from the EEG signals of the M channels, and use the five sub-bands corresponding to the EEG signals of the C channels as the preprocessed EEG signals; the multi-channel parallel convolutional neural network is used to extract the combined sequence features corresponding to the channel features and time features of the preprocessed EEG signals in different frequency bands; the attention network is used to fuse the combined sequence features of different frequency bands to obtain the fused features; the feature extraction network is used to extract the deep features of the fused features; the classification network is used to classify the deep features to identify the corresponding EEG emotions; the five sub-bands correspond to δ waves, θ waves, α waves, β waves and γ waves respectively; M and C are both positive integers, and C is less than or equal to M.

7. The EEG emotion recognition system according to claim 6, characterized in that: The C channels are located in the temporal lobe of the person being measured.

8. The EEG emotion recognition system according to claim 6, characterized in that: The multi-channel parallel convolutional neural network is used to extract the combined sequence features of the pre-processed EEG signals in different frequency bands, specifically: The EEG sequence X f C Input multi-channel parallel convolutional neural network to extract the combined sequence features containing channel features and time features of different frequency bands to obtain features in, is the combined sequence feature of the multi-channel parallel convolutional neural network output in the f frequency band under the C channel combination; F C Represents the set of 5 sub-band features extracted by the multi-channel parallel convolutional neural network under the combination of C channels; ReLU represents the nonlinear excitation function, which increases the nonlinear relationship between network layers; h*g represents the input EEG sequence X f C Perform convolution; Represents the EEG signal of frequency f under the combination of C channels.

9. The EEG emotion recognition system according to claim 8, characterized in that: The attention network is used to fuse the combined sequence features of different frequency bands to obtain the fused features, specifically: From F C The features of n frequency bands are selected and put into the attention network to fuse the combined sequence features of the selected n frequency bands and output the feature F' after fusion of channel, time series and frequency band: Weight k =Sigmoid(q T Mult(Select(F C ) ×n )) F’=Mult(Select(F C ) ×n )*Weight k Among them, Select means from F C n frequency band combinations are selected from the set; Mult is the multiplication of the frequency band combinations; Sigmoid is the activation function, which maps the output to between 0 and 1 and is used as the threshold value of the output weight; q T Similarity calculation; Weight k is the self-attention weight of the selected n frequency bands; F' is the fused feature output by the attention network; n is less than or equal to 5 and is a positive integer.

10. The EEG emotion recognition system according to any one of claims 6 to 9, characterized in that: Also includes: A model training unit is used to predetermine a library of video materials containing three types of video materials that trigger different learning emotions; the three learning emotions are: engaged learning emotion, neutral learning emotion, or bored learning emotion; Determining a set of EEG signals to be trained corresponding to the video material library; The EEG signal set to be trained includes: EEG signals corresponding to engaged learning emotions, EEG signals corresponding to neutral learning emotions, and EEG signals corresponding to bored learning emotions; The engaged learning emotion EEG signal is obtained by the learner watching video materials that trigger engaged learning emotions, the neutral learning emotion is obtained by the learner watching video materials that trigger neutral learning emotions, and the bored learning emotion is obtained by the learner watching video materials that trigger bored learning emotions; and the EEG emotion recognition network model is trained using the EEG signal set to be trained to obtain a trained EEG emotion recognition network model.

Images