A learning emotion classification method and system based on electroencephalogram signals

Abstract

The application provides a learning emotion classification method and system based on electroencephalogram signals, and the method comprises the following steps: acquiring electroencephalogram signals of multiple channels of a learner; different channels correspond to different brain positions; filtering the electroencephalogram signals into multiple different frequency bands through a Butterworth band-pass filter to obtain electroencephalogram signals of multiple frequency bands of multiple channels; inputting the electroencephalogram signals of multiple frequency bands of multiple channels into a trained confusion classification model to obtain a confusion classification result of the learner; if it is judged that the confusion classification result is confusion, the confusion classification result is the learning emotion of the learner; if it is judged that the confusion classification result is not confusion, inputting the electroencephalogram signals of multiple frequency bands of multiple channels into a trained emotion classification model to obtain an emotion classification result of the learner, and the emotion classification result is the learning emotion of the learner. The application can avoid the confusion of the confusion emotion and other emotions, and improve the emotion classification accuracy.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of electroencephalogram emotion classification, and more particularly relates to a learning emotion classification method and system based on electroencephalogram signals. BACKGROUND

[0002] Emotion is a psychological process that can affect physiological behavior and physical activity. Emotion classification has a wide range of applications, such as in the medical field, where emotion classification can assess the emotional state of special patients and help doctors make more accurate diagnoses. Emotion classification also has a wide range of applications in education, such as helping educators better understand and understand the emotional state of students, so that they can dynamically adjust and improve teaching strategies and methods; helping educators better focus on students with emotional problems and taking appropriate measures to intervene in students when necessary, so as to better help students develop and grow.

[0003] Common signals for emotion classification include head posture, facial expression, text, sound, electroencephalogram signal, electrocardiogram signal, etc. Among them, electroencephalogram signals can directly reflect the changes in brain activity and objectively reflect the emotional state of a person. In addition, electroencephalogram signals have the characteristics of non-invasiveness, low cost, portability, and reliability. Therefore, electroencephalogram signals have a significant advantage in emotion classification research. There are many traditional machine learning methods for emotion recognition based on electroencephalogram signals, such as SVM and KNN, but the classification accuracy is still low. SUMMARY

[0004] In view of the defects of the prior art, the purpose of the present application is to provide a learning emotion classification method and system based on electroencephalogram signals, which aims to solve the problem of low classification accuracy of existing emotion classification methods based on electroencephalogram signals.

[0005] To achieve the above-mentioned purpose, in a first aspect, the present application provides a learning emotion classification method based on electroencephalogram signals, comprising:

[0006] S101 obtaining electroencephalogram signals of multiple channels of a learner; different channels correspond to different brain positions;

[0007] S102 filtering the electroencephalogram signals into multiple different frequency bands through a Butterworth band-pass filter to obtain electroencephalogram signals of multiple frequency bands of multiple channels;

[0008] S103 inputting the electroencephalogram signals of multiple frequency bands of multiple channels into a trained confusion classification model to obtain a confusion classification result of the learner;

[0009] S104 If it is judged that the confusion classification result is confusion, the confusion classification result is the learning emotion of the learner; if it is judged that the confusion classification result is not confusion, the electroencephalogram signals of the multiple frequency bands of the multiple channels are input into the trained emotion classification model to obtain the emotion classification result of the learner, and the emotion classification result is the learning emotion of the learner.

[0010] In an optional example, S103 further includes, before S103:

[0011] The classification layer of the initial emotion classification model is adjusted to obtain an initial confusion classification model suitable for binary classification.

[0012] The initial confusion classification model is trained based on the confusion samples and the non-confusion samples to obtain the trained confusion classification model.

[0013] The initial emotion classification model is trained based on the concentration samples, the boredom samples and the neutral samples to obtain the trained emotion classification model.

[0014] In an optional example, S103 specifically includes:

[0015] Features of multiple frequency bands under a channel combination are extracted based on the electroencephalogram signals of the multiple frequency bands of the multiple channels; the channel combination is a combination of channels with strong relevance to emotion classification.

[0016] Features of a frequency band combination under the channel combination are extracted from the features of the multiple frequency bands under the channel combination, the attention weight corresponding to the frequency band combination is generated based on the features of the frequency band combination under the channel combination, the attention weight is assigned to the corresponding features and fused to obtain fused features; the frequency band combination is a combination of frequency bands with strong relevance to emotion classification.

[0017] Confusion classification is performed based on the fused features to obtain the confusion classification result of the learner.

[0018] In an optional example, the channels with strong relevance to emotion classification are determined based on a classification accuracy obtained by performing emotion classification on the electroencephalogram signals of each channel based on a single-channel emotion classification model.

[0019] The single-channel emotion classification model includes a convolution backbone network, an attention unit and a softmax classification unit; the convolution backbone network is used to extract features based on the electroencephalogram signals of each channel, the attention unit is used to generate weights based on the features and assign the weights to the features to obtain weighted features, and the softmax classification unit is used to perform emotion classification based on the weighted features.

[0020] In an optional example, the channel combination includes two channels corresponding to the temporal lobe, two channels corresponding to the positions between the frontal lobe and the temporal lobe, one channel corresponding to the position between the central region and the parietal lobe, and one channel corresponding to the central region.

[0021] In a second aspect, the present application provides a learning emotion classification system based on electroencephalogram signals, comprising:

[0022] An electroencephalogram signal acquisition module is configured to acquire electroencephalogram signals of multiple channels of a learner; different channels correspond to different brain positions.

[0023] A frequency domain decomposition module is configured to filter the electroencephalogram signals into multiple different frequency bands through a Butterworth band-pass filter to obtain electroencephalogram signals of multiple frequency bands of multiple channels.

[0024] A confusion classification module is configured to input the electroencephalogram signals of multiple frequency bands of multiple channels into a trained confusion classification model to obtain a confusion classification result of the learner.

[0025] An emotion classification module is configured to, if it is determined that the confusion classification result is confusion, the confusion classification result is the learning emotion of the learner; if it is determined that the confusion classification result is not confusion, input the electroencephalogram signals of multiple frequency bands of multiple channels into a trained emotion classification model to obtain an emotion classification result of the learner, and the emotion classification result is the learning emotion of the learner.

[0026] In an optional example, the system further comprises a model training module configured to:

[0027] Adjust the classification layer of an initial emotion classification model to obtain an initial confusion classification model suitable for binary classification.

[0028] Train the initial confusion classification model based on confusion samples and non-confusion samples to obtain the trained confusion classification model.

[0029] Train the initial emotion classification model based on focused samples, bored samples, and neutral samples to obtain the trained emotion classification model.

[0030] In an optional example, the confusion classification module is specifically configured to:

[0031] Extract features of multiple frequency bands under a channel combination based on the electroencephalogram signals of multiple frequency bands of multiple channels; the channel combination is a combination of channels with strong relevance to emotion classification.

[0032] The features of the frequency band combination under the channel combination are extracted from the features of the multiple frequency bands under the channel combination, the attention weight corresponding to the frequency band combination is generated based on the features of the frequency band combination under the channel combination, the attention weight is assigned to the corresponding features and fused, and the fused features are obtained; the frequency band combination is a combination of frequency bands with strong relevance to emotion classification;

[0033] The confusion classification result of the learner is obtained based on the fused features.

[0034] In an optional example, the channels with strong relevance to emotion classification adopted in the confusion classification module are determined based on the classification accuracy obtained by performing emotion classification on the electroencephalogram of each channel based on a single-channel emotion classification model;

[0035] The single-channel emotion classification model comprises a convolution backbone network, an attention unit and a softmax classification unit; the convolution backbone network is used to extract features based on the electroencephalogram of each channel, the attention unit is used to generate weights based on the features and assign the weights to the features to obtain weighted features, and the softmax classification unit is used to perform emotion classification based on the weighted features.

[0036] In an optional example, the channel combination adopted in the confusion classification module comprises two channels corresponding to the temporal lobe, two channels corresponding to the positions between the frontal lobe and the temporal lobe, one channel corresponding to the positions between the central region and the parietal lobe, and one channel corresponding to the central region.

[0037] Overall, compared with the prior art, the above technical solutions conceived by the present application have the following beneficial effects:

[0038] The present application provides a learning emotion classification method and system based on electroencephalogram, which identifies the learning emotion of the learner by using the electroencephalogram of multiple frequency bands of multiple channels, understands the state of the learner, separates the confusion emotion by performing a first classification task, performs a second classification task on other emotions, and focuses on the correlation between the frequency bands and channels in both classification tasks, so as to avoid the confusion of the confusion emotion and other emotions, and obtain good classification accuracy for confusion classification and other emotion classification. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 is a flowchart of the learning emotion classification method based on electroencephalogram provided by the present application;

[0040] Figure 2 is an implementation diagram of the learning emotion classification method based on electroencephalogram provided by the present application;

[0041] Figure 3A partial confusion matrix diagram provided by the present application;

[0042] Figure 4 A confusion separation experiment flowchart provided by the present application;

[0043] Figure 5 An architecture diagram of a learning emotion classification system based on electroencephalogram signals provided by the present application. DETAILED DESCRIPTION

[0044] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0045] Confusion is one of the more common emotions in learning emotions. When students encounter problems that are difficult to solve or difficult to understand, they may have a confused emotion. This confusion may stimulate the students' desire to explore and desire to learn, but it may also lead to a loss of confidence. Studies have shown that long-term confusion can lead to frustration and boredom, and there is a strong correlation between confusion and focus\engagement, confusion and boredom. When classifying confusion and other emotions, some confusion may occur, affecting the classification accuracy.

[0046] With the wide application of deep learning, it has shown more significant effects in the field of emotion classification. Generally, features such as time domain, frequency domain, time-frequency domain or spatial domain are extracted based on electroencephalogram signals for emotion classification. In previous related research, most of the data used for emotion classification is full-channel and full-band, but not all channels and frequency bands are closely related to learning emotions. In addition, most studies only study the influence of single frequency band and full frequency band on emotion classification, or only study the influence of channels on frequency bands, ignoring the correlation between frequency bands and channels.

[0047] In summary, there are three problems in most current researches: confusion emotion is easily confused with other emotions in learning emotion classification, affecting the classification accuracy; the correlation between frequency bands and channels is ignored, and the emotion-related features are not fully extracted; the full-channel and full-band information of electroencephalogram signals is used, resulting in a large amount of time consumption.

[0048] In view of the above problems, the present application provides a learning emotion classification method based on electroencephalogram signals, Figure 1 A flowchart of a learning emotion classification method based on electroencephalogram signals provided by the present application is shown in FIG. 1. Figure 1 As shown in the figure, the method comprises the following steps:

[0049] Step S101, acquiring electroencephalogram signals of multiple channels of a learner; different channels correspond to different brain positions;

[0050] Step S102, filtering the brain electrical signals into multiple different frequency bands through a Butterworth band-pass filter to obtain brain electrical signals of multiple channels in multiple frequency bands;

[0051] Step S103, inputting the brain electrical signals of multiple channels in multiple frequency bands into the trained confusion classification model to obtain a confusion classification result of the learner;

[0052] Step S104, if it is judged that the confusion classification result is confusion, the confusion classification result is the learning emotion of the learner; if it is judged that the confusion classification result is not confusion, inputting the brain electrical signals of multiple channels in multiple frequency bands into the trained emotion classification model to obtain an emotion classification result of the learner, and the emotion classification result is the learning emotion of the learner.

[0053] It should be noted that for the common learning emotions of the learner, such as confusion, concentration, boredom, neutrality and the like, through experimental analysis, it is found that compared with the emotion recognition of the brain electrical signals corresponding to other learning emotions, the emotion recognition of the brain electrical signals obtained when watching the confusion-inducing video has a lower recognition accuracy and is more likely to be misjudged as boredom or other learning emotions, and therefore the present application adopts the strategy of first performing a first classification task to separate the confusion emotion and then performing a second classification task to classify other emotions, so as to avoid the confusion of the confusion emotion and other emotions and result in a lower model performance.

[0054] The confusion classification model can judge whether the learning emotion corresponding to the input data is confusion, which belongs to a binary classification model, and the emotion classification model can judge whether the learning emotion corresponding to the input data is concentration, boredom or neutrality, which belongs to a multi-classification model. The confusion classification model and the emotion classification model can both realize the feature extraction of different channels and different frequency bands required for the input brain electrical signals of multiple channels in multiple frequency bands, and can generate attention weights corresponding to different frequency bands for the frequency band dimension, assign the attention weights to the corresponding features and fuse them, and finally realize classification according to the fused features.

[0055] The method provided by the embodiment of the present application can recognize the learning emotion of the learner by using the brain electrical signals of multiple channels in multiple frequency bands, understand the state of the learner, first perform a first classification task to separate the confusion emotion, then perform a second classification task to classify other emotions, and pay attention to the correlation between the frequency bands and the channels in both classification tasks, so as to avoid the classification confusion of the confusion emotion and other emotions, and obtain a good classification accuracy for both the confusion classification and the other emotion classification.

[0056] Based on the above embodiment, step S103 further includes:

[0057] Adjust the classification layer of the initial emotion classification model to obtain an initial confusion classification model suitable for binary classification;

[0058] Train the initial confusion classification model based on the confusion samples and the non-confusion samples to obtain a trained confusion classification model;

[0059] Train the initial emotion classification model based on the focused samples, the boring samples and the neutral samples to obtain a trained emotion classification model.

[0060] The confusion samples are the electroencephalogram signals obtained when watching learning videos that induce confusion emotions, the non-confusion samples can include the focused samples, the boring samples and the neutral samples, the focused samples are the electroencephalogram signals obtained when watching learning videos that induce focused emotions, the boring samples are the electroencephalogram signals obtained when watching learning videos that induce boring emotions, and the neutral samples are the electroencephalogram signals obtained when watching learning videos that induce neutral emotions.

[0061] Preferably, in order to solve the problem of ignoring the correlation between frequency bands and channels, the initial emotion classification model in the present application can adopt an electroencephalogram signal emotion classification network based on attention fusion (Emotion Classification Network Based on Attention Fusion of Multi-Channel Band Features, ECN-AF).

[0062] Based on any of the above embodiments, step S103 specifically includes:

[0063] Extracting features of multiple frequency bands under channel combinations from the electroencephalogram signals of multiple frequency bands under multiple channels; the channel combinations are combinations of channels with strong relevance to emotion classification;

[0064] Extracting features of frequency band combinations under channel combinations from the features of multiple frequency bands under channel combinations, generating attention weights corresponding to the frequency band combinations based on the features of the frequency band combinations under the channel combinations, assigning the attention weights to the corresponding features and fusing them to obtain fused features; the frequency band combinations are combinations of frequency bands with strong relevance to emotion classification;

[0065] Performing confusion classification based on the fused features to obtain the confusion classification results of the learners.

[0066] Specifically, the confusion classification model includes a multi-channel convolution backbone network, a frequency band attention fusion unit and a confusion classification module;

[0067] The multi-channel convolution backbone network can extract features of multiple frequency bands of multiple channels of input electroencephalogram signals and extract the required features of multiple frequency bands under channel combinations from the channel dimension;

[0068] The frequency band attention fusion unit is aimed at the frequency band dimension, extracts the feature of the frequency band combination under the channel combination from the feature of the multiple frequency bands under the channel combination, generates the attention weight corresponding to different frequency bands according to the feature of the frequency band combination under the channel combination, assigns the attention weight to the corresponding feature, and performs feature fusion of different frequency bands under the channel combination to obtain the fused feature;

[0069] Finally, the confusion classification module performs confusion classification according to the fused feature, judges whether the emotion of the learner is confusion, and thus obtains the confusion classification result of the learner.

[0070] The method provided in the embodiment of the application can improve the classification accuracy of the model and reduce the operation time consumption of the confusion classification model by screening the frequency band combination and the channel combination with strong relevance to emotion classification and paying attention to the correlation between the frequency bands and the channels.

[0071] Based on any of the above embodiments, similarly to the confusion classification model, the emotion classification model comprises a multi-channel convolution backbone network, a frequency band attention fusion unit and an emotion classification module.

[0072] The multi-channel convolution backbone network can extract features of multiple frequency bands of multiple channels of the input electroencephalogram and extract the required features of multiple frequency bands under the channel combination from the channel dimension.

[0073] The frequency band attention fusion unit is aimed at the frequency band dimension, extracts the feature of the frequency band combination under the channel combination from the feature of the multiple frequency bands under the channel combination, generates the attention weight corresponding to different frequency bands according to the feature of the frequency band combination under the channel combination, assigns the attention weight to the corresponding feature, and performs feature fusion of different frequency bands under the channel combination to obtain the fused feature.

[0074] Finally, the emotion classification module performs emotion classification according to the fused feature, judges whether the emotion of the learner is confusion, and thus obtains the confusion classification result of the learner.

[0075] Based on any of the above embodiments, the classification accuracy of the single-channel emotion classification model is determined based on the relevance of the channel to emotion classification.

[0076] The single-channel emotion classification model comprises a single-channel convolution backbone network, an attention unit and a softmax classification unit; the single-channel convolution backbone network is used to extract features based on the electroencephalogram of each channel, the attention unit is used to generate weights based on the features and assign the weights to the features to obtain weighted features, and the softmax classification unit is used to perform emotion classification based on the weighted features.

[0077] Preferably, the single-channel emotion classification model can be improved on the basis of the ECN-AF model, and the model suitable for single channel is obtained by replacing the multi-channel convolutional backbone network and the frequency band attention fusion unit in the ECN-AF model with a single-channel convolutional backbone network and an attention unit, and adding a softmax classification unit for emotion classification.

[0078] Based on any of the above embodiments, the channel combination includes two channels corresponding to the temporal lobe, two channels corresponding to the positions between the frontal lobe and the temporal lobe, one channel corresponding to the positions between the central region and the parietal lobe, and one channel corresponding to the central region.

[0079] Preferably, 12 single channels with strong relevance to emotion classification can be found by the single-channel emotion classification model, which are T7 channel of the temporal lobe, FT9 channel between the frontal lobe and the temporal lobe, CP5 channel between the central region and the parietal lobe, FT10 channel between the frontal lobe and the temporal lobe, T8 channel of the temporal lobe, C3 channel of the central region, F7 channel of the frontal lobe, Fp1 channel of the prefrontal lobe, Fp2 channel of the prefrontal lobe, C4 channel of the central region, F3 channel of the frontal lobe, and FC6 channel between the central region and the frontal lobe. The channel combination can use the 12 single channels or the first 6 single channels, both of which can improve the classification accuracy and reduce the time consumption relative to the full channel.

[0080] Based on any of the above embodiments, the frequency band with strong relevance to emotion classification can be determined based on the classification accuracy obtained by the ECN-AF model for emotion classification of single frequency band or multiple frequency band combinations of the electroencephalogram signal. The electroencephalogram signal can be filtered into 5 different frequency bands by a fourth-order Butterworth band-pass filter: δ (1HZ, 4HZ), θ (4HZ, 7HZ), α (8HZ, 13HZ), β (13HZ, 30HZ), γ (31HZ, 50HZ), and the final selected frequency band combination can be α, β, and γ.

[0081] Based on any of the above embodiments, the learning emotion classification method based on the electroencephalogram signal specifically includes the following steps:

[0082] (I) Data processing. The electroencephalogram signal is preprocessed, and then a 10s window containing a 2s overlapping sliding window is used to divide the data, and the divided data is filtered into 5 different frequency bands by a band-pass filter;

[0083] (II) Selection of emotion-related channels and frequency bands. First, models A and B suitable for single-channel data classification are used for training and classification, and the first 12 channels with better classification effect are selected for channel combination to analyze the influence of different channel combinations on classification accuracy and time consumption; second, the influence of single frequency band and multiple frequency band feature attention fusion on classification performance is studied;

[0084] (Three) Analyze the correlation between confusion and other emotions. First, analyze all learning video clips; then analyze the confusion video-induced emotional situation separately; finally, directly use the data without separating confusion to conduct experiments, and analyze the correlation between confusion and other emotions according to the confusion matrix of the classification results.

[0085] (Four) Network model training and completion of multi-label classification task. First, re-label the labels; then, through network model training, conduct two classification tasks, the first classification task separates the confusion emotion, and the second classification task classifies other emotions. The network model uses the frequency band attention feature extraction module and the feature fusion classification module of the electroencephalogram signal emotion classification network based on attention fusion (ECN-AF).

[0086] The frequency band attention feature extraction module includes a multi-channel convolutional backbone network and a frequency band attention fusion unit; the multi-channel convolutional backbone network is constructed by using two layers of CNN, AvgPool1D, Batch Normalization and SpatialDropout1D, and extracts the features of different frequency bands under channel combination C, that is, Fc={F fc}, f∈{δ,θ,α,β,γ}; Fc is input to the frequency band attention fusion unit, which first selects the required frequency band features from Fc, and then generates an attention weight according to the selected feature vector, and then assigns the attention weight to the corresponding feature to obtain the fused feature;

[0087] The feature fusion classification module is a classification network constructed by CNN, AvgPool1D, Batch Normalization, SpatialDropout1D, GlobalAvgPool1D, Dropout and Dense, which inputs the fused feature of the frequency band attention feature extraction module for classification.

[0088] In the above steps, model A suitable for single-channel data classification, i.e. the single-channel emotion classification model described above, is an improvement of ECN-AF, which is changed to a model suitable for single-channel, the input is single-channel data, the frequency band attention fusion unit is changed to directly use attention to assign weights to network output features, and finally the deep feature extraction in the ECN-AF classification network is omitted, and the classification output result is directly classified.

[0089] In the above steps, model B suitable for single-channel data classification is a "CNN+LSTM" model, which uses a two-dimensional convolutional neural network and a bidirectional long short-term memory network (Bi-LSTM) to extract features, and then uses a fully connected layer with softmax to classify using the ECN-AF classification module.

[0090] Based on any of the above embodiments, the present application provides a learning emotion classification method based on electroencephalogram signals, aiming to solve the problem of confusion emotion being easily confused with other emotions, affecting the classification accuracy, and the problem of large operation time consumption, and the specific implementation steps are as follows:

[0091] Figure 2 is the implementation diagram of the learning emotion classification method based on electroencephalogram signals provided by the present application, the emotion classification method proposed by the present application comprises the following modules in the whole: (1) data processing; (2) selection of learning emotion related channels and frequency bands; (3) analysis of the correlation between confusion and other emotions; (4) network model training and completion of multi-label classification task. The specific operation steps involved in each module are as follows:

[0092] (1) Data processing:

[0093] The present application selects a self-built EEG learning emotion dataset, pre-processes the electroencephalogram signals, then divides the data by using a 10s window containing a 2s overlapping sliding window, and extracts 5 different frequency bands: δ (1HZ, 4HZ), θ (4HZ, 7HZ), α (8HZ, 13HZ), β (13HZ, 30HZ), γ (31HZ, 50HZ) through a fourth-order Butterworth band-pass filter for the divided data.

[0094] EEG learning emotion dataset (LE-EEG): 50 learning clips intended to induce concentration, boredom and confusion were selected from a well-known learning website in China. 49 subjects watched the videos and filled out the "Learning State Self-Assessment" questionnaire, using a 5-point scale for scoring. Finally, 44 valid questionnaires were obtained. Through suitability analysis of learning emotion materials, 28 learning videos were selected, including 12 videos to induce concentration\engagement, 4 videos to induce boredom, 6 videos to induce neutral, and 6 videos to induce confusion. In addition, 47 new subjects were recruited for electroencephalogram signal acquisition, and 20 videos (5 for engagement, 4 for boredom, 5 for neutral, and 6 for confusion) were selected from the 28 learning video clips to induce the subjects to produce learning emotions. Finally, 45 subjects had valid experimental data available.

[0095] (2) Selection of learning emotion related channels and frequency bands:

[0096] For electroencephalogram emotion classification, considering that there is redundancy between electrodes, and not all electrodes and frequency bands can play an important role, the influence of channels and frequency bands on emotion classification is analyzed. In order to select the channel with strong correlation with learning emotion, the model A and B are used for training and completing classification. Through the two models, 12 single channels with high accuracy are found, and the top 12 channel classification results are shown in Table 1. The ranking is based on the results of model A. In the case of as little as possible affecting the accuracy, the less channel and frequency band related to learning emotion are selected to save time overhead.

[0097] Table 1 Single channel classification results

[0098]

[0099] In order to further verify the influence of different channel combinations on learning emotion classification, the present application selects two combination modes C1=[T7, FT9, CP5, FT10, T8, C3] and C2=[T7, FT9, CP5, FT10, T8, C3, F7, FP1, Fp2, C4, F3, FC6].

[0100] The present application explores the influence of using single frequency band and multi-band combination on classification accuracy on C1, C2 channel combination and full channel, and the results are shown in Table 2.

[0101] Table 2 Classification accuracy of different frequency bands

[0102]

[0103] From the table, it can be seen that fewer channels can achieve a classification result close to that of full channels; the classification accuracy using 3 frequency bands is better than that using full frequency bands (δ, θ, α, β, γ); the C2 channel combination is almost better than the C1 channel combination. Therefore, the present application applies the C2 channel combination and selects α, β, γ three frequency bands for feature extraction and attention fusion.

[0104] (3) Analyze the correlation of confusion with other emotions:

[0105] By analyzing the effective subjective evaluation questionnaire of the subjects, the present application counts the proportion of induced emotion types in all emotions, as shown in Table 3. From the table, it can be seen that among the four video types, the confusion video has the lowest discrimination rate of target emotion, and is more likely to induce non-target emotions than other emotion videos, for example, the hit rate of confusion video inducing boredom emotion is 17.43%.

[0106] Table 3 Learning video emotion discrimination results

[0107]

[0108]

[0109] The selected 6 confusion emotion videos are analyzed separately, and the average score of part of the emotion-related questionnaire items is calculated by the effective subjective questionnaire of 44 subjects to obtain Table 4. It can be found from Table 4 that confusion may also occur together with other emotions, and even the scores are close, for example, the boredom and confusion emotion scores of No. 04 video (see the 5th row of Table 4) are both 3.07. This indicates that there is confusion between confusion and other emotion labels.

[0110] Table 4: Confusion learning video scores

[0111] Video number Focus Boredom Confusion 01 2.94±0.68 1.03±0.59 2.69±1.12 02 1.32±0.78 2.68±0.8 3.00±0.78 03 2.31±0.86 2.48±0.88 2.81±0.78 04 0.93±0.66 3.07±0.92 3.07±0.69 05 2.55±0.90 1.41±0.91 2.32±0.92 06 1.37±0.61 2.63±0.91 2.95±0.90

[0112] For the self-built data set, ten-fold cross-validation is used for each subject, and the ECN-AF model is used for four classification (confusion, boredom, concentration, and neutral), and the results are shown in Table 5.

[0113] Table 5: Four classification accuracy

[0114]

[0115] In order to more intuitively analyze the recognition ability of various emotions, the confusion matrix of the data classification of each subject is drawn, Figure 3 some confusion matrix graphs are shown in FIG. 6. In the confusion matrix graph, the horizontal axis represents the predicted label, the vertical axis represents the true label, and from left to right and from top to bottom, the concentration \ engagement, boredom, neutral and confusion are represented. It can be seen from the graph that the recognition accuracy of confusion is poor; confusion is easy to be confused with other emotions and is difficult to distinguish.

[0116] In summary, it is necessary to separate the confusion emotion to avoid the confusion of confusion emotion and other emotions, resulting in lower model performance.

[0117] (4) Network model training and completion of multi-label classification task:

[0118] Figure 4This is a flowchart of the confusion separation experiment provided by the present invention. Specifically, the present invention calibrates the collected EEG signal segments with multiple classification labels. First, the EEG signal segments are calibrated as confused or not confused; then the non-confused EEG signal segments are labeled as focused / engaged, bored, and neutral. During the model training and classification process, first, the EEG data labeled as confused and not confused are classified for the first time using the model to achieve the effect of confusion separation; then, all non-confused emotions are aggregated and output, and the samples with incorrect predictions are deleted, that is, the EEG signal samples of the confused class are identified as non-confused; finally, the EEG signals that do not contain the confused class are calibrated for the second time for the second classification, and this classification includes three categories: focused / engaged, bored, and neutral. The results of the first classification (confused / non-confused) are shown in Table 6.

[0119] Table 6 Confusion binary classification

[0120]

[0121] Specifically, in order to verify the effectiveness of the proposed method, the present invention compared its performance with the baseline model. By reproducing the SOGNN (Self-Organized Graph Neural Network) and 4D_CRNN (4D convolutional recurrent neural network) models proposed by the existing method, the results are compared as shown in Table 6. The data required for secondary classification (i.e., data without confused emotions) was used to perform three classifications (focus, boredom, and neutrality). A ten-fold cross-validation method was used for each subject. When the present invention applied the C2 channel combination, 3 frequency bands were used as input, and the accuracy obtained was 97.97%. When the full channel was applied, 3 frequency bands were used as input, and the accuracy obtained was 98.56%. The time consumption of using the full channel (32 channels) is 5 times that of using the C2 channel combination (12 channels), but the classification accuracy only differed by 0.59%. Compared with the baseline model, the present invention uses less data and achieves an improvement in classification accuracy of 7.70%-13.17%.

[0122] In addition, it can be seen from Tables 5, 6, and 7 that the classification accuracy can be improved after the perplexity separation experiment.

[0123] Table 7 Three-classification results

[0124]

[0125] Overall, the present application improves the emotion classification accuracy by separating the confused emotion, and also improves the problem of confusion affecting emotion classification mixed with other emotions. Through verification on the self-built data set, the emotion classification method proposed in the present application can not only show good emotion classification accuracy, but also can well identify the four emotions. Compared with ordinary four classification, the recognition accuracy of each emotion is improved, and a lot of time is saved.

[0126] Based on any of the above embodiments, the present application provides a learning emotion classification system based on electroencephalogram signals, Figure 5 is the architecture diagram of the learning emotion classification system based on electroencephalogram signals provided by the present application, as Figure 5 shown, the system comprises:

[0127] The electroencephalogram signal acquisition module 510 is configured to acquire electroencephalogram signals of multiple channels of the learner; different channels correspond to different brain positions.

[0128] The frequency domain decomposition module 520 is configured to filter the electroencephalogram signals into multiple different frequency bands through a Butterworth band-pass filter, and obtain the electroencephalogram signals of multiple frequency bands of multiple channels.

[0129] The confusion classification module 530 is configured to input the electroencephalogram signals of multiple frequency bands of multiple channels into the trained confusion classification model, and obtain the confusion classification result of the learner.

[0130] The emotion classification module 540 is configured to, if it is judged that the confusion classification result is confusion, the confusion classification result is the learning emotion of the learner; if it is judged that the confusion classification result is not confusion, then input the electroencephalogram signals of multiple frequency bands of multiple channels into the trained emotion classification model, and obtain the emotion classification result of the learner, which is the learning emotion of the learner.

[0131] The system provided by the embodiment of the present application can understand the state of the learner by recognizing the learning emotion of the learner through the electroencephalogram signals of multiple frequency bands of multiple channels, separate the confused emotion in the first classification task, and then classify other emotions in the second classification task, and pay attention to the correlation between frequency bands and channels in both classification tasks, so that the classification confusion of confused emotion and other emotions can be avoided, and good classification accuracy can be obtained for confusion classification and other emotion classification.

[0132] It can be understood that the detailed function implementation of each module can be referred to the introduction in the foregoing method embodiment, which will not be repeated here.

[0133] In addition, the embodiment of the present application provides another learning emotion classification device based on electroencephalogram signals, which comprises a memory and a processor.

[0134] The memory is configured to store a computer program.

[0135] The processor is configured to implement the method in the above embodiments when executing the computer program.

[0136] In addition, the application further provides a computer readable storage medium, and the storage medium stores a computer program. When the computer program is executed by a processor, the method in the above embodiments is implemented.

[0137] Based on the method in the above embodiments, the application provides a computer program product. When the computer program product is run on a processor, the processor executes the method in the above embodiments.

[0138] Those skilled in the art can easily understand that the above description is only the preferred embodiment of the application, and is not intended to limit the application. Any modification, equivalent replacement and improvement made within the spirit and principle of the application shall be included in the protection scope of the application.

Claims

1. A learning emotion classification method based on electroencephalogram signals, characterized in that, The method comprises the steps of: S101 acquiring electroencephalogram signals of multiple channels of a learner; Different channels correspond to different brain positions; S102 filtering the electroencephalogram signals into multiple different frequency bands through a Butterworth band-pass filter to obtain electroencephalogram signals of multiple frequency bands of the multiple channels; S103 inputting the electroencephalogram signals of the multiple frequency bands of the multiple channels into a trained confusion classification model to obtain a confusion classification result of the learner; If it is judged that the confusion classification result is confusion, the confusion classification result is the learning emotion of the learner; If it is judged that the confusion classification result is not confusion, the electroencephalogram signals of the multiple frequency bands of the multiple channels are inputted into a trained emotion classification model to obtain an emotion classification result of the learner, and the emotion classification result is the learning emotion of the learner; S103 specifically comprises: extracting features of multiple frequency bands under a channel combination based on the electroencephalogram signals of the multiple frequency bands of the multiple channels; the channel combination is a combination of channels with strong relevance to emotion classification; extracting features of a frequency band combination under the channel combination from the features of the multiple frequency bands under the channel combination, generating a corresponding attention weight of the frequency band combination based on the features of the frequency band combination under the channel combination, assigning the attention weight to the corresponding features and fusing them to obtain fused features; the frequency band combination is a combination of frequency bands with strong relevance to emotion classification; performing confusion classification based on the fused features to obtain the confusion classification result of the learner.

2. The method of claim 1, wherein, S103 further comprises the following steps: adjusting a classification layer of an initial emotion classification model to obtain an initial confusion classification model suitable for binary classification; training the initial confusion classification model based on confusion samples and non-confusion samples to obtain the trained confusion classification model; training the initial emotion classification model based on concentration samples, boredom samples and neutral samples to obtain the trained emotion classification model.

3. The method of claim 1, wherein, The channels with strong relevance to emotion classification are determined based on a classification accuracy obtained by performing emotion classification on the electroencephalogram signals of each channel through a single-channel emotion classification model; The single-channel emotion classification model comprises a convolution backbone network, an attention unit and a softmax classification unit; the convolution backbone network is used to extract features based on the electroencephalogram signals of each channel, the attention unit is used to generate a weight based on the features and assign the weight to the features to obtain weighted features, and the softmax classification unit is used to perform emotion classification based on the weighted features.

4. The method of claim 1, wherein, The channel combination comprises two channels corresponding to the temporal lobe, two channels corresponding to positions between the frontal lobe and the temporal lobe, one channel corresponding to a position between the central region and the parietal lobe, and one channel corresponding to the central region.

5. A learning emotion classification system based on electroencephalogram signals, characterized by, The method comprises the steps of: an electroencephalogram signal acquisition module for acquiring electroencephalogram signals of multiple channels of a learner; Different channels correspond to different brain positions; a frequency domain decomposition module for filtering the electroencephalogram signals into multiple different frequency bands through a Butterworth band-pass filter to obtain electroencephalogram signals of multiple frequency bands of the multiple channels; a confusion classification module for inputting the electroencephalogram signals of the multiple frequency bands of the multiple channels into a trained confusion classification model to obtain a confusion classification result of the learner; The emotion classification module is configured to: if it is determined that the confusion classification result is confusion, the confusion classification result is the learning emotion of the learner; if it is determined that the confusion classification result is not confusion, the EEG signals of the multiple frequency bands of the multiple channels are input into the trained emotion classification model to obtain the emotion classification result of the learner, and the emotion classification result is the learning emotion of the learner. The confusion classification module is specifically configured to: extract features of multiple frequency bands under a channel combination based on the EEG signals of the multiple frequency bands of the multiple channels; the channel combination is a combination of channels with strong relevance to emotion classification; extract features of a frequency band combination under the channel combination from the features of the multiple frequency bands under the channel combination, generate a corresponding attention weight of the frequency band combination based on the features of the frequency band combination under the channel combination, assign the attention weight to the corresponding features, and fuse the features to obtain fused features; the frequency band combination is a combination of frequency bands with strong relevance to emotion classification; perform confusion classification based on the fused features to obtain the confusion classification result of the learner.

6. The system of claim 5, wherein, The model training module is further configured to: adjust a classification layer of an initial emotion classification model to obtain an initial confusion classification model suitable for binary classification; train the initial confusion classification model based on confusion samples and non-confusion samples to obtain the trained confusion classification model; train the initial emotion classification model based on focused samples, boring samples, and neutral samples to obtain the trained emotion classification model.

7. The system of claim 5, wherein, The channels with strong relevance to emotion classification used in the confusion classification module are determined based on a classification accuracy obtained by performing emotion classification on EEG signals of each channel by a single-channel emotion classification model; The single-channel emotion classification model includes a convolution backbone network, an attention unit, and a softmax classification unit; the convolution backbone network is configured to extract features based on EEG signals of each channel, the attention unit is configured to generate weights based on the features, and assign the weights to the features to obtain weighted features, and the softmax classification unit is configured to perform emotion classification based on the weighted features.

8. The system of claim 5, wherein, The channel combination used in the confusion classification module includes two channels corresponding to the temporal lobe, two channels corresponding to the positions between the frontal lobe and the temporal lobe, one channel corresponding to the positions between the central region and the parietal lobe, and one channel corresponding to the central region.

Images