VR smart classroom teaching method and system based on electroencephalogram emotion recognition

Abstract

The invention relates to a smart classroom teaching method based on electroencephalogram emotion recognition. The method comprises the following steps that VR courses are distributed to VR equipment of a student side, and electroencephalogram signals collected by electroencephalogram collection equipment of the student side are received; the electroencephalogram signals are preprocessed, and then emotion judgment and classroom state evaluation are carried out; and the VR course is adjusted in real time according to the classroom state evaluation result. According to the intelligent classroom teaching method based on electroencephalogram emotion recognition, the learning efficiency and teaching quality of students can be improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of smart classroom teaching, and specifically relates to a smart classroom teaching system based on EEG emotion recognition. Background Art

[0002] VR teaching systems based on brain wave emotion recognition can empower classroom teaching, but they are insufficient in the precise extraction of brain wave spatiotemporal features, and the accurate identification of students' emotions needs to be improved. For example, Chinese patent application CN115657847A (abbreviated as D1) is an adaptive virtual assembly training method based on cognitive load. In order to realize the adaptive virtual assembly training function, an EEG signal acquisition device is used to collect the EEG signals of the trainees, and the EEG signals are processed and features are extracted. Then, a support vector machine classification algorithm with good generalization ability is selected for classification and identification to find the upper and lower bounds of the trainees' cognitive load, and a personal cognitive load classification model is trained using machine learning. This model can classify EEG data into three types of cognitive load: high, medium, and low. The model is embedded in the virtual assembly training system. When the trainee is wearing an EEG device for virtual assembly training, the EEG data collected by the EEG device is transmitted to the personal cognitive load classification model in the system. The system will obtain the real-time cognitive load of the trainee and automatically push the teaching and training content that meets the cognitive load at that time to the trainee according to the level of cognitive load, that is, when the cognitive load is high, the system will push animation training content to the trainee; when the cognitive load is low, the system will push text training content to the trainee; when the cognitive load is moderate, the system will push voice training content to the trainee. However, the patent uses two methods, wavelet transform and sample entropy method, to extract EEG signal features, which cannot accurately process massive data and the steps are cumbersome. Chinese patent application CN115713875A (abbreviated as D2) discloses a virtual reality simulation teaching method based on psychological analysis, which judges the implicit emotional state of teachers and students based on voice information and brain wave data, monitors brain waves based on VR head-mounted devices, and matches them with brain wave feature templates, and uses the matching probability with each emotion as the emotional brain wave matching score. Normalize the emotion text matching score, emotion brain wave matching score and emotion state matching score. Set the weight w, and calculate the hidden emotion state with the highest probability according to the emotion text matching score, emotion brain wave matching score and emotion state matching score. According to the student's emotion curve, judge the student's respect for the teacher, and evaluate the teacher's teaching quality and student psychology. However, the patent uses a CNN network to accurately extract the time characteristics of the EEG signal, and the accurate identification of students' emotions needs to be improved. Chinese patent application CN117520826A (abbreviated as D3) discloses a multimodal emotion recognition method and system based on wearable devices, which uses a bandpass filter to remove high-frequency and low-frequency noise in the EEG signal and retains the EEG signal with a frequency of 4-30Hz; eliminates the DC offset in the EEG signal, that is, the long-term average value, through baseline correction; extracts time domain features from the collected EEG, EMG, EGO and ECG signals, first extracts the mean, standard deviation, maximum and minimum values ​​of each EEG time domain signal, uses fast Fourier transform to obtain the corresponding frequency domain signal, and extracts relevant frequency domain information.After the EEG signal is preprocessed to remove exogenous and endogenous artifacts, the EEG signal can be used alone to extract emotional features through a convolutional network with a self-attention mechanism, or the EEG signal and the facial image can be combined through a convolutional network with a multi-head attention mechanism, a Dropout layer is set, and the fused emotional features are output through an LSTM emotional state encoder (see. Figure 3 ). However, this patent requires the fusion extraction of EEG signals and facial images, and the introduction of the self-attention mechanism and LSTM algorithm cannot accurately capture the spatial correlation in the EEG signal sequence, and is insufficient in accurately judging students' emotions. Summary of the invention

[0003] In view of the above technical problems, the present invention provides a smart classroom teaching method based on EEG emotion recognition, comprising the following steps:

[0004] Distribute VR courses to VR devices at the student end and receive EEG signals collected by EEG collection devices at the student end;

[0005] After preprocessing the EEG signal, emotion discrimination and classroom status assessment are performed;

[0006] The VR course is adjusted in real time according to the classroom status evaluation results.

[0007] Another aspect of the present invention provides an intelligent classroom teaching system based on EEG emotion recognition, the system comprising at least one processor; and a memory storing instructions, which, when executed by at least one processor, implement the steps of the aforementioned method.

[0008] The beneficial effect of the present invention is that the smart classroom teaching method based on EEG emotion recognition can improve students' learning efficiency and teaching quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 : Schematic diagram of the principle of the smart classroom teaching system based on EEG emotion recognition of the present invention;

[0010] Figure 2 : Schematic diagram of the principle of the smart classroom teaching system based on EEG emotion recognition of the present invention;

[0011] Figure 3 : Network model of prior art (CN117520826A);

[0012] Figure 4 : Schematic diagram of comparison of three different network models;

[0013] Figure 5 : Accuracy curves of FFT-CNN model and FFT-LSTM model;

[0014] Figure 6 : Schematic diagram of the participants’ corresponding accuracy;

[0015] Figure 7 : Schematic diagram of the comparison of the accuracy of the three models. DETAILED DESCRIPTION

[0016] The following examples further illustrate the content of the present invention, but should not be construed as limiting the present invention. Without departing from the spirit and substance of the present invention, modifications or substitutions made to the methods, steps or conditions of the present invention all fall within the scope of the present invention.

[0017] like Figure 1 As shown, the patent of this invention proposes a smart classroom teaching system based on EEG emotion recognition. The smart classroom teaching system mainly includes an AI emotion recognition module, a VR teaching module and a smart classroom module. Through the VR module, students can obtain the course content designed for the smart classroom. The AI ​​emotion recognition module can obtain the EEG emotion data of students during class; on this basis, the students' EEG emotion data is identified through the AI ​​algorithm, and then the students' mastery of the knowledge points is evaluated. On the basis of obtaining the mastery situation, the teaching content, teacher-student interaction and other aspects in the smart classroom module are corrected and updated in real time to improve students' learning efficiency and teaching quality. The smart classroom teaching system based on EEG emotion recognition can empower classroom teaching and has a relatively broad application prospect.

[0018] like Figure 2 As shown, some embodiments of the present invention relate to a smart classroom teaching method based on EEG emotion recognition, comprising the following steps:

[0019] Distribute VR courses to VR devices at the student end and receive EEG signals collected by EEG collection devices at the student end;

[0020] After preprocessing the EEG signal, emotion discrimination and classroom status assessment are performed;

[0021] The VR course is adjusted in real time according to the classroom status evaluation results.

[0022] Furthermore, some embodiments of the present invention relate to the preprocessing method including:

[0023] Using a filter to remove artifacts and noise from the EEG signal to obtain EEG data, wherein the filter is selected from one or more of a low-pass filter, a high-pass filter, a band-pass filter and a band-stop filter;

[0024] The EEG data is segmented and baseline corrected, bad leads are removed or interpolated, bad segments are removed, and artifacts are removed.

[0025] Further, some embodiments of the present invention relate to segmentation and baseline correction including the following steps:

[0026] Segmenting the EEG data before and after the stimulation of the course content point;

[0027] Eliminate EEG noise caused by spontaneous EEG waves in segmented EEG data;

[0028] The mean of the data before time point 0 is taken as the baseline.

[0029] Further, some embodiments of the present invention involve removing or interpolating bad leads including the following steps:

[0030] Remove the bad leads from the segmented and baseline-corrected EEG data; or,

[0031] The data of bad leads in the segmented and baseline-corrected EEG data are repaired based on the normal lead data through interpolation method.

[0032] Further, some embodiments of the present invention relate to removing bad segments including the following steps: manually removing bad segments from EEG data from which bad leads have been removed or interpolated;

[0033] Artifact removal includes the following steps: based on ICA, artifacts caused by eye movements and heartbeats in the EEG data are removed to eliminate bad segments.

[0034] Furthermore, some embodiments of the present invention relate to the EEG signal being an EEG time domain signal, and the emotion discrimination method includes:

[0035] The EEG time domain signal is converted into an EEG FFT signal using Fast Fourier Transformation (FFT), and the EEG FFT signal is an EEG frequency domain signal;

[0036] The EEG FFT signal is input into the (FFT-CCLA) model to discriminate and evaluate EEG emotions.

[0037] Furthermore, some embodiments of the present invention relate to a method for constructing the FFT-CCLA model including:

[0038] The EEG FFT signal is input into the CCLA (Channel attention-CNN-LSTM-SelfAttention) network, and the emotion discrimination result output by the CCLA network is matched with the student self-evaluation and the teacher evaluation to construct a CCLA model; the CCLA network uses the SoftMax function for classification evaluation.

[0039] Among them, the channel attention mechanism (Channel Attention) will perform weighted processing on each channel (channel) to increase the attention to important channels, thereby improving the performance of the model. CNN (Convolutional Neural Network) is mainly used to extract spatial information in the signal, and can identify the correlation between EEG signals in different positions. LSTM (Long Short-Term Memory Network) is mainly used to learn the long-term dependencies of EEG signals and effectively capture the changes in emotional states over time. The self-attention mechanism (Self-Attention) is an attention mechanism for processing sequence data. In this mechanism, the model performs weighted processing on each element in the input sequence to increase the attention to important elements. In some specific embodiments, the model involved in the present invention is constructed based on the channel attention mechanism, CNN, LSTM, and self-attention mechanism. After testing, its model accuracy and other aspects of performance are better than the model based on the attention mechanism, LSTM, and multi-head attention mechanism. Figure 3 ).

[0040] Furthermore, some embodiments of the present invention relate to the Channel Attention performing global average pooling on the FFT data of the EEG signal, and then sending the processed vector to the CNN network.

[0041] Furthermore, the CNN network adopts a three-layer convolutional structure with a batch size of 128, and a dropout layer is added to the network to prevent overfitting. Then, the SoftMax classifier is used to evaluate the EEG feature classification, and the evaluation signal is sent to the LSTM network.

[0042] Furthermore, the LSTM network uses the Adam optimization algorithm, the batch size is set to 128, and a dropout layer is added to avoid overfitting. Then the SoftMax classifier is used to evaluate the EEG feature classification and send the signal to the Self-Attention network.

[0043] Furthermore, Self-Attention contains 4 heads and 12 key and query channels. The SoftMax classifier is used to evaluate EEG feature classification and compared with the classroom evaluation of teachers and students.

[0044] Furthermore, based on the evaluation and comparison results, the network parameters are continuously adjusted and the constructed FFT-CCLA network is corrected so that the network evaluation results are consistent with those of the teacher and the student, thus completing the FFT-CCLA network training.

[0045] Furthermore, some embodiments of the present invention relate to the classroom status assessment comprising the following steps: generating a student classroom status assessment result based on an emotion discrimination result; if the discrimination is negative, the status is poor; if the discrimination is positive, the status is excellent.

[0046] The real-time adjustment includes teacher-student interaction and content expansion.

[0047] Other embodiments of the present invention relate to a smart classroom teaching system based on EEG emotion recognition, the system comprising at least one processor; and a memory storing instructions, which, when executed by at least one processor, implement the steps of the aforementioned method.

[0048] It should be pointed out that the network proposed in some embodiments of the present invention has strong generalization ability, and can be used not only for student emotion recognition, but also for more complex human emotion recognition. The improvement of the network proposed in the present invention based on the generalization ability is within the concept of the present invention.

[0049] The technical scheme and technical effects of the present invention are further described below in conjunction with specific embodiments.

[0050] Example 1

[0051] step:

[0052] 1. Students access course content

[0053] 32 healthy students were selected, wearing teaching equipment and receiving the course content.

[0054] 1.1 16 boys and 16 girls

[0055] 1.2 The teaching equipment mainly consists of two parts, namely VR module and AI emotion recognition module. VR module is used for students to receive course content; AI emotion recognition module is used to collect students' EEG emotion data and perform emotion discrimination based on AI algorithm. The smart classroom module divides the course content according to the knowledge graph and divides the classroom content into corresponding segments.

[0056] 1.3 The standard course content is 25 minutes, one class is 45 minutes, and the remaining 20 minutes will change in real time based on class feedback.

[0057] 2. Assessment of students’ learning status

[0058] Obtain students' EEG data, identify their EEG emotions, and then evaluate their learning status.

[0059] Based on obtaining students' EEG data, and continuing EEG emotion data processing.

[0060] 2.1. Use EEG data acquisition equipment to obtain students’ EEG emotional data.

[0061] 2.2. Based on the acquired EEG emotion data, the data is preprocessed and denoised to eliminate the influence of physiological artifacts and non-physiological artifacts.

[0062] 2.2.1. When processing EEG data, it is necessary to select a suitable filter. You can select an appropriate filter based on the frequency range of artifacts and noise. At the same time, you should also select one of the low-pass, high-pass, band-pass, and band-stop filters based on the frequency band you are interested in in the EEG emotion data.

[0063] 2.2.2. Segment and baseline correct the EEG data.

[0064] It is necessary to segment the EEG data before and after the onset of stimulation of the course content point in order to extract the changes in EEG activity after sensory stimulation or cognitive presentation.

[0065] The segmented EEG data needs to eliminate the EEG noise caused by spontaneous EEG waves. Usually, this is done by subtracting an average baseline value, with the mean of the data before the "0 time point" as the baseline.

[0066] 2.2.3. Eliminate or interpolate bad lead data

[0067] The data of bad leads can be repaired through interpolation method based on the normal lead data.

[0068] 2.2.4. Remove bad segments

[0069] When manually removing bad segments, the data was checked multiple times to improve data quality and reliability.

[0070] 2.2.5. Artifact Removal Based on ICA

[0071] In the preprocessing process, ICA is mainly used to remove artifacts caused by eye movements and heartbeats. By observing the topographic map, trial distribution map, and frequency distribution map of each independent component, the components related to the artifacts can be identified and removed.

[0072] Determine students' EEG emotions and then evaluate their learning status.

[0073] 2.2.6 Using FFT-CCLA model to identify and evaluate EEG emotions

[0074] 2.2.6.1 Use Fast Fourier Transform (FFT) to transform the EEG time domain signal into the frequency domain signal to obtain the pattern showing the distribution of EEG power with frequency.

[0075] 2.2.6.2 Input the EEG FFT signals of 24 students into the CCLA network. Use the network to analyze and model the correlation information between EEG signal channels and extract important spatiotemporal emotional features. On this basis, use the SoftMax function for classification evaluation.

[0076] 2.2.6.3 The output of the neural network is matched with the students’ self-evaluation and the teacher’s evaluation, and then the FFT-CCLA model that can be used to evaluate the students’ learning status is constructed. On this basis, the EEG data, students’ self-evaluation and teacher’s evaluation of the remaining 8 students are used to verify the effectiveness of the FFT-CCLA model.

[0077] 2.2.6.4 Based on the judgment result, generate the student's classroom status evaluation result. If the judgment is no, the status is poor; if the judgment is yes, the status is excellent.

[0078] 3. Real-time adjustment of smart courses

[0079] Based on the students' classroom evaluation status, the course is adjusted in real time to form a specific VR teaching scene. The real-time adjustment is mainly divided into two parts: teacher-student interaction and content expansion.

[0080] 3.1 When emotion recognition is excellent: (5 minutes in smart classroom) The teacher can encourage the students and deepen their understanding of specific knowledge points. (5 minutes in smart classroom) On this basis, a test is conducted, and a specific knowledge map is formed for each student based on the test results. Further VR teaching scenes for specific students are formed (10 minutes in smart classroom).

[0081] 3.2 When emotion recognition is poor: (5 minutes in smart classroom) The teacher can encourage students to learn and lay a solid foundation. (5 minutes in smart classroom) On this basis, a test is conducted, and a specific knowledge map is formed for each student based on the test results. A VR teaching scene for a specific student is further formed (10 minutes in smart classroom). Example 2

[0082] Combination Figures 4 to 7 Embodiment 2 is explained below. For matters not explained, please refer to Embodiment 1.

[0083] 1. Data Source

[0084] The data comes from DEAP (DEAP is a multi-channel dataset for studying human emotional states, collected through experiments by institutions such as Queen Mary University of London, University of Twente, University of Geneva, and Swiss Federal Institute of Technology. The dataset can be downloaded from its official website (http: / / www.eecs.qmul.ac.uk / mmv / datasets / deap / ). The dataset contains EEG data of 32 healthy participants (16 men and 16 women). The participants need to watch 40 one-minute videos and record their EEG signals at a sampling frequency of 512Hz.

[0085] 2. Data Results

[0086] like Figure 4 As shown in Figure 2, it is a schematic diagram comparing three different network models.

[0087] 2.1 FFT-CNN model and FFT-LSTM model

[0088] The accuracy curves of the FFT-CNN model and the FFT-LSTM model are as follows: Figure 5 As shown. Figure 5 It can be seen that the average classification accuracy of the FFT-CNN model is 87.79%, and the average classification accuracy of the FFT-LSTM model is 88.30%.

[0089] 2.2 FFT-CCLA model

[0090] In order to more clearly show the accuracy of the FFT-CCLA model, Figure 6 The corresponding accuracy rate for each participant is given. Figure 6 It can be seen that the FFT-CCLA model showed high classification accuracy in most participants, with an average classification accuracy of up to 93.54%.

[0091] 2.3 Accuracy of FFT-CNN, FFT-LSTM, and FFT-CCLA Models

[0092] Comparison of the accuracy of the FFT-CNN model, FFT-LSTM model, and FFT-CCLA model Figure 7 As shown, the FFT-CCLA model has good performance and can accurately identify students' EEG signals in time and space, and then accurately judge students' emotions.

[0093] Although the present invention has been described in detail above by general description, specific implementation methods and experiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. A smart classroom teaching method based on EEG emotion recognition, characterized in that: The steps include: Distribute VR courses to VR devices at the student end and receive EEG signals collected by EEG collection devices at the student end; After preprocessing the EEG signal, emotion discrimination and classroom status assessment are performed; The VR course is adjusted in real time according to the classroom status evaluation results.

2. The method according to claim 1, characterized in that The pretreatment method comprises: Using a filter to remove artifacts and noise from the EEG signal to obtain EEG data, wherein the filter is selected from one or more of a low-pass filter, a high-pass filter, a band-pass filter and a band-stop filter; The EEG data is segmented and baseline corrected, bad leads are removed or interpolated, bad segments are removed, and artifacts are removed.

3. The method according to claim 2, characterized in that Segmentation and baseline correction include the following steps: Segmenting the EEG data before and after the stimulation of the course content point; Eliminate EEG noise caused by spontaneous EEG waves in segmented EEG data; The mean of the data before time point 0 is taken as the baseline.

4. The method according to claim 3, characterized in that Eliminating or interpolating bad leads includes the following steps: Remove the bad leads from the segmented and baseline-corrected EEG data; or, The data of bad leads in the segmented and baseline-corrected EEG data are repaired based on the normal lead data through interpolation method.

5. The method according to claim 4, characterized in that Eliminating bad segments includes the following steps: manually eliminating bad segments in the EEG data from which bad leads have been eliminated or interpolated; Artifact removal includes the following steps: based on ICA, artifacts caused by eye movements and heartbeats in the EEG data are removed to eliminate bad segments.

6. The method according to any one of claims 1 to 5, characterized in that: The EEG signal is an EEG time domain signal, and the emotion discrimination method comprises the following steps: The EEG time domain signal is converted into an EEG FFT signal by using Fast Fourier Transform, wherein the EEG FFT signal is an EEG frequency domain signal; The EEG FFT signal is input into the FFT-CCLA model to identify and evaluate EEG emotions.

7. The method according to claim 6, characterized in that The method for constructing the FFT-CCLA model includes: The EEG FFT signal is input into the CCLA network, and the emotion discrimination result output by the CCLA network is matched with the student's self-evaluation and the teacher's evaluation to construct an FFT-CCLA model; the CCLA network uses the SoftMax function for network evaluation.

8. The method according to claim 7, characterized in that According to the comparison results of the network evaluation, the network parameters are continuously adjusted and the constructed FFT-CCLA network is corrected so that the network evaluation is consistent with the teacher and student evaluation results, and the FFT-CCLA network training is completed.

9. The method according to claim 8, characterized in that The classroom status evaluation comprises the following steps: generating a student classroom status evaluation result based on the emotion discrimination result; if the discrimination is negative, the status is poor; If the judgment is yes, the status is excellent; The real-time adjustment includes teacher-student interaction and content expansion.

10. A smart classroom teaching system based on EEG emotion recognition, characterized in that: The system comprises at least one processor; and a memory storing instructions, which, when executed by the at least one processor, implement the steps of the method according to any one of claims 1 to 9.

Images