A Lesson Plan Generation System and Method Based on Emotion Detection and Large Language Model

Abstract

The present invention discloses a teaching plan generation system and method based on emotion detection and large language model, and relates to the teaching plan generation technology field. The system includes a teacher module, a student physiological signal acquisition module, a multimodal signal processing module, a teaching plan generation module and a display module; the teacher module is connected with the student physiological signal acquisition module, the teaching plan generation module and the display module for receiving teacher information; the student physiological signal acquisition module is used to collect the student's physiological signal information; the multimodal signal processing module is connected with the student physiological signal acquisition module and the teaching plan generation module for processing the physiological signal information and obtaining the multimodal emotion detection result; the teaching plan generation module is connected with the teacher module, the display module and the multimodal signal processing module for generating teaching plans; the display module is used for information display. The present invention can generate teaching plans that are more targeted and more suitable for the emotional state of students.

Description

Technical Field

[0001] The present invention relates to the technical field of lesson plan generation, and in particular to a lesson plan generation system and method based on emotion detection and a large language model. Background Art

[0002] With the rapid development of science and technology, the "people-centered" interaction design principle has become the core concept of the development of the new generation of human-computer interaction. This principle has not only been widely used in the field of human-computer interaction, but also gradually demonstrated its strong potential and value in the field of education and teaching. In the field of education and teaching, lesson plans, as an important part of teaching, are a comprehensive reflection of teachers' teaching plans, ideas and organizational forms. Today, with the continuous development of human-computer interaction technology, the field of education and teaching has also begun to try to integrate these technologies into lesson plan design to provide a more personalized and efficient teaching experience.

[0003] In the process of education and teaching in primary and secondary schools, lesson preparation is the primary link in the teaching process and an important prerequisite for ensuring teaching quality. Lesson plans are an important part of lesson preparation. They are a comprehensive reflection of teachers' teaching plans, teaching ideas and teaching organization forms. They are an important guarantee for achieving teaching goals and improving teaching quality and effectiveness. Nowadays, teachers often encounter the following problems when writing lesson plans:

[0004] (1) Considering the differences in cognitive conditions of students in each class in actual teaching activities, lesson plans should be implemented in accordance with specific conditions to achieve the optimization of teaching effects. In the classroom teaching process, students' emotional state has a great impact on the teacher's teaching process and students' learning effects. Students' emotional state often changes due to subjective and objective factors, resulting in negative emotions such as anxiety, tension, stress, and irritability. Bad emotional state can easily lead to negative emotional interaction between teachers and students, thereby destroying the good atmosphere of teaching and learning and affecting normal educational and teaching activities. In addition, bad emotional state is mostly judged by teachers' experience and is greatly affected by subjective factors. Therefore, it is very necessary to grasp students' emotional state in real time in order to implement "one lesson, one update" to achieve the optimization of teaching effects.

[0005] (2) Considering that teaching activities are multi-level and multi-faceted, teachers need to invest time in teaching, student management, and teaching effect review in addition to lesson preparation. Relying on purely manual methods to achieve "one lesson, one update" will take up teachers' time for other teaching activities and affect teaching effectiveness. Teachers need to complete educational teaching and student management tasks, which are trivial and heavy, and it is easy to reduce the importance of lesson plans, affecting teachers' professional development and the quality of subject teaching. The leader of the lesson plan is the teacher, and the main body is the students. The analysis and design of students' learning situation is an important part of the lesson plan. However, in reality, the multi-level and multi-faceted teaching activities make it difficult for teachers to engage in lesson plan preparation alone. Therefore, the use of automated and intelligent methods to realize the auxiliary compilation of lesson plans has a high practical significance for the implementation of "one lesson, one update".

[0006] (3) For new teachers, they have insufficient understanding of the learning situation and lack efficient teaching methods suitable for groups with different learning backgrounds. Teachers' analysis of students' learning situation is usually based on the accumulation of long-term experience. New teachers have a short working time and lack an objective and comprehensive understanding of their positions and student groups. It is difficult for them to take into account the teaching of classes with different emotional states and to adjust and improve teaching plans in a timely manner according to the real-time changes in students' cognitive states. This has reduced the students' subjective status to a certain extent and ignored the emotional and knowledge needs of students in different states. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide a teaching plan generation system and method based on emotion detection and large language model, by collecting EEG, ECG and facial image signals of students in the classroom, extracting time-space-frequency features from EEG signals, extracting heart rate variability features from ECG signals, extracting optical flow features and frame difference features from facial images after preprocessing, and then further integrating them using multi-scale convolution and residual attention mechanisms, improving the model perception ability, and realizing the detection of students' emotional states. Then, the teaching plan is automatically generated through the large language model to improve the teaching effect and students' learning experience.

[0008] In order to solve the above technical problems, the first aspect of the embodiment of the present invention discloses a teaching plan generation system based on emotion detection and a large language model, the system comprising:

[0009] The system includes a teacher module, a student physiological signal acquisition module, a multimodal signal processing module, a teaching plan generation module and a display module;

[0010] The teacher module is data-connected with the student physiological signal acquisition module, the lesson plan generation module, and the display module, and is used to receive teacher information; the teacher information includes teacher ID login information, lesson plan information, and demand instruction information;

[0011] The student physiological signal acquisition module is data-connected with the teacher module and the multimodal signal processing module to collect the student's physiological signal information; the physiological signal information includes EEG information, ECG information and facial image information;

[0012] The multimodal signal processing module is data-connected with the student physiological signal acquisition module and the teaching plan generation module, and is used to process the physiological signal information to obtain a multimodal emotion detection result;

[0013] The teaching plan generation module is data-connected with the teacher module, the display module, and the multimodal signal processing module to generate a teaching plan;

[0014] The display module is data-connected with the teacher module and the lesson plan generation module, and is used to display lesson plan information, physiological signal information, emotion detection results, and lesson plans customized according to demand.

[0015] As an optional implementation, in the first aspect of the embodiment of the present invention, the teacher module includes a teacher ID login unit, a lesson plan information collection unit and a demand instruction input unit;

[0016] The teacher ID login unit is used to authenticate the teacher;

[0017] The lesson plan information collection unit is used to input lesson plan information; the lesson plan information includes lesson plan name, lesson plan focus, lesson plan difficulty, activity design, common wrong questions and personal teaching reflection;

[0018] The demand instruction input unit is used to process the lesson plan information, obtain a lesson plan generation instruction, and send the lesson plan generation instruction to the lesson plan generation module.

[0019] As an optional implementation, in the first aspect of the embodiment of the present invention, the student physiological signal acquisition module includes an electroencephalogram and electrocardiogram acquisition unit, a wireless Bluetooth headset, a camera and a wireless communication unit;

[0020] The electroencephalogram and electrocardiogram acquisition unit comprises a first dry electrode, a second dry electrode, an infrared light source and an infrared light source receiver;

[0021] The first dry electrode and the second dry electrode are data-connected to the wireless communication unit, and are used to collect the student's EEG information and send the EEG information to the multimodal signal processing module;

[0022] The infrared light source and the infrared light source receiver are data-connected to the wireless communication unit, and are used to collect the electrocardiogram information of the student, and send the electrocardiogram information to the multimodal signal processing module;

[0023] The wireless Bluetooth headset is used to provide prompt information to students;

[0024] The camera is data-connected to the wireless communication unit for collecting facial image information of students and sending the facial image information to the multimodal signal processing module.

[0025] A second aspect of an embodiment of the present invention discloses a teaching plan generation method based on emotion detection and a large language model, the method comprising:

[0026] S1, using the teacher module to process the lesson plan information, obtain a lesson plan generation instruction, and send the lesson plan generation instruction to the lesson plan generation module;

[0027] S2, using the student physiological signal acquisition module to collect the student's physiological signal information, and sending the physiological signal information to the multimodal signal processing module and the display module;

[0028] S3, using the multimodal signal processing module to process the physiological signal information to obtain an emotion detection result, and sending the emotion detection result to the teaching plan generation module;

[0029] S4, using the lesson plan generation module to process the lesson plan generation instruction and the emotion detection result to obtain a lesson plan customized according to demand;

[0030] S5, using the display module to display the lesson plan information, the physiological signal information, the emotion detection result and the customized lesson plan.

[0031] As an optional implementation, in the second aspect of the embodiment of the present invention, the using the multimodal signal processing module to process the physiological signal information to obtain the emotion detection result includes:

[0032] S31, processing the EEG information to obtain EEG characteristic information;

[0033] S32, processing the ECG information to obtain ECG characteristic information;

[0034] S33, processing the facial image information to obtain facial image feature information;

[0035] S34, performing feature fusion on the EEG feature information, the ECG feature information and the facial image feature information to obtain fused feature information;

[0036] S35, using a preset classifier to process the fused feature information to obtain an emotion detection result.

[0037] As an optional implementation, in the second aspect of the embodiment of the present invention, the processing of the EEG information to obtain EEG feature information includes:

[0038] S311, preprocessing the EEG information to obtain preprocessed EEG information;

[0039] S312, performing feature extraction on the preprocessed EEG information to obtain time domain feature information, spatial domain feature information and frequency domain feature information;

[0040] S313, fusing the time domain feature information, the spatial domain feature information, and the frequency domain feature information to obtain EEG fusion information;

[0041] S314, processing the EEG fusion information by using multi-scale convolution to obtain multi-scale EEG fusion information;

[0042] S315, using the residual multi-head attention mechanism, the multi-scale EEG fusion information is processed to obtain EEG feature information.

[0043] As an optional implementation, in the second aspect of the embodiment of the present invention, the processing of the electrocardiogram information to obtain electrocardiogram characteristic information includes:

[0044] S321, preprocessing the ECG information to obtain preprocessed ECG characteristic information;

[0045] S322, extracting features from the preprocessed ECG feature information to obtain heart rate variability information;

[0046] S323, processing the heart rate variability information by using multi-scale convolution to obtain multi-scale heart rate variability information;

[0047] S324, using the residual multi-head attention mechanism, the multi-scale heart rate variability information is processed to obtain ECG feature information.

[0048] As an optional implementation manner, in the second aspect of the embodiment of the present invention, the processing of the facial image information to obtain facial image feature information includes:

[0049] S331, preprocessing the facial image information to obtain preprocessed facial image feature information;

[0050] S332, extracting features from the pre-processed facial image feature information to obtain facial parameter information;

[0051] S333, processing the facial parameter information by using multi-scale convolution to obtain multi-scale facial parameter information;

[0052] S334, using the residual multi-head attention mechanism, the multi-scale facial parameter information is processed to obtain facial image feature information.

[0053] As an optional implementation manner, in the second aspect of the embodiment of the present invention, the preprocessing of the facial image information to obtain preprocessed facial image feature information includes:

[0054] S3311, performing face detection on the facial image information to obtain face information;

[0055] S3312, filtering the face information to obtain filtered face information;

[0056] S3313, performing key point detection on the filtered face information to obtain face key point information;

[0057] S3314, normalize the facial key point information to obtain pre-processed facial image feature information.

[0058] As an optional implementation, in the second aspect of the embodiment of the present invention, the feature fusion of the EEG feature information, the ECG feature information and the facial image feature information to obtain the fused feature information includes:

[0059] S341, fusing the EEG characteristic information and the ECG characteristic information to obtain first characteristic information;

[0060] S342, fusing the electrocardiogram feature information and the facial image feature information to obtain second feature information;

[0061] S343: Fuse the first feature information and the second feature information to obtain fused feature information.

[0062] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0063] (1) The present invention provides a teaching plan generation system and method based on emotion detection and a large language model. On the one hand, the system monitors the emotional state of students by collecting their multimodal physiological signals. On the other hand, the system inputs the teacher's personal teaching experience, reflections, and common mistakes made by students as instruction information into the large language model, and outputs intelligent teaching plans that are more targeted and more suitable for students' emotional states through training.

[0064] (2) The student physiological signal acquisition module of the present invention uses a lightweight, wireless and convenient acquisition device. The EEG and ECG share a set of equipment, which is convenient to wear and will not cause too much burden to students during class. Through the Bluetooth link, the real-time transmission and analysis of data are guaranteed. At the same time, the device integrates a wireless Bluetooth headset for listening to audio. The main function is to transmit the teacher's one-to-one audio reminder or encouraging voice information to students in bad emotional states, which not only protects the students' self-esteem but also can realize real-time reminders and guidance for students in bad states.

[0065] (3) The multimodal signal processing module integrates nerve center signals (EEG), peripheral nerve signals (ECG) and action behavior signals (facial images), extracts multi-domain signals such as time domain, frequency domain, and spatial domain, and uses multi-scale convolution, including the use of convolution kernels of different sizes, to capture patterns and details of different scales, and obtain richer and more effective features at the feature level. Then, it is processed through the residual multi-head self-attention mechanism. The residual connection allows the features to bypass the attention layer directly, reducing the problem of gradient disappearance. Finally, through global average pooling, redundant information is reduced to retain important emotional information. Through the above operations, the information of each single modality complements each other, improving the accuracy of emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0067] Figure 1 It is a structural schematic diagram of a teaching plan generation system based on emotion detection and a large language model disclosed in an embodiment of the present invention;

[0068] Figure 2 is a structural schematic diagram of a student physiological signal acquisition module disclosed in an embodiment of the present invention;

[0069] Figure 3 is a schematic diagram of the structure of a multimodal signal processing module disclosed in an embodiment of the present invention;

[0070] Figure 4 It is a schematic diagram of the structure of a teaching plan generation module disclosed in an embodiment of the present invention;

[0071] Figure 5 is a schematic diagram of the structure of a display module disclosed in an embodiment of the present invention;

[0072] Figure 6 It is a flow chart of a teaching plan generation method based on emotion detection and a large language model disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0073] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0074] The terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, device, product or equipment that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or equipment.

[0075] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0076] The present invention discloses a teaching plan generation system and method based on emotion detection and large language model, the system includes a teacher module, a student physiological signal acquisition module, a multimodal signal processing module, a teaching plan generation module and a display module; the teacher module is connected with the student physiological signal acquisition module, the teaching plan generation module and the display module for receiving teacher information; the teacher information includes teacher ID login information, lesson plan information and demand instruction information; the student physiological signal acquisition module is connected with the teacher module and the multimodal signal processing module for collecting the student's physiological signal information; the physiological signal information includes EEG information, ECG information and facial image information; the multimodal signal processing module is connected with the student physiological signal acquisition module and the teaching plan generation module for processing the physiological signal information to obtain a multimodal emotion detection result; the teaching plan generation module is connected with the teacher module, the display module and the multimodal signal processing module for generating teaching plans; the display module is connected with the teacher module and the teaching plan generation module for information display. The following are detailed descriptions.

[0077] Embodiment 1

[0078] See also Figure 1, Figure 1 : is a structural diagram of a teaching plan generation system based on emotion detection and large language model disclosed in an embodiment of the present invention. Figure 1 The described teaching plan generation system based on emotion detection and large language model is applied to the teaching plan generation technology field, and is used to automatically generate teaching plans through large language model by combining the detection of students' emotional state and teachers' personal experience, after-class reflection, activity design, class characteristics, teaching plan format and other requirements, and the embodiments of the present invention are not limited. Figure 1 As shown, the teaching plan generation system based on emotion detection and large language model may include the following operations:

[0079] The system includes a teacher module, a student physiological signal acquisition module, a multimodal signal processing module, a teaching plan generation module and a display module;

[0080] The teacher module is data-connected with the student physiological signal acquisition module, the lesson plan generation module, and the display module, and is used to receive teacher information; the teacher information includes teacher ID login information, lesson plan information, and demand instruction information;

[0081] The student physiological signal acquisition module is data-connected with the teacher module and the multimodal signal processing module to collect the student's physiological signal information; the physiological signal information includes EEG information, ECG information and facial image information;

[0082] The multimodal signal processing module is data-connected with the student physiological signal acquisition module and the teaching plan generation module, and is used to process the physiological signal information to obtain a multimodal emotion detection result;

[0083] The teaching plan generation module is data-connected with the teacher module, the display module, and the multimodal signal processing module to generate a teaching plan;

[0084] The display module is data-connected with the teacher module and the lesson plan generation module, and is used to display lesson plan information, physiological signal information, emotion detection results, and lesson plans customized according to demand.

[0085] Optionally, the teacher module includes a teacher ID login unit, a lesson plan information collection unit, and a demand instruction input unit;

[0086] The teacher ID login unit is used to authenticate the teacher;

[0087] The lesson plan information collection unit is used to input lesson plan information; the lesson plan information includes lesson plan name, lesson plan focus, lesson plan difficulty, activity design, common wrong questions and personal teaching reflection;

[0088] The demand instruction input unit is used to process the lesson plan information, obtain a lesson plan generation instruction, and send the lesson plan generation instruction to the lesson plan generation module.

[0089] Optionally, the student physiological signal acquisition module includes an electroencephalogram and electrocardiogram acquisition unit, a wireless Bluetooth headset, a camera and a wireless communication unit;

[0090] The electroencephalogram and electrocardiogram acquisition unit comprises a first dry electrode, a second dry electrode, an infrared light source and an infrared light source receiver;

[0091] The first dry electrode and the second dry electrode are data-connected to the wireless communication unit, and are used to collect the student's EEG information and send the EEG information to the multimodal signal processing module;

[0092] The infrared light source and the infrared light source receiver are data-connected to the wireless communication unit, and are used to collect the electrocardiogram information of the student, and send the electrocardiogram information to the multimodal signal processing module;

[0093] The wireless Bluetooth headset is used to provide prompt information to students;

[0094] The camera is data-connected to the wireless communication unit for collecting facial image information of students and sending the facial image information to the multimodal signal processing module.

[0095] Optionally, the teacher module can be used to input the basic information of the school class and teachers into the system in advance, and teachers can log in to the system according to their personal ID number before class. The personal teaching experience of teachers at different stages of professional development, the mastery of the course by previous students, and teaching reflections and other personal key teaching experiences and thoughts can be sent to the lesson plan generation module as instructions.

[0096] Figure 2The present invention is a schematic diagram of the structure of the student physiological signal acquisition module disclosed in the embodiment of the present invention; the EEG and ECG acquisition unit is equipped with an ear-hook EEG and ECG acquisition device, and two dry electrode leads at the mastoid position of the temporal bone are used to record EEG signals; the left and right positions of the tail of the device are respectively an infrared light source and an infrared light source receiver, which are used to record ECG signals; at the same time, the ear-hook EEG and ECG acquisition device is integrated with a wireless Bluetooth headset, which can be used for audio listening. The main function is to transmit the teacher's one-to-one audio reminder or encouraging voice information to students with poor cognitive status, aiming to remind and guide students in real time under the premise of protecting students' self-esteem. EEG signals contain important information such as emotional changes and cognitive load brain neural activity, and ECG signals provide important indicators about heart function and cardiovascular system status, such as heart rate, heart rhythm and ECG waveform, which are of great significance for emotional fluctuations and negative emotion detection. Facial image acquisition is achieved by capturing students' facial expressions using a high frame rate camera to obtain non-verbal behavior signals related to cognitive status. In summary, the three parts of the central nervous system, peripheral nervous system and facial expression work together to provide important physiological data for the system's cognitive status analysis. The student physiological signal acquisition module transmits the collected physiological signal information to the multimodal signal processing module through the wireless communication unit (wireless Bluetooth) for analysis and processing.

[0097] Figure 3 is a schematic diagram of the structure of a multimodal signal processing module disclosed in an embodiment of the present invention; Figure 4 It is a schematic diagram of the structure of the lesson plan generation module disclosed in an embodiment of the present invention. The information input by the teacher includes the course name, the teacher's personal teaching experience, the previous students' mastery of the course, teaching reflections, activity design, lesson plan format and other personal teaching experience and thinking instruction information for prompt word input. Through a large language model such as ChatGPT, lesson plans customized with students' emotional state, course information and teacher's personal teaching experience are generated.

[0098] Figure 5 It is a schematic diagram of the display module structure disclosed in an embodiment of the present invention, and the specific display content includes: 1) Basic information on the teacher side, including the teacher's name, class, subject and basic course information. 2) Search bar, used for outputting teacher instructions and retrieving lesson plan information. 3) Visualization of student cognitive state, used to display a line graph containing time variables of the emotional state and cognitive load state output by EEG, ECG and facial image signal classification, which helps teachers to grasp the cognitive state of students during the whole course on the one hand; on the other hand, it provides more prompt information to enable the large language model to learn more labels. 4) Generate new lesson plans. In order to realize a quick lesson plan generation system that can pay attention to students' emotional state and take into account teachers' experience reflection, it greatly improves the efficiency of teachers' lesson preparation and is an educational auxiliary tool to improve teaching effectiveness and students' learning experience.

[0099] Embodiment 2

[0100] See also Figure 6 , Figure 6 : is a flow chart of a teaching plan generation method based on emotion detection and large language model disclosed in an embodiment of the present invention. Figure 6 The described teaching plan generation method based on emotion detection and large language model is applied to the teaching plan generation technology field, and is used to automatically generate teaching plans through large language model by combining the detection of students' emotional state and teachers' personal experience, after-class reflection, activity design, class characteristics, teaching plan format and other requirements, and the embodiments of the present invention are not limited. Figure 6 As shown, the teaching plan generation method based on emotion detection and large language model may include the following operations:

[0101] S1, using the teacher module to process the lesson plan information, obtain a lesson plan generation instruction, and send the lesson plan generation instruction to the lesson plan generation module;

[0102] S2, using the student physiological signal acquisition module to collect the student's physiological signal information, and sending the physiological signal information to the multimodal signal processing module and the display module;

[0103] S3, using the multimodal signal processing module to process the physiological signal information to obtain an emotion detection result, and sending the emotion detection result to the teaching plan generation module;

[0104] S4, using the lesson plan generation module to process the lesson plan generation instruction and the emotion detection result to obtain a lesson plan customized according to demand;

[0105] S5, using the display module to display the lesson plan information, the physiological signal information, the emotion detection result and the customized lesson plan.

[0106] Optionally, the using the multimodal signal processing module to process the physiological signal information to obtain an emotion detection result includes:

[0107] S31, processing the EEG information to obtain EEG characteristic information;

[0108] S32, processing the ECG information to obtain ECG characteristic information;

[0109] S33, processing the facial image information to obtain facial image feature information;

[0110] S34, performing feature fusion on the EEG feature information, the ECG feature information and the facial image feature information to obtain fused feature information;

[0111] S35, using a preset classifier to process the fused feature information to obtain an emotion detection result.

[0112] Optionally, the processing of the EEG information to obtain EEG feature information includes:

[0113] S311, preprocessing the EEG information to obtain preprocessed EEG information;

[0114] Preprocessing includes: bandpass filtering and downsampling to retain effective information and remove noise and high-frequency electromyography interference;

[0115] The noise removal method is as follows: the signal to be processed is converted into the time-frequency domain using STFT (Short Time Fourier Transform), the window length used is 256 samples, the overlap rate is 75%, and the Hamming window is used. By discarding the frequency samples corresponding to negative frequencies, the length of the spectrum vector can be reduced to 129. A fully convolutional network consisting of 16 convolutional layers is defined. The size of the convolution kernel of the first layer is 9×8, with 18 convolution kernels. The second to 13th convolutional layers are groups of 3 layers repeated 4 times, with convolution kernel widths of 5, 9, and 9, respectively, and the number of convolution kernels is 64. The convolution kernel width of the last convolution layer is 129 and there is only 1 convolution kernel. In this network, convolution is performed only in the frequency dimension direction, and for all layers except the first layer, the convolution kernel width along the time dimension is set to 1. The convolution layer is followed by a batch normalization layer and a ReLU activation function layer, and finally output through a regression layer.

[0116] A convolutional neural network is established to complete the mapping F between the STFT spectrum X(m, k) of the signal to be processed and the STFT spectrum S(m, k) of the clean signal to be processed, so as to achieve the purpose of denoising. The input prediction variable signal and the network target signal are the amplitude spectrum of the received noisy signal to be processed and the clean signal to be processed respectively. The goal of the network is to learn the mapping F so that the network outputs the amplitude spectrum of the noisy signal to be processed. The regression network uses the prediction variable input to minimize the mean square error between its output and the input target. Therefore, the loss function of the denoising network can be expressed as

[0117]

[0118] Among them, B represents the batch size, M and K represent the total number of STFT time frames and frequency points respectively. Finally, the denoised signal to be processed is converted back to the time domain signal using the amplitude spectrum output by the network and the phase of the received signal to be processed.

[0119] S312, performing feature extraction on the preprocessed EEG information to obtain time domain feature information, spatial domain feature information and frequency domain feature information;

[0120] The frequency domain feature information is the power spectral density, which is analyzed in five frequency bands: δ (1-3 Hz), θ (4-7 Hz), α (8-13 Hz), β (14-30 Hz), γ (31-50 Hz). For each trial, the EEG signal is first divided into m non-overlapping short segments, each of which is 5 seconds. The power spectral density is extracted independently from these short segments, and all short segments are averaged. The calculation formula of power spectral density is:

[0121]

[0122] Where N represents the time sampling point in the signal x(n).

[0123] The spatial feature information is the phase-locked value. The formula for calculating the phase-locked value between signals a and b is:

[0124]

[0125] Where i represents the imaginary unit, φ represents the instantaneous phase, and N represents the total number of time sampling points.

[0126] The time-domain feature information is microstates, which are extracted using the EEGLAB Microstate 1.0 plugin. The multi-channel EEG signal is decomposed into a series of transient potential topologies. The global field potential (GFP) is obtained by calculating the standard deviation of the signals on all electrodes. The peak point of the GFP curve represents the moment with the highest signal-to-noise ratio, and the potential of each electrode at that moment is recorded. Then, the agglomerative hierarchical clustering method or K-means clustering algorithm is used to divide the topological map collection into eight microstates. The statistical characteristics of these microstates, such as duration, number of occurrences, contribution, and transition probability, are used for subsequent analysis and classification.

[0127] S313, fusing the time domain feature information, the spatial domain feature information, and the frequency domain feature information to obtain EEG fusion information;

[0128] Optional, fusion method is:

[0129] Get the cluster feature information X and cluster feature information Y to be fused; X is an n′×m′ dimensional matrix, Y is a p×m′ dimensional matrix, m′ represents the number of samples, n′ and p represent the dimensions of the two features, project the two matrices to 1 dimension for linear representation, corresponding to the projection vector a respectively 1 and b 1 , then the feature matrix after projection becomes:

[0130]

[0131] The purpose is to maximize the correlation coefficient between X' and Y', so as to obtain the projection vector a when the correlation coefficient is maximum. 1 and b1 ,Right now:

[0132]

[0133] Before projection, the data is standardized. The purpose of standardization is to make the mean of the data 0 and the variance 1. We can get:

[0134] cov(X',Y')=cov(a 1 T X,b 1 T Y)=E(<a 1 T X,b 1 T Y>)=E((a 1 T X)(b 1 T Y) T )=a 1 T E(XY T )b 1

[0135]

[0136]

[0137] D(X)=cov(X,X)=E(X T X)

[0138] D(Y)=cov(Y,Y)=E(Y T Y)

[0139] cov(X,Y)=E(XY T ),cov(Y,X)=E(YX T )

[0140] S XX =cov(X,X), then the solution objective is transformed into:

[0141]

[0142] Step 1: Calculate the variance S of X and Y XX and S YY , the covariance S of XY and YX XY =S YX T ;

[0143] Step 2: Calculate the matrix

[0144] Step 3: Solve the singular values ​​of M′ to obtain the largest singular value and its preceding and following singular vectors u, v;

[0145] Step 4: Projection vector a of X and Y 1 and b 1 They are:

[0146]

[0147] Step 5: The fusion feature vector Z of X and Y is

[0148] S314, processing the EEG fusion information by using multi-scale convolution to obtain multi-scale EEG fusion information;

[0149] S315, using the residual multi-head attention mechanism, the multi-scale EEG fusion information is processed to obtain EEG feature information.

[0150] Optionally, the processing the ECG information to obtain ECG characteristic information includes:

[0151] S321, preprocessing the ECG information to obtain preprocessed ECG characteristic information;

[0152] Preprocessing includes: bandpass filtering and downsampling to retain effective information and remove noise and high-frequency electromyography interference;

[0153] S322, extracting features from the preprocessed ECG feature information to obtain heart rate variability information;

[0154] The heart rate variability information is a combination of the standard deviation of sinus intervals (SDNN), the root mean square of the differences between adjacent RR intervals (RMSSD), the standard deviation of the differences between adjacent RR intervals (SDSD), the number of adjacent RR intervals with a length greater than 50 ms (NN50), and the ratio of NN50 to the number of all RR intervals (pNN50), which can be expressed as:

[0155] P = k 1 p 1 +k 2 p 2 +k 3 p 3 +k 4 p 4 +k 5p 5

[0156] Among them, P is the heart rate variability information, p 1 is the standard deviation of the sinus interval, p 2 is the root mean square of the difference between the lengths of adjacent RR intervals, p 3 is the standard deviation of the difference between the lengths of adjacent RR intervals (the length between two adjacent R waves on the electrocardiogram), p 4 is the number of adjacent RR intervals longer than 50ms, p 5 is the ratio of NN50 to the number of all RR intervals, k 1 ,k 2 ,k 3 ,k 4 ,k 5 is the weighting coefficient, the sum of the weighting coefficient is 1, the value of the weighting coefficient is determined by experiment and is used to adjust the weight of each parameter.

[0157] The calculation formulas for SDNN, RMSSD and SDSD are:

[0158]

[0159]

[0160]

[0161] Where N is the number of RR intervals, i represents the length of the i-th RR interval, Represents the average length of the RR interval.

[0162] S323, processing the heart rate variability information by using multi-scale convolution to obtain multi-scale heart rate variability information;

[0163] S324, using the residual multi-head attention mechanism, the multi-scale heart rate variability information is processed to obtain ECG feature information.

[0164] Optionally, feature extraction is performed on the preprocessed ECG feature information to obtain heart rate variability information:

[0165] Perform EMD decomposition on the preprocessed ECG feature information to obtain the IMF component c i (t), i=1, 2, ..., N, N is the number of IMF components; the IMF components are heart rate variability information;

[0166] The normalization function is Among them, x max is the maximum value of data information, x minis the minimum value of data information, data information is x(t), and x'(t) is the normalized data information;

[0167] The preprocessed ECG feature information is decomposed into IMFs with different time scales, simple components and relatively stable. All the maximum and minimum points of x'(t) are determined, and the corresponding upper envelope e is obtained by polynomial interpolation. max (t) and the lower envelope e min (t), and the mean value m(t)=(e min (t)+e max (t)) / 2, extract the detail h(t) according to x'(t) and the mean m(t), let h(t) = x'(t)-m(t), if h(t) conforms to IMF, it is the first modal component IMF1 (denoted as c 1 (t)); otherwise, let h(t) = x'(t), and continue to decompose, and finally decompose into Among them, r(t) is the remainder.

[0168] Optionally, the processing the facial image information to obtain facial image feature information includes:

[0169] S331, preprocessing the facial image information to obtain preprocessed facial image feature information;

[0170] S332, extracting features from the pre-processed facial image feature information to obtain facial parameter information;

[0171] Optionally, facial image features are extracted by calculating optical flow features and frame difference features between a peak frame and previous frames frame by frame, and performing feature fusion.

[0172] S333, processing the facial parameter information by using multi-scale convolution to obtain multi-scale facial parameter information;

[0173] S334, using the residual multi-head attention mechanism, the multi-scale facial parameter information is processed to obtain facial image feature information.

[0174] Optionally, the preprocessing the facial image information to obtain preprocessed facial image feature information includes:

[0175] S3311, performing face detection on the facial image information to obtain face information;

[0176] S3312, filtering the face information to obtain filtered face information;

[0177] S3313, performing key point detection on the filtered face information to obtain face key point information;

[0178] S3314, normalize the facial key point information to obtain pre-processed facial image feature information.

[0179] Face detection, filtering, key point detection, and normalization processing are existing technologies in this field.

[0180] Optionally, the performing feature fusion on the EEG feature information, the ECG feature information and the facial image feature information to obtain fused feature information includes:

[0181] S341, fusing the EEG characteristic information and the ECG characteristic information to obtain first characteristic information;

[0182] S342, fusing the electrocardiogram feature information and the facial image feature information to obtain second feature information;

[0183] S343: Fuse the first feature information and the second feature information to obtain fused feature information.

[0184] In order to further extract features with higher correlation with emotions, a deep learning approach is adopted. Multi-scale convolution includes the use of convolution kernels of different sizes to capture patterns and details at different scales, which helps to extract richer features and better characterize emotional states. The residual multi-head self-attention mechanism is used to further process the convolved features. The residual multi-head self-attention mechanism includes: residual connection and normalization. The residual connection allows features to bypass the attention layer directly, reducing the problem of gradient disappearance. Normalization stabilizes the training process and prevents numerical overflow. In addition, the mechanism also includes a fully connected layer for adjusting and reshaping features. The residual multi-head self-attention mechanism can enhance the representation of features and make them more suitable for emotion recognition tasks. Then, the processed features are globally averaged pooled, and the features are compressed by calculating the average value on each feature dimension to achieve the purpose of dimensionality reduction. Finally, the features are input into the support vector machine to segment the data by finding the optimal hyperplane that maximizes the distance between classes, thereby achieving the purpose of classification.

[0185] The specific process is as follows: First, the original EEG signal, ECG signal and facial image are preprocessed by filtering, downsampling and denoising to reduce redundant information while retaining key emotional features. 1 ,x 2 ,…,x n}, there are n input vectors x 1 ,x 2 ,…,x n , each vector has m channels. Multi-scale convolution can be expressed as:

[0186]

[0187] Among them, y i is the i-th output feature, w ij is the weight of the convolution kernel, * is the convolution operation, p j For each modal feature input, convolution kernels of different sizes are used to capture features of different scales.

[0188] Then the residual multi-head self-attention mechanism is used to output the multi-scale convolution features. 1 ,x 2 ,…,x n}∈R N×d , as the input of the residual multi-head self-attention mechanism, where N is the total number of features and d is x i The length of , 1≤i≤N. The attention is calculated by the following formula:

[0189]

[0190]

[0191]

[0192] Then connect all A together and divide each matrix X into subspaces, where X = (X 1 ,X 2 …,X h ,…,X H ), 1≤h≤H,X h represents the subspace, H represents the number of heads, in this embodiment H=3, in each subspace h, calculate the attention A h :

[0193]

[0194] Then connect all H together to produce the final multi-head attention MHA output:

[0195]

[0196] Then, the output of the previous layer is added to the input of this layer through the residual connection, and then normalized. It can be expressed as: Concat represents the concatenation of the multi-head attention mechanism features; layerNorm(x+Sublayer(x)), where layerNorm is normalization and Sublayer is the sublayer in the mechanism.

[0197] Then the output features are concatenated and global average pooling is performed to compress the features by calculating the average value on each feature dimension. It can be expressed as:

[0198]

[0199] Among them, y i is the i-th output feature, m is the number of channels of the input feature, x ij is the i-th element of the j-th channel of the input feature. Global average pooling helps to reduce redundant information and retain important emotional information. The output feature is then input into the linear classifier, and the formula is:

[0200] y=Wx+b

[0201] Where W is the weight matrix, x is the feature vector, b is the bias term, and y is the classification score. These scores are then processed by the softmax function to convert the original scores into probability distributions, thereby achieving 3 intensities (strong, medium, and weak) and 9 emotions (happy, sad, sleepy, calm, surprised, scared, angry, relaxed, and interesting), totaling 27 types of classification outputs.

[0202] The device embodiments described above are only illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, i.e., they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0203] Through the specific description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution can be essentially or partly contributed to the prior art in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0204] Finally, it should be noted that the teaching plan generation system and method based on emotion detection and large language model disclosed in the embodiment of the present invention only discloses the preferred embodiment of the present invention, which is only used to illustrate the technical solution of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions recorded in the aforementioned embodiments can still be modified, or some of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A teaching plan generation system based on emotion detection and large language model, characterized in that: The system includes a teacher module, a student physiological signal acquisition module, a multimodal signal processing module, a teaching plan generation module and a display module; The teacher module is data-connected with the student physiological signal acquisition module, the lesson plan generation module, and the display module, and is used to receive teacher information; the teacher information includes teacher ID login information, lesson plan information, and demand instruction information; The student physiological signal acquisition module is data-connected with the teacher module and the multimodal signal processing module to collect the student's physiological signal information; the physiological signal information includes EEG information, ECG information and facial image information; The multimodal signal processing module is data-connected with the student physiological signal acquisition module and the teaching plan generation module, and is used to process the physiological signal information to obtain a multimodal emotion detection result; The teaching plan generation module is data-connected with the teacher module, the display module, and the multimodal signal processing module to generate a teaching plan; The display module is data-connected with the teacher module and the lesson plan generation module, and is used to display lesson plan information, physiological signal information, emotion detection results, and customized lesson plans as required; The teaching plan generation method applied to the teaching plan generation system based on emotion detection and large language model comprises the following steps: S1, using the teacher module to process the lesson plan information, obtain a lesson plan generation instruction, and send the lesson plan generation instruction to the lesson plan generation module; S2, using the student physiological signal acquisition module to collect the student's physiological signal information, and sending the physiological signal information to the multimodal signal processing module and the display module; S3, using the multimodal signal processing module to process the physiological signal information to obtain an emotion detection result, and sending the emotion detection result to the teaching plan generation module, including: S31, processing the EEG information to obtain EEG feature information, including: S311, preprocessing the EEG information to obtain preprocessed EEG information; Preprocessing includes: bandpass filtering and downsampling to retain effective information and remove noise and high-frequency electromyography interference; The noise removal method is as follows: use STFT (short-time Fourier transform) to convert the signal to be processed into the time-frequency domain, use a window length of 256 samples, an overlap rate of 75%, and use a Hamming window; by discarding frequency samples corresponding to negative frequencies, the length of the spectrum vector can be reduced to 129; define a full convolutional network consisting of 16 convolutional layers, the size of the first convolution kernel is 9×8, there are 18 convolution kernels, the second to 13th convolutional layers are groups of 3 layers repeated 4 times, the convolution kernel widths are 5, 9, and 9 respectively, and the number of convolution kernels is 64, the convolution kernel width of the last convolution layer is 129 and there is only 1 convolution kernel. In this network, convolution is performed only in the frequency dimension direction, and for all layers except the first layer, the convolution kernel width along the time dimension is set to 1, and the convolution layer is followed by a batch normalization layer and a ReLU activation function layer, and finally output through a regression layer; A convolutional neural network is established to complete the mapping F between the STFT spectrum X(m, k) of the signal to be processed and the STFT spectrum S(m, k) of the clean signal to be processed, so as to achieve the purpose of denoising; the input prediction variable signal and the network target signal are the amplitude spectra of the received noisy signal to be processed and the clean signal to be processed respectively. The goal of the network is to learn the mapping F so that the network outputs the amplitude spectrum of the noisy signal to be processed. The regression network uses the prediction variable input to minimize the mean square error between its output and the input target. Therefore, the loss function of the denoising network can be expressed as: Among them, B represents the batch size, M and K represent the total number of STFT time frames and frequency points respectively; finally, the denoised signal to be processed is converted back to the time domain signal using the amplitude spectrum output by the network and the phase of the received signal to be processed; S312, performing feature extraction on the preprocessed EEG information to obtain time domain feature information, spatial domain feature information and frequency domain feature information; The frequency domain feature information is power spectral density, which is analyzed in five frequency bands: δ (1-3 Hz), θ (4-7 Hz), α (8-13 Hz), β (14-30 Hz), γ (31-50 Hz); for each trial, the EEG signal is first divided into m non-overlapping short segments, each of which is 5 seconds; the power spectral density is extracted independently from these short segments, and all short segments are averaged; the calculation formula of power spectral density is: Where N represents the time sampling point in the signal x(n); The spatial feature information is the phase-locked value. The formula for calculating the phase-locked value between signals a and b is: Where i represents the imaginary unit, φ represents the instantaneous phase, and N represents the total number of time sampling points; The time domain feature information is microstates, which are extracted using the EEGLAB Microstate 1.0 plugin; S313, fusing the time domain feature information, the spatial domain feature information, and the frequency domain feature information to obtain EEG fusion information; The fusion method is: Get the cluster feature information X and cluster feature information Y to be fused; X is an n′×m′ dimensional matrix, Y is a p×m′ dimensional matrix, m′ represents the number of samples, n′ and p represent the dimensions of the two features, project the two matrices to 1 dimension for linear representation, corresponding to the projection vectors a1 and b1 respectively, then the feature matrix after projection becomes: The purpose is to maximize the correlation coefficient between X' and Y', so as to obtain the projection vectors a1 and b1 when the correlation coefficient is maximum, that is: Before projection, the data is standardized. The purpose of standardization is to make the mean of the data 0 and the variance 1. We can get: cov(X',Y')=cov(a1 T X,b1 T Y)=E(<a1 T X,b1 T Y>)=E((a1 T X)(b1 T Y) T )=a1 T E(XY T )b1 D(X)=cov(X,X)=E(X T X) D(Y)=cov(Y,Y)=E(Y T Y) cov(X,Y)=E(XY T ),cov(Y,X)=E(YX T ) S XX =cov(X,X), then the solution objective is transformed into: Step 1: Calculate the variance S of X and Y XX and S YY , the covariance S of XY and YX XY =S YX T ; Step 2: Calculate the matrix Step 3: Solve the singular values ​​of M′ to obtain the largest singular value and its preceding and following singular vectors u, v; Step 4: The projection vectors a1 and b1 of X and Y are: Step 5: The fusion feature vector Z of X and Y is S314, processing the EEG fusion information by using multi-scale convolution to obtain multi-scale EEG fusion information; S315, using a residual multi-head attention mechanism, processing the multi-scale EEG fusion information to obtain EEG feature information; S32, processing the ECG information to obtain ECG characteristic information, including: S321, preprocessing the ECG information to obtain preprocessed ECG characteristic information; S322, extracting features from the preprocessed ECG feature information to obtain heart rate variability information; The heart rate variability information is a combination of the standard deviation of the sinus interval, the root mean square of the difference between the lengths of adjacent RR intervals, the standard deviation of the difference between the lengths of adjacent RR intervals, the number of adjacent RR intervals with a length greater than 50 ms (NN50), and the ratio of NN50 to the number of all RR intervals (pNN50), which can be expressed as: P=k1p1+k2p2+k3p3+k4p4+k5p5 Among them, P is the heart rate variability information, p1 is the standard deviation of the sinus interval, p2 is the root mean square of the difference between the lengths of adjacent RR intervals, p3 is the standard deviation of the difference between the lengths of adjacent RR intervals, p4 is the number of adjacent RR intervals longer than 50ms, p5 is the ratio of NN50 to the number of all RR intervals, k1, k2, k3, k4, k5 are weighting coefficients, the sum of the weighting coefficients is 1, and the value of the weighting coefficient is determined by experiment and is used to adjust the weights of various parameters; S323, processing the heart rate variability information by using multi-scale convolution to obtain multi-scale heart rate variability information; Multi-scale convolution can be expressed as: Among them, y i is the i-th output feature, w ij is the weight of the convolution kernel, * is the convolution operation, p j For each modal feature input, convolution kernels of different sizes are used to capture features of different scales; S324, using a residual multi-head attention mechanism, processing the multi-scale heart rate variability information to obtain electrocardiogram feature information; X={x1,x2,…,x n }∈R N×d , as the input of the residual multi-head self-attention mechanism, where N is the total number of features and d is x i The length of , 1≤i≤N; the attention is calculated by the following formula: Then connect all A together and divide each matrix X into subspaces, where X = (X 1 ,X 2 ...,X h ,...,X H ), 1≤h≤H,X h represents the subspace, H represents the number of heads, and in each subspace h, the attention A is calculated h : Then connect all H together to produce the final multi-head attention MHA output: Then, the output of the previous layer is added to the input of this layer through the residual connection, and then normalized; it can be expressed as: Concat represents the concatenation of the multi-head attention mechanism features; layerNorm(x+Sublayer(x)), where layerNorm is normalization and Sublayer is the sublayer in the mechanism; Then the output features are concatenated and global average pooling is performed to compress the features by calculating the average value on each feature dimension; it can be expressed as: Among them, y i is the i-th output feature, m is the number of channels of the input feature, x ij is the i-th element of the j-th channel of the input feature; global average pooling helps to reduce redundant information and retain important emotional information; the output feature is then input into the linear classifier, and the formula is: y=Wx+b Where W is the weight matrix, x is the feature vector, b is the bias term, and y is the classification score; S33, processing the facial image information to obtain facial image feature information, including: S331, preprocessing the facial image information to obtain preprocessed facial image feature information, including: S3311, performing face detection on the facial image information to obtain face information; S3312, filtering the face information to obtain filtered face information; S3313, performing key point detection on the filtered face information to obtain face key point information; S3314, normalizing the facial key point information to obtain pre-processed facial image feature information; S332, extracting features from the pre-processed facial image feature information to obtain facial parameter information; S333, processing the facial parameter information by using multi-scale convolution to obtain multi-scale facial parameter information; S334, using a residual multi-head attention mechanism, processing the multi-scale facial parameter information to obtain facial image feature information; S34, performing feature fusion on the EEG feature information, the ECG feature information and the facial image feature information to obtain fused feature information, including: S341, fusing the EEG characteristic information and the ECG characteristic information to obtain first characteristic information; S342, fusing the electrocardiogram feature information and the facial image feature information to obtain second feature information; S343, fusing the first feature information and the second feature information to obtain fused feature information; S35, using a preset classifier to process the fused feature information to obtain an emotion detection result; S4, using the lesson plan generation module to process the lesson plan generation instruction and the emotion detection result to obtain a lesson plan customized according to demand; S5, using the display module to display the lesson plan information, the physiological signal information, the emotion detection result and the customized lesson plan.

2. The teaching plan generation system based on emotion detection and large language model according to claim 1 is characterized in that: The teacher module includes a teacher ID login unit, a lesson plan information collection unit and a demand instruction input unit; The teacher ID login unit is used to authenticate the teacher; The lesson plan information collection unit is used to input lesson plan information; The demand instruction input unit is used to process the lesson plan information, obtain a lesson plan generation instruction, and send the lesson plan generation instruction to the lesson plan generation module.

3. The teaching plan generation system based on emotion detection and large language model according to claim 1, characterized in that: The student physiological signal acquisition module includes an EEG and ECG acquisition unit, a wireless Bluetooth headset, a camera and a wireless communication unit; The electroencephalogram and electrocardiogram acquisition unit comprises a first dry electrode, a second dry electrode, an infrared light source and an infrared light source receiver; The first dry electrode and the second dry electrode are data-connected to the wireless communication unit, and are used to collect the student's EEG information and send the EEG information to the multimodal signal processing module; The infrared light source and the infrared light source receiver are data-connected to the wireless communication unit, and are used to collect the electrocardiogram information of the student, and send the electrocardiogram information to the multimodal signal processing module; The wireless Bluetooth headset is used to provide prompt information to students; The camera is data-connected to the wireless communication unit for collecting facial image information of students and sending the facial image information to the multimodal signal processing module.

Images