A teaching quality evaluation method and device and a computer readable storage medium
By collecting and integrating multimodal data from students and teachers, and using a multimodal feature fusion classification model to assess teaching quality, the problem of low accuracy in existing methods is solved, and more accurate assessment results are provided to facilitate real-time adjustments to teaching methods.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA UNITED NETWORK COMM GRP CO LTD
- Filing Date
- 2022-10-31
- Publication Date
- 2026-05-12
AI Technical Summary
Existing teaching quality assessment methods often lack accuracy because they rely on only one modality of data or consider only student or teacher-side modal data.
Multimodal data from students and teachers, including brainwave data, behavioral data, and facial expression data, are collected. The data is then fused using a pre-trained multimodal feature fusion classification model to generate a teaching quality assessment score, which is then sent to the classroom terminal for display.
It enables more accurate assessment of teaching quality, helping teachers understand the teaching status in real time and make effective adjustments.
Smart Images
Figure CN115659221B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of teaching technology, and in particular to a method, apparatus, and computer-readable storage medium for evaluating teaching quality. Background Technology
[0002] The evaluation of teachers' teaching quality is a key issue of concern in the education sector, and using big data, AI algorithms, and other methods to analyze various data related to teachers' teaching quality is a development direction that the education sector is actively exploring.
[0003] However, existing teaching quality assessment methods are prone to inaccurate results because they use only one modality of data or only consider modal data from the student or teacher side. Summary of the Invention
[0004] The technical problem to be solved by the present invention is to address the above-mentioned shortcomings of the prior art by providing a method, apparatus and computer-readable storage medium for evaluating teaching quality, so as to solve the problem that the existing teaching quality evaluation methods are prone to low accuracy in evaluation results because they only use one modality data or only consider the modality data of students or teachers.
[0005] In a first aspect, the present invention provides a method for evaluating teaching quality, comprising:
[0006] Collect multimodal data of students and teachers within a preset time period, wherein the multimodal data includes student brainwave data, student behavior data, teacher behavior data, student facial expression data, and teacher facial expression data;
[0007] The multimodal data is input into a pre-trained multimodal feature fusion classification model to obtain the predicted teaching quality assessment score.
[0008] The teaching quality assessment score is sent to the classroom terminal so that the classroom terminal receives and displays the predicted teaching quality assessment score.
[0009] Furthermore, after collecting multimodal data of students and teachers within a preset time period, the method further includes:
[0010] The multimodal data is aligned based on the collection timestamp;
[0011] The step of inputting the multimodal data into a pre-trained multimodal feature fusion classification model to obtain a predicted teaching quality assessment score specifically includes:
[0012] The aligned multimodal data is input into a pre-trained multimodal feature fusion classification model to obtain the predicted teaching quality assessment score.
[0013] Furthermore, before inputting the multimodal data into a pre-trained multimodal feature fusion classification model to obtain the predicted teaching quality assessment score, the method further includes:
[0014] The multimodal data of students and teachers during in-class quizzes are segmented according to a certain time step;
[0015] The multimodal feature fusion classification model is trained based on the segmented multimodal data and the corresponding teaching quality score results to obtain the trained multimodal feature fusion classification model.
[0016] Furthermore, the trained multimodal feature fusion classification model includes a first layer, a second layer, a third layer, a fourth layer, a multimodal feature fusion layer, and a fully connected layer;
[0017] The outputs of the first, second, third, and fourth layers are respectively connected to the input of the multimodal feature fusion layer, and the output of the multimodal feature fusion layer is connected to the input of the fully connected layer.
[0018] The step of inputting the multimodal data into a pre-trained multimodal feature fusion classification model to obtain a predicted teaching quality assessment score specifically includes:
[0019] The student's brainwave data is input into the first layer, and the student's brainwave features are extracted through the first layer.
[0020] The student behavior data and teacher behavior data are input into the second layer, and the fusion features of student behavior and teacher behavior are extracted through the second layer;
[0021] The student facial expression data is input into the third layer, and the teacher behavior and student facial expression fusion features are extracted through the third layer.
[0022] The teacher's facial expression data is input into the fourth layer, and the teacher's facial expression features are extracted through the fourth layer;
[0023] The features extracted from the first, second, third, and fourth layers are input into the multimodal feature fusion layer for normalization processing;
[0024] The normalized features are input into the fully connected layer for full connection to obtain the predicted teaching quality assessment score.
[0025] Furthermore, the first layer includes an activation function layer and a normalization layer. The step of inputting the student's brainwave data into the first layer and extracting the student's brainwave features through the first layer specifically includes:
[0026] The 1*n-dimensional student brainwave data is input into the activation function layer for activation processing, and the output result of the activation function layer is obtained.
[0027] The output of the activation function layer is input into the normalization layer for normalization processing to obtain a normalized 1*n-dimensional feature, and the normalized 1*n-dimensional feature is used as the student's brainwave feature.
[0028] Furthermore, the step of inputting the student behavior data and teacher behavior data into the second layer, and extracting the fusion features of student behavior and teacher behavior through the second layer, specifically includes:
[0029] The 10*m*n dimensional student behavior data and the 10*m*n dimensional teacher behavior data are input into the second layer of the Hourglass model to detect human key points and obtain the detection results of human key points.
[0030] The detection results of the human body key points are input into a recurrent neural network (RNN) for behavior recognition, resulting in 1*n-dimensional student behavior features and 1*n-dimensional teacher behavior features.
[0031] The L2 paradigm is used to fuse the 1*n-dimensional student behavior features and the 1*n-dimensional teacher behavior features to obtain 1*n-dimensional fused behavior features.
[0032] Furthermore, the step of inputting the student facial expression data into the third layer and extracting the fusion features of teacher behavior and student facial expressions through the third layer specifically includes:
[0033] The 10*m*n dimensional student facial expression data is input into the third layer, and the facial expression features are extracted using the convolutional neural network (CNN) model shared between the third and fourth layers to obtain 1*n dimensional student facial expression features.
[0034] Based on prior knowledge, the 1*n-dimensional student facial expression features and the 1*n-dimensional teacher behavior features are fused to obtain 1*n-dimensional teacher behavior and student facial expression fusion features.
[0035] Furthermore, the step of inputting the teacher's facial expression data into the fourth layer and extracting the teacher's facial expression features through the fourth layer specifically includes:
[0036] The 10*m*n dimensional teacher facial expression data is input into the fourth layer, and the CNN model is used to extract facial expression features to obtain 1*n dimensional teacher facial expression features.
[0037] Furthermore, the formula for the normalization process is:
[0038] feature fusion=[λ1d1,λ2(1-d2),λ3(1-d3),λ4d4]
[0039] Where, d i For the features extracted from the i-th layer, λ i Let be the training weights corresponding to the i-th layer.
[0040] Furthermore, the features normalized by the multimodal feature fusion layer are 1*4n dimensional multimodal fusion features. The step of inputting the normalized features from the multimodal feature fusion layer into the fully connected layer for full connection to obtain the predicted teaching quality assessment score specifically includes:
[0041] The 1*4n-dimensional multimodal fusion features are input into a 4n*11-dimensional fully connected layer for full connection to obtain the predicted 1*11-dimensional teaching quality assessment score.
[0042] Secondly, the present invention provides a teaching quality assessment device, comprising:
[0043] The modal data acquisition module is used to collect multimodal data of students and teachers within a preset time period. The multimodal data includes student brainwave data, student behavior data, teacher behavior data, student facial expression data, and teacher facial expression data.
[0044] The evaluation score prediction module is connected to the modal data acquisition module and is used to input the multimodal data into a pre-trained multimodal feature fusion classification model to obtain the predicted teaching quality evaluation score.
[0045] A receiving and display module, connected to the assessment score prediction module, is used to send the teaching quality assessment score to the classroom terminal so that the classroom terminal can receive and display the predicted teaching quality assessment score.
[0046] Thirdly, the present invention provides a teaching quality assessment device, comprising a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to implement the teaching quality assessment method described in the first aspect above.
[0047] Fourthly, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the teaching quality assessment method described in the first aspect above.
[0048] The present invention provides a teaching quality assessment method, device, and computer-readable storage medium that collects multimodal data of students and teachers within a preset time period. This multimodal data includes student brainwave data, student behavior data, teacher behavior data, student facial expression data, and teacher facial expression data. The multimodal data is then input into a pre-trained multimodal feature fusion classification model to obtain highly accurate predicted teaching quality assessment scores. Furthermore, by sending these scores to a classroom terminal, the terminal receives and displays the predicted scores, enabling teachers to easily obtain real-time information on the teaching status and make effective adjustments to their teaching methods. This addresses the problem that existing teaching quality assessment methods often suffer from low accuracy due to the use of only one modality of data or consideration of only one side of the student or teacher's modality. Attached Figure Description
[0049] Figure 1 This is a flowchart of a teaching quality assessment method according to Embodiment 1 of the present invention;
[0050] Figure 2 This is a schematic diagram of the segmentation during the training phase of the multimodal feature fusion classification model according to an embodiment of the present invention;
[0051] Figure 3 This is a schematic diagram of the structure of the multimodal feature fusion classification model according to an embodiment of the present invention;
[0052] Figure 4 This is a schematic diagram of the structure of a teaching quality assessment device according to Embodiment 2 of the present invention;
[0053] Figure 5 This is a schematic diagram of the structure of a teaching quality assessment device according to Embodiment 3 of the present invention. Detailed Implementation
[0054] To enable those skilled in the art to better understand the technical solution of the present invention, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.
[0055] It is understood that the specific embodiments and accompanying drawings described herein are merely for explaining the invention and are not intended to limit the invention.
[0056] It is understood that, without conflict, the various embodiments and features in the embodiments of the present invention can be combined with each other.
[0057] It is understood that, for ease of description, only the parts related to the present invention are shown in the accompanying drawings, while the parts unrelated to the present invention are not shown in the drawings.
[0058] It is understood that each unit or module involved in the embodiments of the present invention may correspond to only one entity structure, or may be composed of multiple entity structures, or multiple units or modules may be integrated into one entity structure.
[0059] It is understood that, without conflict, the functions and steps marked in the flowcharts and block diagrams of this invention may occur in a different order than that marked in the accompanying drawings.
[0060] It is understood that the flowcharts and block diagrams of this invention illustrate the possible architecture, functions, and operations of systems, apparatuses, devices, and methods according to various embodiments of this invention. Each block in the flowchart or block diagram may represent a unit, module, program segment, or code, containing executable instructions for implementing the specified function. Furthermore, each block or combination of blocks in the block diagram and flowchart can be implemented using a hardware-based system to achieve the specified function, or using a combination of hardware and computer instructions.
[0061] It is understood that the units and modules involved in the embodiments of the present invention can be implemented by software or by hardware. For example, the units and modules can be located in a processor.
[0062] Example 1:
[0063] This embodiment provides a method for evaluating teaching quality, such as... Figure 1 As shown, the method includes:
[0064] Step S101: Collect multimodal data of students and teachers within a preset time period.
[0065] In this embodiment, the multimodal data includes student brainwave data, student behavior data, teacher behavior data, student facial expression data, and teacher facial expression data.
[0066] Specifically, students' facial expression data can be collected by cameras installed on student terminals, and the output is an image of each student's facial expression. Teachers' facial expression data can be collected by tracking cameras installed at the back of the classroom, and the output is an image of the teacher's facial expression.
[0067] Specifically, each student wears a sensor device that collects the student's brainwave data, and the output is a brainwave activity value for each student.
[0068] Specifically, cameras placed at the front and back of the classroom can collect the behavior of students and teachers respectively, and output full-body / half-body images of students and teachers.
[0069] Optionally, after collecting multimodal data of students and teachers within a preset time period, the method further includes:
[0070] The multimodal data is aligned based on the collection timestamp.
[0071] In this embodiment, after collecting and storing multimodal data of students and teachers within a certain period of time, the data of each modality is aligned in terms of time dimension according to the collection timestamp.
[0072] Step S102: Input the multimodal data into the pre-trained multimodal feature fusion classification model to obtain the predicted teaching quality assessment score.
[0073] Specifically, the aligned multimodal data can be input into a pre-trained multimodal feature fusion classification model to obtain the predicted teaching quality assessment score.
[0074] Optionally, before inputting the multimodal data into a pre-trained multimodal feature fusion classification model to obtain the predicted teaching quality assessment score, the method further includes:
[0075] The multimodal data of students and teachers during in-class quizzes are segmented according to a certain time step;
[0076] The multimodal feature fusion classification model is trained based on the segmented multimodal data and the corresponding teaching quality score results to obtain the trained multimodal feature fusion classification model.
[0077] In this embodiment, during the model training phase, students will be given in-class quizzes. The teacher's teaching quality will be scored based on the students' quiz scores, and these scores will be used as training labels. The segmentation diagram for the multimodal feature fusion classification model training phase is shown below. Figure 2 As shown, the facial expression data includes facial expression data of teachers and students, and the behavioral data includes behavioral data of teachers and students, such as... Figure 2 As shown, each modal data is segmented with a certain time step (10s is used in this example). The quiz results are divided according to the actual time of the teacher's class. For example, if the first knowledge point is taught within x minutes of the start of class, the test result corresponding to this knowledge point, that is, the teaching quality score result (90 in this example), is the label of the first m corresponding time slices. The test result of 85 at the next y minutes is the label of the n corresponding time slices starting from the (m+1)th slice.
[0078] Optionally, the trained multimodal feature fusion classification model includes a first layer, a second layer, a third layer, a fourth layer, a multimodal feature fusion layer, and a fully connected layer;
[0079] The outputs of the first, second, third, and fourth layers are respectively connected to the input of the multimodal feature fusion layer, and the output of the multimodal feature fusion layer is connected to the input of the fully connected layer.
[0080] The step of inputting the multimodal data into a pre-trained multimodal feature fusion classification model to obtain a predicted teaching quality assessment score specifically includes:
[0081] The student's brainwave data is input into the first layer, and the student's brainwave features are extracted through the first layer.
[0082] The student behavior data and teacher behavior data are input into the second layer, and the fusion features of student behavior and teacher behavior are extracted through the second layer;
[0083] The student facial expression data is input into the third layer, and the teacher behavior and student facial expression fusion features are extracted through the third layer.
[0084] The teacher's facial expression data is input into the fourth layer, and the teacher's facial expression features are extracted through the fourth layer;
[0085] The features extracted from the first, second, third, and fourth layers are input into the multimodal feature fusion layer for normalization processing;
[0086] The normalized features are input into the fully connected layer for full connection to obtain the predicted teaching quality assessment score.
[0087] In this embodiment, the structure of the multimodal feature fusion classification model can be as follows: Figure 3 As shown, the architecture mainly includes a first layer, a second layer, a third layer, a fourth layer, a multimodal feature fusion layer, and a fully connected layer. The first layer extracts student brainwave features, which can be considered a characteristic representing student attention in class. The second layer extracts a fusion feature of student and teacher behavior, which can be considered a characteristic representing the quality of student-teacher interaction. The third layer extracts a fusion feature of teacher behavior and student facial expressions. Based on prior knowledge, the teacher's teaching behavior is likely to affect students' classroom emotions; therefore, this fusion feature can also be considered a characteristic representing the atmosphere of student-teacher classroom interaction. The fourth layer extracts teacher facial expression features separately. Finally, the features extracted from the four layers are concatenated and normalized into a 1*4n feature vector through a multimodal feature fusion layer, and then passed through a 4n*classes fully connected layer to obtain the output vector, where classes is the number of categories, set to 11 here, representing a score of 0-10 for the teacher's teaching quality.
[0088] Optionally, the first layer includes an activation function layer and a normalization layer. The step of inputting the student's brainwave data into the first layer and extracting the student's brainwave features through the first layer specifically includes:
[0089] The 1*n-dimensional student brainwave data is input into the activation function layer for activation processing, and the output result of the activation function layer is obtained.
[0090] The output of the activation function layer is input into the normalization layer for normalization processing to obtain a normalized 1*n-dimensional feature, and the normalized 1*n-dimensional feature is used as the student's brainwave feature.
[0091] In one specific embodiment, the first layer is used to extract brainwave features from students, and the specific settings are as follows:
[0092] (1) The input for this part is the brainwave data of students collected within a 10s time slice, which is a 1*n dimensional vector, with n aligned with the n of all input data in other layers;
[0093] (2) The student brainwave data is input into the activation function layer, and the activation function used is ReLU;
[0094] (3) The results of the activation function layer are input to the normalization layer for normalization processing. The original input data brainwave values are distributed between 0 and 100 for quality inspection. After normalization, the data are mapped to between 0 and 1.
[0095] (4) The normalized 1*n-dimensional data is used as the student's brainwave features and input into the multimodal feature fusion layer.
[0096] Optionally, the step of inputting the student behavior data and teacher behavior data into the second layer, and extracting the fusion features of student behavior and teacher behavior through the second layer, specifically includes:
[0097] The 10*m*n dimensional student behavior data and the 10*m*n dimensional teacher behavior data are input into the second layer of the Hourglass model to detect human key points and obtain the detection results of human key points.
[0098] The detection results of the human body key points are input into a recurrent neural network (RNN) for behavior recognition, resulting in 1*n-dimensional student behavior features and 1*n-dimensional teacher behavior features.
[0099] The L2 paradigm is used to fuse the 1*n-dimensional student behavior features and the 1*n-dimensional teacher behavior features to obtain 1*n-dimensional fused behavior features.
[0100] In one specific embodiment, the second layer is used to extract the fusion features of student behavior and teacher behavior, and is specifically configured as follows:
[0101] (1) The input of this part is the student behavior data and teacher behavior data collected within a 10s time slice. Taking the frequency of sampling one frame per second as an example, each set of input data has 10 images. The input data are all paired inputs. That is, for each student, 10 half-body / full-body images are collected within 10s, and then paired with the collected half-body / full-body images of the teacher. Before input, all images are resized to m*n size and grayscale conversion is performed. Therefore, each pair of inputs consists of two 10*m*n third-order tensors.
[0102] (2) The tensor pairs input in step 1 are passed through the same human keypoint network to detect human keypoints (e.g., using the Hourglass model, or other models can be used depending on the situation).
[0103] It should be noted that the human keypoint detection algorithm is not limited to Hourglass; other similar algorithms that can extract human keypoints can also be used.
[0104] (3) Input the human key point detection results (still an image) output from step 2 into the RNN (Recurrent Neural Network) network for behavior recognition;
[0105] It should be noted that behavior recognition is not limited to RNN networks; LSTM or other recurrent neural networks that can extract time-series information features can also be used.
[0106] (4) After step 3, the behavioral characteristics of students and teachers are obtained respectively, which are 1*n feature vectors;
[0107] (5) The student and teacher behavioral features from step 4 are input into the behavioral feature fusion layer. Because, according to prior knowledge, the consistency between student and teacher behaviors is higher in the classroom, resulting in better learning outcomes, the fusion method here is to use the L2 paradigm to calculate the distance between the two feature vectors. The calculation formula is as follows:
[0108]
[0109] Where, x i Let y be the i-th value in the first feature. i For the i-th value in the second feature, d i Let be the i-th value of the fused behavioral feature.
[0110] The final output is a 1*n dimensional fusion behavior feature.
[0111] Optionally, the step of inputting the student facial expression data into the third layer and extracting the fusion features of teacher behavior and student facial expressions through the third layer specifically includes:
[0112] The 10*m*n dimensional student facial expression data is input into the third layer, and the facial expression features are extracted using the convolutional neural network (CNN) model shared between the third and fourth layers to obtain 1*n dimensional student facial expression features.
[0113] Based on prior knowledge, the 1*n-dimensional student facial expression features and the 1*n-dimensional teacher behavior features are fused to obtain 1*n-dimensional teacher behavior and student facial expression fusion features.
[0114] In one specific embodiment, the third layer is used to extract the fused features of teacher behavior and student facial expressions, and the specific settings are as follows:
[0115] (1) The input is a pair of data, in which the teacher behavior data reuses the data of the second layer and the final feature result, and the student expression data is the student face image captured after detection. It is also resized to m*n size to form a 10*m*n third-order tensor.
[0116] (2) Extract facial features from the student facial expression data in step 1 using a CNN (Convolutional Neural Networks) model and output a 1*n dimensional feature vector.
[0117] It should be noted that facial expression feature extraction is not limited to using CNN models; other models capable of extracting facial features can be used instead.
[0118] (3) Obtain the student facial expression features from step 2 and the teacher behavior features obtained in the second layer, and input them into the feature fusion layer;
[0119] (4) Based on prior knowledge, there is a correlation between teachers’ classroom behavior and students’ classroom emotions. Students’ classroom emotions are also related to teaching quality. Appropriate teacher behavior will have a positive emotional impact on students. Therefore, this feature fusion layer still uses the feature vector distance calculation method proposed in the second layer to obtain a 1*n-dimensional fusion feature of teacher behavior and student expression.
[0120] Optionally, the step of inputting the teacher's facial expression data into the fourth layer and extracting the teacher's facial expression features through the fourth layer specifically includes:
[0121] The 10*m*n dimensional teacher facial expression data is input into the fourth layer, and the CNN model is used to extract facial expression features to obtain 1*n dimensional teacher facial expression features.
[0122] In one specific embodiment, the fourth layer is used to extract the features of the teacher's facial expressions. The input of this layer is 10*m*n teacher facial expression data. It uses the same CNN model as the third layer to extract the teacher's facial expression features and outputs a 1*n-dimensional feature vector.
[0123] Optionally, the normalization formula is:
[0124] feature fusion =[λ1d1,λ2(1-d2),λ3(1-d3),λ4d4]
[0125] Where, d i For the features extracted from the i-th layer, λ i Let be the training weights corresponding to the i-th layer.
[0126] Optionally, the features after normalization processing by the multimodal feature fusion layer are 1*4n-dimensional multimodal fusion features. The step of inputting the normalized features from the multimodal feature fusion layer into the fully connected layer for full connection to obtain the predicted teaching quality assessment score specifically includes:
[0127] The 1*4n-dimensional multimodal fusion features are input into a 4n*11-dimensional fully connected layer for full connection to obtain the predicted 1*11-dimensional teaching quality assessment score.
[0128] In this embodiment, the features output from the four main layers are input into the multimodal feature fusion layer, and the feature fusion layer is calculated as follows:
[0129] feature fusion =[λ1d1,λ2(1-d2),λ3(1-d3),λ4d4]
[0130] Where, d i For the features output by the i-th layer of the model above, λ i To train the weights, since the second layer outputs fused behavioral features and the third layer outputs fused teacher behavior and student facial expressions, both of which are distance features between two sets of data, the closer the distance between the two features, the better the interaction between the student and the teacher. Therefore, the values in the feature vectors output by the second and third layers should be as small as possible. Thus, we subtract the feature vector from 1 and then multiply it by the weights, which transforms the result into a situation where the larger the value in the feature vector, the better, consistent with the first and fourth layers. The features after multiplying by the weights are then directly concatenated, ultimately outputting a 1*4n dimensional multimodal fusion feature.
[0131] The aforementioned 1*4n multimodal fusion features are input into a fully connected layer of size 4n*11, and output a final result of 1*11 dimensions. The median of this vector represents the confidence level of the teacher's teaching quality score from 0 to 10, and the highest confidence level is taken as the teacher's teaching quality score.
[0132] It should be noted that existing teaching quality assessment methods often rely on expert experience to integrate results when making the final judgment. For example, the behavioral result score X is weighted by 'a', and the attention result score Y is weighted by 'b', resulting in a final score of aX + bY. Here, 'a' and 'b' are pre-set based on expert experience or rules. However, while expert experience or rules are effective in most scenarios, real-world situations are constantly changing, and experts cannot consider all situations. Therefore, the pre-set weights often have biases. The embodiments of this invention perform feature-level fusion while the information of each modality is still in the feature stage, allowing the machine to learn the combination of different modality feature quality inspections on the basis of a large amount of data. This is more conducive to dealing with changing situations and can obtain highly accurate assessment results.
[0133] Step S103: Send the teaching quality assessment score to the classroom terminal so that the classroom terminal receives and displays the predicted teaching quality assessment score.
[0134] In this embodiment, the classroom terminal receives and displays the output of the multimodal feature fusion classification model. The teacher obtains the teaching status from the classroom terminal and adjusts the teaching methods accordingly. This invention utilizes five modalities of data—student brainwave data, student behavior data, teacher behavior data, student facial expression data, and teacher facial expression data—to fully consider the interaction between teacher and student classroom behavior and the impact of teacher classroom behavior on student emotions. Therefore, it can obtain highly accurate evaluation results.
[0135] The teaching quality assessment method provided in this invention collects multimodal data of students and teachers within a preset time period. This multimodal data includes student brainwave data, student behavior data, teacher behavior data, student facial expression data, and teacher facial expression data. The multimodal data is then input into a pre-trained multimodal feature fusion classification model to obtain a highly accurate predicted teaching quality assessment score. Furthermore, by sending the predicted teaching quality assessment score to a classroom terminal, the terminal receives and displays the score, allowing teachers to easily obtain real-time information on the teaching status and make effective adjustments to their teaching methods. This solves the problem that existing teaching quality assessment methods often suffer from low accuracy due to using only one modality of data or considering only one side of the student or teacher's modality.
[0136] Example 2:
[0137] like Figure 4 As shown, this embodiment provides a teaching quality assessment device for performing the above-described teaching quality assessment method, including:
[0138] The modal data acquisition module 11 is used to collect multimodal data of students and teachers within a preset time period. The multimodal data includes student brainwave data, student behavior data, teacher behavior data, student facial expression data, and teacher facial expression data.
[0139] The evaluation score prediction module 12 is connected to the modal data acquisition module 11 and is used to input the multimodal data into a pre-trained multimodal feature fusion classification model to obtain the predicted teaching quality evaluation score.
[0140] The receiving and display module 13 is connected to the evaluation score prediction module 12 and is used to send the teaching quality evaluation score to the classroom terminal so that the classroom terminal can receive and display the predicted teaching quality evaluation score.
[0141] Optionally, the device further includes:
[0142] The data alignment module is used to align the multimodal data according to the acquisition timestamp;
[0143] The evaluation score prediction module 12 is specifically used for:
[0144] The aligned multimodal data is input into a pre-trained multimodal feature fusion classification model to obtain the predicted teaching quality assessment score.
[0145] Optionally, the device further includes:
[0146] The data segmentation module is used to segment the multimodal data of students and teachers during in-class quizzes according to a certain time step.
[0147] The model training module is used to train the multimodal feature fusion classification model based on the segmented multimodal data and the corresponding teaching quality score results, so as to obtain the trained multimodal feature fusion classification model.
[0148] Optionally, the trained multimodal feature fusion classification model includes a first layer, a second layer, a third layer, a fourth layer, a multimodal feature fusion layer, and a fully connected layer;
[0149] The outputs of the first, second, third, and fourth layers are respectively connected to the input of the multimodal feature fusion layer, and the output of the multimodal feature fusion layer is connected to the input of the fully connected layer.
[0150] The assessment score prediction module 12 specifically includes:
[0151] The first processing unit is used to input the student's brainwave data into the first layer and extract the student's brainwave features through the first layer.
[0152] The second processing unit is used to input the student behavior data and teacher behavior data into the second layer, and extract the fusion features of student behavior and teacher behavior through the second layer;
[0153] The third processing unit is used to input the student facial expression data into the third layer and extract the fusion features of teacher behavior and student facial expression through the third layer.
[0154] The fourth processing unit is used to input the teacher's facial expression data into the fourth layer and extract the teacher's facial expression features through the fourth layer;
[0155] The fifth processing unit is used to input the features extracted from the first, second, third, and fourth layers into the multimodal feature fusion layer for normalization processing;
[0156] The sixth processing unit is used to input the normalized features into the fully connected layer for full connection to obtain the predicted teaching quality assessment score.
[0157] Optionally, the first layer includes an activation function layer and a normalization layer, and the first processing unit specifically includes:
[0158] An activation processing unit is used to input 1*n-dimensional student brainwave data into the activation function layer for activation processing and obtain the output result of the activation function layer.
[0159] The normalization processing unit is used to input the output of the activation function layer into the normalization layer for normalization processing to obtain a normalized 1*n-dimensional feature, and use the normalized 1*n-dimensional feature as the student's brainwave feature.
[0160] Optionally, the second processing unit specifically includes:
[0161] The key point detection unit is used to input 10*m*n dimensional student behavior data and 10*m*n dimensional teacher behavior data into the second layer of the Hourglass model to detect human key points and obtain the detection results of human key points.
[0162] The behavior recognition unit is used to input the detection results of the human body key points into the recurrent neural network (RNN) for behavior recognition, and obtain 1*n-dimensional student behavior features and 1*n-dimensional teacher behavior features respectively.
[0163] The behavior feature fusion unit is used to fuse the 1*n-dimensional student behavior features and the 1*n-dimensional teacher behavior features using the L2 paradigm to obtain 1*n-dimensional fused behavior features.
[0164] Optionally, the third processing unit specifically includes:
[0165] The facial expression feature extraction unit is used to input 10*m*n dimensional student facial expression data into the third layer, and use the convolutional neural network (CNN) model shared between the third and fourth layers to extract facial expression features, thereby obtaining 1*n dimensional student facial expression features.
[0166] The facial expression and behavior fusion unit is used to fuse the 1*n-dimensional student facial expression features and the 1*n-dimensional teacher behavior features based on prior knowledge to obtain 1*n-dimensional teacher behavior and student facial expression fusion features.
[0167] Optionally, the fourth processing unit is specifically used for:
[0168] The 10*m*n dimensional teacher facial expression data is input into the fourth layer, and the CNN model is used to extract facial expression features to obtain 1*n dimensional teacher facial expression features.
[0169] Optionally, the normalization formula is:
[0170] feature fusion =[λ1d1,λ2(1-d2),λ3(1-d3),λ4d4]
[0171] Where, d i For the features extracted from the i-th layer, λ i Let be the training weights corresponding to the i-th layer.
[0172] Optionally, the normalized features of the multimodal feature fusion layer are 1*4n-dimensional multimodal fusion features;
[0173] Optionally, the sixth processing unit is specifically used for:
[0174] The 1*4n-dimensional multimodal fusion features are input into a 4n*11-dimensional fully connected layer for full connection to obtain the predicted 1*11-dimensional teaching quality assessment score.
[0175] Example 3:
[0176] refer to Figure 5 This embodiment provides a teaching quality assessment device, including a memory 21 and a processor 22. The memory 21 stores a computer program, and the processor 22 is configured to run the computer program to execute the teaching quality assessment method in Embodiment 1.
[0177] The memory 21 is connected to the processor 22. The memory 21 can be a flash memory, a read-only memory or other memory, and the processor 22 can be a central processing unit or a microcontroller.
[0178] Example 4:
[0179] This embodiment provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the teaching quality evaluation method in Embodiment 1 above.
[0180] The computer-readable storage medium includes volatile or non-volatile, removable or non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, computer program modules, or other data). Computer-readable storage media include, but are not limited to, RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory or other memory technologies, CD-ROM (Compact Disc Read-Only Memory), DVD or other optical disc storage, cartridges, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer.
[0181] The teaching quality assessment method, apparatus, and computer-readable storage medium provided in Examples 2 to 4 collect multimodal data of students and teachers within a preset time period. This multimodal data includes student brainwave data, student behavior data, teacher behavior data, student facial expression data, and teacher facial expression data. The multimodal data is then input into a pre-trained multimodal feature fusion classification model to obtain highly accurate predicted teaching quality assessment scores. Furthermore, by sending these scores to a classroom terminal, the terminal receives and displays the predicted scores, allowing teachers to easily obtain real-time information on the teaching status and make effective adjustments to their teaching methods. This solves the problem that existing teaching quality assessment methods often suffer from low accuracy due to the use of only one modality of data or consideration of only one side of the student or teacher's modality.
[0182] It is understood that the above embodiments are merely exemplary implementations used to illustrate the principles of the present invention, and the present invention is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also considered to be within the scope of protection of the present invention.
Claims
1. A method for evaluating teaching quality, characterized in that, include: Collect multimodal data of students and teachers within a preset time period, wherein the multimodal data includes student brainwave data, student behavior data, teacher behavior data, student facial expression data, and teacher facial expression data; The multimodal data is input into a pre-trained multimodal feature fusion classification model to obtain the predicted teaching quality assessment score. The teaching quality assessment score is sent to the classroom terminal so that the classroom terminal receives and displays the predicted teaching quality assessment score. The trained multimodal feature fusion classification model includes a first layer, a second layer, a third layer, a fourth layer, a multimodal feature fusion layer, and a fully connected layer; The outputs of the first, second, third, and fourth layers are respectively connected to the input of the multimodal feature fusion layer, and the output of the multimodal feature fusion layer is connected to the input of the fully connected layer. The step of inputting the multimodal data into a pre-trained multimodal feature fusion classification model to obtain a predicted teaching quality assessment score specifically includes: The student's brainwave data is input into the first layer, and the student's brainwave features are extracted through the first layer. The student behavior data and teacher behavior data are input into the second layer, and the fusion features of student behavior and teacher behavior are extracted through the second layer; The student facial expression data is input into the third layer, and the teacher behavior and student facial expression fusion features are extracted through the third layer. The teacher's facial expression data is input into the fourth layer, and the teacher's facial expression features are extracted through the fourth layer; The features extracted from the first, second, third, and fourth layers are input into the multimodal feature fusion layer for normalization processing; The normalized features are input into the fully connected layer for full connection to obtain the predicted teaching quality assessment score. The step of inputting the student behavior data and teacher behavior data into the second layer, and extracting the fusion features of student behavior and teacher behavior through the second layer, specifically includes: The 10*m*n dimensional student behavior data and the 10*m*n dimensional teacher behavior data are input into the second layer of the Hourglass model to detect human key points and obtain the detection results of human key points. The detection results of the human body key points are input into a recurrent neural network (RNN) for behavior recognition, resulting in 1*n-dimensional student behavior features and 1*n-dimensional teacher behavior features. The L2 paradigm is used to fuse the 1*n-dimensional student behavior features and the 1*n-dimensional teacher behavior features to obtain 1*n-dimensional fused behavior features.
2. The method according to claim 1, characterized in that, After collecting multimodal data of students and teachers within a preset time period, the method further includes: The multimodal data is aligned based on the collection timestamp; The step of inputting the multimodal data into a pre-trained multimodal feature fusion classification model to obtain a predicted teaching quality assessment score specifically includes: The aligned multimodal data is input into a pre-trained multimodal feature fusion classification model to obtain the predicted teaching quality assessment score.
3. The method according to claim 1, characterized in that, Before inputting the multimodal data into a pre-trained multimodal feature fusion classification model to obtain the predicted teaching quality assessment score, the method further includes: The multimodal data of students and teachers during in-class quizzes are segmented according to a certain time step; The multimodal feature fusion classification model is trained based on the segmented multimodal data and the corresponding teaching quality score results to obtain the trained multimodal feature fusion classification model.
4. The method according to claim 1, characterized in that, The first layer includes an activation function layer and a normalization layer. The step of inputting the student's brainwave data into the first layer and extracting the student's brainwave features through the first layer specifically includes: The 1*n-dimensional student brainwave data is input into the activation function layer for activation processing, and the output result of the activation function layer is obtained. The output of the activation function layer is input into the normalization layer for normalization processing to obtain a normalized 1*n-dimensional feature, and the normalized 1*n-dimensional feature is used as the student's brainwave feature.
5. The method according to claim 1, characterized in that, The step of inputting the student facial expression data into the third layer and extracting the fused features of teacher behavior and student facial expressions through the third layer specifically includes: The 10*m*n dimensional student facial expression data is input into the third layer, and the facial expression features are extracted using the convolutional neural network (CNN) model shared between the third and fourth layers to obtain 1*n dimensional student facial expression features. Based on prior knowledge, the 1*n-dimensional student facial expression features and the 1*n-dimensional teacher behavior features are fused to obtain 1*n-dimensional teacher behavior and student facial expression fusion features.
6. The method according to claim 5, characterized in that, The step of inputting the teacher's facial expression data into the fourth layer and extracting the teacher's facial expression features through the fourth layer specifically includes: The 10*m*n dimensional teacher facial expression data is input into the fourth layer, and the CNN model is used to extract facial expression features to obtain 1*n dimensional teacher facial expression features.
7. The method according to claim 1, characterized in that, The formula for the normalization process is: in, The features extracted from the i-th layer, Let be the training weights corresponding to the i-th layer.
8. The method according to claim 7, characterized in that, The normalized features of the multimodal feature fusion layer are 1*4n dimensional multimodal fusion features. The normalized features of the multimodal feature fusion layer are then input into the fully connected layer for full connection to obtain the predicted teaching quality assessment score. Specifically, this includes: The 1*4n-dimensional multimodal fusion features are input into a 4n*11-dimensional fully connected layer for full connection to obtain the predicted 1*11-dimensional teaching quality assessment score.
9. A device for evaluating teaching quality, characterized in that, include: The modal data acquisition module is used to collect multimodal data of students and teachers within a preset time period. The multimodal data includes student brainwave data, student behavior data, teacher behavior data, student facial expression data, and teacher facial expression data. The evaluation score prediction module is connected to the modal data acquisition module and is used to input the multimodal data into a pre-trained multimodal feature fusion classification model to obtain the predicted teaching quality evaluation score. A receiving and display module, connected to the assessment score prediction module, is used to send the teaching quality assessment score to the classroom terminal so that the classroom terminal can receive and display the predicted teaching quality assessment score. The trained multimodal feature fusion classification model includes a first layer, a second layer, a third layer, a fourth layer, a multimodal feature fusion layer, and a fully connected layer; The outputs of the first, second, third, and fourth layers are respectively connected to the input of the multimodal feature fusion layer, and the output of the multimodal feature fusion layer is connected to the input of the fully connected layer. The assessment score prediction module specifically includes: The first processing unit is used to input the student's brainwave data into the first layer and extract the student's brainwave features through the first layer. The second processing unit is used to input the student behavior data and teacher behavior data into the second layer, and extract the fusion features of student behavior and teacher behavior through the second layer; The third processing unit is used to input the student facial expression data into the third layer and extract the fusion features of teacher behavior and student facial expression through the third layer. The fourth processing unit is used to input the teacher's facial expression data into the fourth layer and extract the teacher's facial expression features through the fourth layer; The fifth processing unit is used to input the features extracted from the first, second, third, and fourth layers into the multimodal feature fusion layer for normalization processing; The sixth processing unit is used to input the normalized features into the fully connected layer for full connection to obtain the predicted teaching quality assessment score; The second processing unit specifically includes: The key point detection unit is used to input 10*m*n dimensional student behavior data and 10*m*n dimensional teacher behavior data into the second layer of the Hourglass model to detect human key points and obtain the detection results of human key points. The behavior recognition unit is used to input the detection results of the human body key points into the recurrent neural network (RNN) for behavior recognition, and obtain 1*n-dimensional student behavior features and 1*n-dimensional teacher behavior features respectively. The behavior feature fusion unit is used to fuse the 1*n-dimensional student behavior features and the 1*n-dimensional teacher behavior features using the L2 paradigm to obtain 1*n-dimensional fused behavior features.
10. A device for evaluating teaching quality, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to implement the teaching quality assessment method as described in any one of claims 1-8.
11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method for evaluating teaching quality as described in any one of claims 1-8.