Teaching effect evaluation method and device based on multi-modal interaction analysis, equipment and storage medium

Abstract

The invention discloses a teaching effect evaluation method and device based on multi-modal interaction analysis, equipment and a storage medium, and relates to the technical field of informatization teaching, and the method comprises the steps: collecting multi-modal data of a current teaching scene to construct a classroom behavior data set and a facial expression data set; respectively extracting behavior category features and face category features through a preset behavior recognition model and a preset expression recognition model; the behavior category features and the face category features are spliced, comprehensive classroom performance features corresponding to the students are obtained, then the knowledge point mastering degree of the students is quantitatively scored, and a corresponding teaching optimization strategy is matched according to the scoring result. By integrating the two-dimensional information of the classroom behaviors and the facial expressions, the objectivity and accuracy of teaching evaluation can be remarkably improved, then behavior-expression deep correlation analysis is realized by adopting a cross-modal feature splicing technology, and intelligent matching of teaching strategies is realized.

Description

Technical Field

[0001] The present invention relates to the field of information-based teaching technology, and in particular to a teaching effect evaluation method, device, equipment and storage medium based on multimodal interaction analysis. Background Art

[0002] Traditional teaching evaluation methods mainly rely on teachers' subjective observations and students' test scores. Due to the lack of comprehensive and real-time analysis of students' classroom behavior and emotional state, it is difficult to capture students' subtle performance in class, resulting in delayed teaching feedback and inability to adjust teaching strategies in a timely manner.

[0003] With the development of artificial intelligence, the use of computer vision technology to automatically identify student behavior and emotions can better assist teachers in adjusting teaching strategies. However, while current adaptive teaching systems incorporate some artificial intelligence technologies, they typically rely on data from a single modality, with deficiencies in data collection and analysis. Consequently, they fail to fully reflect students' learning status and lack a deep integration analysis of student behavior and emotions, resulting in low reliability of the resulting teaching strategies. Summary of the Invention

[0004] The main purpose of the present invention is to provide a teaching effectiveness evaluation method, device, equipment and storage medium based on multimodal interaction analysis, aiming to solve the technical problem that the existing method cannot comprehensively and real-time evaluate students' learning status, resulting in low reliability of the matched teaching strategies.

[0005] To achieve the above objectives, the present invention provides a teaching effect evaluation method based on multimodal interaction analysis, the method comprising the following steps:

[0006] Collecting multimodal data of the current teaching scene, and constructing a classroom behavior dataset and a facial expression dataset based on the multimodal data;

[0007] Extracting the behavior category features corresponding to each student from the classroom behavior dataset using a preset behavior recognition model, and extracting the facial category features corresponding to each student from the facial expression dataset using a preset expression recognition model;

[0008] splicing the behavioral category features and the facial category features to obtain comprehensive classroom performance features corresponding to each of the students;

[0009] Based on the comprehensive classroom performance characteristics, each student is quantitatively scored on the degree of mastery of knowledge points, and a corresponding teaching optimization strategy is matched according to the scoring result.

[0010] In one embodiment, the step of collecting multimodal data of the current teaching scene and constructing a classroom behavior dataset and a facial expression dataset based on the multimodal data includes:

[0011] A three-dimensional camera is used to collect teaching video data of the current teaching scene, and a multimodal facial sensor is deployed to capture facial electromyographic signals in real time;

[0012] Performing frame-by-frame processing on the teaching video data, and extracting behavioral image frames at preset time intervals;

[0013] Performing time-frequency conversion on the facial electromyographic signals to generate triboelectric signal patterns corresponding to each facial region, and constructing a facial expression dataset based on each triboelectric signal pattern;

[0014] A spatiotemporal synchronization marking mechanism is established to associate and map the behavioral image frames with the facial electromyography signals in the corresponding time window, and a classroom behavior dataset is constructed based on the associated behavioral image frames.

[0015] In one embodiment, the step of extracting the behavior category features corresponding to each student from the classroom behavior dataset using a preset behavior recognition model includes:

[0016] Input the classroom behavior dataset into the preset behavior recognition model, which is built based on the Yolo model and includes: a data preprocessing module, a feature extraction network, a feature fusion network, and a detection head;

[0017] Performing data enhancement processing and format conversion on the behavior image frames in the classroom behavior dataset through the data preprocessing module to obtain input images corresponding to each student;

[0018] Extracting context features of the input image through the feature extraction network to obtain behavior context features;

[0019] Performing feature fusion on the behavior context features based on the fusion of deep semantic information and shallow position information through the feature fusion network to obtain fusion scale features;

[0020] The detection head predicts and classifies the fused scale features to obtain corresponding behavior category features.

[0021] In one embodiment, the feature extraction network includes a CBS module, a C2F module, and an SPFF module; and the step of extracting the contextual features of the input image through the feature extraction network to obtain the behavior contextual features includes:

[0022] In the CBS module, shallow features corresponding to the input image are extracted through a convolution operation, and the shallow features are normalized and linearly activated to obtain a shallow feature map;

[0023] Slicing the shallow feature map through the C2F module to obtain a first slice feature and a second feature, and splicing the first slice feature with the second slice feature processed by the bottleneck module to obtain a deep feature map;

[0024] The SPFF module performs multi-scale aggregation on the deep feature map to obtain behavioral context features.

[0025] In one embodiment, the step of extracting facial category features corresponding to each student from the facial expression dataset using a preset expression recognition model includes:

[0026] Inputting the facial expression data set into the preset expression recognition model, wherein the preset expression recognition model is constructed based on the ResNet model, and the preset expression recognition model includes: a ResNet network layer, a dual attention layer, an interactive learning layer, and a classification output layer;

[0027] Performing feature extraction on the triboelectric signal graph in the facial expression dataset through the ResNet network layer to obtain a feature matrix;

[0028] The dual attention layer performs feature separation on the feature matrix based on the channel dimension and the spatial dimension respectively to obtain channel dimension features and spatial dimension features;

[0029] Cross-merging the channel dimension features and the space dimension features through the interactive learning layer to obtain fused interactive features;

[0030] The fused interaction features are predicted and classified through the classification output layer to obtain corresponding facial category features.

[0031] In one embodiment, the step of quantitatively scoring each student's degree of mastery of knowledge points based on the comprehensive classroom performance characteristics includes:

[0032] Based on the preset standard classroom behavior, the behavioral weight coefficient corresponding to each focus behavior and the expression weight coefficient corresponding to each facial expression are set;

[0033] Determining the duration of each of the attentive behaviors in the comprehensive classroom performance characteristics, and calculating the corresponding attentiveness score of each of the students in combination with the behavior weight coefficient;

[0034] Determining the expression duration corresponding to each facial expression in the comprehensive classroom performance characteristics, and calculating the emotion score of each student in combination with the expression weight coefficient;

[0035] A mastery score of each student is obtained according to the concentration score and the emotion score.

[0036] In one embodiment, the step of matching the corresponding teaching optimization strategy according to the scoring results includes:

[0037] comparing the mastery score with a preset mastery threshold, and determining a target focused student based on the comparison result;

[0038] When the number of the target focused students exceeds a preset number threshold, updating the database according to the teaching mode in the current teaching scenario;

[0039] When the number of the target focused students does not exceed the preset number threshold, a teaching model related to the teaching content in the current teaching scenario is obtained from the database to generate a teaching optimization strategy.

[0040] In addition, to achieve the above objectives, the present application also proposes a teaching effect evaluation device based on multimodal interaction analysis, the device comprising:

[0041] A data acquisition module is used to collect multimodal data of the current teaching scene and construct a classroom behavior dataset and a facial expression dataset based on the multimodal data;

[0042] A feature extraction module is used to extract the behavior category features corresponding to each student from the classroom behavior dataset using a preset behavior recognition model, and to extract the facial category features corresponding to each student from the facial expression dataset using a preset expression recognition model;

[0043] A feature splicing module, configured to splice the behavioral category features and the facial category features to obtain a comprehensive classroom performance feature corresponding to each of the students;

[0044] The strategy matching module is used to quantitatively score the degree of knowledge mastery of each student based on the comprehensive classroom performance characteristics, and match the corresponding teaching optimization strategy according to the scoring results.

[0045] In addition, to achieve the above-mentioned purpose, the present application also proposes a teaching effect evaluation device based on multimodal interaction analysis, the device including: a memory, a processor, and a teaching effect evaluation program based on multimodal interaction analysis stored on the memory and runnable on the processor, the computer program being configured to implement the steps of the teaching effect evaluation method based on multimodal interaction analysis as described above.

[0046] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and stores a teaching effect evaluation program based on multimodal interaction analysis. When the teaching effect evaluation program based on multimodal interaction analysis is executed by the processor, the steps of the teaching effect evaluation method based on multimodal interaction analysis as described above are implemented.

[0047] The present application discloses a teaching effectiveness evaluation method based on multimodal interaction analysis, which collects multimodal data of the current teaching scene and constructs a classroom behavior dataset and a facial expression dataset based on the multimodal data; extracts the behavior category features corresponding to each student from the classroom behavior dataset through a preset behavior recognition model, and extracts the facial category features corresponding to each student from the facial expression dataset through a preset expression recognition model; splices the behavior category features and the facial category features to obtain the comprehensive classroom performance features corresponding to each student; quantitatively scores the degree of mastery of knowledge points for each student based on the comprehensive classroom performance features, and matches the corresponding teaching optimization strategy according to the scoring results. The present application can effectively integrate the two-dimensional information of classroom behavior and facial expression through multimodal data fusion and dynamic feature analysis, significantly improving the objectivity and accuracy of teaching evaluation; and adopts cross-modal feature splicing technology to realize deep correlation analysis of behavior and expression, breaking through the limitations of traditional single-dimensional evaluation; the dynamic matching mechanism of teaching strategies based on quantitative indicators provides accurate adaptive optimization solutions for personalized teaching. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0049] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0050] Figure 1 This is a flowchart of the first embodiment of the teaching effect evaluation method based on multimodal interaction analysis of this application;

[0051] Figure 2 Schematic diagram of multimodal data collection scenario;

[0052] Figure 3 This is a flow chart of the second embodiment of the teaching effect evaluation method based on multimodal interaction analysis of this application;

[0053] Figure 4This is a schematic diagram of the module structure of the preset behavior recognition model IT_YOLO;

[0054] Figure 5 This is a flowchart of the third embodiment of the teaching effect evaluation method based on multimodal interaction analysis of this application;

[0055] Figure 6 This is a schematic diagram of the module structure of the preset expression recognition model FE_RN;

[0056] Figure 7 This is a schematic diagram of the entire process of the teaching effectiveness evaluation method based on multimodal interaction analysis in this application;

[0057] Figure 8 This is a schematic diagram of the module structure of the first embodiment of the teaching effect evaluation device based on multimodal interaction analysis of this application;

[0058] Figure 9 This is a structural diagram of the teaching effect evaluation device based on multimodal interaction analysis in this application. DETAILED DESCRIPTION

[0059] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0060] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0061] The present invention provides a teaching effect evaluation method based on multimodal interaction analysis. Figure 1 , Figure 1 This is a flow chart of the first embodiment of the teaching effect evaluation method based on multimodal interaction analysis of this application. In this embodiment, the method includes: steps S10 to S40:

[0062] Step S10: Collect multimodal data of the current teaching scene, and construct a classroom behavior dataset and a facial expression dataset based on the multimodal data.

[0063] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, personal computer, mobile phone, teaching system server, etc., or other electronic devices that can achieve the same or similar functions.

[0064] It should be understood that the current teaching scenario can be a scene where students are taking classes in a classroom, that is, a teacher is giving a lecture. Lecture video data can be collected by a pre-installed 3D camera in the classroom, and students' facial expression data can also be collected by pre-deployed facial sensors to obtain multimodal data. The collected data is then organized into a classroom behavior dataset (such as student behavior images) and a facial expression dataset (such as facial electromyography signal images).

[0065] To specifically illustrate how to construct the above two types of data sets, step S10 specifically includes: steps S101 to S104:

[0066] Step S101: using a three-dimensional camera to collect teaching video data of the current teaching scene, and deploying a multimodal facial sensor to capture facial electromyographic signals in real time.

[0067] It should be understood that reference can be made here to Figure 2 Explain the multimodal data collection process in the current teaching scenario. Figure 2 Schematic diagram of the multimodal data collection scenario.

[0068] Depend on Figure 2 As can be seen, while the teacher is teaching, the student's performance in class can be recorded using a 3D camera, thereby obtaining video data of the teaching time. Furthermore, each student can wear a multimodal facial (expression) sensor PSiFI, which can obtain triboelectric signals from each modality sensor installed on it (such as lips, eyes, brow, nose, and chin).

[0069] Step S102: performing frame extraction processing on the teaching video data, and extracting behavior image frames at preset time intervals.

[0070] It should be understood that the teaching video data reflects the students' classroom behavior. The teaching video data can be converted into images frame by frame, and listened to by randomly selecting one image every 10 seconds. Several behavior image frames are obtained and manually labeled using the LabelImg tool to realize the initialization of the classroom behavior dataset.

[0071] For example, the annotation types and annotation standards of the classroom behavior dataset may be shown in Table 1 below. Table 1 is a student classroom behavior definition table.

[0072] Table 1 Definition of students' classroom behavior

[0073] label Name Marking standard (standard) Dt_Xz Lower your head and write Face down, holding a pen Dt_Ks Looking down at a book Face down with a book in hand or in front of Tt_Tk Look up and listen to the class The face is facing the blackboard, and the facial features can be seen Dt nod Up and down movement of the brain Yt Shake your head Turn your head left and right Zt glance right and left The head rotates left and right, and facial features cannot be seen clearly Js Raise your hand Arms raised Zl Standing Stand with your arms hanging naturally or read aloud while holding a textbook Xz_Tl Group Discussion There is a connection between two or more people

[0074] Step S103: performing time-frequency domain conversion on the facial electromyographic signal to generate a triboelectric signal graph corresponding to each facial area, and constructing a facial expression dataset based on each triboelectric signal pattern graph.

[0075] It should be understood that after acquiring triboelectric signals through the song modality sensor on the multimodal facial sensor, different patterns corresponding to different facial expressions, such as happiness, surprise, disgust, anger, and sadness, can be displayed, reflecting facial muscle activity. A facial expression dataset can then be constructed based on these triboelectric signal images.

[0076] Step S104: establishing a spatiotemporal synchronization marking mechanism, performing correlation mapping between the behavior image frames and the facial electromyography signals within the corresponding time window, and constructing a classroom behavior dataset based on the correlated behavior image frames.

[0077] It can be understood that during data collection, timestamp alignment can be performed based on 3D cameras and multimodal sensors to ensure the temporal and spatial consistency of each behavioral image frame and the facial electromyographic signal (frictional electric signal graph), and each behavioral image frame can be associated with a few facial signals, so that the aforementioned initialized classroom behavior dataset is associated with the facial expression dataset, and finally the classroom behavior dataset is obtained.

[0078] It should also be noted that after obtaining the labeled and associated classroom behavior dataset and facial expression dataset, the two types of datasets can be allocated to the training set, validation set, and test set folders in a ratio of 7:2:1 to meet the subsequent model building requirements and provide high-quality data for subsequent feature extraction and analysis.

[0079] Step S20: extracting the behavior category features corresponding to each student from the classroom behavior dataset using a preset behavior recognition model, and extracting the facial category features corresponding to each student from the facial expression dataset using a preset expression recognition model.

[0080] It should be understood that the preset behavior recognition model can be constructed based on the YOLO model, that is, the classroom behavior representation model can be expressed as In-class Teaching YOLO (IT_YOLO). The IT_YOLO model can be trained, verified, and tested based on the aforementioned classroom behavior dataset, and can then identify the behavior category characteristics corresponding to each student.

[0081] The preset expression recognition model can be constructed based on a residual neural network (ResNet), namely, the preset expression recognition model can be expressed as Face Emotion ResNet (FE_RN). The FE_RN model can be trained, verified, and tested based on the aforementioned facial expression dataset to identify the facial category features corresponding to each student.

[0082] Step S30: combining the behavioral category features and the facial category features to obtain comprehensive classroom performance features corresponding to each of the students.

[0083] It should be understood that the result feature vector of the preset behavior model recognition and the result feature vector of the preset expression recognition model can be unified in size through a convolution operation with a convolution kernel of the same size, and then the two feature vectors can be spliced to obtain a comprehensive student classroom expression feature to reflect the comprehensive student classroom learning effect.

[0084] Step S40: Quantitatively score the degree of knowledge mastery of each student based on the comprehensive classroom performance characteristics, and match the corresponding teaching optimization strategy according to the scoring result.

[0085] It is understood that different categories of comprehensive classroom performance characteristics and corresponding quantitative scoring values for knowledge point mastery can be pre-constructed to form a quantitative scoring table. Then, scores can be matched based on the comprehensive classroom performance characteristics within this quantitative scoring table to obtain each student's knowledge point mastery score. Finally, the corresponding teaching optimization strategy is determined based on the total number of students and each student's knowledge point mastery score.

[0086] For example, students can be divided into different mastery categories based on their knowledge point mastery scores: complete mastery, general mastery, basic mastery, and no mastery. Then, a corresponding teaching optimization strategy can be matched based on the ratio of the number of students in each mastery category to the total number of students.

[0087] This embodiment effectively integrates the dual-dimensional information of classroom behavior and facial expressions through multimodal data fusion and dynamic feature analysis, significantly improving the objectivity and accuracy of teaching evaluation; and adopts cross-modal feature splicing technology to realize deep correlation analysis of behavior and expression, breaking through the limitations of traditional single-dimensional evaluation; the dynamic matching mechanism of teaching strategies based on quantitative indicators provides precise adaptive optimization solutions for personalized teaching.

[0088] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above embodiment 1 can be referred to the above introduction and will not be described in detail later. Figure 3 , Figure 3This is a flow chart of the second embodiment of the teaching effectiveness evaluation method based on multimodal interaction analysis of this application.

[0089] In this embodiment, in order to specifically illustrate the behavior feature extraction process of the preset behavior recognition model, step S20 includes: steps S201 to S205:

[0090] Step S201: Input the classroom behavior data set into the preset behavior recognition model, which is constructed based on the YOLO model. The preset behavior recognition model includes: a data preprocessing module, a feature extraction network, a feature fusion network and a detection head.

[0091] It should be understood that this can be combined with Figure 4 The structure and feature extraction process of the preset behavior recognition model IT_YOLO in this embodiment are described. Figure 4 Schematic diagram of the module structure of the preset behavior recognition model IT_YOLO.

[0092] Step S202: performing data enhancement processing and format conversion on the behavior image frames in the classroom behavior dataset through the data preprocessing module to obtain input images corresponding to each student.

[0093] It is understandable that data preprocessing may include: Mosaic, adaptive image scaling, adaptive anchor box scaling and other processes.

[0094] Specifically, the images and labels in the labeled classroom behavior dataset can be input into the IT_YOLO model; the images are subjected to data enhancement operations such as random cropping, flipping and rotation, scaling and scaling jittering, color jittering, blurring and noise; and the pixel values are normalized: the image pixel values are scaled to the [0, 1] interval, usually by dividing by 255, as shown in formula (1):

[0095]

[0096] Then, using Mosaic data augmentation, we stitch the four images together into a single image. We resize the image to the network's required size of 640×640, maintaining the aspect ratio or performing center cropping / padding. We then convert the bounding box format: convert the bounding box coordinates to the format required by the model. The bounding box normalization formulas are shown in Equations (2)-(5). Finally, we convert the dataset annotation format: convert the labels to the YOLO format. Finally, we package the images and labels into batches, which serve as the input for feature extraction, i.e., the input images for each student.

[0097]

[0098] Where x is the horizontal coordinate of the input image, y is the vertical coordinate of the input image, W is the width of the border, and H is the length of the border.

[0099] Step S203: extracting context features of the input image through the feature extraction network to obtain behavior context features.

[0100] It should be noted that the data preprocessed images and labels are input into the feature extraction network, and the input images undergo convolution, normalization, slicing, splicing, SiLU activation function and residual operations to extract the contextual features of students' classroom behavior.

[0101] Depend on Figure 4 It can be seen that the feature extraction network includes CBS module, C2F module and SPFF module, combined with Figure 4 The feature extraction process is described as follows:

[0102] The preprocessed image and labels are input into the feature extraction network. First, the CBS module is used to extract shallow position features. The CBS module includes several convolutional layers for extracting features at different levels. Normalization is then performed to stabilize the training process. The SiLU activation function is then used to introduce a nonlinear activation function formula. The mathematical expression for normalization is shown in Equation (6), and the mathematical expression for the SiLU activation function is shown in Equation (7).

[0103]

[0104] f(x)=x×sigmoid(x) (7)

[0105] Where ∈ is a smoothing term to prevent division by zero. γ and β are learnable parameters. The sigmoid formula is shown in Equation (8).

[0106]

[0107] Among them, x is the input feature map.

[0108] Next, it passes through four CBS and C2f modules, with the second output feature map being denoted as X1 and the third output feature map being denoted as X2. In the C2f module, after extracting features through a CBS module, all the resulting feature maps are sliced and divided into two parts. One part is used for residual connection, and the other part is further extracted through n Bottleneck modules. After splicing these two parts, the features are fused through the CBS module. The Bottleneck module is obtained by element-by-element addition of two CBS operations and residuals. Finally, the SPPF operation aggregates features at multiple scales, and the resulting output is denoted as X3. The SPPF consists of two CBS operations, three maximum pooling operations, and splicing.

[0109] In the specific implementation, the input image is input into the feature extraction network: in the CBS module, the shallow features corresponding to the input image are extracted through convolution operation, and the shallow features are normalized and linearly activated to obtain a shallow feature map; the shallow feature map is sliced through the C2F module to obtain the first slice feature and the second feature, and the first slice feature is spliced with the second slice feature processed by the bottleneck module to obtain a deep feature map; the deep feature map is multi-scale aggregated through the SPFF module to obtain behavioral context features.

[0110] Step S204: performing feature fusion on the behavior context features based on the fusion of deep semantic information and shallow position information through the feature fusion network to obtain fused scale features.

[0111] It should be understood that the feature fusion network is able to input the contextual features of students' classroom behaviors output by the feature extraction network into the feature fusion layer, fusing deep semantic information and shallow position information.

[0112] based on Figure 4 As can be seen, since the feature extraction network outputs feature maps of sizes 320×320, 160×160, and 80×80 as X1, X2, and X3, respectively, we can perform operations such as g0ctConv convolution, batch normalization, and PRElu activation on X1, X2, and X3, and then perform element-wise addition. The CSP module then performs deep fusion and adaptation on the fused features, ultimately outputting feature vectors of three different scales, Y1, Y2, and Y3, to detect images of different sizes.

[0113] Among them, the mathematical formula of the PRelu activation function is shown in formula (9):

[0114]

[0115] Here, α is a parameter that needs to be learned to control the slope of the negative semi-axis. The CSP module consists of CBS, Bottleneck, and splicing.

[0116] Step S205: predicting and classifying the fused scale features through the detection head to obtain corresponding behavior category features.

[0117] It should be understood that the three feature vectors of different scales Y1, Y2, and Y3 output by the feature fusion network are input into the detection head, which can identify and locate students' classroom behaviors, thereby obtaining the predicted classification results of students' classroom behavior images.

[0118] Furthermore, during model training, the difference between the student behavior recognition results and location coordinates obtained and the actual student behavior recognition results and location coordinates marked in the labels can be calculated for regression. Through continuous iterative training, the bounding box, category probability, and confidence level are ultimately output.

[0119] It is understandable that the loss calculation includes classification loss, confidence loss and bounding box loss. For each predicted box, the classification loss calculates the error of the category to which it belongs, and uses binary cross entropy classification loss to handle category imbalance. The formula is shown in Equation (10).

[0120]

[0121] Among them, k is the number of target types, y n is the true classification value, q n is the predicted category probability.

[0122] Confidence loss is used to measure the accuracy of whether the predicted box contains an object, and the formula is shown in Equation (11).

[0123] L df (X i ,X i+1 )=-((q i+1 -q)log(X i )+(qq i )log(X i+1 )) (11)

[0124] Among them, q is the true value, q i is the value on the left side of the true value, q i+1 is the value to the right of the true value, X i is the value on the left side of the predicted value, X i+1 is the value to the right of the predicted value.

[0125] The bounding box loss is used to calculate the difference between the predicted box and the true box, and the formula is shown in Equation (12).

[0126]

[0127] Among them, IoU is the intersection-over-union ratio of the real box and the predicted box. 2 (t,t gt ) is the Euclidean distance between the center points of the real box and the predicted box, and c represents the diagonal length of the minimum closed box covering the predicted box and the real box. Among them, α can be used as a weight parameter, as shown in formula (13). gt and w gt is the length and width of the predicted box, w and h are the length and width of the real box. The formula of ε is shown in Equation (14).

[0128]

[0129]

[0130] This embodiment uses the preset behavior recognition model IT_YOLO to extract students' behavior category features from the classroom behavior dataset. The model includes a data preprocessing module, a feature extraction network, a feature fusion network, and a detection head. The data preprocessing module enhances and converts the behavior image frames to generate a standardized input image; the feature extraction network extracts the contextual features of the image; the feature fusion network fuses deep semantic information and shallow position information to generate multi-scale features; the detection head classifies the fused features and outputs behavior category features. In this way, the robustness of the model is improved through data enhancement, the recognition accuracy is improved through feature fusion, and the real-time classification supports dynamic teaching optimization, which can provide efficient and accurate behavior analysis support for teaching effect evaluation.

[0131] Based on the first and second embodiments of the present application, in the third embodiment of the present application, the same or similar contents as those in the first and second embodiments can be referred to the above introduction and will not be described in detail later. Figure 5 , Figure 5 This is a flow chart of the third embodiment of the teaching effectiveness evaluation method based on multimodal interaction analysis of this application.

[0132] In this embodiment, in order to specifically illustrate the facial feature recognition process of the preset facial expression recognition model, step S20 further includes steps S206 to S210:

[0133] Step S206: Input the facial expression data set into the preset expression recognition model, which is constructed based on the ResNet model. The preset expression recognition model includes: a ResNet network layer, a dual attention layer, an interactive learning layer, and a classification output layer.

[0134] It should be understood that this can be combined with Figure 6 The structure and feature extraction process of the preset expression recognition model FE_RN in this embodiment are described. Figure 6 Schematic diagram of the module structure of the preset expression recognition model FE_RN.

[0135] Step S207: performing feature extraction on the triboelectric signal graph in the facial expression dataset through the ResNet network layer to obtain a feature matrix.

[0136] It should be understood that the facial expression dataset includes the triboelectric signal map of each facial part and the corresponding facial expression label.

[0137] Combine Figure 6It can be seen that, assuming that the input triboelectric signal graph X has been obtained in ∈G 3×H×W , these images will pass through a ResNet model with 5 convolutional layers. The ResNet model converts the input image into a smaller scale high-dimensional feature map through high-dimensional mapping. These output feature maps are represented as feature matrices Where C represents the number of channels, H and W represent the height and width of the facial expression image, respectively.

[0138] Step S208: The feature matrix is subjected to feature separation based on the channel dimension and the spatial dimension respectively through the dual attention layer to obtain channel dimension features and spatial dimension features.

[0139] It should be noted that the results of channel and spatial dimensions can then be combined through the interactive learning mechanism, and the result is Thus, deep fusion of features is achieved through specific algorithms to capture richer information of facial expressions.

[0140] Specifically, high-level features in both channel and spatial dimensions are extracted from the synchronized ResNet results, and the extracted results are expressed as and In the channel dimension, assuming that the reconstructed image is divided into N w groups, each containing C w channels, so that C = C w ×N w The operation process of the overall self-attention mechanism in the channel dimension can be expressed as follows:

[0141]

[0142] In the spatial dimension, the reconstructed image can be divided into M g groups, each containing P g The reconstructed image is divided into multiple non-overlapping patches using a sliding window method, and then these patches are grouped. The size of each patch depends on the desired resolution of the reconstructed image and the size of the original image. The operation process of the overall self-attention mechanism in the spatial dimension is as follows:

[0143]

[0144] Among them, Q, K, V are the reconstructed input image G P×C Obtained through linear transformation, Concat means splicing, P represents the number of blocks, and C represents the number of channels. is the output of the self-attention mechanism.

[0145] Step S209: cross-merging the channel dimension features and the space dimension features through the interactive learning layer to obtain fused interactive features.

[0146] In the specific implementation, the results of these two dimensions (channel dimension features and spatial dimension features) can be combined into First, global spatial information is integrated using global average pooling of individual feature vectors in the channel dimension. Subsequently, the feature vectors are mapped using a feedforward neural network. The resulting mapping is called the implicit dynamic vector. Finally, the feature vector is cross-multiplied with the implicit dynamic vector to complete interactive learning and obtain fused interactive features.

[0147] Step S210: predicting and classifying the fused interaction features through the classification output layer to obtain corresponding facial category features.

[0148] In the specific implementation, the model can use a fully connected layer to generate the classification output vector X out ∈G n×1 This vector represents the classification results of different facial expressions, namely the facial category features, where n is the number of facial expression categories, and ultimately serves as the basis for evaluating students' emotional state and learning adaptability feature vectors.

[0149] Furthermore, after obtaining the facial category features, they can be concatenated with the aforementioned behavioral category features after being resized through a convolution operation to obtain the comprehensive classroom performance features corresponding to each student. Therefore, in order to specifically illustrate how to evaluate students' mastery of corresponding knowledge points based on comprehensive classroom performance features and match corresponding teaching optimization strategies, step S40 includes: steps S401 to S407:

[0150] Step S401: setting a behavior weight coefficient corresponding to each concentration behavior and an expression weight coefficient corresponding to each facial expression based on preset standard classroom behaviors.

[0151] It should be understood that historical experience shows that the degree of mastery of knowledge points is determined by "looking up and listening to the class" as the main action, that is, "looking up and listening to the class" is used as the preset standard classroom behavior.

[0152] For example, the duration of the action of “raising your head to listen to the class” can be recorded as t Tt_Tk , if t Tt_Tk >60s means listening carefully, or Tt_Tk If the duration is less than 60 seconds and the following actions include lowering the head to write, lowering the head to read, raising the hand, standing up to answer questions, or group discussion, it can also be defined as listening attentively; Tt_Tk If the listening time is less than 60 seconds and the subsequent movements include looking around, it means that the listener is not listening carefully.

[0153] The behaviors that indicate students’ concentration in class are mainly “looking up and listening to the class (Tt_Tk)”, followed by “looking down and reading (Dt_Ks)”, “looking down and writing (Dt_Xz)”, “raising hands (Js)”, “standing up and answering questions (Zl)”, and “group discussion (Xz_Tl)”. Therefore, the weights of these six behaviors are assigned as w1, w2, w3, w4, w5, and w6 respectively; the weights of other behaviors are 0.

[0154] It should also be noted that facial expressions can include: happiness (h), surprise (am), disgust (d), anger (a) and sadness (s), among which happiness (h) and surprise (am) are judged as being able to understand the content of the knowledge point and are given weights w e1 ,w e2 Disgust (d), anger (a) and sadness (s) are judged as incomprehensible knowledge points, w e =0.

[0155] Step S402: Determine the behavior duration corresponding to each of the concentration behaviors in the comprehensive classroom performance characteristics, and calculate the concentration score corresponding to each of the students in combination with the behavior weight coefficient.

[0156] It should be understood that the concentration score corresponding to each student can be calculated according to the following concentration score calculation formula, which is shown in formula (17):

[0157]

[0158] It is understandable that when c0>60, it is judged that the student can listen attentively for about 60% of the time in class.

[0159] Step S403: Determine the expression duration corresponding to each facial expression in the comprehensive classroom performance characteristics, and calculate the emotion score of each student in combination with the expression weight coefficient.

[0160] It should be understood that the emotion score corresponding to each student can be calculated according to the following emotion score calculation formula, which is shown in formula (18):

[0161]

[0162] Where, t h and t am The duration of expressions corresponding to happy (h) and surprised (am).

[0163] Step S404: Obtaining a mastery score for each student based on the concentration score and the emotion score.

[0164] It should be understood that the mastery score corresponding to each student can be calculated according to the following mastery score calculation formula, which is shown in formula (19):

[0165]

[0166] Step S405: Compare the mastery score with a preset mastery threshold, and determine the target focused student based on the comparison result.

[0167] It should be noted that the preset mastery threshold can be set to one or more, used to determine each student's mastery level. For example, if q0 > 75 is considered complete mastery; 65 ≤ q0 < 75 is considered mastery; 50 ≤ q0 < 65 is considered basic mastery; and q0 < 50 is considered not mastered, then students with q0 > 50 can be identified as having mastered the knowledge point, i.e., as target focused students.

[0168] Step S406: When the number of the target focused students exceeds a preset threshold, the database is updated according to the teaching mode in the current teaching scenario.

[0169] Step S407: When the number of the target focused students does not exceed the preset number threshold, the teaching mode related to the teaching content in the current teaching scenario is obtained from the database to generate a teaching optimization strategy.

[0170] In a specific implementation, the total number of target-focused students can be counted first, and a preset number threshold of 70% can be set; if the number of target-focused students accounts for more than 70% of the total number of students, the teacher's teaching method is judged to be effective, and the teaching model in the current teaching scenario is stored in the teaching database with the teaching content as a label; otherwise, a teaching model that is consistent or similar (same teaching topic) to the teaching content in the current teaching scenario is matched from the database, the teaching model in the current teaching scenario is updated, and a teaching optimization strategy is generated. In addition, for some students who have not mastered the knowledge points, video and text data of knowledge points related to the teaching content can be matched in the teaching database, and 10-20 after-class exercises for the knowledge points that have not been mastered can be randomly selected as supplementary optimization strategies in the current teaching scenario.

[0171] This embodiment, based on a dual attention mechanism and interactive learning, can improve the accuracy of facial expression recognition and the analysis and classification of complex expressions. Furthermore, it calculates students' concentration and mastery by weighting behavior and expression data. This allows for a comprehensive assessment of student learning outcomes, combining behavioral and expression data. This allows for the generation of targeted teaching strategies based on student mastery scores. This helps improve teachers' teaching effectiveness and meet students' personalized learning needs.

[0172] Further, you can also refer to Figure 7 Explain the whole process of the teaching effect evaluation method based on multimodal interaction analysis. Figure 7 This is a schematic diagram of the entire process of the teaching effectiveness evaluation method based on multimodal interaction analysis in this application.

[0173] exist Figure 7 In this paper, first, we obtain students' classroom video data, extract multiple frames of students' classroom behavior images from them, and construct a classroom behavior dataset; at the same time, we use multimodal facial sensors to obtain triboelectric signal images of students' lips, eyes, eyebrows, nose, and chin areas to form a facial expression dataset.

[0174] Next, the preset behavior recognition model IT_YOLO is used to identify the behavior category features corresponding to each student; and the preset expression recognition model FE_RN is used to identify the facial category features corresponding to each student.

[0175] Subsequently, the students' classroom behavior feature vectors and facial expression feature vectors are combined through a splicing operation to obtain the comprehensive classroom performance characteristics of each student, and then a quantitative score of the degree of mastery of each student's knowledge point is given based on the comprehensive classroom performance characteristics, that is, the mastery score of each student is calculated;

[0176] If a student's mastery score is greater than 50, it is determined that the student has mastered the teaching content in the current teaching scenario. The number of students with a mastery score greater than 50 can be counted. If the proportion exceeds 70% of the total number of students, the teacher's teaching method is judged to be effective, and the teaching model in the current teaching scenario is stored in the teaching database with the teaching content as a label. Otherwise, other teaching models are matched from the teaching database.

[0177] If a student's mastery score is no more than 50, it is determined that the student has not mastered the teaching content in the current teaching scenario. In this case, video and text data of knowledge points related to the teaching content can be matched in the teaching database, and 10-20 after-class exercises for the unmastered knowledge points can be randomly selected as a supplementary optimization strategy in the current teaching scenario.

[0178] Based on the entire process of this application method, adaptive teaching can be achieved based on students' classroom behavior and facial expressions to better meet students' learning needs and styles; real-time feedback on students' classroom performance can be provided to help university teachers understand students' learning status in a timely manner and flexibly adjust teaching strategies to improve teaching effectiveness and learning outcomes; thereby improving teaching quality, promoting student interaction, and improving teaching efficiency through objective data analysis and personalized review plans.

[0179] It should be noted that the above examples are only used to understand this application and do not constitute a limitation on the test question management method based on multimodal adaptive similarity learning of this application. More simple transformations based on this technical concept are all within the scope of protection of this application.

[0180] In addition, this application also provides a teaching effect evaluation device based on multimodal interaction analysis, referring to Figure 8 , Figure 8 This is a schematic diagram of the module structure of the first embodiment of the teaching effect evaluation device based on multimodal interaction analysis of this application; Figure 8 As shown, the device includes:

[0181] The data acquisition module 801 is used to collect multimodal data of the current teaching scene and construct a classroom behavior dataset and a facial expression dataset based on the multimodal data;

[0182] A feature extraction module 802 is configured to extract the behavior category features corresponding to each student from the classroom behavior dataset using a preset behavior recognition model, and to extract the facial category features corresponding to each student from the facial expression dataset using a preset expression recognition model;

[0183] A feature splicing module 803 is used to splice the behavior category features and the facial category features to obtain a comprehensive classroom performance feature corresponding to each student;

[0184] The strategy matching module 804 is used to quantitatively score the degree of knowledge mastery of each student based on the comprehensive classroom performance characteristics, and match the corresponding teaching optimization strategy according to the scoring result.

[0185] Through multimodal data fusion and dynamic feature analysis, this embodiment can effectively integrate the two-dimensional information of classroom behavior and facial expressions, significantly improving the objectivity and accuracy of teaching evaluation; and adopts cross-modal feature splicing technology to realize deep correlation analysis of behavior and expression, breaking through the limitations of traditional single-dimensional evaluation; the dynamic matching mechanism of teaching strategies based on quantitative indicators provides precise adaptive adjustment solutions for personalized teaching.

[0186] In addition, the present application also provides a teaching effect evaluation device based on multimodal interaction analysis, and the teaching effect evaluation device based on multimodal interaction analysis includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor so that the at least one processor can execute the teaching effect evaluation method based on multimodal interaction analysis in the above-mentioned embodiment one.

[0187] Reference below Figure 9 , Figure 9The structure diagram of the teaching effect evaluation device based on multimodal interaction analysis of the present application is shown. The teaching effect evaluation device based on multimodal interaction analysis in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), etc., as well as fixed terminals such as digital TVs and desktop computers. Figure 9 The teaching effect evaluation device based on multimodal interaction analysis shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0188] like Figure 9 As shown, the teaching effect evaluation device based on multimodal interaction analysis may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. Various programs and data required for the operation of the teaching effect evaluation device based on multimodal interaction analysis are also stored in RAM 1004. The processing device 1001, ROM 1002, and RAM 1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the teaching effectiveness evaluation device based on multimodal interaction analysis to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a teaching effectiveness evaluation device based on multimodal interaction analysis with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have instead.

[0189] In addition, the present application also provides a computer-readable storage medium having computer-readable program instructions (i.e., computer program) stored thereon, and the computer-readable program instructions are used to execute the teaching effect evaluation method based on multimodal interaction analysis in the above-mentioned embodiment.

[0190] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0191] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional elements in the process, method, article, or system comprising the element.

[0192] The serial numbers of the above-mentioned embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments. Moreover, they are only some embodiments of the present application and do not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the description and drawings of the present application under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A teaching effect evaluation method based on multimodal interaction analysis, characterized in that: The method comprises: Collecting multimodal data of the current teaching scene, and constructing a classroom behavior dataset and a facial expression dataset based on the multimodal data; Extracting the behavior category features corresponding to each student from the classroom behavior dataset using a preset behavior recognition model, and extracting the facial category features corresponding to each student from the facial expression dataset using a preset expression recognition model; splicing the behavioral category features and the facial category features to obtain comprehensive classroom performance features corresponding to each of the students; Based on the comprehensive classroom performance characteristics, each student is quantitatively scored on the degree of mastery of knowledge points, and a corresponding teaching optimization strategy is matched according to the scoring result.

2. The method according to claim 1, wherein The step of collecting multimodal data of the current teaching scene and constructing a classroom behavior dataset and a facial expression dataset based on the multimodal data includes: A three-dimensional camera is used to collect teaching video data of the current teaching scene, and a multimodal facial sensor is deployed to capture facial electromyographic signals in real time; Performing frame-by-frame processing on the teaching video data, and extracting behavioral image frames at preset time intervals; Performing time-frequency conversion on the facial electromyographic signals to generate triboelectric signal patterns corresponding to each facial region, and constructing a facial expression dataset based on each triboelectric signal pattern; A spatiotemporal synchronization marking mechanism is established to associate and map the behavioral image frames with the facial electromyography signals in the corresponding time window, and a classroom behavior dataset is constructed based on the associated behavioral image frames.

3. The method according to claim 1, wherein The step of extracting the behavior category features corresponding to each student from the classroom behavior dataset using a preset behavior recognition model includes: Inputting the classroom behavior dataset into the preset behavior recognition model, wherein the preset behavior recognition model is constructed based on the YOLO model, and the preset behavior recognition model includes: a data preprocessing module, a feature extraction network, a feature fusion network, and a detection head; Performing data enhancement processing and format conversion on the behavior image frames in the classroom behavior dataset through the data preprocessing module to obtain input images corresponding to each student; Extracting context features of the input image through the feature extraction network to obtain behavior context features; Performing feature fusion on the behavior context features based on the fusion of deep semantic information and shallow position information through the feature fusion network to obtain fusion scale features; The detection head predicts and classifies the fused scale features to obtain corresponding behavior category features.

4. The method according to claim 3, wherein The feature extraction network includes a CBS module, a C2F module, and an SPFF module; the step of extracting the context features of the input image through the feature extraction network to obtain the behavior context features includes: In the CBS module, shallow features corresponding to the input image are extracted through a convolution operation, and the shallow features are normalized and linearly activated to obtain a shallow feature map; Slicing the shallow feature map through the C2F module to obtain a first slice feature and a second feature, and splicing the first slice feature with the second slice feature processed by the bottleneck module to obtain a deep feature map; The SPFF module performs multi-scale aggregation on the deep feature map to obtain behavioral context features.

5. The method according to claim 1, wherein The step of extracting facial category features corresponding to each student from the facial expression dataset using a preset expression recognition model comprises: Inputting the facial expression data set into the preset expression recognition model, wherein the preset expression recognition model is constructed based on the ResNet model, and the preset expression recognition model includes: a ResNet network layer, a dual attention layer, an interactive learning layer, and a classification output layer; Performing feature extraction on the triboelectric signal graph in the facial expression dataset through the ResNet network layer to obtain a feature matrix; The dual attention layer performs feature separation on the feature matrix based on the channel dimension and the spatial dimension respectively to obtain channel dimension features and spatial dimension features; Cross-merging the channel dimension features and the space dimension features through the interactive learning layer to obtain fused interactive features; The fused interaction features are predicted and classified through the classification output layer to obtain corresponding facial category features.

6. The method according to claim 1, wherein The step of quantitatively scoring each student's degree of mastery of knowledge points based on the comprehensive classroom performance characteristics includes: Based on the preset standard classroom behavior, the behavioral weight coefficient corresponding to each focus behavior and the expression weight coefficient corresponding to each facial expression are set; Determining the duration of each of the attentive behaviors in the comprehensive classroom performance characteristics, and calculating the corresponding attentiveness score of each of the students in combination with the behavior weight coefficient; Determining the expression duration corresponding to each facial expression in the comprehensive classroom performance characteristics, and calculating the emotion score of each student in combination with the expression weight coefficient; A mastery score of each student is obtained according to the concentration score and the emotion score.

7. The method according to claim 6, wherein The step of matching the corresponding teaching optimization strategy according to the scoring results includes: comparing the mastery score with a preset mastery threshold, and determining a target focused student based on the comparison result; When the number of the target focused students exceeds a preset number threshold, updating the database according to the teaching mode in the current teaching scenario; When the number of the target focused students does not exceed the preset number threshold, a teaching model related to the teaching content in the current teaching scenario is obtained from the database to generate a teaching optimization strategy.

8. A teaching effect evaluation device based on multimodal interaction analysis, characterized in that: The device comprises: A data acquisition module is used to collect multimodal data of the current teaching scene and construct a classroom behavior dataset and a facial expression dataset based on the multimodal data; A feature extraction module is used to extract the behavior category features corresponding to each student from the classroom behavior dataset using a preset behavior recognition model, and to extract the facial category features corresponding to each student from the facial expression dataset using a preset expression recognition model; A feature splicing module, configured to splice the behavioral category features and the facial category features to obtain a comprehensive classroom performance feature corresponding to each of the students; The strategy matching module is used to quantitatively score the degree of knowledge mastery of each student based on the comprehensive classroom performance characteristics, and match the corresponding teaching optimization strategy according to the scoring results.

9. A teaching effect evaluation device based on multimodal interaction analysis, characterized in that: The device includes: a memory, a processor, and a teaching effect evaluation program based on multimodal interaction analysis stored in the memory and executable on the processor, wherein the teaching effect evaluation program based on multimodal interaction analysis is configured to implement the steps of the teaching effect evaluation method based on multimodal interaction analysis as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that The computer storage medium stores a teaching effect evaluation program based on multimodal interaction analysis. When the teaching effect evaluation program based on multimodal interaction analysis is executed by the processor, the steps of the teaching effect evaluation method based on multimodal interaction analysis as described in any one of claims 1 to 7 are implemented.

Images