A method and system for judging a teacher's teaching style

Abstract

This invention relates to a method and system for determining a teacher's teaching style. The method involves extracting modal features from the teaching video data of the teacher to be predicted, fusing these modal features to obtain multimodal features, and then inputting these multimodal features into a style prediction model to quickly and accurately determine the teacher's teaching style.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and system for judging teachers' teaching styles. Background Technology

[0002] Teaching style refers to a teacher's characteristics in teaching activities. It is a unique, harmonious combination and regular manifestation of a teacher's personality traits, teaching behaviors, and matching teaching abilities in the educational process. The formation of a teaching style is a sign of a teacher's maturing teaching art. A good teaching style plays an important role in helping students achieve academic success. In actual teaching, teachers mainly use multiple channels of verbal and non-verbal behavior to help students achieve learning goals. For teachers, teaching style is not only a specific teaching label but also a symbol of excellent work; for students, it is a reason to like a particular teacher and is also conducive to a deeper understanding of the teaching content. At the same time, teaching style is also a reflection of the quality of a teacher's classroom teaching, and schools can also use a teacher's teaching style to make an initial assessment of the teacher's teaching quality.

[0003] Existing methods for evaluating teacher teaching styles mostly rely on manual assessment and questionnaires. For example, patent CN110070232B discloses a method for predicting student grades based on teacher style. This invention uses a teacher style quantification form filled out manually to determine the teaching style of the course teacher, and then performs intelligent prediction. The drawback of this method is that it is highly dependent on manual labor, wasting human and material resources and incurring additional costs.

[0004] Patent CN105989755A discloses a classroom teaching process data collector, which enables the collection of classroom teaching process data within the classroom. The collected data types include attendance records, electronic hand-raising, answers to objective questions, answers to subjective questions, and in-class notes taken by teachers and students. This data collection allows for the observation of the teaching and learning process between teachers and students, and also accumulates foundational data for big data analysis. This addresses the limitation of existing technologies that do not universally acquire teaching process data, but also fail to record detailed teacher behaviors such as language, facial expressions, and physical actions, leading to inaccurate judgments of teacher style. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for judging teachers' teaching styles, which can improve the accuracy of teacher style prediction.

[0006] To achieve the above objectives, the present invention provides the following solution:

[0007] A method for judging a teacher's teaching style includes:

[0008] Modal features are obtained by extracting features from the teaching video data of the teacher to be predicted; the modal features include: facial expression features, voice features, and motion features; the facial expression features include local facial expression features and overall facial expression features.

[0009] The modal features are fused to obtain multimodal features;

[0010] The multimodal features are input into the style prediction model to obtain the teaching style of the teacher to be predicted.

[0011] Preferably, the step of extracting modal features from the teaching video data of the teacher to be predicted specifically includes:

[0012] FaceNet is used to extract local facial expression features of the teacher to be predicted from the teaching video data; the local facial expression features include: facial key points and the coordinate movement distance between the same facial key points;

[0013] A CNN network is used to identify the overall facial expression features of the teacher to be predicted based on the teaching video data; the overall facial expression features include: expression proportion, true number of expressions, and number of expression changes; the expressions include: serious, disappointed, afraid, happy, sad, and surprised;

[0014] The pyAnalysis tool was used to extract speech features from the teaching video data in real time.

[0015] The OpenPose was used to extract the motion features of the teacher to be predicted from the teaching video data; the motion features included the coordinates of the joints and the confidence level.

[0016] Preferably, before performing feature fusion on the modal features to obtain multimodal features, the method further includes:

[0017] The missing values ​​in the modal features are filled using the average value of the feature vectors;

[0018] The filled modal features are then spliced, normalized, and standardized to obtain the processed modal features.

[0019] Preferably, the step of fusing the modal features to obtain multimodal features specifically includes:

[0020] The modal features are fused using a multimodal fusion model to obtain multimodal features; the multimodal fusion model includes a feature selection structure and a multi-stage feature fusion structure.

[0021] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0022] The method for judging a teacher's teaching style provided by this invention involves extracting modal features from the teaching video data of the teacher to be predicted, fusing the modal features to obtain multimodal features, and then inputting the multimodal features into a style prediction model to quickly and accurately obtain the teaching style of the teacher to be predicted.

[0023] Corresponding to the above-mentioned method for judging teachers' teaching styles, the present invention also provides the following implementation system:

[0024] A system for judging a teacher's teaching style includes:

[0025] The modal feature extraction module is used to extract modal features from the teaching video data of the teacher to be predicted; the modal features include: facial expression features, voice features, and action features; the facial expression features include local facial expression features and overall facial expression features;

[0026] A modal feature fusion module is used to perform feature fusion on the modal features to obtain multimodal features;

[0027] The teaching style prediction module is used to input the multimodal features into the style prediction model to obtain the teaching style of the teacher to be predicted.

[0028] Preferably, the modal feature extraction module includes:

[0029] The facial expression local feature extraction unit is used to extract the facial expression local features of the teacher to be predicted from the teaching video data using FaceNet; the facial expression local features include: facial key points and the coordinate movement distance between the same facial key points;

[0030] The global facial expression feature extraction unit is used to identify the overall facial expression features of the teacher to be predicted based on the teaching video data using a CNN network; the overall facial expression features include: expression proportion, true number of expressions, and number of expression changes; the expressions include: serious, disappointed, afraid, happy, sad, and surprised;

[0031] The speech feature extraction unit is used to extract speech features from the teaching video data in real time using the pyAnalysis tool;

[0032] The motion feature extraction unit is used to extract the motion features of the teacher to be predicted from the teaching video data using OpenPose; the motion features include the coordinates of the joints and the confidence level.

[0033] Preferably, it further includes:

[0034] The missing value imputation module is used to fill the missing values ​​in the modal features with the average value of the feature vectors;

[0035] The modal feature processing module is used to splice, normalize, and standardize the filled modal features to obtain the processed modal features.

[0036] Preferably, the modal feature fusion module includes:

[0037] A modal feature fusion unit is used to fuse the modal features after processing using a multimodal fusion model to obtain multimodal features; the multimodal fusion model includes a feature selection structure and a multi-stage feature fusion structure.

[0038] Since the technical effects achieved by the teacher teaching style judgment system provided by this invention are the same as those achieved by the teacher teaching style judgment method provided above, they will not be described again here. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 A flowchart of the method for judging teachers' teaching styles provided by the present invention;

[0041] Figure 2 An implementation framework diagram of the teacher teaching style judgment method provided by the present invention;

[0042] Figure 3 This is a schematic diagram of the results of local facial expression feature extraction provided in an embodiment of the present invention;

[0043] Figure 4 This is a schematic diagram of the overall facial expression feature extraction results provided in an embodiment of the present invention;

[0044] Figure 5 This is a schematic diagram of key points for posture feature extraction provided in an embodiment of the present invention;

[0045] Figure 6 This is a schematic diagram of the step-by-step feature fusion structure provided in an embodiment of the present invention;

[0046] Figure 7 This is a schematic diagram illustrating the fusion results at different stages of the multi-stage feature fusion model provided in this embodiment of the invention; wherein, Figure 7 (a) is a graph showing the PCA distribution fusion loss rate results; Figure 7 (b) is a graph showing the accuracy results of PCA distributed fusion. Figure 7 (c) is a graph showing the AE distribution fusion loss rate results; Figure 7(d) is a graph showing the accuracy results of AE distribution fusion.

[0047] Figure 8 Service architecture diagram for implementing the teacher teaching style judgment method provided in the embodiments of the present invention;

[0048] Figure 9 A schematic diagram of the structure of the teacher teaching style judgment system provided by the present invention. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] The purpose of this invention is to provide a method and system for judging teachers' teaching styles, which can improve the accuracy of teacher style prediction.

[0051] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0052] like Figure 1 and Figure 2 As shown, the method for judging a teacher's teaching style provided by this invention includes:

[0053] Step 100: Extract modal features from the teaching video data of the teacher to be predicted. Modal features include: facial expression features, speech features, and action features. Facial expression features include local facial expression features and overall facial expression features. The teaching video data used in this invention can be offline teaching videos or online teaching videos. For offline teaching videos, the teacher's teaching video is recorded using equipment such as classroom cameras, and the entire video is used for detection and style judgment using a model. For online teaching videos, the entire video is directly used for detection and style judgment using a model. A general teaching style classification model is trained, and transfer learning is performed on the entire teaching video to obtain a keypoint detection model, wherein the human keypoint detection model is a multi-stage, multi-scale feature fusion human keypoint detection model.

[0054] Step 101: Perform feature fusion on the modal features to obtain multimodal features.

[0055] Different teaching styles exhibit distinct vocal, facial, and posture variations, facilitating the differentiation of teaching styles. After preprocessing and multimodal fusion of all extracted features, the model is trained on videos showcasing known teaching styles. These features are then applied to videos with unknown teaching styles to derive the teaching style of each video. The teaching style classification model comprises feature extraction, data preprocessing, and a multimodal fusion model.

[0056] Step 102: Input the multimodal features into the style prediction model to obtain the teaching style of the teacher to be predicted.

[0057] To further improve the accuracy of teaching style prediction, the specific implementation process of step 100 above is as follows:

[0058] Step 1000: Use FaceNet to extract local facial expression features of the teacher to be predicted from the teaching video data, for example... Figure 3 As shown. Local facial expression features include facial key points and the coordinate movement distance between identical facial key points. This study proposes a novel facial expression representation scheme that considers two different types of features to represent facial expressions: geometric features (local features) and salient expression features (global features). Geometric features capture the occurrence of the most distinctive facial key points in expression recognition and record the coordinate movement distance of identical key points between two frames. Salient expression features are extracted through a neural network to obtain the number of expression changes throughout the video.

[0059] The extraction of local facial features involves extracting the detected facial parts from each frame of the recorded teacher's teaching video. All images are aligned to a common reference frame using MTCNN similarity transformation and tracking facial coordinates. Coordinate information of 23 key points is extracted using FaceNet. Distance calculations are performed between the corresponding key point coordinates of the next frame and the previous frame to obtain the movement distance of each key point per frame, as shown in formula (1). The minimum, maximum, average, and variance of the feature vector S corresponding to all frames in each audio segment are then extracted.

[0060]

[0061] Step 1001: Use a CNN network to identify the overall facial expression features of the teacher to be predicted based on the teaching video data, for example... Figure 4 As shown. The overall characteristics of facial expressions include: the proportion of facial expressions, the number of true facial expressions, and the number of times facial expressions change. Facial expressions include: seriousness, disappointment, fear, happiness, sadness, and surprise.

[0062] The extraction of global facial expression features is as follows: This invention uses a CNN network to identify seven types of facial expressions: serious, disappointed, afraid, happy, sad, surprised, and natural. The recognition process only records cases where faces can be recognized. The total number of frames in the video that recognize expressions and the proportion of each expression are calculated as recognition features. Next, the number of expression changes per lesson is recorded. Finally, the proportion of each expression, the number of expression frames, and the number of expression changes are statistically analyzed across all time periods to obtain facial expression features.

[0063] Step 1002: Use the pyAnalysis tool to extract speech features from the teaching video data in real time. For example, the features extracted using the open-source pyAnalysis tool include short-time energy, short-time zero-crossing rate of the time signal, and Mel-frequency cepstral coefficients (MFCC). Then, using a script to calculate the mean, variance, maximum, and minimum values, the multiple real-time speech feature vectors are transformed into a single feature vector describing the entire video segment.

[0064] Step 1003: Use OpenPose to extract the action features of the teacher to be predicted from the teaching video data, for example... Figure 5 As shown. Motion features include the coordinates of the joints and the confidence level.

[0065] Based on OpenPose, the coordinates and confidence scores of 25 human joints were extracted. This invention selected only 8 key points based on the skeleton. Then, a script was called to calculate the angles formed by three points and the area of ​​the triangles formed by these three points, which were used as the teacher's posture features. The obtained angle features are shown in Table 1 below: the area formed by the hands and neck, the area formed by the hands and head, the area formed by the elbows and neck, the area formed by the elbows and head, the angle formed by the right shoulder, right elbow, and right wrist, the angle formed by the left shoulder, left elbow, and left wrist, the number of times the hands appeared, the number of times the wrists appeared, the number of times the right shoulder, right elbow, and right wrist appeared simultaneously, and the number of times the left shoulder, left elbow, and left wrist appeared simultaneously. Similarly, video-based teacher style research is also a continuous state study. After extracting real-time posture feature vectors from frames, the maximum, minimum, mode, mean, standard deviation, and variance of the posture features at all times in the video segment were calculated as representations of the teacher's teaching style features. Since the area formed by the teacher's wrist and other features is related to their position and distance from the camera, this invention uses the ratio of the joint point recognition area to the teacher's recognition frame area as the basis for comparing the size of the area composed of skeletal points:

[0066]

[0067] To distinguish the angle and area information generated by the wrist and elbow, this paper also uses the number of detected wrists and elbows as one of the features.

[0068] Table 1

[0069]

[0070]

[0071] In practical applications, during feature extraction from the entire video, the process begins with extracting speech features. Using the open-source pyAnalysis tool, the extracted features include short-time energy, short-time zero-crossing rate of the time signal, and Mel-frequency cepstral coefficients (MFCC). These features are then transformed into a feature vector of a certain dimension through statistical operations such as mean, variance, maximum, and minimum values. Next, facial expression features are extracted. A novel facial expression representation scheme is proposed, considering two different types of features: geometric features (local features) and salient expression features (global features). Geometric features capture the most distinctive facial key points in expression recognition, while salient expression features are extracted through a neural network to capture the expression transformation process. Finally, pose features are extracted. Based on the coordinates of a predetermined number of key points (e.g., 8) of bones, angles, areas, and other information are calculated as the teacher's action features.

[0072] Different teaching styles exhibit distinct characteristics in terms of speech, facial expressions, and posture, making it easier to differentiate between teachers' teaching styles. Since the extracted features may contain missing values, and the feature vectors of different modalities have different dimensions, all extracted features need to be preprocessed before step 101. Missing values ​​are filled with the average value of feature vectors from the same teaching style. Then, the speech, action, and facial expression features are concatenated and normalized / standardized.

[0073] Based on the preprocessed modal features described above, this invention employs a novel feature fusion method. Different features contribute to different degrees of results. This invention selects features with higher training contributions and combines them with each mode to increase the contribution of the fused features, establishing a system as follows: Figure 6 The feature fusion model shown is used for multimodal fusion. The multimodal fusion model is a step-by-step fusion model structure based on the main features, which includes a feature selection structure, a multi-stage feature fusion structure, and a prediction structure.

[0074] The feature selection structure uses AE (Aspect-Oriented Expression) and PCA (Program-Based Expression Analysis) as the main structures. PCA selects the following modal feature dimensions: 25 dimensions for speech features, 30 dimensions for local facial expression features, 6 dimensions for overall facial expression features, and 27 dimensions for pose features. AE selects the following modal feature dimensions: 30 dimensions for speech features, 40 dimensions for local facial expression features, 3 dimensions for overall facial expression features, and 25 dimensions for pose features.

[0075] First, modal features are extracted and feature selection is performed. The principal features after dimensionality reduction of the first modality are selected to represent all component vectors of that modality, denoted as Feature1. The first feature is used to predict the result, which is style1. The modality 2 features are combined with Feature1, and then the result is combined with Feature2 to obtain style2. Then, prediction is performed, integrating Feature1 and Feature2, and considering the influence of Feature1 on the final prediction. In the above steps, until all features are fused, all fused features can predict the final style. The principal components with a contribution of 99% extracted by PCA and the principal components of the nonlinear features learned by AE are fed into a contribution-based DNN model for fusion and classification training.

[0076] In each step of the prediction process, a DNN neural network is used as the foundation, and the network parameters are adjusted for different feature data to meet the requirements of the current input feature vector. Considering the data size, this invention establishes a four-layer DNN neural network, with ReLU activation function selected for the activation layer and softmax for the classification function. In the sentiment prediction after each feature fusion, the DNN is still used for classification. Among the two different feature selection methods, PCA has the advantages of speed and efficiency. AE has the advantage of improving the subsequent classification results. When the data volume is large, PCA can be chosen for dimensionality reduction; when accuracy is required, AE can be chosen for dimensionality reduction. The fusion results of different stages of the multi-stage feature fusion model are as follows: Figure 7 As shown.

[0077] The step-by-step multimodal feature fusion structure fuses the main features of speech modality, facial expression modality, and posture modality in stages to improve the ability to recognize the teaching style of the entire video, thereby reducing the limitations of the system.

[0078] Based on this, the entire teacher style prediction process can first utilize the self-made dataset of this invention or other self-made datasets for general training, serving as a pre-trained model for fine-tuning. The feature extraction structure of the pre-trained model is fixed, while the dimensions of feature selection are adjusted for targeted optimization to improve the model's classification accuracy. After training on videos with known different teaching styles, the model is applied to teaching videos with unknown teaching styles to obtain the teaching style of the video.

[0079] Based on the above description, the specific framework of the teacher teaching style judgment method implemented in this invention is as follows: Figure 8As shown, it includes an object layer, a data layer, a technology layer, and an application layer. The object layer represents the teacher to be predicted; the data layer contains the extracted modal features; the technology layer performs speech recognition, facial expression recognition, and posture recognition based on the extracted modal features; and the application layer performs feature analysis based on the recognition results to determine the teacher's teaching style.

[0080] In summary, the beneficial effects of this invention compared to the prior art are:

[0081] 1. This invention constructs a fully annotated dataset, which contains 407 video course records from the China University MOOC platform and, in an educational environment, ultimately obtains 4535 high-quality teaching video clips. The teaching style labels of teachers are defined from three aspects: the degree of teacher facial expression, the degree of posture change, and the degree of voice energy. There are three types: emotional, natural, and serious.

[0082] 2. This invention can identify the style of online and offline teaching videos, which helps to evaluate the quality of teaching and improve students' learning efficiency in the future.

[0083] 3. This invention utilizes a self-constructed dataset to train a general model, and can also use other datasets to fine-tune the model for various types, so as to make the detection more accurate and reduce the dependence on manually labeled data, thus greatly reducing the model training cost.

[0084] 4. This invention utilizes step-by-step multimodal fusion technology, which has better adaptability to different application scenarios and is easy to use and promote.

[0085] In addition, corresponding to the methods for judging teachers' teaching styles provided above, this invention also provides a system for judging teachers' teaching styles, such as... Figure 9 As shown, the system includes: modal feature extraction module 1, modal feature fusion module 2, and teaching style prediction module 3.

[0086] The modal feature extraction module 1 is used to extract modal features from the teaching video data of the teacher to be predicted. Modal features include: facial expression features, speech features, and motion features. Facial expression features include local facial expression features and overall facial expression features.

[0087] Modal feature fusion module 2 is used to perform feature fusion on modal features to obtain multimodal features.

[0088] The teaching style prediction module 3 is used to input multimodal features into the style prediction model to obtain the teaching style of the teacher to be predicted.

[0089] Furthermore, in order to improve the accuracy of the prediction results, the modal feature extraction module used in the present invention may further include: a local facial expression feature extraction unit, a global facial expression feature extraction unit, a speech feature extraction unit, and a motion feature extraction unit.

[0090] The facial expression local feature extraction unit is used to extract local facial expression features of the teacher's face from the teaching video data using FaceNet. These local facial expression features include facial key points and the coordinate movement distance between the same facial key points.

[0091] The global facial expression feature extraction unit is used to identify the overall facial expression features of the teacher to be predicted based on teaching video data using a CNN network. The overall facial expression features include: expression percentage, true number of expressions, and number of expression changes. Expressions include: serious, disappointed, afraid, happy, sad, and surprised.

[0092] The speech feature extraction unit is used to extract speech features from teaching video data in real time using the pyAnalysis tool.

[0093] The motion feature extraction unit is used to extract the motion features of the teacher to be predicted from the teaching video data using OpenPose. The motion features include the coordinates of the joints and the confidence level.

[0094] Furthermore, the teacher teaching style judgment system provided by the present invention may also include: a missing value filling module and a modal feature processing module.

[0095] The missing value imputation module is used to fill missing values ​​in modal features using the average of the feature vectors.

[0096] The modal feature processing module is used to splice, normalize, and standardize the filled modal features to obtain the processed modal features.

[0097] Based on this, the modal feature fusion module provided above may include: a modal feature fusion unit.

[0098] The modal feature fusion unit is used to fuse the modal features processed by the multimodal fusion model to obtain multimodal features. The multimodal fusion model includes: feature selection structure and multi-stage feature fusion structure.

[0099] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0100] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for judging a teacher's teaching style, characterized in that, include: Modal features are obtained by extracting features from the teaching video data of the teacher to be predicted; the modal features include: facial expression features, voice features, and motion features; the facial expression features include local facial expression features and overall facial expression features. The modal features are fused to obtain multimodal features; The multimodal features are input into the style prediction model to obtain the teaching style of the teacher to be predicted; Among them, modal features are obtained by extracting features from the teaching video data of the teachers to be predicted, including: FaceNet is used to extract local facial expression features of the teacher to be predicted from the teaching video data; the local facial expression features include: facial key points and the coordinate movement distance between the same facial key points; A CNN network is used to identify the overall facial expression features of the teacher to be predicted based on the teaching video data; the overall facial expression features include: expression proportion, true number of expressions, and number of expression changes; The pyAnalysis tool is used to extract speech features from the teaching video data in real time. The script calculates the mean, variance, maximum and minimum values ​​to transform multiple real-time speech feature vectors into a single feature vector describing an entire video segment. The OpenPose algorithm was used to extract the action features of the teacher to be predicted from the teaching video data. The action features included the coordinates and confidence scores of the joints. Among them, the coordinates of 8 key points based on the skeleton were selected, and the angle formed by each three points and the area of ​​the triangle formed by the three points were calculated by calling the script as the teacher's posture features. The ratio of the joint point recognition area to the teacher recognition box area was used as the basis for comparing the size of the area formed by the skeleton points. The extracted features included: the area formed by the hands and neck, the area formed by the hands and head, the area formed by the elbows and neck, the area formed by the elbows and head, the angle formed by the right shoulder, right elbow and right wrist, the angle formed by the left shoulder, left elbow and left wrist, the number of times the hands appeared, the number of times the wrists appeared, the number of times the right shoulder, right elbow and right wrist appeared simultaneously, and the number of times the left shoulder, left elbow and left wrist appeared simultaneously. The extraction of local facial features involves: extracting the detected facial parts from each frame of the recorded teacher's teaching video; performing a similarity transformation using MTCNN to align all images to a common reference frame using tracked facial coordinates; extracting the coordinate information of multiple key points using FaceNet; and calculating the distance between the coordinates of the corresponding key points in the next frame and the previous frame to obtain the movement distance of each key point per frame.

2. The method for judging a teacher's teaching style according to claim 1, characterized in that, Before performing feature fusion on the modal features to obtain multimodal features, the method further includes: The missing values ​​in the modal features are filled using the average value of the feature vectors; The filled modal features are then spliced, normalized, and standardized to obtain the processed modal features.

3. The method for judging a teacher's teaching style according to claim 2, characterized in that, The process of fusing the modal features to obtain multimodal features specifically includes: The modal features are fused using a multimodal fusion model to obtain multimodal features; the multimodal fusion model includes a feature selection structure and a multi-stage feature fusion structure.

4. A system for judging a teacher's teaching style, characterized in that, include: The modal feature extraction module is used to extract modal features from the teaching video data of the teacher to be predicted; the modal features include: facial expression features, voice features, and action features; the facial expression features include local facial expression features and overall facial expression features; A modal feature fusion module is used to perform feature fusion on the modal features to obtain multimodal features; The teaching style prediction module is used to input the multimodal features into the style prediction model to obtain the teaching style of the teacher to be predicted. The modality feature extraction module includes: The facial expression local feature extraction unit is used to extract the facial expression local features of the teacher's face to be predicted in the teaching video data using FaceNet. The facial expression local features include: facial key points and the coordinate movement distance between the same facial key points. The extraction of facial expression local features is as follows: extract the detected face parts from each frame of the recorded teacher teaching video; perform similarity transformation through MTCNN to align all images to a common reference frame and track facial coordinates; extract the coordinate information of multiple key points through FaceNet; and perform distance calculation between the coordinate positions of the corresponding key points in the next frame and the previous frame to obtain the movement distance of each key point per frame. The global facial expression feature extraction unit is used to identify the overall facial expression features of the teacher to be predicted based on the teaching video data using a CNN network; the overall facial expression features include: expression proportion, true number of expressions, and number of expression changes; The speech feature extraction unit is used to extract speech features from the teaching video data in real time using the pyAnalysis tool; wherein, through a script that calculates the mean, variance, maximum value, and minimum value, multiple real-time speech feature vectors are transformed into a feature vector describing an entire video segment; The motion feature extraction unit is used to extract the motion features of the teacher to be predicted from the teaching video data using OpenPose. The motion features include the coordinates and confidence scores of key points. Specifically, the coordinates of 8 key points based on skeletons are selected, and the angles formed by three points and the area of ​​the triangles formed by the three points are calculated by calling a script as the teacher's posture features. The ratio of the area of ​​the key point recognition to the area of ​​the teacher's recognition box is used as the basis for comparing the size of the area formed by the skeleton points. The extracted features include: the area formed by the hands and neck, the area formed by the hands and head, the area formed by the elbows and neck, the area formed by the elbows and head, the angles formed by the right shoulder, right elbow and right wrist, the angles formed by the left shoulder, left elbow and left wrist, the number of times the hands appear, the number of times the wrists appear, the number of times the right shoulder, right elbow and right wrist appear simultaneously, and the number of times the left shoulder, left elbow and left wrist appear simultaneously.

5. The teacher teaching style judgment system according to claim 4, characterized in that, Also includes: The missing value imputation module is used to fill the missing values ​​in the modal features with the average value of the feature vectors; The modal feature processing module is used to splice, normalize, and standardize the filled modal features to obtain the processed modal features.

6. The teacher teaching style judgment system according to claim 5, characterized in that, The modality feature fusion module includes: A modal feature fusion unit is used to fuse the modal features after processing using a multimodal fusion model to obtain multimodal features; the multimodal fusion model includes a feature selection structure and a multi-stage feature fusion structure.

Images