A teacher teaching state intelligent evaluation method and system based on multi-modal data and teacher-student interaction

Abstract

The present application belongs to the technical field of teacher teaching state intelligent evaluation, and discloses a teacher teaching state intelligent evaluation method and system based on multi-modal data and teacher-student interaction. Through contactless multi-modal data collection of teacher's voice, facial expression, body posture and teacher-student interaction behavior in the classroom teaching process, a multi-source data set of classroom teaching is constructed. Based on deep learning and multi-modal fusion algorithm, teacher teaching state features and teacher-student interaction features are jointly modeled to realize automatic identification and quantitative evaluation of teacher's classroom performance, and output comprehensive score and structured evaluation report. At the same time, data storage and management are realized through a network system. The overall structure of the system includes a training part and an application part. The training part includes a training data acquisition and processing system and an evaluation model training system. The application part includes a teacher teaching state and teacher-student interaction intelligent evaluation system and a data storage and management system.

Description

Technical Field

[0001] This invention belongs to the field of intelligent assessment technology of teacher teaching behavior, and in particular relates to an intelligent assessment method and system for teacher teaching behavior based on multimodal data and teacher-student interaction. Background Technology

[0002] With the rapid development of new-generation information technologies such as artificial intelligence, big data, computer vision, and speech recognition, the education sector is accelerating its transformation towards digitalization and intelligence. my country's Ministry of Education's "Action Plan for Education Informatization 2.0" explicitly proposes to promote the deep integration of information technology with education and teaching, facilitate the modernization of education, and improve teachers' information technology application capabilities and the quality of teaching. Against this backdrop, teacher teaching ability evaluation, as a crucial link in ensuring education quality, is facing the need to shift from traditional experience-based evaluation to data-driven intelligent evaluation. Among these, teachers' demeanor and classroom interaction directly reflect their teaching organization, classroom management, and teaching effectiveness, serving as important criteria for measuring teaching quality. Currently, teacher evaluation in universities, primary and secondary schools, and various teaching competitions still primarily relies on expert observations, peer review, and student feedback. These methods generally suffer from strong subjectivity, difficulty in unifying evaluation standards, and insufficient quantification. Meanwhile, the widespread adoption of smart classrooms and multimedia teaching equipment enables the effective acquisition of classroom audio modal data, visual modal data, and teacher-student interaction data, providing data support for the objective evaluation of teachers' classroom teaching behavior. Therefore, proposing an intelligent assessment method for teacher teaching behavior that integrates multimodal perception data and teacher-student interaction behavior information has significant practical implications and application value.

[0003] Teacher conduct assessment primarily encompasses three dimensions: verbal expression, visual demeanor, and teacher-student classroom interaction. Verbal expression reflects the teacher's teaching ability and emotional state; visual demeanor reflects the standardization of the teacher's classroom behavior; and classroom interaction reflects the quality of classroom interaction and the degree of student participation. Specifically, teacher conduct and classroom interaction evaluations are mainly conducted using methods such as Expert Observation Evaluation (EOE), Peer Review Evaluation (PRE), and Student Evaluation of Teaching (SET). While evaluations are typically conducted by educational experts using scales, the complexity and dynamic nature of classroom behavior make evaluation results susceptible to subjective experience and differences in standards, resulting in issues such as strong subjectivity and insufficient quantification.

[0004] With the development of artificial intelligence, computer vision, and deep learning technologies, teaching behavior analysis based on classroom multimedia data has become a research hotspot. These methods typically utilize classroom video, audio, and interactive data to extract speech, facial expressions, posture, and interaction features, enabling automatic identification and evaluation of teaching behaviors (e.g., invention patent CN202511429156.6 proposes using teaching posture heatmaps to achieve visual analysis of classroom interaction, providing support for classroom optimization). Related research indicates that these methods can improve the objectivity and automation level of evaluation to some extent, but they still have the following shortcomings:

[0005] (1) The evaluation information comes from a single source and lacks multimodal collaborative analysis;

[0006] (2) The interactive analysis is not deep enough and cannot reflect the teaching effect;

[0007] (3) Lack of unified quantitative standards;

[0008] (4) The evaluation results are not helpful in guiding teaching improvement.

[0009] Therefore, existing technologies cannot simultaneously achieve deep fusion of multimodal information, dynamic temporal modeling of the classroom, personalized longitudinal evaluation of teachers, and quantitative analysis of teaching style and interaction coupling. This is a technical problem that urgently needs to be solved in the field of intelligent classroom assessment for teachers. Summary of the Invention

[0010] To address the problems of existing technologies, this invention provides an intelligent evaluation method and system for teacher conduct based on multimodal data and teacher-student interaction. This solution effectively improves the issues of strong subjectivity and insufficient quantification in traditional evaluation methods. This invention mainly consists of three subsystems: 1) A multimodal classroom behavior data acquisition system: synchronously collects teacher language, facial expressions, body language, and classroom interaction information through classroom video and audio equipment to achieve multimodal data acquisition, including interaction information such as questioning, response, and interaction frequency; 2) An intelligent evaluation system for teacher conduct and teacher-student interaction: based on a multimodal fusion model, it jointly analyzes voice, visual, and behavioral interaction features to achieve quantitative evaluation of the quality of teacher conduct and classroom interaction; 3) A network-based data storage, management, and transmission system: used to remotely store and share classroom multimodal data and evaluation results, supporting structured management of data from different classrooms and teachers, and providing data support for teaching analysis.

[0011] This invention is implemented as follows: an intelligent assessment method for teacher teaching behavior based on multimodal data and teacher-student interaction, comprising:

[0012] Obtain continuous multimodal time-series samples arranged along a unified classroom timeline, wherein the multimodal time-series samples respectively contain video clips, audio clips and classroom interaction events corresponding to the time;

[0013] Speech features, visual teaching features, and classroom interaction features are extracted from the multimodal time-series samples. The speech features, visual teaching features, and classroom interaction features of the current multimodal time-series sample are input into the classroom state perception mechanism, and the classroom state corresponding to the current multimodal time-series sample is output. The classroom behavior sequence composed of continuous multimodal time-series samples is input into the time-series coding mechanism, and the classroom time-series features representing the time dependency relationship between continuous multimodal time-series samples are output.

[0014] The classroom state, classroom timing features, voice features, visual teaching features, and classroom interaction features are input into a dynamic adaptive weight fusion mechanism. The alignment network calculates the modal energy values ​​of the video modality, audio modality, and interaction modality respectively. The energy values ​​of the three modalities are then exponentially calculated. The result of the exponential calculation of any modality is divided by the sum of the exponential calculation results of the three modalities to obtain the dynamic weight of that modality. The sum of the dynamic weights of the three modalities is constrained to be 1.

[0015] According to the dynamic weights, the voice features, visual teaching features, and classroom interaction features are weighted and fused to output classroom fusion time sequence features. The classroom fusion time sequence features are then input into the trained intelligent evaluation model of teacher teaching style and teacher-student interaction to output teacher language expression score, teacher emotion score, teacher action performance score, and classroom interaction quality index. Based on the output, teacher teaching style evaluation results and classroom interaction evaluation results are generated.

[0016] Furthermore, the acquisition of the multimodal time series samples includes:

[0017] Configure unified timestamps for video data, audio data, and classroom interaction data during the classroom teaching process;

[0018] For video data, image enhancement, adaptive illumination compensation, background interference suppression, target region detection, and inter-frame synchronization are performed; for audio data, environmental noise cancellation, speech enhancement, silence removal, and speech segmentation are performed; and for classroom interaction data, abnormal event filtering, duplicate event removal, and time consistency calibration are performed.

[0019] The processed video data, audio data, and classroom interaction data are mapped to a unified classroom timeline, and then divided into segments according to fixed-length sliding time windows. Overlapping areas are set between adjacent sliding time windows so that each sliding time window contains video segments, audio segments, and classroom interaction events corresponding to the time.

[0020] Furthermore, the speech features include speech rate, tone, speech energy, and pause duration; the visual teaching features include facial expression features, human posture features, movement amplitude, and movement trajectory; the classroom interaction features include classroom questioning frequency, student response status, number of interactions, and classroom participation level; and the classroom status includes course introduction, knowledge instruction, classroom questioning, classroom discussion, and course summary.

[0021] Furthermore, it also includes:

[0022] Based on the evaluation results of teachers' history classes, a personalized teaching attitude baseline for teachers is constructed. The current classroom evaluation results are compared with the personalized teaching attitude baseline before the update, and the deviation of each evaluation indicator of the current classroom from the teacher's history level is output.

[0023] Using the teacher's personalized teaching behavior baseline before the update and the current classroom profile as input, the weight of the teacher's personalized teaching behavior baseline before the update is set to 1 minus the model update coefficient, and the weight of the current classroom profile is set to the model update coefficient. The updated teacher's personalized teaching behavior baseline is then calculated according to the corresponding weights.

[0024] The model update coefficient is constrained to be greater than 0 and less than 1, and the updated teacher-personalized teaching baseline is input into the subsequent classroom evaluation process.

[0025] Furthermore, the classroom interaction quality index is input by taking the classroom participation rate, effective response rate, classroom interaction frequency and average response time as inputs. The weighted values ​​corresponding to the classroom participation rate, effective response rate and classroom interaction frequency are added together and the weighted value corresponding to the average response time is subtracted to output the classroom interaction quality index. The sum of the weights corresponding to the four input indicators is constrained to be 1.

[0026] The input for the teacher's overall teaching demeanor score is composed of the teacher's language expression score, teacher's emotional score, and teacher's physical performance score. The overall teaching demeanor score is output by weighting the three scores according to their corresponding weights, and the sum of the weights of the three scores is constrained to be 1.

[0027] The teacher's teaching demeanor comprehensive score and classroom interaction quality index are used as inputs for the coupled evaluation of teacher's teaching demeanor and teacher-student interaction. The coupled evaluation result is output according to the sum of the weighted value corresponding to the teacher's teaching demeanor comprehensive score and the weighted value corresponding to the classroom interaction quality index. The weights corresponding to the two inputs are constrained to be 1. The coupled evaluation result is input into the classroom teaching evaluation report generation module to generate a classroom teaching evaluation report.

[0028] Another objective of this invention is to provide an intelligent evaluation system for teacher teaching behavior based on multimodal data and teacher-student interaction, including a multimodal data preprocessing and sample construction module, a multimodal feature learning module, a model training and dynamic fusion module, and an intelligent evaluation and teaching behavior analysis module.

[0029] The multimodal data preprocessing and sample construction module is used to map video data, audio data and classroom interaction data to a unified classroom timeline, output continuous multimodal time-series samples arranged along the unified classroom timeline, and output the continuous multimodal time-series samples to the multimodal feature learning module.

[0030] The multimodal feature learning module is used to extract speech features, visual teaching features, and classroom interaction features from multimodal time-series samples. It inputs the speech features, visual teaching features, and classroom interaction features of the current multimodal time-series sample into the classroom state perception mechanism and outputs the classroom state corresponding to the current multimodal time-series sample. It inputs the classroom behavior sequence composed of continuous multimodal time-series samples into the temporal coding mechanism and outputs classroom temporal features that represent the temporal dependency relationship between continuous multimodal time-series samples. The classroom state, classroom temporal features, speech features, visual teaching features, and classroom interaction features are then output to the model training and dynamic fusion module.

[0031] The model training and dynamic fusion module is used to calculate the modal energy values ​​of video modality, audio modality and interactive modality respectively through the alignment network, perform exponential operation on the energy values ​​of the three modalities respectively, divide the exponential operation result of any modality by the sum of the exponential operation results of the three modalities to obtain the dynamic weight of the modality, constrain the sum of the dynamic weights of the three modalities to be 1, and perform weighted fusion of speech features, visual teaching features and classroom interaction features according to the dynamic weights to output classroom fusion temporal features;

[0032] The intelligent assessment and teaching demeanor analysis module is used to input the classroom integration time sequence characteristics of the classroom to be assessed into the trained intelligent assessment model of teacher teaching demeanor and teacher-student interaction, and output the teacher's language expression score, teacher's emotion score, teacher's action performance score and classroom interaction quality index, thus forming the teacher teaching demeanor evaluation result and classroom interaction evaluation result.

[0033] Furthermore, it also includes a device deployment and data acquisition module;

[0034] The device deployment and data acquisition module is used to synchronously collect video data, audio data, and classroom interaction data during the classroom teaching process, and to configure a unified timestamp for the video data, audio data, and classroom interaction data.

[0035] The multimodal data preprocessing and sample construction module is used to perform image enhancement, adaptive illumination compensation, background interference suppression, target region detection and inter-frame synchronization on video data, environmental noise cancellation, speech enhancement, silence removal and speech segmentation on audio data, abnormal event filtering, duplicate event removal and time consistency calibration on classroom interaction data, and to divide the time-aligned data into time-aligned data according to a fixed-length sliding time window, and to set overlapping areas between adjacent sliding time windows.

[0036] Furthermore, the model training and dynamic fusion module is also used for:

[0037] The classroom interaction quality index, voice features, visual teaching style features, and classroom interaction features are input into the model to train the network;

[0038] The dynamic weights of video modality, audio modality, and interactive modality are controlled by the classroom state perception mechanism output by the classroom state perception mechanism, and the temporal features of the classroom output by the temporal coding mechanism are used to maintain the temporal dependency between continuous multimodal temporal samples.

[0039] The classroom integration time-series features obtained according to dynamic weights are input into the intelligent evaluation model of teacher's teaching style and teacher-student interaction. Multi-fold cross-validation is used to optimize the model parameters, and the model parameters are determined through iterative training.

[0040] Furthermore, the intelligent assessment and teaching attitude analysis module is also used for:

[0041] Based on the evaluation results of teachers' history classes, a personalized teaching attitude baseline for teachers is constructed. The current classroom evaluation results are compared with the personalized teaching attitude baseline before the update, and the deviation of each evaluation indicator of the current classroom from the teacher's history level is output.

[0042] Using the teacher's personalized teaching behavior baseline before the update and the current classroom profile as input, the weight of the teacher's personalized teaching behavior baseline before the update is set to 1 minus the model update coefficient, and the weight of the current classroom profile is set to the model update coefficient, and the updated teacher's personalized teaching behavior baseline is output.

[0043] The model update coefficient is constrained to be greater than 0 and less than 1, and the updated teacher-personalized teaching baseline is output to the subsequent classroom evaluation process.

[0044] Furthermore, it also includes a teaching suggestion generation module, which includes coupling analysis and report generation functions;

[0045] The coupling analysis and report generation function takes teacher language expression score, teacher emotion score and teacher action performance score as input, outputs teacher teaching attitude comprehensive score according to the weighted sum of the weights corresponding to the three scores, and constrains the sum of the weights corresponding to the three scores to be 1;

[0046] The coupling analysis and report generation function also takes the teacher's teaching attitude comprehensive score and classroom interaction quality index as inputs, and outputs the teacher's teaching attitude and teacher-student interaction coupling evaluation results according to the sum of the weighted value corresponding to the teacher's teaching attitude comprehensive score and the weighted value corresponding to the classroom interaction quality index, and constrains the sum of the weights corresponding to the two inputs to be 1;

[0047] The coupling analysis and report generation function inputs the evaluation results of teacher's teaching demeanor and teacher-student interaction into the classroom teaching evaluation report generation process, and outputs a classroom teaching evaluation report that includes teacher's teaching demeanor score, teacher-student interaction score, comprehensive classroom score, key influencing factor analysis results, and teaching improvement suggestions.

[0048] In conjunction with the above technical solutions and the technical problems they solve, the significant progress and unexpected technical effects achieved by the technical solution for which protection is sought in this invention are specifically as follows:

[0049] This invention proposes an intelligent evaluation method for teacher conduct based on multimodal data and teacher-student interaction. By integrating multi-source information such as teacher voice, facial expressions, body posture, and teacher-student interaction, and incorporating the Interaction Quality Index (IQI), Classroom State Perception Mechanism (CSPM), Temporal Encoding Mechanism (TEM), Dynamic Adaptive Weight Fusion (DAWF), Teacher Profile Model (TPM), and Teacher-Interaction Coupling Model (TICM), this method achieves automatic identification, dynamic analysis, quantitative evaluation, and continuous optimization of classroom teaching behaviors. This method not only improves the objectivity, accuracy, and automation of teaching evaluation but also enables personalized analysis of teacher classroom performance, comprehensive evaluation of classroom interaction quality, and intelligent generation of teaching improvement suggestions. It provides reliable technical support for improving teacher teaching abilities, monitoring classroom teaching quality, and the digitalization of education.

[0050] This invention employs multi-source data analysis, including teacher voice information, facial expression information, human posture information, and teacher-student interaction behavior information. Compared to methods that rely solely on a single behavioral feature or classroom interaction data for evaluation, this invention can simultaneously integrate information from multiple dimensions such as teacher language expression, visual teaching demeanor, and classroom interaction, providing more comprehensive data support for evaluating teachers' classroom teaching status and improving the objectivity, completeness, and reliability of classroom teaching evaluation.

[0051] This invention constructs a multimodal collaborative evaluation system that integrates language expression features, visual teaching posture features, and teacher-student interaction features. It also introduces a classroom state perception mechanism (CSPM) and a temporal coding mechanism (TEM), enabling the identification of different teaching stages in the classroom, characterizing the temporal evolution of teacher behavior and classroom interaction, and achieving dynamic analysis of the entire classroom process. Compared to traditional static analysis methods, this invention has stronger classroom scenario adaptability and temporal modeling capabilities, improving the accuracy of classroom teaching behavior analysis.

[0052] This invention proposes an Individual Classroom Interaction Quality Index (IQI), a Teacher Personalized Teaching Style Baseline Model (TPM), and a Teacher Teaching Style-Teacher-Student Interaction Coupled Evaluation Model (TICM). By establishing quantitative evaluation indicators for classroom interaction, a teacher historical baseline analysis mechanism, and a teaching style-interaction coupled analysis mechanism, it achieves personalized evaluation of teacher classroom performance, analysis of classroom interaction quality, and comprehensive classroom evaluation. It can not only output teacher teaching style scores, classroom interaction scores, and comprehensive classroom scores, but also reveal the impact of teacher teaching style on classroom interaction effectiveness, thereby improving the quantifiability, interpretability, and relevance of classroom teaching evaluation results.

[0053] This invention constructs an intelligent evaluation model of teacher teaching behavior based on multimodal data, and combines a dynamic adaptive weight fusion mechanism (DAWF) to achieve dynamic fusion of different modal features. At the same time, it establishes a classroom teaching data storage and management mechanism and a model continuous update mechanism to achieve unified management of multimodal data, teacher profile data and classroom evaluation results, and supports long-term tracking analysis of teachers' teaching abilities and continuous optimization of the evaluation model. Thus, it provides more reliable data support for intelligent evaluation of teachers' classroom teaching behavior, improvement of teaching quality and educational management decisions.

[0054] The expected benefits and commercial value of the technical solution of this invention after transformation are as follows: This invention can be embedded in smart classrooms, routine recording and broadcasting systems, and information platforms for teacher teaching development, providing automated evaluation services for classroom teaching quality to universities, primary and secondary schools, and educational management institutions. The expected benefits and commercial value are reflected in the following aspects: Compared to the manual classroom evaluation model that relies on expert attendance, this invention can complete classroom evaluation without human intervention, helping to reduce the human and time costs of evaluation and making routine, comprehensive classroom teaching quality monitoring possible; the structured evaluation reports, teacher profiles, and development trend analyses generated by the system can serve teacher training, teaching competition selection, and teaching management decisions, forming a sustainable accumulation of teaching quality data; this invention can be integrated and deployed as a software module with recording and broadcasting equipment manufacturers, smart classroom solution providers, and educational information platforms, and has the conditions for patent licensing, technology transfer, and industry-university-research cooperation transformation, with application scenarios covering basic education, higher education, vocational education, and teacher training.

[0055] The technical solution of this invention fills a technological gap in the industry both domestically and internationally. Represented by existing publicly available solutions (such as invention patent CN202511429156.6), current classroom evaluation technologies are mostly based on single-modal data (such as only teaching posture) or static feature statistics, and have not yet provided the following technical solution: combining a classroom state perception mechanism (CSPM), a temporal coding mechanism (TEM), and a dynamic adaptive weight fusion mechanism (DAWF) to dynamically adjust the fusion weights of the three modalities of speech, vision, and interaction based on real-time identified classroom state; constructing a classroom interaction quality index (IQI) by weighting classroom participation rate, effective response rate, interaction frequency, and average response time to provide a unified quantitative indicator for classroom interaction quality; and combining a teacher-personalized teaching posture baseline model (TPM) with a teacher teaching posture-teacher-student interaction coupled evaluation model (TICM) to achieve longitudinal tracking evaluation of individual teachers and correlation analysis of teaching posture and interaction. This invention provides the above complete technical chain, filling the corresponding technological gap in this field.

[0056] The technical solution of this invention solves a long-standing technical problem that people have long desired to solve but have never been able to achieve: the evaluation of teachers' classroom teaching has long suffered from strong subjectivity, inconsistent standards, difficulty in quantification, inability to cover routinely, and inability to continuously track. The education field has long desired to achieve objective, automatic, and scalable classroom evaluation, but has never succeeded because: (1) the time bases of multi-source heterogeneous classroom data (video, audio, interactive events) are inconsistent, making it difficult to accurately align and integrate; (2) the importance of each modality of information changes dynamically in different teaching stages of the classroom, and the fixed-weight fusion method cannot adapt to it; (3) the quality of classroom interaction lacks a unified quantitative standard, and the evaluation results vary from person to person. This invention solves the problem of multimodal data alignment by constructing a unified timestamp and sliding time window sample, solves the problem of dynamic adaptation of modal weights by using a classroom state perception mechanism (CSPM) and a dynamic adaptive weight fusion mechanism (DAWF), and solves the problem of unified quantification of interaction quality by using an interactive quality index (IQI) and a coupled evaluation model (TICM), thereby realizing an objective, dynamic, quantitative, and traceable evaluation of teachers' classroom teaching behavior. Attached Figure Description

[0057] Figure 1 This is a flowchart of the intelligent evaluation method for teacher teaching behavior based on multimodal data and teacher-student interaction provided in an embodiment of the present invention.

[0058] Figure 2 This is a flowchart of the overall workflow for intelligent assessment of teaching behavior provided in this embodiment of the invention.

[0059] Figure 3 The embodiment of this invention provides a system overall architecture diagram.

[0060] Figure 4This is a diagram of a high-definition PTZ camera provided in an embodiment of the present invention.

[0061] Figure 5 This is a diagram of a wireless microphone and a wireless lavalier microphone provided in an embodiment of the present invention.

[0062] Figure 6 This is a diagram of an actual classroom scene provided in an embodiment of the present invention.

[0063] Figure 7 This embodiment of the invention provides a schematic diagram of the deployment of classroom data acquisition equipment.

[0064] Figure 8 The present invention provides a diagram of an intelligent evaluation system.

[0065] Figure 9 This is a multimodal feature composition diagram provided in the embodiments of the present invention. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0067] To address the problems existing in current teacher classroom performance evaluation processes, such as limited evaluation dimensions, insufficient depth of multimodal information fusion, difficulty in depicting the dynamic evolution of classroom behavior, lack of personalized analysis of evaluation results, and difficulty in continuously tracking changes in teachers' teaching abilities, this invention provides an intelligent evaluation method and system for teacher classroom performance based on multimodal data and teacher-student interaction. This method synchronously collects heterogeneous data from multiple sources, including teacher voice, facial expressions, body posture, and teacher-student interaction behavior, during classroom teaching to construct a unified multimodal classroom dataset. By combining multimodal feature extraction, classroom state perception, temporal dynamic modeling, and dynamic adaptive weight fusion technologies, it achieves joint analysis of teacher classroom performance characteristics and classroom interaction characteristics, establishing a coupled evaluation relationship between teacher classroom performance and teacher-student interaction. Furthermore, by combining historical classroom data, it constructs a personalized teacher classroom performance baseline model to conduct longitudinal analysis of teacher classroom performance and generate targeted suggestions for classroom teaching improvement, achieving objective evaluation, continuous tracking, and intelligent optimization of teacher classroom teaching behavior.

[0068] like Figure 1 As shown in the embodiment of the present invention, the intelligent assessment method for teacher teaching behavior based on multimodal data and teacher-student interaction includes the following steps:

[0069] Step 1: Equipment Deployment and Data Acquisition

[0070] Deploying multi-source sensing devices in actual classroom teaching environments allows for non-contact, multimodal data collection of teachers' entire classroom teaching process, in order to obtain raw teaching behavior data.

[0071] The multi-source sensing devices include video acquisition devices, audio acquisition devices, and classroom interactive information acquisition devices. For example, Figure 4 As shown, the video capture device is preferably one or more of a high-definition PTZ camera, a wide-angle network camera, or an industrial camera, used to capture images of the teacher's facial expressions, changes in body posture, and classroom scenes; such as Figure 5 As shown, the audio acquisition device is preferably one or more of a microphone array, a wireless microphone, or a wireless lavalier microphone, used to acquire the teacher's lecture voice and the voice of teacher-student interaction; the classroom interaction information acquisition device is used to record classroom interaction events such as teacher questions, student answers, classroom discussions, interactive feedback, and classroom participation.

[0072] Furthermore, a unified timestamp mechanism is established for video data, audio data, and classroom interaction events to achieve time synchronization of data from different modalities. A multimodal raw database of classroom data is also constructed to provide a unified data foundation for subsequent data preprocessing, multimodal feature extraction, classroom status recognition, time series modeling, and teacher profiling.

[0073] Step 2: Multimodal data preprocessing and sample construction

[0074] The raw multimodal classroom data acquired in Step 1 is preprocessed. For video data, image enhancement, adaptive illumination compensation, background interference suppression, target region detection, and inter-frame synchronization are performed sequentially. For audio data, environmental noise removal, speech enhancement, silence removal, and speech segmentation are performed. For classroom interaction data, abnormal event filtering, duplicate event removal, and time consistency calibration are performed. Furthermore, a unified classroom timeline is constructed based on a unified timestamp, mapping video data, audio data, and classroom interaction data to the same temporal space to achieve time alignment between different modalities.

[0075] Meanwhile, a fixed-length sliding time window is used to segment the continuous classroom data. Each time window contains the corresponding video clips, audio clips, and classroom interaction events. An overlapping area is set for adjacent windows to reduce information loss caused by classroom behavior segmentation, improve the ability to model continuous behavior, ensure the continuity of classroom behavior changes, and form a unified format of multimodal time series samples.

[0076] Step 3: Multimodal Feature Learning

[0077] The multimodal time-series samples constructed in step 2 were labeled and their multimodal features were learned. A labeling system for teacher demeanor and teacher-student interaction was established based on classroom teaching evaluation standards. A combination of expert labeling and automatic assisted labeling was used to label teachers' language expression, emotional state, behavioral categories, and classroom interaction behaviors. Teacher behaviors included lecturing, blackboard writing, circulating, asking questions, and interacting; teacher emotions included positive, stable, and negative states; and classroom interaction included student responses, classroom discussions, classroom feedback, and interaction frequency.

[0078] Classroom status tags are introduced to describe the changes in the classroom teaching process, including teaching stages such as course introduction, knowledge delivery, classroom questioning, classroom discussion, and course summary, and classroom status tags are associated with corresponding time windows.

[0079] like Figure 9 As shown, based on the labeled multimodal classroom samples, teacher voice features, visual teaching demeanor features, and classroom interaction behavior features are extracted. Voice features include speech rate, tone, voice energy, and pause duration; visual teaching demeanor features include facial expressions, body posture, movement amplitude, and motion trajectory; classroom interaction features include the frequency of classroom questions, student responses, number of interactions, and level of classroom participation. Subsequently, the extracted teacher voice features, visual teaching demeanor features, and classroom interaction features are represented using a unified dimension to construct a multimodal classroom feature set, which serves as input data for subsequent model training.

[0080] To further quantify the effectiveness of classroom interaction, this invention proposes a Classroom Interaction Quality Index (IQI) for comprehensively evaluating the quality of classroom interaction. Its calculation formula is as follows:

[0081] in,

[0082] P represents the class participation rate.

[0083] A represents the Effective Response Rate;

[0084] F represents the frequency of classroom interaction.

[0085] T represents the average response time.

[0086] , , and These represent the weight coefficients of the corresponding indicators;

[0087] And it satisfies: The weighting coefficients can be set based on expert experience or determined using the Analytic Hierarchy Process (AHP) and data-driven optimization methods. IQI, as a quantitative indicator of classroom interaction quality, is input into the subsequent model training process along with classroom multimodal features, providing a unified data foundation for classroom state recognition, multimodal dynamic fusion, and comprehensive classroom evaluation.

[0088] Step 4: Model Training and Dynamic Fusion

[0089] The classroom multimodal feature set obtained in step 3 and the classroom interaction quality index (IQI) are input into the model training network to complete the learning of classroom multimodal features. To enable the model to recognize different classroom teaching stages, a classroom state awareness mechanism (CSPM) is introduced. This mechanism identifies the current classroom state based on teacher voice features, visual teaching posture features, and classroom interaction features. Its calculation process can be represented as follows:

[0090]

[0091] in, This represents the classroom state at time t. Representing speech features; Indicates visual features; represents the classroom interaction characteristics; f() represents the classroom state recognition network. The classroom state includes teaching stages such as course introduction, knowledge instruction, classroom questioning, classroom discussion, and course summary.

[0092] To further study the dynamic characteristics of classroom behavior over time, a temporal coding mechanism (TEM) is introduced. This mechanism constructs a sequence of classroom behaviors from consecutive time windows and extracts the temporal dependencies between these behaviors, represented as follows:

[0093]

[0094] Where X represents a continuous classroom sequence; H represents the classroom temporal characteristics. The classroom temporal characteristics encoded by TEM are used to describe the evolution of teachers' classroom behavior over time.

[0095] After obtaining classroom status information and classroom temporal characteristics, a dynamic adaptive weight fusion mechanism (DAWF) is introduced to dynamically adjust the importance of each modality feature according to the classroom status. The weight calculation formula is as follows:

[0096]

[0097] This represents the attention weight of the m-th modality (video, audio, or interactive behavior), used to measure the contribution of each modality feature to the global fusion. The modal energy score, calculated via the alignment network, is the original scoring metric reflecting modal significance. This represents an exponential summation of all modalities (V, A, and I) used for weighting and normalization. Modalities with higher weights have a higher proportion in the final fusion process, thus enabling adaptive adjustment of the importance of each modality's information under different classroom conditions.

[0098] Based on the calculated dynamic weights, the voice features, visual teaching features, and classroom interaction features are weighted and fused. Finally, the fused multimodal classroom feature representation is used as the input of the intelligent evaluation model of teacher teaching behavior and teacher-student interaction, providing a unified feature representation for intelligent evaluation of classroom teaching behavior.

[0099] Step 5: Intelligent Assessment and Teaching Attitude Analysis

[0100] The classroom integration time-series features obtained in step 4 are input into the intelligent assessment model of teacher demeanor and teacher-student interaction. Based on the classroom state perception results, the model dynamically calls upon the multimodal features corresponding to the classroom scenario to comprehensively analyze the teacher's language expression ability, emotional expression, body language standardization, and classroom interaction quality, and outputs the teacher's language expression score (L), teacher's emotional score (E), and teacher's body language performance score (A). s The evaluation criteria include the teacher's verbal expression score, emotional score, and physical performance score, as well as the classroom interaction quality index (IQI). Further, a comprehensive teacher demeanor score (T) is calculated based on these scores, using the following formula:

[0101]

[0102] Where L represents the teacher's verbal expression score; E represents the teacher's emotional score; A s The score represents the teacher's performance in actions; λ1, λ2, and λ3 represent the weight coefficients corresponding to language expression features, emotional features, and action features, respectively, and satisfy the following conditions: Ultimately, the evaluation results are formed based on teacher language expression, teacher emotion, teacher action, classroom interaction, and overall classroom performance.

[0103] The system invokes the Teacher Personalized Teaching Performance Baseline Model (TPM) to compare and analyze the current classroom evaluation results with the teacher's historical classroom evaluation results. It calculates the deviation of each evaluation indicator from the teacher's historical average level, realizes personalized evaluation of the teacher's classroom performance, and outputs the teacher's teaching performance evaluation results, classroom interaction evaluation results, and comprehensive classroom evaluation results, providing an evaluation basis for subsequent classroom teaching analysis and teaching suggestion generation.

[0104] Step 6: Generating Teaching Suggestions

[0105] Based on the teacher conduct evaluation results, classroom interaction evaluation results, and comprehensive classroom evaluation results obtained in step 5, classroom teaching analysis is conducted using the Teacher Personalized Teaching Conduct Baseline Model (TPM). The Teacher Personalized Teaching Conduct Baseline Model is used to statistically learn from historical classroom evaluation results and dynamically update the teacher's long-term classroom teaching characteristics based on new classroom evaluation results. The update process can be represented as follows:

[0106]

[0107] in, This represents the current baseline model of teachers' individualized teaching styles; This represents the baseline model of the teacher's personalized teaching attitude at the previous moment; This represents the current classroom profile; Indicates the model update coefficients, and The above update method adopts the idea of ​​exponential weighted average, which, while preserving the characteristics of teachers' history classroom teaching, increases the impact of recent classroom evaluation results on the updating of teacher profiles, and realizes dynamic adjustment of teachers' personalized characteristics.

[0108] After obtaining the teacher's overall teaching demeanor score (T) and the classroom interaction quality index (IQI), a teacher demeanor-teacher-student interaction coupled evaluation model (TICM) is further established to comprehensively analyze the correlation between teacher demeanor performance and classroom interaction quality. The calculation formula is as follows:

[0109]

[0110] Where T is the teacher's overall teaching demeanor score, which is calculated by combining the teacher's language expression characteristics, emotional characteristics and action characteristics; IQI is the classroom interaction quality index, used to characterize the quality of classroom interaction; α and β are the corresponding weight coefficients, satisfying (α+β=1).

[0111] Based on the TPM analysis results and TICM evaluation results, the causes of problems in teachers' classroom teaching are analyzed, the strengths and weaknesses of teachers' language expression, emotional expression, body language and classroom interaction are identified, and suggestions for improving language expression, classroom interaction, classroom rhythm, classroom organization and teaching demeanor are automatically generated. At the same time, a classroom evaluation analysis report is generated to provide auxiliary decision-making basis for improving teachers' classroom teaching ability.

[0112] Step 7: Data Management

[0113] The system uniformly stores and manages raw multimodal classroom data, classroom status data, multimodal feature data, teacher conduct evaluation results, classroom interaction evaluation results, comprehensive classroom evaluation results, and teaching suggestion results. It establishes a classroom history database, a teacher profile database, and a personalized teacher conduct baseline database, achieving structured storage and categorized management of classroom teaching data. After each classroom teaching evaluation is completed, the system automatically writes classroom video data, classroom audio data, classroom interaction data, multimodal feature data, classroom evaluation results, classroom profile information, and teaching suggestion results into the corresponding databases. Based on the newly added classroom evaluation data, the system automatically updates the personalized teacher conduct baseline model (TPM). The updated personalized teacher conduct baseline model is redeployed to the intelligent evaluation system, providing a personalized reference baseline for subsequent classroom evaluations.

[0114] Furthermore, the system establishes a unified data index and model version management mechanism to uniformly manage classroom raw data, multimodal feature data, model parameter data, and classroom evaluation results. It supports rapid querying, classification and statistics, historical tracing, and model version management of classroom data, providing data support for subsequent historical analysis, continuous model optimization, and teaching quality management.

[0115] Step 8: Historical Analysis and Network Services

[0116] The system utilizes the classroom history database, teacher profile database, and teacher personalized teaching behavior baseline database established in step 7 to conduct long-term tracking and analysis of teachers' classroom teaching behaviors. Based on the historical classroom evaluation results, the system performs longitudinal trend analysis on the classroom performance of the same teacher in different courses, at different times, and at different teaching stages. It also supports horizontal statistical analysis of classroom evaluation results among different teachers, enabling the assessment of the development and changes in teachers' teaching abilities.

[0117] Furthermore, the system combines the analysis results of the Teacher Personalized Teaching Baseline Model (TPM) and the Teacher Teaching Style-Teacher-Student Interaction Coupled Evaluation Model (TICM) to conduct long-term change analysis on evaluation indicators such as teachers' language expression ability, emotional expression, body language standardization, classroom interaction quality, and classroom organization ability. It automatically generates a report on the development trend of teachers' teaching ability and suggestions for improvement in stages, providing decision support for teachers to continuously optimize classroom teaching.

[0118] Meanwhile, teaching administrators can remotely access the system via the network to query, statistically analyze, and visualize classroom evaluation results, teacher profile information, historical trend analysis results, and model operation status. Teachers can view their individual classroom evaluation results, scores for various evaluation indicators, historical trends, and suggestions for teaching improvement, enabling network sharing, remote access, and continuous monitoring of teaching quality. As historical classroom data accumulates, the system continuously optimizes the parameters of the teacher's personalized teaching baseline model and intelligent evaluation model, achieving continuous learning and performance optimization of the teacher's intelligent classroom evaluation model.

[0119] Step 1 of this invention employs video acquisition devices, audio acquisition devices, and classroom interaction information acquisition devices to perform non-contact multimodal data collection throughout the entire classroom teaching process. Compared to traditional manual evaluation methods such as expert observation, peer review, and student evaluation, this invention can objectively acquire classroom behavior data such as teacher language expression, facial expressions, body language, and classroom interactions, avoiding the influence of subjective human factors on the evaluation results. Simultaneously, by using a unified timestamp to achieve synchronous collection of video, audio, and classroom interaction data, a unified data foundation is provided for subsequent multimodal fusion analysis, improving the completeness, authenticity, and objectivity of classroom evaluation data.

[0120] Step 2 of this embodiment of the invention performs unified preprocessing on video data, audio data, and classroom interaction data, and constructs a unified classroom timeline and sliding time window samples to achieve time synchronization and structured organization between different modalities of data. Compared with direct analysis using raw classroom data, this invention can effectively reduce the impact of environmental noise, lighting changes, and abnormal data, improving data quality and stability. Simultaneously, the unified time-series samples ensure the continuity of classroom behavior changes, providing reliable data support for subsequent multimodal feature learning and dynamic modeling of classroom behavior, thus improving the accuracy and robustness of the overall analysis.

[0121] Step 3 of this invention constructs a multimodal feature set consisting of teacher voice features, visual teaching demeanor features, and classroom interaction behavior features, and establishes a classroom status labeling system. Simultaneously, it proposes a Classroom Interaction Quality Index (IQI) to quantitatively describe the quality of classroom interaction. Compared to traditional evaluation methods that only focus on teacher behavior or student feedback, this invention comprehensively reflects both teacher demeanor and classroom interaction effectiveness, improving the completeness of classroom evaluation dimensions. The proposed IQI objectively describes classroom participation, interaction frequency, and response efficiency, providing a unified quantitative indicator for evaluating classroom interaction quality and enhancing the scientific rigor of classroom teaching quality analysis.

[0122] Step 4 of this embodiment of the invention constructs a multimodal feature learning model, introducing a classroom state awareness mechanism (CSPM) and a temporal coding mechanism (TEM) to achieve dynamic modeling of the entire classroom teaching process. Compared with traditional methods that only analyze static classroom features, this invention can identify different teaching stages such as classroom introduction, knowledge delivery, classroom questioning, classroom discussion, and course summary, and fully explore the evolution of teachers' language, actions, and classroom interaction behaviors over time. This improves the model's ability to understand the classroom teaching process and express temporal features, providing more accurate feature representations for subsequent intelligent assessment.

[0123] In step 5 of this embodiment, a dynamic adaptive weighted fusion mechanism (DAWF) dynamically adjusts the fusion weights of speech features, visual teaching characteristics, and classroom interaction features based on the classroom state. This achieves deep fusion of multimodal information in the classroom and enables intelligent evaluation of teachers' classroom teaching behavior. Compared to traditional fixed-weight fusion methods, this invention can automatically adjust the importance of each modality for different classroom teaching scenarios, making the evaluation results more consistent with the real classroom teaching patterns and improving the accuracy, stability, and environmental adaptability of the evaluation of teachers' language expression, emotional expression, body language, and classroom interaction quality.

[0124] In step 6 of this embodiment of the invention, based on the TPM analysis results and TICM evaluation results, the system performs a deviation analysis on each evaluation indicator in the teacher's classroom teaching process. Specifically, the system calculates the teacher's language expression score (L), teacher's emotional score (E), and teacher's physical performance score (A) for the current classroom. s The classroom interaction quality index (IQI) and the teacher's overall teaching attitude score (T) are compared with the historical baseline values ​​of the corresponding indicators in the Teacher Personalized Teaching Attitude Baseline Model (TPM) to calculate the relative deviation value of each indicator.

[0125] When a certain evaluation indicator decreases beyond a preset deviation threshold relative to its historical baseline, the system determines that the teaching behavior corresponding to that indicator is abnormal or needs improvement, and generates corresponding teaching improvement suggestions based on the indicator type. For example, when the Classroom Interaction Quality Index (IQI) is lower than the teacher's historical baseline and the classroom is in the knowledge delivery or discussion phase, the system generates suggestions for optimizing classroom interaction; when the teacher's Language Expression Score (L) is lower than the historical baseline and the fluctuation in speaking speed exceeds a preset range, the system generates suggestions for adjusting the rhythm of language expression; when the teacher's Action Performance Score (A) is lower than the historical baseline and the fluctuation in speaking speed exceeds a preset range, the system generates suggestions for adjusting the rhythm of language expression; when the teacher's Action Performance Score (A) is lower than the historical baseline and the fluctuation in speaking speed exceeds a preset range, the system generates suggestions for adjusting the rhythm of language expression. s When the teacher's demeanor score is below the historical baseline and visual teaching characteristics indicate insufficient teacher movement range or posture change frequency, the system generates suggestions for optimizing teaching demeanor. When the teacher's emotion score E is below the historical baseline and audio emotion characteristics or facial expression characteristics indicate a low classroom emotional state, the system generates suggestions for improving classroom engagement.

[0126] Furthermore, the system associates the aforementioned items for improvement with the corresponding classroom time windows, classroom status labels, and original multimodal data indexes to generate a structured classroom evaluation and analysis report. This report includes the name of the abnormal indicator, its relative deviation value, the corresponding classroom time segment, relevant modal evidence, causal analysis results, and teaching improvement suggestions, thereby achieving an automatic mapping from quantitative evaluation results to targeted teaching recommendations.

[0127] Step 7 of this embodiment establishes a classroom history database, a teacher profile database, and a teacher personalized teaching behavior baseline database to uniformly store and manage multimodal classroom data, evaluation results, and model data. The teacher personalized teaching behavior baseline model (TPM) is continuously updated based on newly added classroom data. Compared to the problems of traditional scattered storage of evaluation results and difficulty in continuous tracking, this invention enables long-term accumulation and dynamic modeling of teachers' classroom teaching behaviors, providing data support for classroom history analysis, continuous model optimization, and teacher profile construction, thereby improving the system's scalability and continuous learning capabilities.

[0128] Step 8 of this invention, based on a classroom history database, a teacher profile database, and a personalized teacher behavior baseline database, conducts long-term tracking and analysis of teachers' classroom teaching behavior. It also combines TPM and TICM models to perform trend analysis of teacher classroom performance and comprehensive evaluation of classroom teaching quality. Simultaneously, it supports remote network access, historical queries, statistical analysis, and visualization, enabling the sharing and management of classroom evaluation results, teacher profiles, and teaching improvement suggestions. Compared to the difficulty of long-term utilization of traditional classroom evaluation results, this invention enables continuous monitoring and intelligent analysis of teachers' teaching abilities, providing more comprehensive data support for teacher training, classroom teaching quality supervision, and educational management decisions, thereby improving the informatization and intelligentization level of teaching management.

[0129] like Figure 2 As shown, the overall system of this invention adopts a training-application combined architecture, including a training part and an application part. The training part is used to build an intelligent evaluation model of teacher teaching behavior, while the application part is used for classroom teaching behavior analysis, intelligent evaluation of teacher teaching behavior, and management of classroom evaluation results. The two parts share data and update models through a unified data management platform.

[0130] Training section of this invention embodiment:

[0131] (1) Training Data Acquisition and Processing System: The training data acquisition and processing system is used to construct the training dataset required for the intelligent evaluation model of teacher teaching behavior. The system synchronously collects multimodal classroom data through video acquisition devices, audio acquisition devices, and classroom interaction information acquisition devices, and completes processing such as time synchronization, data preprocessing, sliding window segmentation, and sample labeling to establish a standardized multimodal training sample library, providing a data foundation for subsequent model training. The system includes a device deployment and data acquisition module and a multimodal data preprocessing and sample construction module.

[0132] (2) Evaluation Model Training System: The evaluation model training system is used to construct an intelligent evaluation model of teacher teaching behavior. The system first extracts speech, visual, and classroom interaction features, and introduces a temporal coding mechanism (TEM), a classroom state perception mechanism (CSPM), and a dynamic adaptive weight fusion mechanism (DAWF) to achieve multimodal feature learning and dynamic fusion in the classroom. Subsequently, the intelligent evaluation model of teacher teaching behavior is generated through model training and parameter optimization, and then deployed to the application system. This system includes a multimodal feature learning module and a model training and dynamic fusion module.

[0133] Application portion of the embodiments of the present invention:

[0134] (1) Intelligent Evaluation System for Teacher's Teaching Demeanor and Teacher-Student Interaction: This system is used for the analysis of the entire classroom teaching process. The system calls upon a trained evaluation model to jointly analyze teachers' voice, visual, and classroom interaction behaviors, and combines this with a classroom state perception mechanism to achieve dynamic evaluation. Simultaneously, it constructs a personalized teacher teaching demeanor baseline model (TPM) and a teacher teaching demeanor-teacher-student interaction coupled evaluation model (TICM) to complete a comprehensive classroom evaluation and automatically generate a classroom evaluation report and teaching improvement suggestions. The system includes an intelligent evaluation and teaching demeanor analysis module and a teaching suggestion generation module.

[0135] (2) Data Storage and Management System: The data storage and management system is used for the unified storage, management, and sharing of classroom multimodal data, model parameters, teacher profile data, and classroom evaluation results, providing data support for the continuous tracking and analysis of teachers' classroom teaching abilities. The system establishes a classroom history database and a teacher profile database, and uniformly manages classroom videos, classroom audio, classroom interactive behaviors, multimodal features, model parameters, and classroom evaluation results. It also supports historical queries, statistical analysis, and model updates of teacher classroom evaluation results. Simultaneously, the system supports longitudinal and horizontal analysis of teachers' classroom teaching abilities and provides network transmission and remote access functions, providing data support for teacher classroom evaluation, teaching management, and teaching quality monitoring. The system includes: a data management module and a historical analysis and network service module.

[0136] Description of the functional modules of the intelligent teacher performance evaluation system based on multimodal data and teacher-student interaction provided in this embodiment of the invention:

[0137] (1) Equipment deployment and data acquisition module: This module is used to collect multimodal data of teachers in the classroom environment, including video data, audio data and classroom interaction data, and realize the synchronization and alignment of multi-source data through a unified timestamp mechanism to form the original classroom data stream;

[0138] (2) Multimodal data preprocessing and sample construction module: This module is used to denoise, enhance and standardize the collected data, and to segment the classroom data using a sliding time window to construct multimodal training samples containing video, audio and interactive information. At the same time, classroom status labels are introduced to form a structured dataset.

[0139] (3) Multimodal feature learning module: This module is used to extract speech, visual and interactive features and construct time series feature representation. At the same time, it introduces a classroom state perception mechanism to identify classroom teaching stages and combines time series coding to realize dynamic modeling of classroom behavior.

[0140] (4) Model training and dynamic fusion module: This module is used to fuse multimodal features and train the model. It adaptively adjusts the weights of each modality according to the classroom status to achieve dynamic fusion and optimizes the model parameters to improve generalization ability.

[0141] (5) Intelligent assessment and teaching behavior analysis module: This module is used to output the evaluation results of teachers' teaching behavior, construct the personalized teaching behavior baseline of teachers, compare and analyze the current classroom performance, and realize longitudinal evaluation and anomaly detection;

[0142] (6) Teaching suggestion generation module: This module is used to analyze the reasons based on the evaluation results and generate teaching improvement suggestions for teachers' classroom performance, so as to realize the transformation of evaluation results into teaching guidance;

[0143] (7) Data Management Module: This module is used to store and manage classroom multimodal data, model data and evaluation results, and supports data query and statistical analysis;

[0144] (8) Historical Analysis and Network Service Module: This module is used to perform trend analysis on teachers' historical classroom data and provides a remote access interface to realize long-term tracking and management of teachers' teaching behavior.

[0145] This invention provides an intelligent evaluation system for teacher teaching behavior based on multimodal data and teacher-student interaction, including: multimodal classroom behavior data acquisition equipment, server, and terminal display device;

[0146] Multimodal classroom behavior data acquisition equipment includes a high-definition PTZ camera, a wireless microphone / wireless lavalier microphone, and a classroom interaction information acquisition terminal. The high-definition PTZ camera is deployed at the back of the classroom to capture teachers' facial expressions, body postures, and classroom scene images. The wireless lavalier microphone is worn by the teacher, and the wireless desktop microphone is placed on the podium to capture lecture audio and teacher-student interaction audio. The classroom interaction information acquisition terminal is used to input and record interactive events such as teacher questions, student responses, and classroom discussions. The devices are synchronized with a unified timestamp.

[0147] Server: The server's built-in program is used to execute all the evaluation methods in steps 1-8 above. The server integrates: training data acquisition and processing module, evaluation model training module, intelligent evaluation module for teacher's teaching style and teacher-student interaction, data storage management module, and historical analysis network service module; the server's built-in hard disk array constructs a classroom history database, a teacher profile database, and a TPM baseline database.

[0148] like Figure 8 As shown, the terminal display device includes a PC client and a web management terminal, which are used to display the multimodal data import interface, multidimensional radar chart, time series change chart, comprehensive score, teaching diagnosis suggestions, and complete the visual interactive operation.

[0149] Network transmission components: switches and wireless routers, enabling data transmission and remote access between data acquisition devices, servers, and terminals;

[0150] like Figure 6 As shown, the entire system is deployed inside a standard smart classroom. The data collection devices collect teachers' classroom behavior without physical contact. The server is deployed in the campus computer lab, and the terminal devices are available for teachers and teaching administrators to log in and use remotely.

[0151] It should be noted that embodiments of the present invention can be implemented in hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuitry such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., or by software executed by various types of processors, or by a combination of the above-described hardware circuitry and software, such as firmware.

[0152] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for intelligently assessing teacher teaching behavior based on multimodal data and teacher-student interaction, characterized in that, include: Obtain continuous multimodal time-series samples arranged along a unified classroom timeline, wherein the multimodal time-series samples respectively contain video clips, audio clips and classroom interaction events corresponding to the time; Speech features, visual teaching features, and classroom interaction features are extracted from the multimodal time-series samples. The speech features, visual teaching features, and classroom interaction features of the current multimodal time-series sample are input into the classroom state perception mechanism, and the classroom state corresponding to the current multimodal time-series sample is output. The classroom behavior sequence composed of continuous multimodal time-series samples is input into the time-series coding mechanism, and the classroom time-series features representing the time dependency relationship between continuous multimodal time-series samples are output. The classroom state, classroom timing features, voice features, visual teaching features, and classroom interaction features are input into a dynamic adaptive weight fusion mechanism. The alignment network calculates the modal energy values ​​of the video modality, audio modality, and interaction modality respectively. The energy values ​​of the three modalities are then exponentially calculated. The result of the exponential calculation of any modality is divided by the sum of the exponential calculation results of the three modalities to obtain the dynamic weight of that modality. The sum of the dynamic weights of the three modalities is constrained to be 1. According to the dynamic weights, the voice features, visual teaching features, and classroom interaction features are weighted and fused to output classroom fusion time sequence features. The classroom fusion time sequence features are then input into the trained intelligent evaluation model of teacher teaching style and teacher-student interaction to output teacher language expression score, teacher emotion score, teacher action performance score, and classroom interaction quality index. Based on the output, teacher teaching style evaluation results and classroom interaction evaluation results are generated.

2. The intelligent assessment method for teacher teaching behavior based on multimodal data and teacher-student interaction as described in claim 1, characterized in that, The acquisition of the multimodal time series samples includes: Configure unified timestamps for video data, audio data, and classroom interaction data during the classroom teaching process; For video data, image enhancement, adaptive illumination compensation, background interference suppression, target region detection, and inter-frame synchronization are performed; for audio data, environmental noise cancellation, speech enhancement, silence removal, and speech segmentation are performed; and for classroom interaction data, abnormal event filtering, duplicate event removal, and time consistency calibration are performed. The processed video data, audio data, and classroom interaction data are mapped to a unified classroom timeline, and then divided into segments according to fixed-length sliding time windows. Overlapping areas are set between adjacent sliding time windows so that each sliding time window contains video segments, audio segments, and classroom interaction events corresponding to the time.

3. The intelligent assessment method for teacher teaching behavior based on multimodal data and teacher-student interaction as described in claim 1, characterized in that, The speech features include speech rate, tone, speech energy, and pause duration; the visual teaching features include facial expression features, human posture features, movement amplitude, and movement trajectory; the classroom interaction features include classroom questioning frequency, student response status, number of interactions, and classroom participation level; and the classroom status includes course introduction, knowledge instruction, classroom questioning, classroom discussion, and course summary.

4. The intelligent assessment method for teacher teaching behavior based on multimodal data and teacher-student interaction as described in claim 1, characterized in that, Also includes: Based on the evaluation results of teachers' history classes, a personalized teaching attitude baseline for teachers is constructed. The current classroom evaluation results are compared with the personalized teaching attitude baseline before the update, and the deviation of each evaluation indicator of the current classroom from the teacher's history level is output. Using the teacher's personalized teaching behavior baseline before the update and the current classroom profile as input, the weight of the teacher's personalized teaching behavior baseline before the update is set to 1 minus the model update coefficient, and the weight of the current classroom profile is set to the model update coefficient. The updated teacher's personalized teaching behavior baseline is then calculated according to the corresponding weights. The model update coefficient is constrained to be greater than 0 and less than 1, and the updated teacher-personalized teaching baseline is input into the subsequent classroom evaluation process.

5. The intelligent assessment method for teacher teaching demeanor based on multimodal data and teacher-student interaction as described in claim 1, characterized in that: The classroom interaction quality index is input by taking classroom participation rate, effective response rate, classroom interaction frequency and average response time as inputs. The weighted values ​​corresponding to classroom participation rate, effective response rate and classroom interaction frequency are added together and the weighted value corresponding to average response time is subtracted to output the classroom interaction quality index. The sum of the weights corresponding to the four input indicators is constrained to be 1. The input for the teacher's overall teaching demeanor score is composed of the teacher's language expression score, teacher's emotional score, and teacher's physical performance score. The overall teaching demeanor score is output by weighting the three scores according to their corresponding weights, and the sum of the weights of the three scores is constrained to be 1. The teacher's teaching demeanor comprehensive score and classroom interaction quality index are used as inputs for the coupled evaluation of teacher's teaching demeanor and teacher-student interaction. The coupled evaluation result is output according to the sum of the weighted value corresponding to the teacher's teaching demeanor comprehensive score and the weighted value corresponding to the classroom interaction quality index. The weights corresponding to the two inputs are constrained to be 1. The coupled evaluation result is input into the classroom teaching evaluation report generation process.

6. A teacher performance intelligent evaluation system based on multimodal data and teacher-student interaction, characterized in that, It includes modules for multimodal data preprocessing and sample construction, multimodal feature learning, model training and dynamic fusion, and intelligent evaluation and teaching state analysis. The multimodal data preprocessing and sample construction module is used to map video data, audio data and classroom interaction data to a unified classroom timeline, output continuous multimodal time-series samples arranged along the unified classroom timeline, and output the continuous multimodal time-series samples to the multimodal feature learning module. The multimodal feature learning module is used to extract speech features, visual teaching features, and classroom interaction features from multimodal time-series samples. It inputs the speech features, visual teaching features, and classroom interaction features of the current multimodal time-series sample into the classroom state perception mechanism and outputs the classroom state corresponding to the current multimodal time-series sample. It inputs the classroom behavior sequence composed of continuous multimodal time-series samples into the temporal coding mechanism and outputs classroom temporal features that represent the temporal dependency relationship between continuous multimodal time-series samples. The classroom state, classroom temporal features, speech features, visual teaching features, and classroom interaction features are then output to the model training and dynamic fusion module. The model training and dynamic fusion module is used to calculate the modal energy values ​​of video modality, audio modality and interactive modality respectively through the alignment network, perform exponential operation on the energy values ​​of the three modalities respectively, divide the exponential operation result of any modality by the sum of the exponential operation results of the three modalities to obtain the dynamic weight of the modality, constrain the sum of the dynamic weights of the three modalities to be 1, and perform weighted fusion of speech features, visual teaching features and classroom interaction features according to the dynamic weights to output classroom fusion temporal features; The intelligent assessment and teaching demeanor analysis module is used to input the classroom integration time sequence characteristics of the classroom to be assessed into the trained intelligent assessment model of teacher teaching demeanor and teacher-student interaction, and output the teacher's language expression score, teacher's emotion score, teacher's action performance score and classroom interaction quality index, thus forming the teacher teaching demeanor evaluation result and classroom interaction evaluation result.

7. The intelligent teacher evaluation system based on multimodal data and teacher-student interaction as described in claim 6, characterized in that, It also includes equipment deployment and data acquisition modules; The device deployment and data acquisition module is used to synchronously collect video data, audio data, and classroom interaction data during the classroom teaching process, and to configure a unified timestamp for the video data, audio data, and classroom interaction data. The multimodal data preprocessing and sample construction module is used to perform image enhancement, adaptive illumination compensation, background interference suppression, target region detection and inter-frame synchronization on video data, environmental noise cancellation, speech enhancement, silence removal and speech segmentation on audio data, abnormal event filtering, duplicate event removal and time consistency calibration on classroom interaction data, and to divide the time-aligned data into time-aligned data according to a fixed-length sliding time window, and to set overlapping areas between adjacent sliding time windows.

8. The intelligent teacher evaluation system based on multimodal data and teacher-student interaction according to claim 6, characterized in that, The model training and dynamic fusion module is also used for: The classroom interaction quality index, voice features, visual teaching style features, and classroom interaction features are input into the model to train the network; The dynamic weights of video modality, audio modality, and interactive modality are controlled by the classroom state perception mechanism output by the classroom state perception mechanism, and the temporal features of the classroom output by the temporal coding mechanism are used to maintain the temporal dependency between continuous multimodal temporal samples. The classroom integration time-series features obtained according to dynamic weights are input into the intelligent evaluation model of teacher's teaching style and teacher-student interaction. Multi-fold cross-validation is used to optimize the model parameters, and the model parameters are determined through iterative training.

9. The intelligent teacher evaluation system based on multimodal data and teacher-student interaction as described in claim 6, characterized in that, The intelligent assessment and teaching attitude analysis module is also used for: Based on the evaluation results of teachers' history classes, a personalized teaching attitude baseline for teachers is constructed. The current classroom evaluation results are compared with the personalized teaching attitude baseline before the update, and the deviation of each evaluation indicator of the current classroom from the teacher's history level is output. Using the teacher's personalized teaching behavior baseline before the update and the current classroom profile as input, the weight of the teacher's personalized teaching behavior baseline before the update is set to 1 minus the model update coefficient, and the weight of the current classroom profile is set to the model update coefficient, and the updated teacher's personalized teaching behavior baseline is output. The model update coefficient is constrained to be greater than 0 and less than 1, and the updated teacher-personalized teaching baseline is output to the subsequent classroom evaluation process.

10. The intelligent teacher evaluation system based on multimodal data and teacher-student interaction according to claim 6, characterized in that, It also includes a teaching suggestion generation module, which includes coupling analysis and report generation functions; The coupling analysis and report generation function takes teacher language expression score, teacher emotion score and teacher action performance score as input, outputs teacher teaching attitude comprehensive score according to the weighted sum of the weights corresponding to the three scores, and constrains the sum of the weights corresponding to the three scores to be 1; The coupling analysis and report generation function also takes the teacher's teaching attitude comprehensive score and classroom interaction quality index as inputs, and outputs the teacher's teaching attitude and teacher-student interaction coupling evaluation results according to the sum of the weighted value corresponding to the teacher's teaching attitude comprehensive score and the weighted value corresponding to the classroom interaction quality index, and constrains the sum of the weights corresponding to the two inputs to be 1; The coupling analysis and report generation function inputs the evaluation results of teacher's teaching demeanor and teacher-student interaction into the classroom teaching evaluation report generation process, and outputs a classroom teaching evaluation report that includes teacher's teaching demeanor score, teacher-student interaction score, comprehensive classroom score, key influencing factor analysis results, and teaching improvement suggestions.

Images