A teaching evaluation system based on large language model evidence driving

Abstract

The application discloses a kind of teaching evaluation systems based on large language model evidence driving, it is related to education information technology field.System includes multichannel transcription and error correction layer, regularized multimodal analysis layer, quantitative anchor generation layer, LLM determinant formula scoring layer and certainty algorithm score and post-processing layer.Through voice transcription, speaker separation and multilayer error correction, reliable classroom text is obtained, structured quantitative indicators are generated in combination with text, audio, picture, teacher movement, board consistency and other information, and are converted into natural language anchors according to historical statistical distribution.LLM outputs the judgment result with evidence and timestamp according to anchor, determinant and mandatory evidence format, and finally generates the score by certainty formula.The application improves the objectivity, consistency, interpretability and large-scale application efficiency of classroom teaching evaluation.

Description

Technical Field

[0001] This invention relates to the field of educational informatization technology, and in particular to a teaching evaluation system based on evidence-driven large language models. Background Technology

[0002] Classroom teaching evaluation is a crucial component in improving teaching quality and promoting teacher professional development. Traditional classroom evaluation primarily relies on manual methods such as expert observation, curriculum researchers' evaluation, or peer review. This typically requires evaluators to subjectively assess dimensions such as the achievement of teaching objectives, classroom organization, teacher-student interaction, student participation, use of teaching resources, and teaching effectiveness based on classroom observation scales. While this method possesses a certain level of professionalism, it suffers from drawbacks including high evaluation costs, long evaluation cycles, limited classroom coverage, difficulty in achieving consistent evaluation standards, and susceptibility to the influence of evaluators' experience and subjective preferences. Consequently, it fails to meet the needs for large-scale, routine, and objective monitoring of teaching quality and support for teacher development.

[0003] Existing technical solution 1: Classroom teaching evaluation method based on structured rules

[0004] The CSMS system developed by the Guangdong Provincial Academy of Educational Sciences (Chinese authorized patent CN121010485B, authorized on February 27, 2026) is a representative example. This scheme is based on 45 manually designed observation features, which are scored separately by multiple structured models and then weighted and summarized. It does not use a large language model for semantic understanding.

[0005] Existing technical solution 2: Evaluation method for teaching and research discussions based on LLM multi-agent model

[0006] This is exemplified by another authorized patent of CSMS (CN120780835B, authorized on December 5, 2025). Multiple LLM agents are used to simulate the after-class discussion process of a teaching and research group. LLM is used for discussion rather than direct grading.

[0007] Existing technical solution 3: Credibility enhancement method based on historical data post-processing

[0008] This is exemplified by the authorized patent (CN120672216B, authorized on February 3, 2026) jointly granted by Peking University and Shanghai Bibi Information Technology Co., Ltd. It calculates a dynamic smoothing coefficient using historical score sequences after AI scoring is completed. Essentially, it's a post-event verification rather than a real-time self-assessment.

[0009] Existing technical solution 4: Evaluation report generation method based on multi-model integration framework

[0010] The patent application under examination from Beijing University of Aeronautics and Astronautics (CN121303568A, filed October 14, 2025) is a representative example. It integrates object detectors, OCR, ASR, and multiple large models into a processing framework. LLM is only one of the framework components. Summary of the Invention

[0011] The objective of this invention can be achieved through the following technical solution: a teaching evaluation system based on evidence-driven large language models, comprising:

[0012] A multi-channel transcription and error correction layer is used to realize the speech-to-text transcription function and to correct errors in the transcribed text.

[0013] The rule-based multimodal analysis layer is used to output structured quantitative indicators, which refer to objective data calculated by the rule-based multimodal analysis layer for evaluating teaching quality.

[0014] The quantitative anchor point generation layer is used to convert the structured quantitative indicators into Z-score standards based on historical statistical distribution parameters, generate natural language anchor point descriptions, and allocate them according to preset judgment items; the judgment item refers to the smallest evaluation unit used to judge the quality of teaching in a certain dimension.

[0015] The LLM (Large Language Model) decision-based scoring layer is used to output the decision result based on the anchor description, the decision item, and the prompt words of the mandatory evidence output format; LLM stands for Large Language Model.

[0016] A deterministic scoring and post-processing layer is used to convert the determination result into a score.

[0017] A further embodiment of the present invention: the multichannel transcription and error correction layer includes:

[0018] The speech-to-text module is used to convert spoken content in the classroom into text content with timestamps;

[0019] The speaker separation module is used to distinguish different speaking roles from classroom audio and associate the identified role information with the transcribed text.

[0020] As a further aspect of the present invention: the multi-channel transcription and error correction layer runs a four-layer error defense system after transcription to correct errors in the transcribed text.

[0021] L1 layer: Domain dictionary correction: Establish regular expression rules to correct known systematic transcription errors;

[0022] L2 layer: LLM semantic proofreading: The transcribed full text is embedded with a preset proofreading prompt word template and sent to the LLM. The LLM outputs a JSON array, and each element of the JSON array includes a location field, an original text field, a corrected text field, a correction reason field, and a confidence level field.

[0023] L3 layer: Scoring Prompt fault-tolerance instruction: embeds transcription error fault-tolerance rules into the scoring rules;

[0024] L4 Layer: Evidence Post-processing Filtering: Perform keyword matching on all fields in the LLM output that contain deduction evidence. If an entry in the preset speech-to-text error pattern library is matched, the evidence weight is reduced or removed according to the corresponding error type.

[0025] As a further embodiment of the present invention: the regularized multimodal analysis layer includes:

[0026] IRF dialogue chain analysis module: Based on the transcribed text with speaker tags, it identifies initiation, response and feedback in the classroom, classifies the feedback, and calculates the feedback quality score by weighting the feedback type;

[0027] Bloom Question Classification Module: Sets the cognitive levels of teacher questions and student answers, and calculates the Shannon entropy normalized diversity score based on the cognitive levels of teacher questions and student answers;

[0028] Classroom Stage Detection Module: Based on the density of preset keywords in timestamped text paragraphs, classroom stages are identified as different teaching segments formed during the teaching process of a lesson;

[0029] The audio prosody analysis module includes a paragraph layer and a waveform layer. The paragraph layer extracts classroom expression features from the transcribed text, and the waveform layer extracts classroom expression features from the classroom audio.

[0030] Teacher movement analysis module: Analyzes the teacher's spatial movement and physical activities based on continuous screen changes;

[0031] Blackboard Writing-Transcription Consistency Analysis Module: Compares the relationship between visual content and classroom transcription text, and calculates a consistency score based on the degree of consistency between the blackboard writing and the topic being explained, the coverage of key content, and the timing of the appearance of the text during the explanation process;

[0032] Cross-modal event detection module: Integrates text, audio, video, and motion information to identify teaching peaks, student engagement events, teaching transition points, and silent thinking time in the classroom. Specifically, a teaching peak refers to an event occurring simultaneously within a preset time window where a teacher asks a question, changes to the blackboard or PPT, and an increase in teacher movement intensity occur. An increase in teacher movement intensity event refers to an event where the cumulative displacement of the center point of the teacher's detection frame exceeds a preset movement threshold within a unit of time. A student engagement event refers to an event where a student speaks for at least 5 seconds while the teacher remains still. A teacher remaining still refers to a situation where the displacement of the center point of the teacher's detection frame is less than a preset displacement threshold within a consecutive preset time period. A teaching transition point refers to a point in time where the stage boundary coincides with the blackboard / PPT content. A silent thinking time refers to a pause in the video for more than 3 seconds without any changes to the blackboard and the teacher remaining still.

[0033] As a further embodiment of the present invention: the quantitative anchor point generation layer includes:

[0034] The formula for generating anchor point descriptions is:

[0035] Anchor text = indicator name + current value + statistical context + Z-score standard positioning;

[0036] The allocation based on preset judgment criteria includes:

[0037] Based on the evaluation objectives of different judgment items, relevant data indicators are assigned to the corresponding judgment items; each judgment item only receives process data, result data, or reference standards that are directly related to its judgment logic.

[0038] As a further embodiment of the present invention: the LLM judgment scoring layer includes:

[0039] Scoring Dimensions: The scoring dimensions consist of multiple primary dimensions, each of which has several decision items. Each primary dimension and each decision item is assigned a corresponding weight.

[0040] Gear Mapping Table: Different gear types are set for each judgment item, and each different gear type corresponds to a different set of levels. The set of levels includes the judgment item mapping score corresponding to each level, and the set of levels forms the gear mapping table.

[0041] As a further aspect of the present invention: the formula for deterministic scoring and post-processing layer score calculation is as follows:

[0042] Scoring dimension score = ∑ (Mapping score of each judgment item × Weight of judgment item)

[0043] Total score = ∑ (score of each scoring dimension × weight of the first-level dimension).

[0044] As a further aspect of the present invention: the mandatory evidence output format includes: when LLM outputs each judgment result, it must simultaneously provide the corresponding evidence description and timestamp. The evidence description must correspond to a specific segment in the transcribed text, and the timestamp must also originate from the actual mark in the transcribed text and cannot be generated automatically or left blank; wherein, different evidence corresponds to different time positions.

[0045] As a further aspect of the present invention: the LLM judgment scoring layer also includes a real-time self-assessment mechanism for confidence.

[0046] LLM outputs a confidence field for each of the scoring dimensions. The confidence field represents the LLM's self-assessment of the current judgment result, and the self-assessment of confidence is determined according to a preset calibration rule.

[0047] As a further embodiment of the present invention, it also includes a gear boundary generation module, which uses Gaussian kernel density estimation to perform density estimation on historical rating data to obtain a density curve, and determines the gear boundary based on the local minimum points on the estimated density curve.

[0048] As a further aspect of the present invention, it also includes applying the Needleman-Wunsch global sequence alignment algorithm from bioinformatics to the stage sequence alignment of multiple classroom evaluation reports.

[0049] The beneficial effects of this invention are:

[0050] (1) This invention does not rely on subjective scoring by a large language model. Instead, it first extracts structured quantitative indicators such as IRF dialogue chains, Bloom question levels, classroom stages, audio prosody, teacher movement, blackboard consistency, and cross-modal events through a rule-based multimodal analysis layer. Then, a quantitative anchor generation layer converts these indicators into natural language anchor descriptions with historical statistical context. The LLM outputs judgment results based solely on anchors, decision items, and mandatory evidence formats, and the final score is calculated using a deterministic formula. This reduces the discretion and subjective bias of the LLM, making the evaluation results more stable and consistent across different classrooms and batches. Furthermore, each judgment can be associated with specific evidence and timestamps, facilitating review and traceability.

[0051] (2) This invention establishes a four-layer error defense system in the multi-channel transcription and error correction layer, including domain dictionary correction, LLM semantic proofreading, scoring prompt error-tolerant instructions, and post-evidence processing filtering. This mechanism can identify and correct systematic errors, semantic errors, and scoring misuse risks in classroom speech-to-text conversion at multiple stages before and after transcription, avoiding unreasonable deductions or incorrect evaluations due to individual transcription errors. At the same time, the post-evidence processing filtering can reduce the weight or eliminate evidence of suspected speech recognition errors, thereby improving the credibility and stability of teaching evaluation results.

[0052] (3) This invention combines mechanisms such as multimodal objective analysis, historical statistical distribution positioning, LLM judgment evaluation, deterministic scoring, real-time self-assessment of confidence, generation of grade boundaries, and alignment of stage sequences to automatically complete classroom data processing, evidence extraction, judgment output, and score generation. Compared with manual lesson observation or simple AI scoring methods, this invention can significantly reduce evaluation costs, shorten the evaluation cycle, and support continuous monitoring of a large number of classrooms. At the same time, through the confidence field, grade boundary generation, and alignment of stage sequences of multiple classroom reports, it can further discover evaluation uncertainties, differences in different classroom structures, and trends in changes in teachers' teaching behaviors, providing more reliable data for teachers to improve teaching and schools to carry out teaching quality management. Attached Figure Description

[0053] Figure 1 This is a diagram of the overall architecture of a five-layer pipeline for a teaching evaluation system based on evidence driven by a large language model, as described in this invention.

[0054] Figure 2 This is a comparison chart of the judgment-based scoring method of this invention and the traditional numerical scoring method;

[0055] Figure 3 This is a diagram of the four-layer STT transcription error defense system architecture of the present invention;

[0056] Figure 4 This is a flowchart of the quantitative anchor point generation and allocation according to decision items in this invention;

[0057] Figure 5 This is the design decision diagram for the hybrid gear system of this invention;

[0058] Figure 6 This is a schematic diagram of the prompt word structure of the present invention;

[0059] Figure 7 This is a flowchart of the deterministic scoring process of the present invention;

[0060] Figure 8 This is a flowchart of the automatic calibration process for the KDE gauge boundary of the present invention.

[0061] Figure 9This is a schematic diagram of the user interface evidence label and timestamp of the present invention. Detailed Implementation

[0062] The embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that these embodiments are used to illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention. Any equivalent substitutions, combinations, or improvements made based on the technical concept of the present invention should fall within the scope of protection of the present invention.

[0063] like Figure 1 and Figure 2 As shown, this invention provides a teaching evaluation system based on evidence-driven large language model. The system takes classroom video or audio as input and sequentially passes through a multi-channel transcription and error correction layer, a rule-based multimodal analysis layer, a quantitative anchor point generation layer, an LLM-based judgment scoring layer, and a deterministic scoring and post-processing layer. Finally, it generates an evaluation report with timestamped evidence, quantitative indicators, dimensional scores, a total score, and teaching improvement suggestions.

[0064] The core idea of ​​the system is as follows: First, verifiable objective data is obtained through rule-based algorithms. Then, this data is converted into natural language anchors and provided to a large language model. The large language model then makes evidence-based grade determinations, and finally, a deterministic scoring program calculates the score. This reduces the randomness brought about by the large language model directly assigning scores, making the evaluation results both traceable and verifiable.

[0065] Example 1: A Teaching Evaluation System Based on Evidence-Driven Approach of Large Language Models

[0066] Multichannel transcription and error correction layer:

[0067] A multi-channel transcription and error correction layer is used to convert classroom speech into timestamped text and correct systematic errors in the transcription results. This layer includes a speech transcription module, a speaker separation module, and a four-layer error defense system. Specifically:

[0068] Speech-to-text module:

[0069] The speech-to-text module can call at least one automatic speech recognition model. In one implementation, the automatic speech recognition model can be a Chinese speech recognition model based on the Paraformer architecture, a multilingual speech recognition model based on the Transformer architecture, or a cloud-based speech recognition service. For example, FunASR Paraformer, Whisper, or other speech recognition models with timestamp output capabilities can be used.

[0070] The speech-to-text module first extracts audio from the classroom video using FFmpeg and converts it to a uniform 16kHz, mono WAV format. FFmpeg is an open-source computer program licensed under the LGPL / GPL license. The audio is then fed into the Paraformer model for recognition, outputting segmented results containing start and end times and text content.

[0071] When the audio duration exceeds a preset threshold, the speech-to-text module automatically performs segmentation. In GPU mode, segmentation is performed when the audio exceeds 20 minutes; in CPU mode, segmentation is performed when the audio exceeds 40 minutes. A 3-second overlap is maintained between adjacent segments to reduce the probability of speech being truncated at segment boundaries.

[0072] The transcription results undergo two stages of timestamp parsing. The first stage obtains sentence-level timestamps; the second stage further segments the sentence based on punctuation marks at the end of the sentence, including ".", "!", and "?", and interpolates the timestamps of the segmented text segments according to the proportion of character positions. Finally, the speech-to-text module outputs the full text, segmented text, the start and end times of each segment, and the total audio duration.

[0073] Speaker separation module:

[0074] The speaker separation module distinguishes between teachers and students in classroom audio and writes the identified speaker roles into the transcribed text segments. This module supports two implementation schemes: CAM++ and pyannote.audio. In one implementation, the system uses the CAM++ scheme, and the process is as follows:

[0075] The first step is to use the FunASR FSMN VAD model to detect speech regions in the audio. Speech segments with a duration of less than 0.8 seconds are not included in the subsequent clustering process by the speaker separation module.

[0076] The second step is to use the CAM++ speaker model to extract a 192-dimensional speaker embedding vector for each speech region.

[0077] The third step involves using the KMeans algorithm to cluster the speaker embedding vectors into two classes, defaulting to teachers and students. If sklearn is not installed in the runtime environment, the speaker separation module reverts to threshold clustering based on cosine similarity, with the threshold set to 0.3.

[0078] The fourth step involves the speaker separation module calculating the total speaking time for each cluster, marking the cluster with the largest total speaking time as "Teacher", and the remaining clusters as "Student".

[0079] In the fifth step, the speaker separation module assigns speaker labels to each text segment output by ASR through time overlapping voting. Thus, the teacher speech proportion, the student speech proportion, and the speaker identity corresponding to each segment of text are obtained.

[0080] Four-layer error defense system

[0081] Since homophones, terms, symbols and sentence segmentation errors may occur in classroom speech transcription, in order to reduce the impact of these errors on subsequent scoring, as Figure 3 shown, the present invention sets up four layers of error defense systems from L1 to L4.

[0082] L1 layer: Domain dictionary correction:

[0083] The L1 layer uses a domain dictionary and regular expression rules to correct known systematic transcription errors. The domain dictionary contains more than 160 regular expression rules, covering Greek letter transliteration, mathematical terms, formula homophones, trigonometric function homophones, "intersection / glue" confusing words and quadrant homophones.

[0084] For example:

[0085] The regular expression "scattered angle shape" is replaced with "triangle";

[0086] The regular expression "(Afa)" is replaced with "α";

[0087] The regular expression "variance company" is replaced with "variance formula";

[0088] The regular expression "fishing porch" is replaced with "cosine".

[0089] All regular rules are sorted in descending order of expression length. The system matches the longer expressions first, and then matches the shorter expressions, so as to avoid the replacement error caused by matching "analytical expression" before "function analytical expression" is processed.

[0090] L2 layer: LLM semantic proofreading:

[0091] The L2 layer embeds the full transcription text into a preset proofreading prompt template, and sends it to the large language model. In the calling parameters, temperature is set to 0.1, and max_tokens is set to 2048.

[0092] The large language model outputs a JSON array, with each element containing the fields: position, original, corrected, reason, and confidence. The original field stores the original text fragment, with at least 8 characters; the corrected field stores the suggested correction text; the reason field explains the rationale for the correction; and the confidence field represents the model's self-assessed confidence level in the suggested correction.

[0093] The system only applies corrections whose confidence level reaches a preset threshold. All corrections are executed sequentially after being sorted by the position field from largest to smallest to avoid the previous replacement action changing the position of subsequent text.

[0094] L3 layer: Scoring Prompt fault-tolerant instruction:

[0095] Layer L3 incorporates transcription error tolerance rules into the scoring prompts. Large language models must adhere to the following constraints when performing scoring:

[0096] Chinese transliterated words may correspond to mathematical or scientific symbols, and cannot be judged as terminology errors based solely on the literal text.

[0097] The mixing of Chinese and English may be caused by language switching in the speech recognition model and should not be used as a direct basis for deduction.

[0098] Repeated phrasing or excessive use of interjections may originate from echoes or transcription artifacts.

[0099] When judging whether teachers' language is standard, literal distortion caused by speech transcription should be excluded;

[0100] Evidence suspected of being a transcription error will be marked as ignored / invalid evidence, with a score of 0, and will not be included in the deduction calculation.

[0101] L4 Layer: Post-Evidence Processing Filtering

[0102] Layer L4 is executed after the large language model has completed scoring. The system scans all evidence fields containing the meaning of deductions, including cited text and reasons for judgment. The system maintains a list of STT homophone confusion pairs and 10 sets of confusion pairs, such as "company / formula", "glue / interchange", "flip hands / flip books", and "pictograph / quadrant".

[0103] When a piece of evidence for point deduction meets both of the following conditions, the system determines that the evidence was caused by a speech-to-text error:

[0104] The cited text contains incorrect words on the wrong side of a word pair;

[0105] The reasons for the judgment include keywords related to the complaint, such as "terminology," "words," "expression," "misspelling," and "non-standard."

[0106] This approach does not reduce the weight of the deduction, but rather directly eliminates the evidence of deduction triggered by the transcription error.

[0107] Regularized multimodal analysis layer:

[0108] The rule-based multimodal analysis layer is used to extract objective quantitative indicators from text, audio, video footage, and classroom structure. These indicators are calculated directly by the algorithm, without relying on subjective reasoning from a large language model.

[0109] Structured quantitative indicators include the number of teacher questions, the number of independent student responses, the number of complete IRF dialogue chains, IRF chain density, question diversity index, student response cognitive depth score, teacher feedback quality score, teacher speaking speed, start and end times and duration of each teaching stage, and blackboard keyword consistency score. Specifically:

[0110] IRF Dialogue Chain Analysis Module

[0111] The IRF dialogue chain analysis module identifies teacher-initiated responses, student responses, and teacher feedback—the IRF triples—based on transcribed text with speaker tags.

[0112] The system first identifies the teacher's question as I, or Initiation; then it looks for the subsequent student's answer as R, or Response; and finally, it looks for the teacher's feedback on the student's answer as F, or Feedback. If all three parts exist, a complete IRF dialogue chain is formed.

[0113] Teacher feedback was categorized into four types:

[0114] Simple affirmation, in-depth questioning, turning to others, and corrective guidance. Among these:

[0115] Simple positive feedback includes "good," "okay," "right," "correct," "um," "not bad," "very good," "correct," "you got it right," "well said," "great," "okay," and "please sit down." Simple positive feedback is weighted at 30 points.

[0116] In-depth follow-up questions include "Why?", "Why this happened?", "How did you arrive at this conclusion?", "Anything else?", "Continue?", "Go on?", "Can you explain in detail?", "Can you elaborate further?", "What is the basis for this?", and "How was it derived?". In-depth follow-up questions are weighted at 100 points.

[0117] Turning to others for feedback includes options such as "Who can add something?", "Any other opinions?", "What do other students think?", "What do you all think?", "Who else can answer?", "Who else can add something?", "Discuss with your deskmate?", and "Share in your group." Turning to others for feedback is weighted at 60 points.

[0118] Corrective guidance feedback includes phrases such as "Incorrect," "Wrong," "Should be," "Correct is," "Think again," "Look carefully," "There's a problem here," "Pay attention to the steps," and "Pay attention to the concepts." Corrective guidance feedback is weighted at 80 points.

[0119] The system identifies the feedback type of each IRF chain and calculates the teacher feedback quality score according to the weights mentioned above.

[0120] Bloom Problem Classification Module:

[0121] Bloom's question classification module is used to analyze the cognitive level of teachers' questions and the cognitive depth of students' answers.

[0122] Teacher question identification uses a combination of strong and weak markers. Strong markers include "why," "please answer," and "how," which are automatically counted as questions upon being matched. Weak markers include "right or wrong," "is it," "do you know," and "do you understand," which are only counted as valid questions if the text length reaches 8 characters, thus avoiding misjudging verbal confirmations as teaching questions.

[0123] Bloom's question classification module also includes several filtering rules:

[0124] Unmarked segments that last longer than 30 seconds will not be counted in the question;

[0125] If a student's answer lasts longer than 15 seconds, it will not be counted as a single student's answer.

[0126] If a student's answer exceeds 200 characters, it will not be counted as a single student's answer;

[0127] Adjacent paragraphs are deduplicated using the Jaccard similarity algorithm for character triples, and paragraphs with a similarity greater than 0.75 are merged.

[0128] Teacher questions are categorized into six levels according to Bloom's cognitive hierarchy: memory, comprehension, application, analysis, evaluation, and creation. The Bloom question classification module uses regular expression keyword matching, performing the initial matching in the order of "memory, comprehension, application, analysis, evaluation, creation." If no rule is matched, the question is defaulted to the "comprehension" category.

[0129] Examples of keywords for the six types of questions are as follows:

[0130] Memory-based: What is, what is called, definition, state, list, retell, recite, remember, recall, who knows;

[0131] Comprehension type: Explain, summarize, generalize, conclude, compare, what are the differences, what does it mean, and how to understand it in your own words;

[0132] Application-oriented: Please calculate, please solve the problem, apply, use, give examples, for example, how to calculate, how to solve, how to do it;

[0133] Analytical type: why, reason, cause, analysis, reasoning, deduction, argumentation, proof, what relationship;

[0134] Evaluation type: What do you think? What do you feel? What do you think? Evaluation, judgment, what's good about it? Is it reasonable? Do you agree?

[0135] Creative: if, supposing, assuming, envisioning, designing, new methods, new solutions, improving, optimizing, predicting.

[0136] The Bloom question classification module calculates the Shannon entropy based on the distribution of the six question types and normalizes it to obtain the question diversity index.

[0137] Student responses are categorized into five types based on cognitive depth: restatement, explanation, questioning, transfer, and creation. Bloom's question classification module uses the priority of "creation, transfer, questioning, explanation, and restatement" for matching.

[0138] Here are some examples of keywords used by students in their answers:

[0139] Creative: I think, my opinion, my idea, it can be done this way, another, design, solution, new method, improvement, optimization;

[0140] Transfer-type: for example, for instance, in life, in reality, last time, learned before, connect, associate, apply;

[0141] Questioning type: Why, I don't understand, I don't get it, that's not right, I don't think so, I disagree, I still have questions, doubts;

[0142] Explanatory: because, therefore, due to, thus, the reason why, is, means, explains, indicates, firstly, secondly, then;

[0143] Paraphrasing type: Answers from students who did not meet any of the above four rule categories.

[0144] Bloom's question classification module assigns a weighted score to the depth of student responses: restatement is 1 point, explanation is 2 points, questioning is 3 points, transfer is 4 points, and creativity is 5 points.

[0145] Classroom phase assessment module:

[0146] The classroom phase assessment module is used to identify the teaching segments within a lesson. The module predefines eight classroom phases:

[0147] Introduction; Review; Interactive Q&A; Knowledge Explanation; Example Demonstration; Group Discussion; Practice and Consolidation; Summary and Conclusion.

[0148] Each stage corresponds to a keyword library. For example, the "Introduction" stage includes keywords such as "Hello students," "Let's learn today," "Class begins," "Stand up," and "Please sit down"; the "Knowledge Explanation" stage includes keywords such as "What is," "Definition," "Concept," "Formula," "Theorem," "First," "First," and "Key Point"; and the "Summary and Conclusion" stage includes keywords such as "Let's summarize," "What we learned today," and "get out of class dismissed."

[0149] The testing process is as follows:

[0150] Merge consecutive text paragraphs into a context window, with a maximum window length of 60 characters and a maximum interval of 3 seconds between adjacent paragraphs;

[0151] Match each window to a keyword library for each stage and label the stage type;

[0152] Merge paragraphs with consecutive identical phases into a phase block;

[0153] Phase blocks with a duration of less than 30 seconds are merged into adjacent phases;

[0154] If there is an "other" stage block between two identical stages that lasts less than 3 minutes, then that "other" stage block is absorbed and merged.

[0155] Finally, the classroom phase assessment module outputs the start and end times, duration, and phase name for each classroom phase.

[0156] Audio prosody analysis module:

[0157] The audio prosody analysis module includes paragraph-level analysis and waveform-level analysis.

[0158] Paragraph-level analysis is performed directly based on transcribed text and timestamps, calculating teacher speaking speed, number of pauses, average pause duration, paragraph duration variation coefficient, and teacher-student speaking ratio.

[0159] The formula for calculating the coefficient of variation of paragraph duration, tempo_cv, is as follows:

[0160] tempo_cv = Standard deviation (paragraph duration) / Mean (paragraph duration)

[0161] The audio prosody analysis module interprets this metric based on the following thresholds:

[0162] tempo_cv < 0.5 is defined as stable tempo;

[0163] 0.5 ≤ tempo_cv < 1.0 is defined as a detectable change in rhythm;

[0164] tempo_cv ≥ 1.0 is defined as the tempo fluctuation reaching the warning threshold of the audio prosody analysis module.

[0165] Waveform layer analysis extracts RMS energy, F0 fundamental frequency, and spectral centroid from the audio waveform. To control resource consumption, the audio prosody analysis module employs triple memory protection measures:

[0166] Skip waveform layer analysis when available memory is less than 2GB;

[0167] Skip waveform layer analysis if the audio duration is greater than 15 minutes;

[0168] If the audio file is larger than 80MB, skip waveform layer analysis.

[0169] When librosa is not available in the runtime environment, the audio prosody analysis module falls back to the wave module in the Python standard library to extract energy features in a 100Hz block-level RMS manner.

[0170] Teacher mobility analysis module:

[0171] The teacher movement analysis module analyzes the teacher's spatial movement and body movements based on pixel changes between consecutive keyframes of the video.

[0172] The teacher motion analysis module converts adjacent keyframes into grayscale images and standardizes their dimensions. Then, PILImageChops is used to calculate the difference image between the two frames, and the RMS value of the difference image is calculated as the amount of image movement.

[0173] The screen is divided into four quadrants: upper left, upper right, lower left, and lower right. The teacher movement analysis module calculates the movement in each quadrant to form a spatial distribution map of the teacher's activities.

[0174] The teacher movement analysis module also identifies gesture burst points and spatial entropy. When the amount of movement within a certain time window is greater than 1.5 times the average amount of movement throughout the lesson, and this state occurs in consecutive windows, the teacher movement analysis module marks that interval as a gesture burst point.

[0175] Spatial entropy is used to describe the distribution of teachers' activity areas. The teacher movement analysis module calculates simplified entropy values ​​based on the proportion of movement in each of the four quadrants:

[0176] An entropy value > 0.6 indicates that the activity covers three to four quadrants;

[0177] An entropy value of 0.3 ≤ entropy ≤ 0.6 indicates that activity is concentrated in one or two quadrants;

[0178] An entropy value < 0.3 indicates that activity is concentrated in or close to a single quadrant.

[0179] The results of teacher movement analysis will be mapped to the corresponding classroom stage for subsequent judgment of classroom organization, blackboard writing coordination, and peak teaching events.

[0180] Whiteboard writing - Transcription consistency analysis module:

[0181] The blackboard-transcription consistency analysis module is used to compare the degree of consistency between the visual content in the classroom and the text explained by the teacher.

[0182] The whiteboard-transcription consistency analysis module calculates the consistency score from three directions:

[0183] Keyword overlap rate: Frequency-weighted Jaccard alignment of keywords in the whiteboard and transcribed text;

[0184] Thematic coherence: Analyzing the distribution of blackboard content across different stages of the lesson;

[0185] Time alignment: Determine whether the appearance time of the term on the blackboard and the corresponding explanation time in the transcribed text fall within a 30-second window.

[0186] The blackboard-to-transcription consistency analysis module integrates the results from the above three directions and outputs a blackboard-to-transcription consistency score from 0 to 100.

[0187] Cross-modal event detection module:

[0188] like Figure 4 As shown, the cross-modal event detection module integrates text, audio, video, and teacher movement information to identify four types of classroom events:

[0189] Peak teaching events: Three types of signals appear in the same time window: teacher asking questions, changes in blackboard or PPT content, and increased teacher movement.

[0190] Student engagement event: The duration of a student's speech is no less than 5 seconds, and the amount of teacher screen movement is less than the static threshold of the cross-modal event detection module;

[0191] Teaching transition point: The boundary of a class stage coincides with the point in time when the content of the blackboard or PPT changes;

[0192] Silent reflection time: The classroom pause lasts for more than 3 seconds, during which there is no change in the blackboard writing and the teacher remains still.

[0193] These events will serve as quantitative anchors and evidence candidates, which will be provided to subsequent scoring layers.

[0194] Quantitative anchor point generation layer:

[0195] The quantitative anchor generation layer is used to convert the numerical indicators output by the rule-based multimodal analysis layer into natural language anchors that can be directly understood by the large language model.

[0196] The quantitative anchor generation layer first retrieves statistical distribution parameters, including mean and standard deviation, from the historical database for similar classrooms. Then, it calculates the Z-score based on the current classroom indicator values.

[0197] Z-score = (Current value - Historical mean) / Historical standard deviation

[0198] When the historical standard deviation is 0, the preset replacement standard deviation is used.

[0199] The quantitative anchor generation layer then generates anchor text according to the following format:

[0200] Anchor text = Indicator name + Current value + Statistical context + Z-score location description

[0201] For example, if a teacher's speaking speed in a classroom is 4.5 words per second, the built-in quantitative anchor point generation layer has a typical classroom speaking speed range of 3 to 6 words per second. The quantitative anchor point generation layer generates anchor points as follows:

[0202] The teacher's speaking speed was 4.5 words per second, which falls within the normal range of 3 to 6 words per second for regular classes.

[0203] For example, if a teacher asks questions at a frequency of 2.8 times per minute in a certain classroom, while the average for similar history classes is 3.5 times per minute, with a standard deviation of 1.0, then the quantitative anchor point generation layer generates anchor points as follows:

[0204] The teacher's questioning frequency was 2.8 times per minute, which is 0.7 standard deviations lower than the average of similar historical classes.

[0205] Anchor points are not independent scoring results, but rather objective reference information used by large language models to determine grade levels.

[0206] Anchor points are assigned according to decision criteria:

[0207] The quantitative anchor point generation layer assigns anchor points according to 27 decision items. A decision item is the smallest evaluation unit used to determine the quality of teaching in a certain dimension; each decision item only receives data directly related to that decision item.

[0208] For example:

[0209] "Student Participation Breadth" includes statistics on the number of independent speakers, total student speaking time, response pace, and pauses.

[0210] "Whether the speaking rate falls within the norm range" receives the teacher's speaking rate, which is 3 to 6 words per second.

[0211] "Time Allocation and Closed Loop" receives the duration and completeness of each classroom stage;

[0212] "Blackboard writing synchronization" receives the blackboard writing-transcription consistency score;

[0213] "Completeness of teaching process" includes the number of classroom stages, the type of stage, and the order of the stages.

[0214] For "completeness of teaching process", the quantitative anchor point generation layer uses the following rules to generate anchor points:

[0215] If a stage contains both "import" and "summary and generalization", and the number of stage types after deduplication is no less than 4, then a "complete stage" anchor point will be generated.

[0216] If the number of stage types after deduplication does not exceed 2, then an anchor point "insufficient number of stage types" will be generated.

[0217] If the number of stage types after deduplication is 3, then an anchor point for "basic coverage of stages" will be generated.

[0218] The quantitative anchor point generation layer also checks whether the stage sequence conforms to the basic teaching logic of "introduction; review; interactive Q&A; knowledge explanation; example demonstration; group discussion; practice and consolidation; summary and generalization". Deviations will be written into the anchor point description.

[0219] For "student participation breadth," the quantitative anchor generation layer, in addition to counting the number of independent speakers, also uses tempo_cv and pause statistics to describe the pace of classroom responses. The pause statistics record the number of times the interval between adjacent paragraphs is no less than 1.5 seconds and the average pause duration, which are used to reflect students' thinking time and the pace of classroom dialogue.

[0220] LLM-based decision-based scoring layer:

[0221] like Figure 5 As shown, this system employs a hybrid tier design with both 4 and 5 tiers in its LLM judgment-based scoring layer. Different judgment items rely on different types of evidence. Some judgment items can be directly supported by quantitative indicators calculated by the system, such as teacher speaking speed, number of independent student speeches, number of teaching segments, and blackboard writing synchronization. Other judgment items primarily rely on the original classroom text, teaching expression, and contextual semantics, such as the accuracy of knowledge explanation, the appropriateness of examples, and the inspirational nature of teacher responses. Using the same tier structure for both types of judgment items can easily lead to a mismatch between the granularity of judgment and the type of evidence.

[0222] Therefore, this system divides the 27 judgment items into two categories: one category consists of judgment items supported by quantitative anchors, which adopt a 5-level design; the other category consists of judgment items based on textual evidence and semantic judgment, which adopt a 4-level design.

[0223] The 5-level design is used for decision items where the system has already generated clearly defined quantified anchor points. These decision items have calculable objective indicators and can be compared with historical norms, threshold ranges, or statistical distributions, thus allowing for more granular intermediate levels. The decision items using the 5-level design in the system include:

[0224] Student participation breadth is anchored by metrics including the number of independent speakers, student speaking time, response rhythm, and pause statistics.

[0225] The completeness of the teaching process is anchored by the number of classroom stages, the type of stage, whether it includes introduction and summary, and the order of the stages.

[0226] Time allocation and closed loop, with corresponding anchor points including the duration of each teaching stage and the closure status of each stage;

[0227] Whether the speech rate falls within the norm range, the corresponding anchor points include the teacher's speech rate, the norm range is 3 to 6 words / second;

[0228] The synchronization between blackboard writing and transcription is reflected in the corresponding anchor points, including the blackboard writing-transcription consistency score.

[0229] Time management ability, with corresponding anchor points including deviations in session duration, pause statistics, and progress of classroom stages.

[0230] When generating scoring prompts, the system writes the grade type, the set of allowed output options, and the corresponding metric description for each judgment item. For 4-grade judgment items, only A, B, C, and D are allowed in the prompts; for 5-grade judgment items, only A, B, C, D, and E are allowed. After the large language model outputs, the system performs a validity check. If a judgment item outputs an option that does not belong to the grade set for that question, the system marks that item as having a format error and initiates a JSON error recovery or re-request process. Grades that fail the validity check will not be directly included in the scoring.

[0231] The LLM decision-based scoring layer receives three inputs: transcribed text, rubric cue words, and quantitative anchors. The Large Language Model (LLM) does not directly output scores; instead, it outputs an A / B / C / D or A / B / C / D / E rating for each decision item.

[0232] The LLM decision-based rating layer consists of 6 rating dimensions, totaling 27 decision items:

[0233] The knowledge point explanation, with a dimension weight of 20%, includes 5 four-level judgment items;

[0234] Teacher-student interaction, with a dimension weight of 20%, includes four 4-level judgment items and one 5-level judgment item;

[0235] The teaching logic, with a dimension weight of 15%, includes two 4-level judgment items and two 5-level judgment items;

[0236] Language teaching demeanor, with a weight of 15%, includes four 4-level judgment items and one 5-level judgment item;

[0237] The blackboard design, with a dimension weight of 15%, includes three 4-level judgment items and one 5-level judgment item;

[0238] Classroom management, with a dimension weight of 15%, includes three 4-level judgment items and one 5-level judgment item.

[0239] Among them, five judgment criteria are used for data items with quantification anchors, including:

[0240] Breadth of student participation;

[0241] Completeness of teaching process;

[0242] Time allocation and closed loop;

[0243] Does the speech rate fall within the norm range?

[0244] Blackboard writing should be synchronized with the writing process.

[0245] Time management skills.

[0246] The remaining 21 judgment items adopt a 4-level design for qualitative judgment based on textual evidence, including: scientific errors, depth of explanation, quality of examples, emphasis on key points and difficulties, logical progression; diversity of interactive forms, quality of teacher response, timely adjustment, substantive teaching function, smoothness of connection, principle of progression, vividness and appeal, standardization of word choice, frequency of verbal tics, teaching demeanor, structural hierarchy, emphasis on key points, service to teaching, order, attention input, and classroom atmosphere.

[0247] The LLM decision-based scoring layer also includes a mandatory evidence output function:

[0248] Each decision item requires the large language model to output the evidence and timestamp fields. The LLM decision scoring layer sets timestamp rules in the prompt words:

[0249] Each piece of evidence must include a timestamp field;

[0250] The timestamp must come from a timestamp in the transcribed text;

[0251] Different pieces of evidence must not use the same timestamp;

[0252] Timestamps must not be omitted under the pretext of "overall evaluation".

[0253] In the front-end report, each piece of evidence is clickable and will take you to the corresponding moment in the classroom video.

[0254] The LLM-based decision-based scoring layer also includes a real-time self-assessment mechanism for confidence:

[0255] The large language model also needs to output a confidence field for each rating dimension, with a value ranging from 0.0 to 1.0. The LLM decision rating layer provides confidence calibration rules in the prompt words:

[0256] When there are at least three original citations and the evidence directly corresponds to the gauge grade, the confidence level is set between 0.90 and 1.00.

[0257] When there are at least two original citations and most of the evidence directly supports the judgment, the confidence score is set between 0.75 and 0.89.

[0258] When relying primarily on reasoning and citing no more than one original text, confidence should be set between 0.60 and 0.74.

[0259] When there is only textual evidence and no supporting visual or audio data, the maximum confidence score is 0.85.

[0260] When the corresponding classroom segment is missing, the confidence limit for this dimension is 0.70.

[0261] Deterministic scoring and post-processing layer:

[0262] The deterministic scoring and post-processing layer is used to convert the levels output by the large language model into scores and generate the final evaluation report.

[0263] like Figure 6 As shown, the deterministic scoring and post-processing layers use a fixed-level mapping table:

[0264] The four question types are mapped as follows:

[0265] A = 95;

[0266] B = 80;

[0267] C = 60;

[0268] D = 30.

[0269] The five levels of questions are mapped as follows:

[0270] A = 95;

[0271] B = 85;

[0272] C = 75;

[0273] D = 65;

[0274] E = 30.

[0275] like Figure 7 As shown, the formula for calculating the dimension score is:

[0276] Dimensional score = Σ (Decision item mapping score × Decision item weight)

[0277] The formula for calculating the total score is:

[0278] Total score = Σ (dimension score × dimension weight)

[0279] The weights of the decision items are pre-configured by the deterministic scoring and post-processing layers. When there is no special weight configuration within a certain dimension, the decision items within that dimension are calculated with equal weight.

[0280] The post-processing stage includes the following steps:

[0281] Perform JSON fault tolerance recovery on the output of the large language model. The recovery strategies are as follows: direct parsing, removal of Markdown tags, balanced extraction of brackets, truncation repair, and field-by-field extraction using regular expressions.

[0282] Perform L4 layer STT error filtering to eliminate evidence of deduction caused by speech transcription errors;

[0283] Combine the scoring results, judgment details, and evidence citations from all dimensions to generate a complete evaluation report;

[0284] This step generates teaching reflection suggestions based on a pre-set template library and does not call external large language models.

[0285] Save the feature vectors and scoring results for subsequent model training;

[0286] Write the complete report to the database.

[0287] Example 2: Gear Boundary Generation Module

[0288] In another embodiment, the system further includes a gear boundary generation module. For example... Figure 8 As shown, this module uses Gaussian kernel density estimation to find natural dividing points from historical rating data, which helps generate rating level boundaries.

[0289] This module does not correct individual teachers' ratings or smooth individual scores; it is only used to find grade boundaries from historical distributions.

[0290] The algorithm flow is as follows:

[0291] The first step is to perform IQR outlier filtering. The gear position boundary generation module calculates Q1, Q3, and IQR, where IQR = Q3 - Q1. The filtering interval is from Q1 - 1.5 × IQR to Q3 + 1.5 × IQR. If the number of samples after filtering is less than 75% of the original number of samples, the gear position boundary generation module retains the original data.

[0292] The second step involves using `scipy.stats.gaussian_kde` to estimate the kernel density of the filtered scores. The gear boundary generation module generates 200 density grid points in the score range of 0 to 100 and identifies local minima through derivative zero-crossing points. For multiple local minima within a range of 10 grid points, the gear boundary generation module retains the one with the lowest density value.

[0293] The third step involves the grade boundary generation module selecting the first n-1 minimum values ​​based on density values ​​from low to high as the scoring boundaries for n grades. n is determined by the number of target grades; n=4 when there are 4 target grades and n=5 when there are 5 target grades. If the number of boundaries generated by KDE is insufficient, the grade boundary generation module reverts to the percentile scheme, successively using boundary gap constraints of 5.0, 3.0, and 0.0. If sufficient boundaries are still not obtained, tertiary boundary generation is used.

[0294] The fourth step involves the gear boundary generation module detecting degenerate boundaries. If the minimum boundary spacing is less than 1.0 point, the module determines that the boundary originates from a numerical artifact on a near-flat density curve and replaces it with an equidistant tertile.

[0295] The fifth step is to clamp all boundaries to between 5 and 95 points and remove duplicates at a granularity of 2 points.

[0296] The gear boundary generation module adds method markers to the boundary generation results, including kde, kde+percentile, percentile_only, and percentile_concentrated, to record the data processing path used in this boundary generation.

[0297] Example 3: Classroom Stage Sequence Alignment Based on Needleman-Wunsch Algorithm

[0298] In another implementation, the system applies the Needleman-Wunsch global sequence alignment algorithm from bioinformatics to the stage sequence alignment of multiple classroom evaluation reports.

[0299] The system first extracts the sequence of classroom stages from multiple reports. For example, the stage sequence of a report can be represented as "Introduction, Knowledge Explanation, Interactive Q&A, Practice and Consolidation, Summary and Conclusion".

[0300] The system establishes 16 synonymous stage term clusters, covering more than 180 stage name variations. For example, "Import," "Introduction," "Classroom Introduction," "Contextual Introduction," "Review Introduction," and "Problem Introduction" belong to the same term cluster. The stage similarity within the same term cluster is set to 0.8, and the penalty for empty spaces is set to -2.

[0301] After sequence alignment, the system constructs a three-dimensional scoring matrix of "stage × dimension × teacher". Each cell in the matrix stores a teacher's score in a certain stage and on a certain scoring dimension.

[0302] The system then performs three types of analysis:

[0303] Scan the matrix stage by stage to find the teacher with the highest score in each stage and each dimension, and extract the corresponding evidence as the practice explanation for the highest score in that stage;

[0304] The "stage × dimension" combination where all teachers' scores are below a preset threshold is identified as a common area for improvement.

[0305] Calculate the score deviation of each teacher relative to other teachers in each dimension. When the deviation is greater than 1 standard deviation, mark that dimension as the teacher's key teaching strength.

[0306] Finally, the system selects the classroom segment descriptions from the teachers with the highest scores at each stage and splices them together to generate a textual description of a "composite classroom." This description is used for teaching research analysis and does not replace the original evaluation reports of individual teachers.

[0307] Example 4: Unified OpenAI-compatible interface layer

[0308] In another implementation, the system adapts to various AI service providers through a unified OpenAI-compatible interface layer. This interface layer supports eight service types: DeepSeek, OpenAI, SiliconFlow, OpenRouter, Groq, Doubao, Ollama, and MiMo.

[0309] The system uses process-level global semaphores to control the number of concurrent requests:

[0310] Cloud services such as DeepSeek and OpenAI have a concurrent user count of 6.

[0311] The free tier services such as SiliconFlow have a concurrency limit of 4.

[0312] The local Ollam service has a concurrency limit of 1.

[0313] The LLM call uses `stream=True`. The system performs incremental JSON parsing as the response stream arrives. Once the complete result for a certain dimension is parsed, a front-end callback is triggered, allowing the user to see the progress of the dimension-by-dimensional scoring on the page without waiting for all dimensions to be scored.

[0314] Example 5: Keyframe Extraction Process Based on Teaching Semantics

[0315] In another embodiment, the system provides a teaching semantic weighted keyframe extraction method for extracting 15 to 20 keyframes from a 40-minute classroom video. The method is performed in the following five steps.

[0316] Step 1: Identification of Instructional Structure

[0317] The system identifies classroom structure by combining rules and a large language model. The rule component uses predefined keywords, Bloom hierarchy rules, and student response depth rules; the large language model component supplements semantic information that the rules cannot cover.

[0318] The two types of results are combined using a union set. That is, as long as a paragraph is identified as a teaching structure node by either the rules or the large language model, that paragraph will be included in the candidate set.

[0319] Step 2: Multimodal candidate frame extraction

[0320] The system generates candidate keyframes from three sources:

[0321] Scene transition detection;

[0322] Speaker switch detection;

[0323] Write ROI detection on the blackboard.

[0324] Scene transition detection is based on the chi-square distance of the HSV histogram. The system samples video frames at 3-second intervals, with a sampling frequency of 0.33 fps, processing a maximum of 2000 sampling points. The system converts adjacent frames to the HSV color space, calculates the HSV histogram, and uses the following chi-square distance formula to calculate the change:

[0325] chi2 = 0.5 × Σ(diff² / denom)

[0326] When the chi-square distance exceeds 0.35, the system determines it as a scene switching point.

[0327] The whiteboard ROI detection uses the right 40% area of ​​the screen as the candidate whiteboard area and calculates the edge density in this area.

[0328] Step 3: Weighting of Teaching Semantics

[0329] The system calculates a teaching semantic score for each candidate frame. This score is determined by both the importance of the event and the amount of visual information.

[0330] Visual information content includes two sub-indicators:

[0331] Information content on the blackboard: Take the right 40% area of ​​the screen, calculate the proportion of pixels with a gradient magnitude greater than 30, and normalize it to 0 to 1 according to "edge density / 0.12";

[0332] Scene change information: The chi-square distance is normalized to 0 to 1 using "chi2 / 0.7".

[0333] The total score for visual information is obtained by weighting the amount of information written on the blackboard and the amount of information about scene changes, with the weight of blackboard information being 0.20 and the weight of information about scene changes being 0.25.

[0334] Step 4: Diversity-constrained greedy choice

[0335] The system employs a greedy strategy to select the final keyframe from candidate frames. Each time, it selects the candidate frame with the highest current score that satisfies the constraints.

[0336] The constraints include:

[0337] At least two frames should be selected for each teaching stage;

[0338] At least one frame must be selected for each event type;

[0339] The time interval between adjacent keyframes shall not be less than 90 seconds;

[0340] The perceptual hash Hamming distance between adjacent keyframes is no less than 12.

[0341] The formula for calculating the candidate frame score is:

[0342] Candidate frame score = Instructional semantic score + Stage quota reward 0.5 + Event quota reward 0.3

[0343] If fewer than 12 keyframes are selected under the 90-second interval constraint, the system will adjust the interval constraint to 45 seconds and reselect.

[0344] Step 5: Batch Automatic Labeling

[0345] The system sends the final selected keyframes to the visual big language model, which generates an 8-field structured description for each keyframe. This description includes the time point, classroom stage, screen content, blackboard content, teacher behavior, student status, teaching event type, and evaluation-related information. The generated keyframe descriptions are then written into the evaluation report to support evidence analysis across dimensions such as blackboard design, classroom management, teacher-student interaction, and teaching logic.

[0346] like Figure 9 As shown, the system's user interface includes a dimension result display area, a judgment item evidence display area, and a video playback linkage area.

[0347] The top of the interface displays the dimension results, showing the name of the current scoring dimension, its score, and the confidence level of the large language model's assessment of that dimension. For example, the image shows the current dimension as "Knowledge Point Explanation," with a score of 80.00 and a confidence level of 0.85. Users can directly view the overall evaluation results for the current dimension through this area.

[0348] The central part of the interface displays the evidence for each decision item. Each decision item is presented as an independent evidence card, which includes the decision item name, the selected model level, the evidence text, and the corresponding timestamp label. For example, the first evidence card displays "Decision Item 1: Scientific Error, Selection: B", and provides the evidence "The teacher's statement on the Pythagorean theorem is accurate (05:30)". Below this evidence are timestamp labels "[05:30]" and "[18:20]", along with the annotation "→ Video Jump". The second evidence card displays "Decision Item 2: Depth of Explanation, Selection: B", with the evidence being "Using climbing stairs as an analogy for function increment (08:45)", and a timestamp label "[08:45]".

[0349] Each timestamp is linked to a classroom video player. When a user clicks on a timestamp such as "[05:30]", "[18:20]", or "[08:45]", the system sends a location command to the video player, causing the video playback to jump to the corresponding time. This allows users to directly review the original classroom segment from the scoring results and verify the classroom evidence upon which the judgment was based.

[0350] The bottom of the interface features a video playback area that displays the classroom video player and its current playback progress. The example in the image shows the current playback time as 25:30, with a total video duration of 40:00. When the user clicks the timestamp label in the central evidence card, the video player updates its current playback time to the corresponding point in time and plays classroom content from that point.

[0351] Question

[0352] explain

[0353] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

[0354] It should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the present invention.

[0355] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0356] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection, an electrical connection, or a connection that allows communication between them; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0357] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "over," and "on top" of the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

Claims

1. A teaching evaluation system based on evidence-driven large language models, characterized in that, include: A multi-channel transcription and error correction layer is used to realize the speech-to-text transcription function and to correct errors in the transcribed text. The rule-based multimodal analysis layer is used to output structured quantitative indicators, which refer to objective data calculated by the rule-based multimodal analysis layer for evaluating teaching quality. The quantitative anchor point generation layer is used to convert the structured quantitative indicators into Z-score standards based on historical statistical distribution parameters, generate natural language anchor point descriptions, and allocate them according to preset judgment items; the judgment item refers to the smallest evaluation unit used to judge the quality of teaching in a certain dimension. The LLM decision-based scoring layer is used to output the decision result based on the anchor description and the prompts for the decision items and the mandatory evidence output format. A deterministic scoring and post-processing layer is used to convert the determination result into a score.

2. The teaching evaluation system according to claim 1, characterized in that, The multichannel transcription and error correction layer includes: The speech-to-text module is used to convert spoken content in the classroom into text content with timestamps; The speaker separation module is used to distinguish different speaking roles from classroom audio and associate the identified role information with the transcribed text.

3. The teaching evaluation system according to claim 1, characterized in that, The multichannel transcription and error correction layer operates a four-layer error defense system after transcription to correct errors in the transcribed text: L1 layer: Domain dictionary correction: Establish regular expression rules to correct known systematic transcription errors; L2 layer: LLM semantic proofreading: The transcribed full text is embedded with a preset proofreading prompt word template and sent to the LLM. The LLM outputs a JSON array, and each element of the JSON array includes a location field, an original text field, a corrected text field, a correction reason field, and a confidence level field. L3 layer: Scoring Prompt fault-tolerance instruction: embeds transcription error fault-tolerance rules into the scoring rules; L4 Layer: Evidence Post-processing Filtering: Perform keyword matching on all fields in the LLM output that contain deduction evidence. If an entry in the preset speech-to-text error pattern library is matched, the evidence weight is reduced or removed according to the corresponding error type.

4. The teaching evaluation system according to claim 1, characterized in that, The regularized multimodal analysis layer includes: IRF dialogue chain analysis module: Based on the transcribed text with speaker tags, it identifies initiation, response and feedback in the classroom, classifies the feedback, and calculates the feedback quality score by weighting the feedback type; Bloom Question Classification Module: Sets the cognitive levels of teacher questions and student answers, and calculates the Shannon entropy normalized diversity score based on the cognitive levels of teacher questions and student answers; Classroom Stage Detection Module: Based on the density of preset keywords in timestamped text paragraphs, classroom stages are identified as different teaching segments formed during the teaching process of a lesson; The audio prosody analysis module includes a paragraph layer and a waveform layer. The paragraph layer extracts classroom expression features from the transcribed text, and the waveform layer extracts classroom expression features from the classroom audio. Teacher movement analysis module: Analyzes the teacher's spatial movement and physical activities based on continuous screen changes; Blackboard Writing-Transcription Consistency Analysis Module: Compares the relationship between visual content and classroom transcription text, and calculates a consistency score based on the degree of consistency between the blackboard writing and the topic being explained, the coverage of key content, and the timing of the appearance of the text during the explanation process; Cross-modal event detection module: Integrates text, audio, video, and motion information to identify teaching peaks, student engagement events, teaching transition points, and silent thinking time in the classroom. Specifically, a teaching peak refers to an event occurring simultaneously within a preset time window where a teacher asks a question, changes to the blackboard or PPT, and an increase in teacher movement intensity occur. An increase in teacher movement intensity event refers to an event where the cumulative displacement of the center point of the teacher's detection frame exceeds a preset movement threshold within a unit of time. A student engagement event refers to an event where a student speaks for at least 5 seconds while the teacher remains still. A teacher remaining still refers to a situation where the displacement of the center point of the teacher's detection frame is less than a preset displacement threshold within a continuous preset time period. A teaching transition point refers to a point in time where the stage boundary coincides with the blackboard / PPT content. A silent thinking time refers to a pause in the video for more than 3 seconds without any changes to the blackboard and the teacher remaining still.

5. The teaching evaluation system according to claim 1, characterized in that, The quantitative anchor point generation layer includes: The formula for generating anchor point descriptions is: Anchor text = indicator name + current value + statistical context + Z-score standard positioning; The allocation based on preset judgment criteria includes: Based on the evaluation objectives of different judgment items, relevant data indicators are assigned to the corresponding judgment items; each judgment item only receives process data, result data, or reference standards that are directly related to its judgment logic.

6. The teaching evaluation system according to claim 1, characterized in that, The LLM-based decision-based scoring layer includes: Scoring Dimensions: The scoring dimensions consist of multiple primary dimensions, each of which has several decision items. Each primary dimension and each decision item is assigned a corresponding weight. Gear Mapping Table: Different gear types are set for each judgment item, and each different gear type corresponds to a different set of levels. The set of levels includes the judgment item mapping score corresponding to each level, and the set of levels forms the gear mapping table.

7. The teaching evaluation system according to claim 1 or 6, characterized in that, The formulas for deterministic scoring and post-processing layer scoring are as follows: Scoring dimension score = ∑ (Mapping score of each judgment item × Weight of judgment item) Total score = ∑ (score of each scoring dimension × weight of the first-level dimension).

8. The teaching evaluation system according to claim 1, characterized in that, The mandatory evidence output format includes: when LLM outputs each judgment result, it must simultaneously provide the corresponding evidence description and timestamp. The evidence description must correspond to a specific fragment in the transcribed text, and the timestamp must also originate from the real mark in the transcribed text and cannot be generated by itself or left blank; different evidence corresponds to different time positions.

9. The teaching evaluation system according to claim 6, characterized in that, The LLM-based decision-based scoring layer also includes a real-time self-assessment mechanism for confidence: LLM outputs a confidence field for each of the scoring dimensions. The confidence field represents the LLM's self-assessment of the current judgment result, and the self-assessment of confidence is determined according to a preset calibration rule.

10. The teaching evaluation system according to claim 1, characterized in that, It also includes a gear boundary generation module, which uses Gaussian kernel density estimation to perform density estimation on historical rating data to obtain a density curve, and determines the gear boundary based on the local minimum points on the estimated density curve.

Images