Teaching content understanding and intelligent evaluation feedback method and system based on multi-modal large model

Abstract

The application relates to a teaching content understanding and intelligent evaluation feedback method and system based on a multimodal large model, which comprises the following steps: forming a multimodal teaching record by extracting speech text and visual text from a classroom teaching video and performing cross-modal time sequence alignment; separating teaching content and teaching behavior information by performing teaching content perception masking on the speech text; extracting knowledge point relationship triples from the masked text by using a self-feedback mechanism, and constructing a hierarchical teaching knowledge graph; simultaneously fusing semantic features, behavior features and position features to realize teaching behavior category identification; constructing a test question-knowledge point correlation matrix based on student answer data and a hierarchical multi-label classification model, calculating knowledge point mastery probability by using a cognitive diagnosis model, and generating a student diagnosis portrait; adaptively generating an individualized evaluation test question according to the diagnosis portrait; and finally, calling a teaching intelligent agent that has been fine-tuned by a tool to perform a preset teaching aid process, so that a multimodal feedback report is obtained.

Description

Technical Field

[0001] This application relates to the field of large language model technology, and in particular to a method and system for teaching content understanding and intelligent evaluation feedback based on a multimodal large model. Background Technology

[0002] With the rapid development of artificial intelligence technology in the field of education, using intelligent means to analyze the teaching process and evaluate learning outcomes has become an important direction for the digital transformation of education. How to automatically extract teaching knowledge points, identify teaching behaviors, and diagnose students' cognitive levels from multi-source heterogeneous data is a key issue that urgently needs to be addressed in current smart education research.

[0003] Currently, the mainstream teaching assessment method involves manually creating test questions and combining them with traditional cognitive diagnostic models to evaluate student learning. Teachers manually create test questions based on their teaching experience and label the knowledge points tested in each question. The test questions are then distributed to students to answer, and their answer data is collected. Finally, the answer data is input into a cognitive diagnostic model to calculate the students' mastery of each knowledge point.

[0004] The aforementioned technologies rely on manual operation for test question creation and knowledge point annotation, which is time-consuming, labor-intensive, and highly subjective, making it difficult to guarantee the balance of test question quality and the consistency of knowledge point annotation.

[0005] Based on this, this application provides a method and system for teaching content understanding and intelligent evaluation feedback based on a multimodal large model. Summary of the Invention

[0006] To address the issues of test item creation and knowledge point annotation relying on manual operations, which are time-consuming, labor-intensive, and highly subjective, making it difficult to ensure the balance of test item quality and the consistency of knowledge point annotation, this application provides a teaching content understanding and intelligent evaluation feedback method and system based on a multimodal large model.

[0007] Firstly, this application provides a teaching content understanding and intelligent evaluation feedback method based on a multimodal large model, which adopts the following technical solution: including: Collect teaching data, extract audio and visual text, and align them chronologically to form a structured multimodal teaching record; Extract the set of teaching content keywords and the set of original text fragments from the textbook text, and replace the successfully matched keywords and original text fragments in the audio text with preset mask symbols to obtain the masked teaching behavior text; The masked teaching behavior text is input into a multimodal large language model to generate a structured relation description, and the teaching knowledge point entities and their relation triples are extracted to construct a teaching knowledge graph. The speech text and the masked teaching behavior text are vectorized by a text embedding model to obtain semantic feature vectors and behavioral feature vectors. The semantic feature vectors, behavioral feature vectors and location feature vectors are concatenated and fused to identify the teaching behavior category. A response matrix is ​​constructed based on the test question data. Knowledge points are predicted for the test questions based on a hierarchical multi-label classification model, and a test question-knowledge point association matrix is ​​constructed. Based on the response matrix and the test question-knowledge point association matrix, the probability of students mastering each knowledge point is calculated, and a student diagnostic profile is generated. Based on the student diagnostic profile, target knowledge points and adaptive difficulty levels are determined, and relevant knowledge point information is retrieved from the teaching knowledge graph to construct a contextual knowledge base. Using the prompt word generation framework and the contextual knowledge base, structured prompt words are generated and input into the multimodal large language model to generate personalized assessment questions. Based on the personalized assessment questions, a teaching agent is constructed. The multimodal large language model is fine-tuned using tools. The teaching agent is executed according to the preset teaching aid process steps. Based on the teaching behavior category, the corresponding teaching intervention strategy is matched, and a multimodal feedback report is generated.

[0008] Preferably, the process of collecting teaching data, extracting audio and visual text, and performing temporal alignment to form a structured multimodal teaching record includes: Classroom teaching videos are acquired through audio and video acquisition equipment. These videos include footage of the teacher lecturing, blackboard writing, and courseware projection. The audio and video streams of the classroom teaching videos are separated to obtain independent teacher audio data and classroom video frame sequences. The teacher's audio data is transcribed using speech recognition technology. Each transcribed text segment is labeled with a start timestamp and an end timestamp to obtain a speech-text sequence with timestamp information. Keyframes are extracted from the classroom video frame sequence at preset frame intervals to obtain a keyframe image sequence; the multimodal large language model is used to perform visual content analysis on the keyframe image sequence to identify and extract the blackboard text content and the courseware text content. Based on the timestamps corresponding to the keyframe images, timestamp information is marked on the blackboard text content and the courseware text content to obtain blackboard visual text and courseware visual text with timestamp information. Based on a unified timeline, each speech-to-text segment, the blackboard visual text, and the courseware visual text in the speech-text sequence are aligned and merged according to their respective timestamp information to form the structured multimodal teaching record.

[0009] Preferably, the step of extracting the set of teaching content keywords and the set of original text fragments from the textbook text, and replacing the successfully matched keywords and original text fragments in the audio text with preset mask symbols to obtain the masked teaching behavior text includes: The textbook text is segmented into words, and the weight value of each word in the textbook text is calculated based on the inverse document frequency of the word. Words with weight values ​​higher than a preset weight threshold are filtered out to obtain the set of keywords for the teaching content. The textbook text is sliced ​​according to paragraph boundaries to form a set of original text fragments consisting of several teaching text segments; The audio text is matched against each teaching text segment in the set of original text segments in turn. If a match is successful, the matched part is marked as the original text of the classroom content. For the remaining part of the audio text that is not marked as the original text of the class content, perform keyword matching with the keyword set of the teaching content, and replace the successfully matched keywords with preset mask symbols; The portion marked as the original classroom content is concatenated with the remaining portion after masking and replacement in its original order to obtain the masked teaching behavior text.

[0010] Preferably, the step of inputting the masked teaching behavior text into a multimodal large language model to generate a structured relation description, and extracting teaching knowledge point entities and their relation triples to construct a teaching knowledge graph includes: A relation description generator is constructed. The relation description generator uses the multimodal large language model, combined with a preset first prompt template, target relation name and relation description generation example, to initialize and generate the masked teaching behavior text to obtain an initial structured relation description. Based on the preset second prompt template and feedback scoring example, the initial structured relation description is scored in multiple dimensions based on the multimodal large language model. The multidimensional scoring includes entity positioning accuracy score, semantic clarity score and information richness score, and feedback scoring results are generated. Based on the preset third prompt template and iterative optimization example, the initial structured relationship description and the feedback scoring result are input into the multimodal large language model for optimization to generate an optimized structured relationship description; The optimized structured relationship description replaces the initial structured relationship description, and the multi-dimensional scoring and optimization operations are repeated until the feedback scoring result meets the preset scoring threshold or reaches the preset maximum number of iterations, thus obtaining the structured relationship description. The teaching knowledge point entities are identified from the masked teaching behavior text. The teaching knowledge point entities are semantically matched with the structured relation description to determine the relation type between the teaching knowledge point entities. The relation triples are extracted and formed, and each relation triple contains a head entity, a relation type, and a tail entity. Based on the aforementioned relational triples, the teaching knowledge graph is constructed according to the hierarchical structure of courses, knowledge units, knowledge chapters, knowledge blocks, and knowledge points.

[0011] Preferably, the step of constructing a response matrix based on the assessment question data, predicting knowledge points for the questions based on a hierarchical multi-label classification model, and constructing a question-knowledge point association matrix; and calculating the student's mastery probability of each knowledge point based on the response matrix and the question-knowledge point association matrix, and generating a student diagnostic profile, including: Based on the assessment question data and its answer data, the answer data is encoded according to the student dimension and the question dimension to construct the answer response matrix. The rows of the matrix correspond to students, the columns of the matrix correspond to questions, and the matrix elements represent whether the student's answer to the question is correct or not. A hierarchical knowledge point tag system is obtained from the teaching knowledge graph, which includes a subject tag level, a first-level knowledge point tag level, and a second-level knowledge point tag level. Using the hierarchical multi-label classification model, the subject label, first-level knowledge point label and second-level knowledge point label of each test question are predicted in a multi-round dialogue manner. Each round of dialogue is guided by the label predicted in the previous round, and the candidate label set corresponding to the current level is combined to perform hierarchical classification to obtain the complete knowledge point label path of each test question. Construct the question-knowledge point association matrix based on the complete knowledge point tag paths of all questions. The rows of the matrix correspond to the questions, and the columns of the matrix correspond to the knowledge points. When the matrix element takes the first value, it means that the question tests the knowledge point, and when it takes the second value, it means that the question does not test the knowledge point. The response matrix and the question-knowledge point association matrix are input into a preset cognitive diagnostic model. The cognitive diagnostic model calculates the mastery probability of each student for each knowledge point based on whether the student's answers in the response matrix are correct or not and the association relationship between the questions and knowledge points in the question-knowledge point association matrix. The model generates a student diagnostic profile based on the mastery probability, which includes the mastery level of each knowledge point and the identification results of weak knowledge points.

[0012] Preferably, the step of determining target knowledge points and adaptive difficulty levels based on the student diagnostic profile, retrieving relevant knowledge point information from the teaching knowledge graph to construct a contextual knowledge base, generating structured prompts using the prompt word generation framework and the contextual knowledge base, and inputting them into the multimodal large language model to generate personalized assessment questions includes: Extract the mastery probability of each knowledge point from the student diagnostic profile, and identify the knowledge points whose mastery probability is lower than a preset probability threshold as target knowledge points; The appropriate difficulty level corresponding to the target knowledge point is determined based on the numerical range of the mastery probability, and the mastery probability is directly proportional to the adaptive difficulty level. The parent node knowledge point, child node knowledge point, and sibling node knowledge point that have a hierarchical relationship with the target knowledge point are retrieved from the teaching knowledge graph, and the definition description information and typical example information of the target knowledge point are extracted; the retrieved parent node knowledge point, child node knowledge point, sibling node knowledge point, definition description information, and typical example information are combined to form the context knowledge base; Using the aforementioned prompt word generation framework, the character setting information, the context knowledge base, the target knowledge points, the adaptive difficulty level, the number of questions, the question types, and the question format examples are combined according to a preset template structure to generate the structured prompt words. The structured prompt words include a task description field, a background knowledge field, a question attribute field, and a format example field. The structured prompts are input into the multimodal large language model to generate personalized test questions that include the question stem, correct option, distractor option, and detailed explanation.

[0013] Preferably, the step of constructing a teaching agent based on the personalized assessment questions, fine-tuning the multimodal large language model using tools, executing the teaching agent according to preset teaching aid process steps, matching corresponding teaching intervention strategies according to the teaching behavior category, and generating a multimodal feedback report, including: The teaching intelligent agent is constructed, which includes a planning component, a toolset, and a memory component; the toolset includes at least a knowledge point analysis tool, a test question generation tool, a cognitive diagnosis tool, a teaching behavior analysis tool, a knowledge graph query tool, a learning path planning tool, and a textbook explanation tool; The multimodal large language model generates a set of teaching scenario questions containing multiple tool call steps. Tool call sequence data corresponding to each teaching scenario question is collected. The tool call sequence data records the thinking content of the teaching agent in each step, the name and parameters of the tool called, and the observation results returned by the tool in a preset format. Based on the tool call sequence data, the multimodal large language model is fine-tuned to enable it to generate tool call sequences in a preset format; The teaching and learning process steps are pre-set in the system prompts of the teaching intelligence agent. The teaching and learning process steps include: understanding the learning status based on the student diagnostic profile, calling the cognitive diagnostic tool to analyze the student's weak knowledge points, calling the test question generation tool to generate the personalized assessment test questions, evaluating the student's answer and giving feedback, identifying the knowledge points that the student repeatedly makes mistakes on and calling the textbook explanation tool to provide knowledge tutoring, and calling the learning path planning tool to recommend subsequent learning paths. At the same time, the teaching behavior types are matched with a pre-set teaching intervention strategy library to determine the target teaching intervention strategy that is suitable for the current teaching scenario; The execution results of the teaching and learning process steps are summarized to generate the multimodal feedback report. The multimodal feedback report includes at least a knowledge point mastery probability distribution chart, weak knowledge point analysis text, personalized learning path suggestion text, and a targeted practice recommendation list.

[0014] Secondly, this application discloses a teaching content understanding and intelligent evaluation feedback device based on a multimodal large model, which adopts the following technical solution, including: The teaching data module is used to collect teaching data, extract audio and visual text, and perform temporal alignment to form a structured multimodal teaching record. The behavior matching module is used to extract a set of teaching content keywords and a set of original text fragments from the teaching material text, and replace the successfully matched keywords and original text fragments in the audio text with preset mask symbols to obtain the masked teaching behavior text; The knowledge graph module is used to input the masked teaching behavior text into a multimodal large language model, generate a structured relation description, extract teaching knowledge point entities and their relation triples, and construct a teaching knowledge graph. The behavior classification module is used to vectorize the speech text and the masked teaching behavior text respectively through a text embedding model to obtain semantic feature vectors and behavior feature vectors. The semantic feature vectors, behavior feature vectors and location feature vectors are concatenated and fused to identify the teaching behavior category. The student profiling module is used to construct a response matrix based on the assessment question data, predict knowledge points for the questions based on a hierarchical multi-label classification model, and construct a question-knowledge point association matrix; based on the response matrix and the question-knowledge point association matrix, the module calculates the probability of the student's mastery of each knowledge point and generates a student diagnostic profile. The assessment and feedback module is used to determine the target knowledge points and adaptive difficulty level based on the student diagnostic profile, retrieve relevant knowledge point information from the teaching knowledge graph to construct a contextual knowledge base, generate structured prompt words using the prompt word generation framework and the contextual knowledge base, and input them into the multimodal large language model to generate personalized assessment questions. The teaching execution module is used to construct a teaching agent based on the personalized assessment questions, fine-tune the multimodal large language model using tools, execute the teaching agent according to the preset teaching aid process steps, match corresponding teaching intervention strategies according to the teaching behavior categories, and generate multimodal feedback reports.

[0015] Thirdly, this application also provides a control device, the device comprising: It includes a memory and a processor, wherein the memory stores a computer program that can be loaded and executed by the processor, such as the above-described method for understanding teaching content and intelligent evaluation feedback based on a multimodal large model.

[0016] Fourthly, this application also provides a computer-readable storage medium storing a computer program that can be loaded by a processor and executed as described above regarding the teaching content understanding and intelligent evaluation feedback method based on a multimodal large model.

[0017] In summary, this application collects classroom teaching videos, teacher audio, textbook text, and student answer data. Through speech-to-text transcription and visual content extraction, and cross-modal temporal alignment, a structured multimodal teaching record is formed. The audio text undergoes perceptual masking of teaching content to obtain masked teaching behavior text. This masked text is then input into a multimodal large language model, which generates structured relation descriptions through a self-feedback mechanism. Knowledge point entities and their relation triples are extracted to construct a hierarchical teaching knowledge graph. Simultaneously, an embedding model fine-tuned using educational professional corpus is used to vectorize both the audio text and the masked text, fusing semantic, behavioral, and positional features to identify teaching behavior categories. Based on this, a response matrix is ​​constructed using student answer data. This study utilizes a hierarchical multi-label classification model to construct a question-knowledge point association matrix. This matrix is ​​then input into a cognitive diagnostic model to calculate the probability of students mastering each knowledge point, generating a student diagnostic profile. Based on this profile, target knowledge points and adaptive difficulty levels are determined. A contextual knowledge base is constructed using knowledge graph retrieval information, and structured prompts are generated through a prompt word generation framework. These prompts are then input into a multimodal large language model to generate personalized assessment questions. Finally, a teaching intelligence agent is constructed. After fine-tuning the multimodal large language model, it sequentially executes the steps of learning diagnosis, question generation, answer evaluation, and feedback recommendation according to a pre-set teaching aid process. Teaching intervention strategies are matched based on teaching behavior categories, generating a feedback report that includes a knowledge point mastery distribution chart, weakness analysis, and learning suggestions. This achieves automated generation and analysis of teaching plans and questions based on a large language model, thereby improving the intelligence and personalization of teaching assessment and facilitating the popularization of intelligent education. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating a teaching content comprehension and intelligent assessment feedback method based on a multimodal large model.

[0019] Figure 2 This is a structural block diagram of a teaching content comprehension and intelligent evaluation feedback device based on a multimodal large model. Detailed Implementation

[0020] The following combination Figures 1-2 This application will be described in further detail.

[0021] Reference Figure 1 The embodiments of this application include at least steps S10 to S70.

[0022] S10: Collect teaching data, extract audio and visual text, and perform temporal alignment to form a structured multimodal teaching record; S20: Extract the set of teaching content keywords and the set of original text fragments from the textbook text, replace the successfully matched keywords and original text fragments in the audio text with preset mask symbols, and obtain the masked teaching behavior text; S30: Input the masked teaching behavior text into the multimodal large language model to generate a structured relation description, extract the teaching knowledge point entities and their relation triples, and construct a teaching knowledge graph; S40 uses a text embedding model to vectorize the speech text and the masked teaching behavior text to obtain semantic feature vectors and behavioral feature vectors. The semantic feature vectors, behavioral feature vectors and location feature vectors are concatenated and fused to identify the teaching behavior category. S50: Construct a response matrix based on the test question data; predict knowledge points for the test questions based on a hierarchical multi-label classification model; construct a test question-knowledge point association matrix; calculate the probability of students mastering each knowledge point based on the response matrix and the test question-knowledge point association matrix; and generate a student diagnostic profile. S60 determines the target knowledge points and adaptive difficulty level based on the student's diagnostic profile, retrieves relevant knowledge point information from the teaching knowledge graph to build a contextual knowledge base, generates structured prompt words using the prompt word generation framework and contextual knowledge base, and inputs them into a multimodal large language model to generate personalized assessment questions; S70 constructs an intelligent teaching agent based on personalized assessment questions, fine-tunes the multimodal large language model using tools, executes the intelligent teaching agent according to preset teaching aid process steps, matches corresponding teaching intervention strategies based on teaching behavior categories, and generates multimodal feedback reports.

[0023] Specifically, the system synchronously extracts audio text and visual text such as blackboard notes and courseware from classroom teaching videos, and aligns them across modal time sequences to form a unified multimodal teaching record. Based on this, by perceptually masking the teaching content in the audio text, the system effectively separates the two information dimensions of teaching content and teaching behavior. On one hand, it utilizes the self-feedback mechanism of a multimodal large language model to automatically extract knowledge point entities and their relational triples from the masked text, constructing a hierarchical teaching knowledge graph. On the other hand, it integrates semantic features, behavioral features, and location features to automatically identify teaching behavior categories. Next, by combining student answer data with hierarchical multi-label classification models to predict test question knowledge point labels, a test question-knowledge point association matrix is ​​constructed. A cognitive diagnostic model is then used to calculate the probability of students mastering each knowledge point, forming a personalized student diagnostic profile. Finally, driven by diagnostic profiling, a contextual knowledge base is constructed by retrieving related knowledge points from the knowledge graph. A prompt word generation framework guides a multimodal large language model to generate personalized assessment questions with adaptive difficulty. The tool then invokes a finely tuned teaching agent to perform learning diagnosis, test question delivery, answer evaluation, and learning intervention according to a pre-set teaching aid process. The system outputs a multimodal feedback report containing knowledge point mastery analysis, weak point diagnosis, and personalized learning suggestions. This effectively solves the problems of single modality and heavy reliance on manual intervention in mainstream intelligent teaching assessments.

[0024] S10, Multimodal Teaching Data Acquisition and Preprocessing In classroom teaching scenarios, audio and video capture devices deployed in the classroom are used to acquire classroom teaching videos. The content of classroom teaching videos typically includes three categories: the teacher lecturing, the blackboard writing, and the projected slides.

[0025] After acquiring the classroom teaching video, the audio and video streams are separated to obtain independent teacher audio data and a sequence of classroom video frames. The teacher audio data is then transcribed frame-by-frame using automatic speech recognition technology. Automatic speech recognition is an artificial intelligence technology that converts speech signals into text. It uses acoustic and language models to decode the audio signal and output the corresponding text content and time information. During the transcription process, the speech recognition engine segments the audio into several transcribed text segments based on sentences or semantic units, and marks each transcribed text segment with a start and end timestamp, ultimately resulting in a speech-text sequence with timestamp information.

[0026] For the classroom video frame sequence, keyframes are extracted at preset frame intervals to obtain a keyframe image sequence. A multimodal large language model is then used to analyze the visual content of the keyframe image sequence. The multimodal large language model is a large-scale pre-trained language model capable of simultaneously processing multiple modalities such as text, images, and speech. Through pre-training on billions of image-text pairing datasets, it possesses the ability to understand image content and describe image information in text form. During the analysis process, the multimodal large language model distinguishes and identifies handwritten and electronic text regions in the image, extracting the blackboard text and courseware text content.

[0027] Since each keyframe image originates from a specific video time point, the extracted blackboard text and courseware text are labeled with the timestamp information of their source images, resulting in blackboard visual text and courseware visual text with timestamp information. Finally, cross-modal temporal alignment is performed based on a unified timeline: using the timestamp of each speech-to-text segment as a reference, visual text records whose timestamps fall within the same time period are found and merged into a structured multimodal teaching record. Each multimodal teaching record contains speech text fields, blackboard visual text fields, and courseware visual text fields within the same time period.

[0028] Specifically, the system precisely aligns the three previously independent information flows—voice narration, blackboard writing, and presentation slides—in the time dimension, enabling the system to understand the teacher's teaching content at a given moment. When a teacher explains a knowledge point, their blackboard writing and presentation slides often contain the core concepts and key expressions of that knowledge point. These three elements corroborate and complement each other, providing a unified data foundation for subsequent knowledge point relationship extraction and teaching behavior recognition. Compared to traditional single-modal analysis methods that rely solely on voice or presentation slides, multimodal aligned teaching records can capture teaching information that a single modality cannot cover, such as the referential relationships between the teacher's verbal explanation and presentation slides, and the sequential logic between the blackboard derivation process and the voice narration.

[0029] S20, Perceptual Masking Processing of Teaching Content After obtaining structured multimodal teaching records, the audio text in the multimodal teaching records is subjected to teaching content perception masking processing.

[0030] The textbook text is segmented into words: the importance of each word is quantitatively assessed using inverse document frequency (IVF) technology. IVF is a classic text keyword extraction method: the higher the frequency of a word in a document, and the fewer documents containing that word in the entire document set, the more representative that word is of that document. A weight value is obtained by multiplying the word frequency value and the IVF value of each word. Words with weight values ​​higher than a preset weight threshold are selected to form a set of teaching content keywords. The words in this set are typically subject-specific terms and core conceptual nouns.

[0031] Secondly, the textbook text is sliced ​​according to paragraph boundaries, with each natural paragraph as a unit to form several teaching text segments, constituting a set of original text segments. The audio text undergoes dual matching and masking processing. The first layer is original text matching: the audio text is sequentially matched against each teaching text segment in the original text segment set. If a text segment in the audio text continuously matches a teaching text segment and exceeds a preset length threshold, the match is considered successful, and that matched portion is marked as original classroom content. The second layer is keyword matching: for the remaining portion of the audio text not marked as original classroom content, each segment is scanned to see if it contains keywords from the teaching content keyword set. If it does, the keyword is replaced with a preset mask symbol. Finally, the portion marked as original classroom content and the remaining portion after masking are concatenated in their original order to obtain the masked teaching behavior text.

[0032] Specifically, classroom teaching audio carries two distinct types of information: teaching content information and teaching behavior information. Teaching content information is primarily reflected in subject-specific terminology, conceptual expressions, and the original text, while teaching behavior information is mainly manifested in linguistic structural features such as question patterns, instructional tone, and transitional words. In traditional processing methods, these two types of information are mixed together for vector representation. Because vocabulary related to teaching content often carries a stronger semantic weight, teaching behavior features are masked. This causes subsequent teaching behavior clustering or classification tasks to essentially degenerate into teaching content clustering, failing to effectively distinguish based on teaching behavior patterns. Through the masking processing in this embodiment, the semantic influence of teaching content is effectively suppressed, allowing the vector representation output by the text embedding model to reflect teaching behavior features more than teaching content features, thereby significantly improving the accuracy of teaching behavior recognition and knowledge point relationship extraction.

[0033] S30, Construction of Instructional Knowledge Graphs Based on Self-Feedback Mechanisms After obtaining the masked teaching behavior text, a teaching knowledge graph is automatically constructed from the masked teaching behavior text using the self-feedback mechanism of a multimodal large language model. The self-feedback mechanism is a method that iteratively improves the quality of generated content through model self-evaluation and self-correction. It simulates the human thought process of repeatedly examining, reflecting on, and improving when solving problems, and consists of a closed loop of three stages: initial generation, feedback scoring, and iterative optimization.

[0034] During the initialization generation phase, the relation description generator inputs a pre-set first prompt template, the target relation name, and relation description generation examples into the multimodal large language model. The first prompt template guides the model to generate structured relation description text. The target relation name is the name of the relation type to be described. The relation description generation examples are a small number of manually written demonstration cases to help the model understand the output format. Based on this, the multimodal large language model generates an initial structured relation description, which is a templated text with special markers indicating the head and tail entity types.

[0035] In the feedback scoring phase, using a pre-set second prompt template and feedback scoring examples, a multimodal large language model scores the initial structured relation description from multiple dimensions, including entity localization accuracy, semantic clarity, and information richness. The model outputs the score values ​​for each dimension and a brief reasoning for the score, constituting the feedback scoring result.

[0036] In the iterative optimization phase, using a pre-defined third-prompt template and iterative optimization examples, the initial structured relation description and feedback scoring results are input into the multimodal large language model for optimization. The model modifies the shortcomings pointed out in the feedback scoring, generating an optimized structured relation description. The optimized structured relation description replaces the initial structured relation description, and the feedback scoring and iterative optimization are repeated until the feedback scoring results meet a pre-defined scoring threshold or the pre-defined maximum number of iterations is reached. When the iteration terminates, the final optimized structured relation description is used as the structured relation description.

[0037] Specifically, traditional single-step generation methods struggle to guarantee the quality of relation descriptions, potentially resulting in issues such as ambiguous entity localization, unclear semantic representation, or missing information. The self-feedback mechanism, by introducing a cyclical iteration of generation-evaluation-improvement, allows the model to repeatedly examine and correct its output, gradually improving the accuracy of relation descriptions. The three dimensions of the feedback score correspond to different aspects of relation description quality: entity localization accuracy ensures clear type constraints for the head and tail entities; semantic clarity ensures unambiguous relation representation; and information richness ensures the description covers sufficient entity type information rather than being overly simplistic. Relation descriptions optimized through multiple iterations can more accurately depict the semantic relationships between knowledge points.

[0038] After obtaining the structured relation descriptions, teaching knowledge point entities are identified from the masked teaching behavior text. Semantic matching of these entities with the structured relation descriptions determines the relationship types between entities, and relation triples are extracted. Each relation triple contains a head entity, a relation type, and a tail entity. Based on all extracted relation triples, a teaching knowledge graph is constructed according to a hierarchical structure of courses, knowledge units, knowledge chapters, knowledge blocks, and knowledge points. A teaching knowledge graph is a way to organize knowledge using a graph data structure, where nodes represent knowledge point entities and edges represent semantic relationships between knowledge points. The hierarchical structure reflects the organizational logic of knowledge from macro to micro: the course layer provides the most macro-level subject framework, the knowledge point layer provides the most micro-level teaching units, and each intermediate layer is progressively refined. This allows the knowledge graph to not only present the overall picture of the curriculum system at a macro level but also accurately depict the semantic relationships such as dependencies and parallel associations between knowledge points at a micro level, providing structured knowledge support for subsequent test generation and learning path planning.

[0039] S40, Multi-feature Fusion Classification and Recognition of Teaching Behaviors This study utilizes a text embedding model fine-tuned from an educational corpus to vectorize spoken text. A text embedding model is a deep learning model that converts natural language text into fixed-dimensional numerical vectors. It learns semantic representations of words and sentences through pre-training on a large-scale corpus, ensuring that semantically similar texts are close together in the vector space. The educational corpus used for fine-tuning is derived from professional books on curriculum theory, pedagogy, and educational psychology.

[0040] The fine-tuning method involves further pre-training of the masked language model: Building upon the general pre-trained model, additional training is performed using educational corpora. The training task involves randomly masking parts of the text and having the model predict the masked words. This task allows the model to gradually adapt to specialized vocabulary and expressions in the educational field, enhancing its ability to represent language patterns in teaching behaviors. The purpose of this step is that while the general pre-trained model possesses basic semantic understanding capabilities, its training corpus primarily consists of general texts such as news and encyclopedic entries, lacking sensitivity to the specific behavioral expression patterns in teaching scenarios. After fine-tuning with educational corpora, the model can better perceive and distinguish the subtle differences in language expression among different teaching behaviors.

[0041] The original speech text and the masked teaching behavior text are input into a fine-tuned text embedding model. After processing by a multi-layer Transformer encoder and pooling operations, teaching semantic feature vectors and teaching behavior feature vectors are obtained. The Transformer encoder is a neural network structure based on a self-attention mechanism, which can effectively capture long-distance dependencies between words in the text. At the same time, the relative temporal position information of the speech text in the entire classroom teaching is extracted and encoded into positional feature vectors.

[0042] The teaching semantic feature vector, teaching behavior feature vector, and location feature vector are concatenated and fused to form a multi-feature fused teaching behavior vector. Each of the three features captures different dimensions of information about the teaching behavior: the semantic feature vector reflects the semantic content of the text, the behavior feature vector reflects the language pattern information of the teaching behavior after content masking, and the location feature vector reflects the distribution pattern of the teaching behavior in the classroom sequence. The fusion of these three features creates a complementary structure, making the vector representation more comprehensive and robust than using only a single semantic feature. The multi-feature fused teaching behavior vector is then input into a pre-trained teaching behavior classification model for classification. This model employs a multi-scale convolutional neural network structure, which extracts local features of different granularities from the text in parallel using multiple convolutional kernels of different sizes. Smaller kernels capture short-range word collocation features, while larger kernels capture long-range sentence structure features. The classifier outputs the probability distribution for each category, and the category with the highest probability is selected as the teaching behavior category.

[0043] The teaching behaviors include explaining the text content, outlining the article's structure, interpreting emotions, analyzing character portrayals, asking questions and interacting, assigning tasks, and providing learning guidance.

[0044] S50, Learner Cognitive State Analysis Based on Cognitive Diagnosis The assessment question data and its corresponding answers are acquired. The answer data is then encoded according to student and question dimensions to construct an answer response matrix. Rows in the matrix correspond to students, columns to questions, and element values ​​indicate whether the answer is correct or not. The purpose of the answer response matrix is ​​to organize the scattered answer records into a structured data format, enabling subsequent cognitive diagnostic models to efficiently process all students' answer information using matrix operations.

[0045] Secondly, a hierarchical knowledge point label system is obtained from the teaching knowledge graph, which includes subject label levels, first-level knowledge point label levels, and second-level knowledge point label levels. A hierarchical multi-label classification model is used to predict knowledge point labels for each test question. The hierarchical multi-label classification model is a classification model capable of simultaneously predicting multiple hierarchical labels for text. It employs a multi-turn dialogue classification method based on a generative pre-trained language model: transforming the classification task into a multi-turn dialogue interaction process, with each turn predicting a label at one level, and using the prediction result of the previous turn as the conditional input for the next turn, achieving a top-down, progressive classification. Specifically, the first turn predicts the subject label of the test question, the second turn predicts the first-level knowledge point label using the subject label as a condition, the third turn predicts the second-level knowledge point label using the first-level knowledge point label as a condition, ultimately forming a complete knowledge point label path.

[0046] Specifically, traditional one-time planar classification is prone to classification confusion and decreased accuracy when faced with large-scale hierarchical labeling systems due to the sheer number and semantic similarity of the labels. By breaking down the classification task into multiple levels and executing them sequentially, each level only needs to select from the candidate label set of its current level, effectively reducing the classification space. Furthermore, there is a natural constraint relationship between labels at different levels (e.g., once the first-level knowledge point label is determined, the candidate range of second-level knowledge point labels is limited to that first-level knowledge point), thereby improving the accuracy and consistency of label prediction.

[0047] A question-knowledge point association matrix is ​​constructed based on the complete knowledge point tag paths of all test questions. Rows in the matrix correspond to test questions, columns to knowledge points, and element values ​​indicate whether the test question tests that knowledge point. The purpose of the question-knowledge point association matrix is ​​to formally express the testing relationship between test questions and knowledge points. It defines which knowledge points each test question measures and is one of the core inputs for the cognitive diagnostic model to perform parameter estimation and mastery probability calculation.

[0048] The response matrix and the question-knowledge point association matrix are input into a pre-defined cognitive diagnostic model. The cognitive diagnostic model is a psychometric model used to assess learners' knowledge mastery. It transforms students' external answering behavior into internal knowledge point mastery probabilities by probabilistically modeling their answering behavior. Traditional test scoring methods only provide a general total score, failing to reveal students' specific mastery of individual knowledge points: two students with the same total score may have completely different mastery patterns on different knowledge points. The cognitive diagnostic model, through the association between questions and knowledge points, infers the mastery status of each knowledge point from the student's answering patterns, achieving refined differentiation among students with the same total score and providing accurate diagnostic basis for differentiated instruction. The cognitive diagnostic model can also output guessing parameters and error parameters, representing the probability of a student guessing correctly despite not mastering the knowledge point and the probability of answering incorrectly due to carelessness despite mastering the knowledge point, respectively. Based on the mastery probabilities, a student diagnostic profile is generated, including the mastery level of each knowledge point and the identification results of weak knowledge points.

[0049] S60, Knowledge-Enhanced Adaptive Personalized Question Generation The mastery probability of each knowledge point is extracted from the student's diagnostic profile, and knowledge points with mastery probabilities below a preset probability threshold are identified as target knowledge points. An adaptive difficulty level is determined based on the numerical range of the mastery probability, with a positive correlation between mastery probability and adaptive difficulty level. The purpose of adaptive difficulty adjustment is to match the difficulty of the generated test questions with the students' current ability level, avoiding overly difficult questions that lead to frustration or overly easy questions that result in ineffective practice, thus enabling students to learn effectively within their zone of proximal development.

[0050] Using target knowledge points as query criteria, the system retrieves parent, child, and sibling knowledge points from the teaching knowledge graph, extracts their definitions and typical examples, and combines them to form a contextual knowledge base. This contextual knowledge base provides comprehensive and accurate background knowledge support for test question generation. When a large language model is allowed to generate test questions freely, the model may produce inaccurate or incomplete questions due to ambiguous knowledge boundaries. By retrieving accurate definitions and related knowledge point information from the knowledge graph and explicitly providing them to the model, the correctness and relevance of the generated content are effectively constrained, ensuring that the generated test questions closely adhere to the specified knowledge points and conform to subject teaching standards.

[0051] Structured prompts are generated using a prompt generation framework. This framework is a method for standardizing and assembling various input information according to a pre-defined template structure. It organizes role setting information, background knowledge, task parameters, and format examples into structured text using fixed fields, enabling precise guidance for the behavior of a multimodal large language model. The specific assembly process involves filling the corresponding fields of the pre-defined template structure with role setting information, contextual knowledge base, target knowledge points, adaptive difficulty level, number of questions, question type, and question format examples. The generated structured prompts include task description fields, background knowledge fields, question attribute fields, and format example fields. The purpose of the prompt generation framework is to organize scattered input information into a standardized structure, ensuring consistent information format and complete elements each time questions are generated, thus avoiding fluctuations in generation results caused by non-standard prompt design.

[0052] Finally, the structured prompts are input into a multimodal large language model to generate personalized assessment questions. Each personalized assessment question consists of four parts: the stem, the correct answer, distractors, and a detailed explanation. The detailed explanation not only provides the correct answer but, more importantly, helps students understand the reasoning behind the correct answer and the reasons why the distractors are incorrect. This allows students to receive timely and effective learning feedback during practice, improving the efficiency of their self-directed learning.

[0053] S70, Intelligent Agent Evaluation Feedback After generating personalized assessment questions, a teaching intelligent agent system is constructed to achieve proactive, personalized, and interactive assessment feedback. A teaching intelligent agent is an artificial intelligence system capable of autonomously perceiving its environment, planning tasks, executing operations, and interacting with humans. Its core advantage lies in its ability to integrate multiple single-function tools and models into an organically collaborative whole, accomplishing complex teaching tasks that a single model cannot achieve.

[0054] The construction of an intelligent teaching agent comprises three basic components: a planning component, a toolset, and a memory component. The planning component is responsible for developing a task execution plan based on user needs and the current state; the toolset provides the agent with execution capabilities; and the memory component stores dialogue history and user information, enabling the agent to possess contextual memory and personalized response capabilities. The toolset includes at least seven specialized teaching tools: knowledge point analysis tools, test question generation tools, cognitive diagnosis tools, teaching behavior analysis tools, knowledge graph query tools, learning path planning tools, and textbook explanation tools.

[0055] Secondly, the multimodal large language model undergoes tool invocation fine-tuning. During the data construction phase, a set of teaching scenario questions is automatically generated using the multimodal large language model, and tool invocation sequence data corresponding to each question is collected. The tool invocation sequence data is recorded in a reasoning-action-observation format. This format is a data organization method used to train the tool invocation capability of a large language model. It decomposes each step of the agent's operation into three components: the reasoning part records the agent's analysis of the current state and the decision-making process for the next operation; the action part records the name of the tool called by the agent, the input parameters, and the result returned by the tool; and the observation part records the agent's interpretation of the returned result. During the model training phase, the multimodal large language model is fine-tuned based on the tool invocation sequence data, enabling the model to learn the patterns and strategies of tool invocation and to autonomously generate tool invocation sequences according to the reasoning-action-observation format. The purpose of tool invocation fine-tuning is that untuned general-purpose large language models typically lack the awareness and ability to invoke tools. When faced with complex teaching tasks, they often directly generate answers, skipping necessary tool lookup steps, resulting in answers lacking accurate knowledge support. After fine-tuning, the model learns to call the appropriate tools at the right time to obtain accurate information, and then integrates the information to form a final response, thus improving the quality of task completion in complex teaching scenarios.

[0056] The teaching support process steps are pre-defined in the system prompts of the teaching agent. System prompts are pre-set texts used to define the agent's role, behavioral rules, and task flow. The teaching support process steps include: understanding the student's learning status based on the student's diagnostic profile; analyzing weaknesses using cognitive diagnostic tools; generating personalized assessment questions using test question generation tools; evaluating answers and providing guiding feedback; identifying repeatedly missed knowledge points and providing supplementary instruction using textbook explanation tools; and recommending subsequent learning paths using learning path planning tools. The purpose of pre-setting the teaching support process is to enable the teaching agent to proactively advance the teaching process according to the teaching logic of diagnosis-practice-assessment-remediation-planning, rather than passively waiting for students to ask questions, thus realizing a transformation from a passive question-and-answer tool to a proactive teaching assistant.

[0057] Simultaneously, the teaching behavior categories are matched with a pre-set teaching intervention strategy library. This library is a dataset storing the mapping relationships between teaching behavior categories and teaching intervention strategies, used to automatically select appropriate intervention methods based on the identified teaching behavior type. For example, when the current teaching behavior is identified as questioning interaction, a heuristic probing strategy is matched instead of directly providing an answer; when it is identified as learning method guidance, strategies of summarizing learning methods and providing skill tips are matched. The purpose of this step is to ensure that the feedback style of the teaching agent is consistent with the teaching behavior patterns of excellent teachers, providing a more natural and effective teaching interaction experience.

[0058] The entire teaching and learning process is summarized to generate a multimodal feedback report. This report is a comprehensive assessment and feedback document for both students and teachers, containing at least four components: a graph showing the probability distribution of knowledge point mastery, text analyzing weak knowledge points, personalized learning path suggestions, and a list of recommended targeted exercises.

[0059] The implementation principle of a teaching content understanding and intelligent evaluation feedback method based on a multimodal large model in this application is as follows: Classroom teaching videos, teacher voice, textbook text, and student answer data are collected. Through speech-to-text transcription and visual content extraction, and cross-modal temporal alignment, a structured multimodal teaching record is formed. The voice text is then subjected to teaching content perception masking to obtain masked teaching behavior text. The masked text is then input into a multimodal large language model, which generates structured relation descriptions through a self-feedback mechanism, extracting knowledge point entities and their relation triples to construct a hierarchical teaching knowledge graph. Simultaneously, an embedding model fine-tuned using educational professional corpus is used to vectorize the voice text and the masked text respectively, fusing semantic features, behavioral features, and positional features to identify teaching behavior categories. Based on this, the... A response matrix is ​​constructed based on student answer data. A hierarchical multi-label classification model is used to build a question-knowledge point association matrix. This matrix is ​​then input into a cognitive diagnostic model to calculate the probability of students mastering each knowledge point, generating a student diagnostic profile. Based on this profile, target knowledge points and adaptive difficulty levels are determined. A contextual knowledge base is built using knowledge graph retrieval information. Structured prompts are generated through a prompt word generation framework and input into a multimodal large language model to generate personalized assessment questions. Finally, a teaching intelligence agent is constructed. After fine-tuning the multimodal large language model, it executes the steps of learning diagnosis, question generation, answer evaluation, and feedback recommendation according to a pre-set teaching aid process. Teaching intervention strategies are matched based on teaching behavior categories, generating a feedback report that includes a knowledge point mastery distribution chart, weakness analysis, and learning suggestions. This achieves automated generation and analysis of teaching plans and questions based on a large language model, thereby improving the intelligence and personalization of teaching assessment and facilitating the popularization of intelligent education.

[0060] Figure 1This is a flowchart illustrating a teaching content understanding and intelligent assessment feedback method based on a multimodal large model in one embodiment. It should be understood that, although... Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows; unless explicitly stated otherwise, there is no strict order requirement for the execution of these steps, and they can be executed in other orders; and Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0061] Based on the same technical concept, referring to Figure 2 This application also provides a teaching content understanding and intelligent evaluation feedback device based on a multimodal large model, which adopts the following technical solution: The device includes: The teaching data module is used to collect teaching data, extract audio and visual text, and perform temporal alignment to form a structured multimodal teaching record. The behavior matching module is used to extract the set of teaching content keywords and the set of original text fragments from the teaching materials, and replace the successfully matched keywords and original text fragments in the audio text with preset mask symbols to obtain the masked teaching behavior text; The knowledge graph module is used to input the masked teaching behavior text into a multimodal large language model, generate structured relation descriptions, extract teaching knowledge point entities and their relation triples, and construct a teaching knowledge graph. The behavior classification module is used to vectorize the speech text and the masked teaching behavior text respectively through the text embedding model to obtain semantic feature vectors and behavior feature vectors. The semantic feature vectors, behavior feature vectors and location feature vectors are concatenated and fused to identify the teaching behavior category. The student profiling module is used to construct a response matrix based on the assessment question data, predict the knowledge points of the questions based on a hierarchical multi-label classification model, and construct a question-knowledge point association matrix. Based on the response matrix and the question-knowledge point association matrix, the module calculates the probability of students mastering each knowledge point and generates a student diagnostic profile. The assessment and feedback module is used to determine the target knowledge points and adaptive difficulty level based on the student's diagnostic profile, and to retrieve relevant knowledge point information from the teaching knowledge graph to build a contextual knowledge base. It uses prompt words to generate a framework and contextual knowledge base to generate structured prompt words, and inputs them into a multimodal large language model to generate personalized assessment questions. The teaching execution module is used to construct a teaching agent based on personalized assessment questions, fine-tune the multimodal large language model by calling tools, execute the teaching agent according to the preset teaching aid process steps, match corresponding teaching intervention strategies according to the teaching behavior category, and generate multimodal feedback reports.

[0062] In some embodiments, the teaching data module is specifically used to acquire classroom teaching videos through audio and video acquisition devices. The classroom teaching videos include teacher lecturing, blackboard writing, and courseware projection. The audio and video streams of the classroom teaching videos are separated to obtain independent teacher audio data and classroom video frame sequences. The teacher's audio data is transcribed using speech recognition technology. Each transcribed text segment is labeled with a start timestamp and an end timestamp to obtain a speech-text sequence with timestamp information. Keyframes are extracted from the classroom video frame sequence at preset frame intervals to obtain a keyframe image sequence; a multimodal large language model is used to perform visual content analysis on the keyframe image sequence to identify and extract the blackboard text content and the courseware text content. Based on the timestamps corresponding to the keyframe images, timestamp information is added to the blackboard text and courseware text to obtain the blackboard visual text and courseware visual text with timestamp information. Based on a unified timeline, each speech-to-text segment, blackboard visual text, and courseware visual text in the speech-text sequence are aligned and merged according to their respective timestamp information to form a structured multimodal teaching record.

[0063] In some embodiments, the behavior matching module is specifically used to perform word segmentation on the teaching material text, calculate the weight value of each word in the teaching material text based on the inverse document frequency of the word frequency, filter words with weight values ​​higher than a preset weight threshold, and obtain a set of teaching content keywords; The textbook text is sliced ​​according to paragraph boundaries to form a collection of original text fragments consisting of several teaching text segments; The audio text is matched against each teaching text segment in the original text segment set in turn. If a match is successful, the matched part is marked as the original text of the class content. For the remaining parts of the audio text that are not marked as original classroom content, perform keyword matching with the set of keywords of teaching content, and replace the successfully matched keywords with preset mask symbols; The portion marked as the original classroom content is concatenated with the remaining portion after masking and replacement in its original order to obtain the masked teaching behavior text.

[0064] In some embodiments, the knowledge graph module is specifically used to construct a relation description generator. The relation description generator uses a multimodal large language model, combined with a preset first prompt template, target relation name and relation description generation example, to initialize and generate the masked teaching behavior text to obtain an initial structured relation description. Based on the preset second prompt template and feedback scoring example, the initial structured relation description is scored in multiple dimensions based on the multimodal large language model. The multi-dimensional scoring includes entity location accuracy score, semantic clarity score and information richness score, and feedback scoring results are generated. Based on the preset third prompt template and iterative optimization example, the initial structured relation description and feedback scoring results are input into the multimodal large language model for optimization, generating an optimized structured relation description; The optimized structured relationship description is replaced with the initial structured relationship description. Multi-dimensional scoring and optimization operations are repeated until the feedback scoring result meets the preset scoring threshold or reaches the preset maximum number of iterations, thus obtaining the structured relationship description. Identify teaching knowledge point entities from the masked teaching behavior text, perform semantic matching between teaching knowledge point entities and structured relation descriptions, determine the relationship type between teaching knowledge point entities, extract and form relation triples, each relation triple containing a head entity, relation type and tail entity; Based on relation triples, a teaching knowledge graph is constructed according to the hierarchical structure of courses, knowledge units, knowledge chapters, knowledge blocks, and knowledge points.

[0065] In some embodiments, the student profiling module is specifically used to encode the answer data according to the student dimension and the question dimension based on the assessment question data and its answer data, and construct an answer response matrix. The rows of the matrix correspond to students, the columns of the matrix correspond to questions, and the matrix elements represent whether the student's answer to the question is correct or not. A hierarchical knowledge point tagging system is obtained from the teaching knowledge graph. The hierarchical knowledge point tagging system includes subject tag level, first-level knowledge point tag level and second-level knowledge point tag level. Using a hierarchical multi-label classification model, the subject label, first-level knowledge point label, and second-level knowledge point label of each test question are predicted sequentially through a multi-round dialogue. Each round of dialogue is guided by the labels predicted in the previous round, and the test question is classified layer by layer in combination with the candidate label set corresponding to the current level to obtain the complete knowledge point label path of each test question. Construct a question-knowledge point association matrix based on the complete knowledge point tag paths of all questions. The rows of the matrix correspond to the questions, and the columns correspond to the knowledge points. When the matrix element takes the first value, it means that the question tests that knowledge point, and when it takes the second value, it means that the question does not test that knowledge point. Input the response matrix and the question-knowledge point association matrix into the preset cognitive diagnostic model. The cognitive diagnostic model calculates the probability of each student's mastery of each knowledge point based on whether the student's answers in the response matrix are correct or not and the relationship between the questions and knowledge points in the question-knowledge point association matrix. Based on the mastery probability, a student diagnostic profile is generated, which includes the mastery level of each knowledge point and the identification results of weak knowledge points.

[0066] In some embodiments, the assessment feedback module is specifically used to extract the mastery probability of each knowledge point from the student's diagnostic profile, and to identify the knowledge points whose mastery probability is lower than a preset probability threshold as target knowledge points. The appropriate difficulty level for the target knowledge point is determined based on the numerical range of the probability of mastery, and the probability of mastery is directly proportional to the adaptive difficulty level. Retrieve parent, child, and sibling knowledge points that have a hierarchical relationship with the target knowledge point from the teaching knowledge graph, and extract the definition description information and typical example information of the target knowledge point; combine the retrieved parent, child, and sibling knowledge points, definition description information, and typical example information to form a contextual knowledge base; Using a prompt word generation framework, character setting information, contextual knowledge base, target knowledge points, adaptive difficulty level, number of questions, question types, and question format examples are combined according to a preset template structure to generate structured prompt words. The structured prompt words include task description fields, background knowledge fields, question attribute fields, and format example fields. The structured prompts are input into a multimodal large language model to generate personalized test questions that include the question stem, the correct option, the distractors, and detailed explanations.

[0067] In some embodiments, the teaching execution module is specifically used to construct a teaching agent, which includes a planning component, a toolset, and a memory component; the toolset includes at least a knowledge point analysis tool, a test question generation tool, a cognitive diagnosis tool, a teaching behavior analysis tool, a knowledge graph query tool, a learning path planning tool, and a textbook explanation tool; A set of teaching scenario questions containing multiple tool call steps is generated by a multimodal large language model. Tool call sequence data corresponding to each teaching scenario question is collected. The tool call sequence data records the thinking content of the teaching agent in each step, the name and parameters of the tool called, and the observation results returned by the tool in a preset format. Based on tool call sequence data, the tool call fine-tuning of the multimodal large language model is performed, enabling it to generate tool call sequences according to a preset format; The teaching and learning process steps are pre-set in the system prompts of the teaching intelligence agent. The teaching and learning process steps include: understanding the learning status based on the student's diagnostic profile, calling the cognitive diagnostic tool to analyze the student's weak knowledge points, calling the test question generation tool to generate personalized assessment test questions, evaluating the student's answer and giving feedback, identifying the knowledge points that the student repeatedly makes mistakes on and calling the textbook explanation tool to provide knowledge tutoring, and calling the learning path planning tool to recommend subsequent learning paths. At the same time, the teaching behavior types are matched with the pre-set teaching intervention strategy library to determine the target teaching intervention strategies that are suitable for the current teaching scenario; Summarize the execution results of the teaching and learning process steps and generate a multimodal feedback report. The multimodal feedback report should include at least a knowledge point mastery probability distribution chart, weak knowledge point analysis text, personalized learning path suggestion text, and a targeted practice recommendation list.

[0068] This application also discloses a control device.

[0069] Specifically, the control device includes a memory and a processor. The memory stores a computer program that can be loaded and executed by the processor to implement the aforementioned teaching content understanding and intelligent evaluation feedback method based on a multimodal large model.

[0070] This application also discloses a computer-readable storage medium.

[0071] Specifically, the computer-readable storage medium stores a computer program that can be loaded and executed by a processor, such as the above-mentioned teaching content understanding and intelligent evaluation feedback method based on a multimodal large model. The computer-readable storage medium includes, for example, various media that can store program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0072] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A method for instructional content comprehension and intelligent assessment feedback based on a multimodal large model, characterized in that, include: Collect teaching data, extract audio and visual text, and align them chronologically to form a structured multimodal teaching record; Extract the set of teaching content keywords and the set of original text fragments from the textbook text, and replace the successfully matched keywords and original text fragments in the audio text with preset mask symbols to obtain the masked teaching behavior text; The masked teaching behavior text is input into a multimodal large language model to generate a structured relation description, and the teaching knowledge point entities and their relation triples are extracted to construct a teaching knowledge graph. The speech text and the masked teaching behavior text are vectorized by a text embedding model to obtain semantic feature vectors and behavioral feature vectors. The semantic feature vectors, behavioral feature vectors and location feature vectors are concatenated and fused to identify the teaching behavior category. A response matrix is ​​constructed based on the test question data. Knowledge points are predicted for the test questions based on a hierarchical multi-label classification model, and a test question-knowledge point association matrix is ​​constructed. Based on the response matrix and the test question-knowledge point association matrix, the probability of students mastering each knowledge point is calculated, and a student diagnostic profile is generated. Based on the student diagnostic profile, target knowledge points and adaptive difficulty levels are determined, and relevant knowledge point information is retrieved from the teaching knowledge graph to construct a contextual knowledge base. Using the prompt word generation framework and the contextual knowledge base, structured prompt words are generated and input into the multimodal large language model to generate personalized assessment questions. Based on the personalized assessment questions, a teaching agent is constructed. The multimodal large language model is fine-tuned using tools. The teaching agent is executed according to the preset teaching aid process steps. Based on the teaching behavior category, the corresponding teaching intervention strategy is matched, and a multimodal feedback report is generated.

2. The teaching content understanding and intelligent evaluation feedback method based on a multimodal large model according to claim 1, characterized in that, The collected teaching data includes extracting audio and visual text, aligning them temporally, and forming a structured multimodal teaching record, comprising: Classroom teaching videos are acquired through audio and video acquisition equipment. These videos include footage of the teacher lecturing, blackboard writing, and courseware projection. The audio and video streams of the classroom teaching videos are separated to obtain independent teacher audio data and classroom video frame sequences. The teacher's audio data is transcribed using speech recognition technology. Each transcribed text segment is labeled with a start timestamp and an end timestamp to obtain a speech-text sequence with timestamp information. Keyframes are extracted from the classroom video frame sequence at preset frame intervals to obtain a keyframe image sequence; the multimodal large language model is used to perform visual content analysis on the keyframe image sequence to identify and extract the blackboard text content and the courseware text content. Based on the timestamps corresponding to the keyframe images, timestamp information is marked on the blackboard text content and the courseware text content to obtain blackboard visual text and courseware visual text with timestamp information. Based on a unified timeline, each speech-to-text segment, the blackboard visual text, and the courseware visual text in the speech-text sequence are aligned and merged according to their respective timestamp information to form the structured multimodal teaching record.

3. The teaching content understanding and intelligent evaluation feedback method based on a multimodal large model according to claim 2, characterized in that, The process involves extracting a set of teaching content keywords and a set of original text fragments from the textbook text, replacing the successfully matched keywords and original text fragments in the audio text with a preset mask symbol, and obtaining the masked teaching behavior text, including: The textbook text is segmented into words, and the weight value of each word in the textbook text is calculated based on the inverse document frequency of the word. Words with weight values ​​higher than a preset weight threshold are filtered out to obtain the set of keywords for the teaching content. The textbook text is sliced ​​according to paragraph boundaries to form a set of original text fragments consisting of several teaching text segments; The audio text is matched against each teaching text segment in the set of original text segments in turn. If a match is successful, the matched part is marked as the original text of the classroom content. For the remaining part of the audio text that is not marked as the original text of the class content, perform keyword matching with the keyword set of the teaching content, and replace the successfully matched keywords with preset mask symbols; The portion marked as the original classroom content is concatenated with the remaining portion after masking and replacement in its original order to obtain the masked teaching behavior text.

4. The teaching content understanding and intelligent evaluation feedback method based on a multimodal large model according to claim 3, characterized in that, The process of inputting the masked teaching behavior text into a multimodal large language model to generate a structured relation description, extracting teaching knowledge point entities and their relation triples, and constructing a teaching knowledge graph includes: A relation description generator is constructed. The relation description generator uses the multimodal large language model, combined with a preset first prompt template, target relation name and relation description generation example, to initialize and generate the masked teaching behavior text to obtain an initial structured relation description. Based on the preset second prompt template and feedback scoring example, the initial structured relation description is scored in multiple dimensions based on the multimodal large language model. The multidimensional scoring includes entity positioning accuracy score, semantic clarity score and information richness score, and feedback scoring results are generated. Based on the preset third prompt template and iterative optimization example, the initial structured relationship description and the feedback scoring result are input into the multimodal large language model for optimization to generate an optimized structured relationship description; The optimized structured relationship description replaces the initial structured relationship description, and the multi-dimensional scoring and optimization operations are repeated until the feedback scoring result meets the preset scoring threshold or reaches the preset maximum number of iterations, thus obtaining the structured relationship description. The teaching knowledge point entities are identified from the masked teaching behavior text. The teaching knowledge point entities are semantically matched with the structured relation description to determine the relation type between the teaching knowledge point entities. The relation triples are extracted and formed, and each relation triple contains a head entity, a relation type, and a tail entity. Based on the aforementioned relational triples, the teaching knowledge graph is constructed according to the hierarchical structure of courses, knowledge units, knowledge chapters, knowledge blocks, and knowledge points.

5. The teaching content understanding and intelligent evaluation feedback method based on a multimodal large model according to claim 1, characterized in that, The process involves constructing a response matrix based on the assessment question data, predicting knowledge points for the questions using a hierarchical multi-label classification model, and constructing a question-knowledge point association matrix. Based on the response matrix and the question-knowledge point association matrix, the probability of students mastering each knowledge point is calculated, and a student diagnostic profile is generated, including: Based on the assessment question data and its answer data, the answer data is encoded according to the student dimension and the question dimension to construct the answer response matrix. The rows of the matrix correspond to students, the columns of the matrix correspond to questions, and the matrix elements represent whether the student's answer to the question is correct or not. A hierarchical knowledge point tag system is obtained from the teaching knowledge graph, which includes a subject tag level, a first-level knowledge point tag level, and a second-level knowledge point tag level. Using the hierarchical multi-label classification model, the subject label, first-level knowledge point label and second-level knowledge point label of each test question are predicted in a multi-round dialogue manner. Each round of dialogue is guided by the label predicted in the previous round, and the candidate label set corresponding to the current level is combined to perform hierarchical classification to obtain the complete knowledge point label path of each test question. Construct the question-knowledge point association matrix based on the complete knowledge point tag paths of all questions. The rows of the matrix correspond to the questions, and the columns of the matrix correspond to the knowledge points. When the matrix element takes the first value, it means that the question tests the knowledge point, and when it takes the second value, it means that the question does not test the knowledge point. The response matrix and the question-knowledge point association matrix are input into a preset cognitive diagnostic model. The cognitive diagnostic model calculates the mastery probability of each student for each knowledge point based on whether the student's answers in the response matrix are correct or not and the association relationship between the questions and knowledge points in the question-knowledge point association matrix. The model generates a student diagnostic profile based on the mastery probability, which includes the mastery level of each knowledge point and the identification results of weak knowledge points.

6. The teaching content understanding and intelligent evaluation feedback method based on a multimodal large model according to claim 5, characterized in that, The process involves determining target knowledge points and adaptive difficulty levels based on the student's diagnostic profile, retrieving relevant knowledge point information from the teaching knowledge graph to construct a contextual knowledge base, generating structured prompts using a prompt word generation framework and the contextual knowledge base, and inputting these prompts into the multimodal large language model to generate personalized assessment questions, including: Extract the mastery probability of each knowledge point from the student diagnostic profile, and identify the knowledge points whose mastery probability is lower than a preset probability threshold as target knowledge points; The appropriate difficulty level corresponding to the target knowledge point is determined based on the numerical range of the mastery probability, and the mastery probability is directly proportional to the adaptive difficulty level. The parent node knowledge point, child node knowledge point, and sibling node knowledge point that have a hierarchical relationship with the target knowledge point are retrieved from the teaching knowledge graph, and the definition description information and typical example information of the target knowledge point are extracted; the retrieved parent node knowledge point, child node knowledge point, sibling node knowledge point, definition description information, and typical example information are combined to form the context knowledge base; Using the aforementioned prompt word generation framework, the character setting information, the context knowledge base, the target knowledge points, the adaptive difficulty level, the number of questions, the question types, and the question format examples are combined according to a preset template structure to generate the structured prompt words. The structured prompt words include a task description field, a background knowledge field, a question attribute field, and a format example field. The structured prompts are input into the multimodal large language model to generate personalized test questions that include the question stem, correct option, distractor option, and detailed explanation.

7. The teaching content understanding and intelligent evaluation feedback method based on a multimodal large model according to claim 6, characterized in that, The process involves constructing a teaching intelligence agent based on the personalized assessment questions, fine-tuning the multimodal large language model using tools, executing the teaching intelligence agent according to preset teaching aid process steps, matching corresponding teaching intervention strategies based on the teaching behavior categories, and generating a multimodal feedback report, including: The teaching intelligent agent is constructed, which includes a planning component, a toolset, and a memory component; the toolset includes at least a knowledge point analysis tool, a test question generation tool, a cognitive diagnosis tool, a teaching behavior analysis tool, a knowledge graph query tool, a learning path planning tool, and a textbook explanation tool; The multimodal large language model generates a set of teaching scenario questions containing multiple tool call steps. Tool call sequence data corresponding to each teaching scenario question is collected. The tool call sequence data records the thinking content of the teaching agent in each step, the name and parameters of the tool called, and the observation results returned by the tool in a preset format. Based on the tool call sequence data, the multimodal large language model is fine-tuned to enable it to generate tool call sequences in a preset format; The teaching and learning process steps are pre-set in the system prompts of the teaching intelligence agent. The teaching and learning process steps include: understanding the learning status based on the student diagnostic profile, calling the cognitive diagnostic tool to analyze the student's weak knowledge points, calling the test question generation tool to generate the personalized assessment test questions, evaluating the student's answer and giving feedback, identifying the knowledge points that the student repeatedly makes mistakes on and calling the textbook explanation tool to provide knowledge tutoring, and calling the learning path planning tool to recommend subsequent learning paths. At the same time, the teaching behavior types are matched with a pre-set teaching intervention strategy library to determine the target teaching intervention strategy that is suitable for the current teaching scenario; The execution results of the teaching and learning process steps are summarized to generate the multimodal feedback report. The multimodal feedback report includes at least a knowledge point mastery probability distribution chart, weak knowledge point analysis text, personalized learning path suggestion text, and a targeted practice recommendation list.

8. A teaching content comprehension and intelligent assessment feedback device based on a multimodal large model, characterized in that, The device includes: The teaching data module is used to collect teaching data, extract audio and visual text, and perform temporal alignment to form a structured multimodal teaching record. The behavior matching module is used to extract a set of teaching content keywords and a set of original text fragments from the teaching material text, and replace the successfully matched keywords and original text fragments in the audio text with preset mask symbols to obtain the masked teaching behavior text; The knowledge graph module is used to input the masked teaching behavior text into a multimodal large language model, generate a structured relation description, extract teaching knowledge point entities and their relation triples, and construct a teaching knowledge graph. The behavior classification module is used to vectorize the speech text and the masked teaching behavior text respectively through a text embedding model to obtain semantic feature vectors and behavior feature vectors. The semantic feature vectors, behavior feature vectors and location feature vectors are concatenated and fused to identify the teaching behavior category. The student profiling module is used to construct a response matrix based on the assessment question data, predict knowledge points for the questions based on a hierarchical multi-label classification model, and construct a question-knowledge point association matrix; based on the response matrix and the question-knowledge point association matrix, the module calculates the probability of the student's mastery of each knowledge point and generates a student diagnostic profile. The assessment and feedback module is used to determine the target knowledge points and adaptive difficulty level based on the student diagnostic profile, retrieve relevant knowledge point information from the teaching knowledge graph to construct a contextual knowledge base, generate structured prompt words using the prompt word generation framework and the contextual knowledge base, and input them into the multimodal large language model to generate personalized assessment questions. The teaching execution module is used to construct a teaching agent based on the personalized assessment questions, fine-tune the multimodal large language model using tools, execute the teaching agent according to the preset teaching aid process steps, match corresponding teaching intervention strategies according to the teaching behavior categories, and generate multimodal feedback reports.

9. A control device, characterized in that, The device includes: A memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executed as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer program is stored that can be loaded by a processor and executed as described in any one of claims 1 to 7.

Images