Classroom evaluation system based on large model agent architecture and retrieval enhancement generation

Abstract

The invention belongs to the technical field of teaching systems, and particularly discloses a classroom evaluation system based on large model proxy architecture and retrieval enhancement generation, comprising: a retrieval module used for determining a target document block by analyzing semantic similarity based on a word embedding vector and an evaluation standard query vector corresponding to a document block of teaching design; the classroom semantic information extraction module is used for inputting and extracting cue words to the large language model and obtaining an extraction result output by the large language model; the evaluation module is used for analyzing a target evaluation index suitable for the actual classroom teaching content in the evaluation standard through a large language model based on the extraction result, and determining an evaluation standard query vector through word embedding; and based on the extraction result and the target document block, analyzing the alignment degree between the target document block and the actual classroom teaching content through a large language model, determining a score corresponding to the target evaluation index, and generating a feedback suggestion. According to the invention, the classroom teaching content can be evaluated efficiently and reliably.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of education, and more particularly, to a classroom evaluation system based on large model agent architecture and retrieval enhanced generation. BACKGROUND

[0002] Teaching design plays a key role in the teaching process by promoting students' learning and development. It involves making a systematic and comprehensive teaching plan by setting clear learning goals, using effective teaching strategies, and implementing strict evaluation methods, and is a key educational tool that reflects teachers' teaching design methods and teaching strategies. Although teaching design is a breakthrough in improving teaching content, the challenge is still the effective implementation of teachers. Therefore, it is particularly important to evaluate the implementation of teaching design in actual classroom teaching. Accurate evaluation results and targeted feedback suggestions can help teachers better understand the teaching situation in the classroom and further promote the realization of teaching goals. The main purpose of evaluating the implementation quality of teaching design in classroom teaching is to improve teaching quality and also to provide a basis for assisting teachers in teaching improvement. How to efficiently and reliably evaluate classroom teaching content is a technical problem that needs to be solved in the field. SUMMARY

[0003] In view of the defects of the prior art, the purpose of the present application is to efficiently and reliably evaluate classroom teaching content.

[0004] To achieve the above purpose, in a first aspect, the present application provides a classroom evaluation system based on large model agent architecture and retrieval enhanced generation, which comprises a retrieval module, a classroom semantic information extraction module and an evaluation module (based on Think-Evaluate-Suggestion thought chain); The retrieval module is used to determine the target document block by analyzing the semantic similarity based on the word embedding vector corresponding to the document block of the teaching design (the document block of the teaching design can be obtained by cutting the teaching design document) and the evaluation standard query vector, and the semantic similarity between the word embedding vector corresponding to the target document block and the evaluation standard query vector satisfies the preset similarity condition; The classroom semantic information extraction module is used to input the extraction prompt word to the large language model and obtain the extraction result output by the large language model, and the extraction prompt word is used to instruct the large language model to extract the classroom semantic text from the actual classroom teaching content as the extraction result; The evaluation module is configured to analyze target evaluation indexes suitable for the actual teaching content in the evaluation standard based on the extraction result by using the large language model, determine an evaluation standard query vector based on the target evaluation indexes by using word embedding, and provide the evaluation standard query vector to the retrieval module; analyze the alignment degree between the target document block and the actual teaching content in the classroom based on the extraction result and the target document block fed back by the retrieval module by using the large language model, and determine a score corresponding to the target evaluation index; and generate a feedback suggestion based on the evaluation standard and the score corresponding to the target evaluation index by using the large language model.

[0005] Specifically, the classroom semantic information extraction module extracts key semantic text from the actual classroom content by using the extraction prompt word to guide the large language model, converts unstructured teaching information into a structured analysis object, and provides the structured analysis object as the extraction result.

[0006] The evaluation module adopts a Think-Evaluate-Suggestion thought chain, intelligently matches the evaluation standard and the actual teaching content in the classroom by using the large language model as an expert agent, obtains target evaluation indexes, generates corresponding query vectors, feeds back the query vectors to the retrieval module, combines the extraction result and the target document block fed back by the retrieval module, quantitatively analyzes the alignment degree between the teaching design and the actual execution by using the deep semantic understanding capability of the large language model, generates an objective score, and finally generates specific and operable feedback suggestions based on the evaluation standard and the score result.

[0007] The retrieval module analyzes the semantic similarity between the word embedding vector of the teaching design document block and the evaluation standard query vector, accurately locates the target document block highly related to the current evaluation target, and ensures the pertinence of the evaluation basis.

[0008] The whole process realizes efficient automation of the evaluation process and professional reliability of the result by using the information conversion based on semantic extraction, the logical reasoning based on the thought chain, and the accurate positioning based on vector retrieval, thereby improving the evaluation efficiency by relying on the natural language understanding and generation capability of the large model, and ensuring the explainability of the accurate matching and alignment analysis of the teaching design by using retrieval enhancement.

[0009] In a possible implementation, the semantic similarity index used by the retrieval module includes one or more of the following indexes: cosine similarity, Manhattan distance, and Euclidean distance.

[0010] In a possible implementation, the analysis of the semantic similarity includes: Based on the word embedding vector corresponding to the document block of the teaching design and the evaluation standard query vector, a first similarity is calculated to determine a candidate set, the candidate set is composed of one or more candidate document blocks, the first similarity between the word embedding vector corresponding to the candidate document block and the evaluation standard query vector satisfies a first preset similarity condition, and the first similarity is cosine similarity. The target document block is filtered from the candidate set by calculating a second similarity based on the candidate set and the evaluation standard query vector, the second similarity between the word embedding vector corresponding to the target document block and the evaluation standard query vector satisfies a second preset similarity condition, and the second similarity is Manhattan distance or Euclidean distance.

[0011] In a possible implementation, the extraction prompt word includes: an extraction task description text, a plurality of few-shot question and answer sample pairs, actual classroom teaching content, and an extraction type. The extraction task description text is used to instruct the large language model to extract classroom semantic text from the actual classroom teaching content according to the specified extraction type (teaching method, teaching evaluation).

[0012] In a possible implementation, the few-shot question and answer sample pair includes: an actual classroom teaching content sample and a corresponding extraction result label.

[0013] It can be understood that the extraction prompt word guides the large language model to accurately complete the classroom semantic information extraction task through structured design.

[0014] Specifically, the extraction task description text clearly instructs the large language model to extract key semantic information from the actual classroom teaching content according to the specified extraction type (such as teaching method, teaching evaluation, etc.), providing clear task guidance for the model; the plurality of few-shot question and answer sample pairs show the mapping relationship between the "actual classroom teaching content sample" and the "corresponding extraction result label", so that the large language model learns the extraction rules and semantic patterns in a small number of examples, thereby quickly adapting to complex teaching scenarios; finally, the model combines the patterns in the task description and the sample pair to convert the input actual classroom teaching content into structured classroom semantic text.

[0015] Therefore, this design not only ensures the pertinence of the extraction target through the task description, but also improves the model's understanding of specific semantics in the teaching field through the few-shot sample pair, significantly improving the accuracy and consistency of the extraction result.

[0016] In a possible implementation, the evaluation prompt word used by the evaluation module includes: an evaluation task description text and an evaluation thought chain description text. The evaluation task description text includes: role setting content, task theme, task content, and scoring range; the role setting content is used to instruct the large language model to act as an evaluation expert. The evaluation thought chain description text is used to instruct the large language model to perform the evaluation steps.

[0017] In a possible implementation, the evaluation thought chain description text includes: first step indication information, second step indication information, and third step indication information. The first step indication information is used to indicate that the large language model analyzes the target evaluation index suitable for the actual teaching content in the classroom in the evaluation standard; The second step indication information is used to indicate the alignment degree between the target document block and the actual teaching content in the classroom analyzed by the large language model; The third step indication information is used to indicate that the large language model generates feedback suggestions.

[0018] It can be understood that the evaluation prompt word adopted by the evaluation module realizes professional reasoning and efficient decision-making of the large language model in classroom teaching evaluation through the structured task description and the thinking chain guiding mechanism.

[0019] Specifically, the evaluation task description text positions the large language model as an evaluation expert through role setting content, and combines the task theme, task content and scoring range to clearly define the evaluation target and boundary, providing a professional context for the model. The evaluation thinking chain description text constructs a "Think-Evaluate-Suggestion" reasoning framework through step-by-step indication information - the first step indication information guides the model to analyze the target evaluation index suitable for the current classroom content in the evaluation standard, ensuring the relevance of the evaluation dimension; the second step indication information drives the model to compare the target document block with the actual teaching content in the classroom, quantitatively analyze the alignment degree of the two in terms of teaching design consistency, content coverage, etc., and realize the objectivity of the score; the third step indication information prompts the model to generate specific and operable feedback suggestions based on the foregoing analysis.

[0020] Therefore, the design enhances the professional output of the model through role setting, and the step-by-step guidance of the thinking chain significantly reduces the reasoning difficulty of complex evaluation tasks, avoids the logical jumps or omissions that may occur when the model directly generates results, and improves the transparency and explainability of the evaluation process. In addition, the step-by-step instruction design enables the model to accurately focus on the three core links of evaluation index selection, alignment analysis and suggestion generation, not only improving the accuracy and consistency of the evaluation results, but also enhancing the generalization ability of the system to different teaching scenarios, providing technical support for automated teaching evaluation with efficiency and reliability.

[0021] In one possible implementation, the method further includes: obtaining the large language model through a model fine-tuning module; The model fine-tuning module is configured to: Based on the actual teaching content samples in the classroom and the corresponding comment labels, a data set is constructed, and the comment labels are used to represent the scores and feedback suggestions of the actual teaching content samples in the classroom; Based on the data set, the pre-set model (pre-set large language model) is fine-tuned to obtain the fine-tuned large language model.

[0022] Specifically, a special dataset is constructed based on actual teaching content samples and their corresponding review labels (including scores and feedback suggestions), so that the model learns specific semantic patterns and evaluation logic in the teaching evaluation field. Subsequently, through a supervised fine-tuning method, the preset large language model is optimized in parameters using the dataset, i.e., adjusting the model weights to enable it to master the mapping relationship from the classroom content to the evaluation results, with the goal of minimizing the difference between the model output and the review labels.

[0023] It can be understood that fine-tuning significantly improves the domain adaptability of the model in the teaching evaluation task: on the one hand, the model can generate scores and suggestions that better meet the needs of the educational scene by learning professional terms, evaluation dimensions, and feedback styles in real review data, reducing the generalization bias that may occur in general large models; on the other hand, the fine-tuning process enhances the model's sensitivity to key features such as teaching design and classroom interaction, making its performance in sub-tasks such as evaluation index selection and alignment analysis closer to the level of human experts, while improving the consistency and explainability of the output results. In addition, compared to methods that rely on prompt engineering, the fine-tuned model can more efficiently handle complex or edge teaching cases, reducing the dependence on artificially designed prompts, and providing an automated classroom evaluation system with professional, robust, and scalable technical support.

[0024] In a second aspect, the present application provides a classroom evaluation method based on a large model agent architecture and retrieval-enhanced generation, comprising: inputting an extraction prompt to the large language model to obtain an extraction result output by the large language model, the extraction prompt being used to instruct the large language model to extract classroom semantic text from actual teaching content in the classroom as the extraction result; based on the extraction result, analyzing the target evaluation index suitable for the actual teaching content in the classroom through the large language model, and determining the evaluation standard query vector through word embedding based on the target evaluation index; based on the word embedding vector corresponding to the document block of the teaching design and the evaluation standard query vector, determining the target document block by analyzing the semantic similarity, the semantic similarity between the word embedding vector corresponding to the target document block and the evaluation standard query vector satisfying a preset similarity condition; based on the extraction result and the target document block, analyzing the alignment degree between the target document block and the actual teaching content in the classroom through the large language model to determine the score corresponding to the target evaluation index; and generating feedback suggestions through the large language model based on the evaluation standard and the score corresponding to the target evaluation index.

[0025] In one possible implementation, the large language model is obtained by the following steps: based on actual teaching content samples and corresponding review labels, a dataset is constructed, the review labels being used to represent the scores and feedback suggestions of the actual teaching content samples in the classroom; Based on the dataset, fine-tune the preset model to obtain the fine-tuned large language model.

[0026] It can be understood that the beneficial effects of the above-mentioned second aspect can be referred to the related description in the above-mentioned first aspect, which will not be repeated here.

[0027] Overall, compared with the prior art, the above technical solutions conceived by the present application have the following beneficial effects: (1) The present application is based on the method of Retrieval Augmented Generation (RAG), which performs block processing in the process of processing the instructional design document, constructs a vector database through an embedding model, and uses a multi-path retrieval algorithm to accurately retrieve the instructional design blocks.

[0028] (2) The present application proposes a generative structural information extraction scheme, which uses the fine-tuned model to extract structured information from the semantic information contained in the classroom teaching video, and specifies the returned data format to ensure the accuracy of subsequent evaluation content.

[0029] (3) The present application proposes a Think-Evaluate-Suggestion prompting method, which requires the LLM to imitate the evaluation method of human experts in the classroom, evaluates multiple dimensions according to the classroom evaluation indicators, and gives feedback suggestions. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 is a system evaluation flowchart provided by an embodiment of the present application; Figure 2 is a teaching design document retrieval flowchart provided by an embodiment of the present application; Figure 3 is a comparison diagram of the evaluation system and the manual evaluation process provided by an embodiment of the present application; Figure 4 is a flowchart of the classroom evaluation method based on the large model agent architecture and retrieval augmented generation provided by an embodiment of the present application; Figure 5 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to facilitate a clearer understanding of the embodiments of the present application, first some related background knowledge is introduced as follows.

[0032] While human expert classroom observation evaluation methods have intuitive value in capturing teacher-student interaction and teaching dynamics, they also present challenges. This method is resource-intensive and susceptible to the observer's personal judgment and level of engagement, which can greatly affect the reliability of the results. While expert comments on course materials can provide in-depth insights, considering time and manpower, it makes large-scale assessment impractical. In addition, expert evaluation can be influenced by personal preferences, potentially affecting the objectivity and effectiveness of the evaluation.

[0033] The main shortcomings of existing evaluation methods for teaching design implementation in classroom teaching are as follows: (1) Based on traditional manual measurement and evaluation methods, teaching evaluation analysis has a rich theoretical basis, but due to the complexity, subjectivity, and consumption of classroom teaching, manual evaluation methods cannot be popularized, resulting in fewer relevant authoritative scales and evaluation indicators, and research is difficult to break through.

[0034] (2) The cost of manual evaluation analysis is high, and it is highly subjective, lacking objective and automatic data processing models and analysis tools. In the process of traditional teaching evaluation analysis, comprehensive human judgment of classroom audio and video is required, especially a large amount of data information processing, therefore, the existing teaching evaluation analysis reflects complexity and difficulty.

[0035] In view of the above two points, the present application proposes a classroom teaching content evaluation system based on a large model (or large language model, LLM) agent architecture and retrieval enhancement generation. The system evaluates the alignment degree of the teaching design and the classroom teaching content extraction result according to the constructed evaluation framework, and returns the evaluation result according to the evaluation standard, which includes the alignment degree score (1~5) and feedback suggestions. In order to ensure the accuracy of the evaluation system output and the rationality of the feedback suggestions, the evaluation system performs generative information extraction (classroom teaching content extraction result) on the relevant content in the classroom teaching content, and uses the word embedding (embedding) model to segment the teaching design. The segmented vector and the evaluation standard constitute an evaluation retrieval framework, and finally the TES prompt method is used to make the large language model (LLM) simulate the evaluation of the implementation of the teaching design in the classroom teaching content by experts.

[0036] In order to make the purpose, technical scheme and advantages of the present application more clear and understandable, the following will further describe the present application in detail combined with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and not to limit the present application.

[0037] The terms "first" and "second" and the like in the description and claims of the application are used for distinguishing between similar elements and not necessarily for describing a specific sequential or chronological order. For example, the first similarity and the second similarity are used for distinguishing between the similarities and not necessarily for describing a specific sequential or chronological order.

[0038] In the embodiments of the present application, the words "exemplary" and "for example" are used to mean serving as an example, instance, or illustration, at 99 2 / 27 / 2014 11:45 AM least with respect to the particular aspect being discussed. For example, any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of the present application should not be construed as preferred or advantageous over other embodiments or design schemes. Rather, use of the words "exemplary" or "for example" is intended to present concepts in a concrete manner.

[0039] In the description of the embodiments of the present application, unless otherwise specified, "a plurality of" means two or more, for example, a plurality of processing units means two or more processing units, and the like, a plurality of elements means two or more elements, and the like.

[0040] The embodiments of the present application will be described below in detail with reference to the accompanying drawings.

[0041] The base model adopted by the classroom evaluation system is the open source Qwen2.5-8B open source model. In order to ensure that the evaluation system has a matching subject background knowledge in the evaluation of different knowledge background classrooms, the Qwen2.5-8B model is defined as an expert agent that focuses on evaluating the implementation degree of teaching design in classroom teaching. The data set used for fine-tuning to improve the evaluation ability of the Qwen2.5-8B model contains part of the middle school subjects (here only as an example, the present application does not limit the subject), which ensures the accuracy and stability of the output results of the system in the classroom evaluation. Specifically, the purpose is to explore whether the classroom evaluation system can effectively help teachers improve the implementation quality of teaching design in the classroom, and provide a new scheme for personalized classroom evaluation.

[0042] The main modules of this classroom assessment system are threefold. (1) Retrieval module: The instructional design is divided into blocks, and a vector database is constructed through the embedding model, which together with the classroom evaluation criteria constitutes the retrieval module. (2) Classroom semantic information extraction module: The structured information extraction of semantic information contained in the classroom teaching video is performed using the fine-tuned model (Qwen2.5-8B), and the data format of the return is specified. (3) Assessment module: This module is implemented based on a Think-Evaluate-Suggestion prompting method, which requires the LLM (as Agent) to imitate the assessment method of human experts in the classroom, to assess from multiple dimensions according to the classroom evaluation indicators, and to provide feedback suggestions. In addition, the lack of a unified assessment standard for classroom assessment in the absence of suitable assessment tools and evaluation standards is a current challenge. In order to solve this limitation, this application invited experts to construct a hierarchical model containing 5 primary indicators and 13 secondary indicators, and constructed an evaluation standard for the quality of implementation of the teaching plan in classroom teaching. This evaluation standard considers a wide range of factors, including the realization of learning objectives in classroom teaching, the completeness of the teaching process, and the implementation of teaching strategies. It then provides a more comprehensive assessment of the implementation of instructional design in the classroom through a systematic approach, and offers suggestions for teachers and curriculum researchers. Some components of the evaluation standard are shown in Table 1. The system evaluation process is as follows: Figure 1 As shown.

[0043] Table 1. Example of Evaluation Criteria

[0044] (a) Search module; The evaluation of classroom teaching content in the field of education must ensure its accuracy. The feedback and suggestions provided should be based on reliable data and scientific methods to ensure their effectiveness and credibility. In order to solve the illusion problem caused by the lack of quantifiable evaluation criteria when evaluating classroom teaching content in large models, this application uses retrieval augmented generation (RAG) technology to divide the instructional design into document blocks (or text blocks). Based on the relevant document blocks, the completion of the predetermined teaching objectives in the classroom teaching content can be better evaluated. The retrieval module mainly consists of two parts: (1) multi-path retrieval generation of instructional design; (2) evaluation criteria, which are used to evaluate the degree of alignment between instructional design and classroom teaching content.

[0045] The following section explains the process of segmenting and retrieving instructional design documents.

[0046] For example, this application first divides the collected 50 instructional design documents into blocks to reduce the context length of each block. Then, it uses the open-source word embedding model bge-base-zh-v1.5 to convert these document blocks into 5038-dimensional numerical vectors (word embedding vectors). The advantage of the embedding model is its ability to capture the semantic information of words and phrases in the instructional designs, allowing for matching and retrieving relevant information based on semantic rather than literal meaning. These embedded document blocks are stored in a vector database for easy searching. The model's function is to map document blocks into vector form, facilitating similarity comparison and retrieval.

[0047] In the vector retrieval process, the embedding model maps the instructional design text to a vector space and calculates the semantic similarity between texts to identify and evaluate the criterion query vectors. The most similar search information (target document blocks), among which This application addresses the input queries generated by an LLM (as an agent) when mimicking expert evaluations of classroom teaching content. It employs three methods—cosine similarity, Manhattan distance, and Euclidean distance—to calculate semantic distance.

[0048] For cosine similarity, the query vector can be obtained by calculating cos(θ). Similarity between vectors corresponding to document blocks. The closer the value is to 1, the more semantically similar they are.

[0049] For the Manhattan distance, calculate the query vector. The Manhattan distance is the sum of linear distances across all feature dimensions to the vectors corresponding to document blocks. A smaller Manhattan distance indicates smaller differences across feature dimensions and semantic similarity.

[0050] The Euclidean distance calculation evaluates the actual distance between the query vector and the vector corresponding to the document block in the multidimensional semantic space. The smaller the Euclidean distance, the closer the semantic positions of the two are.

[0051] Cosine similarity calculation evaluation criteria query vector Information vector of instructional design documents The formula for calculating semantic distance is: ; Manhattan distance calculation evaluation criteria query vector Information vector of instructional design documents The formula for calculating semantic distance is: ; Euclidean distance calculation evaluation criteria query vector Information vector of instructional design documents The formula for calculating semantic distance is: ; The most relevant instructional design document block vector to the query vector is obtained through multi-path retrieval as external knowledge Used in the subsequent LLM (as Agent) evaluation of the alignment degree of instructional design and teaching content. The corresponding document block is called the target document block.

[0052] The evaluation retrieval framework developed in this application uses Chroma, an open-source vector database specifically designed to store embedded document blocks and perform semantic search. When the LLM (as Agent) begins to mimic the evaluation of teaching content by human experts, the evaluation utterance (evaluation utterance is the output of the Think step in Think-Evaluate-Suggestion) is converted into a series of numerical vectors (query vectors) using the same embedding model (bge-base-zh-v1.5). The embedded evaluation utterance is used to retrieve the most relevant vectors from the Chroma vector database through multi-path retrieval . Then the multi-path retrieved target document block and the evaluation utterance are sent to the LLM (as Agent).

[0053] For the strategy of multi-path semantic distance retrieval, cascade retrieval can be used: the core idea is to filter in stages, first using a method with low computational cost and fast speed for coarse screening (cosine similarity), and then using a method with high computational cost but more accurate for fine screening (Manhattan / Euclidean). This way, while ensuring retrieval quality, it reduces the consumption of computing resources.

[0054] Exemplarily, the first stage retrieval (fast coarse screening): calculate the cosine similarity of the query vector and all document vectors. The first preset similarity condition: according to the similarity ranking, select the top K1 document blocks as the candidate set. K1=5, after multiple experiments, the first coarse screening takes the top 5 as the candidate set, which is the best retrieval effect. The second stage retrieval (accurate fine screening): for the document vectors in the candidate set, calculate the Euclidean distance or Manhattan distance with the query vector respectively. The second preset similarity condition: according to the distance ranking, select the top K2 document blocks as the final retrieval result. K2 is a positive integer.

[0055] In terms of multi-path retrieval, the evaluation retrieval framework developed in this application uses an open-source Python library called LlamaIndex to implement the RAG method. Specifically, LLM (as Agent) uses LlamaIndex to generate questions based on the relevant vectors retrieved from the Chroma vector database and the evaluation speech. Then, LLM (as Agent) asks Qwen2.5-8B to answer the questions based on the relevant blocks and the evaluation speech. Since Qwen2.5-8B generates its response from the multi-path retrieval of instructional design relevant vectors, the likelihood of LLM producing fabricated answers is greatly reduced. In addition, when evaluating classroom instructional content through Qwen2.5-8B, it sets the "temperature" parameter of Qwen2.5-8B, which controls the randomness level in Qwen2.5-8B's response, to make its response as deterministic as possible. The instructional design document retrieval process is as shown in Figure 2 .

[0056] (ii) Classroom semantic information extraction module; The structured information extraction target is to extract the content that can be input into the model evaluation from the classroom semantic text and present it according to the specified structure template. Information extraction through large models converts the information extraction task into a text generation task, which can better handle the complex text structure and diverse semantic information in classroom semantic text.

[0057] The extraction prompt words input to the large model include: instruction, few-shot prompt, and input message (Input Message).

[0058] To solve the problem of insufficient generalization ability of large models when processing domain-specific unstructured text such as classroom semantic text, this application uses instruction for the model: according to the definition of the information extraction task, the background of the entire information extraction task is specified. This step is to provide a clear task background and context for the large model, helping the model understand the text to be processed and the extraction target. Few-shot prompt: Introduce In-Context Learning (ICL), provide original text, extraction type, and extraction result label, to provide context prompts. This step guides the model to understand the extraction target and format of the specific task through a small number of samples. Input message (Input Message): Combine the original text and extraction type to generate a query in the same format as few-shot, and let the large model output the corresponding extraction result. This step is to input specific text into the model (Qwen2.5-8B) to obtain the extraction result.

[0059] Examples of "instruction" above are as follows: "The text you will process is a semantic text of a middle school classroom. Such texts have domain-specificity and may contain various interactions, explanations, activities, and other content during the teaching process. The goal of this information extraction task is to extract two types of key information from the classroom text: teaching methods and teaching evaluations. Teaching methods refer to various teaching means and ways adopted by teachers in the classroom, such as lecture method, discussion method, demonstration method, etc. Teaching evaluations refer to the evaluation and feedback of the teaching process or results, including teacher evaluation of students, student evaluation of teachers, or mutual evaluation among students. Extracting these information helps to analyze teaching quality and classroom interaction. Please note that the extracted information should accurately reflect the content of the text, maintain its original semantics and context."

[0060] Exemplarily, "in_context_learning" above contains "original text", "extraction type", "extraction result label".

[0061] Examples of "original text" in "in_context_learning" are as follows: "In the physics class, the teacher first showed several sets of experimental data on mechanics through PPT, then explained the principles in detail, then let the students discuss in groups and put forward their own opinions. Finally, the teacher commented on each group's speech, pointing out the advantages and areas for improvement."

[0062] Examples of "extraction type" in "in_context_learning" are as follows: "Teaching methods, teaching evaluations."

[0063] Examples of "extraction result label" in "in_context_learning" are as follows: "Teaching methods: [{"method name": "show experimental data", "specific description": "The teacher showed several sets of experimental data on mechanics through PPT, allowing students to intuitively understand the experimental phenomena and data results. This display method can help students establish a perceptual understanding of physical concepts and lay a foundation for subsequent theoretical learning."}, { "Method name: explain principles", "specific description": "The teacher explained the physical principles contained in the mechanics experimental data in detail, including Newton's laws of motion, energy conservation law, etc. During the explanation, the teacher used simple language and combined with real-life examples to make abstract physical principles vivid and easy to understand for students."}, { "Method Name": "Group Discussion", "Detailed Description": "The teacher divides the students into several groups and encourages them to discuss the principles of mechanics. The students actively participate, share their ideas and understanding, and broaden their perspectives through communication and collision. The teacher guides and inspires the groups to ensure the effectiveness and depth of the discussion."], "Teaching Evaluation": [{ "Evaluator": "Teacher", "Object of Evaluation": "Student Groups", "Content of Evaluation": "The teacher commented on each group's speech, pointing out strengths and areas for improvement. For example, for Group A, the teacher praised their in-depth analysis of experimental data and their ability to accurately apply physical principles, but also pointed out that their expression was not clear enough and their logic needed to be improved. For Group B, the teacher praised their unique insights and innovative thinking, but reminded them to pay attention to team collaboration and fully utilize the strengths of each member."}],

[0064] An example of "input_message" is as follows: {"Original Text": "In the physics class, the teacher first showed several sets of experimental data on mechanics through PPT, then explained the principles in detail, then divided the students into groups for discussion and put forward their own opinions. Finally, the teacher commented on each group's speech, pointing out strengths and areas for improvement.", "Extraction Type": "Teaching Method, Teaching Evaluation"}.

[0065] (Three) Evaluation Module (TES Logical Evaluation Thinking Chain); This application studies the process of education experts evaluating classroom teaching and summarizes the general strategies of expert evaluation. The research finds that the strategies of classroom teaching evaluation are highly consistent with the steps of evaluation cognitive process. Therefore, the research attempts to correspond the cognitive process with specific evaluation steps, and based on these two steps, summarizes three basic actions to improve the evaluation system's ability to evaluate the implementation of teaching design in classroom teaching. This application proposes an innovative prompting mechanism TES, namely Think-Evaluate-Suggestion prompt, which requires LLM (as Agent) to mimic the evaluation method of human experts, and to conduct overall evaluation based on the structured information extracted from the target document block and the content of classroom teaching. This method will effectively integrate the advantages of prompting strategies into the evaluation system. The comparison between the evaluation system and the artificial evaluation process is shown in Figure 3 .

[0066] Think-Evaluate-Suggestion consists of three steps. (1) Think: Guide the LLM (as an Agent) to think about which part of the evaluation standard the current obtained classroom teaching content extraction result (information extracted by the classroom semantic information extraction module) belongs to, and determine the target evaluation index in the corresponding secondary index and evaluation standard. After that, according to the target evaluation index, multiple retrieval of the corresponding teaching design is carried out; (2) Evaluate: Evaluate whether the current retrieved teaching design block (i.e. target document block) matches the obtained classroom teaching content extraction result on the specified evaluation index, evaluate the alignment degree of the teaching design block in the corresponding classroom teaching content, and calculate the result according to the alignment index; (3) Suggestion: According to the alignment index, summarize the process and score of the evaluation thinking in the previous step, and give feedback suggestions according to the score and evaluation standard, to ensure the rationality of the feedback suggestions output by the system.

[0067] Exemplarily, the prompt words related to Think-Evaluate-Suggestion (evaluation prompt words) include: "task theme", "role setting", "task content", "scoring range", "evaluation steps".

[0068] Examples of "task theme" are as follows: "single-dimensional teaching design implementation evaluation".

[0069] Examples of "role setting" are as follows: "you are an evaluation expert of {a certain subject}".

[0070] Examples of "task content" are as follows: "evaluate the implementation of {index} in {extracted classroom teaching content} according to the evaluation retrieval framework".

[0071] Examples of "scoring range" are as follows: "1-5 points".

[0072] Exemplarily, "evaluation steps" include: "Step 1: Think", "Step 2: Evaluate", "Step 3: Suggestion".

[0073] Examples of "Step 1: Think" are as follows: "action": ["identify primary indicators: according to the classroom content, determine which primary indicator (teaching objectives, key points, teaching process, teaching strategies, or evaluation methods) of the evaluation retrieval framework is involved."], "refine secondary indicators: under this primary indicator, analyze and lock the corresponding secondary indicators with the classroom content."], "multi-path retrieval design block: extract relevant teaching design elements (such as objectives, activity design, evaluation methods, etc.) from teaching materials, and preliminarily match them with classroom content."], "evaluate matching potential: according to the specific evaluation standard of the secondary index, judge whether these design blocks can effectively correspond to the classroom implementation."].

[0074] Examples of "Step 2: Evaluation" are as follows: "Action": ["Evaluate the match: Compare the extracted design blocks with the course content piece by piece, and score strictly according to the evaluation criteria (1-5 points). Pay attention to adjusting the evaluation method according to the characteristics of the subject (science / humanities)." "Calculate the matching index: According to the weight rules and scoring guidelines of the evaluation criteria, calculate the overall matching score to ensure scientificity and consistency."].

[0075] Examples of "Step 3: Suggestions" are as follows: "Action": ["Summarize the evaluation basis: Clearly explain the analysis process of Step 1 and Step 2, including the use of primary / secondary indicators, design block matching results, and the reason for the final score." "Propose improvements: For low-scoring areas, give practical improvement suggestions based on evaluation criteria, focusing on improving the consistency of teaching design and classroom implementation, while considering feasibility and subject limitations." "Emphasize rationality: Ensure that the suggestions meet the requirements of the evaluation criteria and are in line with the actual classroom needs."].

[0076] In order to make the evaluation knowledge background of LLM more consistent with that of human experts, the present application further defines the role of LLM as an expert in a certain subject, where the field depends on which subject the course being evaluated belongs to. The role played by LLM as an expert improves the model's ability to effectively derive information related to the role from the training data (the dataset used for fine-tuning in the foregoing) to understand the role, thereby producing more accurate and specific evaluations.

[0077] The present application adopts the idea of COT (Chain of Thought) to generate a logical chain from Think, Evaluate to Suggestion. A series of logical chains can be regarded as the cognitive process of the alignment degree of LLM evaluation. However, illusions and factual errors in the internal knowledge acquisition of LLM can trigger non-true or non-critical reasoning chains, ultimately leading to incorrect answers. In order to solve this problem, it is necessary to make LLM reflect the complete reasoning process as much as possible, display the acquisition and transfer of effective information between steps with language, and refine the logic.

[0078] (Four) Experimental analysis; The evaluation system's score for each indicator of the classroom teaching content will be used as the basis for the final evaluation of the entire class. When the system produces its final quality evaluation, the present application requires it to provide corresponding explanations and feedback suggestions for its own judgment. LLM has the ability to express its confidence in the evaluation it has made. Certain reliability in the decision-making process can be ensured.

[0079] Tables 2-6 show the evaluation results of the evaluation system for a middle school mathematics class that explains the equation of a first-degree equation. The evaluation results are based on the evaluation results of five primary indicators and feedback suggestions according to the teaching design of the mathematics class.

[0080] Table 2 One of the result tables of the evaluation system for evaluating middle school mathematics

[0081] Table 3 Second of the result tables of the evaluation system for evaluating middle school mathematics

[0082] Table 4 Third of the result tables of the evaluation system for evaluating middle school mathematics

[0083] Table 5 Fourth of the result tables of the evaluation system for evaluating middle school mathematics

[0084] Table 6 Fifth of the result tables of the evaluation system for evaluating middle school mathematics

[0085] Overall, (1) the present application is based on the method of retrieval enhancement generation (RAG), which is processed in blocks during the processing of the instructional design document, and a vector database is constructed by embedding the model, and a multi-path retrieval algorithm is used for accurate retrieval of the instructional design blocks. (2) The present application proposes a generative structural information extraction scheme, which uses a fine-tuned model to extract structured information from the semantic information contained in the classroom teaching video, and specifies the returned data format to ensure the accuracy of the subsequent evaluation content. (3) The present application proposes a Think-Evaluate-Suggestion prompt method, which requires the LLM to imitate the evaluation method of human experts in the classroom, evaluates multiple dimensions according to the classroom evaluation indicators, and gives feedback suggestions.

[0086] The present application also provides a classroom evaluation method based on a large model agent architecture and retrieval enhancement generation, as shown in Figure 4 The method comprises the following steps S101 to S104.

[0087] S101, inputting an extraction prompt word to a large language model to obtain an extraction result output by the large language model, the extraction prompt word being used to instruct the large language model to extract a classroom semantic text from actual classroom teaching content, as the extraction result; S102, based on the extraction result, analyzing a target evaluation index suitable for the actual classroom teaching content in the evaluation standard by the large language model, and determining an evaluation standard query vector by word embedding based on the target evaluation index; S103, based on the word embedding vector corresponding to the document block of the instructional design and the evaluation standard query vector, the semantic similarity is analyzed to determine the target document block, and the semantic similarity between the word embedding vector corresponding to the target document block and the evaluation standard query vector meets the preset similarity condition; S104, based on the extraction result and the target document block, the alignment degree between the target document block and the actual teaching content in the classroom is analyzed by the large language model to determine the score corresponding to the target evaluation index; based on the evaluation standard and the score corresponding to the target evaluation index, the feedback suggestion is generated by the large language model.

[0088] Specifically, the key semantic text is extracted from the actual classroom content by guiding the large language model through the extraction prompt word, and the unstructured teaching information is converted into a structured analysis object as the extraction result.

[0089] The Think-Evaluate-Suggestion thinking chain is adopted, the large language model (as an expert agent) is used to intelligently match the evaluation standard and the actual teaching content in the classroom to obtain the target evaluation index, and the corresponding query vector is generated, and then the extraction result and the target document block are combined to quantitatively analyze the alignment degree of the instructional design and the actual execution by using the deep semantic understanding ability of the large language model, and finally the specific and operable feedback suggestion is generated based on the evaluation standard and the score result.

[0090] Based on the semantic similarity analysis of the word embedding vector of the instructional design document block and the evaluation standard query vector, the target document block highly related to the current evaluation target is accurately positioned, and the pertinence of the evaluation basis is ensured.

[0091] The whole process realizes the efficient automation of the evaluation process and the professional reliability of the result through the information transformation of semantic extraction, the logical reasoning of the thinking chain and the accurate positioning of the vector retrieval, that is, relying on the natural language understanding and generation ability of the large model to improve the evaluation efficiency, and through retrieval enhancement to ensure the explainability of the accurate matching and alignment analysis of the instructional design, thereby realizing the efficient automation of the evaluation process and the professional reliability of the result.

[0092] Based on the method in the above embodiment, the electronic device provided by the embodiment of the present application comprises: Figure 5 is a structural schematic diagram of the electronic device provided by the embodiment of the present application, as Figure 5 shown, the electronic device can include a processor (Processor) 810, a communications interface (Communications Interface) 820, a memory (Memory) 830 and a communications bus 840, wherein the processor 810, the communications interface 820, the memory 830 complete mutual communication through the communications bus 840. The processor 810 can call the logic instruction in the memory 830 to execute the method in the above embodiment.

[0093] In addition, the logical instructions in the memory 830 described above can be implemented in the form of a software function unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present application.

[0094] Based on the method in the above embodiments, the embodiments of the present application provide a computer readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method in the above embodiments.

[0095] Based on the method in the above embodiments, the embodiments of the present application provide a computer program product, which runs on a processor. When the computer program product runs on the processor, the processor executes the method in the above embodiments.

[0096] It can be understood that the processor in the embodiments of the present application can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor.

[0097] The method steps in the embodiments of the present application can be implemented by means of hardware, or by means of a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in a Random Access Memory (RAM), a flash memory, a Read-only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), a register, a hard disk, a mobile hard disk, a CD-ROM, or any other form of storage medium well known in the art. An exemplary storage medium is coupled to a processor, such that the processor can read information from, and write information to, the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0098] In the above embodiments, all or some of the steps can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or some of the steps can be implemented by one or more computer programs. When implemented by computer programs, the computer programs can be stored in a computer readable storage medium, and executed by a processor. The computer readable storage medium can be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), a register, a hard disk, a mobile hard disk, a CD-ROM, or any other form of storage medium well known in the art. The computer readable storage medium can be coupled to the processor, such that the processor can read information from, and write information to, the computer readable storage medium.

[0099] It can be understood that various numerical numbers involved in the embodiments of the present application are only used for differentiation for convenience of description, and are not used to limit the scope of the embodiments of the present application.

[0100] Those skilled in the art can easily understand that the above description is only the preferred embodiment of the present application, and is not used to limit the present application, and any modification, equivalent replacement and improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A classroom evaluation system based on a large model agent architecture and retrieval augmented generation, characterized in that, The method comprises the following steps: The retrieval module, the classroom semantic information extraction module and the evaluation module; The retrieval module is used for determining a target document block based on the semantic similarity by analyzing the semantic similarity between the word embedding vector corresponding to the document block based on the teaching design and the evaluation standard query vector; the semantic similarity between the word embedding vector corresponding to the target document block and the evaluation standard query vector meets a preset similarity condition; The classroom semantic information extraction module is used for inputting an extraction prompt word into a large language model to obtain an extraction result output by the large language model; the extraction prompt word is used to instruct the large language model to extract classroom semantic text from the actual classroom teaching content as the extraction result; The evaluation module is used for analyzing a target evaluation index suitable for the actual classroom teaching content in the evaluation standard based on the extraction result by using the large language model; the evaluation standard query vector is determined based on the target evaluation index and is provided to the retrieval module by using the word embedding; the alignment degree between the target document block and the actual classroom teaching content is analyzed based on the extraction result and the target document block fed back by the retrieval module by using the large language model to determine the score corresponding to the target evaluation index; and the feedback suggestion is generated based on the evaluation standard and the score corresponding to the target evaluation index by using the large language model.

2. The system according to claim 1, wherein, The semantic similarity index used by the retrieval module includes one or more of the following indexes: cosine similarity, Manhattan distance and Euclidean distance.

3. The system according to claim 2, wherein, The analysis of the semantic similarity comprises the following steps: The candidate set is determined by calculating the first similarity based on the word embedding vector corresponding to the document block based on the teaching design and the evaluation standard query vector; the candidate set is composed of one or more candidate document blocks; the first similarity between the word embedding vector corresponding to the candidate document block and the evaluation standard query vector meets a first preset similarity condition; and the first similarity is cosine similarity; The target document block is selected from the candidate set by calculating the second similarity based on the candidate set and the evaluation standard query vector; the second similarity between the word embedding vector corresponding to the target document block and the evaluation standard query vector meets a second preset similarity condition; and the second similarity is Manhattan distance or Euclidean distance.

4. The system according to claim 1, wherein, The extraction prompt word includes: an extraction task description text, a plurality of question and answer sample pairs, actual classroom teaching content and an extraction type; The extraction task description text is used to instruct the large language model to extract the classroom semantic text from the actual classroom teaching content according to the specified extraction type.

5. The system according to claim 4, wherein, The question and answer sample pair includes: a classroom actual teaching content sample and a corresponding extraction result label.

6. The system according to claim 1, wherein, The evaluation prompt word used by the evaluation module includes: an evaluation task description text and an evaluation thought chain description text; The evaluation task description text includes: role setting content, task theme, task content and score range; the role setting content is used to instruct the large language model to act as an evaluation expert; The evaluation thought chain description text is used to instruct the large language model to perform the evaluation steps.

7. The system according to claim 6, wherein, The evaluation thought chain description text includes: first step indication information, second step indication information and third step indication information; The first step indication information is used to instruct the large language model to analyze the target evaluation index suitable for the actual classroom teaching content in the evaluation standard; The second step indication information is used to indicate the alignment degree between the target document block and the actual classroom teaching content analyzed by the large language model; The third step indication information is used to indicate the feedback suggestion generated by the large language model.

8. The system according to claim 1, wherein, Further comprising: a model fine-tuning module, the large language model is obtained through the model fine-tuning module; The model fine-tuning module is used to: Based on the actual classroom teaching content samples and the corresponding comment labels, a data set is constructed, and the comment labels are used to represent the scores and feedback suggestions of the actual classroom teaching content samples; Based on the data set, the preset model is fine-tuned to obtain the fine-tuned large language model.

9. A classroom evaluation method based on a large model agent architecture and retrieval enhanced generation, characterized in that, It includes: Input extraction prompt words to the large language model to obtain the extraction result output by the large language model, and the extraction prompt words are used to instruct the large language model to extract classroom semantic text from the actual classroom teaching content as the extraction result; Based on the extraction result, the target evaluation index suitable for the actual classroom teaching content in the evaluation standard is analyzed by the large language model, and the target evaluation index is used to determine the evaluation standard query vector through word embedding; Based on the word embedding vector corresponding to the document block of the teaching design and the evaluation standard query vector, the semantic similarity is analyzed to determine the target document block, and the semantic similarity between the word embedding vector corresponding to the target document block and the evaluation standard query vector meets the preset similarity condition; Based on the extraction result and the target document block, the alignment degree between the target document block and the actual classroom teaching content is analyzed by the large language model to determine the score corresponding to the target evaluation index; Based on the evaluation standard and the score corresponding to the target evaluation index, the feedback suggestion is generated by the large language model.

10. The method of claim 9, wherein, The large language model is obtained through the following steps: Based on the actual classroom teaching content samples and the corresponding comment labels, a data set is constructed, and the comment labels are used to represent the scores and feedback suggestions of the actual classroom teaching content samples; Based on the data set, the preset model is fine-tuned to obtain the fine-tuned large language model.

Images