A Method and System for Managing Teaching Evaluation Data Based on Locally Deployed Large Language Models
By deploying a large quantitative language model locally and combining it with structured prompt text and a keyword rule base, the issues of protecting student answer data privacy and hardware barriers were resolved, achieving closed-loop management of teaching evaluation and improving the credibility and interpretability of evaluation results.
Patent Information
- Application Number
- CN202611389673.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-08
- Publication Date
- 2026-11-17
AI Technical Summary
In existing technologies, student responses cannot be uploaded to third-party servers due to privacy protection requirements, which renders cloud-based large language model evaluation services unusable. At the same time, deploying a complete large language model locally requires high hardware configuration, which is difficult for local servers or edge devices in ordinary teaching institutions to handle. Furthermore, the evaluation results directly output by the large language model lack traceable scoring criteria, making it difficult for instructors to judge whether the evaluation conclusions are reasonable.
By deploying a quantitative large language model on a local server or edge device, the system constructs initial evaluation results using structured prompt text, and performs rule verification in conjunction with a pre-set keyword rule base to generate final teaching evaluation results that include evaluation scores, evaluation levels, and interpretable evaluation evidence.
This allows for the initial evaluation of student responses locally, eliminating the need to upload them to a third-party server. This resolves privacy concerns while enhancing the credibility and interpretability of the evaluation results, thus meeting the standardization requirements of teaching management.
Smart Images

Figure CN123118320A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of natural language processing technology, and in particular to a method and system for managing teaching evaluation data based on locally deployed large language models. Background Technology
[0002] With the deepening application of artificial intelligence technology in the field of teaching evaluation, automatic evaluation methods based on large language models are gradually gaining attention. Large language models can understand complex semantics and generate structured evaluation content, offering significant advantages over traditional manual grading and rule-matching methods when handling open-ended tasks such as short-answer questions, lab reports, and programming problems. During the teaching evaluation process, student-submitted data contains sensitive information such as personal details, learning process records, and knowledge acquisition. Once this data is uploaded to a third-party server, there is a risk of data leakage and unauthorized use, especially for high-security teaching institutions such as military academies, where student data must not leave the domain.
[0003] Existing large language model evaluation services are typically provided via cloud APIs, requiring student responses to be uploaded to third-party servers to obtain evaluation results. This is unusable in scenarios with strict data privacy requirements, and the evaluation service becomes completely unavailable when the network is unstable or down. Deploying the complete large language model locally requires high-end hardware, which is difficult for typical teaching institutions' local servers or edge devices to handle. Furthermore, the evaluation results directly output by the large language model often only provide scores and comments, lacking traceable scoring evidence. This makes it difficult for instructors to judge the reasonableness of the evaluation conclusions, resulting in insufficient credibility and interpretability of the evaluation results. Summary of the Invention
[0004] This invention provides a teaching evaluation data management method and system based on locally deployed large language models. It improves the technical problems that cloud-based large language model evaluation services cannot be used in scenarios where student answer data cannot be uploaded to third-party servers due to privacy protection requirements, and locally deployed complete large language models have high hardware configuration requirements and are difficult to run on local servers or edge devices.
[0005] The present invention discloses the following technical solution: In a first aspect, the present invention provides a method for managing teaching evaluation data based on a locally deployed large language model, the method comprising: Acquire the teaching data to be evaluated and the answer data submitted by students corresponding to the learning task data, wherein the teaching data to be evaluated includes learning task data with preset scoring standards or reference answers; The teaching data to be evaluated is input into a large language model deployed on a local server or edge device. The large language model performs an initial evaluation of the answer data according to the preset scoring criteria or reference answers to obtain a structured initial evaluation result. The structured initial evaluation result includes an initial score, evaluation keywords, and evaluation basis. According to the preset keyword rule base, the evaluation keywords are validated to obtain the rule validation results; Based on the structured preliminary evaluation results and the rule verification results, a final teaching evaluation result is generated. The final teaching evaluation result includes an evaluation score, an evaluation grade, and interpretable evaluation evidence. The interpretable evaluation evidence consists of the matching relationship between the evaluation keywords and the preset keyword rule base.
[0006] Secondly, this invention provides a teaching evaluation data management system based on a locally deployed large language model, the system comprising: The data acquisition module is used to acquire the teaching data to be evaluated and the answer data submitted by students corresponding to the learning task data. The teaching data to be evaluated includes learning task data with preset scoring standards or reference answers. The initial evaluation execution module is used to input the teaching data to be evaluated into a large language model deployed on a local server or edge device. The large language model performs an initial evaluation on the answer data according to the preset scoring criteria or reference answers to obtain a structured initial evaluation result. The structured initial evaluation result includes an initial score, evaluation keywords, and evaluation basis. The rule verification module is used to perform rule verification on the evaluation keywords according to the preset keyword rule library and obtain the rule verification result; The result generation module is used to generate a final teaching evaluation result based on the structured preliminary evaluation result and the rule verification result. The final teaching evaluation result includes an evaluation score, an evaluation grade, and interpretable evaluation evidence. The interpretable evaluation evidence consists of the matching relationship between the evaluation keywords and the preset keyword rule base.
[0007] One or more technical solutions provided in this invention have at least the following technical effects or advantages: The technical solution of this invention provides a teaching evaluation data management method based on a locally deployed large language model. First, by acquiring the teaching data to be evaluated and the answer data submitted by students corresponding to the learning task data, it achieves complete acquisition and structured organization of the evaluation object data. This solves the technical problem that the learning task information and student answer data are scattered in teaching evaluation and cannot be directly used as effective input to the large language model, thus providing a data foundation for subsequent localized preliminary evaluation.
[0008] Furthermore, by inputting the teaching data to be evaluated into a large language model deployed on a local server or edge device, the large language model performs a preliminary evaluation of the answer data according to the preset scoring criteria or reference answers, obtaining a structured preliminary evaluation result. This enables the student's answer data to be evaluated locally without uploading to a third-party server, solving the technical problem that student data cannot be used for cloud-based large language model evaluation services due to privacy protection requirements. At the same time, the output of the structured preliminary evaluation result provides a structured input for subsequent rule verification.
[0009] Furthermore, by performing rule verification on the evaluation keywords according to the preset keyword rule base, the rule verification results are obtained, realizing keyword-level verification and quantitative correction of the initial evaluation results of the large language model. This solves the possible bias and randomness problems in the initial scoring of the large language model and provides rule constraints for the accuracy of the final evaluation results.
[0010] Finally, based on the structured preliminary evaluation results and the rule verification results, a final teaching evaluation result is generated. The final teaching evaluation result includes an evaluation score, an evaluation level, and interpretable evaluation evidence. The interpretable evaluation evidence consists of the matching relationship between the evaluation keywords and the preset keyword rule base. This achieves the traceability and interpretability of the evaluation results, solves the technical problems of large language models directly outputting evaluation results without traceable scoring basis and teaching staff having difficulty judging the rationality of evaluation conclusions, and improves the credibility of the evaluation results and the standardization level of teaching management.
[0011] Therefore, this invention achieves closed-loop management of teaching evaluation under the condition that student answer data does not leave the domain by organically combining localized processing of teaching data to be evaluated, construction of structured prompt text, local initial evaluation by large language model, keyword rule verification, and generation of interpretable evaluation evidence. It solves the data privacy risks of cloud evaluation and the hardware threshold of local deployment, while improving the interpretability of evaluation results and system reliability. Attached Figure Description
[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0013] Figure 1 This is a schematic diagram of the overall process of the teaching evaluation data management method based on a locally deployed large language model provided in an embodiment of the present invention.
[0014] Figure 2The flowchart illustrates the initial score correction process for a teaching evaluation data management method based on a locally deployed large language model, as provided in this embodiment of the invention.
[0015] Figure 3 This is a schematic diagram of the structure of a teaching evaluation data management system based on a locally deployed large language model, provided in an embodiment of the present invention.
[0016] The components represented by each number in the attached diagram are described as follows: data acquisition module 11, preliminary evaluation execution module 12, rule verification module 13, and result generation module 14. Detailed Implementation
[0017] This invention provides a teaching evaluation data management method and system based on a locally deployed large language model. It is used to solve the technical problem that in scenarios where student answer data cannot leave the domain and the large language model needs to complete the evaluation locally, the evaluation results directly output by the large language model lack traceable scoring basis, making it difficult for teachers to judge whether the evaluation conclusions are reasonable, resulting in insufficient credibility of the evaluation results.
[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. It should be noted that the numerical values in the embodiments are for illustrative purposes only and do not constitute a limitation on the present invention.
[0019] Example 1, as shown in the appendix Figure 1 As shown, this invention provides a method for managing teaching evaluation data based on a locally deployed large language model. The method includes the following steps: S100: Obtain the teaching data to be evaluated and the answer data submitted by students corresponding to the learning task data. The teaching data to be evaluated includes learning task data with preset scoring standards or reference answers.
[0020] In teaching evaluation scenarios, student responses are sensitive data and cannot be uploaded to third-party servers under strict data privacy protection requirements. However, learning task data and response data are diverse in form, with significant differences in response data corresponding to different task types. The acquired raw data is scattered and lacks a unified organizational format, making it difficult to use directly as effective input for large language models, thus hindering the implementation of localized automated evaluation. Therefore, it is necessary to acquire and combine learning task data and student response data in a structured manner to form standardized teaching data to be evaluated, providing a data foundation for subsequent localized initial evaluation.
[0021] Step S100 in the method provided in this embodiment of the invention includes: Acquire learning task data, which includes task description, preset scoring criteria or reference answers, score settings, and task type identifiers. The task type identifiers include at least one of the following: objective question type, programming question type, lab report type, or short answer question type. Acquire student-submitted answer data, which corresponds to the learning task data. The answer data includes at least one of the following: text answer content, program code, lab data records, or structured form data. Combine the task description, preset scoring criteria or reference answers, score settings, and task type identifiers from the learning task data, along with the answer data, into teaching data to be evaluated.
[0022] Specifically, the learning task data is a complete dataset describing the content of the teaching evaluation task, including the task description, preset scoring criteria or reference answers, score settings, and task type identifier. The task description is a textual explanation of the background, requirements, and objectives of the learning task, used to enable the large language model to understand the specific content and evaluation objectives of the task. The preset scoring criteria or reference answers are the scoring basis pre-set by the instructors, where the preset scoring criteria are detailed scoring rules for each item, and the reference answers are the standard answer content.
[0023] The score is set as the total score for the learning task and the score allocation information for each scoring point. The task type identifier is the classification information that identifies the type of learning task, which includes at least one of the following: objective question type, programming question type, lab report type, or short answer question type. Different task types correspond to different answer data formats and evaluation methods.
[0024] The response data refers to the assignment data submitted by students for the learning tasks, including at least one of the following: text responses, program code, experimental data records, or structured form data. The teaching data to be evaluated is a complete dataset formed by combining the learning task data and the response data, which serves as the input data for the large language model.
[0025] In this step, the first step is to acquire the learning task data. This data includes the task description, preset scoring criteria or reference answers, score settings, and task type identifier.
[0026] It should be noted that preset scoring criteria and reference answers can coexist, or only one of them may exist. Objective questions typically provide both scoring criteria and reference answers; short-answer questions and lab reports typically provide only preset scoring criteria; programming questions provide both reference answers and scoring criteria. Attributes belonging to the same learning task, including task description, preset scoring criteria or reference answers, score settings, and task type identifiers, are associated using the unique identifier of the learning task during data acquisition.
[0027] Furthermore, the system obtains the student-submitted response data. This response data corresponds to the learning task data and includes at least one of the following: text responses, program code, experimental data records, or structured form data.
[0028] It should be noted that the format of the answer data differs depending on the task type. Answer data for objective questions is typically text-based or structured form data; answer data for programming questions is typically program code; answer data for lab reports is typically a combination of text-based answers and lab data records; and answer data for short-answer questions is typically text-based.
[0029] Furthermore, the task description, preset scoring criteria or reference answers, score settings, and task type identifiers in the learning task data, as well as the answer data, are combined into teaching data to be evaluated.
[0030] It should be noted that learning task data is entered or imported by instructors into the local teaching management platform, stored in a local database, and retrieved through the interface provided by the local teaching management platform; it is not uploaded to a third-party server. The teaching data to be evaluated is associated and grouped according to the unique identifier of the learning task. Each piece of teaching data to be evaluated contains complete learning task information and a student's answer data, used for the initial evaluation input of the large language model.
[0031] Therefore, this step, by acquiring learning task data and student response data and associating and combining them according to the unique identifier of the learning task to form the teaching data to be evaluated, achieves complete acquisition and structured organization of the evaluation object data. It solves the technical problem that learning task information and student response data are scattered in teaching evaluation and cannot be directly used as effective input for large language models, thus providing a data foundation for subsequent localized preliminary evaluation.
[0032] S200: Input the teaching data to be evaluated into a large language model deployed on a local server or edge device. The large language model will conduct an initial evaluation of the answer data according to the preset scoring criteria or reference answers, and obtain the structured initial evaluation results.
[0033] Due to limited hardware resources on local servers or edge devices, large language models are difficult to deploy and run directly due to their large number of parameters and computational demands. Furthermore, the free text evaluation results directly output by large language models are not uniformly formatted, making them difficult to process programmatically. Additionally, the evaluation keywords extracted by large language models are not categorized and cannot be directly used for rule validation and quantitative correction of evaluation results. Therefore, it is necessary to lower the deployment threshold of the model through quantization processing and constrain the model's output format through structured prompt text to provide structured input for subsequent rule validation.
[0034] Step S200 in the method provided in this embodiment of the invention includes: A role setting section is created, which is used to set the large language model as a teaching evaluator and instruct the large language model to output an initial score, evaluation keywords, and evaluation basis according to a preset scoring standard or reference answer; a task description section is created, which contains the task description in the learning task data; a scoring standard section is created, which contains the preset scoring standard or reference answer in the learning task data; a response content section is created, which contains the response data; the role setting section, the task description section, the scoring standard section, and the response content section are concatenated in a preset order to obtain structured prompt text; The structured prompt text is input into a large language model deployed on a local server or edge device. The large language model performs an initial evaluation of the answer data in the answer content segment based on the scoring criteria segment in the structured prompt text, and outputs a structured initial evaluation result. The structured initial evaluation result includes an initial score, evaluation keywords, and evaluation criteria. The evaluation keywords are extracted by the large language model from the answer data and are used to indicate the main points of conformity and deviation of the answer data relative to the preset scoring criteria or reference answer. The evaluation criteria are generated by the large language model based on the text fragments in the answer data that correspond to the evaluation keywords.
[0035] Specifically, the role setting section is the role setting part of the structured prompt text, used to assign the large language model the role of the teaching evaluator, and instructing the large language model to output an initial score, evaluation keywords, and evaluation criteria based on preset scoring standards or reference answers. The task description section is a prompt text paragraph containing the task description from the learning task data, enabling the large language model to understand the specific content of the task and the evaluation objectives. The scoring standard section is a prompt text paragraph containing preset scoring standards or reference answers from the learning task data, serving as the scoring basis for the large language model's initial evaluation.
[0036] The response content segment is a prompt text paragraph containing trainee response data, and it is the evaluation object for the initial assessment by the large language model. The structured prompt text is a complete prompt text formed by splicing the role setting segment, task description segment, scoring criteria segment, and response content segment in a preset order. The structured initial assessment result is a structured evaluation result output by the large language model, containing the initial score, evaluation keywords, and evaluation criteria.
[0037] Evaluation keywords are keywords extracted from the response data by the large language model, used to indicate the main points of agreement and deviation between the response data and the preset scoring criteria or reference answers. Evaluation basis consists of explanatory text generated by the large language model from the text fragments in the response data corresponding to the evaluation keywords, used to explain the source of the evaluation keyword determination.
[0038] In this step, the first step is to construct the structured prompt text. Create a role setting section, a task description section, a scoring criteria section, and a response content section, and then concatenate these four sections in a preset order to form the structured prompt text.
[0039] It should be noted that the preset order is: role setting section, task description section, scoring criteria section, and answer content section. The prompt text is organized according to the logical order of role awareness, task comprehension, scoring basis, and evaluation object, so that the large language model can evaluate the answer content after understanding its own role and scoring basis. The role setting section explicitly instructs the large language model to output a structured initial evaluation result in JSON format, including three fields: initial score, evaluation keyword list, and evaluation basis. The JSON format facilitates parsing and processing by the subsequent rule validation module.
[0040] The structured prompt text is input into a large language model deployed on a local server or edge device. The large language model performs an initial evaluation of the answer data in the answer content segment according to the scoring criteria, and outputs a structured initial evaluation result. The structured initial evaluation result includes an initial score, evaluation keywords, and evaluation criteria. The evaluation keywords are extracted by the large language model from the answer data, and the evaluation criteria are generated by the large language model based on the text fragments in the answer data that correspond to the evaluation keywords.
[0041] Furthermore, step S200 in the method provided in this embodiment of the invention further includes: Obtain the original weights of the open-source large language model, wherein the original weights are high-precision floating-point numbers; convert the original weights into low-bit-width numerical representations to obtain quantized weights; keep the model structure of the open-source large language model unchanged, replace the quantized weights in the open-source large language model to obtain a quantized large language model, and use the quantized large language model as the large language model.
[0042] More specifically, weight quantization is the process of converting the original high-precision floating-point weights of a large language model into low-bit-width numerical representations, reducing model storage space and inference computation overhead while maintaining the model structure. The original weights are the high-precision floating-point weight parameters saved after the open-source large language model is trained, typically 32-bit or 16-bit floating-point numbers. Quantized weights are the weight parameters obtained after converting the original weights into low-bit-width numerical representations, using 8-bit integers. This is based on the principle that 8-bit integer quantization can significantly reduce storage usage and inference computation while maintaining controllable loss of model accuracy, enabling the quantized model to run on local servers or edge devices. The quantized large language model is the model obtained by replacing the quantized weights in the open-source large language model while maintaining its original model structure, and is then deployed as a large language model on local servers or edge devices.
[0043] In this step, firstly, the raw weights of the open-source large language model are obtained; these raw weights are high-precision floating-point numbers. Then, the raw weights are converted to 8-bit integers to obtain quantized weights. Finally, keeping the model structure unchanged, the quantized weights are replaced in the open-source large language model to obtain a quantized large language model, which is then used as the large language model deployed on a local server or edge device.
[0044] It should be noted that after using 8-bit integer quantization, the loss of model evaluation accuracy is controlled within an acceptable range. Accuracy verification is performed by calibrating the dataset after quantization, and the consistency between the evaluation results and the original model reaches more than 95%.
[0045] Furthermore, step S200 in the method provided in this embodiment of the invention further includes: When a text fragment in the answer data meets the preset scoring criteria or the scoring points in the reference answer, the keyword corresponding to the text fragment is determined as a positive keyword; when a text fragment in the answer data deviates from the preset scoring criteria or the scoring points in the reference answer, the keyword corresponding to the text fragment is determined as a negative keyword; when a text fragment in the answer data contains information that is not directly related to the score but affects the comprehensiveness of the evaluation, the keyword corresponding to the text fragment is determined as a neutral keyword.
[0046] More specifically, evaluation keywords are categorized into three types based on their correlation with the corresponding text fragments of the answer data and the preset scoring criteria or reference answers: positive keywords, negative keywords, and neutral keywords. Positive keywords are those corresponding to text fragments in the answer data that satisfy the scoring points of the preset scoring criteria or reference answers; negative keywords are those corresponding to text fragments in the answer data that deviate from the scoring points of the preset scoring criteria or reference answers; and neutral keywords are those corresponding to text fragments in the answer data that contain information not directly related to the scoring but affecting the overall evaluation.
[0047] In this step, after the structured preliminary evaluation results are generated, the evaluation keywords are classified according to the relationship between the text fragments and the preset scoring criteria or reference answers: if the text fragments meet the scoring criteria, the corresponding keywords are classified as positive keywords; if the text fragments deviate from the scoring criteria, the corresponding keywords are classified as negative keywords; if the text fragments are not directly related to the scoring but affect the comprehensiveness of the evaluation, the corresponding keywords are classified as neutral keywords.
[0048] It should be noted that the granularity of evaluation keyword extraction is determined by the large language model based on the semantic content of the response data. Each keyword corresponds to a scoring point or deviation point in the response data, and the same text fragment corresponds to only one keyword category. The number of evaluation keywords is related to the richness of the response data content, and typically 3 to 10 evaluation keywords are extracted from each response data point.
[0049] For example, consider the deployment and initial evaluation of a large language model on a local server. The original model uses the Qwen-7B open-source large language model, with original weights of 16-bit floating-point numbers and 7 billion parameters. Converting the original weights to 8-bit integers reduces the model's storage space from approximately 14GB to approximately 7GB. The deployment environment is a local server configured with a single GPU with 16GB of video memory, allowing the quantized model to run normally.
[0050] Using a calibration dataset to compare the evaluation outputs of the model before and after quantization, the consistency between the evaluation results and the original model reached over 95%. The teaching data to be evaluated for a short-answer learning task was input into the quantified large language model. The structured prompt text was concatenated in the order of role setting, task description, scoring criteria, and answer content. The model output the following structured preliminary evaluation results: an initial score of 82 points. Evaluation keywords included complete explanation of principles, lack of example analysis, clear conclusions, and clear logical argumentation. The evaluation basis was that the answer data fully explained the basic principles but did not provide practical application examples; the conclusion was clearly stated; and the overall logical argumentation was clear.
[0051] Therefore, this step involves constructing structured prompt text and inputting the teaching data to be evaluated into a large language model deployed on a local server or edge device. The large language model then performs an initial evaluation of the answers based on preset scoring criteria or reference answers, resulting in a structured initial evaluation result containing the initial score, evaluation keywords, and evaluation criteria. This allows student answers to be evaluated locally without uploading to a third-party server, solving the technical problem that prevents student data from leaving the domain due to privacy protection requirements from using cloud-based large language model evaluation services. Furthermore, the output of the structured initial evaluation result and the classification of evaluation keywords provide structured input for subsequent rule validation.
[0052] S300: Based on the preset keyword rule base, perform rule verification on the evaluation keywords and obtain the rule verification results.
[0053] The evaluation keywords output by the initial evaluation of the large language model are in free text form, lacking standardized category attributes and quantified weight information. Therefore, they cannot be directly used for revising evaluation scores or validating the rules of the evaluation results. A keyword rule base needs to be established to standardize and quantify the initial evaluation results, ensuring their accuracy and consistency. Thus, it is necessary to match and validate the evaluation keywords against a pre-set keyword rule base, statistically analyze keyword distribution, and calculate weighted validation values to provide a rule-based basis for revising the final evaluation scores.
[0054] Step S300 in the method provided in this embodiment of the invention includes: Each evaluation keyword in the structured initial evaluation result is matched against a preset keyword rule base, which stores keywords, keyword categories, corresponding score adjustment directions, and rule weights. When an evaluation keyword successfully matches in the preset keyword rule base, the evaluation keyword, the matched keyword category, the score adjustment direction, and the rule weight are recorded as a matching rule item. Based on all matching rule items, the number of positive keywords, negative keywords, and neutral keywords are counted, and a weighted verification value is calculated based on the rule weight of each matching rule item. The number of positive keywords, the number of negative keywords, the number of neutral keywords, the weighted verification value, and all matching rule items are used as the rule verification result; wherein, the preset keyword rule base is constructed in the following way: obtaining historical teaching evaluation data; the historical teaching evaluation data includes historical learning task data, historical answer data, and corresponding historical score labels; extracting historical evaluation keywords from the historical teaching evaluation data, and determining the positive category, negative category, or neutral category corresponding to each historical evaluation keyword according to the historical score labels; The frequency of each historical evaluation keyword in the positive category sample set, negative category sample set, and neutral category sample set is counted, and the rule weight of each historical evaluation keyword is determined based on the frequency of occurrence. Historical evaluation keywords that meet the preset frequency of occurrence conditions, along with their keyword categories, score adjustment directions, and rule weights, are stored in the preset keyword rule base.
[0055] Specifically, the preset keyword rule base is a rule database that stores keywords, keyword categories, corresponding scoring adjustment directions, and rule weights. It is used to perform rule-based validation of the evaluation keywords output by the large language model. The rule base can be continuously expanded and updated based on historical teaching evaluation data. Keyword categories are classification information that identifies the evaluation attributes of keywords, including positive, negative, and neutral categories, corresponding to positive, negative, and neutral keywords in the evaluation keywords.
[0056] The scoring adjustment direction refers to the direction of adjustment to the initial score after a keyword hit. Positive category keywords are adjusted in a positive direction, negative category keywords are adjusted in a negative direction, and neutral category keywords are not adjusted. Rule weight is the quantified weight value of each keyword in the rule base, reflecting the degree of influence of the keyword on the evaluation result. The value is a real number between 0 and 1, determined through statistical calculation based on historical scoring annotations.
[0057] The matching rule item is a rule record item formed when the evaluation keyword successfully matches in the preset keyword rule library. It includes the evaluation keyword, the category of the matched keyword, the direction of score adjustment, and the rule weight. The weighted verification value is a quantitative value obtained by summing the rule weights in all matching rule items by their corresponding score adjustment coefficients. It reflects the overall direction and degree of influence of all matched keywords on the score.
[0058] Historical teaching evaluation data comprises learning task data, student response data, and corresponding historical scoring labels accumulated during the historical teaching evaluation process. This data is used to construct and update a pre-set keyword rule base. Rule validation results are structured validation outputs containing the number of positive keywords, negative keywords, neutral keywords, weighted validation values, and all matching rule items.
[0059] In this step, each evaluation keyword in the structured preliminary evaluation results is first matched with a preset keyword rule base.
[0060] It should be noted that the matching method is exact matching, meaning a successful match is determined when the text of the evaluation keyword exactly matches a keyword stored in the rule base. If an evaluation keyword does not match successfully in the rule base, that keyword will not be included in the calculation of the weighted check value, but it will still be recorded in the rule check results for later review.
[0061] Furthermore, when an evaluation keyword successfully matches a keyword in the preset keyword rule library, the evaluation keyword, the matched keyword category, the scoring adjustment direction, and the rule weight are recorded as a matching rule item.
[0062] Furthermore, based on all matching rule items, the number of positive keywords, negative keywords, and neutral keywords are counted, and a weighted verification value is calculated according to the rule weight of each matching rule item. The number of positive keywords, negative keywords, neutral keywords, the weighted verification value, and all matching rule items are used as the rule verification result.
[0063] It should be noted that the weighted check value = The rule weights are multiplied by their corresponding score adjustment coefficients, and then summed over all matching rule items. Keyword score adjustments for positive categories are positive, for negative categories are negative, and for neutral categories there is no adjustment.
[0064] Furthermore, the construction process of the preset keyword rule base is as follows: acquire historical teaching evaluation data, extract historical evaluation keywords from the historical teaching evaluation data, determine the positive category, negative category or neutral category corresponding to each historical evaluation keyword according to the historical scoring labels, count the frequency of occurrence of each historical evaluation keyword in the sample set of each category, determine the rule weight according to the frequency of occurrence, and store the historical evaluation keywords that meet the preset frequency of occurrence conditions, their keyword categories, scoring adjustment directions and rule weights into the preset keyword rule base.
[0065] It should be noted that the historical teaching evaluation data comes from historical evaluation records archived in the local teaching management platform, including learning task data, student answer data, and corresponding historical score annotations covering at least two complete teaching cycles. The preset frequency condition is that the keyword appears at least 10 times in the historical teaching evaluation data, based on the premise that keywords with too low an frequency have insufficient statistical significance, making it difficult to reliably determine their category and rule weight. For positive category keywords, the rule weight is the proportion of samples with historical score annotations higher than the average when the keyword appears; for negative category keywords, the rule weight is the proportion of samples with historical score annotations lower than the average when the keyword appears; for neutral category keywords, the rule weight is 0.
[0066] Furthermore, step S300 in the method provided in this embodiment of the invention further includes: Count the number of matching rules with positive keyword categories to obtain the number of positive keywords; count the number of matching rules with negative keyword categories to obtain the number of negative keywords; count the number of matching rules with neutral keyword categories to obtain the number of neutral keywords; multiply the rule weights of all matching rules by their corresponding score adjustment coefficients, and sum the products to obtain the weighted verification value. The score adjustment coefficient is incremented by 1 when the score adjustment direction is positive and decremented by 1 when the score adjustment direction is negative.
[0067] More specifically, the number of positive keywords is the number of matching rule items whose keyword category is positive. The number of negative keywords is the number of matching rule items whose keyword category is negative. The number of neutral keywords is the number of matching rule items whose keyword category is neutral. The score adjustment coefficient is a numerical coefficient corresponding to the score adjustment direction; it is +1 when the score adjustment direction is positive and -1 when the score adjustment direction is negative, and is used to calculate the weighted check value.
[0068] In this step, firstly, the number of matching rule items with keyword categories of positive, negative, and neutral is counted, yielding the number of positive keywords, negative keywords, and neutral keywords, respectively. Then, the rule weights in all matching rule items are multiplied by their corresponding score adjustment coefficients, and the products are summed to obtain the weighted verification value.
[0069] It should be noted that the formula for calculating the weighted check value is: Weighted check value = Where i is the index of the matching rule item, Let i be the rule weight of the i-th matching rule item. Let be the score adjustment coefficient for the i-th matching rule item. This represents the sum of the products of the rule weights and the score adjustment coefficients for all matching rule items. The score adjustment coefficient is incremented by 1 when the score adjustment direction is positive and decremented by 1 when the score adjustment direction is negative. Neutral category keywords are not included in the weighted verification value calculation.
[0070] For example, consider the construction process of a preset keyword rule base and a rule matching and verification process. Obtain 500 historical evaluation records of short-answer questions from the past two teaching cycles from the local teaching management platform. These records include learning task data, student answer data, and corresponding historical rating labels. Extract 120 historical evaluation keywords from the historical answer data and rating labels. Sort by frequency of occurrence, and select 45 keywords that appear at least 10 times.
[0071] Based on historical scoring, keyword categories were determined, including 20 positive keywords, 18 negative keywords, and 7 neutral keywords. Taking "complete explanation of principle" as an example, the proportion of samples with a score higher than the average when this keyword appeared in historical samples was 0.85, and the rule weight was set to 0.85. Forty-five keywords, their categories, scoring adjustment directions, and rule weights were stored in a pre-set keyword rule base. In a preliminary evaluation, the evaluation keywords were "complete explanation of principle," "lack of example analysis," "clear conclusion," and "clear argumentation logic." "Complete explanation of principle," "clear conclusion," and "clear argumentation logic" successfully matched in the rule base and were recorded as positive category matching rules, while "lack of example analysis" did not match successfully.
[0072] Therefore, this step verifies the evaluation keywords according to the preset keyword rule library, records the successfully matched keywords as matching rule items, counts the number of positive keywords, negative keywords and neutral keywords, and calculates the weighted verification value to obtain the rule verification result. This realizes the keyword-level verification and quantitative correction of the initial evaluation results of the large language model, solves the possible bias and randomness problems in the initial scoring of the large language model, and provides rule constraints for the accuracy of the final evaluation results.
[0073] S400: Generate the final teaching evaluation results based on the structured preliminary evaluation results and the rule verification results.
[0074] The structured preliminary assessment results and rule-based validation results generate initial scores and keyword validation information, respectively. However, these two have not yet been integrated into the final evaluation conclusion. The initial scores may be biased, either too high or too low, and the evaluation level and evaluation basis are not yet clear. Instructors cannot obtain a complete evaluation result containing quantitative scores, level determinations, and a chain of evidence. The traceability and interpretability of the evaluation results are insufficient. Therefore, it is necessary to integrate the structured preliminary assessment results and rule-based validation results to generate a final teaching evaluation result that includes evaluation scores, evaluation levels, and interpretable evaluation evidence.
[0075] Step S400 in the method provided in this embodiment of the invention includes: Based on the rule verification results, the initial score is corrected to obtain an evaluation score; the evaluation level is determined based on the evaluation score and the preset level division rules; all matching rule items in the rule verification results are obtained, and for each matching rule item, the frequency of the corresponding evaluation keyword in the answer data is counted; the evidence strength of each matching rule item is determined based on the frequency of occurrence; each matching rule item and its corresponding evidence strength are considered as an interpretable evaluation evidence item, and all interpretable evaluation evidence items constitute interpretable evaluation evidence; the evaluation score, the evaluation level, and the interpretable evaluation evidence are combined to form the final teaching evaluation result. The initial score correction process in this step is as follows: Figure 2 As shown.
[0076] Specifically, the evaluation score is the final score obtained after correcting the initial score based on the rule verification results. The evaluation level is the level result determined based on the evaluation score and the preset level division rules. Explainable evaluation evidence is an evidence set consisting of all explainable evaluation evidence items. Each explainable evaluation evidence item includes a matching rule item and its corresponding evidence strength, used to explain the basis and source of the evaluation score correction. Evidence strength is determined by the frequency of occurrence of the evaluation keywords corresponding to the matching rule item in the answer data. The higher the frequency, the stronger the evidence strength. Evidence strength is divided into three levels: strong, medium, and weak. A frequency of 3 or more occurrences is strong, a frequency of 2 occurrences is medium, and a frequency of 1 occurrence is weak.
[0077] Interpretable evaluation evidence items are evidence entries formed by combining each matching rule item and its corresponding strength of evidence. The final teaching evaluation result is a complete evaluation output formed by the combination of evaluation scores, evaluation grades, and interpretable evaluation evidence, which serves as the final conclusion of the teaching evaluation.
[0078] In this step, firstly, the initial score is corrected based on the rule verification results to obtain the evaluation score. Then, the evaluation level is determined based on the evaluation score and the preset level division rules.
[0079] It should be noted that the preset grading rules are as follows: an evaluation score of 90 or above is excellent, 80 or above and less than 90 is good, 60 or above and less than 80 is satisfactory, and less than 60 is unsatisfactory. This is based on the grading standards commonly used in teaching evaluation.
[0080] Furthermore, all matching rule items in the rule validation results are obtained. For each matching rule item, the frequency of the corresponding evaluation keyword in the response data is counted. The evidence strength of each matching rule item is determined based on the frequency of occurrence.
[0081] It should be noted that the strength of evidence is determined by the frequency of the evaluation keywords corresponding to the matching rule items in the response data. A frequency of 3 or more occurrences indicates strong evidence, 2 occurrences indicates medium evidence, and 1 occurrence indicates weak evidence.
[0082] Furthermore, each matching rule item and its corresponding strength of evidence are treated as an interpretability evaluation evidence item, and all interpretability evaluation evidence items constitute interpretable evaluation evidence. The evaluation score, evaluation grade, and interpretable evaluation evidence are then combined to form the final teaching evaluation result.
[0083] It should be noted that interpretable evaluation evidence, as a component of the final teaching evaluation results, is output along with the evaluation score and evaluation grade. Instructors can use interpretable evaluation evidence to view the keywords, categories, scoring adjustment directions, rule weights, and evidence strength of each matching rule item, thereby tracing the basis for the revision of the evaluation score.
[0084] Furthermore, step S400 in the method provided in this embodiment of the invention further includes: Obtain the number of positive keywords, the number of negative keywords, and the weighted verification value from the rule verification result; when the number of positive keywords is greater than the number of negative keywords, the initial score is lower than the first score threshold, and the weighted verification value is positive, calculate the difference between the number of positive keywords and the number of negative keywords, multiply the difference by a preset unit correction amount to obtain an upward correction amount, and use the upward correction amount to correct the initial score; When the number of positive keywords is greater than the number of negative keywords, and the initial score is lower than the first score threshold, but the weighted verification value is negative, the initial score remains unchanged; when the number of negative keywords is greater than the number of positive keywords, and the initial score is higher than the second score threshold, and the weighted verification value is negative, the difference between the number of negative keywords and the number of positive keywords is calculated, the difference is multiplied by a preset unit correction amount to obtain a downward correction amount, and the downward correction amount is used to correct the initial score. When the number of negative keywords is greater than the number of positive keywords, and the initial score is higher than the second score threshold, but the weighted verification value is positive, the initial score remains unchanged; when the number of positive keywords is equal to the number of negative keywords, or the initial score is between the first score threshold and the second score threshold, the initial score remains unchanged; the corrected initial score is used as the evaluation score; wherein, the preset unit correction amount is a pre-set score correction value corresponding to the difference in the number of keywords per unit.
[0085] More specifically, the first score threshold, set at 60, is used to determine whether the initial score is too low. This threshold is based on the passing grade; an initial score below 60 indicates that the initial evaluation of the large language model is too strict. The second score threshold, set at 90, is used to determine whether the initial score is too high. This threshold is based on the excellent grade; an initial score above 90 indicates that the initial evaluation of the large language model is too lenient.
[0086] The preset unit correction amount is a pre-defined score correction value corresponding to the difference in the number of keywords per unit, set to 1 point. This is based on the assumption that the score impact of each keyword quantity difference is 1 point, consistent with the granularity of individual scoring points in the scoring criteria. The upward correction amount is the correction value obtained by multiplying the difference between the number of positive and negative keywords by the preset unit correction amount; it is used to correct the initial score upwards. The downward correction amount is the correction value obtained by multiplying the difference between the number of negative and positive keywords by the preset unit correction amount; it is used to correct the initial score downwards.
[0087] In this step, firstly, the number of positive keywords, the number of negative keywords, and the weighted verification value are obtained from the rule verification results. Then, based on the comparison relationship of the number of keywords and the score threshold conditions, it is determined whether to perform score correction.
[0088] It should be noted that the following situations allow for maintaining the initial score unchanged: the number of positive keywords is greater than the number of negative keywords and the initial score is lower than the first score threshold but the weighted verification value is negative; the number of negative keywords is greater than the number of positive keywords and the initial score is higher than the second score threshold but the weighted verification value is positive; the number of positive keywords is equal to the number of negative keywords; and the initial score is between the first score threshold and the second score threshold.
[0089] Furthermore, when the number of positive keywords is greater than the number of negative keywords, the initial score is lower than the first score threshold of 60 points, and the weighted verification value is positive, the difference between the number of positive keywords and the number of negative keywords is calculated, the difference is multiplied by a preset unit correction amount of 1 point to obtain the upward correction amount, and the upward correction amount is used to correct the initial score.
[0090] Furthermore, when the number of negative keywords exceeds the number of positive keywords, the initial score is higher than the second score threshold of 90, and the weighted verification value is negative, the difference between the number of negative keywords and the number of positive keywords is calculated. This difference is multiplied by a preset unit correction of 1 point to obtain a downward correction amount, which is then used to correct the initial score. Finally, the corrected initial score is used as the evaluation score.
[0091] It should be noted that the corrected evaluation score should be no lower than 0 and no higher than the total score set in the learning task data. If the corrected result exceeds this range, it will be truncated according to the boundary value.
[0092] For example, consider the process of generating the final teaching evaluation result for a short-answer question. The initial score of the large language model is 82 points. The rule verification result shows 3 positive keywords and 0 negative keywords, with a weighted verification value of 0.85×1 + 0.75×1 + 0.60×1 = 2.20, which is positive. Although the number of positive keywords is greater than the number of negative keywords, the initial score of 82 points is higher than the first score threshold of 60 points, which does not meet the upward correction condition. Therefore, the initial score remains unchanged, and the evaluation score is 82 points.
[0093] According to the preset grading rules, 82 points corresponds to a "good" grade. Among the matching rule items, "complete explanation of the principle" appeared twice in the response data, indicating moderate evidence strength; "clear conclusion" appeared once in the response data, indicating weak evidence strength; and "clear logical argumentation" appeared three times in the response data, indicating strong evidence strength. The interpretability evaluation evidence consists of the above three matching rule items and their evidence strengths. The final teaching evaluation result is an evaluation score of 82 points, an evaluation grade of "good," and the interpretability evaluation evidence includes three evidence items.
[0094] Therefore, this step involves correcting the initial score based on the rule verification results to obtain the evaluation score, determining the evaluation level based on the evaluation score, and determining the strength of evidence based on the frequency of occurrence of matching rule items in the response data. This generates interpretable evaluation evidence consisting of all interpretable evaluation evidence items. The evaluation score, evaluation level, and interpretable evaluation evidence are combined to form the final teaching evaluation result. This achieves traceability and interpretability of the evaluation results, solves the technical problems of large language models directly outputting evaluation results without traceable scoring basis and teaching staff having difficulty judging the rationality of evaluation conclusions, and improves the credibility of evaluation results and the standardization of teaching management.
[0095] Through the specific implementation methods described above, the embodiments of the present invention achieve the following technical effects: In summary, this embodiment achieves structured organization of evaluation object data by acquiring learning task data and student answer data and combining them into teaching data to be evaluated; by constructing structured prompt text, the teaching data to be evaluated is input into a large language model that has been weighted and deployed on a local server or edge device for preliminary evaluation, resulting in a structured preliminary evaluation result containing preliminary scores, evaluation keywords, and evaluation criteria, thus enabling the student answer data to complete the preliminary evaluation locally without uploading it to a third-party server; Furthermore, by matching and verifying the evaluation keywords with the preset keyword rule base, counting the number of keywords in the three categories and calculating the weighted verification value, the rule verification result is obtained. The initial score is corrected based on the rule verification result to obtain the evaluation score, the evaluation level is determined, and the strength of evidence is determined based on the frequency of occurrence of the matching rule item in the answer data, thus generating interpretable evaluation evidence, which is combined into the final teaching evaluation result.
[0096] In this embodiment, weight quantization reduces model storage space and inference computation overhead, enabling large language models to run on local hardware; structured prompt text is organized in a logical order of role cognition, task understanding, scoring criteria, and evaluation object, constraining the model output format; keyword rule verification corrects the initial score through matching rule items and weighted verification values, triggering correction only when the initial score is in a low or high range and the verification value direction is consistent, avoiding blind adjustments; interpretable evaluation evidence fully presents the basis for correction, making the evaluation conclusion traceable and reliable.
[0097] Example 2, as shown in the appendix Figure 3 As shown, based on the inventive concept of the teaching evaluation data management method based on a locally deployed large language model provided in Embodiment 1, this invention also provides a teaching evaluation data management system based on a locally deployed large language model, specifically including: The data acquisition module 11 is used to acquire the teaching data to be evaluated and the answer data submitted by the students corresponding to the learning task data. The teaching data to be evaluated includes learning task data with preset scoring standards or reference answers. The initial evaluation execution module 12 is used to input the teaching data to be evaluated into a large language model deployed on a local server or edge device. The large language model performs an initial evaluation on the answer data according to the preset scoring criteria or reference answers to obtain a structured initial evaluation result. The structured initial evaluation result includes an initial score, evaluation keywords, and evaluation basis. The rule verification module 13 is used to perform rule verification on the evaluation keywords according to the preset keyword rule library and obtain the rule verification result; The result generation module 14 is used to generate a final teaching evaluation result based on the structured preliminary evaluation result and the rule verification result. The final teaching evaluation result includes an evaluation score, an evaluation grade, and interpretable evaluation evidence. The interpretable evaluation evidence consists of the matching relationship between the evaluation keywords and the preset keyword rule base.
[0098] In one embodiment, the data acquisition module 11 is further configured to: Acquire learning task data, which includes task description, preset scoring criteria or reference answers, score settings and task type identifier, wherein the task type identifier includes at least one of objective question type, programming question type, experiment report type or short answer question type; Obtain the answer data submitted by the students, the answer data corresponding to the learning task data, and the answer data including at least one of text answer content, program code, experimental data records or structured form data; The task description, preset scoring criteria or reference answers, score settings, and task type identifier in the learning task data, along with the answer data, are combined to form the teaching data to be evaluated.
[0099] In one embodiment, the initial evaluation execution module 12 is further configured to: Create a role setting section, which is used to set the large language model as a teaching evaluator and instruct the large language model to output an initial score, evaluation keywords and evaluation basis according to a preset scoring standard or reference answer; Create a task description segment, which contains a task description from the learning task data; Create a scoring criteria segment, which includes preset scoring criteria or reference answers from the learning task data; Create a response content segment, which contains the response data; The character setting section, the task description section, the scoring criteria section, and the answer content section are concatenated in a preset order to obtain structured prompt text; The structured prompt text is input into a large language model deployed on a local server or edge device. The large language model performs a preliminary evaluation of the answer data in the answer content segment based on the scoring criteria segment in the structured prompt text, and outputs a structured preliminary evaluation result. The structured preliminary evaluation results include preliminary scores, evaluation keywords, and evaluation criteria. The evaluation keywords are extracted from the response data by the large language model and are used to indicate the main points of agreement and deviation of the response data relative to the preset scoring criteria or reference answers. The evaluation criteria are generated by the large language model based on the text fragments in the response data that correspond to the evaluation keywords.
[0100] Furthermore, the preliminary evaluation execution module 12 is also used for: Obtain the original weights of the open-source large language model, where the original weights are high-precision floating-point numbers; The original weights are converted into low-bit-width numerical representations to obtain quantized weights; Keeping the model structure of the open-source large language model unchanged, the quantization weights are replaced in the open-source large language model to obtain the quantized large language model, which is then used as the large language model.
[0101] Furthermore, the preliminary evaluation execution module 12 is also used for: When a text fragment in the answer data meets the preset scoring criteria or the scoring points in the reference answer, the keyword corresponding to the text fragment is identified as a positive keyword; When a text fragment in the answer data deviates from the preset scoring criteria or the scoring points in the reference answer, the keyword corresponding to the text fragment is identified as a negative keyword; When a text fragment in the response data contains information that is not directly related to the score but affects the overall evaluation, the keyword corresponding to the text fragment is identified as a neutral keyword.
[0102] In one embodiment, the rule verification module 13 is further configured to: Each evaluation keyword in the structured preliminary evaluation results is matched with a preset keyword rule base, wherein the preset keyword rule base stores keywords, keyword categories, scoring adjustment directions corresponding to keywords, and rule weights. When an evaluation keyword successfully matches in the preset keyword rule base, the evaluation keyword, the matched keyword category, the scoring adjustment direction, and the rule weight are recorded as a matching rule item. Based on all matching rule items, count the number of positive keywords, negative keywords, and neutral keywords, and calculate the weighted verification value according to the rule weight of each matching rule item; The number of positive keywords, the number of negative keywords, the number of neutral keywords, the weighted verification value, and all matching rule items are used as the rule verification result; The preset keyword rule base is constructed in the following way: Acquire history teaching evaluation data; the history teaching evaluation data includes history learning task data, history answer data, and corresponding history scoring labels; Historical evaluation keywords are extracted from the historical teaching evaluation data, and the positive, negative, or neutral category corresponding to each historical evaluation keyword is determined based on the historical scoring labels. The frequency of each historical evaluation keyword in the positive category sample set, negative category sample set, and neutral category sample set is counted, and the rule weight of each historical evaluation keyword is determined based on the frequency of occurrence. Historical evaluation keywords that meet the preset frequency conditions, along with their keyword categories, scoring adjustment directions, and rule weights, are stored in the preset keyword rule library.
[0103] Furthermore, the rule validation module 13 is also used for: The number of positive keywords is obtained by counting the number of matching rule items whose keyword category is positive among all matching rule items; The number of negative keywords is obtained by counting the number of matching rule items with negative keyword categories among all matching rule items; The number of neutral keywords is obtained by counting the number of matching rule items with neutral keyword categories among all matching rule items; Multiply the rule weights in all matching rule items by their corresponding score adjustment coefficients, and sum the products to obtain the weighted verification value. When the score adjustment direction is positive, the score adjustment coefficient is +1, and when the score adjustment direction is negative, the score adjustment coefficient is -1.
[0104] In one embodiment, the result generation module 14 is further configured to: Based on the rule verification results, the initial score is corrected to obtain an evaluation score; based on the evaluation score and the preset level division rules, the evaluation level is determined; all matching rule items in the rule verification results are obtained, and for each matching rule item, the frequency of the corresponding evaluation keyword in the answer data is counted; The strength of evidence for each matching rule item is determined based on its frequency of occurrence; each matching rule item and its corresponding strength of evidence are taken as an interpretability evaluation evidence item, and all interpretability evaluation evidence items constitute interpretable evaluation evidence; the evaluation score, the evaluation level, and the interpretable evaluation evidence are combined to form the final teaching evaluation result.
[0105] Furthermore, the result generation module 14 is also used for: Obtain the number of positive keywords, the number of negative keywords, and the weighted verification value from the rule verification results; When the number of positive keywords is greater than the number of negative keywords, the initial score is lower than the first score threshold, and the weighted verification value is positive, the difference between the number of positive keywords and the number of negative keywords is calculated, the difference is multiplied by a preset unit correction amount to obtain an upward correction amount, and the upward correction amount is used to correct the initial score. When the number of positive keywords is greater than the number of negative keywords, and the initial score is lower than the first score threshold, but the weighted verification value is negative, the initial score remains unchanged. When the number of negative keywords is greater than the number of positive keywords, the initial score is higher than the second score threshold, and the weighted verification value is negative, the difference between the number of negative keywords and the number of positive keywords is calculated, the difference is multiplied by a preset unit correction amount to obtain a downward correction amount, and the downward correction amount is used to correct the initial score. When the number of negative keywords is greater than the number of positive keywords, and the initial score is higher than the second score threshold, but the weighted verification value is positive, the initial score remains unchanged. When the number of positive keywords equals the number of negative keywords, or when the initial score is between the first score threshold and the second score threshold, the initial score remains unchanged. The revised initial score will be used as the evaluation score. The preset unit correction amount is a pre-set score correction value corresponding to the difference in the number of keywords per unit.
[0106] This invention provides a teaching evaluation data management method and system based on a locally deployed large language model, which can be applied to scenarios such as teaching evaluation in colleges and universities, homework review on online learning platforms, and training assessment. It can be integrated into a local teaching management platform or edge computing node to achieve localized automatic evaluation of students' answers.
[0107] This invention achieves closed-loop management of teaching evaluation without requiring student responses to leave the domain. This is achieved through structured organization of the teaching data to be evaluated, local deployment of a large quantitative language model, initial evaluation with structured prompt text, keyword rule verification, and generation of interpretable evaluation evidence. This helps protect student data privacy and security, reduces dependence on the network environment, and improves the interpretability and credibility of evaluation results, providing reliable technical support for the digital and intelligent management of teaching evaluation. For specific implementation details of this invention, please refer to Embodiment 1.
[0108] It should be noted that the order of the embodiments in this invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Specific embodiments of the present invention have been described above. Furthermore, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.
Claims
1. A method for managing teaching evaluation data based on a locally deployed large language model, characterized in that: The method includes: Acquire the teaching data to be evaluated and the answer data submitted by students corresponding to the learning task data, wherein the teaching data to be evaluated includes learning task data with preset scoring standards or reference answers; The teaching data to be evaluated is input into a large language model deployed on a local server or edge device. The large language model performs an initial evaluation of the answer data according to the preset scoring criteria or reference answers to obtain a structured initial evaluation result. The structured initial evaluation result includes an initial score, evaluation keywords, and evaluation basis. According to the preset keyword rule base, the evaluation keywords are validated to obtain the rule validation results; Based on the structured preliminary evaluation results and the rule verification results, a final teaching evaluation result is generated. The final teaching evaluation result includes an evaluation score, an evaluation grade, and interpretable evaluation evidence. The interpretable evaluation evidence consists of the matching relationship between the evaluation keywords and the preset keyword rule base.
2. The teaching evaluation data management method based on a locally deployed large language model according to claim 1, characterized in that, Obtain the teaching data to be evaluated and the student-submitted response data corresponding to the learning task data, including: Acquire learning task data, which includes task description, preset scoring criteria or reference answers, score settings and task type identifier, wherein the task type identifier includes at least one of objective question type, programming question type, experiment report type or short answer question type; Obtain the answer data submitted by the students, the answer data corresponding to the learning task data, and the answer data including at least one of text answer content, program code, experimental data records or structured form data; The task description, preset scoring criteria or reference answers, score settings, and task type identifier in the learning task data, along with the answer data, are combined to form the teaching data to be evaluated.
3. The teaching evaluation data management method based on a locally deployed large language model according to claim 1, characterized in that, The teaching data to be evaluated is input into a large language model deployed on a local server or edge device. The large language model performs an initial evaluation of the response data based on the preset scoring criteria or reference answers, obtaining structured initial evaluation results, including: Create a role setting section, which is used to set the large language model as a teaching evaluator and instruct the large language model to output an initial score, evaluation keywords and evaluation basis according to a preset scoring standard or reference answer; Create a task description segment, which contains a task description from the learning task data; Create a scoring criteria segment, which includes preset scoring criteria or reference answers from the learning task data; Create a response content segment, which contains the response data; The character setting section, the task description section, the scoring criteria section, and the answer content section are concatenated in a preset order to obtain structured prompt text; The structured prompt text is input into a large language model deployed on a local server or edge device. The large language model performs a preliminary evaluation of the answer data in the answer content segment based on the scoring criteria segment in the structured prompt text, and outputs a structured preliminary evaluation result. The structured preliminary evaluation results include preliminary scores, evaluation keywords, and evaluation criteria. The evaluation keywords are extracted from the response data by the large language model and are used to indicate the main points of agreement and deviation of the response data relative to the preset scoring criteria or reference answers. The evaluation criteria are generated by the large language model based on the text fragments in the response data that correspond to the evaluation keywords.
4. The teaching evaluation data management method based on a locally deployed large language model according to claim 3, characterized in that, The large language model is an open-source large language model that has undergone weight quantization processing. The large language model is deployed on a local server or edge device and runs in an offline environment. The weight quantization processing includes: Obtain the original weights of the open-source large language model, where the original weights are high-precision floating-point numbers; The original weights are converted into low-bit-width numerical representations to obtain quantized weights; Keeping the model structure of the open-source large language model unchanged, the quantization weights are replaced in the open-source large language model to obtain the quantized large language model, which is then used as the large language model.
5. The teaching evaluation data management method based on a locally deployed large language model according to claim 3, characterized in that, The evaluation keywords include positive keywords, negative keywords, and neutral keywords: When a text fragment in the answer data meets the preset scoring criteria or the scoring points in the reference answer, the keyword corresponding to the text fragment is identified as a positive keyword; When a text fragment in the answer data deviates from the preset scoring criteria or the scoring points in the reference answer, the keyword corresponding to the text fragment is identified as a negative keyword; When a text fragment in the response data contains information that is not directly related to the score but affects the overall evaluation, the keyword corresponding to the text fragment is identified as a neutral keyword.
6. The teaching evaluation data management method based on a locally deployed large language model according to claim 1, characterized in that, Based on a preset keyword rule base, the evaluation keywords are subjected to rule validation to obtain rule validation results, including: Each evaluation keyword in the structured preliminary evaluation results is matched with a preset keyword rule base, wherein the preset keyword rule base stores keywords, keyword categories, scoring adjustment directions corresponding to keywords, and rule weights. When an evaluation keyword successfully matches in the preset keyword rule base, the evaluation keyword, the matched keyword category, the scoring adjustment direction, and the rule weight are recorded as a matching rule item. Based on all matching rule items, count the number of positive keywords, negative keywords, and neutral keywords, and calculate the weighted verification value according to the rule weight of each matching rule item; The number of positive keywords, the number of negative keywords, the number of neutral keywords, the weighted verification value, and all matching rule items are used as the rule verification result; The preset keyword rule base is constructed in the following way: Acquire history teaching evaluation data; the history teaching evaluation data includes history learning task data, history answer data, and corresponding history scoring labels; Historical evaluation keywords are extracted from the historical teaching evaluation data, and the positive, negative, or neutral category corresponding to each historical evaluation keyword is determined based on the historical scoring labels. The frequency of each historical evaluation keyword in the positive category sample set, negative category sample set, and neutral category sample set is counted, and the rule weight of each historical evaluation keyword is determined based on the frequency of occurrence. Historical evaluation keywords that meet the preset frequency conditions, along with their keyword categories, scoring adjustment directions, and rule weights, are stored in the preset keyword rule library.
7. The teaching evaluation data management method based on a locally deployed large language model according to claim 6, characterized in that, Based on all matching rule items, count the number of positive keywords, negative keywords, and neutral keywords, and calculate a weighted verification value according to the rule weight of each matching rule item, including: The number of positive keywords is obtained by counting the number of matching rule items whose keyword category is positive among all matching rule items; The number of negative keywords is obtained by counting the number of matching rule items with negative keyword categories among all matching rule items; The number of neutral keywords is obtained by counting the number of matching rule items with neutral keyword categories among all matching rule items; Multiply the rule weights in all matching rule items by their corresponding score adjustment coefficients, and sum the products to obtain the weighted verification value. When the score adjustment direction is positive, the score adjustment coefficient is +1, and when the score adjustment direction is negative, the score adjustment coefficient is -1.
8. The teaching evaluation data management method based on a locally deployed large language model according to claim 1, characterized in that, Based on the structured preliminary assessment results and the rule validation results, the final teaching evaluation results are generated, including: Based on the verification results of the rules, the initial score is corrected to obtain the evaluation score; The evaluation level is determined based on the evaluation score and the preset level division rules; Obtain all matching rule items in the rule verification result. For each matching rule item, count the frequency of the evaluation keyword corresponding to the matching rule item in the answer data. The strength of evidence for each matching rule item is determined based on its frequency of occurrence. Each matching rule item and its corresponding strength of evidence are treated as an interpretability evaluation evidence item, and all interpretability evaluation evidence items constitute interpretability evaluation evidence. The evaluation scores, evaluation grades, and interpretable evaluation evidence are combined to form the final teaching evaluation result.
9. The teaching evaluation data management method based on a locally deployed large language model according to claim 8, characterized in that, Based on the rule verification results, the initial score is corrected to obtain an evaluation score, including: Obtain the number of positive keywords, the number of negative keywords, and the weighted verification value from the rule verification results; When the number of positive keywords is greater than the number of negative keywords, the initial score is lower than the first score threshold, and the weighted verification value is positive, the difference between the number of positive keywords and the number of negative keywords is calculated, the difference is multiplied by a preset unit correction amount to obtain an upward correction amount, and the upward correction amount is used to correct the initial score. When the number of positive keywords is greater than the number of negative keywords, and the initial score is lower than the first score threshold, but the weighted verification value is negative, the initial score remains unchanged. When the number of negative keywords is greater than the number of positive keywords, the initial score is higher than the second score threshold, and the weighted verification value is negative, the difference between the number of negative keywords and the number of positive keywords is calculated, the difference is multiplied by a preset unit correction amount to obtain a downward correction amount, and the downward correction amount is used to correct the initial score. When the number of negative keywords is greater than the number of positive keywords, and the initial score is higher than the second score threshold, but the weighted verification value is positive, the initial score remains unchanged. When the number of positive keywords equals the number of negative keywords, or when the initial score is between the first score threshold and the second score threshold, the initial score remains unchanged. The revised initial score will be used as the evaluation score. The preset unit correction amount is a pre-set score correction value corresponding to the difference in the number of keywords per unit.
10. A teaching evaluation data management system based on a locally deployed large language model, characterized in that: The system is used to execute the teaching evaluation data management method based on a locally deployed large language model as described in any one of claims 1 to 9, the system comprising: The data acquisition module is used to acquire the teaching data to be evaluated and the answer data submitted by students that correspond to the learning task data; The initial evaluation execution module is used to input the teaching data to be evaluated into a large language model deployed on a local server or edge device. The large language model performs an initial evaluation on the answer data according to the preset scoring criteria or reference answers to obtain structured initial evaluation results. The rule verification module is used to perform rule verification on the evaluation keywords according to the preset keyword rule library and obtain the rule verification result; The results generation module is used to generate the final teaching evaluation results based on the structured preliminary evaluation results and the rule verification results.
