A Smart Homework Grading Method Based on a Large Language Model
By combining multi-dimensional feature filtering and large language models, representative answer sheets and boundary sample pairs are generated, which solves the problems of low efficiency in generating scoring rules and weak iterative optimization capabilities in existing intelligent homework grading technologies. This achieves an efficient, accurate, and flexible scoring process and improves the level of intelligence in homework grading.
Patent Information
- Application Number
- CN202611204658.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-10
- Publication Date
- 2026-11-10
AI Technical Summary
Existing intelligent job grading technologies suffer from problems such as low efficiency in generating scoring details, weak iterative optimization capabilities, insufficient scoring accuracy, high reliance on manual intervention, poor versatility, lack of guaranteed scoring consistency, and poor process interpretability, making it difficult to meet the needs of large-scale, high-precision, intelligent, and universal job grading.
By filtering multi-dimensional features, representative samples of textual features, semantic features, word count features, and score features are obtained. A representative questionnaire set and boundary sample pairs are constructed. Scoring rules are generated by combining a large language model. The scoring rules are iteratively optimized through semantic error analysis to ensure the diversity of samples and the consistency of scoring.
It improved the efficiency of generating scoring details, enhanced scoring accuracy and consistency, reduced manual annotation costs, enabled flexible adaptation to different subjects and question types, and ensured the interpretability and credibility of the scoring results.
Smart Images

Figure CN123070087A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent homework correction technology, specifically to an intelligent homework correction method based on a large language model. Background Technology
[0002] With the widespread adoption of artificial intelligence technology, intelligent homework correction technology has gradually replaced traditional manual and machine-based rule-based correction methods. However, intelligent homework correction technology still has the following shortcomings in practical applications:
[0003] (1) Existing intelligent grading technologies generally process all answer sheets directly, lacking sample screening mechanisms and boundary information retention mechanisms. In the process of grading assignments, such schemes process all answer sheet samples indiscriminately, without identifying and screening the representativeness and typicality of the answer sheet samples. A large number of invalid, homogeneous, and low-reference-value answer sheet samples will occupy the model's computing resources. At the same time, key grading boundary information is easily lost in the full-processing process, which directly leads to high redundancy and weak pertinence in the generation process of scoring rules, greatly reducing the generation efficiency of scoring rules and making it difficult to quickly adapt to different batches and different quality assignment grading scenarios.
[0004] (2) Most of the current mainstream grading schemes adopt the mode of generating grading rules in a one-time fixed manner. After the initial grading rules are generated and the homework is graded, the grading rules cannot be iterated and optimized independently based on the actual grading results. The grading rules always remain in the initial state and cannot adapt to the diverse student answer situations.
[0005] A few solutions with iterative optimization capabilities rely solely on simple numerical error calculation methods to fine-tune model parameters, which can only correct superficial data deviations. They lack a semantic error evaluation system and an AI-driven optimization mechanism, making it difficult to guarantee scoring accuracy. Their simple numerical error calculation can only capture the quantitative deviation of the grading results, but cannot uncover the deep semantic reasons for scoring errors, the deviations in answer logic, and the loopholes in rule adaptation. This results in inaccurate and incomplete iterative optimization of scoring details. With long-term use, scoring errors continue to accumulate, and the overall accuracy of homework grading cannot be effectively guaranteed.
[0006] (3) Existing technologies heavily rely on manually predefined scoring rules and lack the ability to independently analyze descriptive reference answers and automatically extract core scoring points. Before grading assignments, it is necessary to manually sort out and predefine a complete set of scoring rules for different subjects and question types. Faced with the large-scale and large-volume assignment grading needs in teaching scenarios, the manual predefined rule model has a large workload and low implementation efficiency. At the same time, the manually predefined rules cannot flexibly adapt to the answer logic and scoring standards of different subjects and question types, and it is difficult to adaptively optimize the grading logic according to the needs of the scenario. This results in poor system universality and scalability, and it cannot adapt to large-scale and diversified assignment grading scenarios.
[0007] (4) Existing intelligent grading technology lacks a sound scoring consistency guarantee mechanism, and the grading process has obvious black box characteristics: Existing solutions mainly rely on the autonomous generation capability of large language models to complete the grading and scoring. The entire scoring reasoning process has no standardized constraints and no consistency verification mechanism. The generation of the model is highly random. For homogeneous answers, it is easy to have inconsistent scoring standards before and after and large fluctuations in scoring deviation. Its scoring consistency and stability are difficult to control. At the same time, the interpretability of the entire grading reasoning process is poor, forming a typical technical black box. It is impossible to trace the generation link of scoring deviation and to optimize the grading logic in a targeted manner, which reduces the credibility and practicality of intelligent homework grading results.
[0008] In summary, current intelligent homework correction technologies generally suffer from numerous technical shortcomings, such as low efficiency in generating scoring details, weak iterative optimization capabilities, insufficient scoring accuracy, high reliance on manual intervention, poor versatility, lack of guaranteed scoring consistency, and poor process interpretability. These shortcomings make it difficult to meet the current needs for large-scale, high-precision, intelligent, and universal homework correction. Summary of the Invention
[0009] To overcome the shortcomings and deficiencies of existing technologies, this invention provides an intelligent homework grading method based on a large language model. This invention selects representative answer sheets and boundary sample pairs from multiple dimensions, including semantics, word count, and score. It automatically extracts scoring points from unstructured reference answers and generates scoring rules by combining optimized prompts. Then, it performs semantic error analysis using a large language model to uncover the root causes of errors and iteratively updates the rules, continuously improving scoring accuracy. Scoring strictly adheres to the scoring rules, and the semantic understanding capabilities of the large model uniformly distinguish compliant answers with different expressions. This method not only improves the efficiency and differentiation of rule generation by relying on highly representative samples, but also enhances iteration efficiency and scoring accuracy through semantic error optimization. Simultaneously, it eliminates the scoring black box, ensuring scoring consistency. It can efficiently process large batches of answer sheets and is adaptable to scoring scenarios for various subjects and question types.
[0010] To achieve the above objectives, the present invention adopts the following technical solution:
[0011] This invention provides an intelligent homework grading method based on a large language model, comprising the following steps:
[0012] Receive question descriptions, reference answers, and answer sheets;
[0013] Construct a bag-of-words vector for each answer sheet, cluster the bag-of-words vectors of all answer sheets, calculate the text feature values of each answer sheet, and select representative samples of text features.
[0014] Construct a semantic vector for each answer sheet, cluster the semantic vectors of all answer sheets, calculate the semantic feature value of each answer sheet, and select representative samples of semantic features.
[0015] Divide the answer sheets into multiple word count intervals based on the word count distribution, calculate the probability density function value of the word count of each answer sheet under the normal distribution of the corresponding word count interval, and select representative samples with word count characteristics.
[0016] Divide the answer sheets into multiple score intervals based on the score distribution, calculate the probability density function value of each answer sheet score under the normal distribution of the corresponding score interval, and select representative samples of score characteristics.
[0017] Based on the feature values, representative samples for each feature are selected cyclically to construct a representative questionnaire set;
[0018] Boundary sample pairs are selected based on preset comprehensive similarity thresholds and score difference thresholds;
[0019] Based on the information of boundary sample pairs, prompt words for generating scoring rules are constructed. Scoring rules are generated based on a large language model. Scoring rules are optimized based on the difference between the scoring results and the standard scores to obtain the final scoring rules.
[0020] The system retrieves student answer sheets to be graded, constructs grading prompts based on the final grading criteria, performs grading based on a large language model, generates grading results based on the grading criteria and student answer sheets, parses and verifies the grading results, and obtains the homework grading results.
[0021] As a preferred technical solution, a bag-of-words vector is constructed for each answer sheet, the bag-of-words vectors of all answer sheets are clustered, the text feature values of each answer sheet are calculated, and representative samples of the text features are selected, specifically including:
[0022] Cluster the bag-of-words vectors of all answer sheets, treat each answer sheet as an independent cluster, calculate the cosine similarity between two clusters, and merge the clusters when the cosine similarity is greater than the preset similarity threshold to obtain the number of categories;
[0023] Calculate the average IoU index between each answer sheet and other answer sheets in the same category as its textual feature value;
[0024] For each category, prompt words are constructed to assess the representativeness of the samples, and a representativeness score is obtained through a large language model;
[0025] The fused text feature values are calculated by weighting the text feature values and representative scores.
[0026] All answer sheets are sorted based on the fused text feature values, and representative samples of text features are selected.
[0027] As a preferred technical solution, the average IoU index between each answer sheet and other answer sheets within the same category is calculated as its textual feature value, expressed as:
[0028] ;
[0029] ;
[0030] in, and Answer sheets and answer sheets a set of words, Indicates the number of elements in the set. Indicates the first Categories Represents the character feature value. This represents the average IoU index;
[0031] The fused text feature value is calculated based on the weighted average of the text feature value and the representativeness score, and is expressed as follows:
[0032] ;
[0033] in, and These are the weighting coefficients. Indicates representative rating. This represents the feature values of the merged text.
[0034] As a preferred technical solution, a semantic vector for each answer sheet is constructed, the semantic vectors of all answer sheets are clustered, the semantic feature value of each answer sheet is calculated, and representative samples of semantic features are selected, specifically including:
[0035] Cluster the semantic vectors of all answer sheets, treat each answer sheet as an independent cluster, calculate the semantic vector similarity between two clusters, and merge the clusters when the semantic vector similarity is greater than the preset similarity threshold to obtain the number of categories;
[0036] Calculate the average semantic vector similarity between each questionnaire and other questionnaires within the same class as the semantic feature value;
[0037] All questionnaires are sorted based on semantic feature values, and representative samples with semantic features are selected.
[0038] As a preferred technical solution, the average semantic vector similarity between each answer sheet and other answer sheets within the same class is calculated as a semantic feature value, expressed as:
[0039] ;
[0040] ;
[0041] in, Represents semantic feature values. This represents the average semantic vector similarity. Indicates the first Categories Semantic vectors and semantic vectors The cosine similarity.
[0042] As a preferred technical solution, the probability density function value of the word count of each answer sheet under the normal distribution in the corresponding word count interval is calculated, and used as the word count feature value, expressed as:
[0043] ;
[0044] ;
[0045] ;
[0046] in, Represents the character count feature value. , Indicates the range of word count Mean and standard deviation of the number of words in the internal answer sheet. word count range The number of answer sheets within, Indicates the range of word count The i-th answer sheet This represents the number of words in the i-th answer sheet;
[0047] All answer sheets were sorted based on word count as a feature value, and representative samples based on word count as a feature value were obtained.
[0048] As a preferred technical solution, the probability density function value of each answer sheet score under the normal distribution in the corresponding score interval is calculated and used as the score feature value, expressed as:
[0049] ;
[0050] ;
[0051] ;
[0052] in, Represents the characteristic value of the fraction. , Represents a fraction interval The mean and standard deviation of the internal answer sheet scores. For the fractional interval The number of answer sheets within, Represents a fraction interval The i-th answer sheet This represents the score of the i-th answer sheet;
[0053] All answer sheets are sorted according to their score features, and representative samples based on these features are obtained.
[0054] As a preferred technical solution, representative samples for each feature are selected cyclically based on feature value sorting to construct a representative questionnaire set, specifically including:
[0055] Calculate the number of samples selected for each score interval The proportional allocation strategy is adopted, which is expressed as:
[0056] ;
[0057] in, For the fractional interval The number of answer sheets within, The total number of answer sheets This represents the preset threshold for the number of representative samples;
[0058] Obtain the set of all categories for each feature, iterate through the samples in each feature class in descending order of feature value, and select samples to add to the representative questionnaire set until the representative sample count threshold is reached.
[0059] As a preferred technical solution, boundary sample pairs are selected based on preset comprehensive similarity thresholds and score difference thresholds, specifically including:
[0060] The overall similarity of each pair of samples in the representative questionnaire set is calculated and expressed as:
[0061] ;
[0062] in, , , These represent the answer sheets. and answer sheets The overall similarity, textual similarity, and semantic similarity. and These are the weighting coefficients;
[0063] The system presets a comprehensive similarity threshold and a score difference threshold, adds the identified boundary sample pairs to the boundary sample pair list, and if the number of boundary sample pairs exceeds the preset number threshold, it selects boundary samples to add from high to low similarity.
[0064] For each boundary sample pair, record its difference characteristics.
[0065] As a preferred technical solution, the scoring rules are optimized based on the difference between the scoring results and the standard scores to obtain the final scoring rules, which specifically include:
[0066] Obtain the standard score of the representative answer sheets, score the representative answer sheets according to the current scoring rules, and obtain the predicted score;
[0067] Construct prompt words for semantic error assessment and input them into a large language model to obtain semantic error reports;
[0068] Root cause analysis prompts are constructed based on error reports, input into a large language model to obtain root cause analysis results, and an optimization priority list is generated based on the root cause analysis results, sorting the scoring details that need to be modified according to the degree of impact.
[0069] Based on the root cause analysis results, optimization suggestions are constructed and prompt words are generated, which are then input into a large language model to obtain optimization suggestions.
[0070] Based on the optimization suggestions, we construct detailed update prompt words and input them into the large language model to obtain the updated scoring details;
[0071] Record the modification history of the iteration, calculate the numerical error and determine whether the preset convergence threshold has been reached. If it is met, it is determined to be numerical convergence. Construct semantic convergence judgment prompt words and input them into the large language model. Determine whether the scoring quality meets the preset conditions. Stop the iteration when the convergence conditions or the maximum number of iterations are met, and output the final scoring details.
[0072] Compared with the prior art, the present invention has the following advantages and beneficial effects:
[0073] (1) Existing technologies process all answer sheets directly without considering the representativeness of the samples and lack a sample screening mechanism, resulting in low efficiency in generating scoring rules. However, this invention obtains representative samples with textual features, semantic features, word count features, and score features through multi-dimensional feature screening, ensuring that the samples are sufficiently representative in terms of content diversity, semantic diversity, word count representativeness, and score coverage. Furthermore, based on the diverse sample selection with score perception, a representative answer sheet set and a list of boundary sample pairs are obtained, which selects the most representative samples from a large number of answer sheets, greatly improving the efficiency of scoring rule generation. Moreover, the boundary sample pairs provide a distinguishing signal for scoring rule generation, enhancing the distinguishing ability of the scoring rules.
[0074] (2) Existing technologies use a one-time generation of scoring rules, which cannot automatically optimize the scoring rules based on the scoring results. Even if there is iterative optimization, it only uses simple numerical error calculation and parameter adjustment, which makes it difficult to guarantee the scoring accuracy. However, this invention generates a structured error report, infers the root cause of the error, generates semantic optimization suggestions and updates the scoring rules through a large language model. It achieves continuous improvement in scoring accuracy by iteratively optimizing the scoring rules through semantic error assessment. Semantic error assessment uses text description to replace numerical error, which is more efficient than pure numerical error assessment and can capture the deep cause of the error, making the error analysis more accurate.
[0075] (3) Existing technologies rely on the manual pre-definition of scoring rules, while the present invention automatically generates scoring rules based on prompt word optimization and boundary sample information. It automatically extracts scoring points from unstructured reference answers, identifies the semantics of reference answers and generates boundary case processing rules, thereby reducing the cost of manual annotation and improving the efficiency of scoring rule generation.
[0076] (4) Existing technologies mainly rely on the model’s generation capabilities and lack a mechanism to ensure scoring consistency. The scoring process has a black box characteristic. However, this invention, while strictly following the scoring rules, utilizes the semantic understanding capabilities of a large language model to flexibly handle answers with diverse expressions but correct semantics, achieving accurate and consistent scoring. It can efficiently process large-scale data and does not depend on specific question types or subjects, thus adapting to different scoring needs. Attached Figure Description
[0077] Figure 1 This is a flowchart illustrating the intelligent homework grading method based on a large language model according to the present invention. Detailed Implementation
[0078] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0079] Example
[0080] like Figure 1 As shown, this embodiment provides an intelligent homework grading method based on a large language model, including the following steps:
[0081] S1: Receives the question description, reference answers, and a batch of imported student answer sheets from the teacher user. , The total number of answers is [number]. The question description includes information such as the question stem, question type, and score. The reference answer can be a standard answer text or a descriptive guide.
[0082] S2: Select representative samples based on text features, specifically including:
[0083] S21: For each answer sheet Use the TF-IDF method to construct bag-of-words vectors ;
[0084] word The formula for calculating the TFIDF weights is:
[0085] ;
[0086] ;
[0087] in, Indicator In answering the questions word frequency in Indicator Inverse document frequency;
[0088] According to the words The TFIDF weights are used to form the numerical vector of each answer sheet, resulting in the bag-of-words vector;
[0089] S22: Use a hierarchical clustering algorithm (such as AGNES) to cluster the bag-of-words vectors of all questionnaires. Hierarchical clustering adopts a bottom-up aggregation strategy. Initially, each questionnaire is an independent cluster, and then the clusters with the highest similarity are gradually merged.
[0090] S23: By setting a similarity threshold The number of categories is adaptively determined. Merging stops when the similarity between two clusters is less than a threshold. The inter-cluster similarity is calculated using the average linking method.
[0091] ;
[0092] in, and There are two clusters. For answering questions and answer sheets Cosine similarity;
[0093] S24: For each category The text-based IoU (Interchange of Union) between any two responses within a class is calculated and represented as follows:
[0094] ;
[0095] in, and Answer sheets and a set of words, This indicates the number of elements in the set.
[0096] S25: For each answer sheet Calculate its average IoU with other responses within the same category as its textual feature value:
[0097] ;
[0098] in, Represents the character feature value. This represents the average IoU index;
[0099] In this embodiment, if there is an extreme case where there is only one answer sheet, the following can be performed:
[0100] Single-point clusters are filtered out, and clusters with a size of 1 are directly discarded and not included in the text feature calculation;
[0101] Define default values for individual point clusters, for example, let This means that there are no other samples in the cluster and the average similarity with the same type is 0.
[0102] After clustering, small clusters are merged by setting a minimum cluster size threshold and merging clusters with fewer than 2 samples into other clusters.
[0103] S26: For each category Construct cue words to assess the representativeness of the sample;
[0104] The following are examples of prompts used in this embodiment to assess the representativeness of the sample:
[0105] "You are a seasoned education assessment expert. Please evaluate the representativeness of the following questionnaire sample for generating the scoring criteria:"
[0106] Title: {{Title Content}};
[0107] Suggested answer: {{Suggested answer}};
[0108] Please conduct a representativeness assessment for each sample, including the following dimensions:
[0109] Completeness of content: Does the answer sheet cover the main points tested in the questions?
[0110] Diversity of expression: Does the written expression in the answer sheet possess uniqueness?
[0111] Scoring value: The value of the answer sheet in differentiating students from different score ranges;
[0112] Please output the evaluation results in JSON format:
[0113] "evaluations":
[0114] "sample_id": "Answer sheet ID";
[0115] "representativeness_score": A representative score of 1-10;
[0116] "content_completeness": "Content completeness assessment (high / medium / low)";
[0117] "expression_diversity": "Assessment of expression diversity (high / medium / low)";
[0118] "grading_value": "Grading value assessment (high / medium / low)";
[0119] "reason": "Evaluation rationale (one-sentence description)";
[0120] "category_summary": "Overall feature description of samples in this category";
[0121] S27: Input the constructed prompt words into the large language model to obtain a representative score for each sample. ;
[0122] S28: For each answer sheet Calculate the feature values of the merged text:
[0123] ;
[0124] in, and Let be the weighting coefficient, satisfying Preferably, set , Give AI evaluation a higher weight;
[0125] S29: For each category All answer sheets within this category are analyzed according to the merged text feature values. Sort the samples from highest to lowest to obtain representative samples of text features;
[0126] S3: Based on vectors from a pre-trained model (such as BGE), hierarchical clustering is performed on the questionnaires. The number of categories is adaptively obtained through a similarity threshold. Vector similarity is calculated within each category. The semantic feature value of each sample is the mean cosine similarity between it and other sample vectors in the same category. Specifically, this includes:
[0127] S31: For each answer sheet Using pre-trained language models (such as BGE), semantic vectors that can capture the deep semantic information of the answers are extracted. ;
[0128] S32: Use a hierarchical clustering algorithm to cluster the semantic vectors of all responses, adopting the same bottom-up aggregation strategy as the text features;
[0129] S33: By setting a similarity threshold The number of categories is adaptively determined, and merging stops when the similarity between two clusters is less than a threshold.
[0130] S34: For each category Calculate the semantic vector similarity between any two answer sheets within the same class, using cosine similarity:
[0131] ;
[0132] S35: For each answer sheet The average similarity between the answer sheet and other answer sheets within the same class is calculated as its semantic feature value:
[0133] ;
[0134] in, Represents semantic feature values. Indicates average similarity;
[0135] S36: For each category All responses within this category are sorted according to their semantic feature values. Sort the samples from highest to lowest to obtain representative samples of semantic features;
[0136] S4: Divide the answer sheet into word count intervals as classes, calculate the word count of each answer sheet, and each answer sheet will fall into a word count interval. For samples within each interval class, calculate the probability density function value of the normal distribution within the interval. The word count feature value of each sample is the probability density function value of the normal distribution, specifically including:
[0137] S41: For each answer sheet Calculate its word count ;
[0138] S42: Divide the answer sheet into sections based on the word count distribution. The word count range can be divided using either equal intervals or adaptive intervals. The range for equal intervals is as follows:
[0139] ;
[0140] in, , The interval width, and These are the minimum and maximum character counts, respectively.
[0141] S43: For each answer sheet According to its word count Assign it to the corresponding word count range ;
[0142] S44: For each word count range Calculate the average number of words in the answer sheet within this interval. and standard deviation :
[0143] ;
[0144] ;
[0145] in, word count range The number of answer sheets within;
[0146] S45: For each answer sheet Calculate the probability density function value of its word count under the normal distribution in this interval as its word count feature value:
[0147] ;
[0148] S46: For each word count range All answer sheets within this range are sorted by word count. Sort the samples from highest to lowest word count to obtain representative samples;
[0149] S5: Divide the score intervals into classes. Each answer sheet will fall into a score interval. For samples within each interval class, calculate the probability density function value of the normal distribution within that interval. The score feature value of each sample is the probability density function value of the normal distribution, specifically including:
[0150] S51: Based on the score distribution of the answer sheets, divide the answer sheets into... There are several score intervals, which can be divided using equal intervals. Let's assume the maximum score is... The range of the equally spaced intervals is:
[0151] ;
[0152] in, ;
[0153] S52: For each answer sheet According to its score Assign it to the corresponding score range ;
[0154] S53: For each score interval Calculate the mean score of the answer sheet within this interval. and standard deviation :
[0155] ;
[0156] ;
[0157] in, For the fractional interval The number of answer sheets within;
[0158] S54: For each answer sheet Calculate the probability density function value of its score under the normal distribution in this interval as its score characteristic value:
[0159] ;
[0160] S55: For each score interval All answer sheets within this interval are sorted according to their score characteristics. Sort the samples from highest to lowest to obtain representative samples based on their score features;
[0161] S6: Set the number of representative samples for the output. This refers to the number of samples in the final representative questionnaire set, while also setting the number of boundary sample pairs. ;
[0162] S7: Employ a score-aware, diversified sample selection strategy to ensure representative samples for each score range, while retaining similar but different sample pairs, specifically including:
[0163] S71: Construct an empty representative questionnaire set and the selected sample set ;
[0164] S72: Calculate the number of samples to be selected for each score interval, using a proportional allocation strategy:
[0165] ;
[0166] in, For the fractional interval The number of answer sheets within, This represents the total number of answer sheets.
[0167] S73: Cyclicly select each feature type , Specifically:
[0168] Get the set of all categories for this feature , This represents the number of categories for this feature;
[0169] For each category The samples within a category are traversed in descending order of feature values. This embodiment sets a maximum selection limit for each category, preventing any single category from filling the entire sample set. Selection is performed in a round-robin fashion, taking turns selecting from each category. , , , Instead of iterating through all samples of a single feature type at once, one sample is selected from each feature type.
[0170] If the current sample Not in the selected sample set If the sample is found to be in the middle, then add it to the representative questionnaire set. and the selected sample set middle;
[0171] If representative answer sheet collection The number of samples has reached If so, the selection process will stop;
[0172] S74: For representative answer sheets Each pair of samples Calculate their overall similarity:
[0173] ;
[0174] in, and As a weighting coefficient, preferably, set to , ;
[0175] S75: Find sample pairs that satisfy the following conditions :
[0176] High similarity: Overall similarity ;
[0177] Large score difference: score difference ;
[0178] in, This represents the overall similarity threshold. This indicates the score difference threshold; the score used in the current step is the score from manual grading.
[0179] These sample pairs are boundary sample pairs, representing the boundary situations that the scoring rules need to distinguish in detail;
[0180] S76: Add the identified boundary sample pairs to the boundary sample pair list. In the middle, if the number of boundary sample pairs exceeds a preset threshold... Then select the first one from the highest to the lowest similarity. right;
[0181] S77: For each boundary sample pair, record its difference features, including textual differences, semantic differences, score differences, and speculation on the reasons for the differences;
[0182] S78: Output the final representative set of responses. and boundary sample pair list This will be used for subsequent generation and iterative optimization of scoring rules;
[0183] S8: Construct prompt words for generating scoring details, incorporate boundary sample pair information into the prompt words, input the constructed prompt words into the large language model, call the model's generation capability to generate scoring details, parse the scoring details generated by the large language model, convert them into executable data structures, verify whether the generated scoring details are complete, if the verification fails, reconstruct the prompt words and call the large language model to generate new scoring details, and finally output the verified scoring details for subsequent iterative optimization and batch execution.
[0184] The following is an example of the prompt words used to generate scoring details in this embodiment:
[0185] "The following are boundary cases that need to be distinguished:"
[0186] Boundary sample pair 1:
[0187] Sample A (Score: 18 points): {{Answer Sheet Content}};
[0188] Sample B (Score: 12 points): {{Answer Sheet Content}};
[0189] Similarity: 0.85 (high similarity);
[0190] Score difference: 6 points;
[0191] Difference characteristics: {{Difference characteristics analyzed by AI}};
[0192] Speculated reasons for the differences: {{AI-specified reasons for the score differences}};
[0193] Based on the above boundary cases, please clearly define the distinguishing criteria in the scoring rules to ensure that similar situations receive correct scores.
[0194] In this embodiment, the scoring rules adopt a structured format, including scoring dimensions, scoring points, score weights, deduction criteria, and rules for handling boundary cases. Correspondingly, the completeness of the generated scoring rules is verified, that is, whether the scoring dimensions cover all aspects of the question requirements, whether the scoring points are clear and specific, whether the score weights are reasonable, whether the deduction criteria are clear, and whether the rules for handling boundary cases are complete.
[0195] S9: Through semantic error assessment and closed-loop feedback mechanisms, based on the difference between the scoring results and the preset standard scores, the causes of errors are analyzed and the scoring rules are automatically optimized to achieve continuous improvement in scoring accuracy. Specifically, this includes:
[0196] S91: Obtaining Standard Scores for Representative Answer Sheets According to the current scoring rules Representative responses were scored to obtain predicted scores. ;
[0197] S92: Construct prompt words for semantic error assessment, input the constructed prompt words into the large language model, obtain semantic error reports, organize the error reports output by the large language model, and generate structured error report documents;
[0198] The following are examples of prompt words used for semantic error evaluation in this embodiment:
[0199] "You are a seasoned education assessment expert. Please analyze the discrepancy between the following rating results and the standard scores."
[0200] Title: {{Title Content}};
[0201] Scoring details: {{Current scoring details}};
[0202] Answer sheet grading results:
[0203] The scoring details for each questionnaire include: questionnaire ID, questionnaire content, standard score, predicted score, and scores for each dimension.
[0204] Please perform an error analysis on each questionnaire and output an error report in JSON format:
[0205] "answer_id": "Answer sheet ID";
[0206] "standard_score": Standard score;
[0207] "predicted_score": Predicted score;
[0208] "error_magnitude": Error magnitude (high / medium / low);
[0209] "error_description": "A detailed textual description of the error, including: the difference between the predicted score and the standard score, which dimensions are biased, the specific manifestations of the bias, and the possible causes of the bias";
[0210] "affected_rules": ["Affected scoring rules"];
[0211] "suggested_fix": "Specific modification suggestions";
[0212] "overall_analysis":
[0213] "common_errors": ["Common error types"];
[0214] "root_causes": ["The root cause of the error"];
[0215] "priority_fixes": ["priority fixes"];
[0216] In this embodiment, the error report includes a single-response error description, a multi-level error magnitude assessment, and 3. Identification of affected rules and modification suggestions;
[0217] S93: Construct root cause analysis prompts, input the constructed prompts into the large language model to obtain in-depth root cause analysis results, generate an optimization priority list based on the root cause analysis results, sort the scoring details that need to be modified according to the degree of impact, and generate specific and targeted semantic optimization suggestions based on error analysis and root cause analysis results.
[0218] The following is an example of root cause analysis prompts in this embodiment:
[0219] "You are a seasoned education assessment expert. Based on the following error report, please conduct an in-depth analysis of the root causes of the errors."
[0220] Error report: {{generated error report}};
[0221] Scoring details: {{Current scoring details}};
[0222] Please analyze:
[0223] 1. Which clauses in the scoring criteria led to systematic error?
[0224] 2. What are the problems with these clauses? (Too strict / Too lenient / Vague wording / Incomplete coverage)
[0225] 3. What are the relationships between the errors?
[0226] 4. Which clauses should be modified to minimize errors?
[0227] Please output the analysis results in JSON format:
[0228] "root_cause_analysis":
[0229] "rule_id": "Rating details clause ID";
[0230] "rule_content": "Terms and conditions";
[0231] "problem_type": "Problem type (too strict / too lenient / vague / incomplete)";
[0232] "problem_description": "Detailed description of the problem";
[0233] "affected_answers": ["Affected answer IDs"];
[0234] "impact_score": Impact score (1-10);
[0235] "fix_priority": Fix priority (high / medium / low);
[0236] "correlation_analysis":
[0237] "correlated_errors": ["correlated error types"];
[0238] "cascade_effects": ["cascade effects"];
[0239] "optimization_strategy":
[0240] "high_priority_fixes": ["High-priority fixes"];
[0241] "expected_improvement": "Expected improvement effect";
[0242] S94: Build optimization suggestions and generate prompt words. Input the built prompt words into the large language model to obtain specific optimization suggestions;
[0243] The following is an example of the optimized suggestion prompt words generated in this embodiment:
[0244] "You are a seasoned education assessment expert. Based on the following analysis results, please generate optimization suggestions for the scoring criteria."
[0245] Current scoring rules: {{current scoring rules}};
[0246] Root cause analysis results: {{Root cause analysis results of step S83}};
[0247] Boundary sample pairs: {{boundary sample pair information}};
[0248] For each clause that needs modification, please generate specific optimization suggestions:
[0249] "optimization_suggestions":
[0250] "target_rule_id": "Target rule ID";
[0251] "current_content": "Current content";
[0252] "suggested_content": "Suggested revised content";
[0253] "modification_type": "Modification type (adjust score / modify description / add clause / delete clause)";
[0254] "reason": "the reason for the modification";
[0255] "expected_effect": "expected effect";
[0256] "affected_answers": ["the IDs of the answers that will benefit"];
[0257] "new_rules_suggestions":
[0258] "suggested_rule": "Suggested new clauses";
[0259] "reason": "Added reason";
[0260] "placement": "Suggested placement location";
[0261] S95: Construct detailed rules and update prompt words. Input the constructed prompt words into the large language model to obtain the updated scoring rules.
[0262] The following is an example of the detailed update prompt words in this embodiment:
[0263] "You are a seasoned education assessment expert. Please update the scoring details based on the following optimization suggestions."
[0264] Current scoring rules: {{current scoring rules}};
[0265] Optimization suggestions: {{Optimization suggestions for step S84}};
[0266] Require:
[0267] 1. Maintain structural consistency in the scoring criteria;
[0268] 2. Adopt high-priority optimization suggestions;
[0269] 3. Ensure the revised rules are clearer and more actionable;
[0270] 4. Ensure the sum of the scores for each dimension equals the total score;
[0271] Please provide the updated and complete scoring details:
[0272] {{Updated scoring rules}}”;
[0273] S96: Record the modification history of the iteration, calculate the numerical error and determine whether the preset convergence threshold has been reached. If it is met, it is determined to be numerical convergence. Construct semantic convergence judgment prompt words and input them into the large language model. Determine whether the scoring quality meets the requirements. Stop the iteration when the convergence condition or the maximum number of iterations is met, and output the final scoring details.
[0274] In this embodiment, the modification history of the iteration is recorded, including the modified terms, the modified content, the reasons for the modification, and the expected effects;
[0275] The semantic convergence judgment prompts in this embodiment are shown below:
[0276] "You are a seasoned education assessment expert. Please determine whether the current scoring criteria have reached an acceptable level of quality."
[0277] Error Report: {{Latest Error Report}}
[0278] Scoring details: {{Current scoring details}}
[0279] Please determine:
[0280] 1. Is the error within an acceptable range?
[0281] 2. Are the scoring criteria clear and easy to follow?
[0282] 3. Are there still any significant systematic biases?
[0283] Please output the judgment result in JSON format:
[0284] "is_converged": true / false;
[0285] "quality_score": Quality score (1-10);
[0286] "remaining_issues": ["Remaining issues"];
[0287] "recommendation": "Suggestion (continue iteration / stop iteration)";
[0288] S10: Obtain student answer sheets to be graded, construct grading prompts based on the final grading rules, input the constructed grading prompts into the large language model, call the model's generation capabilities for grading, the large language model generates grading results based on the grading rules and student answer sheets, parse the grading results generated by the large language model, convert them into structured data format, and verify the rationality of the grading results. If the verification fails, reconstruct the grading prompts and call the large language model for grading, and finally output the verified grading results.
[0289] In this embodiment, the correction prompt includes the following elements:
[0290] (1) Role definition: The instruction model plays the role of a senior scoring expert;
[0291] (2) Task description: The model is required to score students' answers according to the given scoring criteria;
[0292] (3) Scoring details: including scoring dimensions, description of scoring points, weight of scores, deduction criteria, rules for handling boundary cases, etc.;
[0293] (4) Student answer sheets: The contents of student answer sheets awaiting grading;
[0294] (5) Output format specifications: The model is required to output in a specific structure, including the score of each dimension, the score of each score point, the deduction items, the scoring basis, and the summary comments.
[0295] In this embodiment, the scoring results include: the score for each dimension, the score for each scoring point, the deduction items and reasons for deduction, detailed scoring basis, and summary comments; correspondingly, the reasonableness of the scoring results is verified, including: whether the sum of the scores for each dimension equals the total score, whether the scores for each scoring point are within a reasonable range, whether the deduction items are reasonable, and whether the scoring basis is clear.
[0296] This invention selects representative answer sheets and boundary sample pairs from multiple dimensions such as semantics, word count, and score. It automatically extracts scoring points from unstructured reference answers and generates scoring rules by combining optimized prompt words. Then, it uses a large language model to perform semantic error analysis, dig out the root causes of errors, and iteratively updates the rules to continuously improve scoring accuracy. Scoring strictly adheres to the scoring rules and uses the semantic understanding capabilities of the large model to uniformly distinguish compliant answers with different expressions. It can improve the efficiency of rule generation and enhance the distinction of rules by relying on highly representative samples, and improve the iteration efficiency and scoring accuracy by optimizing semantic errors. At the same time, it eliminates the scoring black box, ensures scoring consistency, and can efficiently process a large number of answer sheets, adapting to scoring scenarios of various subjects and question types.
[0297] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.
Claims
1. An intelligent homework grading method based on a large language model, characterized in that, Includes the following steps: Receive question descriptions, reference answers, and answer sheets; Construct a bag-of-words vector for each answer sheet, cluster the bag-of-words vectors of all answer sheets, calculate the text feature values of each answer sheet, and select representative samples of text features. Construct a semantic vector for each answer sheet, cluster the semantic vectors of all answer sheets, calculate the semantic feature value of each answer sheet, and select representative samples of semantic features. Divide the answer sheets into multiple word count intervals based on the word count distribution, calculate the probability density function value of the word count of each answer sheet under the normal distribution of the corresponding word count interval, and select representative samples with word count characteristics. Divide the answer sheets into multiple score intervals based on the score distribution, calculate the probability density function value of each answer sheet score under the normal distribution of the corresponding score interval, and select representative samples of score features; Based on the feature values, representative samples for each feature are selected cyclically to construct a representative questionnaire set; Boundary sample pairs are selected based on preset comprehensive similarity thresholds and score difference thresholds; Based on the information of boundary sample pairs, prompt words for generating scoring rules are constructed. Scoring rules are generated based on a large language model. Scoring rules are optimized based on the difference between the scoring results and the standard scores to obtain the final scoring rules. The system retrieves student answer sheets to be graded, constructs grading prompts based on the final grading criteria, performs grading based on a large language model, generates grading results based on the grading criteria and student answer sheets, parses and verifies the grading results, and obtains the homework grading results.
2. The intelligent homework grading method based on a large language model according to claim 1, characterized in that, Construct a bag-of-words vector for each answer sheet, cluster the bag-of-words vectors of all answer sheets, calculate the text feature values for each answer sheet, and select representative samples based on the text features. Specifically, this includes: Cluster the bag-of-words vectors of all answer sheets, treat each answer sheet as an independent cluster, calculate the cosine similarity between two clusters, and merge the clusters when the cosine similarity is greater than the preset similarity threshold to obtain the number of categories; Calculate the average IoU index between each answer sheet and other answer sheets in the same category as its textual feature value; For each category, prompt words are constructed to assess the representativeness of the samples, and a representativeness score is obtained through a large language model; The fused text feature values are calculated by weighting the text feature values and representative scores. All answer sheets are sorted based on the fused text feature values, and representative samples of text features are selected.
3. The intelligent homework grading method based on a large language model according to claim 2, characterized in that, Calculate the average IoU (Interaction-Oriented Union) between each answer sheet and other answer sheets within the same category as its textual feature value, expressed as: ; ; in, and Answer sheets and answer sheets a set of words, Indicates the number of elements in the set. Indicates the first Categories Represents the character feature value. This represents the average IoU index; The fused text feature value is calculated based on the weighted average of the text feature value and the representativeness score, and is expressed as follows: ; in, and These are the weighting coefficients. Indicates representative rating. This represents the feature values of the merged text.
4. The intelligent homework grading method based on a large language model according to claim 1, characterized in that, Construct a semantic vector for each questionnaire, cluster the semantic vectors of all questionnaires, calculate the semantic feature value for each questionnaire, and select representative samples of semantic features, specifically including: Cluster the semantic vectors of all answer sheets, treat each answer sheet as an independent cluster, calculate the semantic vector similarity between two clusters, and merge the clusters when the semantic vector similarity is greater than the preset similarity threshold to obtain the number of categories; Calculate the average semantic vector similarity between each questionnaire and other questionnaires within the same class as the semantic feature value; All questionnaires are sorted based on semantic feature values, and representative samples with semantic features are selected.
5. The intelligent homework grading method based on a large language model according to claim 1, characterized in that, The average semantic vector similarity between each questionnaire and other questionnaires within the same class is calculated as a semantic feature value, expressed as: ; ; in, Represents semantic feature values. This represents the average semantic vector similarity. Indicates the first Categories Represents semantic vectors and semantic vectors The cosine similarity.
6. The intelligent homework grading method based on a large language model according to claim 1, characterized in that, Calculate the probability density function value of the word count for each answer sheet under the normal distribution within the corresponding word count interval, and use it as the word count feature value, expressed as: ; ; ; in, Represents the character count feature value. , Indicates the range of word count Mean and standard deviation of the number of words in the internal answer sheet. word count range The number of answer sheets within, Indicates the range of word count The i-th answer sheet, This represents the number of words in the i-th answer sheet; All answer sheets were sorted based on word count as a feature value, and representative samples based on word count as a feature value were obtained.
7. The intelligent homework grading method based on a large language model according to claim 1, characterized in that, Calculate the probability density function value of each answer sheet score under the normal distribution in the corresponding score interval, and use it as the score feature value, expressed as: ; ; ; in, Represents the characteristic value of the fraction. , Represents a fraction interval The mean and standard deviation of the internal answer sheet scores. For the fractional interval The number of answer sheets within, Represents a fraction interval The i-th answer sheet, This represents the score of the i-th answer sheet; All answer sheets are sorted according to their score features, and representative samples based on these features are obtained.
8. The intelligent homework grading method based on a large language model according to claim 1, characterized in that, Based on the feature values, representative samples for each feature are selected cyclically to construct a representative questionnaire set, specifically including: Calculate the number of samples selected for each score interval The proportional allocation strategy is adopted, which is expressed as: ; in, For the fractional interval The number of answer sheets within, The total number of answer sheets This represents the preset threshold for the number of representative samples; Obtain the set of all categories for each feature, iterate through the samples in each feature class in descending order of feature value, and select samples to add to the representative questionnaire set until the representative sample count threshold is reached.
9. The intelligent homework grading method based on a large language model according to claim 8, characterized in that, Boundary sample pairs are selected based on preset comprehensive similarity thresholds and score difference thresholds, specifically including: The overall similarity of each pair of samples in the representative questionnaire set is calculated and expressed as: ; in, , , These represent the answer sheets. and answer sheets The overall similarity, textual similarity, and semantic similarity. and These are the weighting coefficients; The system presets a comprehensive similarity threshold and a score difference threshold, adds the identified boundary sample pairs to the boundary sample pair list, and if the number of boundary sample pairs exceeds the preset number threshold, it selects boundary samples to add from high to low similarity. For each boundary sample pair, record its difference characteristics.
10. The intelligent homework grading method based on a large language model according to claim 1, characterized in that, The scoring criteria were refined based on the difference between the scoring results and the standard scores, resulting in the final scoring criteria, which include: Obtain the standard score of the representative answer sheets, score the representative answer sheets according to the current scoring rules, and obtain the predicted score; Construct prompt words for semantic error assessment and input them into a large language model to obtain semantic error reports; Root cause analysis prompts are constructed based on error reports, input into a large language model to obtain root cause analysis results, and an optimization priority list is generated based on the root cause analysis results, sorting the scoring details that need to be modified according to the degree of impact. Based on the root cause analysis results, optimization suggestions are constructed and prompt words are generated, which are then input into a large language model to obtain optimization suggestions. Based on the optimization suggestions, we construct detailed update prompt words and input them into the large language model to obtain the updated scoring details; Record the modification history of the iteration, calculate the numerical error and determine whether the preset convergence threshold has been reached. If it is met, it is determined to be numerical convergence. Construct semantic convergence judgment prompt words and input them into the large language model. Determine whether the scoring quality meets the preset conditions. Stop the iteration when the convergence conditions or the maximum number of iterations are met, and output the final scoring details.
