Automatic scoring method and system based on comparative error sample mining and model fine-tuning

Abstract

The present application belongs to the technical field of natural language processing and intelligent education, and particularly relates to an automatic scoring method and system based on comparative error sample mining and model fine-tuning. The method comprises: constructing a full-tuning model and a Few-shot model to score the to-be-scored answer sample data, obtaining a first scoring result and a second scoring result; based on the first scoring result and the second scoring result, obtaining an inconsistent score and a full-tuning model scoring deviation; based on the inconsistent score and the full-tuning model scoring deviation, combining a dynamic threshold mechanism to construct an error-prone sample set; based on the error-prone sample set, using LoRA technology to correct the full-tuning model to obtain a corrected full-tuning model; and based on the corrected full-tuning model, obtaining a final scoring result of the to-be-scored answer sample data. The present application proposes a comparative error sample mining and fine-tuning framework, which significantly improves the robustness and generalization ability of the model in difficult scenarios.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing and intelligent education technology, specifically involving an automatic scoring method and system based on comparative error sample mining and model fine-tuning. Background Technology

[0002] With the rapid development of artificial intelligence technology, especially the widespread application of Natural Language Processing (NLP) and Large Scale Language Models (LLM), automated scoring systems are playing an increasingly important role in education. These systems not only significantly reduce the workload of teachers in grading subjective questions, but also improve teaching efficiency and student learning motivation through rapid and accurate feedback mechanisms. In recent years, the powerful language understanding and generation capabilities of large models such as GPT-4, QWen, and LLama have shown great promise in open-ended answer scoring tasks. More and more research is dedicated to applying large models to the automatic evaluation of student answers. For example, Zhan et al. (2024) proposed using GPT-4 with carefully designed prompt word templates for essay scoring, pioneering a new paradigm of prompt word engineering-driven scoring. LI et al. (2024) added scoring criteria to the prompt words, allowing large models to score more accurately. Sun et al. (2024), Quan et al. (2024), and LI et al. (2025) used fine-tuning techniques to fine-tune large models to enable scoring experts to score short answers.

[0003] However, despite the initial success of large-scale models on multiple educational scoring benchmarks, they still face three key challenges in practical scoring. First, large-scale model scoring results are prone to inconsistencies and biases when faced with complex, ambiguous, or novel responses, limiting their reliability and fairness. Second, currently popular few-shot cue learning methods are unstable when dealing with long-tailed distributions and highly semantically complex samples, easily influenced by cue word construction and example selection. Finally, existing scoring methods often treat all training samples equally, failing to pay attention to "high-value error-prone samples" (e.g., samples with ambiguous right-or-wrong boundaries and strong model bias) that have a crucial influence on model behavior, resulting in the retention of systematic biases even after fine-tuning. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention provides an automatic scoring method and system based on comparative error sample mining and model fine-tuning, aiming to solve the aforementioned problems existing in the prior art.

[0005] To achieve the above objectives, the present invention provides the following solution:

[0006] An automatic scoring method based on comparative error sample mining and model fine-tuning, the method comprising:

[0007] We construct a full-scale fine-tuning model and a Few-shot model to score the response sample data to be scored, and obtain the first scoring result and the second scoring result;

[0008] Based on the first and second scoring results, the inconsistent scoring and the full-scale fine-tuning model scoring bias are obtained;

[0009] Based on the scoring bias of inconsistent scoring and full-scale fine-tuning model scoring, and combined with a dynamic threshold mechanism, a set of error-prone samples is constructed.

[0010] Based on the error-prone sample set, the LoRA technique is used to correct the full-scale fine-tuning model, resulting in the corrected full-scale fine-tuning model.

[0011] Based on the revised full-scale fine-tuning model, the final scoring results of the answer sample data to be scored are obtained.

[0012] Preferably, the method for constructing a full-scale fine-tuning model and a Few-shot model to score the response sample data to obtain the first and second scoring results includes:

[0013] Full-scale fine-tuning model predictions: ;

[0014] Few-shot model predictions: ;

[0015] in, For full-scale fine-tuning model Sample responses to scoring The first in The rating result of each sample is denoted as the first rating result. For Few-shot model Sample responses to scoring The first in The rating result of each sample is denoted as the second rating result. For full-scale fine-tuning model The parameters, for K Number of prompt samples.

[0016] Preferably, the methods for obtaining inconsistent scores and full-scale fine-tuning model score bias based on the first and second scoring results include:

[0017] Inconsistency rating: ;

[0018] Full-scale fine-tuning model scoring bias: ;

[0019] in, For the first i Full-sample fine-tuning model and Few-shot model differences For full-scale fine-tuning model Deviation from the actual score For the first i Human expert ratings in a sample.

[0020] Preferred methods for constructing error-prone sample sets based on inconsistent scoring and full-scale fine-tuning model scoring bias, combined with a dynamic threshold mechanism, include:

[0021] ;

[0022] ;

[0023] ;

[0024] in, For error-prone sample sets, and The difference in scores for all samples Deviation from scores of all samples The mean, and These are their respective standard deviations; and These are all sensitivity adjustment parameters. For full-scale fine-tuning model and Few-shot model The difference screening threshold, For full-scale fine-tuning model The screening threshold for deviation from the true score.

[0025] Preferably, the method for correcting the full-scale fine-tuning model based on a set of error-prone samples and using LoRA technology to obtain the corrected full-scale fine-tuning model includes:

[0026] ;

[0027] in, To correct the weight matrix of the model before full fine-tuning, This is the weight matrix of the corrected full-scale fine-tuning model. It is a low-rank weight matrix. and All of them are trainable matrices.

[0028] The present invention also provides an automatic scoring system based on comparative error sample mining and model fine-tuning. The system is used to implement the aforementioned method and includes: a dual-model scoring module, a scoring difference acquisition module, an error-prone sample set construction module, a model correction module, and a scoring acquisition module.

[0029] The dual-model scoring module is used to construct a full-scale fine-tuning model and a Few-shot model to score the response sample data to be scored, and obtain the first scoring result and the second scoring result.

[0030] The scoring difference acquisition module is used to obtain inconsistent scores and full-scale fine-tuning model scoring bias based on the first and second scoring results.

[0031] The error-prone sample set construction module is used to construct an error-prone sample set based on the bias between inconsistent scoring and full-scale fine-tuning model scoring, combined with a dynamic threshold mechanism.

[0032] The model correction module is used to correct the full fine-tuning model based on the error-prone sample set and using LoRA technology to obtain the corrected full fine-tuning model.

[0033] The scoring acquisition module is used to obtain the final scoring results of the answer sample data to be scored based on the corrected full-scale fine-tuning model.

[0034] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the aforementioned method.

[0035] The present invention also provides a computer-readable storage medium storing a computer program that, when executed, implements the aforementioned method.

[0036] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0037] This invention proposes a Comparative Error Sample Mining and Fine-tuning Framework (CEM-FT), which automatically identifies high-value, difficult samples by analyzing the scoring discrepancy between the full-sample fine-tuning model and the few-sample model, and trains a lightweight LoRA adapter on this basis to achieve model performance correction with low resource overhead.

[0038] This invention automatically mines high-value difficult samples with large score discrepancies by comparing the scoring results of the full-scale fine-tuning model and the few-sample model, enabling targeted data reuse and model correction, and significantly improving the robustness and generalization ability of the model in difficult scenarios.

[0039] This invention reduces computational resources by integrating LoRA technology into model accuracy correction. Traditional full-scale fine-tuning requires significant computational resources and time, while this invention trains a lightweight LoRA adapter only on automatically selected key samples, avoiding repeated training of the entire model. This achieves superior performance compared to full-scale fine-tuning while maintaining low computational overhead and ease of deployment.

[0040] This invention significantly improves scoring accuracy and consistency. It achieves significant advantages on multiple standard datasets, with the quadratic weighted Kappa (QWK) index improving by up to 3.9% compared to traditional fine-tuning, and also showing a significant improvement compared to the few-shot inference baseline. It effectively reduces the scoring bias of large models on uncertain samples and improves the consistency and reliability of scoring results.

[0041] This invention offers scalable and universally applicable value. The framework can be seamlessly integrated into automated scoring systems across different disciplines and multiple languages, exhibiting excellent cross-domain adaptability and scalability. It provides a feasible technical approach for building efficient, stable, and sustainably optimized automated student answer scoring systems in real-world educational scenarios. Attached Figure Description

[0042] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a schematic diagram of the automatic scoring method based on comparative error sample mining and model fine-tuning in an embodiment of the present invention. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0046] Example 1

[0047] like Figure 1 As shown, this invention provides an automatic scoring method based on comparative error sample mining and model fine-tuning, including:

[0048] We construct a full-scale fine-tuning model and a Few-shot model to score the response sample data to be scored, and obtain the first scoring result and the second scoring result;

[0049] Based on the first and second scoring results, the inconsistent scoring and the full-scale fine-tuning model scoring bias are obtained;

[0050] Based on the scoring bias of inconsistent scoring and full-scale fine-tuning model scoring, and combined with a dynamic threshold mechanism, a set of error-prone samples is constructed.

[0051] Based on the error-prone sample set, the LoRA technique is used to correct the full-scale fine-tuning model, resulting in the corrected full-scale fine-tuning model.

[0052] Based on the revised full-scale fine-tuning model, the final scoring results of the answer sample data to be scored are obtained.

[0053] The specific implementation process of this invention is as follows:

[0054] Phase 1: Dual-model scoring

[0055] Using Qwen3-8B as an example, two scoring models are constructed: a full-scale fine-tuning model and a Few-shot model, to provide a multi-perspective answer scoring mechanism. Different models make judgments based on their learned different "perspectives" or "biases," representing the scoring difficulties. The two scoring models output the answer's score:

[0056] 1) Full-scale fine-tuning model based on Qwen3-8B .

[0057] 2) Few-shot model based on Qwen3-8B few-shot hints .

[0058] Given a set of training samples ,in = { , } represents the sample data of responses to be scored. Indicates the first i The problem in the sample Indicates the first i Student response texts from a sample For the first i Human expert ratings in a sample, n For the sample size. The fine-tuning instructions will be constructed. Input into the full fine-tuning model The following is an example of an instruction template for training:

[0059] instruction = [

[0060] {"role": "system", "content": You are a professional and fair teacher, and now you need to grade the students' answers.}

[0061] {"role": "user", "content": Question: Question content, Student answer: Answer content} ];

[0063] Full-scale fine-tuning of the trained model Used for inference. Few-shot model No training.

[0064] Bundle Input full fine-tuning model Obtain the prediction results ,Bundle Input Few-shot model Obtain the prediction results The formula is described as follows:

[0065] Full-scale fine-tuning model predictions: ;

[0066] Few-shot model predictions: ;

[0067] in, For full-scale fine-tuning model Sample data of responses to be scored The first in The rating result of each sample is denoted as the first rating result. For Few-shot model Sample responses to scoring The first in The rating result of each sample is denoted as the second rating result. For full-scale fine-tuning model The parameters, for K Number of prompt samples This means using a 3-shot prompt rating.

[0068] The scoring model of this invention can also be replaced with other large language models, such as QWen3-32B, Llama3-8B, Mistral-7B, etc.

[0069] Phase 2: Comparative Error Discovery

[0070] The prediction results provided by the dual models reflect the differences in the scoring judgments of the models under different training strategies. This invention constructs a scoring difference index to quantify the degree of divergence between the two scoring strategies at the sample level.

[0071] In practical applications, the main reason for model scores deviating from human expert scores often lies in certain difficult samples. These samples may have complex language structures, implicit reasoning logic, or fuzzy scoring criteria, leading to discrepancies in predictions from different models. To address this, this invention designs a risk sample mining mechanism based on model score differences:

[0072] Inconsistency rating: ;

[0073] in, belong This strategy is used to measure the degree of score discrepancy between two models.

[0074] Full-scale fine-tuning model scoring bias: ;

[0075] This strategy is used to quantize the full fine-tuning model. Predict the degree of deviation from expert scores.

[0076] in, For the first i Full-sample fine-tuning model and Few-shot model differences For full-scale fine-tuning model The deviation from the actual score.

[0077] Existing technologies typically select "difficult samples" based solely on the deviation between model predictions and human labels, or solely on inconsistencies between different models. Reinforcement learning or human preference alignment techniques usually require manual selection of high-quality samples. This invention, through a risk sample mining strategy, can automatically filter out risk samples to construct a difficult sample set, i.e., a set of error-prone samples.

[0078] Error-prone sample set Construction rules:

[0079] .

[0080] To reasonably determine which samples belong to the high-risk category, this invention introduces a dynamic threshold mechanism, which adaptively sets screening criteria based on the global statistical characteristics of the samples:

[0081] ;

[0082] ;

[0083] in, and The difference in scores for all samples Scoring bias across all samples The mean, and These are their respective standard deviations; and These are all sensitivity adjustment parameters, used to control the stringency of the screening. For full-scale fine-tuning model and Few-shot model The difference screening threshold, For full-scale fine-tuning model The screening threshold for deviation from the true score.

[0084] Phase 3: Model Fine-tuning

[0085] This invention introduces the LoRA (Low-Rank Adaptation) technique, using a set of error-prone samples. Lightweight LoRA fine-tuning is performed on the full-scale fine-tuning model from Stage 1. Using the same input instruction template, efficient parameter fine-tuning restricts parameter updates to a low-dimensional subspace, enabling targeted correction of the scoring accuracy of the full-scale fine-tuning model. Medium weight matrix :

[0086] ;

[0087] in, To correct the weight matrix of the model before full fine-tuning, This is the weight matrix of the corrected full-scale fine-tuning model. It is a low-rank weight matrix. and All are trainable matrices, where, It is a rank constraint parameter. d , k This represents the dimension parameter.

[0088] LoRA fine-tuning technology is used to quickly adapt a model to downstream tasks while keeping the original parameters of the large model largely unchanged. Its core idea is to learn only the incremental parameters of a low-rank matrix while keeping the pre-trained weights frozen, thus significantly reducing training costs and memory overhead. By integrating LoRA technology into the accuracy correction module of the full-scale fine-tuning model, this invention can achieve targeted fine-tuning on identified high-risk samples, significantly improving the scoring ability of the full-scale fine-tuned model on complex samples.

[0089] In summary, this invention proposes a Contrastive Error Sample Mining and Fine-Tuning Framework (CEM-FT). By analyzing the score discrepancy between the full-scale fine-tuning model and the few-shot model, it automatically identifies high-value, difficult samples and trains a lightweight LoRA adapter based on this discrepancy, achieving model performance correction with low resource overhead. By comparing the score results of the full-scale fine-tuning model and the few-shot model, this invention automatically mines high-value, difficult samples with significant score discrepancies, enabling targeted data reuse and model correction, and significantly improving the model's robustness and generalization ability in difficult scenarios.

[0090] Example 2

[0091] Based on the same inventive concept, the present invention also provides an automatic scoring system based on comparative error sample mining and model fine-tuning, for implementing the method described in the foregoing embodiments. The system includes: a dual-model scoring module, a scoring difference acquisition module, an error-prone sample set construction module, a model correction module, and a scoring acquisition module.

[0092] The dual-model scoring module is used to construct a full-scale fine-tuning model and a Few-shot model to score the response sample data to be scored, and obtain the first scoring result and the second scoring result.

[0093] The scoring difference acquisition module is used to obtain inconsistent scores and full-scale fine-tuning model scoring bias based on the first and second scoring results.

[0094] The error-prone sample set construction module is used to construct an error-prone sample set based on the bias between inconsistent scoring and full-scale fine-tuning model scoring, combined with a dynamic threshold mechanism.

[0095] The model correction module is used to correct the full fine-tuning model based on the error-prone sample set and using LoRA technology to obtain the corrected full fine-tuning model.

[0096] The scoring acquisition module is used to obtain the final scoring results of the answer sample data to be scored based on the corrected full-scale fine-tuning model.

[0097] Furthermore, the dual-model scoring module includes:

[0098] Full-scale fine-tuning model predictions: ;

[0099] Few-shot model predictions: ;

[0100] in, For full-scale fine-tuning model Sample responses to scoring The first in The rating result of each sample is denoted as the first rating result. For Few-shot model Sample responses to scoring The first in The rating result of each sample is denoted as the second rating result. For full-scale fine-tuning model The parameters, for K Number of prompt samples.

[0101] Furthermore, the rating difference acquisition module includes:

[0102] Inconsistency rating: ;

[0103] Full-scale fine-tuning model scoring bias: ;

[0104] in, For the first i Full-sample fine-tuning model and Few-shot model differences For full-scale fine-tuning model Deviation from the actual score For the first i Human expert ratings in a sample.

[0105] Furthermore, the error-prone sample set construction module includes:

[0106] ;

[0107] ;

[0108] ;

[0109] in, For error-prone sample sets, and The difference in scores for all samples Deviation from scores of all samples The mean, and These are their respective standard deviations; and These are all sensitivity adjustment parameters. For full-scale fine-tuning model and Few-shot model The difference screening threshold, For full-scale fine-tuning model The screening threshold for deviation from the true score.

[0110] Furthermore, the model correction module includes:

[0111] ;

[0112] in, To correct the weight matrix of the model before full fine-tuning, This is the weight matrix of the corrected full-scale fine-tuning model. It is a low-rank weight matrix. and All of them are trainable matrices.

[0113] Example 3

[0114] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in any of the above embodiments.

[0115] Example 4

[0116] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, the present invention also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the methods described in any of the above embodiments.

[0117] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. An automated scoring method based on contrastive error samples mining and model fine-tuning, characterized in that, The method comprises: The full fine-tuning model and the Few-shot model are used to score the to-be-scored answer sample data, and first scoring results and second scoring results are obtained; Based on the first scoring results and the second scoring results, inconsistent scores and full fine-tuning model scoring biases are obtained; Based on the inconsistent scores and the full fine-tuning model scoring biases, a dynamic threshold mechanism is combined to construct an easy-to-mistake sample set; Based on the easy-to-mistake sample set, the LoRA technology is used to correct the full fine-tuning model, and a corrected full fine-tuning model is obtained; Based on the corrected full fine-tuning model, final scoring results of the to-be-scored answer sample data are obtained; The method for obtaining inconsistent scores and full fine-tuning model scoring biases based on the first scoring results and the second scoring results comprises: Inconsistent score: ; Full fine-tuning model score bias: ; in, The first score result This is the second scoring result. For the first i Full-sample fine-tuning model and Few-shot model differences For full-scale fine-tuning model Deviation from the actual score For the first i Human expert ratings in a sample; The method for constructing an easy-to-mistake sample set based on the inconsistent scores and the full fine-tuning model scoring biases and combining a dynamic threshold mechanism comprises: ； ； ； wherein, is the set of easy-to-error samples, and is the mean of the score difference of all samples, and the score deviation of all samples, and are the standard deviations of their corresponding values, respectively; and are sensitivity adjustment parameters, is the difference screening threshold of the full-fine-tuning model and the few-shot model is the screening threshold of the full-fine-tuning model and the real score deviation.​​ 2. The method of claim 1, wherein, The method for constructing a full fine-tuning model and a Few-shot model to score to-be-scored answer sample data and obtaining first scoring results and second scoring results comprises: Full fine-tuned model predictions: ; Few-shot model predictions: ; wherein, is a full fine-tuning model is a scoring result of a sample in the answer sample to be scored, is a scoring result of a sample in the answer sample to be scored, is a first scoring result, is a few-shot model is a scoring result of a sample in the answer sample to be scored, is a scoring result of a sample in the answer sample to be scored, is a second scoring result, is a parameter of the full fine-tuning model is a parameter of the full fine-tuning model is a prompt sample, K is a prompt sample, K is a number of prompt samples.

3. The method of claim 1, wherein, The method for correcting a full fine-tuning model based on an easy-to-mistake sample set and using the LoRA technology to obtain a corrected full fine-tuning model comprises: ； wherein, is the weight matrix of the full fine-tuned model before the correction, is the weight matrix of the full fine-tuned model after the correction, is the low-rank weight matrix, and are trainable matrices.

4. An automated scoring system based on contrastive error samples mining and model fine-tuning, the system being configured to implement the method of any one of claims 1-3, characterized in that, The system comprises a double-model scoring module, a scoring difference acquisition module, an easy-to-mistake sample set construction module, a model correction module, and a scoring acquisition module. The double-model scoring module is configured to construct a full fine-tuning model and a Few-shot model to score to-be-scored answer sample data, and obtain first scoring results and second scoring results. The scoring difference acquisition module is configured to obtain inconsistent scores and full fine-tuning model scoring biases based on the first scoring results and the second scoring results. The easy-to-mistake sample set construction module is configured to construct an easy-to-mistake sample set based on the inconsistent scores and the full fine-tuning model scoring biases and combine a dynamic threshold mechanism. The model correction module is configured to correct a full fine-tuning model based on an easy-to-mistake sample set and use the LoRA technology to obtain a corrected full fine-tuning model. The scoring acquisition module is configured to obtain final scoring results of to-be-scored answer sample data based on the corrected full fine-tuning model.

5. An electronic device, comprising: The computer readable storage medium stores a computer program, and when the computer program is executed, the method of any one of claims 1-3 is implemented.

6. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and when the computer program is executed, the method of any one of claims 1-3 is implemented.

Images