A course learning-based large model rapid scoring and error analysis method and system

Abstract

The application discloses a large model rapid scoring and error analysis method and system based on course learning, and belongs to the technical field of artificial intelligence. The method comprises the following steps: input information of an error analysis model is constructed; the input information is embedded into a prompt template, and scoring of a response text is obtained based on the error analysis model. The application can guarantee the large-scale evaluation efficiency of a to-be-evaluated model and can also provide deep error cause diagnosis.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a method and system for rapid scoring and error analysis of large-scale models based on course learning. Background Technology

[0002] With the rapid development of large language models (LLMs), their performance in various chat-based tasks has significantly improved, making the need for efficient, reliable, and sophisticated question-answering (QA) evaluation of generated responses increasingly urgent. An effective QA evaluation model that can both accurately score and analyze errors is crucial for measuring model quality and guiding iterative optimization.

[0003] Existing evaluation methods can be broadly categorized into two types: rule-based methods are fast and stable, but rely heavily on shallow surface features, limiting their ability to evaluate complex reasoning and open-ended generative capabilities. The second type is the currently mainstream LLM-as-a-Judge method, which utilizes LLM to replace manual judgment. Specifically, it can be divided into three strategies: Prompt-based methods primarily utilize context learning and inference chains to achieve training-free evaluation, while recent research has enhanced flexibility by introducing diverse evaluation roles and principles or employing multi-turn interactions. Tuning-based methods adjust large language models through supervised fine-tuning or methods such as DPO (preference alignment technique). Post-processing methods optimize output through probabilistic calibration, multi-turn aggregation, or word-level scoring transformation to ensure accuracy and reliability. LLM-as-a-Judge has been widely applied in various fields, such as general natural language processing tasks, medicine, law, finance, education, and RAG system evaluation.

[26] wait.

[0004] However, existing methods face several challenges in question-answering evaluation. First, rule-based matching often underestimates model performance due to its limited semantic understanding. Second, most methods using large language models as evaluation tools rely on commercial APIs (such as GPT-4o), which, while offering good performance, are costly and have long response times, limiting their large-scale application. Open-source large language models require significant computational resources; smaller models are more efficient but prone to illusions and misjudgments. Furthermore, most methods lack sophisticated error analysis, focusing primarily on binary "correct / incorrect" decisions without identifying error types, which limits developers' ability to diagnose and improve models. Summary of the Invention

[0005] To address the aforementioned challenges, this invention discloses a method and system for rapid scoring and error analysis of large-scale models based on curriculum learning. Using Qwen3-VL-4B as the base model, it trains a two-level evaluation model through a progressively deeper curriculum design. This design ensures both high efficiency in large-scale evaluation of the models being evaluated and provides in-depth diagnosis of error causes.

[0006] To achieve the above objectives, the technical solution of the present invention includes the following:

[0007] A rapid scoring and error analysis method based on a large-scale model of course learning, the method comprising: Input information for constructing an error analysis model; wherein, the input information includes an image, a question posed based on the image, the actual answer, the response text of the model to be evaluated, and analysis instructions, and the error analysis model is constructed based on a large language model; The input information is embedded into a prompt template, and based on an error analysis model, a score for the response text is obtained, as well as the error analysis results corresponding to a score of partially correct or incorrect, wherein the score includes correct, partially correct, or incorrect. The training process of the error analysis model includes: By combining the rating tags of the response text with the training of the large language model, a fast rating model is obtained; Based on the rapid scoring model, the rapid scoring model is trained by combining the scoring labels and error reason labels of the response text to obtain the error analysis model.

[0008] Furthermore, by combining the rating labels of the response text with the training of the large language model, a fast rating model is obtained, including: Construct a first training sample, which includes an image, a question based on the image, the real answer, the response text of the model to be evaluated, and a pre-defined scoring rule; Calculate the category weights based on the true rating label distribution of the first training sample; The first training sample is fed into the large language model to obtain the predicted probability of the true rating label. Calculate the focus loss based on the category weight and the predicted probability. ; Backpropagation is performed based on focus loss to update the parameters of the large language model.

[0009] Furthermore, based on the true rating label distribution of the training data, the class weights are calculated, including: Calculate the quotient between the total number of training samples and the number of training samples in each category; The square root of the quotient is used as the class weight for that category.

[0010] Furthermore, the focal loss ;in, Indicates the first The category weights of each category This represents the predicted probability of the true rating label. Indicates the focus parameter.

[0011] Furthermore, by combining the true rating labels and true error reason labels of the response text, the fast rating model is trained to obtain an error analysis model, including: Construct a first training sample and a second training sample. The first training sample includes an image, a question based on the image, the real answer, the response text of the model to be evaluated, and a pre-defined scoring rule. The second training sample includes an image, a question based on the image, the real answer, the response text of the model to be evaluated, and analysis instructions. The first training sample is sampled and combined with the second training sample to generate a mixed dataset; The mixed data is fed into the fast rating model, and the maximum likelihood estimation loss is calculated by combining the rating labels and error reason labels of the response text. Nonlikelihood loss ; Loss based on maximum likelihood estimation Backpropagation is performed using the non-likelihood loss to update the parameters of the fast scoring model.

[0012] Furthermore, the maximum likelihood estimation loss ;in, Indicates the length of the true rating label or the true reason for error label. This represents the summation operation; This represents the current position step number in the true rating label or true error reason label, log represents the logarithmic function, and P represents the predicted probability calculation. This represents the label at the t-th position in the true rating label or true error reason label, where x represents the input data. This refers to the label preceding the t-th position in the true rating label or the true error reason label.

[0013] Furthermore, the nonlikelihood loss Among them, the normalized sequence probability , Represented by natural constant An exponential function with base 0. Indicates the length of the negative rating label or negative error reason label; This indicates the first negative rating label or negative error reason label. Labels for each location, The parameter is The predicted probability of the error analysis model at that time. This indicates the first negative rating label or negative error reason label. The label before each position.

[0014] A rapid scoring and error analysis system based on a large-scale model of course learning, characterized in that the system comprises: An information construction module is used to construct the input information for the error analysis model; wherein, the input information includes an image, a question posed based on the image, the actual answer, the response text of the model to be evaluated, and analysis instructions, and the error analysis model is constructed based on a large language model; The information reasoning module is used to embed the input information into the prompt template and, based on the error analysis model, obtain a score for the response text, as well as the error analysis result corresponding to the score being partially correct or incorrect, wherein the score includes correct, partially correct, or incorrect. The training process of the error analysis model includes: By combining the rating tags of the response text with the training of the large language model, a fast rating model is obtained; Based on the rapid scoring model, the rapid scoring model is trained by combining the scoring labels and error reason labels of the response text to obtain the error analysis model.

[0015] A computer device includes: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements the large-scale model-based rapid scoring and error analysis method for curriculum learning as described above.

[0016] A computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the large-scale model-based rapid scoring and error analysis method for curriculum learning as described above.

[0017] Compared with the prior art, the present invention has at least the following beneficial effects.

[0018] 1) Employing a lightweight foundation and hierarchical learning framework to achieve efficient deployment and evaluation with zero cost: This invention is based on a lightweight open-source foundation model (such as Qwen3-VL-4B) combined with LoRA fine-tuning technology to construct a two-stage hierarchical evaluation system (rapid scoring and error analysis) progressing from simple to complex. This technical architecture eliminates the dependence of traditional methods on expensive commercial closed-source large model APIs, achieving zero cost throughout the evaluation process and supporting efficient deployment on a single 48GB GPU. Simultaneously, the hierarchical framework significantly reduces computational redundancy, enabling the evaluation of approximately 2000 question-and-answer data points to take only about 400 seconds, saving approximately 25 times the inference time compared to closed-source large models, thus achieving efficient evaluation of large-scale data.

[0019] 2) Introducing a square-root smoothed class-balanced focal loss to significantly improve the accuracy of minority class scoring: Addressing the common class imbalance problem in question-answering evaluation data—a large number of "correct / incorrect" samples and a small number of "partially correct" samples—this invention innovatively introduces a square-root smoothed class-balanced focal loss in the first stage of training. This mechanism effectively avoids gradient explosion caused by a few extremely rare samples, while preventing the model's decisions from being biased towards the majority class. This significantly improves the model's scoring accuracy for difficult and minority class samples, resulting in an overall evaluation accuracy superior to existing closed-source large models.

[0020] 3) Robust and precise error diagnosis achieved by combining experience replay and contrastive nonlikelihood optimization: In the training of the error analysis model in the second stage, this invention introduces an experience replay mechanism to effectively prevent the model from catastrophically forgetting its basic scoring capabilities when learning complex error attributions. Simultaneously, to address illusions and misjudgments that are prone to occur in multimodal analysis tasks, this invention introduces a contrastive nonlikelihood optimization mechanism. This mechanism makes the decision boundary for error classification clearer by explicitly suppressing the generation probability of other competing error categories. Based on the above mechanisms, this model, while providing a score, can accurately attribute errors to five categories: reasoning errors, image misunderstandings, overthinking, etc., providing highly interpretive and clear guidance for the iterative optimization of the evaluated model. Attached Figure Description

[0021] Figure 1 Flowchart of a rapid scoring and error analysis method based on a large-scale model of course learning.

[0022] Figure 2 Large model training flowchart.

[0023] Figure 3 An image from one embodiment of the present invention.

[0024] Figure 4 Block diagram of a large-scale model-based rapid scoring and error analysis system for curriculum learning.

[0025] Figure 5 A block diagram of computer equipment. Detailed Implementation

[0026] The present invention will be further described below with reference to possible accompanying drawings and specific embodiments, but this does not constitute any limitation on the present invention.

[0027] The present invention provides a rapid scoring and error analysis method for a large-scale model based on course learning, such as... Figure 1 As shown, the input information for the error analysis model is first constructed, then this input information is embedded into a prompt template, and based on the error analysis model, the score of the response text and the error analysis results corresponding to the score being partially correct or incorrect are obtained; wherein, the error analysis model is constructed based on a large language model. The training of the large model is divided into two stages. In the first stage, to address the common class imbalance problem in the evaluation data, this invention uses LoRA fine-tuning and innovatively introduces square root smoothed class balancing Focal Loss to train a fast scoring model that can quickly output Correct / Partial / Wrong three-level scores based on the question, the standard answer, and the model's response. In the second stage, this invention further fine-tunes the fast scoring model to construct the error analysis model. To prevent the model from forgetting its basic scoring ability when learning complex attributions, this invention adopts an experience replay mechanism; at the same time, contrastive unlikely optimization is introduced to clarify the decision boundary of the model and improve the robustness of attribution. The final model can not only score, but also subdivide errors into five categories: reasoning error, image misunderstanding, overthinking, unanswerable, and other, thus achieving the integration of scoring and error analysis.

[0028] Specifically, the training process of this large model, such as Figure 2 As shown, it includes the following steps S1 to S2.

[0029] Step S1: Train the large language model by combining the rating labels of the response text to obtain a fast rating model.

[0030] Input data for the HiEval-QS rapid scoring model The evaluation process includes an image, a question based on that image, the actual answer, the model's response to be evaluated, and predefined scoring rules. The output is one of the following: {Correct, Partial, Wrong}; where Correct represents completely correct, Partial represents partially correct, and Wrong represents incorrect. Let... To indicate a problem, Let represent the set of true answers to the question. This represents the set of actual answers encompassed at the semantic level by the model's response to be evaluated. The scoring logic is defined as follows: Here, Score represents the evaluation score, symbolized by " "Represents logical AND, , symbol" " indicates logical OR, Represents the set of true answers The supplement, This represents the empty set. Specifically, the score "Partial" refers only to the set of answers semantically covered by the model's response. It is a collection of real answers. Non-empty proper subsets (denoted as It will only be activated when (i.e., when) the model's response contains elements of incorrect answers (i.e., when) the model's intent is correct, but it fails to cover the entire solution space of the problem. The model categorizes it as "Wrong".

[0031] In rapid scoring tasks, training data often suffers from imbalance, with an excessive number of "Correct" and "Wrong" samples and a relatively small number of "Partial" samples. Traditional cross-entropy loss functions tend to bias the model towards the class with the larger sample size. To address this issue, this invention proposes a Class-Balanced Focal Loss with Square Root Smoothing.

[0032] To alleviate the class imbalance problem, this invention first calculates class weights based on the true label distribution in the training set. Let... Represents the total number of samples. Indicates category (in The number of samples in the (). To prevent a sharp increase in weight values ​​due to extremely scarce samples in some categories, this invention employs a square root smoothing strategy to calculate the inverse class frequency: Subsequently, the invention standardizes the weights and introduces a cutoff threshold for training stability. This yields the final category weights. : In the above formula, This represents a function that takes the minimum value. This represents the average inverse class frequency across all categories; Represents a set of categories The total number of categories in this invention ); ∑ represents the summation operation on the elements in the set; Represents a set of categories The inverse class frequency of the j-th class. This smoothing and pruning mechanism ensures that the model focuses on the minority class (“Partial”) while avoiding training divergence due to excessively large gradients generated by individual extreme samples. This invention uses the aforementioned weights... This is incorporated into the Focal Loss. For each generated rating token, the final loss function of this invention is... Defined as: in, It is the model's predicted probability of the target's true label. These are the focus parameters. Through this design, during backpropagation, for samples belonging to the majority class and with high prediction confidence, their loss weights will be... Significantly reduce the impact. Conversely, for challenging samples from the minority class that are poorly predicted (with weights of ), The gradient contribution is thus preserved or even enhanced.

[0033] Step S2: Based on the fast scoring model, train the fast scoring model by combining the scoring labels and error reason labels of the response text to obtain the error analysis model.

[0034] After the model has acquired basic scoring capabilities, the goal of this step is to expand its functionality from "coarse-grained classification" to "refined analysis." Specifically, for samples classified as "Wrong" or "Partial," the model must accurately identify the specific error type. Unlike the rapid scoring task, this step is defined as a multimodal analysis process. Formally, this invention defines the input to this task as a quintuple. Specifically, it includes: Indicates "input image". Indicates "problem" Indicates the "true answer". This refers to the "response text of the model to be evaluated," that is, the response text of a given external large language model or multimodal large language model upon receiving the aforementioned input image. and the problem The generated response text; This indicates the analysis instructions. The instruction text predefines specific scoring rules, detailed definitions and descriptions of various error types, and standardized output format requirements, which are used to guide the model of this invention to perform scoring and error analysis tasks.

[0035] The goal of this model is to learn the mapping function. Output sequence Includes a rating result And an error analysis category The target output format is defined as follows: in, Error classification set This indicates text concatenation. This definition explicitly states that, in the actual evaluation application (i.e., the reasoning / testing phase), the analysis model of this invention will also receive data containing the true answers. The complete quintuple is taken as input, and must be based on the true answer. and the response of the model to be evaluated The differences between them, combined with images and the problem ,exist The diagnostic results are generated under the guidance of the model. The error types are as follows: (1) Reasoning Error, which involves errors in logic, calculation or conditional understanding; (2) Image Misunderstanding, which is the misidentification of image content or the generation of illusions; (3) Unanswerable, which is the model's incorrect judgment that the question cannot be answered; (4) Overthinking, which mainly occurs in reasoning-based models and is manifested as redundant or circular reasoning leading to truncation; (5) Other: which mainly involves situations where the error cannot be accurately classified into the aforementioned specific categories of reasoning error, image misunderstanding, unanswerable and overthinking due to reasons such as the response text generated by the evaluated model being too short or the output format not conforming to the preset specifications.

[0036] In one embodiment, the above five-tuple is composed of: image :like Figure 3 As shown; question "There are more gray squares than white squares in the picture"; Model response to be evaluated "There are 36 gray squares and 16 white squares in the diagram. 36 - 16 = 20. There are 20 more gray squares than white squares." The real answer "9"; Scoring instructions "Please perform the following dual tasks: 1. Scoring: Evaluate the correctness of the model's response by combining the image and the real answer. Output is limited to..." 2. Error Attribution: If the score is or Please identify the main error categories from the following options: (1) Reasoning Error; (2) Image Misunderstanding; (3) Unanswerable; (4) Overthinking; (5) Other. Evaluation results " ;Image Misunderstanding".

[0037] Furthermore, when models transition from simple scoring tasks to complex error analysis tasks, they face the risk of catastrophic forgetting—that is, the model may learn to analyze errors but lose its basic scoring accuracy. To address this issue, this invention introduces an experience replay mechanism. This invention constructs a hybrid dataset. This includes data for detailed analysis tasks. ) and data used for rapid scoring tasks ( ), according to proportion Perform sampling: In the above formula, This represents the mixed dataset used for the second stage of training; This represents the target dataset for the second-stage error analysis task; The union operation represents the set operation; Represents a random sampling function; This represents the empirical dataset for the first-stage rapid scoring task. This represents the sampling ratio coefficient. Through this experience replay mechanism, the present invention learns from new task data in the second phase. At the same time, proportionally Resampling and incorporating empirical data from the first phase This ensures that the model improves the precision of error diagnosis without losing basic scoring accuracy, effectively solving the catastrophic forgetting problem in deep learning.

[0038] The standard supervised fine-tuning (SFT) method optimizes the model by minimizing the negative log-likelihood value: In the above formula, This represents the maximum likelihood estimation loss function; Indicates the length of the true rating label or the true reason for error label. This represents the summation operation; This represents the current position step number in the true rating label or true error reason label, log represents the logarithmic function, and P represents the predicted probability calculation. This represents the label at the t-th position in the true rating label or true error reason label, where x represents the input data. This refers to the label preceding the t-th position in the true rating label or the true error reason label.

[0039] The core of this objective function is to teach the model "what is correct". However, in multimodal analysis tasks, the model faces the problem of fine-grained classification. Training solely using maximum likelihood estimation (MLE) can easily lead to the model inflating the true labels. The probability of the error is underestimated, but the probability of other competing error categories is not sufficiently suppressed, thus blurring the decision boundary.

[0040] To address this problem, this invention introduces a comparative impossibility mechanism to explicitly teach the model "what is wrong." For each sample containing error analysis... Its real label is (Including error categories) ). Utilizing the mutual exclusivity of labels, this invention extracts information from a set of categories. Remove from Then select a negative class To construct negative sequences .

[0041] This invention defines a "normalized sequence probability" with the aim of minimizing the probability of generating negative sequences: In the above formula, represents the standardized sequence probability; exp represents an exponential function with the natural constant e as the base. Indicates the length of the negative rating label or negative error reason label; This indicates the first negative rating label or negative error reason label. Labels for each location, The parameter is The predicted probability of the error analysis model at that time. This indicates the first negative rating label or negative error reason label. The labels preceding each position. The corresponding nonlikelihood loss function is defined as follows: In the above formula, This represents the non-likelihood loss function. The purpose of this loss function is to minimize the probability of generating negative sequences through inverse penalty. In the second stage, the ultimate optimization objective is to combine general generative capability (applicable to all data) with discriminative constraints (applicable to error analysis data): In the above formula, This represents the final optimization objective loss function in the second stage; This represents the maximum likelihood estimation loss function, used to constrain the model's general generative capability; x represents the input sample. This represents the true label sequence corresponding to the sample; Indicates the balance coefficient; This indicates an indicator function, meaning it takes the value 1 when the condition inside the parentheses is true, and 0 otherwise. This represents the set of error samples in the error analysis task; This represents the non-likelihood loss function, used to provide discriminative constraints. It is the balance coefficient. To ensure the model can smoothly generate scores and perform correct analysis, This is used as a regularization term specifically to punish hallucinations produced by the model. It is the set of error samples in the error analysis task.

[0042] In summary, this invention, based on the Qwen3-VL-4B model, trains a HiEval-QS model for rapid scoring of model responses. During training, LoRA fine-tuning is employed, and a square root smoothing class-balancing Focal Loss is introduced. The trained model can quickly output Correct / Half / Wrong three-level scores based on the question, the standard answer, and the model's response.

[0043] This invention further refines the HiEval-QS model to obtain the HiEval-EA model, which can both quickly score and analyze the causes of errors. During training, an experience replay mechanism is used to prevent the model from forgetting its basic scoring capabilities, while contrastive likelihood optimization is introduced to clarify the model's decision boundaries. The causes of errors are categorized into five types: Reasoning Error, Image Misunderstanding, Overthinking, Unanswerable, and Other.

[0044] Based on the same concept, this invention also discloses a rapid scoring and error analysis system based on a large-scale model of course learning, such as... Figure 4 As shown, the system includes: An information construction module is used to construct the input information for the error analysis model; wherein, the input information includes an image, a question posed based on the image, the actual answer, the response text of the model to be evaluated, and analysis instructions, and the error analysis model is constructed based on a large language model; The information reasoning module is used to embed the input information into the prompt template and, based on the error analysis model, obtain a score for the response text, as well as the error analysis result corresponding to the score being partially correct or incorrect, wherein the score includes correct, partially correct, or incorrect. The training process of the error analysis model includes: By combining the rating tags of the response text with the training of the large language model, a fast rating model is obtained; Based on the rapid scoring model, the rapid scoring model is trained by combining the scoring labels and error reason labels of the response text to obtain the error analysis model.

[0045] Based on the same concept, this invention also discloses a computer device, which may be a terminal, a laptop computer, a desktop computer, a server, a computer cluster, or other types of computer devices. For example... Figure 5 As shown, the computer device may include at least one processor and memory. The processor can execute instructions stored in the memory. The processor is communicatively connected to the memory via a data bus. In addition to the memory, the processor can also be communicatively connected to input devices, output devices, and communication devices via the data bus.

[0046] The processor can be any conventional processor. Processors may include central processing units (CPUs), graphics processing units (GPUs), field-programmable gate arrays (FPGAs), systems on chips (SoCs), application-specific integrated circuits (ASICs), or combinations thereof.

[0047] Memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0048] In this embodiment of the invention, an executable instruction is stored in a memory. The processor can read the executable instruction from the memory and execute the instruction to implement all or part of the steps of the method of the invention.

[0049] Based on the same concept, the present invention also discloses a computer-readable storage medium including a computer program product or storing the computer program product. The computer product includes computer program instructions that can be executed by a processor to perform all or part of the steps described in the exemplary embodiments above.

[0050] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages, and scripting languages ​​(e.g., Python). The program code can be executed entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0051] Computer-readable storage media can take the form of any combination of one or more readable media. A readable medium can be a readable signal medium or a readable storage medium. A readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media include: static random access memory (SRAM) having one or more electrically connected wires; electrically erasable programmable read-only memory (EEPROM); erasable programmable read-only memory (EPROM); programmable read-only memory (PROM); read-only memory (ROM); magnetic storage; flash memory; magnetic disk or optical disk; or any suitable combination thereof.

[0052] The above embodiments are merely illustrative of the technical solutions of the present invention and are not intended to limit it. Those skilled in the art can modify or make equivalent substitutions to the above technical solutions based on the concept of the present invention, and such modifications or equivalent substitutions should all be covered within the protection scope of the present invention. The protection scope of the present invention is defined by the claims.

Claims

1. A rapid scoring and error analysis method based on a large-scale model of course learning, characterized in that, The method includes: Input information for constructing an error analysis model; wherein, the input information includes an image, a question posed based on the image, the actual answer, the response text of the model to be evaluated, and analysis instructions, and the error analysis model is constructed based on a large language model; The input information is embedded into a prompt template, and based on an error analysis model, a score for the response text is obtained, as well as the error analysis results corresponding to a score of partially correct or incorrect, wherein the score includes correct, partially correct, or incorrect. The training process of the error analysis model includes: By combining the rating tags of the response text with the training of the large language model, a fast rating model is obtained; Based on the rapid scoring model, the rapid scoring model is trained by combining the scoring labels and error reason labels of the response text to obtain the error analysis model.

2. The method according to claim 1, characterized in that, By training a large language model using rating labels from the response text, a fast rating model is obtained, including: Construct a first training sample, which includes an image, a question based on the image, the real answer, the response text of the model to be evaluated, and a pre-defined scoring rule; Calculate the category weights based on the true rating label distribution of the first training sample; The first training sample is fed into the large language model to obtain the predicted probability of the true rating label. Calculate the focus loss based on the category weight and the predicted probability. ; Backpropagation is performed based on focus loss to update the parameters of the large language model.

3. The method according to claim 2, characterized in that, Based on the true rating label distribution of the training data, calculate the class weights, including: Calculate the quotient between the total number of training samples and the number of training samples in each category; The square root of the quotient is used as the class weight for that category.

4. The method according to claim 2, characterized in that, The focus loss ;in, Indicates the first The category weights of each category This represents the predicted probability of the true rating label. Indicates the focus parameter.

5. The method according to claim 1, characterized in that, By combining the true rating labels and true error reason labels from the response text, a fast rating model is trained to obtain an error analysis model, including: Construct a first training sample and a second training sample. The first training sample includes an image, a question based on the image, the real answer, the response text of the model to be evaluated, and a pre-defined scoring rule. The second training sample includes an image, a question based on the image, the real answer, the response text of the model to be evaluated, and analysis instructions. The first training sample is sampled and combined with the second training sample to generate a mixed dataset; The mixed data is fed into the fast rating model, and the maximum likelihood estimation loss is calculated by combining the rating labels and error reason labels of the response text. Nonlikelihood loss ; Loss based on maximum likelihood estimation Backpropagation is performed using the non-likelihood loss to update the parameters of the fast scoring model.

6. The method according to claim 5, characterized in that, The maximum likelihood estimation loss ;in, Indicates the length of the true rating label or the true reason for error label. This represents the summation operation; This represents the current position step number in the true rating label or true error reason label, log represents the logarithmic function, and P represents the predicted probability calculation. This represents the label at the t-th position in the true rating label or true error reason label, where x represents the input data. This refers to the label preceding the t-th position in the true rating label or the true error reason label.

7. The method according to claim 5, characterized in that, The nonlikelihood loss Among them, the normalized sequence probability , Represented by natural constant An exponential function with base 0. Indicates the length of the negative rating label or negative error reason label; This indicates the first negative rating label or negative error reason label. The label for each location, The parameter is The predicted probability of the error analysis model at that time. This indicates the first negative rating label or negative error reason label. The label before each position.

8. A rapid scoring and error analysis system based on a large-scale model of course learning, characterized in that, The system includes: An information construction module is used to construct the input information for the error analysis model; wherein, the input information includes an image, a question posed based on the image, the actual answer, the response text of the model to be evaluated, and analysis instructions, and the error analysis model is constructed based on a large language model; The information reasoning module is used to embed the input information into the prompt template and, based on the error analysis model, obtain a score for the response text, as well as the error analysis result corresponding to the score being partially correct or incorrect, wherein the score includes correct, partially correct, or incorrect. The training process of the error analysis model includes: By combining the rating tags of the response text with the training of the large language model, a fast rating model is obtained; Based on the rapid scoring model, the rapid scoring model is trained by combining the scoring labels and error reason labels of the response text to obtain the error analysis model.

9. A computer device, characterized in that, The computer device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the large-scale model rapid scoring and error analysis method based on course learning as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the fast scoring and error analysis method for large-scale learning models based on course learning as described in any one of claims 1-7.

Images