A lightweight large language model evaluator training method based on reinforcement learning

Abstract

The present application relates to the technical field of natural language processing, large language model automatic evaluation and reinforcement learning, in particular to a lightweight large language model evaluator training method based on reinforcement learning. The present application constructs the judging prompt relying on the paired evaluation sample, adopts the lightweight large language model to complete the evaluation output, generates the reward with the consistency of the advantage and disadvantage relationship, and completes the reinforcement learning update through the same prompt group sampling, the group reward mean variance calculation advantage value, effectively solves the problems of high cost, indirect training target and poor training stability of the existing evaluation scheme; using the lightweight model reduces the training and deployment cost, without relying on the high-cost large model; taking the matching degree of the answer advantage and disadvantage as the reward signal, and with the help of the group sampling normalized advantage value and the reinforcement learning update, the stable training process is realized, and finally the lightweight evaluation model can be widely used in open domain question and answer evaluation, model answer comparison, automatic evaluation and other scenes.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing, automated evaluation of large language models, and reinforcement learning, and in particular to a lightweight method for training a large language model evaluator based on reinforcement learning. Background Technology

[0002] With the widespread application of large language models in scenarios such as question answering, summarizing, code generation, intelligent customer service, and content creation, how to conduct stable, low-cost, and scalable quality assessment of model-generated content has become a key issue in the research, development, and deployment of large language models. While traditional manual evaluation has high reliability, it typically requires professional annotators to compare each item individually, resulting in high costs, long cycles, and difficulties in ensuring consistency, making it unsuitable for the needs of rapid model iteration and large-scale evaluation.

[0003] To reduce evaluation costs, a technique called LLM-as-Judge (Large Language Model as-Judge) has gradually emerged. This approach typically inputs the question to be evaluated and candidate answers into a powerful large language model, which then outputs a comparison conclusion or score. This technique can, to some extent, simulate the judging preferences of humans or strong models and has been used for tasks such as open-domain question answering evaluation, instruction following ability assessment, and model-aligned data construction.

[0004] However, existing large language model evaluation methods still have several shortcomings. First, directly calling powerful closed-source models as evaluators is costly and limited in terms of data privacy, controllability, and reproducibility. Second, some open-source evaluation models rely on large parameter scales or large-scale supervised fine-tuning data, resulting in high deployment costs and making them unsuitable for use in resource-constrained environments. Third, existing methods often use the score value itself as the primary fitting target, while in pairwise answer comparison scenarios, the more crucial factor is whether the judgment of the superiority / inferiority relationship between the answers is correct. If the training objective fails to directly align with this core evaluation objective, it can easily lead to insufficient training efficiency and evaluation stability.

[0005] Therefore, overcoming the shortcomings of the existing technology is an urgent problem to be solved in this technical field. Summary of the Invention

[0006] The technical problem to be solved by this invention is: how to solve the problems of high evaluation cost of large language models, insufficiently direct training objectives of lightweight evaluation models, and insufficient evaluation stability in the existing technology.

[0007] To achieve the above objectives, the present invention employs the following means: Firstly, a lightweight method for training a large language model evaluator based on reinforcement learning is provided, including: Construct a pairwise answer evaluation data sample, wherein the evaluation data sample includes at least the user question, the first candidate answer, the second candidate answer, and the reference evaluation result; Based on the evaluation data sample, a referee prompt is generated, which is used to instruct the large language model to be trained to compare, analyze and score the two candidate answers; The judge's prompts are input into a lightweight large language model to obtain the model output, which includes the reasoning process and two scores corresponding to the first candidate answer and the second candidate answer, respectively. The two scores in the model output are analyzed, and the relative merits of the model's predicted answers are determined based on the relationship between the two scores. The reward is calculated based on whether the relationship between the superiority and inferiority of the answers is consistent with the relationship between the superiority and inferiority of the answers as indicated by the reference evaluation results; Group sampling is performed for the same referee's prompts, and the advantage value is calculated based on the mean and variance of the rewards in the same group. The lightweight large language model is updated using reinforcement learning based on the rewards and advantage values ​​to obtain the trained lightweight large language model evaluation model.

[0008] Preferably, the reference evaluation results include the first candidate answer score and the second candidate answer score given by the reference model or manual annotation. The two scores are used to determine three types of relationships, including the first candidate answer being better than the second candidate answer, the first candidate answer being worse than the second candidate answer, or the two being of comparable quality.

[0009] Preferably, the referee prompts include system prompts and user prompts. The system prompts are used to define the evaluation criteria, output structure, and scoring range, while the user prompts are used to carry the user's question and the main text of the two candidate answers.

[0010] Preferably, the model output meets a preset format, which includes a reasoning field for carrying the reasoning analysis content and an answer field for carrying the two scores.

[0011] Preferably, the parsing model output includes: extracting the first predicted score and the second predicted score from the answer field, determining whether the two predicted scores are integers or values ​​within a preset score range, and determining the model output as invalid output when the output format does not meet the requirements, the score is missing, or the score exceeds the limit.

[0012] Preferably, the reward is determined by the following relationship: A positive reward is given when the model's predicted relationship between the quality of the answers matches the relationship between the quality of the answers as indicated by the reference evaluation results, and the model's output meets the preset format; otherwise, no positive reward is given.

[0013] Preferably, the reward The function is represented as: ; in, and These represent the first and second predicted scores in the output of the i-th model, respectively. and These represent the first and second reference scores in the evaluation results, respectively. This indicates whether the output of the i-th model satisfies the preset format and scoring range constraints. Indicates an indicator function, Represents a symbolic function.

[0014] Preferably, the same referee's prompt is sampled multiple times to obtain multiple model outputs and their corresponding rewards, and the advantage value is calculated based on the mean and standard deviation of the rewards of the same group of samples; The advantage value The calculation formula is: ; Where G represents the number of sampled outputs corresponding to the same referee's prompt. As a reward, It is a mean function. It is a function of standard deviation; The reward corresponding to the i-th sampled output and the relative advantage value of the group used by the t-th token in the i-th output are both calculated according to the above formula.

[0015] Preferably, the reinforcement learning update includes optimizing the objective using GRPO (Group Relative Policy Optimization), the GRPO objective function being expanded as follows: ;

[0016] ; ; in, Let be the policy objective function to be maximized by the GRPO algorithm. For mathematical expectation, Provides suggestions for a single input field. Let G represent the distribution of all training prompt samples, and let G be the number of sampled outputs corresponding to the same referee prompt. For the same prompt q, generate G model output texts. For the old strategy model, Given the cue q, the model generates a conditional distribution for the output o. The total token length of the i-th output text. KL penalty coefficient, For the group normalized dominance value, Output the t-th token for the i-th item. For PPO symmetric clipping hyperparameters, This is the KL divergence regularization term between the current model and the baseline model. To freeze the SFT baseline reference model.

[0017] Preferably, after the lightweight large language model is trained, a consistency index between the evaluation model and the reference evaluation result is calculated on the validation set, and model parameters for deployment or subsequent evaluation are selected based on the consistency index. The lightweight large language model evaluation model is used for at least one of the following tasks: open-domain question answering quality assessment, model answer comparison, automated evaluation, preference data filtering, and model training feedback.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention constructs judge prompts based on paired evaluation samples, uses a lightweight large language model to complete the judgment output, generates rewards based on the consistency of the superiority-inferiority relationship, and completes reinforcement learning updates by sampling from the same prompt group and calculating the advantage value based on the mean and variance of group rewards. This effectively solves the problems of high cost, indirect training objectives, and poor training stability of existing judgment schemes. On the one hand, the use of a lightweight model reduces training and deployment costs, eliminating the need to rely on high-cost large models. On the other hand, the matching degree of answer superiority-inferiority is used as the reward signal, and the training objective is directly aligned with the core requirements of paired comparison judgment. At the same time, the use of group sampling normalized advantage value to cooperate with reinforcement learning updates stabilizes the training process. The resulting lightweight judgment model can be widely used in open-domain question answering evaluation, model answer comparison, automated evaluation, and other scenarios. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating a lightweight large language model evaluator training method based on reinforcement learning provided in an embodiment of the present invention. Figure 2This is a schematic diagram illustrating the principle of a lightweight large language model evaluator training method based on reinforcement learning provided in an embodiment of the present invention. Figure 3 This is a schematic diagram of a process for constructing paired response evaluation data samples provided in an embodiment of the present invention; Figure 4 This is a flowchart illustrating the construction of a referee prompt template according to an embodiment of the present invention; Figure 5 This is a schematic diagram of a process for generating model evaluation output provided by an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0022] Unless the context otherwise requires, throughout the specification and claims, the term "comprising" is interpreted as openly inclusive, meaning "including, but not limited to." In the description of the specification, terms such as "one embodiment," "some embodiments," "exemplary embodiment," "example," "specific example," or "some examples" are intended to indicate that a particular feature, structure, material, or characteristic associated with that embodiment or example is included in at least one embodiment or example of this disclosure. The illustrative representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics mentioned may be included in any suitable manner in any one or more embodiments or examples; that is, although they may be incorporated into embodiments or examples using the above terms for reasons such as order and position, it does not limit them to be incorporated in combination by a single embodiment or example.

[0023] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this disclosure, unless otherwise stated, "a plurality of" means two or more. Furthermore, for example, the description may use the prefix "A" or "B" to describe the same type of nouns as two independent entities. In this case, the corresponding features defined with "A" and "B" are used only to distinguish between similar entities and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features.

[0024] In describing some embodiments, the terms "coupled," "coupled," and "connected," and their derivative expressions, may be used. For example, the term "connected" may be used in describing some embodiments to indicate that two or more components have direct physical or electrical contact with each other. Similarly, the term "coupled" may be used in describing some embodiments to indicate that two or more components have direct physical or electrical contact. However, the terms "connected" or "coupled" may also refer to two or more components that do not have direct contact with each other but still cooperate or interact with each other, such as "optical coupling," "wireless connection," etc. The embodiments disclosed herein are not necessarily limited to the scope of this invention.

[0025] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0026] Example 1: This embodiment first explains and clarifies the technical terminology of the present invention: A large language model evaluator refers to a large language model capable of receiving questions and candidate answers, and comparing, scoring, or ranking the quality of the candidate answers. Paired answer evaluation data refers to a data sample where one user question corresponds to two candidate answers, along with a reference evaluation result. This reference evaluation result may include the scores of the two candidate answers, their relative merits, or the reasons for the evaluation.

[0027] To address the problems of existing technologies, this embodiment proposes a lightweight large language model evaluator training method based on reinforcement learning. In one implementation, such as... Figure 1 and Figure 2 As shown, the method includes: Step 101: Construct a pairwise answer evaluation data sample, wherein the evaluation data sample includes at least the user question, the first candidate answer, the second candidate answer, and the reference evaluation result.

[0028] Among them, such as Figure 3 As shown, the construction of paired response evaluation data samples specifically includes: Step 1011: Obtain the user's question, the first candidate answer, and the second candidate answer. The candidate answers can come from different large language models, or from the output of the same model under different sampling conditions.

[0029] Step 1012: Obtain the reference evaluation results. The reference evaluation results include a first reference score and a second reference score, denoted as y1 and y2 respectively.

[0030] Step 1013: Determine a reference advantage-disadvantage relationship according to the reference scores. If y1>y2, the reference advantage-disadvantage relationship is that the first candidate answer is superior to the second candidate answer; if y1<y2, the reference advantage-disadvantage relationship is that the second candidate answer is superior to the first candidate answer; if y1=y2, the reference advantage-disadvantage relationship is that the two answers have comparable quality.

[0031] In one embodiment, the reference evaluation result comprises a score for the first candidate answer and a score for the second candidate answer given by a reference model or manual annotation, and the two scores are used to determine three types of relationships: the first candidate answer is superior to the second candidate answer, the first candidate answer is inferior to the second candidate answer, or the two have comparable quality.

[0032] Step 102: Generate a judge prompt based on the evaluation data sample, where the judge prompt is used to instruct the large language model to be trained to compare, analyze and score the two candidate answers.

[0033] wherein, as Figure 4 shown, constructing the judge prompt template specifically comprises the following sub-steps: Step 1021: Write an evaluation task description in the system prompt, requiring the model to act as an impartial judge and compare the two candidate answers from the aspects of accuracy, relevance, completeness and helpfulness.

[0034] Step 1022: Sequentially write the user question, the first candidate answer and the second candidate answer in the user prompt, and use boundary markers to distinguish different text segments.

[0035] Step 1023: Specify the output format of the model, so that the model outputs the reasoning process first, and then outputs two scores in the answer field.

[0036] In one embodiment, the judge prompt comprises a system prompt and a user prompt, the system prompt is used to define the evaluation criteria, output structure and scoring range, and the user prompt is used to carry the user question and the main content of the two candidate answers.

[0037] Step 103: Input the judge prompt into a lightweight large language model to obtain a model output, where the model output comprises a reasoning process and two scores respectively corresponding to the first candidate answer and the second candidate answer.

[0038] wherein, the model output satisfies a preset format, and the preset format comprises a reasoning field for carrying reasoning analysis content and an answer field for carrying the two scores.

[0039] In one embodiment, as Figure 5 shown, generating the model evaluation output comprises the following steps: Step 1031: Input the judge prompt into the lightweight large language model to be trained, to obtain text generated by the model.

[0040] Step 1032: Extract the reasoning process and answer field from the text generated by the model, wherein the answer field contains the first prediction score and the second prediction score, denoted as s1 and s2 respectively.

[0041] Step 1033: Determine whether the model output meets the format constraints and scoring range constraints. If the output does not contain a valid answer field, has missing scores, or has scores that exceed the range, then the output is determined to be invalid.

[0042] Step 104: Analyze the two scores in the model output and determine the relative merits of the model's predicted answers based on the relationship between the two scores.

[0043] In one implementation, parsing the model output includes: extracting a first predicted score and a second predicted score from the answer field, determining whether the two predicted scores are integers or numerical values ​​within a preset score range, and determining the model output as invalid output when the output format does not meet the requirements, the score is missing, or the score exceeds the limit.

[0044] Step 105: Calculate the reward based on whether the relationship between the superiority and inferiority of the answers is consistent with the relationship between the superiority and inferiority of the answers as indicated by the reference evaluation results.

[0045] The reward is determined by the following relationship: when the relationship between the quality of the answer predicted by the model is consistent with the relationship between the quality of the answer represented by the reference evaluation result, and the model output meets the preset format, a positive reward is given; otherwise, no positive reward is given.

[0046] In one implementation, calculating the consistency reward specifically includes: calculating the model's predicted relationship based on the relationship between the first and second predicted scores; calculating the reference relationship based on the relationship between the first and second reference scores; and setting the reward to 1 when the model's predicted relationship matches the reference relationship and the model output meets a preset format, otherwise setting the reward to 0.

[0047] In one implementation, the reward The function is represented as: ; in, and These represent the first and second predicted scores in the output of the i-th model, respectively. and These represent the first and second reference scores in the evaluation results, respectively. This indicates whether the output of the i-th model satisfies the preset format and scoring range constraints. Indicates an indicator function, Represents a symbolic function.

[0048] Step 106: Perform group sampling for the same referee's prompts, and calculate the advantage value based on the mean and variance of the rewards in the same group.

[0049] Group sampling specifically includes the following sub-steps: For the same referee prompt x, generate G candidate judge outputs using the current policy model, denoted as o1, o2, ..., oG. Calculate the reward r1, r2, ..., rG for each candidate judge output. Calculate the advantage value of each candidate output based on the mean and standard deviation of the rewards in the same group. Sample the same referee prompt multiple times to obtain multiple model outputs and their corresponding rewards, and calculate the advantage value based on the mean and standard deviation of the rewards in the same group of samples. The advantage value The calculation formula is: ; Where G represents the number of sampled outputs corresponding to the same referee's prompt. As a reward, It is a mean function. The standard deviation function is used; the reward corresponding to the i-th sampled output and the group relative advantage value used by the t-th token in the i-th output are both calculated according to the above formula.

[0050] Step 107: Based on the reward and advantage value, perform reinforcement learning to update the lightweight large language model to obtain the trained lightweight large language model evaluation model.

[0051] The process of updating model parameters based on the policy-based pruning objective function includes the following sub-steps: Calculate the probability ratio of the current policy relative to the old policy at the token level. Construct a policy optimization objective based on the group relative advantage value and the token-level probability ratio. Prune the probability ratio to limit the magnitude of a single policy update, thereby reducing the risk of training divergence.

[0052] In one implementation, the reinforcement learning update includes optimizing the objective using a GRPO relative policy, the GRPO objective function being expanded as follows: ;

[0053] ; ; in, Let be the policy objective function to be maximized by the GRPO algorithm. For mathematical expectation, Provides suggestions for a single input field. Let G represent the distribution of all training prompt samples, and let G be the number of sampled outputs corresponding to the same referee prompt. For the same prompt q, generate G model output texts. For the old strategy model, Given the cue q, the model generates a conditional distribution for the output o. The total token length of the i-th output text. KL penalty coefficient, For the group normalized dominance value, Output the t-th token for the i-th token. For PPO symmetric clipping hyperparameters, This is the KL divergence regularization term between the current model and the baseline model. To freeze the SFT baseline reference model.

[0054] In one implementation, DAPO (Decoupled Clip and DynamicsAmpling Policy Optimization) is used to eliminate KL, asymmetric clipping, and the token-level policy gradient loss objective, expressed by the following formula:

[0055] ; ; in: ; In this invention, DAPO normalizes the loss using the total number of all output tokens and filters out all correct or all incorrect sample groups through dynamic sampling constraints. In the evaluation task of this invention, the equivalence judgment function can be replaced by the consistency judgment of the superiority or inferiority of the answers.

[0056] In one implementation, after the lightweight large language model is trained, a consistency index between the evaluation model and the reference evaluation result is calculated on the validation set, and model parameters for deployment or subsequent evaluation are selected based on the consistency index; the lightweight large language model evaluation model is used for at least one task among open domain question answering quality assessment, model answer comparison, automated evaluation, preference data filtering, and model training feedback.

[0057] Input a user question and two candidate answers into the validation set. The trained lightweight evaluation model generates two predicted scores. Calculate at least one of the following metrics based on the consistency between the predicted score relationship and the reference score relationship: accuracy, precision, recall, and F1 score. Compare the validation set performance of the model parameters at different training stages, and select the best-performing model parameters as the final lightweight evaluation model.

[0058] In summary, this embodiment constructs judge prompts based on paired evaluation samples, uses a lightweight large language model to complete the judgment output, generates rewards based on the consistency of the superiority-inferiority relationship, and completes reinforcement learning updates by sampling from the same prompt group and calculating the advantage value from the mean and variance of group rewards. This effectively solves the problems of high cost, indirect training objectives, and poor training stability of existing judgment schemes. On the one hand, the use of a lightweight model reduces training and deployment costs, eliminating the need to rely on high-cost large models; on the other hand, the matching degree of answer superiority-inferiority is used as the reward signal, and the training objective is directly aligned with the core requirements of paired comparison judgment. At the same time, the use of group sampling to normalize the advantage value in conjunction with reinforcement learning updates stabilizes the training process. The resulting lightweight judgment model can be widely used in open-domain question answering evaluation, model answer comparison, automated evaluation, and other scenarios.

[0059] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A lightweight method for training a large language model evaluator based on reinforcement learning, characterized in that, include: Construct a pairwise answer evaluation data sample, wherein the evaluation data sample includes at least the user question, the first candidate answer, the second candidate answer, and the reference evaluation result; Based on the evaluation data sample, a referee prompt is generated, which is used to instruct the large language model to be trained to compare, analyze and score the two candidate answers; The judge's prompt is input into a lightweight large language model to obtain the model output, which includes the reasoning process and two scores corresponding to the first candidate answer and the second candidate answer, respectively. The two scores in the model output are analyzed, and the relative merits of the model's predicted answers are determined based on the relationship between the two scores. The reward is calculated based on whether the relationship between the superiority and inferiority of the answers is consistent with the relationship between the superiority and inferiority of the answers as indicated by the reference evaluation results; Group sampling is performed for the same referee's prompts, and the advantage value is calculated based on the mean and variance of the rewards in the same group. The lightweight large language model is updated using reinforcement learning based on the rewards and advantage values ​​to obtain the trained lightweight large language model evaluation model.

2. The method for training a lightweight large language model evaluator based on reinforcement learning as described in claim 1, characterized in that, The reference evaluation results include the scores of the first candidate answer and the second candidate answer given by the reference model or manual annotation. The two scores are used to determine three types of relationships, including the first candidate answer being better than the second candidate answer, the first candidate answer being worse than the second candidate answer, or the two being of comparable quality.

3. The method for training a lightweight large language model evaluator based on reinforcement learning as described in claim 1, characterized in that, The referee's prompts include system prompts and user prompts. The system prompts are used to define the evaluation criteria, output structure, and scoring range, while the user prompts are used to carry the user's question and the main text of the two candidate answers.

4. The method for training a lightweight large language model evaluator based on reinforcement learning as described in claim 1, characterized in that, The model output meets a preset format, which includes a reasoning field for carrying the reasoning analysis content and an answer field for carrying the two scores.

5. The method for training a lightweight large language model evaluator based on reinforcement learning as described in claim 1, characterized in that, The parsing model output includes: extracting the first and second predicted scores from the answer field, determining whether the two predicted scores are integers or numerical values ​​within the preset score range, and classifying the model output as invalid output if the output format does not meet the requirements, the score is missing, or the score exceeds the limit.

6. The method for training a lightweight large language model evaluator based on reinforcement learning as described in claim 1, characterized in that, The reward is determined by the following relationship: A positive reward is given when the model's predicted relationship between the quality of the answers matches the relationship between the quality of the answers as indicated by the reference evaluation results, and the model's output meets the preset format; otherwise, no positive reward is given.

7. The method for training a lightweight large language model evaluator based on reinforcement learning as described in claim 1, characterized in that, The reward The function is represented as: ； in, and These represent the first and second predicted scores in the output of the i-th model, respectively. and These represent the first and second reference scores in the evaluation results, respectively. This indicates whether the output of the i-th model satisfies the preset format and scoring range constraints. Indicates an indicator function, Represents a symbolic function.

8. The method for training a lightweight large language model evaluator based on reinforcement learning as described in claim 1, characterized in that, Multiple samples were taken from the same referee's prompt to obtain multiple model outputs and their corresponding rewards, and the advantage value was calculated based on the mean and standard deviation of the rewards of the same group of samples. The advantage value The calculation formula is: ； Where G represents the number of sampled outputs corresponding to the same referee's prompt. As a reward, It is a mean function. It is a function of standard deviation; The reward corresponding to the i-th sampled output and the relative advantage value of the group used by the t-th token in the i-th output are both calculated according to the above formula.

9. The method for training a lightweight large language model evaluator based on reinforcement learning as described in claim 1, characterized in that, The reinforcement learning update includes optimizing the objective using a GRPO relative policy, the GRPO objective function being expanded as follows: ； ； ； in, Let be the policy objective function to be maximized by the GRPO algorithm. For mathematical expectation, Provides suggestions for a single input field. Let G represent the distribution of all training prompt samples, and let G be the number of sampled outputs corresponding to the same referee prompt. For the same prompt q, generate G model output texts. For the old strategy model, Given the cue q, the model generates a conditional distribution for the output o. Let be the total token length of the i-th output text. KL penalty coefficient, For the group normalized dominance value, Output the t-th token for the i-th token. For PPO symmetric clipping hyperparameters, This is the KL divergence regularization term between the current model and the baseline model. To freeze the SFT baseline reference model.

10. The method for training a lightweight large language model evaluator based on reinforcement learning as described in claim 1, characterized in that, After the lightweight large language model is trained, a consistency index between the evaluation model and the reference evaluation result is calculated on the validation set, and model parameters for deployment or subsequent evaluation are selected based on the consistency index. The lightweight large language model evaluation model is used for at least one of the following tasks: open-domain question answering quality assessment, model answer comparison, automated evaluation, preference data filtering, and model training feedback.

Images