Reward model training method, answer evaluation method and device

Abstract

The invention provides a reward model training method and device and an answer evaluation method and device, and relates to the technical field of artificial intelligence. The reward model training method comprises the following steps: inputting an original training sample and a prompt text into an initial reward model to obtain a first reasoning principle and a first score output by the initial reward model; the prompt text comprises a reasoning principle generation requirement text, the reasoning principle generation requirement text comprises reasoning principle generation requirement information, and the reasoning principle is used for indicating reasoning logic which the initial reward model needs to follow during scoring; based on the original training sample, the first reasoning principle and the first score, finely adjusting the initial reward model to obtain a reward model; wherein the initial reward model is a pre-trained large language model. The performance of the reward model can be improved, and finally the performance of the answer generation model enhanced based on the reward model is improved, namely, the generation accuracy of the answer generation model is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a reward model training method, an answer evaluation method and a device. Background Art

[0002] Reinforced Learning with Human Feedback (RLHF) refers to the use of human feedback to optimize the learning process of the generative model, that is, to improve and optimize the performance of the model by introducing human feedback. This method combines the advantages of reinforcement learning and human supervision, enabling the model to better understand human intentions and preferences during the learning process.

[0003] At present, reinforcement learning based on human feedback usually relies on a large amount of manually annotated data, and then uses the collected human feedback data to train a reward model. However, the training of the reward model in the existing method relies on a large amount of manually annotated data, which is easily affected by human subjective bias, resulting in the performance of the generative model reinforced by the reward model is not high, that is, the generation accuracy of the generative model is not high; and the output of the reward model obtained through human feedback training only includes the score of the output answer of the generative model, that is, the output information of the reward model is small, resulting in the performance of the generative model reinforced by the reward model is not high, that is, the generation accuracy of the generative model is not high. Summary of the invention

[0004] The present invention provides a reward model training method, an answer evaluation method and a device, which are used to solve the defect of poor reward model training effect in the prior art, realize the training of a high-performance reward model, and further realize a high-performance generation model.

[0005] The present invention provides a training method for a reward model, comprising: Inputting the original training sample and the prompt text into the initial reward model, obtaining the first reasoning principle and the first score output by the initial reward model; the original training sample includes the original sample question and the original sample answer, the prompt text includes the reasoning principle generation requirement text, the reasoning principle generation requirement text includes the reasoning principle generation requirement information, and the reasoning principle is used to indicate the reasoning logic to be followed by the initial reward model when scoring; Fine-tuning the initial reward model based on the original training sample, the first reasoning principle, and the first score to obtain a reward model; Among them, the initial reward model is a pre-trained large language model; the reward model is used to evaluate the answers output by the answer generation model, and the answer generation model is used to generate answers corresponding to input questions.

[0006] According to a training method for a reward model provided by the present invention, the original training sample also includes a scoring label corresponding to the original sample answer; The fine-tuning of the initial reward model based on the original training sample, the first reasoning principle and the first score comprises: When the first score is consistent with the score label, generating fine-tuning sample data based on the original training sample, the first reasoning principle and the first score; In the case where the first score is inconsistent with the score label, the original training sample, the prompt text and the score label are input into the initial reward model to obtain a second reasoning principle and a second score output by the initial reward model, and fine-tuning sample data is generated based on the original training sample, the second reasoning principle and the second score; The initial reward model is fine-tuned based on the fine-tuning sample data.

[0007] According to a reward model training method provided by the present invention, the initial reward model is fine-tuned based on the fine-tuning sample data, including: Inputting the original training samples in the fine-tuning sample data into the initial reward model to obtain the prediction reasoning principle and prediction score output by the initial reward model; Determine the reasoning principle in the fine-tuning sample data as a true reasoning principle label, and determine the score in the fine-tuning sample data as a true score label; The initial reward model is fine-tuned based on a difference between the predicted inference principle and the true inference principle label, and a difference between the predicted score and the true score label.

[0008] According to a method for training a reward model provided by the present invention, the original training sample and the prompt text are input into the initial reward model to obtain a first reasoning principle and a first score output by the initial reward model, including: Obtaining an original sample set; the original sample set includes a plurality of the original training samples and a plurality of original samples, any of the original samples including an original sample question and an original sample answer; For each of the original training samples, input the original training sample and the prompt text into an initial reward model to obtain a first reasoning principle and a first score output by the initial reward model; The prompt text is determined based on the plurality of original samples.

[0009] According to a reward model training method provided by the present invention, the prompt text also includes an output format requirement text of the initial reward model, and the output format requirement text is used to indicate the output format of the reasoning principle and the output format of the score.

[0010] According to a reward model training method provided by the present invention, the prompt text also includes output example text of the initial reward model, and the output example text includes an output example of the reasoning principle and an output example of the score.

[0011] The present invention also provides an answer evaluation method, comprising: Get the output answer of the answer generation model to the input question; The input question and the output answer are input into a reward model to obtain an evaluation result of the output answer output by the reward model, wherein the reward model is trained based on any of the reward model training methods described above.

[0012] The present invention also provides a training device for a reward model, comprising: An input module, used to input the original training sample and the prompt text into the initial reward model, and obtain the first reasoning principle and the first score output by the initial reward model; the original training sample includes the original sample question and the original sample answer, the prompt text includes the reasoning principle generation requirement text, the reasoning principle generation requirement text includes the reasoning principle generation requirement information, and the reasoning principle is used to indicate the reasoning logic that the initial reward model needs to follow when scoring; A fine-tuning module, configured to fine-tune the initial reward model based on the original training sample, the first reasoning principle and the first score to obtain a reward model; Among them, the initial reward model is a pre-trained large language model; the reward model is used to evaluate the answers output by the answer generation model, and the answer generation model is used to generate answers corresponding to input questions.

[0013] The present invention also provides an answer evaluation device, comprising: An acquisition module, used for acquiring an output answer to an input question output by an answer generation model; An evaluation module is used to input the input question and the output answer into a reward model to obtain an evaluation result of the output answer output by the reward model, wherein the reward model is trained based on any of the reward model training methods described above.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the processor implements a training method for a reward model as described above or implements an answer evaluation method as described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a training method for a reward model as described in any one of the above or implements an answer evaluation method as described in any one of the above.

[0016] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements a training method for a reward model as described in any one of the above or implements an answer evaluation method as described in any one of the above.

[0017] The training method, answer evaluation method and device of the reward model provided by the present invention input the original training sample and the prompt text into the initial reward model to obtain the first reasoning principle and the first score output by the initial reward model, and the prompt text includes the reasoning principle generation requirement text, and the reasoning principle generation requirement text includes the reasoning principle generation requirement information, and the reasoning principle is used to indicate the reasoning logic that the initial reward model must follow when scoring, so that based on the prompt text, the initial reward model can be guided to generate scores according to the established logic, thereby improving the accuracy and consistency of the scores, and then improving the performance of the reward model, and finally improving the performance of the answer generation model strengthened by the reward model, that is, improving the generation accuracy of the answer generation model; and based on the original training sample, the first reasoning principle and the first score, the initial reward model is fine-tuned to obtain the reward model, so that the trained reward model can not only output the answer The score corresponding to the case can also be output to infer the reasoning principle of the score, thereby providing a more comprehensive reward signal to the answer generation model, thereby improving the performance of the answer generation model strengthened by the reward model, that is, improving the generation accuracy of the answer generation model; at the same time, fine-tuning the initial reward model based on the first reasoning principle can continuously stimulate the reasoning ability of the reward model, thereby improving the performance of the reward model, so that the reward model provides a denser and more accurate reward signal to the answer generation model, thereby improving the performance of the answer generation model strengthened by the reward model, that is, improving the generation accuracy of the answer generation model; and the initial reward model is a pre-trained large language model, so that the training of the reward model does not need to rely on a large amount of manually labeled data, so it is not easily affected by human subjective bias, thereby improving the performance of the answer generation model strengthened by the reward model, that is, improving the generation accuracy of the answer generation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is one of the flow charts of the training method of the reward model provided by the present invention.

[0020] Figure 2 This is the second flow chart of the training method of the reward model provided by the present invention.

[0021] Figure 3 This is the third flow chart of the training method of the reward model provided by the present invention.

[0022] Figure 4 It is a flow chart of the answer evaluation method provided by the present invention.

[0023] Figure 5 It is a structural schematic diagram of the training device of the reward model provided by the present invention.

[0024] Figure 6 It is a structural schematic diagram of the answer evaluation device provided by the present invention.

[0025] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] Reinforced Learning with Human Feedback (RLHF) refers to the use of human feedback to optimize the learning process of the generative model, that is, to improve and optimize the performance of the model by introducing human feedback. This method combines the advantages of reinforcement learning and human supervision, enabling the model to better understand human intentions and preferences during the learning process.

[0028] Especially with the rapid development of artificial intelligence technology, the application scope of large models is becoming more and more extensive. In the field of reinforcement learning based on human feedback, how to effectively use human feedback to optimize the learning process of large models has become a key challenge. In this learning process, the learning of the reward model has become a crucial link, which determines the optimization direction of the large model in reinforcement learning.

[0029] At present, reinforcement learning based on human feedback usually relies on a large amount of manually annotated data, and then uses the collected human feedback data to train a reward model. However, the training of the reward model in the existing method relies on a large amount of manually annotated data, which is easily affected by human subjective bias, resulting in the performance of the generative model reinforced based on the reward model is not high, that is, the generation accuracy of the generative model is not high; and the output of the reward model obtained through human feedback training only includes the score of the output answer of the generative model, that is, the output information of the reward model is small, that is, the reward signal output by the reward model is sparse, which results in the performance of the generative model reinforced based on the reward model is not high, that is, the generation accuracy of the generative model is not high.

[0030] For example, using reinforcement learning based on human feedback, the partial order of the reward model is mainly used to learn reward signals from human feedback. The specific process is as follows: First, by comparing multiple outputs generated by the generative model, humans evaluate the quality of these outputs; according to human feedback, the outputs are sorted according to the relationship of quality to form a partial order, which indicates which outputs are more popular than other outputs; these partial order data are used to train the reward model so that it can predict the relative quality of the output based on the features; the generative model optimizes its performance by maximizing the reward signal consistent with the human evaluation. And the reward model is usually a classification model or regression model that aims to predict the quality score of a given output.

[0031] In addition, human feedback-based reinforcement learning directly uses human feedback as training data, which helps the reward model align with human values ​​and preferences because it directly reflects human choices and preferences. However, since human preferences are affected by personal subjective factors, resulting in biases in training data, obtaining high-quality human preference labels is a key bottleneck, which may involve high labor costs, resulting in excessively high training costs for reward models.

[0032] Even if the method of reinforcement learning with AI Feedback (RLAIF) is used, that is, the feedback of the reward signal is mainly realized through a pre-trained large language model, the specific process is as follows: first, a pre-trained large model is selected, that is, a large language model with strong performance is used as a benchmark to evaluate the output generated by the generative model; the feedback provided by the large language model is used to guide the training process of the generative model, and the output of the generative model is optimized to better meet the expected quality standards. However, the output of the reward model trained in this way only includes the score of the output answer of the generative model, that is, the output information of the reward model is small, that is, the reward signal output by the reward model is sparse, which leads to the performance of the generative model reinforced by the reward model is not high, that is, the generation accuracy of the generative model is not high; and the effect of RLAIF depends on the quality of the pre-trained large model used to generate the score. If the pre-trained large model itself is biased or inaccurate, it may affect the final result, that is, the performance of the generative model reinforced by the reward model is not high, that is, the generation accuracy of the generative model is not high.

[0033] In view of the above problems, the present invention proposes the following embodiments. Figure 1-Figure 3 Describe the training method of the reward model of the present invention. The embodiment of the present invention can be applicable to the scenario of evaluating the text output by any generation model, and in particular, it can be applicable to the scenario of evaluating the factual correctness or knowledge correctness of the output answer of a large language model. The executor of the training method of the reward model can be an electronic device such as a computer, a terminal device, a server, a server cluster, or a specially designed reward model training device, or it can be a reward model training device set in the electronic device. The reward model training device can be implemented by software, hardware, or a combination of both.

[0034] Figure 1 is one of the flow charts of the training method of the reward model provided by the present invention, such as Figure 1 As shown, the training method of the reward model includes the following steps 110 and 120.

[0035] Step 110: input the original training sample and the prompt text into the initial reward model to obtain the first reasoning principle and the first score output by the initial reward model.

[0036] The original training samples include original sample questions and original sample answers.

[0037] The original sample question can be a question collected by the user input to the answer generation model to ask, or it can be other questions collected from the Internet. The original sample answer is the answer obtained after the original sample question is input into the answer generation model, that is, there is an association relationship (correspondence relationship) between the original sample question and the original sample answer in an original training sample.

[0038] The prompt text includes a reasoning principle generation requirement text, the reasoning principle generation requirement text includes generation requirement information of the reasoning principle, and the reasoning principle is used to indicate the reasoning logic that the initial reward model needs to follow when scoring.

[0039] Here, the reasoning principle generation requirement text is the core of the text prompt project. The reasoning principle generation requirement information defines the reasoning logic that the initial reward model must follow when scoring. Furthermore, the reasoning logic can include multiple dimensions, such as accuracy, richness, simplicity, etc. Furthermore, the selection of specific dimensions depends on the type of data that needs to be scored, that is, different prompt texts correspond to different task scenarios, that is, the prompt text is the prompt text corresponding to the original training sample. These dimensions together constitute the basic principles of scoring, which aim to guide the initial reward model to generate scoring criteria according to the established logic. And the reasoning principle generation requirement text is input into the initial reward model to prompt the initial reward model how to output the corresponding reasoning principle.

[0040] It should be understood that based on the prompt text, the initial reward model can be guided to generate scores according to a predetermined logic, thereby improving the accuracy and consistency of the scores.

[0041] It should be understood that users can improve the output content of the reward model according to actual needs by setting and adjusting the prompt text, that is, these text-based feedback can also provide a basis for subsequent use of the reward model to improve data quality.

[0042] The initial reward model is a pre-trained large language model, which may be, for example, the iFlytek Spark model or other large language models capable of generating reasoning principles and scoring.

[0043] Based on this, since the initial reward model is already a pre-trained large language model, the training of the reward model does not need to rely on a large amount of manually annotated data, so it is not easily affected by human subjective bias, thereby improving the performance of the generative model reinforced by the reward model, that is, improving the generation accuracy of the generative model. In other words, you only need to fine-tune the initial reward model later to get a high-performance reward model.

[0044] Here, the first reasoning principle is used to indicate the reasoning logic followed by the initial reward model to score the first score. Since the reasoning principle generation requirement text is given to the initial reward model as a prompt, the initial reward model can know how to output the first reasoning principle.

[0045] It should be noted that since the input reasoning principle generates a requirement text, and the reasoning principle generation requirement text includes the generation requirement information of the reasoning principle, and the reasoning principle is used to indicate the reasoning logic that the initial reward model must follow when scoring, based on this, the first score output by the initial reward model can include dimensional scores of multiple dimensions, and is not limited to a comprehensive score, thereby providing the generation model with a more comprehensive reward signal, thereby improving the performance of the generation model based on the reward model reinforcement, that is, improving the generation accuracy of the generation model.

[0046] In a specific embodiment, multiple original training samples are obtained; for each original training sample, the original training sample and the prompt text are input into the initial reward model to obtain the first reasoning principle and the first score output by the initial reward model. In other words, each original training sample (each question-answer pair) and the prompt text are input into the initial reward model (the large language model to be fine-tuned) to obtain the reasoning principle and score pair (i.e., the first reasoning principle and the first score) corresponding to each original training sample output by the initial reward model.

[0047] Step 120: fine-tune the initial reward model based on the original training sample, the first reasoning principle and the first score to obtain a reward model.

[0048] Specifically, the original training samples are used as input samples of the initial reward model, the first reasoning principle and the first score are used as labels of the input samples, and the initial reward model is fine-tuned.

[0049] In a specific embodiment, the original training sample is input into the initial reward model to obtain the predicted reasoning principle and predicted score output by the initial reward model; the first reasoning principle is determined as the real reasoning principle label, and the first score is determined as the real score label; based on the difference between the predicted reasoning principle and the real reasoning principle label, and the difference between the predicted score and the real score label, the initial reward model is fine-tuned. Further, the original training sample and the prompt text are input into the initial reward model to obtain the predicted reasoning principle and predicted score output by the initial reward model.

[0050] It should be noted that the number of the original training samples can be multiple, based on which, based on multiple original training samples, multiple first reasoning principles and multiple first scores, the initial reward model is fine-tuned, that is, the model parameters of the initial reward model are adjusted iteratively, thereby continuously improving the scoring performance of the reward model. In addition, it should be noted that even if the number of original training samples is multiple, since the task of the answer generation model is the same, the number of prompt texts can be only one, that is, different original training samples and the same prompt text are input into the initial reward model.

[0051] Here, fine-tuning the initial reward model means adjusting the model parameters of the initial reward model to train the reward model. In one embodiment, the initial reward model is fine-tuned using LoRA (Low-Rank Adaptation) fine-tuning technology.

[0052] The reward model is used to evaluate the answer output by the answer generation model, and the answer generation model is used to generate the answer corresponding to the input question. The evaluation result of the reward model evaluating the answer output by the answer generation model may include a reasoning principle similar to the first reasoning principle and a score similar to the first score.

[0053] In one embodiment, the answer generation model may be a pre-trained large language model. In another embodiment, the answer generation model is trained based on sample questions and their corresponding answer labels.

[0054] It should be noted that based on the trained reward model, under the framework of reinforcement learning, the answer generation model generates a new output answer and obtains feedback from the reward model, and continues to train and improve the answer generation model based on the feedback, so that the reward model can continuously optimize the output of the answer generation model.

[0055] It should be understood that the reward model trained in the above manner can not only output the score corresponding to the answer, but also output the reasoning principle for inferring the score, so that it can output specific analysis information on the quality of the answer response, so as to provide a more direct and comprehensive improvement signal to the answer generation model, that is, to provide the generation model with a more comprehensive reward signal, thereby improving the performance of the generation model strengthened by the reward model, that is, to improve the generation accuracy of the generation model.

[0056] It should be understood that fine-tuning the initial reward model based on the first reasoning principle can continuously stimulate the reasoning ability of the reward model, thereby improving the performance of the reward model, so that the reward model can provide a denser and more accurate reward signal to the answer generation model, thereby improving the performance of the generation model reinforced by the reward model, that is, improving the generation accuracy of the generation model.

[0057] It should be understood that since the initial reward model is a pre-trained large language model, it is only necessary to fine-tune the initial reward model based on a small amount of original training samples, thereby reducing the cost of acquiring training samples and further reducing the training cost of the reward model.

[0058] It should be understood that even if the initial reward model is a pre-trained large language model, and the pre-trained large language model itself has deviations or inaccuracies, the embodiment of the present invention will fine-tune the pre-trained large language model in the above manner, thereby improving the robustness of the reward model, and then improving the performance of the generative model reinforced by the reward model, that is, improving the generation accuracy of the generative model.

[0059] The training method of the reward model provided by the embodiment of the present invention inputs the original training sample and the prompt text into the initial reward model to obtain the first reasoning principle and the first score output by the initial reward model, and the prompt text includes the reasoning principle generation requirement text, and the reasoning principle generation requirement text includes the reasoning principle generation requirement information, and the reasoning principle is used to indicate the reasoning logic that the initial reward model must follow when scoring, so that based on the prompt text, the initial reward model can be guided to generate scores according to the established logic, thereby improving the accuracy and consistency of the scores, and then improving the performance of the reward model, and finally improving the performance of the answer generation model strengthened by the reward model, that is, improving the generation accuracy of the answer generation model; and based on the original training sample, the first reasoning principle and the first score, fine-tune the initial reward model to obtain the reward model, so that the trained reward model can not only output the answer corresponding to the answer, but also can improve the performance of the reward model, and finally improve the performance of the answer generation model strengthened by the reward model, that is, improve the generation accuracy of the answer generation model; and fine-tune the initial reward model based on the original training sample, the first reasoning principle and the first score ..., and finally improve the performance of the reward model, so that the reward model can generate the answer corresponding to the answer, and finally improve the performance of the reward model, and finally improve the performance of the reward model, and finally improve the performance of the reward model, and finally improve the performance of the reward model, and finally improve the performance of the reward The score can also be outputted to obtain the reasoning principle of the score, thereby providing a more comprehensive reward signal to the answer generation model, thereby improving the performance of the answer generation model strengthened by the reward model, that is, improving the generation accuracy of the answer generation model; at the same time, fine-tuning the initial reward model based on the first reasoning principle can continuously stimulate the reasoning ability of the reward model, thereby improving the performance of the reward model, so that the reward model provides a denser and more accurate reward signal to the answer generation model, thereby improving the performance of the answer generation model strengthened by the reward model, that is, improving the generation accuracy of the answer generation model; and the initial reward model is a pre-trained large language model, so that the training of the reward model does not need to rely on a large amount of manually annotated data, so it is not easily affected by human subjective bias, thereby improving the performance of the answer generation model strengthened by the reward model, that is, improving the generation accuracy of the answer generation model.

[0060] Based on any of the above embodiments, in the method, the original training sample also includes a scoring label corresponding to the original sample answer.

[0061] It should be noted that the score labels can be annotated manually, that is, human feedback data on the original sample answers is collected. It should be understood that since the initial reward model is a pre-trained large language model, it is only necessary to fine-tune the initial reward model based on a small number of original training samples. Therefore, even if the score labels still need to be annotated, there is no need to consume a lot of annotation costs, thereby reducing the training cost of the reward model.

[0062] Accordingly, Figure 2 This is a second flow chart of the training method of the reward model provided by the present invention, such as Figure 2 As shown, the above step 120 includes step 121, step 122 and step 123.

[0063] Step 121 : when the first score is consistent with the score label, generate fine-tuning sample data based on the original training sample, the first reasoning principle and the first score.

[0064] It should be noted that when the first score is consistent with the score label, it means that the first score output by the initial reward model is correct, and the first score is scored based on the first reasoning principle, so the first reasoning principle is also correct. Based on this, the fine-tuning sample data is generated directly based on the original training sample, the first reasoning principle and the first score.

[0065] Specifically, the original training sample is used as the input sample of the initial reward model, and the first reasoning principle and the first score are used as the label of the input sample. That is, the fine-tuning sample data includes the original training sample, the first reasoning principle and the first score.

[0066] Step 122, when the first score is inconsistent with the score label, the original training sample, the prompt text and the score label are input into the initial reward model to obtain a second reasoning principle and a second score output by the initial reward model, and fine-tuning sample data is generated based on the original training sample, the second reasoning principle and the second score.

[0067] Considering the limitation of generating fine-tuning sample data based only on step 121, that is, training is performed only on training samples with correct answers, the improvement stops when the reward model cannot solve new problems in the training set. This is essentially because no training signal can be obtained from failed examples. Based on this, in the embodiment of the present invention, when the first score is inconsistent with the score label, that is, for those cases in which errors occur in the initial reward model in terms of reasoning principle generation and score generation, the correct score label is provided as prompt information to the initial reward model, and the initial reward model is guided to generate a new second reasoning principle and a new second score.

[0068] It should be noted that since the initial reward model is a pre-trained large language model, and given that the large language model has the ability of reverse reasoning, the correct score label is provided to the initial reward model as prompt information, so that the initial reward model can more easily generate the correct second reasoning principle and second score.

[0069] Specifically, the original training sample is used as the input sample of the initial reward model, and the second reasoning principle and the second score are used as the label of the input sample. That is, the fine-tuning sample data includes the original training sample, the second reasoning principle and the second score.

[0070] It should be understood that, through the above method, additional training data can be provided for the initial reward model, and these training data come from the data that the initial reward model failed to process correctly. Based on this, not only can fine-tuning sample data be generated through step 121, but also fine-tuning sample data can be generated through step 122, thereby forming a more comprehensive and diverse fine-tuning sample data set, that is, the expansion of fine-tuning sample data can be achieved. This new fine-tuning sample data set will be used to fine-tune the initial reward model in the hope of improving its performance and accuracy when dealing with new problems, thereby improving the robustness of the reward model, and then improving the performance of the generative model based on the reward model reinforcement, that is, improving the generation accuracy of the generative model.

[0071] Considering that RLAIF needs to use a ready-made pre-trained large model to mark preferences instead of relying on humans, RLAIF can reduce the cost of data collection due to the reduced dependence on human annotation. However, the preferences generated by the large model may deviate from the true preferences of humans, resulting in low performance of the reward model. Based on this, the score label corresponding to the original sample answer is used to judge whether the first score and the first reasoning principle output by the initial reward model are correct. If not, the score label is input as a prompt to the initial reward model, so as to finally obtain the second score that is consistent with the true preference of humans, thereby improving the robustness of the reward model, and then improving the performance of the generative model based on the reinforcement of the reward model, that is, improving the generation accuracy of the generative model.

[0072] Step 123: fine-tune the initial reward model based on the fine-tuning sample data.

[0073] It should be noted that the number of the fine-tuning sample data can be multiple. Based on this, the initial reward model is fine-tuned based on multiple fine-tuning sample data, that is, the model parameters of the initial reward model are continuously iteratively adjusted, so as to continuously improve the scoring performance of the reward model.

[0074] In the training method of the reward model provided by an embodiment of the present invention, the original training sample also includes a score label corresponding to the original sample answer, so that when the first score is consistent with the score label, fine-tuning sample data is generated based on the original training sample, the first reasoning principle and the first score, and when the first score is inconsistent with the score label, the original training sample, the prompt text and the score label are input into the initial reward model to obtain the second reasoning principle and the second score output by the initial reward model, and the fine-tuning sample data is generated based on the original training sample, the second reasoning principle and the second score, so as to finally obtain a score consistent with the real preference of human beings, thereby improving the robustness of the reward model, and then improving the performance of the generative model based on the reward model reinforcement, that is, improving the generation accuracy of the generative model; at the same time, when the first score is inconsistent with the score label, fine-tuning sample data is also generated based on the original training sample, the second reasoning principle and the second score, so as to form a more comprehensive and diversified fine-tuning sample data set, that is, the expansion of the fine-tuning sample data can be achieved, thereby improving the robustness of the reward model, and then improving the performance of the generative model based on the reward model reinforcement, that is, improving the generation accuracy of the generative model.

[0075] Based on any of the above embodiments, in the method, the above step 123 includes: Inputting the original training samples in the fine-tuning sample data into the initial reward model to obtain the prediction reasoning principle and prediction score output by the initial reward model; Determine the reasoning principle in the fine-tuning sample data as a true reasoning principle label, and determine the score in the fine-tuning sample data as a true score label; The initial reward model is fine-tuned based on a difference between the predicted inference principle and the true inference principle label, and a difference between the predicted score and the true score label.

[0076] Furthermore, the original training samples and prompt texts are input into the initial reward model to obtain the prediction reasoning principle and prediction score output by the initial reward model.

[0077] Exemplarily, the initial reward model is fine-tuned in an autoregressive manner. This fine-tuning usually adjusts the model parameters by minimizing the loss function. The basic principle is to use the obtained fine-tuning sample data to train the initial reward model, with the goal of reducing the difference between the predicted value and the true label. Commonly used loss functions include cross entropy loss.

[0078] The training method of the reward model provided by the embodiment of the present invention, since the reasoning principles and scores in the fine-tuning sample data are correct, that is, the scores included therein are consistent with the real preferences of humans, thereby determining the reasoning principles in the fine-tuning sample data as the real reasoning principle labels, and determining the scores in the fine-tuning sample data as the real score labels to fine-tune the initial reward model, can improve the robustness of the reward model, and further improve the performance of the generative model strengthened based on the reward model, that is, improve the generation accuracy of the generative model.

[0079] Based on any of the above embodiments, in the method, the above step 110 includes: Obtaining an original sample set; the original sample set includes a plurality of the original training samples and a plurality of original samples, any of the original samples including an original sample question and an original sample answer; For each of the original training samples, the original training sample and the prompt text are input into an initial reward model to obtain a first reasoning principle and a first score output by the initial reward model.

[0080] The prompt text is determined based on the plurality of original samples.

[0081] In other words, some samples in the original sample set are used as original training samples to fine-tune the initial reward model, and some samples in the original sample set are used as original samples to determine the prompt text. That is, the text prompt project is designed using a few-sample prompt method, which can not only guide the initial reward model to score specific data, but also ensure that a certain amount of original training samples are used to fine-tune the initial reward model.

[0082] The training method of the reward model provided in an embodiment of the present invention can obtain original training samples for fine-tuning based on the original sample set, and can also obtain original samples for determining prompt texts through the above-mentioned method. Since the original sample set is a sample set for the same task scenario, the accuracy of determining the prompt text is improved, that is, it is ensured that the prompt text is the prompt text for the current task scenario, thereby improving the robustness of the reward model, and then improving the performance of the generation model strengthened based on the reward model, that is, improving the generation accuracy of the generation model.

[0083] Based on any of the above embodiments, in this method, the prompt text also includes an output format requirement text of the initial reward model, and the output format requirement text is used to indicate the output format of the reasoning principle and the output format of the score.

[0084] Here, the output format requirement text is used to prompt the initial reward model to generate reasoning principles and scores according to the specified output format. Based on this, a unified output format is not only conducive to the subsequent fine-tuning of the initial reward model, but also improves the readability and consistency of the reward model output.

[0085] In the training method of the reward model provided by an embodiment of the present invention, the prompt text also includes an output format requirement text of the initial reward model, and the output format requirement text is used to indicate the output format of the reasoning principle and the output format of the score, thereby improving the accuracy and consistency of the output reasoning principle and the score, thereby improving the performance of the reward model, and ultimately improving the performance of the answer generation model strengthened based on the reward model, that is, improving the generation accuracy of the answer generation model.

[0086] Based on any of the above embodiments, in the method, the prompt text also includes output example text of the initial reward model, and the output example text includes an output example of the reasoning principle and an output example of the score.

[0087] It should be noted that in order to ensure that the initial reward model can accurately understand and apply the reasoning principles to generate requirement text and output format text, a small number of examples written by human experts can be introduced, that is, output examples of reasoning principles and output examples of scores written by human experts, that is, these examples include descriptions of reasoning principles and corresponding scoring examples, which provide specific references for the initial reward model and help the initial reward model better understand and learn how to score according to the requirements of the prompt text.

[0088] In the training method for a reward model provided by an embodiment of the present invention, the prompt text also includes an output example text of the initial reward model, and the output example text includes an output example of the reasoning principle and an output example of the score, thereby helping the initial reward model to better understand and learn how to score according to the requirements of the prompt text, thereby improving the accuracy and consistency of the score, and then improving the performance of the reward model, and ultimately improving the performance of the answer generation model strengthened based on the reward model, that is, improving the generation accuracy of the answer generation model.

[0089] To facilitate understanding of the above embodiments, a specific embodiment is described herein. Figure 3 As shown, the original sample question, the original sample answer and the prompt text are first input into the large language model (initial reward model) to obtain the reasoning principle and score output by the large language model. When the output of the large language model is correct, that is, when the score is consistent with the score label, fine-tuning sample data is generated based on the original sample question, the original sample answer, the reasoning principle and the score to fine-tune the large language model. When the output of the large language model is incorrect, that is, when the score is inconsistent with the score label, the original sample question, the original sample answer, the prompt text and the score label are input into the large language model again to obtain the reasoning principle and score output by the large language model. Based on the original sample question, the original sample answer, the reasoning principle and the score, fine-tuning sample data is generated to fine-tune the large language model.

[0090] Based on the above embodiments, the present invention proposes a reward model learning method based on the principle of self-learning reasoning, which can stimulate the reasoning ability of the reward model, thereby providing a denser and more accurate reward signal. Specifically, the reward model learning method based on self-learning reasoning proposed by the present invention is a scalable bootstrap method that allows the reward model to learn to generate its own reasoning principles, and also learn how to score according to the reasoning process. More specifically, the following process is repeated: in each iteration, first, fine-tuning sample data is generated by trying to use the reasoning principle generation ability of the initial reward model to construct a fine-tuning sample data set; then, this fine-tuning sample data set is expanded using rationalization screening, and the reasoning principle is corrected for the problem that the initial reward model fails to score correctly; finally, the initial reward model is fine-tuned on the fine-tuning sample data set to obtain a reward model.

[0091] Based on any of the above embodiments, the present invention also provides an answer evaluation method. Figure 4 It is a flow chart of the answer evaluation method provided by the present invention, such as Figure 4 As shown, the answer evaluation method includes the following steps 410 and 420.

[0092] Step 410, obtaining the output answer to the input question output by the answer generation model.

[0093] Specifically, after obtaining the input question input by the user, the input question can be input into the answer generation model, so as to obtain the output answer output by the answer generation model.

[0094] In one embodiment, the answer generation model may be a pre-trained large language model. In another embodiment, the answer generation model is trained based on sample questions and their corresponding answer labels.

[0095] Step 420: input the input question and the output answer into a reward model to obtain an evaluation result of the output answer output by the reward model.

[0096] Wherein, the reward model is trained based on the reward model training method described in any of the above embodiments.

[0097] Here, the evaluation result may include a reasoning principle similar to the first reasoning principle and a rating similar to the first rating.

[0098] In other words, based on the trained reward model, under the framework of reinforcement learning, the answer generation model generates new output answers and obtains feedback from the reward model, and continues to train and improve the answer generation model based on the feedback, so that the reward model can continuously optimize the output of the answer generation model.

[0099] The answer evaluation method provided by the embodiment of the present invention obtains the output answer for the input question output by the answer generation model, inputs the input question and the output answer into the reward model, and obtains the evaluation result of the output answer output by the reward model. And because the reward model is trained based on the training method of the reward model as described in any of the above embodiments, during the training process of the reward model, the original training sample and the prompt text are input into the initial reward model to obtain the first reasoning principle and the first score output by the initial reward model, and the prompt text includes the reasoning principle generation requirement text, and the reasoning principle generation requirement text includes the reasoning principle generation requirement information, and the reasoning principle is used to indicate the reasoning logic that the initial reward model must follow when scoring, so that based on the prompt text, the initial reward model can be guided to generate scores according to the established logic, thereby improving the accuracy and consistency of the scores, and then improving the performance of the reward model, and finally improving the performance of the answer generation model based on the reward model reinforcement, that is, improving the generation accuracy of the answer generation model; and based on the original training sample, the first reasoning principle and the first score, the initial reward model is fine-tuned to obtain the reward model, so as to train the obtained The reward model can not only output the score corresponding to the answer, but also output the reasoning principle for inferring the score, thereby providing a more comprehensive reward signal to the answer generation model, thereby improving the performance of the answer generation model strengthened by the reward model, that is, improving the generation accuracy of the answer generation model; at the same time, fine-tuning the initial reward model based on the first reasoning principle can continuously stimulate the reasoning ability of the reward model, thereby improving the performance of the reward model, so that the reward model provides a denser and more accurate reward signal to the answer generation model, thereby improving the performance of the answer generation model strengthened by the reward model, that is, improving the generation accuracy of the answer generation model; and the initial reward model is a pre-trained large language model, so that the training of the reward model does not need to rely on a large amount of manually annotated data, and is not easily affected by human subjective bias, thereby improving the performance of the answer generation model strengthened by the reward model, that is, improving the generation accuracy of the answer generation model.

[0100] The training device of the reward model provided by the present invention is described below. The training device of the reward model described below and the training method of the reward model described above can be referred to each other.

[0101] Figure 5 is a schematic diagram of the structure of the training device of the reward model provided by the present invention, such as Figure 5 As shown, the training device of the reward model includes an input module 510 and a fine-tuning module 520.

[0102] The input module 510 is used to input the original training sample and the prompt text into the initial reward model to obtain the first reasoning principle and the first score output by the initial reward model; the original training sample includes the original sample question and the original sample answer, the prompt text includes the reasoning principle generation requirement text, the reasoning principle generation requirement text includes the reasoning principle generation requirement information, and the reasoning principle is used to indicate the reasoning logic that the initial reward model must follow when scoring.

[0103] The fine-tuning module 520 is used to fine-tune the initial reward model based on the original training sample, the first reasoning principle and the first score to obtain a reward model.

[0104] Among them, the initial reward model is a pre-trained large language model; the reward model is used to evaluate the answers output by the answer generation model, and the answer generation model is used to generate answers corresponding to input questions.

[0105] The training device of the reward model provided by the embodiment of the present invention inputs the original training sample and the prompt text into the initial reward model to obtain the first reasoning principle and the first score output by the initial reward model, and the prompt text includes the reasoning principle generation requirement text, and the reasoning principle generation requirement text includes the generation requirement information of the reasoning principle, and the reasoning principle is used to indicate the reasoning logic that the initial reward model must follow when scoring, so that based on the prompt text, the initial reward model can be guided to generate scores according to the established logic, thereby improving the accuracy and consistency of the scores, and then improving the performance of the reward model, and finally improving the performance of the answer generation model strengthened by the reward model, that is, improving the generation accuracy of the answer generation model; and based on the original training sample, the first reasoning principle and the first score, the initial reward model is fine-tuned to obtain the reward model, so that the trained reward model can not only output the answer corresponding to the answer, but also can output the answer corresponding to the answer corresponding to the answer. The score can also be outputted to obtain the reasoning principle of the score, thereby providing a more comprehensive reward signal to the answer generation model, thereby improving the performance of the answer generation model strengthened by the reward model, that is, improving the generation accuracy of the answer generation model; at the same time, fine-tuning the initial reward model based on the first reasoning principle can continuously stimulate the reasoning ability of the reward model, thereby improving the performance of the reward model, so that the reward model provides a denser and more accurate reward signal to the answer generation model, thereby improving the performance of the answer generation model strengthened by the reward model, that is, improving the generation accuracy of the answer generation model; and the initial reward model is a pre-trained large language model, so that the training of the reward model does not need to rely on a large amount of manually annotated data, so it is not easily affected by human subjective bias, thereby improving the performance of the answer generation model strengthened by the reward model, that is, improving the generation accuracy of the answer generation model.

[0106] Based on any of the above embodiments, the original training sample also includes a scoring label corresponding to the original sample answer; the fine-tuning module 520 is specifically used for: When the first score is consistent with the score label, generating fine-tuning sample data based on the original training sample, the first reasoning principle and the first score; In the case where the first score is inconsistent with the score label, the original training sample, the prompt text and the score label are input into the initial reward model to obtain a second reasoning principle and a second score output by the initial reward model, and fine-tuning sample data is generated based on the original training sample, the second reasoning principle and the second score; The initial reward model is fine-tuned based on the fine-tuning sample data.

[0107] Based on any of the above embodiments, the fine-tuning module 520 is specifically used for: Inputting the original training samples in the fine-tuning sample data into the initial reward model to obtain the prediction reasoning principle and prediction score output by the initial reward model; Determine the reasoning principle in the fine-tuning sample data as a true reasoning principle label, and determine the score in the fine-tuning sample data as a true score label; The initial reward model is fine-tuned based on a difference between the predicted inference principle and the true inference principle label, and a difference between the predicted score and the true score label.

[0108] Based on any of the above embodiments, the input module 510 is specifically used for: Obtaining an original sample set; the original sample set includes a plurality of the original training samples and a plurality of original samples, any of the original samples including an original sample question and an original sample answer; For each of the original training samples, input the original training sample and the prompt text into an initial reward model to obtain a first reasoning principle and a first score output by the initial reward model; The prompt text is determined based on the plurality of original samples.

[0109] Based on any of the above embodiments, the prompt text further includes an output format requirement text of the initial reward model, and the output format requirement text is used to indicate an output format of the reasoning principle and an output format of the score.

[0110] Based on any of the above embodiments, the prompt text also includes output example text of the initial reward model, and the output example text includes an output example of the reasoning principle and an output example of the score.

[0111] The answer evaluation device provided by the present invention is described below. The answer evaluation device described below and the answer evaluation method described above can be referenced to each other.

[0112] Figure 6 is a schematic diagram of the structure of the answer evaluation device provided by the present invention, such as Figure 6 As shown, the answer evaluation device includes an acquisition module 610 and an evaluation module 620.

[0113] The acquisition module 610 is used to obtain the output answer to the input question output by the answer generation model.

[0114] The evaluation module 620 is used to input the input question and the output answer into the reward model to obtain the evaluation result of the output answer output by the reward model, and the reward model is trained based on the training method of the reward model described in any of the above embodiments.

[0115] The answer evaluation device provided in the embodiment of the present invention obtains the output answer for the input question output by the answer generation model, inputs the input question and the output answer into the reward model, and obtains the evaluation result of the output answer output by the reward model. And because the reward model is trained based on the training method of the reward model as described in any of the above embodiments, during the training process of the reward model, the original training sample and the prompt text are input into the initial reward model to obtain the first reasoning principle and the first score output by the initial reward model, and the prompt text includes the reasoning principle generation requirement text, and the reasoning principle generation requirement text includes the reasoning principle generation requirement information, and the reasoning principle is used to indicate the reasoning logic that the initial reward model must follow when scoring, so that based on the prompt text, the initial reward model can be guided to generate scores according to the established logic, thereby improving the accuracy and consistency of the scores, and then improving the performance of the reward model, and finally improving the performance of the answer generation model based on the reward model reinforcement, that is, improving the generation accuracy of the answer generation model; and based on the original training sample, the first reasoning principle and the first score, the initial reward model is fine-tuned to obtain the reward model, so as to train the obtained The reward model can not only output the score corresponding to the answer, but also output the reasoning principle for inferring the score, thereby providing a more comprehensive reward signal to the answer generation model, thereby improving the performance of the answer generation model strengthened by the reward model, that is, improving the generation accuracy of the answer generation model; at the same time, fine-tuning the initial reward model based on the first reasoning principle can continuously stimulate the reasoning ability of the reward model, thereby improving the performance of the reward model, so that the reward model provides a denser and more accurate reward signal to the answer generation model, thereby improving the performance of the answer generation model strengthened by the reward model, that is, improving the generation accuracy of the answer generation model; and the initial reward model is a pre-trained large language model, so that the training of the reward model does not need to rely on a large amount of manually annotated data, and is not easily affected by human subjective bias, thereby improving the performance of the answer generation model strengthened by the reward model, that is, improving the generation accuracy of the answer generation model.

[0116] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the training method of the reward model, the method comprising: inputting the original training sample and the prompt text into the initial reward model to obtain the first reasoning principle and the first score output by the initial reward model; the original training sample comprises the original sample question and the original sample answer, the prompt text comprises the reasoning principle generation requirement text, the reasoning principle generation requirement text comprises the reasoning principle generation requirement information, the reasoning principle is used to indicate the reasoning logic to be followed by the initial reward model when scoring; based on the original training sample, the first reasoning principle and the first score, fine-tune the initial reward model to obtain the reward model; wherein the initial reward model is a pre-trained large language model; the reward model is used to evaluate the answer output by the answer generation model, and the answer generation model is used to generate the answer corresponding to the input question. Alternatively, an answer evaluation method is executed, the method comprising: obtaining an output answer to an input question output by an answer generation model; inputting the input question and the output answer into a reward model, and obtaining an evaluation result of the output answer output by the reward model, wherein the reward model is trained based on the training method of the reward model described in any of the above embodiments.

[0117] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0118] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the training method of the reward model provided by the above methods, the method including: inputting the original training sample and the prompt text into the initial reward model to obtain the first reasoning principle and the first score output by the initial reward model; the original training sample includes the original sample question and the original sample answer, the prompt text includes the reasoning principle generation requirement text, the reasoning principle generation requirement text includes the reasoning principle generation requirement information, and the reasoning principle is used to indicate the reasoning logic to be followed by the initial reward model when scoring; based on the original training sample, the first reasoning principle and the first score, fine-tuning the initial reward model to obtain a reward model; wherein the initial reward model is a pre-trained large language model; the reward model is used to evaluate the answer output by the answer generation model, and the answer generation model is used to generate an answer corresponding to the input question. Alternatively, a computer can execute the answer evaluation method provided by the above methods, the method comprising: obtaining an output answer to an input question output by an answer generation model; inputting the input question and the output answer into a reward model, and obtaining an evaluation result of the output answer output by the reward model, wherein the reward model is trained based on the training method of the reward model described in any of the above embodiments.

[0119] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the training method of the reward model provided by the above-mentioned methods, the method comprising: inputting an original training sample and a prompt text into an initial reward model to obtain a first reasoning principle and a first score output by the initial reward model; the original training sample comprises an original sample question and an original sample answer, the prompt text comprises a reasoning principle generation requirement text, the reasoning principle generation requirement text comprises reasoning principle generation requirement information, and the reasoning principle is used to indicate the reasoning logic to be followed by the initial reward model when scoring; based on the original training sample, the first reasoning principle and the first score, fine-tuning the initial reward model to obtain a reward model; wherein the initial reward model is a pre-trained large language model; the reward model is used to evaluate the answer output by an answer generation model, and the answer generation model is used to generate an answer corresponding to the input question. Alternatively, execute the answer evaluation method provided by the above methods, the method comprising: obtaining an output answer to an input question output by an answer generation model; inputting the input question and the output answer into a reward model, and obtaining an evaluation result of the output answer output by the reward model, wherein the reward model is trained based on the training method of the reward model described in any of the above embodiments.

[0120] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0121] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A training method for a reward model, characterized in that: include: Inputting the original training sample and the prompt text into the initial reward model to obtain a first reasoning principle and a first score output by the initial reward model; The original training sample includes an original sample question and an original sample answer, the prompt text includes a reasoning principle generation requirement text, the reasoning principle generation requirement text includes generation requirement information of the reasoning principle, and the reasoning principle is used to indicate the reasoning logic that the initial reward model should follow when scoring; Fine-tuning the initial reward model based on the original training sample, the first reasoning principle, and the first score to obtain a reward model; Among them, the initial reward model is a pre-trained large language model; the reward model is used to evaluate the answers output by the answer generation model, and the answer generation model is used to generate answers corresponding to input questions.

2. The training method of the reward model according to claim 1, characterized in that: The original training sample also includes a scoring label corresponding to the original sample answer; The fine-tuning of the initial reward model based on the original training sample, the first reasoning principle and the first score comprises: When the first score is consistent with the score label, generating fine-tuning sample data based on the original training sample, the first reasoning principle and the first score; In the case where the first score is inconsistent with the score label, the original training sample, the prompt text and the score label are input into the initial reward model to obtain a second reasoning principle and a second score output by the initial reward model, and fine-tuning sample data is generated based on the original training sample, the second reasoning principle and the second score; The initial reward model is fine-tuned based on the fine-tuning sample data.

3. The reward model training method according to claim 2, wherein fine-tuning the initial reward model based on the fine-tuning sample data comprises: Inputting the original training samples in the fine-tuning sample data into the initial reward model to obtain the prediction reasoning principle and prediction score output by the initial reward model; Determine the reasoning principle in the fine-tuning sample data as a true reasoning principle label, and determine the score in the fine-tuning sample data as a true score label; The initial reward model is fine-tuned based on a difference between the predicted inference principle and the true inference principle label, and a difference between the predicted score and the true score label.

4. The training method of the reward model according to claim 1, characterized in that: The inputting the original training sample and the prompt text into the initial reward model to obtain the first reasoning principle and the first score output by the initial reward model includes: Obtaining an original sample set; the original sample set includes a plurality of the original training samples and a plurality of original samples, any of the original samples including an original sample question and an original sample answer; For each of the original training samples, input the original training sample and the prompt text into an initial reward model to obtain a first reasoning principle and a first score output by the initial reward model; The prompt text is determined based on the plurality of original samples.

5. The training method of the reward model according to any one of claims 1 to 4, characterized in that: The prompt text further includes an output format requirement text of the initial reward model, wherein the output format requirement text is used to indicate an output format of the reasoning principle and an output format of the score.

6. The training method of the reward model according to any one of claims 1 to 4, characterized in that: The prompt text also includes output example text of the initial reward model, and the output example text includes an output example of reasoning principle and an output example of scoring.

7. A method for evaluating an answer, characterized in that: include: Get the output answer of the answer generation model to the input question; The input question and the output answer are input into a reward model to obtain an evaluation result of the output answer output by the reward model, wherein the reward model is trained based on the training method of the reward model as described in any one of claims 1 to 6.

8. A training device for a reward model, characterized in that: include: An input module, used to input the original training sample and the prompt text into the initial reward model, and obtain the first reasoning principle and the first score output by the initial reward model; the original training sample includes the original sample question and the original sample answer, the prompt text includes the reasoning principle generation requirement text, the reasoning principle generation requirement text includes the reasoning principle generation requirement information, and the reasoning principle is used to indicate the reasoning logic that the initial reward model needs to follow when scoring; A fine-tuning module, configured to fine-tune the initial reward model based on the original training sample, the first reasoning principle and the first score to obtain a reward model; Among them, the initial reward model is a pre-trained large language model; the reward model is used to evaluate the answers output by the answer generation model, and the answer generation model is used to generate answers corresponding to input questions.

9. An answer evaluation device, characterized in that: include: An acquisition module, used for acquiring an output answer to an input question output by an answer generation model; An evaluation module is used to input the input question and the output answer into a reward model to obtain an evaluation result of the output answer output by the reward model, wherein the reward model is trained based on the training method of the reward model as described in any one of claims 1 to 6.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, it implements the training method of the reward model as described in any one of claims 1 to 6 or the answer evaluation method as described in claim 7.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the training method of the reward model as described in any one of claims 1 to 6 or the answer evaluation method as described in claim 7.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the training method of the reward model as described in any one of claims 1 to 6 or the answer evaluation method as described in claim 7.

Images