Method, device, equipment and storage medium for training reward model

Abstract

The embodiments of the present disclosure provide a method, apparatus, device and computer-readable storage medium for training a reward model. The method obtains prompt instructions based on multi-level classification including a decreasing classification granularity to balance the category distribution of a large language model, and generates multiple prompt responses for each prompt instruction to balance the result diversity distribution of the large language model, wherein the generated prompt responses may include various prompt responses including baseline prompt responses, prompt responses generated by different language models, prompt responses generated by a large language model at different training stages, and prompt responses controlled by different temperature coefficients, so as to train the reward model based on these prompt responses, so as to maximize the difference between the score of the reward model for the better prompt response and the score of the worse prompt response, thereby generating a trained reward model and improving the generalization of the reward model.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and more specifically, to a method, apparatus, device and storage medium for training a reward model. Background Art

[0002] In today's artificial intelligence field, large language models are increasingly used, and the reward model (RM), as an important technical means, is of great significance for improving the performance and generalization ability of large language models. For large language models, the reward model is a training technology that guides the model to learn specific behaviors or performances by introducing additional reward signals. In large language models, the reward model can be used to train the model to generate text that better meets specific standards or expectations, thereby improving the model's generation ability and adaptability.

[0003] The reward model can help the large language model better understand the context, semantics, and logic, thereby improving the generation ability and performance of the large language model, and being able to better adapt to specific tasks or better express the desired behavior. In addition, the reward model can also help solve problems that may arise in the generation process of the large language model, such as improving the generation diversity of the large language model, reducing generation errors, and improving the relevance of generation. Through the reward model, the large language model can be guided to learn in the desired direction, so that the text it generates is more in line with specific standards or expected styles. It can also help the large language model better adapt to different tasks and scenarios, so that it performs better in specific application fields. By introducing reward signals, the large language model can learn and adapt to new tasks faster, improving its generalization ability and adaptability.

[0004] Therefore, an effective reward model training method is needed to generate an optimized reward model to improve the generation ability, adaptability and performance of large language models so that they can better meet the needs of specific tasks and scenarios. Summary of the invention

[0005] In order to solve the above problems, the present invention optimizes the training of the reward model by improving the acquisition of prompt instructions and the generation process of corresponding prompt responses, thereby generating a reward model that can balance the category distribution and result diversity distribution of the large language model and can be applicable to large language models of different scales and capabilities, and can produce good results on these models.

[0006] Embodiments of the present disclosure provide a method, apparatus, device, and computer-readable storage medium for training a reward model.

[0007] An embodiment of the present disclosure provides a method for training a reward model, wherein the reward model is associated with a large language model, and the method comprises: obtaining a prompt instruction set based on a multi-level classification system, wherein the multi-level classification system comprises a multi-level classification with a decreasing classification granularity; collecting a plurality of prompt responses for each prompt instruction in the prompt instruction set, wherein the plurality of prompt responses comprises at least two of the following: prompt responses generated by different language models based on the prompt instructions; prompt responses generated by the large language model based on the prompt instructions at different training stages, including prompt responses generated by the large language model based on the prompt instructions after supervised fine-tuning, prompt responses generated by the large language model based on the prompt instructions during reinforcement learning, and prompt responses generated by the large language model based on the prompt instructions during reinforcement learning. the prompt response generated by the large language model based on the prompt instruction under the control of different temperature coefficients, wherein the temperature coefficient is used to control the diversity of the prompt responses generated by the large language model; for each prompt instruction in the prompt instruction set, a plurality of prompt response pairs are constructed based on a plurality of prompt responses corresponding to the prompt instruction; and based on all prompt response pairs associated with the prompt instruction set, the reward model is trained by maximizing the difference between the score given by the reward model to the better prompt response in each prompt response pair and the score given to the worse prompt response in the prompt response pair, wherein the better prompt response and the worse prompt response in the prompt response pair are determined by ranking the prompt responses in the prompt response pair.

[0008] An embodiment of the present disclosure provides a device for training a reward model, wherein the reward model is associated with a large language model, and the device comprises: an instruction acquisition module, configured to acquire a prompt instruction set based on a multi-level classification system, wherein the multi-level classification system comprises a multi-level classification with a decreasing classification granularity; a response generation module, configured to collect a plurality of prompt responses for each prompt instruction in the prompt instruction set, wherein the plurality of prompt responses comprises at least two of the following: prompt responses generated by different language models based on the prompt instructions; prompt responses generated by the large language model based on the prompt instructions at different training stages, including prompt responses generated by the large language model based on the prompt instructions after supervised fine-tuning, prompt responses generated by the large language model based on the prompt instructions during reinforcement learning, and prompt responses generated by the large language model based on the prompt instructions after reinforcement learning. The invention relates to a method for generating prompt responses according to the present invention; and a prompt response generated by the large language model based on the prompt instruction under the control of different temperature coefficients, wherein the temperature coefficient is used to control the diversity of the prompt responses generated by the large language model; a response pair construction module, configured to construct a plurality of prompt response pairs for each prompt instruction in the prompt instruction set based on a plurality of prompt responses corresponding to the prompt instruction; and a model training module, configured to train the reward model based on all prompt response pairs associated with the prompt instruction set by maximizing the difference between the score given by the reward model to the better prompt response in each prompt response pair and the score given to the worse prompt response in the prompt response pair, wherein the better prompt response and the worse prompt response in the prompt response pair are determined by ranking the prompt responses in the prompt response pair according to their merits.

[0009] An embodiment of the present disclosure provides a device for training a reward model, comprising: one or more processors; and one or more memories, wherein a computer executable program is stored in the one or more memories, and when the computer executable program is executed by the processor, the method for training a reward model as described above is performed.

[0010] An embodiment of the present disclosure provides a computer-readable storage medium having computer-executable instructions stored thereon, which are used to implement the method for training a reward model as described above when executed by a processor.

[0011] The embodiments of the present disclosure provide a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the method for training a reward model according to an embodiment of the present disclosure.

[0012] The method provided by the embodiments of the present disclosure obtains prompt instructions based on multi-level classification including a decreasing classification granularity, and generates multiple prompt responses for each prompt instruction, wherein the generated prompt responses may include various prompt responses including baseline prompt responses, prompt responses generated by different language models, prompt responses generated by a large language model at different training stages, and prompt responses controlled by different temperature coefficients, so as to train a reward model based on these prompt responses, so as to maximize the gap between the score of the reward model for a better prompt response and the score of the worse prompt response, thereby generating a trained reward model.

[0013] Through the method provided by the embodiments of the present disclosure, prompt instructions can be obtained by using a multi-level classification system covering multiple categories of different granularities to balance the category distribution of the large language model, so that the number of samples in different categories is relatively balanced, thereby improving the learning effect of the large language model on each category, and by generating multiple prompt responses on different dimensions, the result diversity distribution of the large language model can be balanced, and the generalization of the reward model can be improved, so that the trained reward model can be applicable to large language models of different scales and capabilities, and can produce good results on these models. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some exemplary embodiments of the present disclosure, and a person of ordinary skill in the art can obtain other drawings based on these drawings without creative work.

[0015] Figure 1 is a flow chart illustrating a method 100 of training a reward model according to an embodiment of the present disclosure;

[0016] Figure 2 is a schematic flow chart illustrating a method for training a reward model according to an embodiment of the present disclosure;

[0017] Figure 3 is a schematic diagram illustrating a multi-level classification system according to an embodiment of the present disclosure;

[0018] Figure 4 is a schematic flow chart showing the order of merit for prompt responses generated for prompt instructions according to an embodiment of the present disclosure;

[0019] Figure 5 is a schematic flow chart illustrating the accuracy of a reward model on two prompt responses with different scoring gaps according to an embodiment of the present disclosure;

[0020] Figure 6 is a schematic diagram showing an apparatus for training a reward model according to an embodiment of the present disclosure;

[0021] Figure 7 A schematic diagram showing a device for training a reward model according to an embodiment of the present disclosure; and

[0022] Figure 8 A schematic diagram illustrating the architecture of an exemplary computing device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present disclosure more obvious, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.

[0024] In this specification and the accompanying drawings, substantially the same or similar steps and elements are represented by the same or similar reference numerals, and repeated descriptions of these steps and elements will be omitted. At the same time, in the description of the present disclosure, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance or ranking.

[0025] In the embodiments of the present disclosure, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0026] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present disclosure pertains. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0027] To facilitate description of the present disclosure, concepts related to the present disclosure are introduced below.

[0028] The method of training the reward model disclosed in the present invention can be based on artificial intelligence (AI). Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence.

[0029] The method for training the reward model disclosed in the present invention can be based on the reinforcement learning from human feedback (RLHF) method. RLHF refers to the use of feedback information from human experts to reinforce the learning of the model to improve the performance of the model. In the alignment process, the quality of the model-generated text can be improved by collecting human experts' evaluations of the model-generated text, and then using this evaluation information to guide the training of the model. For example, in the dialogue generation task, users' evaluations of the dialogue content generated by the model can be collected, and then this evaluation information can be used to guide the training of the model, so that the model generates dialogue content that is more in line with user expectations.

[0030] The method for training a reward model disclosed in the present invention can be based on a reward model (RM). The reward model algorithm is an algorithm based on reinforcement learning, which is used to train a language model to generate more accurate and fluent text. In this algorithm, the goal of the model is to maximize a given reward function, which is usually defined by a human expert or an automatic evaluation method. The goal of the reward function is to evaluate the quality of the text generated by the model, such as grammatical correctness, coherence, information accuracy, etc.

[0031] In summary, the solutions provided by the embodiments of the present disclosure involve technologies such as artificial intelligence, reinforcement learning from human feedback, and reward models. The embodiments of the present disclosure will be further described below in conjunction with the accompanying drawings.

[0032] After the large language model completes supervised fine-tuning (SFT), the next stage is to build a reward model to score the question-answer pair. The reward model is derived from the reward function in reinforcement learning and can score the current state of the large language model to quantify the value of the state. The reward model in the large language model fine-tuning calculates a score for the input question and answer. The higher the match between the input answer and the question, the higher the score output by the reward model.

[0033] As an important technical means, reward model training is of great significance for improving the performance and generalization ability of large language models. However, in practical applications, there may be many problems. For example, in the reinforcement learning process using the proximal policy optimization (PPO) algorithm, the application of PPO can help the model better learn complex language rules and structures, but unpredictable spaces may be explored during the PPO sampling process. If the generalization ability of the reward model is not enough, it will lead to deviations in the learning of the reward model. The generalization ability of the reward model is crucial for the training of large language models. As a means of evaluating model behavior, the generalization ability of the reward model directly affects the performance and stability of the model. How to better maintain the generalization of the reward model under different model sizes and capabilities is also an important issue. This involves the design and training of the reward model, as well as how to maintain a certain consistency and robustness under different model sizes and capabilities. Therefore, it is necessary to design a more general and effective training strategy to improve the applicability and performance of the model in different scenarios.

[0034] Based on this, the present disclosure provides a method for training a reward model, which optimizes the training of the reward model by improving the acquisition of prompt instructions and the generation process of corresponding prompt responses, thereby generating a reward model that can balance the category distribution and result diversity distribution of a large language model and can be applicable to large language models of different scales and capabilities.

[0035] The method provided by the embodiments of the present disclosure obtains prompt instructions based on multi-level classification including a decreasing classification granularity, and generates multiple prompt responses for each prompt instruction, wherein the generated prompt responses may include various prompt responses including baseline prompt responses, prompt responses generated by different language models, prompt responses generated by a large language model at different training stages, and prompt responses controlled by different temperature coefficients, so as to train a reward model based on these prompt responses, so as to maximize the gap between the score of the reward model for a better prompt response and the score of the worse prompt response, thereby generating a trained reward model.

[0036] Through the method provided by the embodiments of the present disclosure, prompt instructions can be obtained by using a multi-level classification system covering multiple categories of different granularities to balance the category distribution of the large language model, so that the number of samples in different categories is relatively balanced, thereby improving the learning effect of the large language model on each category, and by generating multiple prompt responses on different dimensions, the result diversity distribution of the large language model can be balanced, and the generalization of the reward model can be improved, so that the trained reward model can be applicable to large language models of different scales and capabilities, and can produce good results on these models.

[0037] Figure 1 is a flow chart illustrating a method 100 of training a reward model according to an embodiment of the present disclosure. Figure 2 is a schematic flowchart illustrating a method for training a reward model according to an embodiment of the present disclosure.

[0038] like Figure 1 As shown, in step S101, a prompt instruction set may be acquired based on a multi-level classification system, wherein the multi-level classification system may include multi-level classifications with decreasing classification granularity.

[0039] Optionally, in an embodiment of the present disclosure, in order to improve the generalization of the trained reward model, prompt instructions for training the reward model can be collected from a large number of prompt instruction data sets based on a multi-level classification system, such as Figure 2 As shown in 201, the multi-level classification system may include multi-level classification to comprehensively cover all types of user needs. For example, classification may be performed level by level according to the decreasing relationship of classification granularity, so as to sample samples for reward model training from the data set for each classification, i.e., prompt instructions, to form the above-mentioned prompt instruction set. Optionally, these data sets may also adopt the classification architecture of the multi-level classification system of the present application to classify various prompt instructions for their application in various task scenarios.

[0040] Figure 3 is a schematic diagram showing a multi-level classification system according to an embodiment of the present disclosure. Figure 3 As shown, the multi-level classification system may include a first-level classification, a second-level classification, a third-level classification, etc., with the classification granularity in a decreasing relationship. Specifically, the multi-level classification system may include multiple (e.g., M) first-level classifications, the first-level classification being the highest level of the entire classification system, each first-level classification category may represent the main direction of a type of user demand, the second-level classification may be a more specific subdivision based on the first-level classification, each first-level classification may include multiple (e.g., N or other numbers) second-level classifications, and the third-level classification may be a further subdivision based on the second-level classification, each second-level classification may include multiple (e.g., P, Q or other numbers) third-level classifications, and so on.

[0041] As an example, the multi-level classification system can be, for example, a three-level classification system, that is, including three levels of classification. For example, the three-level classification system can include 6 first-level classifications, 30 second-level classifications, and more than 200 third-level classifications. Therefore, the prompt instruction set can include multiple prompt instructions under each three-level classification. These three-level classifications can further refine user needs to meet the user's specific needs for different types of prompts, thereby designing a classification system for all prompt instructions that comprehensively covers all types of user needs. Of course, it should be understood that the above-mentioned three-level classification system is only used as an example and not a limitation in the present disclosure, and the method of the present disclosure can also adopt a multi-level classification system of other levels.

[0042] Therefore, from the user's perspective, this multi-level classification system can help users more accurately select and locate the prompts they need, thereby improving the user experience. In addition, from the perspective of reward model training, the prompt instructions in each category should be diverse enough to ensure that the reward model can cover various application scenarios well. That is to say, under each three-level classification, a variety of prompt instructions covering different aspects can be included so that the trained reward model can generalize well to different types of inputs. For example, in the second-level classification of novels under the category of literary works, various types of novel prompts can be included, covering prompts of different themes and styles such as suspense, romance, history, science fiction, etc., to ensure that the reward model can adapt to various types of novel generation tasks.

[0043] Therefore, based on the multi-level classification system that comprehensively covers all types of user needs as described above, and ensures that the prompts in each category have sufficient diversity, the user experience can be improved and a more generalized reward model can be trained. Such a classification system can help users better locate the prompts they need, while also providing rich training data for the reward model, improving the performance and adaptability of the model.

[0044] After the collection of the prompt instruction set is completed, the prompt response may be collected for each prompt instruction in the prompt instruction set.

[0045] In step S102, multiple prompt responses can be collected for each prompt instruction in the prompt instruction set, wherein the multiple prompt responses may include at least two of the following: prompt responses generated by different language models based on the prompt instructions; prompt responses generated by the large language model based on the prompt instructions at different training stages, including prompt responses generated by the large language model based on the prompt instructions after supervised fine-tuning, prompt responses generated by the large language model based on the prompt instructions during reinforcement learning, and prompt responses generated by the large language model after reinforcement learning; and prompt responses generated by the large language model based on the prompt instructions under the control of different temperature coefficients, wherein the temperature coefficient can be used to control the diversity of prompt responses generated by the large language model.

[0046] Optionally, in the embodiments of the present disclosure, different prompt responses to the same prompt instruction can be obtained in a variety of ways to respectively achieve different training objectives of the reward model, such as Figure 2 As shown in 202 and 203.

[0047] Optionally, in order to improve the generalization of the trained reward model, for the same prompt instruction, prompt responses generated by different language models, prompt responses generated by the large language model at different training stages, and prompt responses generated by the large language model under the control of different temperature coefficients can be collected.

[0048] According to an embodiment of the present disclosure, the prompt responses generated by different language models based on the prompt instruction may include prompt responses generated by language models of different sizes and different performances based on the prompt instruction. Optionally, in order to ensure the generalization of the trained reward model on language models of different sizes and different performances, the prompt responses generated by these various language models for the same prompt instruction can be collected for training the reward model. Among them, these different language models can also include language models with the current best performance (for example, GPT, lamma, baichuan and other language models), so that the reward model can learn the optimal results that the language model can generate.

[0049] Optionally, a large language model at different training stages (e.g., supervised fine-tuning stage, reinforcement learning stage) can also be used to generate different prompt responses for the same prompt instruction to enhance the diversity of responses.

[0050] According to an embodiment of the present disclosure, the prompt response generated by the large language model based on the prompt instruction under the control of different temperature coefficients may include a prompt response generated by adjusting the temperature coefficient in the process of generating the prompt response based on the prompt instruction using the large language model. Optionally, using the same large language model, different prompt responses can also be generated by adjusting the temperature coefficient in the large language model to increase the diversity of prompt responses. Among them, the temperature coefficient is a hyperparameter for adjusting the creativity and diversity of the large language model when generating text. By adjusting the temperature coefficient, the probability distribution of the large language model can be affected to control the text generated by the large language model to be more concentrated or more diverse. For example, the temperature coefficient can be a value greater than 0, usually between 0 and 1. The temperature coefficient can affect the probability distribution of the sampled predicted vocabulary of the large language model when generating text. For example, when the temperature coefficient of the large language model is high (such as 0.8, 1 or higher), the large language model will be more inclined to select from a more diverse and different vocabulary, which makes the generated text riskier and more creative. When the temperature coefficient is low (such as 0.2, 0.3, etc.), the model will mainly select from vocabulary with higher probability, resulting in smoother and more coherent text.

[0051] In addition, according to an embodiment of the present disclosure, the multiple prompt responses may further include: a benchmark prompt response corresponding to the prompt instruction. The reward model training may also be performed in combination with a benchmark prompt response of the same prompt instruction, wherein the benchmark prompt response may be real, well-labeled, and reliable data (e.g., a standard response to a prompt instruction) used for model training and evaluation to provide a reliable benchmark to measure the performance of the reward model. By utilizing the benchmark prompt response to participate in the training of the reward model, the optimization of the reward model may be better guided, so that the reward model may better understand the objectives and correct behaviors of the task, thereby improving the generalization performance and practical application effect of the reward model.

[0052] Of course, the above-mentioned several prompt response generation methods are only used as examples and not limitations in the present disclosure, and the method of the present disclosure can also use other generation methods to collect prompt responses.

[0053] Therefore, by collecting prompt instructions and their corresponding prompt responses, e.g. Figure 2 As shown in 201-203 in , training data for the reward model can be further constructed based on the collected prompt instructions and their corresponding prompt responses, such as Figure 2 As shown in 204.

[0054] In step S103, for each prompt instruction in the prompt instruction set, a plurality of prompt response pairs may be constructed based on a plurality of prompt responses corresponding to the prompt instruction.

[0055] Optionally, for multiple prompt responses corresponding to the same prompt instruction, the relative ranking between these prompt responses can be used to train the reward model, that is, instead of scoring each prompt response, these prompt responses can be labeled with a ranking of good and bad, so that the reward model learns to automatically score each prompt response by trying to maximize the difference between the better prompt response scores and the worse prompt response scores.

[0056] According to an embodiment of the present disclosure, constructing a plurality of prompt response pairs based on a plurality of prompt responses corresponding to the prompt instruction may include: constructing a prompt response pair for any two prompt responses among the plurality of prompt responses, and ranking the two prompt responses in the prompt response pair to determine a better prompt response and a worse prompt response in the prompt response pair; wherein, for each prompt response, a baseline prompt response corresponding to the prompt instruction is determined as a better prompt response in any prompt response pair associated with the prompt response.

[0057] In order to avoid excessive subjectivity in data annotation and unify the scoring standards, in an embodiment of the present disclosure, the absolute score annotation task can be converted into a relative ranking task. Figure 4 It is a schematic flow chart showing the ranking of prompt responses generated for prompt instructions according to an embodiment of the present disclosure.

[0058] like Figure 4 As shown in 401 and 402 in FIG. 1 , for the same prompt instruction (e.g., “What is a banana?”), multiple prompt-response pairs can be constructed, for example, Figure 4 One of the prompt-response pairs is shown in , which may include prompt response 1 (for example, "Banana is a yellow fruit that usually grows on trees. It is a favorite fruit of monkeys.") and prompt response 2 (for example, "Banana is very sour and rich in mineral elements."). Therefore, for this prompt instruction, according to the labeling task of absolute score, different scores may be obtained when different scoring methods are used (for example, when different human experts perform the scoring), which in turn makes it difficult for the reward model to clearly understand the pros and cons of each prompt response when learning. In an embodiment of the present disclosure, the prompt responses in the prompt-response pair can be processed according to the relative pros and cons sorting task, that is, the relative pros and cons of the prompt-response pair can be ranked in different scoring methods, so as to obtain a unified labeling result. For example, as Figure 4As shown in 403, when different human experts 1 and 2 rank the prompt response pairs, although the two human experts may have different scoring criteria, their results of ranking the prompt responses relative to each other should be consistent (for example, both A>B). Therefore, the determined annotation results can be unified.

[0059] Therefore, multiple different prompt responses to the same prompt instruction, e.g. Figure 2 As shown in 204, multiple prompt-response pairs and their ranking results can be obtained through ranking.

[0060] Furthermore, for the collection of training data for the reward model (e.g., the above-mentioned prompt instructions and corresponding prompt response pairs), the training data can also be obtained in a variety of ways as a candidate source of training data for the reward model. According to an embodiment of the present disclosure, the method may include at least two of the following: collecting training data for the reward model by human experts, including collecting the prompt instruction set from a prompt instruction database, generating multiple prompt responses based on the collected prompt instruction set, and manually ranking the two prompt responses in each prompt response pair; supplementing the training data collected by human experts through a pre-trained language model, including generating prompt responses based on directional prompt instructions; ranking the two prompt responses in each prompt response pair by a pre-trained language model; and using open source reward model training data as training data for the reward model.

[0061] As described above, the collection method of training data for the reward reward model may include but is not limited to manual annotation, directional generation, machine annotation, and open source data. Specifically, for example, manual annotation can screen prompt instructions from various databases, generate different prompt responses in the manner described above, and generate prompt response pairs through artificial ranking. For the directional generation method, the existing more mature large language model can be used to carry out directional reinforcement for the parts that have not been covered in the manually annotated data, including the directional generation of prompt responses based on directional prompt instructions. Similarly, such a more mature large language model can also be used to rank the generated prompt responses to generate prompt response pairs. For the collection method using open source data, the training data of the currently open source reward model can be obtained for the training of the reward model to improve the coverage of the prompt instruction-prompt response space.

[0062] Therefore, by referring to Figure 2 Steps 201-204 and Figure 3 and Figure 4 The description of can complete the collection of training data for the reward model. Next, the reward model can be trained based on the collected training data.

[0063] In step S104, the reward model may be trained based on all prompt response pairs associated with the prompt instruction set by maximizing the difference between the score given by the reward model to the better prompt response in each prompt response pair and the score given to the worse prompt response in the prompt response pair, wherein the better prompt response and the worse prompt response in the prompt response pair may be determined by ranking the prompt responses in the prompt response pair. Figure 4 For the prompt response pair shown in , the better prompt response in the prompt response pair may be prompt response 1, and the worse prompt response may be prompt response 2.

[0064] According to an embodiment of the present disclosure, training the reward model may include: for each prompt instruction in the prompt instruction set, based on each prompt-response pair corresponding to the prompt instruction and the quality ranking results of the prompt response pairs, training the reward model by maximizing the difference between the score given by the reward model to the better prompt response in the prompt response pair and the score given to the worse prompt response in the prompt response pair.

[0065] Optionally, a loss function can be constructed for reward model training that maximizes the difference between the reward model's score for the better prompt response in each prompt response pair and the score for the worse prompt response in the prompt response pair, so that the scalar score corresponding to the prompt response with a higher ranking is as high as possible than the scalar score corresponding to the prompt response with a lower ranking. Therefore, by maximizing the difference between the reward model's score for the better prompt response and the score for the worse prompt response in the prompt response pair, the distinction accuracy of the reward model can be improved. For example, for six different prompt responses for the same prompt instruction, the difference between the results of the ranking of any two prompt responses can fall in the set {1, 2, 3, 4, 5}, and the difference between the results of the ranking of the two prompt responses can reflect the difficulty of distinguishing the two prompt responses, and evaluating the consistency of the scores of different scoring gaps can determine the accuracy of the current reward model in subdividing the scoring gaps.

[0066] Figure 5 is a schematic flow chart showing the accuracy of the reward model according to an embodiment of the present disclosure on two prompt responses with different scoring gaps. Figure 5 As shown, different scoring gaps can correspond to the significance of the difference between the two prompt responses, for example, Figure 5The score gaps are 1, 2, 3, 4, and 5, and their significance can correspond to uncertain, almost the same, slightly better, better, and significantly better situations, respectively. Therefore, based on the prompt-response pairs under such score gaps, the reward model has different discrimination accuracies accordingly, such as Figure 5 As shown, for the prompt-response pairs with a higher score gap, the reward model has a higher discrimination accuracy, that is, the scoring results of the reward model have a higher scoring consistency with the superiority and inferiority ranking results of the prompt-response pairs.

[0067] In addition, in an embodiment of the present disclosure, targeted optimization can also be performed for specific problems that arise in the large language model in subsequent training stages, and the targeted optimization can be achieved by taking data related to such problems as input to optimize the reward model. For example, for problems that arise in the large language model during reinforcement learning, according to an embodiment of the present disclosure, training the reward model may include: when the multiple prompt responses include a prompt response generated by the large language model based on the prompt instruction during reinforcement learning, training the reward model for deviations in the prompt response generated by the large language model based on the prompt instruction during reinforcement learning.

[0068] Optionally, for problems that arise in the large language model during reinforcement learning, when it is determined that these problems are caused by the scoring results of the reward model, prompt-response pairs related to these problems can be added to the input of the reward model to optimize the reward model, thereby correcting the deviations that arise in the large language model during reinforcement learning.

[0069] For example, when the reward model gives high scores to rejection answers during reinforcement learning of a large language model, such rejection answer data generated by the large language model can be input into the training of the reward model to optimize the reward model, thereby solving the problem of inaccurate learning of rejection answers by the reward model.

[0070] Therefore, by referring to the above Figure 2-Figure 5 The described method for training a reward model can obtain prompt instructions by using a multi-level classification system covering multiple categories of different granularities to balance the category distribution of the large language model, so that the number of samples in different categories is relatively balanced, thereby improving the learning effect of the large language model on each category, and by generating multiple prompt responses on different dimensions, it can balance the result diversity distribution of the large language model, improve the generalization of the reward model, and make the trained reward model applicable to large language models of different sizes and capabilities, and can produce good results on these models.

[0071] Figure 6 is a schematic diagram showing an apparatus 600 for training a reward model according to an embodiment of the present disclosure.

[0072] According to an embodiment of the present disclosure, the apparatus 600 for training a reward model may include an instruction acquisition module 601 , a response generation module 602 , a response pair construction module 603 and a model training module 604 .

[0073] The instruction acquisition module 601 may be configured to acquire a prompt instruction set based on a multi-level classification system, wherein the multi-level classification system may include multi-level classifications with decreasing classification granularity. Optionally, the instruction acquisition module 601 may perform the operations described above with reference to step S101.

[0074] For example, in order to improve the generalization of the trained reward model, prompt instructions for training the reward model can be collected based on a multi-level classification system, wherein the multi-level classification system may include multi-level classifications to comprehensively cover all types of user needs. For example, classification can be performed step by step according to the decreasing relationship of classification granularity. Optionally, the multi-level classification system may be, for example, a three-level classification system, that is, including three levels of classification. For example, the three-level classification system may include 6 first-level classifications, 30 second-level classifications, and more than 200 third-level classifications. Therefore, the prompt instruction set may include multiple prompt instructions under each three-level classification, and these three-level classifications may further refine user needs to meet the user's specific needs for different types of prompts, thereby designing a classification system for all prompt instructions that comprehensively covers all types of user needs. Of course, it should be understood that the above-mentioned three-level classification system is only used as an example and not a limitation in the present disclosure, and the method of the present disclosure may also adopt a multi-level classification system of other levels.

[0075] Based on the multi-level classification system described above that comprehensively covers all types of user needs and ensures that the prompts in each category have sufficient diversity, the user experience can be improved and a more generalized reward model can be trained. Such a classification system can help users better locate the prompts they need, while also providing rich training data for the reward model, improving the performance and adaptability of the model.

[0076] After the collection of the prompt instruction set is completed in the instruction acquisition module 601, the prompt response may be collected for each prompt instruction in the prompt instruction set.

[0077] The response generation module 602 may be configured to collect multiple prompt responses for each prompt instruction in the prompt instruction set, wherein the multiple prompt responses may include at least two of the following: prompt responses generated by different language models based on the prompt instruction; prompt responses generated by the large language model based on the prompt instruction at different training stages, including prompt responses generated by the large language model based on the prompt instruction after supervised fine-tuning, prompt responses generated by the large language model based on the prompt instruction during reinforcement learning, and prompt responses generated by the large language model after reinforcement learning; and prompt responses generated by the large language model based on the prompt instruction under the control of different temperature coefficients, wherein the temperature coefficient may be used to control the diversity of prompt responses generated by the large language model. Optionally, the response generation module 602 may perform the operations described above with reference to step S102.

[0078] Optionally, in an embodiment of the present disclosure, different prompt responses to the same prompt instruction may be obtained in a variety of ways to respectively achieve different training objectives of the reward model.

[0079] For example, in order to improve the generalization of the trained reward model, for the same prompt instruction, the prompt responses generated by different language models, the prompt responses generated by the large language model at different training stages, and the prompt responses generated by the large language model under the control of different temperature coefficients can be collected.

[0080] For example, in order to ensure the generalization of the trained reward model on language models of different sizes and performances, prompt responses generated by these various language models for the same prompt instruction can be collected for training the reward model. Among them, these different language models can also include a language model with the best performance at present (for example, GPT-4), so that the reward model can learn the best results that the language model can generate.

[0081] For example, using the same large language model, different prompt responses can also be generated by adjusting the temperature coefficient in the large language model to increase the diversity of prompt responses. Among them, the temperature coefficient is a hyperparameter used to adjust the creativity and diversity of the large language model when generating text. By adjusting the temperature coefficient, the probability distribution of the large language model can be affected to control the text generated by the large language model to be more concentrated or more diverse. For example, the temperature coefficient can be a value greater than 0, usually between 0 and 1. The temperature coefficient can affect the probability distribution of the sampled predicted vocabulary of the large language model when generating text. For example, when the temperature coefficient of the large language model is high (such as 0.8, 1 or higher), the large language model will be more inclined to select from a more diverse and different vocabulary, which makes the generated text riskier and more creative, while when the temperature coefficient is low (such as 0.2, 0.3, etc.), the model will mainly select from vocabulary with higher probability, thereby producing smoother and more coherent text.

[0082] For example, the reward model training can also be combined with the benchmark prompt response of the same prompt instruction, wherein the benchmark prompt response can be real, well-labeled, and reliable data used for model training and evaluation (for example, a standard response to the prompt instruction) to provide a reliable benchmark to measure the performance of the reward model. By using the benchmark prompt response to participate in the training of the reward model, the optimization of the reward model can be better guided, so that the reward model can better understand the goal of the task and the correct behavior, thereby improving the generalization performance and practical application effect of the reward model.

[0083] Of course, the above-mentioned several prompt response generation methods are only used as examples and not limitations in the present disclosure. The response generation module 602 of the present disclosure can also use other generation methods to collect prompt responses.

[0084] Therefore, by collecting the prompt instructions and their corresponding prompt responses in the response generation module 602, training data for the reward model can be further constructed based on the collected prompt instructions and their corresponding prompt responses. The response pair construction module 603 can be configured to construct multiple prompt response pairs for each prompt instruction in the prompt instruction set based on multiple prompt responses corresponding to the prompt instruction. Optionally, the response pair construction module 603 can perform the operations described above with reference to step S103.

[0085] For example, for multiple prompt responses corresponding to the same prompt instruction, the relative ranking between these prompt responses can be used to train the reward model, that is, instead of scoring each prompt response, these prompt responses can be labeled with a ranking of good and bad, so that the reward model learns to automatically score each prompt response by trying to maximize the difference between the better prompt response scores and the worse prompt response scores.

[0086] For example, in order to avoid excessive subjectivity in data annotation and unify the scoring standards, in an embodiment of the present disclosure, the absolute score annotation task can be converted into a relative ranking task. Specifically, for the same prompt instruction, according to the absolute score annotation task, different scores may be obtained when different scoring methods are used (for example, when different human experts perform the scoring), which in turn makes it difficult for the reward model to clearly understand the pros and cons of each prompt response when learning. Therefore, in an embodiment of the present disclosure, the prompt responses in the prompt response pair can be processed according to the relative ranking task, that is, the relative pros and cons of the prompt response pair can be ranked in different scoring methods, so as to facilitate obtaining a unified annotation result.

[0087] Therefore, for multiple different prompt responses to the same prompt instruction, multiple prompt response pairs and their ranking results can be obtained by ranking by merit in the response pair construction module 603 .

[0088] Furthermore, with respect to the collection of training data for the reward model, the training data may be obtained in a variety of ways as candidate sources of the training data for the reward model.

[0089] The model training module 604 may be configured to train the reward model based on all prompt response pairs associated with the prompt instruction set by maximizing the difference between the score of the reward model for the better prompt response in each prompt response pair and the score of the worse prompt response in the prompt response pair, wherein the better prompt response and the worse prompt response in the prompt response pair may be determined by ranking the prompt responses in the prompt response pair. Optionally, the model training module 604 may perform the operations described above with reference to step S104.

[0090] For example, a loss function can be constructed for reward model training to maximize the difference between the reward model's score for the better prompt response in each prompt response pair and the score for the worse prompt response in the prompt response pair, so that the scalar score corresponding to the prompt response with a higher ranking is as high as possible. The scalar score corresponding to the prompt response with a lower ranking. Therefore, by maximizing the difference between the reward model's score for the better prompt response and the score for the worse prompt response in the prompt response pair, the distinction accuracy of the reward model can be improved. For example, for six different prompt responses for the same prompt instruction, the difference between the results of the ranking of any two prompt responses can fall in the set {1,2,3,4,5}, and the difference between the results of the ranking of the two prompt responses can reflect the difficulty of distinguishing the two prompt responses, and evaluating the consistency of the scores of different scoring gaps can determine the accuracy of the current reward model in subdividing the scoring gaps.

[0091] In addition, in the embodiments of the present disclosure, targeted optimization can also be performed for specific problems that arise in the subsequent training phase of the large language model, and this targeted optimization can be achieved by taking data related to such problems as input to optimize the reward model. For example, for problems that arise in the large language model during reinforcement learning, when it is determined that these problems are caused by the scoring results of the reward model, prompt response pairs related to these problems can be added to the input of the reward model to optimize the reward model, thereby correcting the deviations that occur in the large language model during reinforcement learning. For example, for the situation in which the reward model gives a high score to the refusal to answer during reinforcement learning of the large language model, this type of refusal to answer data generated by the large language model can be input into the training of the reward model to optimize the reward model, thereby solving the problem of inaccurate learning of the reward model for refusal to answer.

[0092] Therefore, the device for training the reward model disclosed in the present invention can obtain prompt instructions by using a multi-level classification system covering multiple categories of different granularities to balance the category distribution of the large language model, so that the number of samples in different categories is relatively balanced, thereby improving the learning effect of the large language model on each category, and by generating multiple prompt responses on different dimensions, it can balance the result diversity distribution of the large language model, improve the generalization of the reward model, so that the trained reward model can be applicable to large language models of different scales and capabilities, and can produce good results on these models.

[0093] According to yet another aspect of the present disclosure, a device for training a reward model is also provided. Figure 7 A schematic diagram of a device 2000 for training a reward model according to an embodiment of the present disclosure is shown.

[0094] like Figure 7As shown, the device 2000 for training a reward model may include one or more processors 2010, and one or more memories 2020. The memory 2020 stores a computer-readable code, and when the computer-readable code is run by the one or more processors 2010, the method for training a reward model as described above may be executed.

[0095] The processor in the embodiments of the present disclosure may be an integrated circuit chip having signal processing capabilities. The processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc., and may be an X86 architecture or an ARM architecture.

[0096] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the embodiments of the present disclosure are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0097] For example, the method or device according to the embodiment of the present disclosure may also be implemented by Figure 8 The architecture of the computing device 3000 shown in FIG. Figure 8 As shown, the computing device 3000 may include a bus 3010, one or more CPUs 3020, a read-only memory (ROM) 3030, a random access memory (RAM) 3040, a communication port 3050 connected to a network, an input / output component 3060, a hard disk 3070, etc. The storage device in the computing device 3000, such as the ROM 3030 or the hard disk 3070, may store various data or files used for processing and / or communication of the method for training a reward model provided in the present disclosure, as well as program instructions executed by the CPU. The computing device 3000 may also include a user interface 3080. Of course, Figure 8 The architecture shown is only exemplary and can be omitted according to actual needs when implementing different devices. Figure 8 One or more components of a computing device are shown.

[0098] According to another aspect of the present disclosure, a computer-readable storage medium is also provided. Computer-readable instructions are stored on the computer storage medium. When the computer-readable instructions are executed by a processor, the method for training a reward model according to an embodiment of the present disclosure described with reference to the above figures can be executed. The computer-readable storage medium in the embodiment of the present disclosure may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronously linked dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM). It should be noted that memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory. It should be noted that memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0099] The embodiments of the present disclosure also provide a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device performs the method for training a reward model according to the embodiments of the present disclosure.

[0100] Embodiments of the present disclosure provide a method, apparatus, device, and computer-readable storage medium for training a reward model.

[0101] The method provided by the embodiments of the present disclosure obtains prompt instructions based on multi-level classification including a decreasing classification granularity, and generates multiple prompt responses for each prompt instruction, wherein the generated prompt responses may include various prompt responses including baseline prompt responses, prompt responses generated by different language models, prompt responses generated by a large language model at different training stages, and prompt responses controlled by different temperature coefficients, so as to train a reward model based on these prompt responses, so as to maximize the gap between the score of the reward model for a better prompt response and the score of the worse prompt response, thereby generating a trained reward model.

[0102] Through the method provided by the embodiments of the present disclosure, prompt instructions can be obtained by using a multi-level classification system covering multiple categories of different granularities to balance the category distribution of the large language model, so that the number of samples in different categories is relatively balanced, thereby improving the learning effect of the large language model on each category, and by generating multiple prompt responses on different dimensions, the result diversity distribution of the large language model can be balanced, and the generalization of the reward model can be improved, so that the trained reward model can be applicable to large language models of different scales and capabilities, and can produce good results on these models.

[0103] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, program segment, or a part of the code contains at least one executable instruction for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0104] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the embodiments of the present disclosure are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0105] The exemplary embodiments of the present disclosure described in detail above are merely illustrative and not restrictive. It should be understood by those skilled in the art that various modifications and combinations may be made to these embodiments or their features without departing from the principles and spirit of the present disclosure, and such modifications should fall within the scope of the present disclosure.

Claims

1. A method for training a reward model, wherein the reward model is associated with a large language model, the method comprising: Based on a multi-level classification system, a prompt instruction set is obtained, wherein each prompt instruction in the prompt instruction set is a text sequence, and the multi-level classification system includes multi-level classifications with decreasing classification granularity; For each prompt instruction in the prompt instruction set, multiple prompt responses are collected, wherein each prompt response is a text sequence, and the multiple prompt responses include at least two of the following: Prompt responses generated by different language models based on the prompt instructions; The prompt responses generated by the large language model at different training stages based on the prompt instructions include prompt responses generated by the large language model based on the prompt instructions after supervised fine-tuning, prompt responses generated by the large language model based on the prompt instructions during reinforcement learning, and prompt responses generated by the large language model after reinforcement learning; and the large language model generating a prompt response based on the prompt instruction under the control of different temperature coefficients, wherein the temperature coefficients are used to control the diversity of the prompt responses generated by the large language model; For each prompt instruction in the prompt instruction set, constructing a plurality of prompt response pairs based on a plurality of prompt responses corresponding to the prompt instruction; and The reward model is trained based on all prompt-response pairs associated with the prompt instruction set by maximizing the difference between the score given by the reward model to the better prompt response in each prompt-response pair and the score given to the worse prompt response in the prompt-response pair, wherein the better prompt response and the worse prompt response in the prompt-response pair are determined by ranking the prompt responses in the prompt-response pairs according to their merits.

2. The method of claim 1, wherein: The prompt responses generated by different language models based on the prompt instruction include prompt responses generated by language models of different sizes and different performances based on the prompt instruction.

3. The method of claim 1, wherein: The prompt response generated by the large language model based on the prompt instruction under the control of different temperature coefficients includes a prompt response generated by adjusting the temperature coefficient in the process of using the large language model to generate the prompt response based on the prompt instruction.

4. The method of claim 1, wherein: Based on a plurality of prompt responses corresponding to the prompt instruction, constructing a plurality of prompt response pairs comprises: For any two prompt responses in the plurality of prompt responses, construct a prompt response pair, and rank the two prompt responses in the prompt response pair to determine a better prompt response and a worse prompt response in the prompt response pair; Wherein, for each prompt response, a benchmark prompt response corresponding to the prompt instruction is determined as a better prompt response in any prompt response pair associated with the prompt response.

5. The method of claim 4, wherein: Training the reward model includes: For each prompt instruction in the prompt instruction set, based on each prompt-response pair corresponding to the prompt instruction and the quality ranking results of the prompt-response pairs, the reward model is trained by maximizing the difference between the score given by the reward model to the better prompt response in the prompt-response pair and the score given to the worse prompt response in the prompt-response pair.

6. The method of claim 1, wherein: Training the reward model includes: When the plurality of prompt responses include prompt responses generated by the large language model during reinforcement learning based on the prompt instructions, the reward model is trained for deviations in the prompt responses generated by the large language model during reinforcement learning based on the prompt instructions.

7. The method of claim 1, wherein: The plurality of prompt responses also include a reference prompt response corresponding to the prompt instruction.

8. The method of claim 1, wherein: The method further comprises at least two of the following: Collecting training data for the reward model by a human expert, including collecting the prompt instruction set from a prompt instruction database, generating a plurality of prompt responses based on the collected prompt instruction set, and manually ranking the two prompt responses in each prompt response pair; Supplementing said training data collected by human experts with a pre-trained language model, including generating prompt responses based on directed prompt instructions; The pre-trained language model is used to rank the two prompt responses in each prompt-response pair. as well as The open source reward model training data is used as the training data for the reward model.

9. A device for training a reward model, the reward model being associated with a large language model, the device comprising: The instruction acquisition module is configured to acquire a prompt instruction set based on a multi-level classification system, wherein each prompt instruction in the prompt instruction set is a text sequence, and the multi-level classification system includes multi-level classifications with decreasing classification granularity; The response generation module is configured to collect a plurality of prompt responses for each prompt instruction in the prompt instruction set, wherein each prompt response is a text sequence, and the plurality of prompt responses include at least two of the following: Prompt responses generated by different language models based on the prompt instructions; The prompt responses generated by the large language model at different training stages based on the prompt instructions include prompt responses generated by the large language model based on the prompt instructions after supervised fine-tuning, prompt responses generated by the large language model based on the prompt instructions during reinforcement learning, and prompt responses generated by the large language model after reinforcement learning; and the large language model generating a prompt response based on the prompt instruction under the control of different temperature coefficients, wherein the temperature coefficients are used to control the diversity of the prompt responses generated by the large language model; a response pair construction module configured to construct, for each prompt instruction in the prompt instruction set, a plurality of prompt response pairs based on a plurality of prompt responses corresponding to the prompt instruction; and A model training module is configured to train the reward model based on all prompt-response pairs associated with the prompt instruction set by maximizing the difference between the score given by the reward model to the better prompt response in each prompt-response pair and the score given to the worse prompt response in the prompt response pair, wherein the better prompt response and the worse prompt response in the prompt response pair are determined by ranking the prompt responses in the prompt response pairs.

10. A device for training a reward model, comprising: one or more processors; as well as One or more memories storing a computer executable program, wherein when the computer executable program is executed by the processor, the method according to any one of claims 1 to 8 is performed.

11. A computer program product, stored on a computer-readable storage medium, and comprising computer instructions which, when executed by a processor, cause a computer device to perform the method of any one of claims 1 to 8.

12. A computer-readable storage medium having computer-executable instructions stored thereon, wherein the instructions are used to implement the method according to any one of claims 1 to 8 when executed by a processor.

Images