Method and apparatus for determining training data set of large reward model, and electronic device

Abstract

This disclosure provides a method, apparatus, and electronic device for determining the training dataset of a large-scale reward model, relating to the field of artificial intelligence technology, particularly deep learning, natural language processing, and large-scale models. The specific implementation scheme is as follows: acquiring candidate question texts and corresponding response requirements; determining at least one candidate answer text and its scoring data based on the candidate question texts and their corresponding response requirements; selecting a target answer text from the at least one candidate answer text based on its scoring data; and constructing a training dataset for the large-scale reward model based on the target answer text's scoring data and the corresponding candidate question text, for training the large-scale reward model. The training dataset used for training the large-scale reward model is generated by an electronic device combining candidate question texts and corresponding response requirements, resulting in high accuracy and improved accuracy and generalization of the trained large-scale reward model, thereby improving the accuracy of the dialogue model obtained through reinforcement learning based on the large-scale reward model.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, particularly to the fields of deep learning, natural language processing, and large models, and especially to a method, apparatus, and electronic device for determining the training dataset of a large reward model. Background Technology

[0002] Currently, in task-based dialogue generation technologies based on dialogue models, reinforcement learning is applied to the dialogue model in conjunction with a reward model. However, the training dataset used for training the reward model is manually labeled, resulting in poor labeling accuracy. This leads to accuracy and generalization issues in the trained reward model, potentially causing over-optimization of rewards during reinforcement learning and reducing the accuracy of the trained dialogue model. Summary of the Invention

[0003] This disclosure provides a method, apparatus, and electronic device for determining the training dataset of a large reward model.

[0004] According to one aspect of this disclosure, a method for determining a training dataset for a large-scale reward model is provided. The method includes: acquiring candidate question texts and corresponding response requirements; determining at least one candidate answer text and its rating data based on the candidate question texts and their corresponding response requirements; selecting a target answer text from the at least one candidate answer text based on its rating data; and constructing a training dataset for a large-scale reward model based on the target answer text's rating data and the candidate question texts corresponding to the target answer text, for training the large-scale reward model.

[0005] According to another aspect of this disclosure, an apparatus for determining a training dataset for a large reward model is provided. The apparatus includes: a first acquisition module for acquiring candidate question texts and corresponding response requirements; a determination module for determining at least one candidate answer text and its rating data based on the candidate question texts and the corresponding response requirements; a selection module for selecting a target answer text from the at least one candidate answer text based on its rating data; and a construction module for constructing a training dataset for a large reward model based on the target answer text's rating data and the corresponding candidate question text, for training the large reward model.

[0006] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to said at least one processor; wherein the memory stores instructions executable by said at least one processor, said instructions being executed by said at least one processor to enable said at least one processor to perform the method for determining the training dataset of the large reward model proposed above in this disclosure.

[0007] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing a computer to execute the method for determining the training dataset of the large reward model proposed above in this disclosure.

[0008] According to another aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the steps of the method for determining the training dataset of the large reward model proposed above in this disclosure.

[0009] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0011] Figure 1 This is a schematic diagram based on the first embodiment of the present disclosure;

[0012] Figure 2 This is a schematic diagram according to the second embodiment of the present disclosure;

[0013] Figure 3 This is a schematic diagram according to the third embodiment of the present disclosure;

[0014] Figure 4 This is a schematic diagram according to the fourth embodiment of the present disclosure;

[0015] Figure 5 This is a block diagram of an electronic device used to implement a method for determining a training dataset for a large reward model according to embodiments of the present disclosure. Detailed Implementation

[0016] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0017] Currently, in task-based dialogue generation technologies based on dialogue models, reinforcement learning is applied to the dialogue model in conjunction with a reward model. However, the training dataset used for training the reward model is manually labeled, resulting in poor labeling accuracy. This leads to accuracy and generalization issues in the trained reward model, potentially causing over-optimization of rewards during reinforcement learning and reducing the accuracy of the trained dialogue model.

[0018] To address the aforementioned issues, this disclosure proposes a method, apparatus, and electronic device for determining the training dataset of a large-scale reward model.

[0019] Figure 1 The diagram illustrates a first embodiment of this disclosure. It should be noted that the method for determining the training dataset of a large reward model according to this disclosure can be applied to a device for determining the training dataset of a large reward model. This device can be configured in an electronic device so that the electronic device can perform the function of determining the training dataset of a large reward model. The following embodiments use an electronic device as an example for illustration.

[0020] The electronic device can be any device with computing capabilities, such as a personal computer (PC), mobile terminal, server, etc. Mobile terminals can be, for example, in-vehicle devices, mobile phones, tablets, personal digital assistants, wearable devices, smart speakers, servers, server clusters, and other hardware devices with various operating systems, touchscreens, and / or displays. In the following embodiments, an electronic device is used as an example to illustrate the determination of the training dataset for a large reward model.

[0021] like Figure 1 As shown, the method for determining the training dataset for this large reward model may include the following steps:

[0022] Step 101: Obtain the candidate question text and the corresponding response requirements.

[0023] In this embodiment of the disclosure, different objects may give different answers to the same candidate question text, or different objects may require different answers. These differences in answers reflect differences in object preferences. Therefore, the response requirements corresponding to the candidate question text can be determined based on the object's preference data. Specifically, the response requirements corresponding to the candidate question text can be selected from at least one response requirement of the object.

[0024] Preference data includes, for example, search preferences, content preferences, and shopping preferences. For a single candidate question text, multiple answer texts can be provided. The response requirements corresponding to the candidate question text refer to the requirements for the answer to the candidate question text; that is, what requirements the answer text needs to meet.

[0025] The following examples illustrate this. Suppose the candidate question text is "Who is Qin Shi Huang?". Some users require the answer text to include Qin Shi Huang's name; some users require the answer text to include Qin Shi Huang's name and deeds; some users require the answer text to include Qin Shi Huang's name and life story; and some users require the answer text to reflect Qin Shi Huang's personality.

[0026] Specifically, by combining the object's preference data to determine the response requirements corresponding to the candidate question text, and then determining the answer text corresponding to the candidate question text, the determined answer text can reflect the object's preferences, which in turn allows the constructed training dataset to reflect the object's preferences, and the large reward model trained based on the training dataset can reflect the object's preferences, thereby improving the accuracy of the trained large reward model.

[0027] Step 102: Based on the candidate question text and the corresponding response requirements, determine at least one candidate answer text and at least one candidate answer text's scoring data.

[0028] In this embodiment, the scoring data for candidate answer texts includes, for example, the relevance and matching degree between candidate question texts and candidate answer texts. The scoring data for candidate answer texts can also include, for example, the relevance and matching degree between candidate answer texts and the corresponding response requirements of candidate question texts. These can be set according to actual needs.

[0029] Step 103: Select the target answer text from at least one candidate answer text based on the scoring data of at least one candidate answer text.

[0030] In this embodiment of the disclosure, the scoring data may be, for example, a score value. Correspondingly, the electronic device performing step 103 may, for example, select the maximum score value among at least one score value; determine whether the maximum score value is greater than or equal to a preset score threshold; if the maximum score value is greater than or equal to the preset score threshold, determine the candidate answer text corresponding to the maximum score value as the target answer text; if the maximum score value is less than the preset score threshold, determine that the target answer text does not exist among at least one candidate answer text, that is, the target answer text is not selected.

[0031] In addition, electronic devices can also perform the following process: if the target answer text is not selected, the candidate answer text is re-determined and selected based on the candidate question text and the corresponding response requirements.

[0032] In cases where the target answer text is not selected, re-determining and selecting candidate answer texts to select the target answer text can improve the accuracy of the determined target answer text, thereby improving the accuracy of the constructed training dataset.

[0033] Step 104: Based on the scoring data of the target answer text and the candidate question text corresponding to the target answer text, construct a training dataset for the large reward model, which is used to train the large reward model.

[0034] Large-scale reward models refer to reward models that are considered large-scale models. Large-scale models, in particular, are neural network models with a large number of parameters and complex structures. In other words, large-scale reward models are reward models with a large number of parameters and complex structures.

[0035] In this embodiment of the disclosure, the electronic device may perform step 104 as follows: obtain the historical training dataset of the large reward model; add the score data of the target answer text and the candidate question text corresponding to the target answer text to the historical training dataset to obtain the training dataset of the large reward model.

[0036] The historical training dataset for the large-scale reward model can be a training dataset that has already been used to train the large-scale reward model. The methods for obtaining the historical answer text corresponding to the historical question text in the historical training dataset can include at least one of the following: generation by a question-and-answer dialogue model, search in a knowledge base, or capture of the dialogue process. The methods for obtaining the rating data of the historical answer text in the historical training dataset can include at least one of the following: determination based on the similarity between the historical answer text and the historical question text, determination based on the occurrence frequency or occurrence of question-and-answer pairs containing the historical answer text and the historical question text in the knowledge base, or annotation.

[0037] In particular, real-time supplementation of the training dataset can improve the accuracy of the training dataset, thereby improving the accuracy of the large reward model trained.

[0038] In this embodiment of the disclosure, the electronic device may also perform the following process: when the candidate question text is a historical question text in the historical training dataset of a large reward model, the historical answer text and the score data of the historical answer text corresponding to the candidate question text in the historical training dataset are replaced according to the target answer text and the score data of the target answer text.

[0039] Among these improvements, fixing the training dataset can further enhance its accuracy, which in turn can further improve the accuracy of the large reward model trained on it.

[0040] In this embodiment of the disclosure, after step 104, the electronic device may further perform the following processes: obtaining an initial large reward model; training the initial large reward model according to the training dataset to obtain a trained large reward model; and training the initial dialogue model according to the trained large reward model and the training dialogue dataset.

[0041] The training process of the large reward model by the electronic device can be as follows: inputting sample question texts and sample answer texts from the training dataset into the large reward model to obtain the predicted score data output by the large reward model; determining the value of the loss function of the large reward model based on the score data of the sample answer texts and the predicted score data; and adjusting the parameters of the large reward model based on the value of the loss function to obtain the trained large reward model.

[0042] The training dialogue dataset may include sample dialogue questions. Correspondingly, the training process of the electronic device for the dialogue model can be as follows: input the sample dialogue questions from the training dialogue dataset into the dialogue model to obtain the predicted dialogue answers output by the model; input the sample dialogue questions and predicted dialogue answers into a large reward model to obtain the predicted score data output by the model; determine the value of the loss function of the dialogue model based on the predicted score data; and adjust the parameters of the dialogue model based on the value of the loss function to achieve training.

[0043] It should be noted that the initial dialogue model can be a pre-trained dialogue model; or, the initial dialogue model can be a pre-trained and fine-tuned dialogue model.

[0044] In this process, electronic devices train a large reward model using a predetermined training dataset; then, by combining the trained large reward model with the training of the dialogue model, the accuracy of the trained large reward model can be improved, which in turn further improves the accuracy of the trained dialogue model.

[0045] The method for determining the training dataset of a large-scale reward model according to this embodiment involves: acquiring candidate question texts and corresponding response requirements; determining at least one candidate answer text and its score data based on the candidate question texts and their corresponding response requirements; selecting a target answer text from the at least one candidate answer text based on its score data; and constructing a training dataset for the large-scale reward model based on the target answer text's score data and the corresponding candidate question text, for training the large-scale reward model. The training dataset for training the large-scale reward model is generated by an electronic device combining candidate question texts and corresponding response requirements, resulting in high accuracy and improved accuracy and generalization of the trained large-scale reward model, thereby improving the accuracy of the dialogue model obtained through reinforcement learning based on the large-scale reward model.

[0046] The electronic device can acquire the original question text and at least one candidate response requirement; based on the category of the original question text, it selects one candidate response requirement from the at least one candidate response requirement, thereby determining the candidate question text and its corresponding response requirement, thus ensuring the matching degree between the candidate question text and the corresponding response requirement. For example... Figure 2 As shown, Figure 2 This is a schematic diagram based on the second embodiment of the present disclosure. Figure 2 The illustrated embodiment may include the following steps:

[0047] Step 201: Obtain the original question text and at least one candidate response requirement.

[0048] In this embodiment of the disclosure, the original question text may be obtained in at least one of the following ways: by crawling from web page text, by extracting from chat logs, etc.

[0049] At least one candidate response requirement can be determined based on the object's preference data. The specific response requirement corresponding to the candidate question text can be selected from at least one response requirement of the object.

[0050] Step 202: Determine the category of the original question text.

[0051] In this embodiment of the disclosure, the electronic device performing step 202 may, for example, input the original question text into a classification model and obtain the category output by the classification model. The category of the original question text may refer to the domain to which the original question text belongs and / or the question-and-answer type of the original question text. The domain to which the original question text belongs may be, for example, the communications domain, the biological domain, the modeling domain, etc., and can be set according to actual needs. Furthermore, the domain to which the original question text belongs may also be a sub-domain among the aforementioned domains.

[0052] The question-and-answer types of the original question text include, for example, knowledge-based question-and-answer, translation, choice, and judgment types.

[0053] In the case where the category of the original question text refers to the domain to which the original question text belongs and the question-and-answer type of the original question text, the category can be set according to actual needs, such as the knowledge question-and-answer category in the field of communication, the translation category in the field of communication, the choice category in the field of biology, and the judgment category in the field of modeling.

[0054] Step 203: Select a candidate response requirement from at least one candidate response requirement according to the category.

[0055] In this embodiment of the disclosure, the electronic device may perform step 203 as follows: determine the relevance between the category and at least one candidate response requirement; and select a candidate response requirement whose relevance is greater than or equal to the relevance threshold.

[0056] In one example, the relevance between a category and a candidate response requirement can be determined based on the semantic similarity between the two. Semantic similarity refers to the feature similarity between the semantic features of a category and the semantic features of a candidate response requirement.

[0057] In another example, the process by which the electronic device determines the relevance between the category and the candidate response requirement can be as follows: the electronic device inputs the category and the candidate response requirement into a relevance model, obtains the relevance output by the relevance model, and determines this relevance as the relevance between the category and the candidate response requirement.

[0058] Among these methods, selecting candidate response requirements based on the relevance between the category and at least one candidate response requirement, and then determining the candidate question text and the corresponding response requirement, can improve the matching degree between the candidate question text and the corresponding response requirement.

[0059] Step 204: If candidate response requirements are selected, the original question text is determined as the candidate question text, and the selected candidate response requirements are determined as the response requirements corresponding to the candidate question text.

[0060] In this embodiment of the disclosure, as an alternative to steps 201 to 204, the electronic device may perform the following process: acquiring the original question text and at least one candidate response requirement; determining the relevance between the original question text and at least one candidate response requirement; selecting a candidate response requirement from the at least one candidate response requirement based on the relevance; and, if a candidate response requirement is selected, determining the original question text as the candidate question text and determining the selected candidate response requirement as the response requirement corresponding to the candidate question text.

[0061] The process by which the electronic device selects a candidate response from at least one candidate response based on relevance can be, for example, by identifying candidate response responses with a relevance greater than or equal to a preset relevance threshold as selected candidate response responses. If the relevance between at least one candidate response and the original question text is less than the preset relevance threshold, then it is determined that no candidate response has been selected.

[0062] In this embodiment of the disclosure, the process by which the electronic device determines the relevance between the original question text and at least one candidate response requirement can be, for example, inputting the original question text and the candidate response requirement to be processed into a relevance classification model for each candidate response requirement to be processed among the at least one candidate response requirements, obtaining the relevance output by the relevance classification model; and determining the relevance output by the relevance classification model as the relevance between the original question text and the candidate response requirement to be processed.

[0063] In this process, the electronic device combines the relevance between the original question text and at least one candidate response requirement, selects one candidate response requirement from at least one candidate response requirement, and then determines the candidate question text and the corresponding response requirement, which can further ensure the matching degree between the candidate question text and the corresponding response requirement.

[0064] The use of a relevance classification model can improve the accuracy of determining the relevance between the candidate response requirements and the original question text, thereby further ensuring the matching degree between the candidate question text and the corresponding response requirements.

[0065] In addition, the electronic device can also perform the following process: if no candidate response requirement is selected, stop identifying the original question text as a candidate question text. Specifically, stopping the identification of the original question text as a candidate question text when no candidate response requirement is selected can further improve the matching degree between the identified candidate question texts and the corresponding response requirements.

[0066] Step 205: Based on the candidate question text and the corresponding response requirements, determine at least one candidate answer text and at least one candidate answer text's scoring data.

[0067] Step 206: Select the target answer text from at least one candidate answer text based on the scoring data of at least one candidate answer text.

[0068] Step 207: Based on the scoring data of the target answer text and the candidate question text corresponding to the target answer text, construct a training dataset for the large reward model, which is used to train the large reward model.

[0069] It should be noted that for details of steps 205 to 207, please refer to [link / reference needed]. Figure 1 Steps 102 to 104 in the illustrated embodiment will not be described in detail here.

[0070] The method for determining the training dataset of a large-scale reward model according to embodiments of this disclosure involves: acquiring an original question text and at least one candidate response requirement; determining the category of the original question text; selecting a candidate response requirement from the at least one candidate response requirement based on the category; if a candidate response requirement is selected, determining the original question text as the candidate question text and the selected candidate response requirement as the response requirement corresponding to the candidate question text; determining at least one candidate answer text and the scoring data of the at least one candidate answer text based on the candidate question text and the response requirement corresponding to the candidate question text; selecting a target answer text from the at least one candidate answer text based on the scoring data of the at least one candidate answer text; and constructing a training dataset for the large-scale reward model based on the scoring data of the target answer text and the candidate question text corresponding to the target answer text, for training the large-scale reward model. The electronic device, by combining the category of the original question text with the selection of a candidate response requirement from the at least one candidate response requirement, thereby determining the candidate question text and the corresponding response requirement, ensures the matching degree between the candidate question text and the corresponding response requirement, thereby improving the accuracy of the determined training dataset, improving the accuracy and generalization of the trained large-scale reward model, and thus improving the accuracy of the dialogue model obtained by reinforcement learning based on the large-scale reward model.

[0071] In this system, the electronic device can determine the prompt words for the candidate question text by combining the category of the candidate question text and the corresponding response requirements. Then, based on these prompt words, it can determine at least one candidate answer text corresponding to the candidate question text, thereby generating candidate answer text that matches the response requirements and improving the matching degree between the generated candidate answer text and the candidate question text. For example... Figure 3 As shown, Figure 3 This is a schematic diagram based on the third embodiment of the present disclosure. Figure 3 The illustrated embodiment may include the following steps:

[0072] Step 301: Obtain the candidate question text and the corresponding response requirements for the candidate question text.

[0073] Step 302: Obtain the category of the candidate question text.

[0074] Step 303: Determine the prompt words for the candidate question text based on the category of the candidate question text and the corresponding response requirements.

[0075] In this embodiment of the disclosure, the electronic device may perform step 303 as follows: input the category of the candidate question text and the corresponding response requirement of the candidate question text into the prompt word generation model, obtain the prompt words output by the prompt word generation model, and determine the prompt words output by the prompt word generation model as the prompt words of the candidate question text.

[0076] The prompt word generation model can be used to generate prompt words that contain the category and the response requirement, or to generate prompt words that contain keywords that match the category and the response requirement.

[0077] In this process, the electronic device combines the category of the candidate question text, the corresponding response requirements of the candidate question text, and the prompt word generation model to generate prompt words. The prompt word generation model has a large computational load and high computational accuracy, which can improve the accuracy of the generated prompt words, and thus improve the accuracy of the generated candidate answer text.

[0078] Step 304: Based on the candidate question text and the prompts in the candidate question text, determine at least one candidate answer text corresponding to the candidate question text.

[0079] In this embodiment of the disclosure, the electronic device performing step 304 may, for example, involve inputting the candidate question text and its prompts into the answer generation model to obtain a candidate answer text output by the answer generation model; then, inputting the candidate question text and its prompts back into the answer generation model to obtain another candidate answer text output by the answer generation model. If the required number of candidate answer texts is greater than two, the above process is repeated, and the device obtains the required number of candidate answer texts.

[0080] Step 305: Determine the scoring data for at least one candidate answer text based on the candidate question text and the corresponding response requirements.

[0081] In one example of this disclosure, the electronic device performing step 305 may, for example, input the candidate question text, the corresponding response requirements of the candidate question text, and at least one candidate answer text into a sorting model to obtain the sorting result of at least one candidate answer text; and determine the scoring data of at least one candidate answer text based on the sorting result.

[0082] Before inputting the candidate question text, the corresponding response requirements, and at least one candidate answer text into the ranking model, the number of candidate answer texts corresponding to the candidate question text can be determined. If the number is one, the scoring data for that candidate answer text is determined using the scoring model. Alternatively, if the number is two or more, the electronic device inputs the candidate question text, the corresponding response requirements, and multiple candidate answer texts into the ranking model, and obtains the ranking result of the multiple candidate answer texts output by the ranking model.

[0083] In the ranking results, candidate answer texts with higher scores are listed first, while those with lower scores are listed last. Therefore, the score data for each candidate answer text can be determined by combining the ranking order of the candidate answer texts in the ranking results.

[0084] Specifically, the scoring data for at least one candidate answer text is determined by combining a ranking model, candidate question texts, corresponding response requirements for candidate question texts, and at least one candidate answer text. The ranking model has a large computational load but high computational accuracy, thereby improving the accuracy of the scoring data for the determined candidate answer texts.

[0085] In another example of this disclosure, the electronic device may perform step 305 as follows: for each candidate answer text to be processed, input the candidate question text, the corresponding response requirement of the candidate question text, and the candidate answer text to be processed into the scoring model, and obtain the scoring data of the candidate answer text to be processed output by the scoring model.

[0086] Specifically, the scoring data for at least one candidate answer text is determined by combining the scoring model, candidate question text, corresponding response requirements for the candidate question text, and at least one candidate answer text. The scoring model has a large computational load and high accuracy, which can further improve the accuracy of the scoring data of the determined candidate answer text.

[0087] Step 306: Select the target answer text from at least one candidate answer text based on the scoring data of at least one candidate answer text.

[0088] Step 307: Based on the scoring data of the target answer text and the candidate question text corresponding to the target answer text, construct a training dataset for the large reward model, which is used to train the large reward model.

[0089] It should be noted that for details regarding steps 301, 306, and 307, please refer to [the relevant documentation / reference]. Figure 1 Steps 101, 103 to 104 in the illustrated embodiment will not be described in detail here.

[0090] The method for determining the training dataset of a large reward model according to embodiments of this disclosure includes: acquiring candidate question texts and corresponding response requirements; acquiring the category of the candidate question texts; determining prompt words for the candidate question texts based on their category and response requirements; determining at least one candidate answer text based on the candidate question texts and their prompt words; determining the scoring data of at least one candidate answer text based on the candidate question texts and their corresponding response requirements; selecting a target answer text from the at least one candidate answer text based on its scoring data; and then selecting a target answer text based on its scoring data. A training dataset for a large-scale reward model is constructed by combining the candidate question texts corresponding to the target answer text with the candidate question texts. This dataset is used to train the large-scale reward model. The electronic device determines the prompt words of the candidate question texts by combining the category of the candidate question texts with the corresponding response requirements. Then, it determines at least one candidate answer text corresponding to the candidate question texts by combining the prompt words. This enables the generation of candidate answer texts that match the response requirements, improves the matching degree between the generated candidate answer texts and the candidate question texts, and thus improves the accuracy of the determined training dataset. This improves the accuracy and generalization of the trained large-scale reward model, thereby improving the accuracy of the dialogue model obtained by reinforcement learning based on the large-scale reward model.

[0091] To implement the above embodiments, this disclosure also provides an apparatus for determining the training dataset of a large reward model. For example... Figure 4 As shown, Figure 4 This is a schematic diagram according to the fourth embodiment of the present disclosure. The device 40 for determining the training dataset of the large reward model may include: a first acquisition module 401, a determination module 402, a selection module 403, and a construction module 404.

[0092] The system includes a first acquisition module 401, used to acquire candidate question texts and corresponding response requirements; a determination module 402, used to determine at least one candidate answer text and its score data based on the candidate question texts and their corresponding response requirements; a selection module 403, used to select a target answer text from the at least one candidate answer text based on its score data; and a construction module 404, used to construct a training dataset for a large-scale reward model based on the target answer text's score data and the corresponding candidate question text, for training the large-scale reward model.

[0093] As one possible implementation of this disclosure, the first acquisition module 401 is specifically configured to: acquire the original question text and at least one candidate response requirement; determine the category of the original question text; select a candidate response requirement from the at least one candidate response requirement according to the category; and, if a candidate response requirement is selected, determine the original question text as the candidate question text and determine the selected candidate response requirement as the response requirement corresponding to the candidate question text.

[0094] As one possible implementation of this disclosure, the first acquisition module 401 is further configured to: determine the relevance between the category and the at least one candidate response requirement; and select candidate response requirements whose relevance is greater than or equal to the relevance threshold.

[0095] As one possible implementation of this disclosure, the first acquisition module 401 is specifically configured to: acquire the original question text and at least one candidate response requirement; determine the relevance between the original question text and the at least one candidate response requirement; select a candidate response requirement from the at least one candidate response requirement based on the relevance; and, if a candidate response requirement is selected, determine the original question text as the candidate question text and determine the selected candidate response requirement as the response requirement corresponding to the candidate question text.

[0096] As one possible implementation of this disclosure, the first acquisition module 401 is further configured to: for each candidate response requirement to be processed in the at least one candidate response requirement, input the original question text and the candidate response requirement to be processed into a relevance classification model, and obtain the relevance output by the relevance classification model; and determine the relevance output by the relevance classification model as the relevance between the original question text and the candidate response requirement to be processed.

[0097] As one possible implementation of this disclosure, the first acquisition module 401 is further configured to stop determining the original question text as the candidate question text when no candidate response requirement is selected.

[0098] As one possible implementation of this disclosure, the at least one candidate response requirement is determined based on the object's preference data.

[0099] As one possible implementation of this disclosure, the determining module 402 includes an acquisition unit, a first determining unit, a second determining unit, and a third determining unit; the acquisition unit is used to acquire the category of the candidate question text; the first determining unit is used to determine the prompt words of the candidate question text according to the category of the candidate question text and the corresponding response requirements of the candidate question text; the second determining unit is used to determine at least one candidate answer text corresponding to the candidate question text according to the candidate question text and the prompt words of the candidate question text; the third determining unit is used to determine the scoring data of the at least one candidate answer text according to the candidate question text and the corresponding response requirements of the candidate question text.

[0100] As one possible implementation of this disclosure, the first determining unit is specifically configured to: input the category of the candidate question text and the response requirement corresponding to the candidate question text into the prompt word generation model, obtain the prompt words output by the prompt word generation model; and determine the prompt words output by the prompt word generation model as the prompt words of the candidate question text.

[0101] As one possible implementation of this disclosure, the third determining unit is specifically used to input the candidate question text, the response requirements corresponding to the candidate question text, and the at least one candidate answer text into a sorting model to obtain a sorting result for the at least one candidate answer text; and to determine the scoring data of the at least one candidate answer text based on the sorting result.

[0102] As one possible implementation of this disclosure, the third determining unit is specifically used to input the candidate question text, the corresponding response requirement of the candidate question text, and the candidate answer text to be processed into a scoring model for each candidate answer text to be processed of the candidate question text, and to obtain the scoring data of the candidate answer text to be processed output by the scoring model.

[0103] As one possible implementation of this disclosure, the apparatus further includes: a re-execution module, configured to, when the target answer text is not selected, re-determine and select candidate answer text based on the candidate question text and the response requirements corresponding to the candidate question text.

[0104] As one possible implementation of this disclosure, the construction module 404 is specifically used to: obtain the historical training dataset of the large reward model; add the scoring data of the target answer text and the candidate question text corresponding to the target answer text to the historical training dataset to obtain the training dataset of the large reward model.

[0105] As one possible implementation of this disclosure, the construction module 404 is further configured to, when the candidate question text is a historical question text in the historical training dataset of the large reward model, replace the historical answer text corresponding to the candidate question text and the score data of the historical answer text in the historical training dataset according to the target answer text and the score data of the target answer text.

[0106] As one possible implementation of this disclosure, the apparatus further includes a second acquisition module, a first training module, and a second training module; the second acquisition module is used to acquire an initial large reward model; the first training module is used to train the initial large reward model according to the training dataset to obtain a trained large reward model; the second training module is used to train an initial dialogue model according to the trained large reward model and the training dialogue dataset.

[0107] As one possible implementation of this disclosure, the training dialogue dataset includes sample dialogue questions; the second training module is specifically configured to: input the sample dialogue questions from the training dialogue dataset into the dialogue model to obtain the predicted dialogue answers output by the dialogue model; input the sample dialogue questions and the predicted dialogue answers into the large reward model to obtain the predicted score data output by the large reward model; determine the value of the loss function of the dialogue model based on the predicted score data; and perform parameter adjustment processing on the dialogue model based on the value of the loss function to achieve training.

[0108] The apparatus for determining the training dataset of a large-scale reward model according to an embodiment of this disclosure acquires candidate question texts and corresponding response requirements; determines at least one candidate answer text and its score data based on the candidate question texts and their corresponding response requirements; selects a target answer text from the at least one candidate answer text based on its score data; and constructs a training dataset for the large-scale reward model based on the target answer text's score data and the corresponding candidate question text, for training the large-scale reward model. The training dataset for training the large-scale reward model is generated by an electronic device combining candidate question texts and corresponding response requirements, resulting in high accuracy and improved accuracy and generalization of the trained large-scale reward model, thereby improving the accuracy of the dialogue model obtained through reinforcement learning based on the large-scale reward model.

[0109] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision, and disclosure of users' personal information are all carried out with the consent of the users, and all comply with the provisions of relevant laws and regulations, and do not violate public order and good morals.

[0110] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0111] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0112] like Figure 5 As shown, device 500 includes a computing unit 501, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 502 or a computer program loaded from storage unit 508 into random access memory (RAM) 503. RAM 503 may also store various programs and data required for the operation of device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. Input / output (I / O) interface 505 is also connected to bus 504.

[0113] Multiple components in device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0114] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as the method for determining the training dataset of a large reward model. For example, in some embodiments, the method for determining the training dataset of a large reward model can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the method for determining the training dataset of a large reward model described above can be performed. Alternatively, in other embodiments, computing unit 501 may be configured by any other suitable means (e.g., by means of firmware) to perform a method for determining the training dataset of a large reward model.

[0115] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0116] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0117] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0118] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0119] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0120] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0121] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0122] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for determining the training dataset for a large reward model, the method comprising: Obtain candidate question texts and corresponding response requirements for the candidate question texts; The response requirement is determined based on the object's preference data; The candidate question text is the original question text from which the response requirement is obtained by selecting from at least one candidate response requirement; The response requirements are selected from at least one candidate response requirements based on the relevance between the original question text and the at least one candidate response requirements, or they are selected from at least one candidate response requirements based on the category of the original question text. Based on the candidate question text and the corresponding response requirements, determine at least one candidate answer text and the scoring data of the at least one candidate answer text. The at least one candidate answer text is determined by combining the prompt words of the candidate question text; the prompt words are determined according to the category of the candidate question text and the response requirements; the scoring data is determined by combining the ranking results of the at least one candidate answer text by the ranking model, or by the scoring model combining the candidate answer text, the candidate question text and the response requirements. Based on the scoring data of the at least one candidate answer text, select the target answer text from the at least one candidate answer text; Based on the scoring data of the target answer text and the candidate question text corresponding to the target answer text, a training dataset for a large-scale reward model is constructed for training the large-scale reward model.

2. The method according to claim 1, wherein, The process of obtaining candidate question texts and corresponding response requirements includes: Obtain the original question text and at least one candidate response requirement; Determine the category of the original question text; Select a candidate response requirement from the at least one candidate response requirement according to the category; If candidate response requirements are selected, the original question text is determined as the candidate question text, and the selected candidate response requirements are determined as the response requirements corresponding to the candidate question text.

3. The method according to claim 2, wherein, The step of selecting a candidate response requirement from the at least one candidate response requirement according to the category includes: Determine the relevance between the category and the at least one candidate response requirement; Select candidate responses whose relevance is greater than or equal to the relevance threshold.

4. The method according to claim 1, wherein, The process of obtaining candidate question texts and corresponding response requirements includes: Obtain the original question text and at least one candidate response requirement; Determine the relevance between the original question text and the at least one candidate response requirement; Based on the relevance, a candidate response requirement is selected from the at least one candidate response requirement; If candidate response requirements are selected, the original question text is determined as the candidate question text, and the selected candidate response requirements are determined as the response requirements corresponding to the candidate question text.

5. The method according to claim 4, wherein, Determining the relevance between the original question text and the at least one candidate response requirement includes: For each candidate response requirement to be processed in the at least one candidate response requirement, the original question text and the candidate response requirement to be processed are input into the relevance classification model to obtain the relevance output by the relevance classification model; The relevance output by the relevance classification model is determined as the relevance between the original question text and the candidate response requirement to be processed.

6. The method according to claim 2 or 4, wherein, The method further includes: If no candidate response is selected, the original question text is no longer identified as the candidate question text.

7. The method according to claim 2 or 4, wherein, The at least one candidate response requirement is determined based on the object's preference data.

8. The method according to claim 1, wherein, The step of determining at least one candidate answer text and its scoring data based on the candidate question text and the corresponding response requirements includes: Obtain the category of the candidate question text; Based on the category of the candidate question text and the corresponding response requirements, determine the prompt words for the candidate question text; Based on the candidate question text and the prompt words of the candidate question text, determine at least one candidate answer text corresponding to the candidate question text; Based on the candidate question text and the corresponding response requirements, the scoring data of the at least one candidate answer text is determined.

9. The method according to claim 8, wherein, The step of determining the prompt words for the candidate question text based on its category and the corresponding response requirements includes: Input the category of the candidate question text and the corresponding response requirements into the prompt word generation model, and obtain the prompt words output by the prompt word generation model; The prompt words output by the prompt word generation model are determined as the prompt words for the candidate question text.

10. The method according to claim 8, wherein, The step of determining the scoring data for at least one candidate answer text based on the candidate question text and the corresponding response requirements includes: Input the candidate question text, the corresponding response requirements of the candidate question text, and the at least one candidate answer text into the sorting model to obtain the sorting result of the at least one candidate answer text; Based on the sorting results, the scoring data for the at least one candidate answer text is determined.

11. The method according to claim 8, wherein, The step of determining the scoring data for at least one candidate answer text based on the candidate question text and the corresponding response requirements includes: For each candidate answer text to be processed in the candidate question text, the candidate question text, the corresponding response requirement of the candidate question text, and the candidate answer text to be processed are input into the scoring model to obtain the scoring data of the candidate answer text to be processed output by the scoring model.

12. The method according to claim 1, wherein, The method further includes: If the target answer text is not selected, the candidate answer text is determined and selected again based on the candidate question text and the corresponding response requirements.

13. The method according to claim 1, wherein, The step of constructing a training dataset for a large reward model based on the scoring data of the target answer text and the candidate question texts corresponding to the target answer text includes: Obtain the historical training dataset of the large reward model; The scoring data of the target answer text and the candidate question text corresponding to the target answer text are added to the historical training dataset to obtain the training dataset of the large reward model.

14. The method according to claim 1 or 13, wherein, The step of constructing a training dataset for a large reward model based on the scoring data of the target answer text and the candidate question texts corresponding to the target answer text includes: If the candidate question text is a historical question text in the historical training dataset of the large reward model, the historical answer text corresponding to the candidate question text and the score data of the historical answer text in the historical training dataset are replaced according to the target answer text and the score data of the target answer text.

15. The method according to claim 1, wherein, The method further includes: Obtain the initial large-scale reward model; Based on the training dataset, the initial large reward model is trained to obtain a trained large reward model. Based on the pre-trained large reward model and the training dialogue dataset, the initial dialogue model is trained.

16. The method according to claim 15, wherein, The training dialogue dataset includes sample dialogue questions; The step of training the initial dialogue model based on the trained large reward model and the training dialogue dataset includes: Input the sample dialogue questions from the training dialogue dataset into the dialogue model, and obtain the predicted dialogue answer output by the dialogue model; Input the sample dialogue question and the predicted dialogue answer into the large reward model to obtain the predicted score data output by the large reward model; Based on the predicted scoring data, determine the value of the loss function of the dialogue model; Based on the value of the loss function, the parameters of the dialogue model are adjusted to achieve training.

17. An apparatus for determining a training dataset for a large reward model, the apparatus comprising: The first acquisition module is used to acquire candidate question texts and corresponding response requirements for the candidate question texts; The response requirement is determined based on the object's preference data; The candidate question text is the original question text from which the response requirement is obtained by selecting from at least one candidate response requirement; The response requirements are selected from at least one candidate response requirements based on the relevance between the original question text and the at least one candidate response requirements, or they are selected from at least one candidate response requirements based on the category of the original question text. The determining module is used to determine at least one candidate answer text corresponding to the candidate question text and the scoring data of the at least one candidate answer text based on the candidate question text and the corresponding response requirements; The at least one candidate answer text is determined by combining the prompt words of the candidate question text; the prompt words are determined according to the category of the candidate question text and the response requirements; the scoring data is determined by combining the ranking results of the at least one candidate answer text by the ranking model, or by the scoring model combining the candidate answer text, the candidate question text and the response requirements. The selection module is used to select the target answer text from the at least one candidate answer text based on the scoring data of the at least one candidate answer text; The construction module is used to construct a training dataset for a large-scale reward model based on the scoring data of the target answer text and the candidate question text corresponding to the target answer text, and to train the large-scale reward model.

18. The apparatus according to claim 17, wherein, The first acquisition module is specifically used for, Obtain the original question text and at least one candidate response requirement; Determine the category of the original question text; Select a candidate response requirement from the at least one candidate response requirement according to the category; If candidate response requirements are selected, the original question text is determined as the candidate question text, and the selected candidate response requirements are determined as the response requirements corresponding to the candidate question text.

19. The apparatus according to claim 18, wherein, The first acquisition module is also specifically used for, Determine the relevance between the category and the at least one candidate response requirement; Select candidate responses whose relevance is greater than or equal to the relevance threshold.

20. The apparatus according to claim 17, wherein, The first acquisition module is specifically used for, Obtain the original question text and at least one candidate response requirement; Determine the relevance between the original question text and the at least one candidate response requirement; Based on the relevance, a candidate response requirement is selected from the at least one candidate response requirement; If candidate response requirements are selected, the original question text is determined as the candidate question text, and the selected candidate response requirements are determined as the response requirements corresponding to the candidate question text.

21. The apparatus according to claim 20, wherein, The first acquisition module is also specifically used for, For each candidate response requirement to be processed in the at least one candidate response requirement, the original question text and the candidate response requirement to be processed are input into the relevance classification model to obtain the relevance output by the relevance classification model; The relevance output by the relevance classification model is determined as the relevance between the original question text and the candidate response requirement to be processed.

22. The apparatus according to claim 18 or 20, wherein, The first acquisition module is further configured to stop identifying the original question text as the candidate question text if no candidate response requirement is selected.

23. The apparatus according to claim 18 or 20, wherein, The at least one candidate response requirement is determined based on the object's preference data.

24. The apparatus according to claim 17, wherein, The determining module includes an acquisition unit, a first determining unit, a second determining unit, and a third determining unit; The acquisition unit is used to acquire the category of the candidate question text; The first determining unit is used to determine the prompt words of the candidate question text according to the category of the candidate question text and the corresponding response requirements of the candidate question text; The second determining unit is configured to determine at least one candidate answer text corresponding to the candidate question text based on the candidate question text and the prompt words of the candidate question text; The third determining unit is used to determine the scoring data of the at least one candidate answer text based on the candidate question text and the corresponding response requirements.

25. The apparatus according to claim 24, wherein, The first determining unit is specifically used for, Input the category of the candidate question text and the corresponding response requirements into the prompt word generation model, and obtain the prompt words output by the prompt word generation model; The prompt words output by the prompt word generation model are determined as the prompt words for the candidate question text.

26. The apparatus according to claim 24, wherein, The third determining unit is specifically used for, Input the candidate question text, the corresponding response requirements of the candidate question text, and the at least one candidate answer text into the sorting model to obtain the sorting result of the at least one candidate answer text; Based on the sorting results, the scoring data for the at least one candidate answer text is determined.

27. The apparatus according to claim 24, wherein, The third determining unit is specifically used for, For each candidate answer text to be processed in the candidate question text, the candidate question text, the corresponding response requirement of the candidate question text, and the candidate answer text to be processed are input into the scoring model to obtain the scoring data of the candidate answer text to be processed output by the scoring model.

28. The apparatus according to claim 17, wherein, The device further includes a re-execution module, used to re-determine and select candidate answer texts based on the candidate question texts and the corresponding response requirements when the target answer text is not selected.

29. The apparatus according to claim 17, wherein, The building module is specifically used for, Obtain the historical training dataset of the large reward model; The scoring data of the target answer text and the candidate question text corresponding to the target answer text are added to the historical training dataset to obtain the training dataset of the large reward model.

30. The apparatus according to claim 17 or 29, wherein, The building module is also specifically used for, If the candidate question text is a historical question text in the historical training dataset of the large reward model, the historical answer text corresponding to the candidate question text and the score data of the historical answer text in the historical training dataset are replaced according to the target answer text and the score data of the target answer text.

31. The apparatus according to claim 17, wherein, The device further includes a second acquisition module, a first training module, and a second training module; The second acquisition module is used to acquire the initial large-scale reward model; The first training module is used to train the initial large reward model based on the training dataset to obtain a trained large reward model. The second training module is used to train the initial dialogue model based on the trained large reward model and the training dialogue dataset.

32. The apparatus according to claim 31, wherein, The training dialogue dataset includes sample dialogue questions; the second training module is specifically used for, Input the sample dialogue questions from the training dialogue dataset into the dialogue model, and obtain the predicted dialogue answer output by the dialogue model; Input the sample dialogue question and the predicted dialogue answer into the large reward model to obtain the predicted score data output by the large reward model; Based on the predicted scoring data, determine the value of the loss function of the dialogue model; Based on the value of the loss function, the parameters of the dialogue model are adjusted to achieve training.

33. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 16.

34. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 16.

35. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 16.

Images