Replay generation model training method and related device

Abstract

The invention discloses a reply generation model training method and related device.The reply generation model training method comprises the steps that a dialogue training set of a current round is obtained, the dialogue training set of the current round comprises at least one first dialogue, and the first dialogue comprises a first query and a corresponding first reply; the at least one first dialogue comprises a first dialogue corresponding to the current round, and a first reply in the first dialogue corresponding to the current round is generated based on a corresponding first query by a reinforcement learning large model obtained in the previous round of the current round; performing supervised fine tuning on the pre-trained large model by using the dialogue training set of the current round to obtain a supervised fine tuning large model of the current round; performing reinforcement learning by using the supervised fine-tuning large model of the current round to obtain a reinforcement learning large model of the current round; wherein the reinforcement learning large model obtained in the final round serves as a reply generation model. According to the scheme, the training effect and the training efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of natural language processing technology, and in particular to a response generation model training method, a response generation method, a response generation model training device, a response generation device, an electronic device, and a computer-readable storage medium. Background Art

[0002] A response generation model is a large model that generates responses (keys) based on user-entered queries (queries). This model can be applied to scenarios such as conversational interaction, content generation, and information processing. Conversational interaction can include intelligent customer service and virtual assistants. Content generation can include news reporting, literary creation (such as essay generation, style adaptation, novel writing, and official document writing). Information processing can include document summarization and semantic search.

[0003] Before using the response generation model to generate responses in an application scenario, it needs to be trained. The training goal of the response generation model is to ensure that the response quality generated by the response generation model for the query meets the expected quality. Meeting the expected quality means that the style of the generated responses meets or closely approximates the target style preferred by humans.

[0004] However, existing response generation model training methods have poor training effects and low training efficiency. Summary of the Invention

[0005] The present application provides a reply generation model training method, a reply generation method, a reply generation model training device, a reply generation device, an electronic device and a computer-readable storage medium, which can solve the problems of poor training effect and low training efficiency of existing reply generation model training methods.

[0006] The present application provides a response generation model training method, including: obtaining a dialogue training set for a current round, wherein the dialogue training set for the current round includes at least one first dialogue, the first dialogue includes a first query and a corresponding first reply, the at least one first dialogue includes a first dialogue corresponding to the current round, and the first reply in the first dialogue corresponding to the current round is generated by a reinforcement learning large model obtained in the previous round of the current round based on the corresponding first query; using the dialogue training set for the current round to perform supervised fine-tuning on a pre-trained large model to obtain a supervised fine-tuned large model for the current round; using the supervised fine-tuned large model for the current round to perform reinforcement learning to obtain a reinforcement learning large model for the current round; wherein the reinforcement learning large model obtained in the final round is used as the response generation model.

[0007] The present application provides a reply generation method, comprising: obtaining a query to be processed; generating a target reply corresponding to the query to be processed using a reply generation model; wherein the reply generation model is trained based on a reply generation model training method.

[0008] The present application provides a response generation model training device, comprising: an acquisition module, a supervised fine-tuning module, and a reinforcement learning module. The acquisition module is used to acquire a dialogue training set for the current round, wherein the dialogue training set for the current round includes at least one first dialogue, the first dialogue includes a first query and a corresponding first reply, the at least one first dialogue includes the first dialogue corresponding to the current round, and the first reply in the first dialogue corresponding to the current round is generated by the reinforcement learning large model obtained in the previous round of the current round based on the corresponding first query; the supervised fine-tuning module is used to perform supervised fine-tuning on the pre-trained large model using the dialogue training set for the current round to obtain the supervised fine-tuned large model for the current round; the reinforcement learning module is used to perform reinforcement learning using the supervised fine-tuned large model for the current round to obtain the reinforcement learning large model for the current round; wherein the reinforcement learning large model obtained in the final round is used as the response generation model.

[0009] The present application provides a reply generation device, comprising: an acquisition module and a generation module. The acquisition module is used to acquire a query to be processed; the generation module is used to generate a target reply corresponding to the query to be processed using a reply generation model; wherein the reply generation model is trained based on a reply generation model training method.

[0010] The present application provides an electronic device, including a memory and a processor, wherein the processor is configured to execute program instructions stored in the memory to implement the above method.

[0011] The present application provides a computer-readable storage medium having program instructions stored thereon, which implement the above method when the program instructions are executed by a processor.

[0012] The above scheme divides the training of the reply generation model into multiple rounds. During the training process of the current round, at least the first dialogue corresponding to the current round is used to perform supervised fine-tuning on the pre-trained large model to obtain the supervised fine-tuned large model of the current round. The supervised fine-tuned large model of the current round is then used for reinforcement learning to obtain the reinforcement learning large model of the current round. The reinforcement learning large model obtained in the final round is used as the reply generation model. On the one hand, because the first reply in the first dialogue corresponding to the current round is generated by the reinforcement learning large model obtained in the previous round of the current round, the first dialogue can be automatically generated for supervised fine-tuning of the pre-trained large model without manual labeling. On the other hand, as the number of training rounds increases, the quality of the first reply generated by the reinforcement learning large model obtained in the previous round of the current round improves, which can improve the supervised fine-tuning effect of the pre-trained large model, thereby improving the performance of the supervised fine-tuned large model of the current round, and further improving the performance of the reinforcement learning large model of the current round obtained by reinforcement learning of the supervised fine-tuned large model of the current round. Therefore, the reply generation model training method provided in this application can improve training effect and has high training efficiency without the need for manual labeling.

[0013] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and, together with the specification, are used to illustrate the technical solutions of the present application.

[0015] Figure 1 This is the architecture diagram of the model training method for generating responses to this application;

[0016] Figure 2 This is a flow chart of an embodiment of the reply generation model training method provided by the present application;

[0017] Figure 3 This is a flowchart of another embodiment of the reply generation model training method provided by the present application;

[0018] Figure 4 This is a flowchart of another embodiment of the reply generation model training method provided by the present application;

[0019] Figure 5 This is a flow chart of another embodiment of the reply generation model training method provided by the present application;

[0020] Figure 6 This is a flow chart of another embodiment of the reply generation model training method provided by the present application;

[0021] Figure 7This is a flowchart of a specific example of the training method for the application response generation model;

[0022] Figure 8 This is a flowchart of an embodiment of a reply generation method provided by this application;

[0023] Figure 9 This is a structural diagram of an embodiment of a response generation model training device provided by the present application; Figure 10 This is a structural diagram of an embodiment of a reply generation device provided by the present application;

[0024] Figure 11 This is a structural diagram of an embodiment of an electronic device of the present application;

[0025] Figure 12 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0026] The following describes the embodiments of the present application in detail with reference to the accompanying drawings.

[0027] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0028] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the previous and next associated objects are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0029] A response generation model is a large model that can generate responses (keys) based on the query (query) entered by the user.

[0030] The training goal of the response generation model is to ensure that the response quality it generates for a given query meets the expected quality. This quality means that the style of the generated responses meets or closely approximates the target style preferred by humans (for example, semantic accuracy, engaging expression, expressiveness, and literary quality).

[0031] In related technologies, the training of the response generation model is divided into pre-training, supervised fine-tuning (SFT), and reinforcement learning (RLHF) stages.

[0032] Pre-training phase: Pre-train the initial large model to obtain a pre-trained large model. Pre-training is to enable the initial large model to learn the general rules of language and master vocabulary, grammar, semantics, and contextual understanding.

[0033] Supervised fine-tuning stage: Perform supervised fine-tuning on the pre-trained large model to obtain a supervised fine-tuned large model.

[0034] Reinforcement learning stage: Reinforcement learning is performed on the supervised fine-tuning large model to obtain the reinforcement learning large model. The reinforcement learning large model is the response generation model.

[0035] After long-term research, the inventors of this application found that the above-mentioned response generation model has at least the following technical problems:

[0036] 1. The supervised fine-tuning stage requires manual annotation and construction of training dialogues (including queries and corresponding responses) to fine-tune the pre-trained large model. Manually annotating training dialogues presents at least the following technical issues: First, annotation efficiency is low, leading to low training efficiency; second, annotation quality is limited by the annotator's ability, which limits the performance of the supervised fine-tuned large model obtained through fine-tuning the pre-trained large model.

[0037] 2. The starting point of reinforcement learning is supervised fine-tuning of large models, so reinforcement learning is limited by the performance of supervised fine-tuning large models.

[0038] In order to solve at least some of the above technical problems, the present application provides a response generation model training method.

[0039] The reply generation model training method provided in the present application divides the training of the reply generation model into multiple rounds. The starting point of the overall training of the reply generation model is the initial pre-trained large model, and the end point of the overall training is the reinforcement learning large model obtained in the final round. That is, the reinforcement learning large model obtained in the final round is used as the reply generation model. The starting point of each round of training of the reply generation model is the initial pre-trained large model, and the end point of each round of training is the reinforcement learning large model of that round. That is, each round is retrained starting from the initial pre-trained large model. Specifically, each round of training of the reply generation model includes a supervised fine-tuning stage and a reinforcement learning stage. The supervised fine-tuning stage is used to perform supervised fine-tuning on the pre-trained large model to obtain the supervised fine-tuning large model of that round, and the reinforcement learning stage is used to further perform reinforcement learning on the supervised fine-tuning large model of that round to obtain the reinforcement learning large model of that round.

[0040] Combine Figure 1 For example, Figure 1 This is the architecture diagram of the model training method for the application response generation. Figure 1As shown, the training of the response generation model is divided into n rounds. In the first round, supervised fine-tuned large model 1 is obtained based on the pre-trained large model, and reinforcement learning large model 1 is obtained based on supervised fine-tuned large model 1. Then, in the second round, supervised fine-tuned large model 2 is obtained based on the pre-trained large model, and reinforcement learning large model 2 is obtained based on supervised fine-tuned large model 2. This continues in this way until the nth round, supervised fine-tuned large model n is obtained based on the pre-trained large model, and reinforcement learning large model n is obtained based on supervised fine-tuned large model n. Reinforcement learning large model n is used as the response generation model.

[0041] An embodiment of the reply generation model training method provided in this application is introduced as follows.

[0042] Figure 2 This is a flow chart of an embodiment of the reply generation model training method provided by this application. Figure 2 As shown, in this embodiment, the reply generation model training method may include the following steps:

[0043] S110: Obtain the dialogue training set for the current round.

[0044] Among them, the dialogue training set of the current round includes at least one first dialogue, the first dialogue includes a first query and a corresponding first reply, at least one first dialogue includes the first dialogue corresponding to the current round, and the first reply in the first dialogue corresponding to the current round is generated by the reinforcement learning large model obtained in the previous round of the current round based on the corresponding first query.

[0045] The execution subject of this embodiment is a reply generation model training device, and the reply generation model training device can be any electronic device with reply generation model training capability.

[0046] The query mentioned in this application can be text, pictures, files, etc.

[0047] The current round can be any round other than the first round. The reinforcement learning model obtained in any round before the final round can be used to construct the first dialogue. Constructing the first dialogue means generating a corresponding first response based on the first query. The quality of the first response in the first dialogue constructed by the reinforcement learning model obtained in the later training round is higher than the quality of the first response in the first dialogue constructed by the reinforcement learning model obtained in the earlier training round. For example, the quality of the first response in the first dialogue constructed by the reinforcement learning model obtained in the second round is higher than the quality of the first response in the first dialogue constructed by the reinforcement learning model obtained in the first round.

[0048] In some embodiments, the at least one first dialogue includes only the first dialogue corresponding to the current round, wherein the first dialogue corresponding to the current round is the first dialogue constructed using the reinforcement learning large model obtained in the round before the current round.

[0049] In some embodiments, at least one first conversation further includes a first conversation corresponding to a reference historical round. The first response in the first conversation corresponding to the reference historical round is generated by the reinforcement learning model obtained in the previous round of the reference historical round based on the corresponding first query. The reference historical round is at least one round before the current round. The first conversation corresponding to the reference historical round is the first conversation constructed by the reinforcement learning model obtained in the previous round of the reference historical round. The first conversation corresponding to the reference historical round is obtained in the same manner as the first conversation corresponding to the current round.

[0050] In some embodiments, the reference historical rounds may be a preset number of historical rounds preceding the current round. The preset number may vary with the round number of the current round. Specifically, a round threshold may be set. If the round number of the current round is less than the round threshold, the preset number is the current round's round number minus 1, meaning that the reference historical rounds are all historical rounds preceding the current round. If the round number of the current round is greater than or equal to the round threshold, the preset number is the round threshold, meaning that the reference historical rounds are the threshold number of historical rounds preceding the current round. For example, if the round threshold is 3 and the current round is round 2, the preset number is 1, and the reference historical round is round 1. If the current round is greater than or equal to 3, the preset number is 2, and the reference historical rounds are the two most recent historical rounds preceding the current round. For another example, if the round threshold is 4 and the current round is round 2, the preset number is 1, and the reference historical rounds are round 1. If the current round is round 3, the preset number is 2, and the reference historical rounds are rounds 1 and 2. When the number of the current round is greater than or equal to 4, the preset number is 3, and the reference historical rounds are the latest 3 historical rounds before the current round. It can be understood that in the early stage of training (the number of the current round is less than the round threshold), the number of historical rounds before the current round is small, and all historical rounds before the current round are used as reference historical rounds, which can obtain a larger number of first conversations. In the late stage of training (the number of the current round reaches the round threshold), the number of historical rounds before the current round is large, and only the latest round threshold historical rounds before the current round are used as reference historical rounds, which can select a sufficient number of high-quality first conversations for training the current round, thereby improving the training effect.

[0051] In some embodiments, the conversation training set for the current round only includes at least one first conversation. In this case, it can be considered that the conversation training set for the current round is constructed using the reinforcement learning large model obtained before the current round.

[0052] In some embodiments, the conversation training set for the current round also includes at least one second conversation and / or at least one third conversation. The second conversation includes a second query and a corresponding first-quality response, and the third conversation includes a third query and a corresponding third-quality response. The quality of the first-quality response is higher than that of the third-quality response. In this case, the first conversation constructed using the large reinforcement learning model obtained before the current round can be considered to have expanded into the conversation training set for the previous round.

[0053] S 120: Use the current round of dialogue training set to supervise fine-tune the pre-trained large model to obtain the current round of supervised fine-tuning large model.

[0054] Supervised fine-tuning aims to learn the response paradigm of the target style.

[0055] In some embodiments, the replies in each conversation in the current round of conversation training set can be used as true replies, and a pre-trained large model can be used to generate predicted replies for queries in each conversation in the current round of conversation training set. Based on the difference between the predicted replies and the corresponding true replies, a supervised fine-tuning loss function is constructed; the parameters of the pre-trained large model are adjusted based on the supervised fine-tuning loss function; and the foregoing process is repeated until the supervised fine-tuning loss function converges.

[0056] S130: Perform reinforcement learning using the current round of supervised fine-tuning large model to obtain the current round of reinforcement learning large model.

[0057] Among them, the final round of reinforcement learning model serves as the response generation model.

[0058] It is understandable that the target style of the replies from supervised fine-tuning of large models is not obvious, and reinforcement learning aims to strengthen the target style of the replies.

[0059] In some embodiments, algorithms such as PPO (Proximal Policy Optimization), DPO (Direct Policy Optimization), and rejection sampling can be applied to implement reinforcement learning of supervised fine-tuning large models.

[0060] Through the implementation of this embodiment, the present application divides the training of the reply generation model into multiple rounds. During the training process of the current round, at least the first dialogue corresponding to the current round is used to perform supervised fine-tuning on the pre-trained large model to obtain the supervised fine-tuned large model of the current round. The supervised fine-tuned large model of the current round is then used for reinforcement learning to obtain the reinforcement learning large model of the current round. The reinforcement learning large model obtained in the final round is used as the reply generation model. On the one hand, because the first reply in the first dialogue corresponding to the current round is generated by the reinforcement learning large model obtained in the previous round of the current round, the first dialogue can be automatically generated for supervised fine-tuning of the pre-trained large model without manual labeling. On the other hand, as the number of training rounds increases, the quality of the first reply generated by the reinforcement learning large model obtained in the previous round of the current round improves, which can improve the supervised fine-tuning effect of the pre-trained large model, thereby improving the performance of the supervised fine-tuned large model of the current round, and further improving the performance of the reinforcement learning large model of the current round obtained by reinforcement learning of the supervised fine-tuned large model of the current round. Therefore, the reply generation model training method provided by the present application can improve training effect and has high training efficiency without the need for manual labeling.

[0061] Figure 3 This is a flow chart of another embodiment of the reply generation model training method provided by this application. In this embodiment, Figure 3 As shown, the step of generating the first reply in the first dialogue corresponding to the current round in S110 may include:

[0062] S210: Generate multiple candidate responses corresponding to the first query using the reinforcement learning model obtained in the previous round of the current round.

[0063] S220: Obtain a quality score for each candidate response using the reward model.

[0064] The reward (RM) model can be any model with quality scoring capability.

[0065] S230: Selecting a first reply corresponding to the first query from the candidate replies based on the quality scores of the candidate replies.

[0066] The quality score of a candidate response represents the quality of the candidate response. A higher quality score indicates higher quality. Quality indicates preference; higher quality indicates higher preference.

[0067] In some embodiments, the candidate response with the highest quality score may be selected from the candidate responses as the first response corresponding to the first query.

[0068] In some embodiments, a candidate response having a quality score that is neither the highest nor the lowest may be selected from the candidate responses as the first response corresponding to the first query.

[0069] Unlike the previous embodiment, this embodiment uses the reinforcement learning model from the previous round to generate multiple candidate responses for the first query in the first conversation of the current round. The first response is then selected from these candidates based on their quality scores. Compared to generating only one response as the first response, this improves the quality of the first response, thereby enhancing the training effect of the current round.

[0070] Furthermore, in some embodiments, the reward model may be trained before S220 so that the quality scoring capability of the reward model meets expectations, that is, it has obvious preference guidance characteristics.

[0071] Figure 4 This is a flow chart of another embodiment of the reply generation model training method provided by this application. Figure 4 As shown, the training steps for the reward model may include the following steps:

[0072] S310: Obtain a reward training set.

[0073] The reward training set includes multiple response pairs, where a response pair includes a response of a first quality and a response of a second quality, wherein the quality of the response of the first quality is higher than the quality of the response of the second quality.

[0074] The first-quality response and the second-quality response in a response pair are responses to the same query, which may or may not be the second query mentioned below.

[0075] The reward training set includes response pairs in the public response set and / or response pairs processed using a pre-trained large model.

[0076] In some embodiments, the reward training set includes the reply pairs in the public reply set. In this case, S310 may include: adding at least one reply pair in the public reply set to the reward training set.

[0077] In some embodiments, the reward training set includes reply pairs processed using a pre-trained large model. In this case, S310 may include S321-S323. S321: Obtain a second conversation, wherein the second conversation includes a second query and a corresponding first-quality reply. S322: Generate a second-quality reply corresponding to the second query using the pre-trained large model. S323: Combine the first-quality reply and the corresponding second-quality reply into a reply pair. Thus, the second-quality reply in the reply pair processed using the pre-trained large model is generated by the pre-trained large model based on the second query.

[0078] In some embodiments, the first query in the second conversation is collected through a public channel, and the first quality reply is obtained by manual annotation.

[0079] In some embodiments, the first quality responses in the second conversation are collected through public channels, and the second query is generated by reverse engineering the first quality responses.

[0080] S320: Train the reward model using multiple reply pairs.

[0081] It is understandable that in the related art, the reward training set obtained by manual annotation is needed to train the reward model. For example, for the same query, it is necessary to manually annotate to obtain replies of different qualities corresponding to the query to form a reply pair. Or, for the same query, it is necessary to manually sort the quality of multiple replies corresponding to the query according to preference to form a reply pair. Or, for the same query, it is necessary to manually sort the multiple replies corresponding to the query according to preference, and then edit the reply with the highest quality to obtain the ideal reply, and then construct the reply pair in the order of the ideal reply quality being greater than the best reply quality, and the best reply quality being greater than the other reply qualities. This method of manually annotating to obtain the reward training set not only has low annotation efficiency, which affects the training efficiency, but also because it is necessary to sort and edit the replies according to preference, the annotation quality is limited by the literary accomplishment of the annotator, which affects the training effect.

[0082] Unlike the previous embodiment, this embodiment ensures that the quality of the first-quality responses is sufficiently high because the first-quality responses are not generated based on the second query but are instead collected from public channels. Furthermore, since the first-quality responses can be collected from public channels and the second-quality responses can be automatically generated based on the second query by a pre-trained large model, manual annotation can be reduced or eliminated, improving annotation efficiency. This improves training efficiency and reduces the risk that annotation quality will be limited by the annotator's literary literacy.

[0083] Furthermore, in some embodiments, in the early stages of reward model training, the reward model can be trained using manually annotated responses to ensure that the reward model has a relatively stable quality scoring capability. In the later stages of training, the reward model is trained through S310-S320 to further optimize the quality scoring capability.

[0084] Alternatively, in the early stage of training, the reward model is trained through S310-S320 to ensure that the reward model has a relatively stable quality scoring capability. In the later stage of training, the reward model is trained using manually labeled response pairs to further optimize the quality scoring capability.

[0085] Figure 5 This is a flow chart of another embodiment of the reply generation model training method provided by this application. In this embodiment, Figure 5 As shown, the step of obtaining the second conversation includes the following steps:

[0086] S410: Collect a number of first quality responses.

[0087] In some embodiments, high-quality text resources can be collected from public channels; the high-quality text resources can be regularized using a text regularization model to obtain a number of first-quality responses.

[0088] S420: Reversely generate a second query corresponding to the first quality response.

[0089] In some embodiments, a prompt strategy may be used to reverse-write each first-quality response to obtain a second query corresponding to each first-quality response.

[0090] In some embodiments, each first quality reply may be reversely generated based on a query generation model to obtain a second query corresponding to each first quality reply.

[0091] In some embodiments, after S420 , the method may further include: checking or reconstructing the second query distribution to ensure the balance of the second query.

[0092] S430: Combine the second query and the first quality reply into a second conversation.

[0093] Unlike the previous embodiment, this embodiment first collects the first-quality replies, then reverses them to obtain the second query, and combines the first-quality replies and the second query into a second conversation. Since the first-quality replies are collected first, the quality of the first-quality replies can be guaranteed to be high.

[0094] Furthermore, in some embodiments, the step of obtaining the third conversation includes: using a conversation from a public conversation set as the third conversation. The public conversation set can be an industry-wide, non-manually annotated conversation set, such as Alpaca, Dolly, or ShareGPT. Alternatively, a plurality of third queries can be collected, and a pre-trained large model can be used to generate a third-quality response corresponding to each third query.

[0095] Figure 6 This is a flow chart of another embodiment of the response generation model training method provided by this application. In this embodiment, the conversation training set of the current round includes the first conversations corresponding to all historical rounds before the current round. Figure 6 As shown, S110 may include the following steps:

[0096] S510: Obtain at least one first query.

[0097] The first query may be a query in a public query set, or a query generated by a query generation model, etc.

[0098] S520: Process at least one first query using the reinforcement learning large model obtained in the previous round of the current round, and obtain at least one first reply accordingly.

[0099] In some embodiments, for each first query, a reinforcement learning model obtained in the previous round of the current round can be used to generate a reply corresponding to the first query, and the reply can be used as the first reply corresponding to the first query.

[0100] In some embodiments, for each first query, the reinforcement learning model obtained in the previous round of the current round can be used to generate multiple candidate responses corresponding to the first query, and the candidate response with the highest quality is selected from the multiple candidate responses as the first response corresponding to the first query.

[0101] S530: Combine at least one first query and at least one corresponding first reply into at least one first dialogue corresponding to the current round.

[0102] S540: Add at least one first dialogue corresponding to the current round to the dialogue training set of the previous round of the current round to obtain the dialogue training set of the current round.

[0103] The method of obtaining the dialogue training set of the previous round of the current round is the same as that of obtaining the dialogue training set of the current round, which will not be repeated here. In addition, the dialogue training set of the first round includes at least one second dialogue and / or at least one third dialogue.

[0104] Different from the above-mentioned embodiment, this embodiment uses the reinforcement learning large model obtained in the previous round of the current round to automatically construct the first dialogue corresponding to the current round, and adds the first dialogue corresponding to the current round to the dialogue training set of the previous round of the current round to expand the dialogue training set of the previous round of the current round to obtain the dialogue training set of the current round.

[0105] It's understandable that when the first-round conversation training set includes the second / third conversations, the second conversation, as the seed conversation, is primarily responsible for ensuring the consistency of the response style generated by the first-round supervised fine-tuning large model, while the third conversation is primarily responsible for ensuring that the first-round supervised fine-tuning large model closely follows the instructions. Furthermore, the non-first-round conversation training set is derived from the second / third conversations, using the first conversations constructed using the reinforcement learning large model from non-first-round historical rounds. This can further improve the performance of the non-first-round supervised fine-tuning large model. Improving the performance of the supervised fine-tuning large model is equivalent to improving the performance of the reinforcement learning starting point, thereby improving the performance of the final reinforcement learning large model.

[0106] In order to better understand the response generation model training method provided in this application, the following is an explanation in the form of a specific example:

[0107] Figure 7This is a flowchart of a specific example of the training method for the response generation model of this application. Figure 7 As shown, the reply generation model training method includes:

[0108] 1. Get the first round of training dialogue set.

[0109] 1) Obtain several second conversations.

[0110] High-quality text resources are collected; the text regularization model is used to regularize the high-quality text resources to obtain several first-quality responses; each first-quality response is reversed using a prompt strategy to obtain the second query corresponding to each first-quality response. The second query and the corresponding first-quality response are combined into a second dialogue.

[0111] 2) Obtain several third conversations.

[0112] Include the dialogue from the public dialogue set as the third dialogue.

[0113] 3) Each second dialogue and each third dialogue constitutes the first round of dialogue training set.

[0114] 2. Use the first round of dialogue training set for the first round of training.

[0115] The pre-trained large model is supervised and fine-tuned using the first round of dialogue training set to obtain supervised fine-tuned large model 1; and reinforcement learning is performed using supervised fine-tuned large model 1 to obtain reinforcement learning large model 1.

[0116] 3. Get the reward training set.

[0117] For each second conversation, generate a second-quality response corresponding to the second query based on the pre-trained large model. Combine the first-quality response and the second-quality response corresponding to the second query into a response pair. Furthermore, obtain at least one response pair from the public response set. Each response pair constitutes a reward training set.

[0118] 4. Use the reward training set to train the reward model.

[0119] Utilize the reward model for each reply pair to train.

[0120] 5. Obtain the dialogue training set for the second round.

[0121] 1) Use the reinforcement learning model 1 to build the first dialogue.

[0122] Collect several first queries; use the reinforcement learning model 1 to generate multiple candidate replies corresponding to each first query; for each first query, use the reward model to obtain the quality score of each candidate reply corresponding to the first query; select the candidate reply with the highest quality score as the corresponding first reply for each first query; and combine the first query and the corresponding first reply into a first conversation.

[0123] 2) The first dialogue constructed by the reinforcement learning large model 1 is added to the first round of dialogue training set to obtain the second round of dialogue training set.

[0124] 6. Use the second round of dialogue training set for the second round of training.

[0125] The pre-trained large model is fine-tuned using the second round of dialogue training set to obtain supervised fine-tuned large model 2; and reinforcement learning is performed using supervised fine-tuned large model 2 to obtain reinforcement learning large model 2.

[0126] 7. Obtain the dialogue training set for round 3.

[0127] 8. Use the dialogue training set from the third round to conduct the third round of training.

[0128] And so on, until the nth round of training is completed, the reinforcement learning model n obtained from the nth round of training is used as the response generation model.

[0129] In the above specific example, in the first round, the high-quality second dialogue and the existing third dialogue form the first-round dialogue training set. This first-round dialogue training set is used to perform supervised fine-tuning on the pre-trained large model to obtain supervised fine-tuned large model 1. Reinforcement learning is then performed on supervised fine-tuned large model 1 to obtain reinforcement learning large model 1. In the second round, reinforcement learning large model 1 is used to construct the first dialogue, which is then added to the first-round dialogue training set to obtain the second-round dialogue training set. This second-round dialogue training set is used to perform supervised fine-tuning on the pre-trained large model to obtain supervised fine-tuned large model 2. Reinforcement learning is then performed on supervised fine-tuned large model 2 to obtain reinforcement learning large model 2. This process continues in this manner, and reinforcement learning large model n obtained in round n is used as the response generation model.

[0130] Among them, for the supervised fine-tuning stage, in the first round, a combination of high-quality second dialogues (seed dialogues) and existing stock third dialogues is used to train the pre-trained large model to obtain the supervised fine-tuning large model 1. Therefore, in the supervised fine-tuning stage, the focus is not on the response generation effect of the supervised fine-tuning large model 1, but on ensuring that the supervised fine-tuning large model 1 has the ability to generate responses that conform to the target style and can closely follow the instructions. Specifically, the second dialogue, as a seed dialogue, is mainly responsible for ensuring the consistency of the response style generated by the supervised fine-tuning large model 1, and the third dialogue is mainly responsible for ensuring that the supervised fine-tuning large model 1 can closely follow the instructions. In the non-first round, a combination of high-quality second dialogues, existing stock third dialogues, and first dialogues is used to train the pre-trained large model. Therefore, the dialogue training set can be further expanded on the basis of the first round, and the supervised fine-tuning effect can be improved.

[0131] Furthermore, to construct the reward training set for response pairs, we firstly collect first-quality responses from public channels, reverse-write the first-quality responses to obtain the second query, and then use the pre-trained large model to generate the second-quality response corresponding to the second query. The first-quality response and the second-quality response constitute a response pair. Secondly, we obtain at least one response pair from the public response set. This eliminates the need for manual annotation, improves both annotation and training efficiency, and increases training effectiveness by making annotation quality independent of the annotator's literary literacy.

[0132] Figure 8 This is a flow chart of an embodiment of the reply generation method provided by this application. Figure 8 As shown, the reply generation method may include the following steps:

[0133] S610: Obtain pending queries.

[0134] S620: Generate a target reply corresponding to the query to be processed using the reply generation model.

[0135] The reply generation model is trained based on the aforementioned reply generation model training method.

[0136] For other detailed descriptions of the reply generation method, please refer to the previous embodiments, which will not be repeated here.

[0137] Figure 9 This is a structural diagram of an embodiment of the response generation model training device provided by this application. Figure 9 As shown, the response generation model training device 70 may include an acquisition module 71, a supervised fine-tuning module 72 and a reinforcement learning module 73.

[0138] An acquisition module 71 is used to obtain a dialogue training set for the current round, wherein the dialogue training set for the current round includes at least one first dialogue, the first dialogue includes a first query and a corresponding first reply, the at least one first dialogue includes the first dialogue corresponding to the current round, and the first reply in the first dialogue corresponding to the current round is generated by a reinforcement learning large model obtained in the previous round of the current round based on the corresponding first query.

[0139] The supervised fine-tuning module 72 is used to perform supervised fine-tuning on the pre-trained large model using the dialogue training set of the current round to obtain the supervised fine-tuned large model of the current round.

[0140] The reinforcement learning module 73 is used to perform reinforcement learning using the supervised fine-tuning large model of the current round to obtain the reinforcement learning large model of the current round.

[0141] Among them, the reinforcement learning large model obtained in the final round is used as the response generation model.

[0142] For other detailed descriptions of the response generation model training device 70, please refer to the previous embodiments, which will not be repeated here.

[0143] Figure 10 This is a schematic diagram of the structure of an embodiment of the reply generation device provided by this application. Figure 10 As shown, the reply generating device 80 may include an acquiring module 81 and a generating module 82 .

[0144] The acquisition module 81 is used to acquire queries to be processed.

[0145] The generation module 82 is configured to generate a target reply corresponding to the query to be processed using a reply generation model.

[0146] The reply generation model is trained based on the aforementioned reply generation model training method.

[0147] For other detailed descriptions of the reply generating device 80 , please refer to the previous embodiments, which will not be repeated here.

[0148] Figure 11 This is a schematic diagram of the structure of an embodiment of the electronic device of the present application. Figure 12 As shown, electronic device 90 includes memory 91 and processor 92. Processor 92 is configured to execute program instructions stored in memory 91 to implement the steps of any of the above method embodiments. In a specific implementation scenario, electronic device 90 may include, but is not limited to, a microcomputer and a server. Furthermore, electronic device 90 may also include a carrier device such as a laptop computer and a tablet computer, which are not limited herein.

[0149] Specifically, the processor 92 is used to control itself and the memory 91 to implement the steps in any of the above method embodiments. The processor 92 can also be called a CPU (Central Processing Unit). The processor 92 may be an integrated circuit chip with signal processing capabilities. The processor 92 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 92 can be implemented by an integrated circuit chip.

[0150] See also Figure 12 , Figure 12 The computer-readable storage medium 100 stores program instructions 101, which, when executed by a processor, implement the steps of any of the above method embodiments.

[0151] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0152] The above description of the various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced with each other and will not be repeated herein for the sake of brevity.

[0153] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation methods described above are only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. For example, units or components can be combined or integrated into another system, or some features can be ignored or not executed. In another image position, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0154] In addition, the functional units in the various embodiments of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

Claims

1. A response generation model training method, characterized in that: include: Obtaining a conversation training set for the current round, wherein the conversation training set for the current round includes at least one first conversation, the first conversation including a first query and a corresponding first reply, the at least one first conversation including a first conversation corresponding to the current round, and the first reply in the first conversation corresponding to the current round is generated by a reinforcement learning large model obtained in a round previous to the current round based on the corresponding first query; Using the current round of dialogue training set to supervise fine-tune the pre-trained large model, to obtain the current round of supervised fine-tuning large model; Perform reinforcement learning using the current round of supervised fine-tuning large model to obtain the current round of reinforcement learning large model; Among them, the reinforcement learning large model obtained in the final round is used as the response generation model.

2. The method according to claim 1, characterized in that The step of generating a first reply in the first dialogue corresponding to the current round includes: Generate multiple candidate responses corresponding to the first query using the reinforcement learning model obtained in the previous round of the current round; Obtaining a quality score for each candidate response using a reward model; Based on the quality scores of the candidate replies, a first reply corresponding to the first query is selected from the candidate replies.

3. The method according to claim 2, characterized in that Before obtaining the quality score of each candidate response using the reward model, the following steps are included: Acquire a reward training set, wherein the reward training set includes a plurality of response pairs, wherein the response pair includes a first quality response and a second quality response, wherein the quality of the first quality response is higher than the quality of the second quality response; The reward model is trained using the plurality of reply pairs.

4. The method according to claim 3, characterized in that The obtaining of the reward training set includes: Obtaining a second conversation, wherein the second conversation includes a second query and a corresponding first-quality reply; generating a second quality response corresponding to the second query using the pre-trained large model; combining the first quality response and the corresponding second quality response into the response pair; And / or, obtaining the reward training set includes: At least one response pair in the public response set is added to the reward training set.

5. The method according to claim 1, wherein The current round of dialogue training set also includes at least one second dialogue and / or at least one third dialogue, wherein the second dialogue includes a second query and a corresponding first-quality response, and the third dialogue includes a third query and a corresponding third-quality response, and the quality of the first-quality response is higher than the quality of the third-quality response.

6. The method according to claim 4 or 5, characterized in that The step of obtaining the second conversation includes: Collect a number of first-quality responses; Reversely generate the second query corresponding to the first quality response; combining the second query and the first quality response into the second conversation; And / or, the step of obtaining the third conversation includes: The conversation in the public conversation set is used as the third conversation.

7. The method according to claim 1, characterized in that The at least one first conversation also includes the first conversation corresponding to a reference historical round, and the first reply in the first conversation corresponding to the reference historical round is generated by the reinforcement learning large model obtained in the previous round of the reference historical round based on the corresponding first query, and the reference historical round is at least one round before the current round.

8. The method according to claim 7, characterized in that The step of obtaining the current round of dialogue training set includes: Obtain at least one first query; Processing the at least one first query using the reinforcement learning large model obtained in the previous round of the current round, and obtaining at least one first reply accordingly; Combining the at least one first query and the corresponding at least one first reply into at least one first dialogue corresponding to the current round; Add at least one first dialogue corresponding to the current round to the dialogue training set of the previous round of the current round to obtain the dialogue training set of the current round.

9. A reply generation method, characterized in that: include: Get pending queries; generating a target response corresponding to the query to be processed using a response generation model; Wherein, the response generation model is trained based on any one of the methods in claims 1-8.

10. A response generation model training device, characterized in that: include: an acquisition module, configured to acquire a dialogue training set for a current round, wherein the dialogue training set for the current round includes at least one first dialogue, the first dialogue including a first query and a corresponding first reply, the at least one first dialogue including a first dialogue corresponding to the current round, and the first reply in the first dialogue corresponding to the current round being generated by a reinforcement learning large model obtained in a round previous to the current round based on the corresponding first query; A supervised fine-tuning module is used to perform supervised fine-tuning on the pre-trained large model using the dialogue training set of the current round to obtain a supervised fine-tuned large model of the current round; A reinforcement learning module, configured to perform reinforcement learning using the supervised fine-tuning large model of the current round to obtain the reinforcement learning large model of the current round; Among them, the reinforcement learning large model obtained in the final round is used as the response generation model.

11. A reply generating device, characterized in that: include: The acquisition module is used to obtain pending queries; A generation module, configured to generate a target response corresponding to the query to be processed using a response generation model; Wherein, the response generation model is trained based on any one of the methods in claims 1-8.

12. An electronic device, characterized in that: The system comprises a memory and a processor, wherein the processor is configured to execute program instructions stored in the memory to implement the method according to any one of claims 1 to 8.

13. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images