Training method and device of reply enhancement-based dialogue query generation model

Abstract

The application discloses a training method and device of a dialogue query generation model based on reply enhancement, the method comprising: obtaining a training set; constructing a first language model and a second language model, training the first language model and the second language model by using the training set to obtain a query generation model and a reply enhancement query generation model; generating a query set by using the query generation model and generating a pseudo query by using the reply enhancement query generation model; obtaining corresponding similarity scores according to the query set and the pseudo query, constructing pseudo instances according to the similarity scores, and performing second-stage training on the query generation model and the reply enhancement query generation model according to the pseudo instances; then, scoring candidate queries obtained by sampling the query generation model by using the reply enhancement query generation model to obtain reward scores, and performing third-stage reinforcement learning training on the query generation model according to the reward scores; thus, the trained query generation model can obtain more accurate dialogue queries.

Description

Technical Field

[0001] This application relates to the field of human-computer dialogue technology, and in particular to a training method for a dialogue query generation model based on response enhancement, a computer-readable storage medium, a computer device, and a training apparatus for a dialogue query generation model based on response enhancement. Background Technology

[0002] In related technologies, dialogue query generation is a key module of search engine knowledge-assisted dialogue systems. The dialogue queries predicted by the model are used to obtain relevant external knowledge documents from knowledge sources such as search engines to assist in the generation of subsequent dialogue responses. The training of existing dialogue query generation models usually uses manually labeled dialogue query data to fine-tune the pre-trained language model to obtain the final dialogue query generation model. However, manually labeled dialogue query data is still relatively scarce, while a large amount of publicly available unlabeled dialogue data has not been fully utilized. Furthermore, the domain of manually labeled datasets is limited, and the model's cross-domain generalization ability is weak, resulting in the generally low quality of the trained dialogue query generation model. Summary of the Invention

[0003] This application aims to at least partially address one of the technical problems in the aforementioned technologies. To this end, one objective of this application is to propose a training method for a response-enhanced dialogue query generation model. Through a three-stage model training framework, it fully utilizes knowledge of dialogue responses, providing multi-granular training guidance to the query generation model, thereby making it suitable for dialogue query generation in cross-domain and low-resource scenarios, resulting in more accurate dialogue queries.

[0004] The second objective of this application is to provide a computer-readable storage medium.

[0005] The third objective of this application is to propose a computer device.

[0006] The fourth objective of this application is to propose a training device for a response-enhanced dialogue query generation model.

[0007] To achieve the above objectives, a first aspect of this application proposes a training method for a response-enhanced dialogue query generation model, comprising the following steps: obtaining a training sample set, wherein the training sample set includes dialogue data with labeled dialogue queries and dialogue data with unlabeled dialogue queries, the dialogue data including dialogue history and dialogue responses; constructing a first language model and a second language model, training the first language model using the dialogue history from the labeled dialogue query data to obtain a query generation model, and training the second language model using the dialogue history and dialogue responses from the labeled dialogue query data to obtain a response-enhanced query generation model; inputting the dialogue history from the unlabeled dialogue query data into the query generation model to obtain a response-enhanced query generation model. A query set is generated, and the dialogue history and dialogue responses from the unlabeled dialogue query data are input into the response-enhanced query generation model to generate pseudo queries. A similarity score is obtained for each pseudo query based on the query set and the pseudo queries. Pseudo instances are constructed by selecting each pseudo query with a similarity score greater than a preset threshold and its corresponding dialogue history and dialogue responses, so that the query generation model and the response-enhanced query generation model can be trained in the second stage based on the pseudo instances. The response-enhanced query generation model trained in the second stage is used to score the candidate queries sampled by the query generation model trained in the second stage to obtain a reward score, so that the query generation model can be trained in the third stage based on the reward score.

[0008] According to the training method of the response-enhanced dialogue query generation model according to the embodiments of this application, firstly, a training sample set is obtained, wherein the training sample set includes dialogue data with labeled dialogue queries and dialogue data with unlabeled dialogue queries, and the dialogue data includes dialogue history and dialogue responses; then, a first language model and a second language model are constructed, the first language model is trained using the dialogue history in the labeled dialogue query data to obtain a query generation model, and the second language model is trained using the dialogue history and dialogue responses in the labeled dialogue query data to obtain a response-enhanced query generation model; next, the dialogue history in the unlabeled dialogue query data is input into the query generation model to generate a query set, and the dialogue history and dialogue responses in the unlabeled dialogue query data are input into the response-enhanced query generation model. The system first generates pseudo-queries. Then, based on the query set and the pseudo-queries, it obtains the similarity score for each pseudo-query. Each pseudo-query with a similarity score greater than a preset threshold, along with its corresponding dialogue history and responses, is selected to construct pseudo-instances. These pseudo-instances are then used to train the query generation model and the response-enhanced query generation model in the second stage. Finally, the response-enhanced query generation model trained in the second stage is used to score the candidate queries sampled by the query generation model trained in the second stage to obtain reward scores. These reward scores are then used to train the query generation model in the third stage of reinforcement learning. Thus, through this three-stage model training framework, the knowledge of dialogue responses is fully utilized, providing multi-granular training guidance for the query generation model. This makes it suitable for cross-domain and low-resource scenarios of dialogue query generation, resulting in more accurate dialogue queries.

[0009] In addition, the training method for the response-enhanced dialogue query generation model proposed in the above embodiments of this application may also have the following additional technical features:

[0010] Optionally, the similarity score for each pseudo-query is obtained according to the following formula:

[0011]

[0012] Where Fsim represents the text similarity function. This indicates a pseudo-query. This represents the x-th query in the query set. This represents a query set consisting of N queries.

[0013] Optionally, the response-enhanced query generation model trained in the second stage is used to score the candidate queries sampled by the query generation model trained in the second stage to obtain a reward score. This includes: inputting the dialogue history from the dialogue data of the unlabeled dialogue query into the query generation model trained in the second stage to generate a candidate query set; obtaining the length-normalized log probability of each candidate query based on the candidate query set, normalizing the length-normalized log probability of all candidate queries to obtain a predicted probability distribution, and using it as a random sampling strategy for reinforcement learning; inputting the dialogue history and dialogue responses from the dialogue data of the unlabeled dialogue query into the response-enhanced query generation model trained in the second stage to calculate the length-normalized log probability corresponding to each candidate query, and using the length-normalized log probability as a reward score.

[0014] Optionally, the response-enhanced query generation model trained in the second stage is used to score the candidate queries sampled by the query generation model trained in the second stage to obtain a reward score. This includes: inputting the dialogue history from the dialogue data of the unlabeled dialogue query into the query generation model trained in the second stage to generate a candidate query set; obtaining the length-normalized log probability of each candidate query based on the candidate query set, normalizing the length-normalized log probability of all candidate queries to obtain a predicted probability distribution, and using it as a random sampling strategy for reinforcement learning; inputting the dialogue history and dialogue responses from the dialogue data of the unlabeled dialogue query into the response-enhanced query generation model trained in the second stage to calculate the length-normalized log probability corresponding to each candidate query; and sorting all candidate queries in descending order based on the length-normalized log probability to obtain a reward score.

[0015] To achieve the above objectives, a second aspect of this application provides a computer-readable storage medium storing a training program for a response-enhanced dialogue query generation model, which, when executed by a processor, implements the training method for the response-enhanced dialogue query generation model as described above.

[0016] According to embodiments of this application, a computer-readable storage medium stores a training program for a response-enhanced dialogue query generation model. This enables a processor to implement the aforementioned training method for the response-enhanced dialogue query generation model when executing the training program. Thus, through a three-stage model training framework, knowledge of dialogue responses is fully utilized, providing multi-granular training guidance for the query generation model. This makes the model suitable for dialogue query generation in cross-domain and low-resource scenarios, resulting in more accurate dialogue queries.

[0017] To achieve the above objectives, a third aspect of this application provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the training method for the response-enhanced dialogue query generation model as described above.

[0018] According to the computer device of the present application embodiment, the training of the response-enhanced dialogue query generation model is stored in the memory, so that when the processor executes the training program of the response-enhanced dialogue query generation model, the training method of the response-enhanced dialogue query generation model as described above is implemented. Thus, through the three-stage model training framework, the knowledge of dialogue responses is fully utilized, and multi-granular training guidance is given to the query generation model, thereby making it suitable for dialogue query generation in cross-domain and low-resource scenarios, and obtaining more accurate dialogue queries.

[0019] To achieve the above objectives, a fourth aspect of this application proposes a training apparatus for a response-enhanced dialogue query generation model, comprising: a first acquisition module, configured to acquire a training sample set, wherein the training sample set includes dialogue data of labeled dialogue queries and dialogue data of unlabeled dialogue queries, the dialogue data including dialogue history and dialogue responses; a first training module, configured to construct a first language model and a second language model, train the first language model using the dialogue history in the labeled dialogue query data to obtain a query generation model, and train the second language model using the dialogue history and dialogue responses in the labeled dialogue query data to obtain a response-enhanced query generation model; and a second acquisition module, configured to input the dialogue history in the dialogue data of the unlabeled dialogue queries into the query generation model. The model generates a query set and inputs the dialogue history and dialogue responses from the unlabeled dialogue query data into the response-enhanced query generation model to generate pseudo queries; the second training module is used to obtain a similarity score for each pseudo query based on the query set and the pseudo queries, select each pseudo query with a similarity score greater than a preset threshold and construct pseudo instances with the corresponding dialogue history and dialogue responses, so as to perform a second stage of training on the query generation model and the response-enhanced query generation model based on the pseudo instances; the third training module is used to score the candidate queries sampled by the query generation model trained in the second stage using the response-enhanced query generation model trained in the second stage to obtain a reward score, so as to perform a third stage of reinforcement learning training on the query generation model based on the reward score.

[0020] The training apparatus for the dialogue query generation model based on response enhancement according to the embodiments of this application fully utilizes the knowledge of dialogue responses through a three-stage model training framework, providing multi-granular training guidance for the query generation model, thereby making it suitable for dialogue query generation in cross-domain and low-resource scenarios and obtaining more accurate dialogue queries.

[0021] In addition, the training apparatus for the response-enhanced dialogue query generation model proposed in the above embodiments of this application may also have the following additional technical features:

[0022] Optionally, the similarity score for each pseudo-query is obtained according to the following formula:

[0023]

[0024] Where Fsim represents the text similarity function. This indicates a pseudo-query. This represents the x-th query in the query set. This represents a query set consisting of N queries.

[0025] Optionally, the third training module is further configured to input the dialogue history from the dialogue data of the unlabeled dialogue query into the query generation model trained in the second stage to generate a candidate query set; obtain the length-normalized log probability of each candidate query based on the candidate query set, normalize the length-normalized log probability of all candidate queries to obtain a predicted probability distribution, and use it as a random sampling strategy for reinforcement learning; input the dialogue history and dialogue responses from the dialogue data of the unlabeled dialogue query into the response-enhancing query generation model trained in the second stage to calculate the length-normalized log probability corresponding to each candidate query, and use the length-normalized log probability as a reward score.

[0026] Optionally, the third training module is further configured to input the dialogue history from the dialogue data of the unlabeled dialogue query into the query generation model trained in the second stage to generate a candidate query set; obtain the length-normalized log probability of each candidate query based on the candidate query set, normalize the length-normalized log probability of all candidate queries to obtain a predicted probability distribution, and use it as a random sampling strategy for reinforcement learning; input the dialogue history and dialogue responses from the dialogue data of the unlabeled dialogue query into the response-enhancing query generation model trained in the second stage to calculate the length-normalized log probability corresponding to each candidate query; and sort all candidate queries in descending order based on the length-normalized log probability to obtain a reward score. Attached Figure Description

[0027] Figure 1This is a flowchart illustrating the training method of a response-enhanced dialogue query generation model according to an embodiment of this application;

[0028] Figure 2 This is a schematic diagram of the model training framework according to an embodiment of this application;

[0029] Figure 3 This is a block diagram of a training apparatus for a response-enhanced dialogue query generation model according to an embodiment of this application. Detailed Implementation

[0030] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0031] To better understand the above technical solutions, exemplary embodiments of this application will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0032] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0033] Figure 1 This is a flowchart illustrating the training method of the response-enhanced dialogue query generation model according to an embodiment of this application, as shown below. Figure 1 As shown, the training method for this response-enhanced dialogue query generation model includes the following steps:

[0034] S101, Obtain the training sample set, which includes labeled dialogue data and unlabeled dialogue data. The dialogue data includes dialogue history and dialogue responses.

[0035] Among them, the dialogue history is u ＜i =u1, ...,u i-1 The dialogue reply is u i .

[0036] S102, construct a first language model and a second language model. Train the first language model using the dialogue history in the labeled dialogue query data to obtain a query generation model, and train the second language model using the dialogue history and dialogue responses in the labeled dialogue query data to obtain a response-enhanced query generation model.

[0037] In other words, here we use supervised learning to train the query generation model QP and the response-enhanced query generation model RA using dialogue query labeled data.

[0038] As an example, a specific generative pre-trained language model, such as T5, is selected as the first language model and the second language model, i.e., as the base model.

[0039] It should be noted that, given the dialogue history and dialogue response, both the first language model and the second language model need to predict the target query q. The difference between the two is that one only uses the dialogue history as the model input, while the other also needs to introduce the dialogue response as an additional model input; however, the dialogue response is not available in actual reasoning application scenarios.

[0040] In other words, given the dialogue history u with query annotations ＜i =u1, ...,u i-1 The reply to the conversation is u i The first language model is trained using dialogue history to obtain the query generation model QP, and the second language model is trained using dialogue history and dialogue responses to obtain the response-enhanced query generation model RA.

[0041] For each training instance, the standard cross-entropy loss function is used as the model training objective:

[0042] L qp =-logp(q|u ＜i ;θ qp )

[0043] L ra = -log p(q|u ≤i ;θ ra )

[0044] Where θ qp and θ ra These represent the model parameters for QP and RA, respectively.

[0045] S103, input the dialogue history from the dialogue data of the unlabeled dialogue query into the query generation model to generate a query set, and input the dialogue history and dialogue responses from the dialogue data of the unlabeled dialogue query into the response-enhanced query generation model to generate pseudo queries.

[0046] In other words, for unlabeled dialogue corpus, given the dialogue history u ＜i And reply to the conversation u i Use the Response Augmentation Query Generation (RA) model to generate pseudo queries. At the same time, given the dialogue history u ＜i As input to the query generation model QP, generate N queries using the query generation model QP.

[0047] S104. Based on the query set and pseudo-queries, obtain the similarity score for each pseudo-query. Select each pseudo-query with a similarity score greater than a preset threshold and construct pseudo-instances with the corresponding dialogue history and dialogue responses. Then, conduct the second-stage training of the query generation model and the response-enhanced query generation model based on the pseudo-instances.

[0048] In other words, this method uses unlabeled dialogue data to generate pseudo-queries and constructs high-quality pseudo-instances for semi-supervised learning. In the second stage of training, the dialogue history from the unlabeled dialogue data in the pseudo-queries is input into the query generation model for training, and the dialogue history and dialogue responses from the unlabeled dialogue data in the pseudo-queries are input into the response-enhanced query generation model for training.

[0049] It should be noted that during query generation, it was observed that dialogue responses and dialogue queries are often closely related. The response-enhanced query generation model (RA) outperformed the query generation model (QP). Therefore, RA was used to generate more accurate pseudo-queries for this stage of semi-supervised learning. During query selection, considering that RA may over-rely on response information and ignore dialogue history, resulting in some pseudo-queries containing irrelevant concepts, which is not conducive to QP training, data filtering is required. At the same time, it was noted that the confidence level of the model's predicted probability may not be reliable. Therefore, a similarity-based query selection method is needed to select higher-quality pseudo-queries.

[0050] As a specific example, such as Figure 2 As shown, the pseudo-queries generated by RA are evaluated using the following similarity scores. Quality:

[0051]

[0052] Among them, F sim This represents a text similarity function that can return a specific quantitative metric (such as Unigram F1, BLEU, ROUGE) score or a sentence similarity model score (such as Sentence-BERT). It's important to note that when RA is excessively influenced by response information... This will result in a lower similarity score if irrelevant concepts are included in the response; This indicates a pseudo-query. This represents the x-th query in the query set. This represents a query set consisting of N queries.

[0053] Next, select high-quality pseudo-queries generated by RA; if and only if When the similarity score is greater than the preset threshold α, retain Therefore, each pseudo-query obtained through filtering is utilized. And its corresponding dialogue history u ＜i Construct pseudo-instances; finally, use the pseudo-instances constructed above to further train QP with standard cross-entropy loss.

[0054] Specifically, the training strategy used in this stage varies slightly depending on the scenario. Specifically, in cross-domain scenarios, the optimal QP model checkpoint obtained by S102 is fine-tuned directly on the pseudo-instances labeled with RA. In low-resource scenarios, the QP is retrained using the original pre-trained model. It is also important to note that pseudo-queries are introduced when constructing pseudo-instances. Corresponding dialogue reply u i The RA is further trained in the above manner, and additional fine-tuning is performed on the RA using labeled data in low-resource scenarios to facilitate subsequent training.

[0055] S105, the response enhancement query generation model trained in the second stage is used to score the candidate queries sampled by the query generation model trained in the second stage to obtain a reward score, so as to carry out the third stage of reinforcement learning training of the query generation model based on the reward score.

[0056] It should be noted that reinforcement learning guided by RA is used here to further enhance QP. First, the small number of pseudo queries obtained by S103 may still be of low quality, which may have a negative impact on the model. Second, due to the difference in model input between QP and RA, QP still has difficulty obtaining sufficient response information guidance by only using pseudo queries generated by RA. Therefore, the REINFORCE algorithm is introduced in this stage to solve the above problems.

[0057] There are several ways to score candidate queries sampled by the second-stage trained query generation model using the response enhancement query generation model trained in the second stage in order to obtain reward scores.

[0058] As an example, the dialogue history from the dialogue data of unlabeled dialogue queries is input into the query generation model trained in the second stage to generate a candidate query set. The length-normalized log probability of each candidate query is obtained from the candidate query set. The length-normalized log probability of all candidate queries is normalized to obtain the predicted probability distribution, which is used as the random sampling strategy for reinforcement learning. The dialogue history and dialogue responses from the dialogue data of unlabeled dialogue queries are input into the response-enhancing query generation model trained in the second stage to calculate the length-normalized log probability corresponding to each candidate query, and the length-normalized log probability is used as the reward score.

[0059] As another example, the dialogue history from the dialogue data of unlabeled dialogue queries is input into the query generation model trained in the second stage to generate a candidate query set; the length-normalized log probability of each candidate query is obtained from the candidate query set, and the length-normalized log probability of all candidate queries is normalized to obtain a predicted probability distribution, which is used as a random sampling strategy for reinforcement learning; the dialogue history and dialogue responses from the dialogue data of unlabeled dialogue queries are input into the response-enhancing query generation model trained in the second stage to calculate the length-normalized log probability corresponding to each candidate query; all candidate queries are sorted in descending order according to the length-normalized log probability to obtain a reward score.

[0060] As a specific example, such as Figure 2 As shown, for each dialogue history u in the unlabeled dialogue corpus ＜i First, it is input into QP. Based on the predicted probability distribution of QP, N is obtained through beam search. c One candidate query; for each candidate query Calculate the length-normalized log probability of QP as follows:

[0061]

[0062] in This represents the j-th query tag; further, for all candidate queries... After performing softmax normalization, a prediction probability distribution is finally obtained, which serves as... The reinforcement learning random sampling strategy.

[0063] Next, as an example, the following two types of rewards Both are usable:

[0064] Probability-based rewards: and The calculation method is similar, using the dialogue history u ＜i And reply to the conversation u i Inputting the data into the RA (Automatic Retrieval System), each candidate query is calculated. The corresponding length-normalized log probability is denoted as Use this probability directly as the reward:

[0065] Ranking-based rewards: based on Sort all candidate queries in descending order and obtain the following reward:

[0066]

[0067] in, It is based on Returns the input candidate query A function that sorts N in descending order. c Normalizing candidate queries can reduce the variance of reinforcement learning gradient estimation; obviously, this normalization step will penalize those candidate queries that rank lower.

[0068] Ultimately, QP undergoes reinforcement learning training guided by reward points:

[0069]

[0070] Therefore, compared to the pseudo-query obtained in S103, the reward provided by RA in this stage is a more granular training signal.

[0071] In summary, the training method for the response-enhanced dialogue query generation model according to the embodiments of this application firstly obtains a training sample set, wherein the training sample set includes dialogue data with labeled dialogue queries and dialogue data with unlabeled dialogue queries, and the dialogue data includes dialogue history and dialogue responses; then, a first language model and a second language model are constructed, the first language model is trained using the dialogue history in the labeled dialogue query data to obtain the query generation model, and the second language model is trained using the dialogue history and dialogue responses in the labeled dialogue query data to obtain the response-enhanced query generation model; finally, the dialogue history in the unlabeled dialogue query data is input into the query generation model to generate... The query set, along with the dialogue history and responses from the unlabeled dialogue query data, is input into the response-enhanced query generation model to generate pseudo-queries. Next, a similarity score is obtained for each pseudo-query based on the query set and pseudo-queries. Pseudo-instances are constructed by selecting each pseudo-query with a similarity score greater than a preset threshold and its corresponding dialogue history and responses. These pseudo-instances are then used to train the query generation model and the response-enhanced query generation model in the second stage. Finally, the response-enhanced query generation model trained in the second stage is used to score the candidate queries sampled by the query generation model trained in the second stage to obtain a reward score. This reward score is then used to train the query generation model in the third stage of reinforcement learning.

[0072] To implement the above embodiments, this application proposes a computer-readable storage medium storing a training program for a response-enhanced dialogue query generation model. When the training program for the response-enhanced dialogue query generation model is executed by a processor, it implements the training method for the response-enhanced dialogue query generation model as described above.

[0073] According to embodiments of this application, a computer-readable storage medium stores a training program for a response-enhanced dialogue query generation model. This enables a processor to implement the aforementioned training method for the response-enhanced dialogue query generation model when executing the training program. Thus, through a three-stage model training framework, knowledge of dialogue responses is fully utilized, providing multi-granular training guidance for the query generation model. This makes the model suitable for dialogue query generation in cross-domain and low-resource scenarios, resulting in more accurate dialogue queries.

[0074] To implement the above embodiments, this application proposes a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the training method for the response-enhanced dialogue query generation model as described above.

[0075] According to the computer device of the present application embodiment, the training of the response-enhanced dialogue query generation model is stored in the memory, so that when the processor executes the training program of the response-enhanced dialogue query generation model, the training method of the response-enhanced dialogue query generation model as described above is implemented. Thus, through the three-stage model training framework, the knowledge of dialogue responses is fully utilized, and multi-granular training guidance is given to the query generation model, thereby making it suitable for dialogue query generation in cross-domain and low-resource scenarios, and obtaining more accurate dialogue queries.

[0076] To implement the above embodiments, this application also proposes a training device for a dialogue query generation model based on response enhancement, such as... Figure 3 As shown, the training device for the response-enhanced dialogue query generation model includes: a first acquisition module 10, a first training module 20, a second acquisition module 30, a second training module 40, and a third training module 50.

[0077] The first acquisition module 10 is used to acquire a training sample set, which includes dialogue data for labeled dialogue queries and dialogue data for unlabeled dialogue queries. The dialogue data includes dialogue history and dialogue responses. The first training module 20 is used to construct a first language model and a second language model. The first language model is trained using the dialogue history from the labeled dialogue query data to obtain a query generation model, and the second language model is trained using the dialogue history and dialogue responses from the labeled dialogue query data to obtain a response-enhanced query generation model. The second acquisition module 30 is used to input the dialogue history from the unlabeled dialogue query data into the query generation model to generate a query set, and to input the dialogue history from the unlabeled dialogue query data into the query generation model to generate a query set. The dialogue history and dialogue responses in the query data are input into the response-enhanced query generation model to generate pseudo queries; the second training module 40 is used to obtain the similarity score of each pseudo query based on the query set and pseudo queries, and select each pseudo query with a similarity score greater than a preset threshold and its corresponding dialogue history and dialogue responses to construct pseudo instances, so as to perform the second stage training of the query generation model and the response-enhanced query generation model based on the pseudo instances; the third training module 50 is used to score the candidate queries sampled by the query generation model trained in the second stage using the response-enhanced query generation model trained in the second stage to obtain reward scores, so as to perform the third stage reinforcement learning training of the query generation model based on the reward scores.

[0078] As an example, the similarity score for each pseudo-query is obtained according to the following formula:

[0079]

[0080] Where Fsim represents the text similarity function. This indicates a pseudo-query. This represents the a-th query in the query set. This represents a query set consisting of N queries.

[0081] As an example, the third training module 50 is further used to input the dialogue history in the dialogue data of the unlabeled dialogue query into the query generation model trained in the second stage to generate a candidate query set; obtain the length-normalized log probability of each candidate query based on the candidate query set, normalize the length-normalized log probability of all candidate queries to obtain a predicted probability distribution, and use it as a random sampling strategy for reinforcement learning; input the dialogue history and dialogue responses in the dialogue data of the unlabeled dialogue query into the response-enhancing query generation model trained in the second stage to calculate the length-normalized log probability corresponding to each candidate query, and use the length-normalized log probability as a reward score.

[0082] As an example, the third training module 50 is further configured to input the dialogue history from the dialogue data of the unlabeled dialogue query into the query generation model trained in the second stage to generate a candidate query set; obtain the length-normalized log probability of each candidate query based on the candidate query set, normalize the length-normalized log probability of all candidate queries to obtain a predicted probability distribution, and use it as a random sampling strategy for reinforcement learning; input the dialogue history and dialogue responses from the dialogue data of the unlabeled dialogue query into the response-enhancing query generation model trained in the second stage to calculate the length-normalized log probability corresponding to each candidate query; and sort all candidate queries in descending order according to the length-normalized log probability to obtain a reward score.

[0083] It should be noted that the above regarding Figure 1 The example of the training method for the response-enhanced dialogue query generation model is also applicable to the training device for the response-enhanced dialogue query generation model, and will not be elaborated here.

[0084] In summary, the training apparatus for the response-enhanced dialogue query generation model according to the embodiments of this application, through a three-stage model training framework, fully utilizes the knowledge of dialogue responses and provides multi-granular training guidance for the query generation model, thereby making it suitable for dialogue query generation in cross-domain and low-resource scenarios and obtaining more accurate dialogue queries.

[0085] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0086] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0087] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0088] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0089] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. This application can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0090] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0091] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

[0092] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0093] In this application, unless otherwise expressly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0094] In this application, unless otherwise expressly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "on top of," and "over" the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0095] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. The illustrative expressions of the above terms in this specification should not be construed as necessarily referring to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0096] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A training method for a dialogue query generation model based on response enhancement, characterized in that, Includes the following steps: Obtain a training sample set, wherein the training sample set includes dialogue data with labeled dialogue queries and dialogue data with unlabeled dialogue queries, and the dialogue data includes dialogue history and dialogue responses; A first language model and a second language model are constructed. The first language model is trained using the dialogue history in the dialogue data of the labeled dialogue query to obtain a query generation model. The second language model is trained using the dialogue history and dialogue responses in the dialogue data of the labeled dialogue query to obtain a response-enhanced query generation model. The dialogue history in the dialogue data of the unlabeled dialogue query is input into the query generation model to generate a query set, and the dialogue history and dialogue responses in the dialogue data of the unlabeled dialogue query are input into the response-enhanced query generation model to generate a pseudo query. Based on the query set and the pseudo-queries, a similarity score is obtained for each pseudo-query. Each pseudo-query with a similarity score greater than a preset threshold is selected, and a pseudo-instance is constructed along with the corresponding dialogue history and dialogue response. This pseudo-instance is then used to train the query generation model and the response-enhanced query generation model in the second stage. The response enhancement query generation model trained in the second stage is used to score the candidate queries sampled by the query generation model trained in the second stage to obtain a reward score, so as to perform a third stage of reinforcement learning training on the query generation model based on the reward score.

2. The training method for the dialogue query generation model based on response enhancement as described in claim 1, characterized in that, The similarity score for each pseudo-query is obtained using the following formula: Among them, F sim This represents a text similarity function. This indicates a pseudo-query. This represents the x-th query in the query set. This represents a query set consisting of N queries.

3. The training method for the dialogue query generation model based on response enhancement as described in claim 1, characterized in that, The response-enhanced query generation model trained in the second stage is used to score the candidate queries sampled by the query generation model trained in the second stage to obtain a reward score, including: The dialogue history in the dialogue data of the unlabeled dialogue query is input into the query generation model trained in the second stage to generate a candidate query set. The length-normalized log probability of each candidate query is obtained from the candidate query set. The length-normalized log probability of all candidate queries is normalized to obtain the predicted probability distribution, which is then used as a random sampling strategy for reinforcement learning. The dialogue history and dialogue responses from the unlabeled dialogue query data are input into the response enhancement query generation model trained in the second stage, so as to calculate the length-normalized log probability corresponding to each candidate query, and the length-normalized log probability is used as the reward score.

4. The training method for the dialogue query generation model based on response enhancement as described in claim 1, characterized in that, The response-enhanced query generation model trained in the second stage is used to score the candidate queries sampled by the query generation model trained in the second stage to obtain a reward score, including: The dialogue history in the dialogue data of the unlabeled dialogue query is input into the query generation model trained in the second stage to generate a candidate query set. The length-normalized log probability of each candidate query is obtained from the candidate query set. The length-normalized log probability of all candidate queries is normalized to obtain the predicted probability distribution, which is then used as a random sampling strategy for reinforcement learning. The dialogue history and dialogue responses in the dialogue data of the unlabeled dialogue query are input into the response enhancement query generation model trained in the second stage, so as to calculate the length-normalized log probability corresponding to each candidate query. All candidate queries are sorted in descending order based on the length-normalized logarithmic probability to obtain the reward score.

5. A computer-readable storage medium, characterized in that, It stores a training program for a response-enhanced dialogue query generation model, which, when executed by a processor, implements the training method for a response-enhanced dialogue query generation model as described in any one of claims 1-4.

6. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the training method for the dialogue query generation model based on response enhancement as described in any one of claims 1-4.

7. A training device for a dialogue query generation model based on response enhancement, characterized in that, include: The first acquisition module is used to acquire a training sample set, wherein the training sample set includes labeled dialogue query data and unlabeled dialogue query data, and the dialogue data includes dialogue history and dialogue replies; The first training module is used to construct a first language model and a second language model. The first language model is trained using the dialogue history in the dialogue data of the labeled dialogue query to obtain a query generation model, and the second language model is trained using the dialogue history and dialogue responses in the dialogue data of the labeled dialogue query to obtain a response-enhanced query generation model. The second acquisition module is used to input the dialogue history in the dialogue data of the unlabeled dialogue query into the query generation model to generate a query set, and to input the dialogue history and dialogue replies in the dialogue data of the unlabeled dialogue query into the reply-enhanced query generation model to generate a pseudo query. The second training module is used to obtain the similarity score of each pseudo-query based on the query set and the pseudo-query, select each pseudo-query with a similarity score greater than a preset threshold and construct pseudo-instances with the corresponding dialogue history and dialogue response, so as to perform a second stage of training on the query generation model and the response-enhanced query generation model based on the pseudo-instances; The third training module is used to score the candidate queries sampled by the query generation model trained in the second stage using the response enhancement query generation model trained in the second stage to obtain a reward score, so as to perform the third stage of reinforcement learning training on the query generation model based on the reward score.

8. The training apparatus for the dialogue query generation model based on response enhancement as described in claim 7, characterized in that, The similarity score for each pseudo-query is obtained using the following formula: Among them, F sim This represents a text similarity function. This indicates a pseudo-query. This represents the x-th query in the query set. This represents a query set consisting of N queries.

9. The training apparatus for the dialogue query generation model based on response enhancement as described in claim 7, characterized in that, The third training module is also used for: The dialogue history in the dialogue data of the unlabeled dialogue query is input into the query generation model trained in the second stage to generate a candidate query set. The length-normalized log probability of each candidate query is obtained from the candidate query set. The length-normalized log probability of all candidate queries is normalized to obtain the predicted probability distribution, which is then used as a random sampling strategy for reinforcement learning. The dialogue history and dialogue responses from the unlabeled dialogue query data are input into the response enhancement query generation model trained in the second stage, so as to calculate the length-normalized log probability corresponding to each candidate query, and the length-normalized log probability is used as the reward score.

10. The training apparatus for the dialogue query generation model based on response enhancement as described in claim 7, characterized in that, The third training module is also used for: The dialogue history in the dialogue data of the unlabeled dialogue query is input into the query generation model trained in the second stage to generate a candidate query set. The length-normalized log probability of each candidate query is obtained from the candidate query set. The length-normalized log probability of all candidate queries is normalized to obtain the predicted probability distribution, which is then used as a random sampling strategy for reinforcement learning. The dialogue history and dialogue responses in the dialogue data of the unlabeled dialogue query are input into the response enhancement query generation model trained in the second stage, so as to calculate the length-normalized log probability corresponding to each candidate query. All candidate queries are sorted in descending order based on the length-normalized logarithmic probability to obtain the reward score.

Images