Dialogue Data Sample Enhancement Method, Dialogue Text Generation Model Pretraining Method and System

Abstract

The present invention relates to the technical field of artificial intelligence dialogue text generation, and particularly relates to a method for enhancing dialogue data samples, a method and a system for pre-training a dialogue text generation model. A single-round dialogue is selected from the collected dialogue sample data set, and the single-round dialogue includes dialogue content, dialogue response and the best dialogue knowledge. Among them, the best dialogue knowledge is the external knowledge with the best relevance to the single-round dialogue topic and assisting in generating the dialogue response; based on cross-rotation, the dialogue content, dialogue response and the best dialogue knowledge in the single-round dialogue are exchanged to generate a new extended single-round dialogue; the original single-round dialogue and the new extended single-round dialogue are spliced to obtain a dialogue sample data set after data enhancement. The present invention can strengthen the utilization of existing knowledge dialogue data, realize data enhancement by a simple method without increasing the data annotation cost, improve the model performance, and enable the model to obtain the ability to generate responses from different perspectives during the training process.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence dialogue text generation technology, and in particular to a dialogue data sample enhancement method and a dialogue text generation model pre-training method and system, which enhance the dialogue data samples by cross-rotating dialogue content sentences from multiple perspectives to achieve training optimization of the dialogue text generation model. Background Art

[0002] With the rapid development of internet information technology, the application of artificial intelligence to intelligent conversational question-answering systems is making human-computer interaction a reality. With the application of deep learning in fields such as image and language processing, neural network-based language models have rapidly become a research hotspot. In recent years, generative models based on large-scale language models have acquired domain knowledge through extensive pre-training to generate high-quality text, such as answers to questions or pluralized content. Conversational question-answering generative models are neural network-based machine translation models that model historical conversation data and related knowledge bases to generate novel natural language text such as news, daily conversations, text messages, and emails. For example, the sequence-to-sequence Seq2Seq model converts input sequences into output sequences, enabling the generation of text of arbitrary length and complexity. In practical applications, the accuracy of conversational text generation models is affected by the effectiveness of model training. A good intelligent conversational question-answering system cannot do without the assistance of external knowledge, but it is not easy to annotate and collect standard knowledge. At the same time, previous data augmentation methods, such as character-level deletion, swapping, and insertion, word-level noise addition and embedding substitution, phrase- and sentence-level cropping, and document-level back translation, lack consideration of external conversational knowledge and cannot adapt well to knowledge conversation data, which in turn affects the quality of enhanced samples in model training. Summary of the Invention

[0003] To this end, the present invention provides a conversation data sample enhancement method based on multi-perspective sentence cross-rotation and a conversation text generation model pre-training method and system, which effectively solves the problem that the existing data enhancement methods lack external knowledge considerations, thus affecting the quality of the enhanced samples. By strengthening the utilization of existing knowledge conversation data and achieving sample data enhancement without increasing data annotation costs, the model training effect is improved.

[0004] According to the design scheme provided by the present invention, on the one hand, a method for enhancing dialogue data samples based on multi-perspective sentence rotation is provided, which includes the following contents:

[0005] Collecting conversation sample data to form a sample dataset, and selecting a single-turn conversation from the sample dataset. The single-turn conversation includes conversation content, conversation responses, and optimal conversation knowledge. The optimal conversation knowledge is external knowledge that is most relevant to the topic of the single-turn conversation and assists in generating conversation responses.

[0006] Exchange the conversation content, conversation response, and best conversation knowledge in a single-round conversation based on cross-rotation to generate a new extended single-round conversation;

[0007] Concatenate the original single-round conversation and the new extended single-round conversation to obtain a conversation sample dataset with enhanced data.

[0008] Furthermore, collect conversation samples and form a sample dataset, including:

[0009] First, select the gold knowledge from the external knowledge set as the best conversation knowledge according to the conversation content;

[0010] Next, use the conversation content and the corresponding best conversation knowledge as the model input, and use the pre-trained knowledge-aware generative conversation model to generate conversation responses;

[0011] Then, combine the conversation content, conversation response, and best conversation knowledge to obtain conversation sample data.

[0012] Furthermore, select the gold knowledge from the external knowledge set as the best conversation knowledge according to the conversation content, including:

[0013] First, use the pre-trained language model to encode the conversation content and the text of the external knowledge set into corresponding text embedded representation vectors respectively;

[0014] Then, obtain the conditional probability distribution of the external knowledge corresponding to the conversation content based on the text embedded representation vectors, and select the gold knowledge corresponding to the conversation content by maximizing the posterior probability.

[0015] Furthermore, exchange the conversation content, conversation response, and best conversation knowledge in a single-round conversation based on cross-rotation to generate a new extended single-round conversation, including:

[0016] Generate a new single-round conversation based on the cross-rotation operation, where the cross-rotation operation includes: conversation cross-rotation between the conversation content and the conversation response, and / or knowledge cross-rotation between the best conversation knowledge and the conversation content and the conversation response.

[0017] Furthermore, the conversation cross-rotation includes: exchanging the conversation content and the conversation response in a single-round conversation to use the original conversation response as the new conversation content and the original conversation content as the new conversation response.

[0018] Furthermore, the knowledge cross-rotation includes: forward-exchanging the conversation content and the best conversation knowledge in a single-round conversation to use the original conversation content as the new best conversation knowledge and the original best conversation knowledge as the new conversation content; and / or, backward-exchanging the conversation response and the best conversation knowledge in a single-round conversation to use the original conversation response as the new best conversation knowledge and the original best conversation knowledge as the new conversation response.

[0019] On the other hand, in combination with the above-mentioned method for enhancing dialogue data samples based on multi-perspective sentence cross-rotation, the present invention also provides a method for pre-training a dialogue text generation model, including:

[0020] Obtaining model training data samples by using the above-mentioned method for enhancing dialogue data samples;

[0021] Training and optimizing the dialogue text generation model by using the model training data samples to obtain a target dialogue text generation model for a dialogue question-and-answer system.

[0022] On yet another hand, in combination with the above-mentioned method for enhancing dialogue data samples based on multi-perspective sentence cross-rotation, the present invention also provides a pre-training system for a dialogue text generation model, including: a data collection module and a model training module, where

[0023] The data collection module is used to obtain model training data samples by using the above-mentioned method for enhancing dialogue data samples;

[0024] The model training module is used to train and optimize the dialogue text generation model by using the model training data samples to obtain a target dialogue text generation model for a dialogue question-and-answer system.

[0025] Advantages of the present invention:

[0026] The present invention strengthens the utilization of existing knowledge dialogue data, enhances the sample data by performing cross-rotation operations on the conversation content, conversation response, and best conversation knowledge in different sentences, realizes data enhancement with a simple method without increasing the data annotation cost, improves the model performance, and enables the model to obtain the ability to generate responses from different perspectives during the training process. In order to conform to the real dialogue scenario, by selecting good external knowledge, the enhanced data can enable the model to better learn the content in the dialogue data, accurately understand the connection between knowledge and conversation content, improve the accuracy of the knowledge selection module in the model, and enhance the model training effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Schematic diagram of the process for enhancing dialogue data samples based on multi-perspective sentence cross-rotation in the embodiment;

[0028] Figure 2Schematic diagram of the principle of the knowledge-aware dialogue generation model in the embodiment;

[0029] Figure 3 Schematic diagram of the sentence cross-rotation process in the embodiment;

[0030] Figure 4 Example of sentence cross-rotation in the embodiment. Detailed implementation manners

[0031] To make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings and technical solutions.

[0032] In view of the situation that the existing dialogue sample data enhancement method described in the background technology cannot well adapt to knowledge dialogue data and affects the training effect of the speech recognition model, in an embodiment of the present invention, as shown in Figure 1 shown, a dialogue data sample enhancement method based on multi-perspective sentence cross-rotation is provided, including the following contents:

[0033] S101. Collect dialogue sample data and form a sample data set, and select single-turn dialogues from the sample data set. The single-turn dialogues include dialogue content, dialogue responses, and the best dialogue knowledge. Among them, the best dialogue knowledge is the external knowledge with the best relevance to the single-turn dialogue theme and assisting in generating dialogue responses.

[0034] Specifically, collecting dialogue samples and forming a sample data set can be designed to include the following contents:

[0035] First, select golden knowledge from the external knowledge set as the best dialogue knowledge according to the dialogue content;

[0036] Next, use the dialogue content and the corresponding best dialogue knowledge as the model input, and use the pre-trained knowledge-aware generative dialogue model to generate dialogue responses;

[0037] Then, combine the dialogue content, dialogue responses, and the best dialogue knowledge to obtain dialogue sample data.

[0038] Among them, selecting golden knowledge from the external knowledge set as the best dialogue knowledge according to the dialogue content may include:

[0039] First, use the pre-trained language model to encode the text of the dialogue content and the external knowledge set into corresponding text embedded representation vectors respectively;

[0040] Then, obtain the conditional probability distribution of the external knowledge corresponding to the dialogue content based on the text embedded representation vectors, and select the golden knowledge corresponding to the dialogue content by maximizing the posterior probability.

[0041] Such as Figure 2As shown in the figure, the input of the knowledge-aware generative dialogue model includes the dialogue content and external knowledge. When the model generates an answer, the external knowledge representation will assist the context representation to jointly generate a response. In actual conversations, the golden knowledge cannot be directly obtained, and the best current golden knowledge needs to be selected from the obtained external knowledge set. Therefore, the training of a knowledge selection module is added to the knowledge-aware generative dialogue model, that is Figure 2 the yellow dotted-line module in the figure. Through this module, the model's ability to select knowledge is exercised. Generally, external knowledge includes structured knowledge graphs and unstructured documents. For knowledge graphs, the knowledge selection module aims to select the best triples, while for unstructured documents, the best text information that can support dialogue generation needs to be selected. In the embodiments of this case, the external knowledge can be set as unstructured documents, that is, the external knowledge includes multiple texts with a relatively high degree of relevance to the dialogue topic.

[0042] In a specific task, given the input sentence X = (x1...x n ) and the external knowledge set K = {k1,..., k l}), existing language models such as the BERT model can be used to encode the input sentence and the external knowledge set into and Then, based on the encoded embedding vector representation, the best knowledge is selected. As shown in Equation (1) for the knowledge conditional probability distribution, the golden knowledge K gold is selected by maximizing the posterior probability P(k = k gold |x). After that, based on the input sentence and knowledge, the response Y = (y1...y m ) can be generated according to the probability distribution shown in Equation (2).

[0043] k ~ P(k = k gold |x) (1)

[0044] y ~ P(y|x, k gold ) (2)

[0045] S102. Exchange the dialogue content, dialogue response, and dialogue best knowledge in a single-round dialogue based on cross-rotation to generate a new extended single-round dialogue.

[0046] Specifically, exchanging the dialogue content, dialogue response, and dialogue best knowledge in a single-round dialogue based on cross-rotation to generate a new extended single-round dialogue can be designed to include the following content:

[0047] Based on cross-rotation operations, a new single-round dialogue is generated, where the cross-rotation operations include: dialogue cross-rotation between the dialogue content and the dialogue response, and / or knowledge cross-rotation between the dialogue best knowledge and the dialogue content and the dialogue response.

[0048] Among them, the dialogue cross-rotation may include: exchanging the dialogue content and dialogue response in a single-round dialogue, so as to use the original dialogue response as the new dialogue content and the original dialogue content as the new dialogue response. The knowledge cross-rotation may include: forward-exchanging the dialogue content and the best dialogue knowledge in a single-round dialogue, so as to use the original dialogue content as the new best dialogue knowledge and the original best dialogue knowledge as the new dialogue content; and / or backward-exchanging the dialogue response and the best dialogue knowledge in a single-round dialogue, so as to use the original dialogue response as the new best dialogue knowledge and the original best dialogue knowledge as the new dialogue response.

[0049] As Figure 3 shown, three rotation operations are performed on the original data set, namely dialogue content cross-rotation and knowledge cross-rotation. Among them, the knowledge rotation is further divided into forward knowledge cross-rotation and backward knowledge cross-rotation. The data newly added through the rotation operation is added to the original data set to expand it into a new data set. A specific example of data augmentation is as Figure 4 shown. Take a single-round dialogue content dialogA(Src,Gold K,Tgt) in the original data set and perform the Figure 3 operations shown in ①, ②, and ③ in it: dialogue content cross-rotation, that is, exchanging the input Src and the output Tgt of the dialogue; forward knowledge cross-rotation, that is, exchanging the input Src and the best knowledge Gold K; and backward knowledge cross-rotation, that is, exchanging the output Tgt and the best knowledge Gold K, so as to obtain extended new dialogue data.

[0050] S103. Concatenate the original single-round dialogue and the new extended single-round dialogue to obtain an enhanced dialogue sample data set.

[0051] As Figure 3 and 4 shown, based on the original dialogue dialogA, three new dialogue data are extended, namely dialogA1, dialogA2, and dialogA3. Data augmentation is performed based on the original dialogue and the extended new dialogue data. That is, the new enhanced sample data set includes four different combinations, namely [(Src,Gold K,Tgt),(Tgt,Gold K,Src),(Gold K,Src,Tgt),(Src,Tgt,Gold K)], and each dialogue combination in the sample data set is associated with the existing knowledge, so that the enhanced data sample can well adapt to the intelligent knowledge dialogue, enabling the model to better learn the content in the dialogue data and more accurately understand the connection between knowledge and dialogue content.

[0052] Furthermore, based on the above dialogue data sample enhancement method based on multi-perspective statement cross-rotation, an embodiment of the present invention further provides a pre-training method for a dialogue text generation model, including:

[0053] Obtain model training data samples by using the above-mentioned dialogue data sample enhancement method;

[0054] Use the model training data samples to train and optimize the dialogue text generation model, so as to obtain the target dialogue text generation model for the dialogue question-answering system.

[0055] Furthermore, in combination with the above-mentioned dialogue data sample enhancement method based on multi-perspective sentence cross-rotation, the embodiment of the present invention also provides a dialogue text generation model pre-training system, including: a data collection module and a model training module, wherein,

[0056] The data collection module is used to obtain model training data samples by using the above-mentioned dialogue data sample enhancement method;

[0057] The model training module is used to train and optimize the dialogue text generation model by using the model training data samples, so as to obtain the target dialogue text generation model for the dialogue question-answering system.

[0058] To verify the effectiveness of the solution of this case, the following further explanation is made in combination with theoretical analysis and experimental data:

[0059] 1. Theoretical analysis: The conditional distribution of knowledge under the original data is shown in Equation (1), which represents that the model, under the condition of the known input sentence, uses the maximum posterior probability criterion to find the best knowledge. Based on the best knowledge selected by the model and the input sentence, training and optimization are carried out with Equation (2) as the objective function, and the model can generate responses.

[0060]

[0061] After the sentence cross-rotation operation, the knowledge selection module of the model is equivalent to adding three types of tasks: given the best knowledge, finding the input sentence from the candidate knowledge mixed with the input sentence; given the output sentence, finding the best knowledge from the candidate knowledge; and given the best knowledge, finding the output sentence from the candidate knowledge mixed with the output sentence. Their conditional distributions are shown in the following Equations (4), (5), and (6) respectively.

[0062] k~P(k=x|k gold ) (4)

[0063] k~P(k=y gold |y) (5)

[0064] k~P(k=y|x) (6)

[0065] Based on the knowledge distributions described in Equations (4), (5), and (6), the model selects the required knowledge, and then obtains the probability distribution of the corresponding generated sentences, and its loss functions are shown in Equations (7), (8), and (9).

[0066]

[0067]

[0068]

[0069] Starting from the knowledge selection process, through the exchange of dialogue content and best knowledge, the understanding of the internal information connection among the source sentence, target sentence, and best knowledge is further strengthened. Since knowledge helps generate responses for the dialogue and has a very high correlation with the dialogue topic, among many candidate knowledge, the one with the highest probability should be the golden knowledge or the rotated input or output sentence, which improves the quality of the enhanced model training samples and facilitates the model to better understand the connection between knowledge and dialogue, so as to improve the model training effect.

[0070] 2. Experimental analysis: Use Wizard-of-Wikipedia and Holl-E as the experimental datasets for data augmentation in the experiment. Train and test according to the division of the original dataset into training / validation / test sets, and filter and fine-tune the content. Process the original dataset into two forms: single-round dialogue and multi-round dialogue. At the same time, dialogue content, dialogue candidate knowledge, and best knowledge are required in the experiment, and dialogues without using knowledge can be deleted.

[0071] Based on the existing posterior-guided knowledge selection POSTKS, the encoder and decoder can be designed to both have a two-layer GRU structure, with 800 hidden states in each layer and no shared parameters. Set the word embedding size to 300 and initialize it using GloVe. The vocabulary size is 20,000 words. Use the Adam optimizer, with a batch size of 64, a dropout of 0.3, a learning rate of 0.0005, a data scale coefficient α of 1.0, and an augmented data ratio β of 1.0. Train the model for 10 epochs on an A5000 machine. During the training process, after each epoch, save a model, and select the model with the minimum loss in the current epoch for evaluation based on the total loss as the selection criterion.

[0072] Combined with the specific scheme of the embodiments of this case, the data augmentation algorithm MSCR in the experiment can be designed to include the following steps:

[0073] Step 1: Dataset normalization. Form a json file in the dictionary form of the source dialogue sentence, knowledge, target dialogue sentence, and knowledge label.

[0074] Step 2: Read the data in the file.

[0075] Step 3: Start the loop. Each loop targets a dialogue data dialogA(Src, Gold K, Tgt).

[0076] Step 4: Cross-rotate the dialogue content, that is, exchange the dialogue input Src and the output Tgt to obtain the new data dialogA1(Tgt, Gold K, Src).

[0077] Step 5: Cross-rotate the forward knowledge, that is, exchange the input Src and the best knowledge Gold K to obtain the new data dialogA2(Gold K, Src, Tgt).

[0078] Step 6: Cross-rotate the backward knowledge, that is, exchange the output Tgt and the best knowledge Gold K to obtain the new data dialogA3(Src, Tgt, Gold K).

[0079] Step 7: Store the new data set.

[0080] Compare the existing RAW method, BT method, and EDA data augmentation methods including SR, RS, RI, and RD. The experimental results are shown in Table 1.

[0081] Table 1 Performance under different data augmentation methods

[0082]

[0083] MSCR achieved the optimal results in terms of ACC, BLEU1 / 2 / 3 / 4, DIST1 / 2, K-P / R / F1, and PPL metrics. The overall performance of the four strategies under the EDA method was at a sub-optimal level, while BT did not have an improvement effect on the model performance. The experimental results showed that compared with other data augmentation methods, the data augmentation scheme in this case could improve the generation accuracy ACC, word coverage BLEU-1 / 2 / 3 rates, diversity DIST-1 / 2, knowledge selection K-P / R / F1, and model fitting ability PPL. In contrast, other data augmentation methods participating in the comparison could only improve a few metrics. The above data further proved the effectiveness of the scheme in this case. At the same time, the scheme in this case could be improved on various models and data sets.

[0084] Unless otherwise specifically stated, the relative steps, numerical expressions, and values of the components and steps described in these embodiments do not limit the scope of the present invention.

[0085] In the present specification, the various embodiments are described in a progressive manner. Each embodiment focuses on the differences from other embodiments, and for the same or similar parts among the various embodiments, reference can be made to each other. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and for the relevant parts, reference can be made to the description in the method section.

[0086] The units and method steps of the various examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. A person of ordinary skill in the art can use different methods to implement the described functions for each specific application, but such implementation is not considered to exceed the scope of the present invention.

[0087] A person of ordinary skill in the art can understand that all or part of the steps in the above methods can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a magnetic disk, or an optical disc, etc. Optionally, all or part of the steps of the above embodiments can also be implemented using one or more integrated circuits. Correspondingly, the various modules / units in the above embodiments can be implemented in the form of hardware or in the form of software function modules. The present invention is not limited to any specific form of the combination of hardware and software.

[0088] Finally, it should be noted that the above-described embodiments are only specific implementation manners of the present invention, used to illustrate the technical solutions of the present invention, rather than to limit it. The protection scope of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, a person of ordinary skill in the art should understand that: any person skilled in the art within the technical scope disclosed by the present invention can still modify the technical solutions recorded in the foregoing embodiments, or can easily think of changes, or perform equivalent replacements on some of the technical features; and these modifications, changes, or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A method for enhancing dialogue data samples, characterized in that, It includes the following content: Collect dialogue sample data and form a sample data set, select single-turn dialogues from the sample data set, where the single-turn dialogue includes dialogue content, dialogue response, and the best dialogue knowledge. The best dialogue knowledge is the external knowledge with the best relevance to the single-turn dialogue topic and assisting in generating the dialogue response; Based on the cross-rotation operation, exchange the dialogue content, dialogue response, and the best dialogue knowledge in the single-turn dialogue to generate a new extended single-turn dialogue. The cross-rotation operation includes: dialogue cross-rotation between the dialogue content and the dialogue response, and knowledge cross-rotation between the best dialogue knowledge and the dialogue content and the dialogue response. The dialogue cross-rotation includes: exchanging the dialogue content and the dialogue response in the single-turn dialogue to use the original dialogue response as the new dialogue content and the original dialogue content as the new dialogue response. The knowledge cross-rotation includes: forward exchange between the dialogue content and the best dialogue knowledge in the single-turn dialogue to use the original dialogue content as the new best dialogue knowledge and the original best dialogue knowledge as the new dialogue content; and backward exchange between the dialogue response and the best dialogue knowledge in the single-turn dialogue to use the original dialogue response as the new best dialogue knowledge and the original best dialogue knowledge as the new dialogue response; Concatenate the original single-turn dialogue and the new extended single-turn dialogue to obtain a data-augmented dialogue sample data set.

2. The method for enhancing the dialogue data sample according to claim 1, wherein Collect dialogue samples and form a sample data set, including: First, select the golden knowledge from the external knowledge set as the best dialogue knowledge according to the dialogue content; Next, use the dialogue content and the corresponding best dialogue knowledge as model inputs, and use the pre-trained knowledge-aware generative dialogue model to generate dialogue responses; Then, combine the dialogue content, dialogue response, and the best dialogue knowledge to obtain dialogue sample data.

3. The method for enhancing the dialogue data sample according to claim 2, wherein Select the golden knowledge from the external knowledge set as the best dialogue knowledge according to the dialogue content, including: First, use the pre-trained language model to encode the text of the dialogue content and the external knowledge set into corresponding text embedded representation vectors respectively; Then, obtain the conditional probability distribution of the external knowledge corresponding to the dialogue content based on the text embedded representation vectors, and select the golden knowledge corresponding to the dialogue content by maximizing the posterior probability.

4. A method for pre-training a dialogue text generation model, characterized in that It includes the following content: Use the dialogue data sample enhancement method described in claim 1 to obtain model training data samples; Use the model training data samples to train and optimize the dialogue text generation model to obtain the target dialogue text generation model required for the dialogue question-answering system.

5. A pre-training system for a dialogue text generation model, characterized in that, It includes: a data collection module and a model training module, where The data collection module is used to obtain model training data samples by using the dialogue data sample enhancement method described in claim 1; The model training module is used to train and optimize the dialogue text generation model by using the model training data samples to obtain the target dialogue text generation model required for the dialogue question-answering system.

6. An electronic device, characterized in that, It includes a memory and a processor, and the processor and the memory communicate with each other through a bus; the memory stores program instructions executable by the processor, and the processor can execute the method according to any one of claims 1 to 4 by invoking the program instructions.

7. A computer-readable storage medium, characterized in that, A computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images