A dialogue assembly method and device based on skeleton guidance

Abstract

The present invention proposes a dialogue assembly method and device based on skeleton guidance, which belongs to the field of natural language technology. The skeleton words covering key information are extracted from the original dialogue to form a dialogue skeleton; the components and skeleton words in the original dialogue are combined in different ways to obtain multiple candidate assembled sentences; the probability of generating each candidate assembled sentence is calculated according to the original dialogue skeleton as a fluency feature; the number of words that do not belong to the skeleton words contained in each candidate assembled sentence is counted as the number of irrelevant words to obtain a semantic consistency feature; each candidate assembled sentence is spliced ​​with the original dialogue, input into a classifier to predict the probability, and obtain an information coverage feature; by combining the above three features, each candidate assembled sentence is evaluated to obtain the best assembled sentence. The present invention can generate assembled sentences that are fluent, have high information coverage, and are semantically consistent with the original dialogue using only a small amount of annotated data, thereby alleviating the problem of scarce annotated data.

Description

Technical Field

[0001] The invention relates to a dialogue assembly method and device based on skeleton guidance, belonging to the technical field of natural language processing. Background Art

[0002] People tend to use short, fragmented utterances rather than longer, complete sentences when they are talking. Therefore, conversations in real scenarios often consist of multiple, context-dependent, informal sentences, in which omissions, references, and redundancy are very common. In addition, since there may be some speech recognition errors when transcribing conversations into text, conversation corpora in real scenarios are often noisy. Therefore, many sentences in a conversation need to be understood in the context of the conversation. For example, given the conversation "Who sang Zhou Taiya?" "I said Planting the Sun." "Who composed it?", it contains the error of recognizing "Planting the Sun" as "Zhou Taiya", and the "Who composed it?" needs to be combined with the conversation context to be correctly understood as "Who composed it?".

[0003] However, most natural language processing systems are designed for well-formed, context-independent text input, which makes them unsuitable for handling informal, context-dependent conversations. Current research is mostly focused on developing conversation-specific systems, designing complex mechanisms to extend specific downstream tasks from single-sentence input to multi-round conversation input, so that the system can take into account richer contextual information. Conversation-specific systems can be mainly divided into two categories: one is to encode historical conversations and previous model output results with the current conversation sentence by designing complex context-aware encoders to provide context information to the decoder; the other is to fuse or modify previous model predictions to generate outputs corresponding to the current conversation sentence by designing complex context-aware decoders. However, conversation-specific systems are usually difficult to design, and the time and cost of building conversation-specific systems for a large number of downstream tasks is also unacceptable. Therefore, how to more effectively handle informal, context-dependent sentences in conversations is a key challenge.

[0004] Another idea is to assemble the noisy and interdependent informal sentences in multiple rounds of dialogue into formal, context-independent and semantically complete sentences. However, dialogue assembly is a challenging task: first, in order to integrate relevant information in interdependent and noisy dialogues, the dialogue assembly model needs to solve problems such as omissions, references, speech recognition errors and redundancy in the dialogue; second, in order to meet the input requirements of different downstream natural language processing systems, the assembled sentences must cover all the key information in the dialogue, keep the semantics consistent with the original dialogue, and be a fluent natural language sentence; third, the dialogue assembly model needs to be trained in a data-efficient way, because building large-scale manually annotated data is very expensive and time-consuming. Summary of the invention

[0005] In order to solve the inconsistency problem between multi-round dialogues and text inputs of current natural language processing systems, the present invention provides a dialogue assembly method and device based on skeleton guidance. The method combines the key skeleton information in the dialogue with a large-scale pre-trained text generation model. Only a small amount of labeled data is needed to generate assembled sentences that are fluent, have high information coverage and are semantically consistent with the original dialogue, thereby alleviating the problem of scarce labeled data.

[0006] The technical solution adopted by the present invention is as follows:

[0007] A dialogue assembly method based on skeleton guidance includes the following steps:

[0008] Extract skeleton words covering key information from the original dialogue, and combine the extracted skeleton words into a dialogue skeleton;

[0009] The components in the original dialogue and the skeleton words in the dialogue skeleton are combined in different ways, and multiple candidate assembled sentences are obtained through multi-step sampling;

[0010] Calculate the probability of generating each candidate assembled sentence based on the original dialogue skeleton as a fluency feature;

[0011] The number of words that do not belong to the skeleton words in each candidate assembled sentence is counted as the number of irrelevant words, and the negative value of the number of irrelevant words is used as the semantic consistency feature between the candidate assembled sentence and the original dialogue;

[0012] Each candidate assembled sentence is concatenated with the original dialogue, and the concatenated sentence is input into the classifier. The classifier predicts the probability based on the key information contained in the original dialogue skeleton, and uses the predicted probability as the information coverage feature;

[0013] By combining the fluency feature, semantic consistency feature and information coverage feature, each candidate assembled sentence is evaluated to obtain the best assembled sentence.

[0014] Furthermore, the last sentence in the original conversation is marked, and the original conversation and the last sentence in the original conversation are encoded using the encoder of the pre-trained RoBERTa, BERT or BART model, and a binary classifier is used to determine whether each encoded word is key information. If so, it is extracted as a skeleton word.

[0015] Furthermore, a Top-k sampling algorithm is used for multi-step sampling.

[0016] Furthermore, the concatenated sentences are input into a classifier based on the pre-trained RoBERTa, BERT or BART model.

[0017] A dialogue assembly device based on skeleton guidance, comprising:

[0018] The skeleton extraction model is used to extract skeleton words covering key information from the original dialogue and combine the extracted skeleton words into a dialogue skeleton;

[0019] The skeleton-guided generator is used to combine the components in the original dialogue with the skeleton words in the dialogue skeleton in different ways through a large-scale pre-trained language model that can generate fluent natural language, and obtain multiple candidate assembled sentences through multi-step sampling;

[0020] A skeleton-aware sorter is used to calculate the probability of generating each candidate assembled sentence according to the original dialogue skeleton through a large-scale pre-trained language model that can generate fluent natural language, as a fluency feature; count the number of words that do not belong to the skeleton words in each candidate assembled sentence as the number of irrelevant words, and use the negative value of the number of irrelevant words as the semantic consistency feature between the candidate assembled sentence and the original dialogue; concatenate each candidate assembled sentence with the original dialogue, input the concatenated sentence into a classifier, and the classifier predicts the probability according to the key information contained in the original dialogue skeleton, and uses the predicted probability as the information coverage feature; and evaluate each candidate assembled sentence by combining the fluency feature, the semantic consistency feature and the information coverage feature to obtain the best assembled sentence.

[0021] Furthermore, the skeleton extraction model includes an encoder and a binary classifier based on a pre-trained RoBERTa, BERT or BART model. The encoder encodes the original conversation and the last sentence in the original conversation; the binary classifier determines whether each encoded word is key information, and if so, it is extracted as a skeleton word.

[0022] Furthermore, the skeleton extraction model is trained by optimizing the cross entropy loss.

[0023] Furthermore, the skeleton-guided generator uses a Top-k sampling algorithm to sample a predetermined number of candidate assembly sentences from the output distribution.

[0024] Furthermore, large-scale pre-trained language models include BERT, GPT-2, or T5.

[0025] Furthermore, the concatenated sentences are input into a classifier based on the pre-trained RoBERTa, BERT or BART model.

[0026] Furthermore, the skeleton extraction model is trained with the training corpus to extract the dialogue skeleton, the extracted dialogue skeleton is used to train the skeleton-guided generator to generate candidate assembled sentences according to the dialogue, and the skeleton-aware sorter is trained based on the generated candidate assembled sentences to obtain the optimal assembled sentence.

[0027] The beneficial effects of the present invention are:

[0028] 1) Use limited labeled data to train an effective skeleton extraction model, combine the key skeleton information in the conversation with a large-scale pre-trained text generation model to generate high-quality assembled sentences, and effectively solve the problem of data sparsity.

[0029] 2) A skeleton extraction model is used to extract the key information that needs to be covered in the assembled sentence from the dialogue and form it into a skeleton, which effectively improves the coverage of the assembled sentence for key information.

[0030] 3) A skeleton-guided generator is adopted, which uses a large-scale pre-trained language model to generate multiple candidate sentences under the guidance of the dialogue skeleton, ensuring the fluency of assembled sentences.

[0031] 4) A skeleton-aware sorter is used to evaluate the fluency, coverage, and semantic consistency of each candidate sentence with the original dialogue under the guidance of the dialogue skeleton, ensuring that the model output is fluent, has high information coverage, and is semantically consistent with the original dialogue. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A framework diagram of a dialogue assembly process based on skeleton guidance provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to make the above features and advantages of the present invention more obvious and easy to understand, embodiments are given below and described in detail with reference to the accompanying drawings.

[0034] The embodiment of the present invention proposes a dialogue assembly method and device based on skeleton guidance, such as Figure 1 As shown in the figure, the device is a deep neural network encoder-decoder framework, which mainly includes: 1) a skeleton extraction model, which is used to identify the key information in the dialogue that needs to be covered by the assembled sentences; 2) a skeleton-guided generator; 3) a skeleton-aware sorter; the latter two are used to generate smooth, high-information coverage and semantically consistent integrated sentences with the original dialogue under the guidance of the skeleton. The technical solution is as follows:

[0035] 1. Skeleton extraction model

[0036] The purpose of skeleton extraction is to extract the key information that needs to be covered in the sentence assembly from a given dialogue to form a dialogue skeleton. The present invention converts skeleton extraction into a word-level sequence labeling problem. As shown below:

[0037]

[0038] Given a dialogue D = (S1, ..., S n ) and the last sentence that needs to be assembled S n , the input of the skeleton extraction model is:

[0039] D E =[S1,S2,...,[M]S n [ / M]];

[0040] [M] and [ / M] are special markers that indicate the beginning and end of the last sentence to be assembled. Then, an encoder based on the pre-trained RoBERTa, BERT or BART model is used to encode the input, and a binary classifier is applied to each encoded word to predict whether each word is key information. The predicted label is 1 if the word is a skeleton word that needs to be covered by the assembled sentence, otherwise it is 0. Continuous skeleton words are extracted and combined into a dialogue skeleton C = (c1, c2, ..., c k ), c represents the skeleton word, and k represents the number of words. The embodiment of the present invention trains the skeleton extraction model by optimizing the cross entropy loss:

[0041]

[0042] Where P i represents the probability of predicting that the i-th word is a skeleton word, y i =1 means that the i-th word appears in the answer of the assembled sentence, otherwise y i =0; n represents the total number of words, and the log base is e.

[0043] 2. Skeleton-guided Generator

[0044] Given a dialogue D = (S1, ..., S n ) and its dialogue skeleton C = (c1, c2, ..., c k ), the goal of the skeleton-guided generator is to generate candidate assembled sentences S′=(x′1,...,x′) under the guidance of the dialogue skeleton C. l ). In order to utilize the ability of large-scale pre-trained language models (such as BERT, GPT-2, T5, etc.) to generate fluent sentences, the skeleton-guided generator of the embodiment of the present invention adopts an encoder-decoder architecture based on the pre-trained T5 model, and uses the skeleton to ensure the information coverage and semantic consistency of its output. As shown below:

[0045]

[0046] The input of the skeleton-guided generator consists of two parts: the dialogue and the dialogue skeleton:

[0047] D G =[[CLS],S1,...,[M]S n [ / M]c1[SEP]c2,...,[SEP]c k ];

[0048] [CLS] indicates the start of a conversation, and [SEP] is used to separate different skeleton information. G The generator guided by the input skeleton is used to output candidate assembled sentences S′:

[0049]

[0050] where x′ t represents the word generated in step i, x′ ＜t Indicates that at x′ t In order to improve the diversity of candidate sentences and reduce the error propagation caused by skeleton extraction, the embodiment of the present invention uses the Top-k sampling algorithm to sample a predetermined number of candidate assembly sentences (6 in the embodiment of the present invention) from the output distribution of the generator model.

[0051] 3. Skeleton-aware Sorter

[0052] Given a candidate assembled sentence S′, the skeleton-aware ranker, guided by the dialogue skeleton C, utilizes three features to measure its fluency, information coverage, and semantic consistency with the original dialogue D, and selects the best assembled sentence by combining these features.

[0053] (1) Fluency: Since large-scale pre-trained language models (such as BERT, GPT-2, T5, etc.) have the ability to generate fluent natural language expressions, the present invention directly uses the probability of generating each candidate sentence based on the T5 generator model as the fluency feature f Flu (S′|D, C), to measure its fluency.

[0054] (2) Semantic consistency: Because the extracted dialogue skeleton can effectively cover the key information that needs to appear in the assembled sentence, the embodiment of the present invention counts the number of non-stop words that are not in the skeleton word set contained in each candidate assembled sentence as the number of irrelevant words, and takes the negative value of the number of irrelevant words as a measure of the semantic consistency feature between the candidate assembled sentence and the original dialogue. con (S′|C).

[0055] (3) Information coverage: This embodiment of the present invention transforms the information coverage evaluation into a natural language reasoning problem, evaluating whether the candidate assembled sentences can contain the key information of the original conversation. Specifically, the candidate assembled sentences are regarded as the premise of the natural language reasoning task, and the original conversation is regarded as the hypothesis of the natural language reasoning task. The two are concatenated and input into a classifier based on a pre-trained RoBERTa, BERT or BART model. In order to utilize the key information of the conversation contained in the conversation skeleton C, this embodiment of the present invention connects the skeleton prediction label of each word with the word embedding representation to guide the classifier which words are key information. Finally, the model prediction probability under the guidance of the conversation skeleton C is used as the information coverage feature f Cov (D, S′|C).

[0056] The final skeleton-aware ranker evaluates each candidate assembled sentence by combining the above three features:

[0057] Score=λ1f Flu (S′|D,C)+λ2f con (S′|C)+λ3f Cov (D, S′|C);

[0058] Among them, λ1+λ2+λ3=1, representing the weights of the three features respectively, and the optimal weights are 0.25, 0.25 and 0.5.

[0059] The following is an example of training and testing the device of the present invention:

[0060] Scenario:

[0061] Training corpus:

[0062] Training example 1: [“Will the bite of an ant be poisonous?”, “A bite from a black flying ant”, “Black flying ants”], the target assembly sentence is “Will the bite of a black flying ant be poisonous?”.

[0063] Test corpus:

[0064] Test Example 1: ["How to record a kitten", "How to anger"]

[0065] Implementation:

[0066] (I) The embodiment of the present invention first extracts the dialogue skeleton ("black flying ants", "bitten", "poisonous") according to the dialogue and the target assembly sentence, and uses the training corpus as input to train the skeleton extraction model to extract the dialogue skeleton.

[0067] (ii) In the embodiment of the present invention, under the guidance of the dialogue skeleton in (i), the skeleton-guided generator is trained to generate an assembled sentence "Will the bite of a black flying ant be poisonous?" according to the dialogue.

[0068] (iii) Sample multiple candidate assembly sentences from the model output of the skeleton-guided generator and train a skeleton-aware ranker to rank the best assembly sentence first.

[0069] (IV) The embodiment of the present invention first extracts the dialogue skeleton in the test corpus to obtain ("angry", "kitten").

[0070] (V) The embodiment of the present invention generates a plurality of candidate assembled sentences using a skeleton-guided generator under the guidance of a dialogue skeleton, namely [“how to record a kitten”, “how to irritate a kitten”, “how to irritate a cat”, “how to record a cat”].

[0071] (VI) The embodiment of the present invention uses a skeleton-aware sorter to evaluate each candidate assembled sentence and outputs the sentence with the highest score, “How to irritate a kitten”.

[0072] The above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. A person skilled in the art may modify or make equivalent substitutions for the technical solutions of the present invention without departing from the spirit and scope of the present invention. The protection scope of the present invention shall be subject to the claims.

Claims

1. A dialogue assembly method based on skeleton guidance, characterized in that: The following steps are involved: Mark the last sentence in the original conversation, use the encoder of the pre-trained RoBERTa, BERT or BART model to encode the original conversation and the last sentence in the original conversation, and use a binary classifier to determine whether each encoded word is key information. If so, extract it as a skeleton word and combine the extracted skeleton words into a conversation skeleton; The components in the original dialogue and the skeleton words in the dialogue skeleton are combined in different ways, and multiple candidate assembled sentences are obtained through multi-step sampling; Calculate the probability of generating each candidate assembled sentence based on the original dialogue skeleton as a fluency feature; The number of words that do not belong to the skeleton words in each candidate assembled sentence is counted as the number of irrelevant words, and the negative value of the number of irrelevant words is used as the semantic consistency feature between the candidate assembled sentence and the original dialogue; Each candidate assembled sentence is concatenated with the original dialogue, and the concatenated sentence is input into the classifier. The classifier predicts the probability based on the key information contained in the original dialogue skeleton, and uses the predicted probability as the information coverage feature; By combining the fluency feature, semantic consistency feature and information coverage feature, each candidate assembled sentence is evaluated to obtain the best assembled sentence.

2. The method according to claim 1, characterized in that Use the Top-k sampling algorithm for multi-step sampling.

3. The method according to claim 1, characterized in that Input the concatenated sentence into a classifier based on a pre-trained RoBERTa, BERT, or BART model.

4. A dialogue assembly device based on skeleton guidance, characterized in that: include: The skeleton extraction model is used to extract skeleton words covering key information from the original dialogue and combine the extracted skeleton words into a dialogue skeleton; The skeleton extraction model includes an encoder and a binary classifier based on the pre-trained RoBERTa, BERT or BART model. The encoder encodes the original conversation and the last sentence in the original conversation. The binary classifier determines whether each encoded word is key information. If so, it is extracted as a skeleton word. The skeleton extraction model is trained by optimizing the cross entropy loss. The skeleton-guided generator is used to combine the components in the original dialogue with the skeleton words in the dialogue skeleton in different ways through a large-scale pre-trained language model that can generate fluent natural language, and obtain multiple candidate assembled sentences through multi-step sampling; A skeleton-aware sorter that computes the probability of generating each candidate assembled sentence from the original dialogue skeleton using a large-scale pre-trained language model that can generate fluent natural language, as a fluency feature; The number of words that do not belong to the skeleton words in each candidate assembled sentence is counted as the number of irrelevant words, and the negative value of the number of irrelevant words is used as the semantic consistency feature between the candidate assembled sentence and the original dialogue; each candidate assembled sentence is spliced ​​with the original dialogue, and the spliced ​​sentence is input into the classifier. The classifier predicts the probability based on the key information contained in the skeleton of the original dialogue, and the predicted probability is used as the information coverage feature; each candidate assembled sentence is evaluated by combining the fluency feature, the semantic consistency feature and the information coverage feature to obtain the best assembled sentence.

5. The device according to claim 4, characterized in that The skeleton-guided generator uses a Top-k sampling algorithm to sample a predetermined number of candidate assembly sentences from the output distribution.

6. The device according to claim 4, characterized in that Large-scale pre-trained language models include BERT, GPT-2, or T5.

7. The device according to claim 4, characterized in that Input the concatenated sentence into a classifier based on a pre-trained RoBERTa, BERT, or BART model.

8. The device according to claim 4, characterized in that The skeleton extraction model is trained with the training corpus to extract the dialogue skeleton. The extracted dialogue skeleton is used to train the skeleton-guided generator to generate candidate assembled sentences based on the dialogue. The skeleton-aware sorter is trained based on the generated candidate assembled sentences to obtain the optimal assembled sentence.

Images