Generative dialogue abstract method fusing dual-granularity knowledge fine tuning

Abstract

The invention discloses a generative dialogue abstract method fusing dual-granularity knowledge fine tuning, and relates to the technical field of text abstracts.The method is characterized by comprising the following steps that 1, original dialogue data is labeled with a DialoGPT model, redundant sentences in the original dialogue data are labeled, and subject division is conducted; 2) screening the most critical dialogue for each topic of each dialogue based on the marked topic, and rewriting to obtain an enhanced dialogue; 3) generating a group of entity-level common knowledge and a group of paragraph-level common knowledge based on the standard abstract of each group of dialogues; and 4) inputting enhanced dialogue data into the pre-trained language model, and supervising fine tuning of the model by using the standard abstract and the two groups of common knowledge. And 5) completing a dialogue abstract generation task by using the pre-training language model after fine tuning. According to the method, semantic features of dialogue data can be remarkably enhanced, dialogue semantics are deeply understood based on the features, and therefore the fluency and accuracy of the generative dialogue abstract are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of text summarization, and in particular to the technical field of generative dialogue summarization based on a pre-trained language model. Background Art

[0002] Currently, with the rapid development of Internet technology, text data is showing an explosive growth trend, covering various forms such as news, reports, blogs, academic papers, and daily conversations. In this context, it has become particularly important and urgent to efficiently extract core information from massive texts. Text summarization technology has emerged as the times require. It can generate a refined version of the original text, accurately capture and convey the main idea and core points of the article (paragraph).

[0003] At present, pre-trained language models based on the Transformer structure have been widely used. Models based on the encoder-decoder architecture and their improvements have made significant progress in the field of generative summarization and have also been widely used in the field of dialogue summarization. The existing generative dialogue summarization methods usually use the dialogue structure to design multiple views to extract dialogue features, or manually construct knowledge graphs to assist the model in generation.

[0004] The drawback of the generative dialogue summarization methods in the prior art is that many generative summary models only focus on the spoken features in the dialogue data. When external knowledge is introduced into the dialogue, it is difficult for the first few layers of encoders to understand the relevant knowledge, which easily wastes the performance of the first few layers of encoders. Some generative models ignore the topic jumping phenomenon in the dialogue data, the diversity of expressions, and the knowledge of different granularities contained in the dialogue, making it difficult to achieve better performance.

[0005] As mentioned in the review, there is an urgent need for a generative dialogue summarization method that can better integrate knowledge to address the shortcomings of existing methods and improve the quality of generated summaries. Summary of the invention

[0006] The purpose of the present invention is to provide a generative dialogue summarization method that integrates dual-granularity knowledge fine-tuning. First, the dialogue data is annotated and enhanced based on the topic at the data level, the original dialogue data is divided into several topics, and a variety of expressions and rich semantic expressions are formed on each topic. Secondly, at the sentence understanding level, entity-level common sense knowledge and paragraph-level common sense knowledge are introduced respectively, multi-decoder multi-task training is performed, and external knowledge is injected into the model to assist the model to better understand these expressions.

[0007] The above technical objectives of the present invention are achieved through the following technical solutions: A generative dialogue summarization method integrating dual-granularity knowledge fine-tuning, comprising the following main steps:

[0008] 1) Use the DialoGPT model to annotate the original dialogue data, mark its redundant sentences and divide them into topics;

[0009] 2) Based on the annotated topics, the most critical dialogues are selected for each topic of each conversation, and rewritten to obtain enhanced dialogue data;

[0010] 3) Generate a set of entity-level and paragraph-level common sense knowledge based on the standard summary of each dialogue to assist the pre-trained language model in understanding the semantics of various dialogues;

[0011] 4) Input the enhanced dialogue data into the pre-trained language model, and use the standard summary and the generation of the two sets of common sense knowledge supervision models mentioned above, and use the negative log-likelihood loss to fine-tune the pre-trained language model to make it more suitable for the dialogue summary generation task;

[0012] 5) Use the fine-tuned pre-trained language model to complete the conversation summary generation task.

[0013] In summary, the present invention has the following beneficial effects: it can annotate and enhance the conversation data based on themes at the data level, form diversified expressions on each theme, and introduce external knowledge at the sentence understanding level to assist the model to better understand these expressions. This can significantly improve the performance of the generative conversation summary task and improve the quality of the generative conversation summary. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a flow chart in an embodiment of the present invention;

[0015] Figure 2 It is a schematic diagram showing the main operation process in an embodiment of the present invention;

[0016] Figure 3 Schematic diagram of multi-task learning in an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The following is combined with Figure 1-3 The present invention is described in further detail.

[0018] Example: A generative dialogue summarization method integrating dual-granularity knowledge fine-tuning, such as Figure 1 , Figure 2 and Figure 3 As shown, the following steps are included:

[0019] 1) For the original set of conversation data D = {u1,u2,...,u n}, the dialogue data annotated by the DialoGPT model is D'={(u1,RD),(u2,RD)...,(u i ,RD),TS(u i+1,RD)...(u j ,RD)TS(u j+1 ,RD)...,(u n ,RD)}, where RD is a redundancy marker, indicating whether the dialogue is a redundant sentence, 0 represents a non-redundant sentence, 1 represents a redundant sentence, and TS is a topic division marker. The dialogues separated by TS belong to different topics;

[0020] 2) Based on the annotated dialogues, the dialogue data D' is divided into several topics according to the TS tags, and the key dialogues of each topic are selected for rewriting to obtain enhanced dialogues D*, in which sentences marked as redundant are not included in the enhancement.

[0021] Preferably, the key dialogues are screened by calculating the LIX score, which depends on the total number of sentences, word length and number of complex words in the dialogue. That is, the higher the LIX score, the more content the dialogue states, the more complex words are used, and the more expressive information it contains.

[0022] The calculation formula of LIX score is as follows, where ω is the total number of words in the dialogue round, s is the total number of sentences in the dialogue round, and l is the number of long words, that is, the number of words with a length of more than 6 characters in the dialogue round.

[0023]

[0024] According to the LIX scores of each dialogue round, the dialogue with the highest score for each topic is rewritten to obtain the enhanced dialogue D*;

[0025] 3) Based on the standard summary, two knowledge generation models, COMET and PARACOMET, are used to generate two sets of external inference knowledge of the corresponding summaries. During fine-tuning, each set of external knowledge is used as multi-task generation supervision. The former can understand and generate information related to specific entities by learning a large amount of common sense knowledge, which we call entity-level knowledge. The latter, based on the former, dynamically stores and updates paragraph-level information during the generation process, and can generate coherent common sense inferences from the narrative, which we call paragraph-level knowledge.

[0026] 4) Add two decoders D to the pre-trained language model C1 (Entity Decoder), D C2 (ParaDecoder), which is used to learn and generate the two external knowledge mentioned in step 3). In the fine-tuning stage, each piece of data includes a dialogue, a summary, and the entity-level knowledge and paragraph-level knowledge corresponding to the summary;

[0027] The main task of the fine-tuning model is the summary generation task. Preferably, the BART model is used as the pre-trained language model. First, the dialogue data D * Encoded into a vector, the decoder is then used to generate the target summary S, while the model learns the optimal parameters θ and minimizes the negative log-likelihood loss as shown below:

[0028]

[0029] Where N is the total number of samples in each batch, S i is the i-th word in the summary generated by the decoder, and T is the length of the summary, that is, the total number of words, S i,<t Indicates summary S i All words before the tth word in P(S i,t |S i,<t, D * ) is the conditional probability of the model generating the current word in a sample by inputting the dialogue data and the previously generated words;

[0030] The two auxiliary tasks of the fine-tuned pre-trained language model are the generation of common sense knowledge at two different granularities, and the model is supervised by the two sets of knowledge at different granularities mentioned in step 3).

[0031] Using the above two target knowledge, two knowledge summary decoders D are trained C1 , D C2 To minimize the two negative log-likelihood losses shown below, and learn the optimal parameters θ on each task DC1 ,θ DC2 :

[0032]

[0033] Finally, the above loss functions are linearly combined to obtain the final loss function as shown below:

[0034] L total =λ a L main +λ b L cs1 +(1-λ a -λ b )L cs2

[0035] Where L main , L cs1 , L cs2 They are the main task loss, entity-level knowledge generation loss, paragraph-level knowledge generation loss, and λ a , b are the hyperparameters that control the weights of the main task loss and entity-level knowledge generation loss, respectively.

[0036] 5) Use the above fine-tuned pre-trained language model to complete the generative dialogue summarization task, that is, given a dialogue with n words: D = {u1,..., u n}, the goal is to generate a summary with m words: T = {s1,..., s m}, where m << n, and the generated summary can better summarize the main content of the original dialogue. When generating the dialogue summary, only use the summary generation encoder for output, while the knowledge generation encoder does not participate in the output during the summary generation task.

[0037] The following are the verification experiments of this embodiment:

[0038] This method was compared on two public datasets, SAMSum and DialogSum, and the statistical data of the two datasets are shown in Table 1.

[0039] Table 1 Data Statistics

[0040]

[0041] The implementation details are as follows:

[0042] For the SAMSum dataset, the learning rate during fine-tuning is set to 2e-5, dropout is set to 0.1, and linear warmup is set to 300 training steps, and fine-tuning is performed for 15 epochs on the training sets before and after SAMSum augmentation; for the DialogSum dataset, the learning rate during fine-tuning is set to 2e-5, dropout is set to 0.15, and linear warmup is set to 400 training steps, and fine-tuning is performed for 20 epochs on the training sets before and after DialogSum augmentation. The fine-tuned pre-trained language model was compared with existing generative dialogue summarization methods, including Transformer, BART(D (ALL) ), LA-BART, SICK++, and the results are shown in Tables 2 and 3.

[0043] Table 2 Experimental Comparison on SAMSum Dataset

[0044]

[0045] Table 3 Experimental Comparison on DialogSum Dataset

[0046]

[0047] Experimental results show that the proposed model outperforms most existing models on different datasets. In addition, the summaries generated by the proposed method achieve excellent results in the bigram recall (R-2) score on both datasets, further demonstrating the effectiveness of the proposed method in improving the quality of generated dialogue summaries.

[0048] The above verification experiment proves that rewriting key sentences based on the topic in the conversation enriches the semantic expression. On this basis, two sets of common sense knowledge with different granularity of summaries are generated, which enables the pre-trained model to further learn the characteristics of the conversation data, thereby optimizing the generation of summaries. The verification experiment tested our method on two open source datasets. Compared with the existing generative summarization methods, this method can significantly improve the quality of generated summaries.

[0049] This specific embodiment is merely an explanation of the present invention and is not a limitation of the present invention. After reading this specification, those skilled in the art may make non-creative modifications to the present embodiment as needed. However, as long as they are within the scope of the claims of the present invention, they are protected by the patent law.

Claims

1. A generative dialogue summarization method integrating dual-granularity knowledge fine-tuning, specifically comprising the following steps: 1) For the original set of conversation data D = {u1,u2,...,u n }, after being annotated by the DialoGPT model, we get a new set of dialogue data D'={(u1,RD),(u2,RD)...,(u i ,RD),TS(u i+1 ,RD)...(u j ,RD)TS(u j+1 ,RD)...,(u n ,RD)}, where RD is a redundancy marker, indicating whether the dialogue is a redundant sentence, 0 represents a non-redundant sentence, 1 represents a redundant sentence, and TS is a topic division marker. The dialogues separated by TS belong to different topics; 2) Based on the annotated dialogues, the dialogue data D' is divided into several topics according to the TS tags, and the key dialogues of each topic are selected for rewriting to obtain enhanced dialogue data D*; 3) Generate a set of entity-level knowledge and a set of paragraph-level knowledge based on the standard summary of each set of dialogue data to assist the pre-trained language model in understanding the semantics in the dialogue data. The entity-level knowledge is used to understand and generate information related to a specific entity, while the paragraph-level knowledge dynamically stores and updates the paragraph-level information based on the entity-level knowledge to generate coherent common sense inferences from the paragraph. 4) The enhanced dialogue data D* is input into the pre-trained language model, and the pre-trained language model is fine-tuned on the generation task using the standard summary and entity-level knowledge and paragraph-level knowledge; 5) Use the above fine-tuned pre-trained language model to complete the generative dialogue summarization task, that is, given a set of dialogue data D = {u1, u2,..., u n}, the goal is to generate a summary T = {s1, s2..., s m}, where m << n, and the generated summary can better summarize the main content described in the original dialogue data.

2. The generative dialogue summarization method integrating dual-granularity knowledge fine-tuning according to claim 1 is characterized by: The specific screening method for key conversations in step 2) is: The LIX score of each dialogue extracted above is calculated. The score depends on the total number of sentences, word length and number of complex words in the dialogue. A high LIX score means that the dialogue round contains more content, more complex words and more information. The dialogue with the highest LIX score for each topic is then used as the key dialogue to obtain the enhanced dialogue D*.

3. The generative dialogue summarization method integrating dual-granularity knowledge fine-tuning according to claim 1 is characterized by: In step 4), three tasks are used to fine-tune the pre-trained language model. The main task is the standard summary generation task, and the two auxiliary tasks are knowledge generation tasks of different granularities, namely the entity-level knowledge generation task and the paragraph-level knowledge generation task. The generation of the model is then supervised using standard summaries and commonsense knowledge of two different granularities.

Images