A text summarization method based on semantic matching and improved cross-encoding model

Abstract

The application designs a text summarization method based on semantic matching and an improved cross coding model, and belongs to the field of natural language processing; candidate summaries are acquired, and are sorted in descending order based on a ROUGE evaluation index; a matching model is designed and trained to score the candidate summaries, the candidate summary with the highest score is selected as the final output of the text summarization model, and a sorting loss function is used to constantly optimize the matching model, so that the model can identify the most matched candidate summary; the application follows a two-stage framework, improves the matching stage from multiple aspects, fully acquires fine-grained semantic interaction information between the candidate summaries and the document, so that the semantic similarity of the candidate summaries and the document can be more accurately calculated in the matching stage, and experiments show that the application has good performance in the text summarization task.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of natural language processing, and particularly relates to a text summarization method based on semantic matching and an improved cross-encoding model. BACKGROUND

[0002] Automatic text summarization can quickly summarize key information from redundant texts by machine understanding of text meaning. In practical applications, text summarization technology can be used for generation of news headlines, generation of academic literature summaries, generation of product review summaries, etc. In addition, it is also a technical support for question and answer systems and search engine systems. Generative summarization is more difficult because it contains more text underlying processing methods, such as text understanding, sentence rewriting, synonym conversion, etc. However, it is more promising because it is more in line with the way of manually writing summaries. In particular, with the advent of deep learning and large-scale datasets, generative summarization has become a research focus in the field of automatic text summarization. Generative text summarization is mainly based on the Seq2Seq framework with attention mechanism, and can use RNN, CNN, Transformers to build models. In recent years, with the advent of pre-training tasks, more and more text summarization models using pre-trained language models have been proposed. Through the use of the language capabilities of pre-trained language models, the text summarization task has made a breakthrough.

[0003] However, the traditional generative model uses beam search algorithm to generate a summary, which can easily lead to non-optimal output results. In addition, since the model is trained in a teacher-forcing manner under maximum likelihood estimation, there is a problem of inconsistency between the target function and the evaluation index. The training target is the token-level loss of the generated summary and the reference summary, and the evaluation index is the global similarity between the generated summary and the document. The two-stage framework of "generating" first and "matching" second alleviates the above two problems. The SimCLS model follows the two-stage framework paradigm and improves the performance of the text summarization task to a new level. The "generation stage" adjusts the beam width of the pre-trained summary model to allow the model to generate multiple candidate summaries. The "matching stage" uses text matching technology to match the candidate summaries and the document to find the most relevant candidate summary as the final output.

[0004] The matching stage of the SimCLS model simply encodes the candidate summary and the source document to obtain a single vector representation of each, and then calculates the semantic similarity of the two vectors as the final score. The encoding method is difficult to obtain the complex interaction features between the document and the candidate summary. And compressing each text into a single vector is not enough to represent the rich semantic features of the text, which will lead to poor matching results. SUMMARY

[0005] In view of the deficiencies of the prior art, the present application proposes a text summarization method based on semantic matching and an improved cross-encoding model, which splices the candidate summary and the document into a long text, then uses the same encoder to obtain the interaction features of the two, and differentiates between different candidates, performs semantic correction on the candidate summary and the document in the model, fully obtains the fine-grained semantic interaction information between the candidate summary and the document, so as to more accurately calculate the semantic similarity of the candidate summary and the document in the matching stage.

[0006] A text summarization method based on semantic matching and an improved cross-encoding model, specifically comprising the following steps:

[0007] Step 1: adopt a generation model g(·), which is a sequence-to-sequence model, and train it by minimizing the negative log-likelihood of the generated summary S and the reference summary ;

[0008] Step 2: use the model g(·) pre-trained in step 1 to generate multiple candidate summaries S i , S i ∈S1,...,S n , where n is the number of candidate summaries; n is an integer greater than or equal to 1;

[0009] Step 3: sort the generated multiple candidate summaries in descending order according to the ROUGE evaluation index ;

[0010] Step 4: design a matching model h(·) to score different candidate summaries, then select the candidate summary with the highest score as the final output of the text summarization model;

[0011] Step 4.1: use the cross-encoder in the semantic matching task to cross-encode the public dataset Chinese document D and the candidate summary S i ; first splice the document D and the candidate summary S i into a single long text; then feed it into the semantic interaction model t(·), and take the embedding vector of the first word [cls] as the final representation vector of the candidate-document after encoding by the model t(·); the semantic interaction model t(·) is an encoder based on the Transformer structure, and its parameters are initialized using the pre-trained self-attention model RoBERTa, which is used to encode S i and D, and finally obtains the similarity score of S i and D;

[0012]

[0013]

[0014]

[0015] where the first word [cls] is an extra input for the semantic interaction model to learn the interaction representation information between the candidate and the document, and [sep] is a separator between different texts; the Pooling(·) function selects the first vector as the final representation information; the linear layer W1 transforms each vector into a scalar, which is used to represent the similarity between the document D and the candidate summary S i

[0016] Considering the essential difference between the candidate summary matching and the semantic matching task, two strategies are proposed to assist the training of the matching model h(·):

[0017] The first strategy: since the multiple candidate summaries generated in step 2 are obtained from the same document, they have high similarity. In order to let the matching model h(·) obtain the difference between different candidate summaries, the relevant representation information of each candidate summary S i is directly used as additional information to estimate the semantic matching score, and the part of the output related to the candidate summary S i encoded by the semantic interaction model t(·) is used to calculate the representation information, which enhances the final output result.

[0018]

[0019]

[0020] where the AvgPooling(·) function divides the candidate-document encoding representation into two groups, corresponding to S i and D, respectively, and then performs average pooling on all representation vectors in each group to obtain the embedding representation information of each group; W2 is another linear layer used to transform the embedding representation information of the candidate summary into a scalar as an additional score; the additional score is reweighted by the hyperparameter a and added to the main score to produce the final score:

[0021] h(S i , D) = a h1(S i , D) + (1-a) h2(S i )

[0022] The second strategy: the candidate summaries generated in the generation stage are highly condensed from the document, so theoretically the semantics of each candidate summary S i and the document D are very similar; based on this, a helper function is designed to enhance the candidate summary S​​​i and the semantic similarity of document D; the embedding representation information of Si and D obtained by average pooling in the first strategy v D The semantic correction process is performed to further enhance their semantic consistency by maximizing their similarity:

[0023]

[0024] where the Similarity(·) function adopts the cosine similarity loss function to approximate and v D the semantic representation information calculated by the correction model;

[0025] Step 4.2: Select the candidate summary with the highest score as the final summary S generated by the text summary model:

[0026] S = argmax h1(S i , D)

[0027] Step 5: In the above training process, use the ranking loss function L rank to learn the ranking of the real evaluation indicators:

[0028]

[0029] where reflects the difference between the reference summary and the candidate summary S i , ∑ i ∑ j＞i max(0, h(D, S j )-h(D, S i )+λ ij ) is used to increase the gap between different candidate summaries; candidate summaries S1,..., S n are ranked in descending order according to the ROUGE indicator, and λ ii is a hyperparameter used to adjust the difference between different candidate summaries S i and S j ; add the auxiliary loss function L cos-sim to the main loss function L rank to form the final loss function:

[0030] L(Θ) = L rank + L cos-sim

[0031] By using the above loss function, the training of the matching model h(·) proposed in step 3 is completed, so that the model can identify the most matching candidate summary.

[0032] The present application has beneficial technical effects:

[0033] The present application proposes a new two-stage text summary model, which considers semantic interaction in the second stage and designs a new matching model to capture the fine-grained (token-level) interaction between the candidate summary and the document in the matching stage.

[0034] The present application proposes a new two-stage text summary model. The model considers the fine-grained interaction between the candidate summary and the document, and designs two strategies to enhance the ability of the interaction module to distinguish multiple candidates and constrain the semantic consistency of the candidate and the document. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 The present application is a text summary method based on semantic matching and improved cross-encoding model flowchart;

[0036] Figure 2 The present application is a text summary method based on semantic matching and improved cross-encoding model architecture diagram;

[0037] Figure 3 The present application is a comparison between the architecture of the present method and the prior art simCLS. Figure a is simCLS, and figure b is the present application;

[0038] Figure 4 The present application is an experimental result diagram on the CNN / Daily Mail dataset;

[0039] Figure 5 The present application is an experimental result diagram on the XSum dataset. DETAILED DESCRIPTION

[0040] The present application will be further described below in conjunction with the drawings and examples;

[0041] A text summary method based on semantic matching and improved cross-encoding model follows a two-stage framework, in which the matching stage is improved from multiple aspects, Figure 1 is the technical solution flowchart of the present application, Figure 2 which shows the overall structure of the present application, specifically including the following steps:

[0042] Step 1: using the generation model g(·), which is a sequence-to-sequence model, to train it by minimizing the negative log-likelihood of the generated summary S and the reference summary ;

[0043] Step 2: using the model g(·) pre-trained in step 1 to generate multiple candidate summaries S i i S n , S i 1,..., S i n, where n is the number of candidate summaries; n takes an integer greater than or equal to 1, and experiments show that n = 16 has the best performance;

[0044] Step 3: sorting the generated multiple candidate summaries in descending order according to the ROUGE evaluation index ;

[0045] Step 4: designing a matching model h(·) to score different candidate summaries, and then selecting the candidate summary with the highest score as the final output of the text summary model; Figure 3 is the structure comparison between the present invention and the existing model SimCLS in the matching stage.

[0046] Step 4.1: using the cross-encoder model in the semantic matching task to cross-encode the public dataset CNN / DM or XSum Chinese document D and the candidate summary S i ; first, concatenate the document D and the candidate summary S i into a single long text; then feed it into the semantic interaction model t(·), and take the embedding vector of the first word [cls] as the final representation vector of the candidate-document after encoding by the model t(·); the semantic interaction model t(·) is an encoder based on the Transformer structure, and its parameters are initialized using the pre-trained self-attention model RoBERTa, which is used to encode S i and D, and finally get the similarity score of S i and D;

[0047]

[0048]

[0049]

[0050] where the first word [cls] is an additional input to let the semantic interaction model learn the interaction representation information of the candidate-document, and [sep] is a separator between different texts; the Pooling(·) function selects the first vector as the final representation information; the linear layer W1 vectors transformed into a scalar to represent the similarity between the document D and the candidate summary S i ;

[0051] Considering the essential difference between the candidate summary matching and the semantic matching task, two strategies are proposed to assist the training of the matching model h(·):

[0052] The first strategy: since multiple candidate summaries generated in step 2 are obtained from the same document, they have high similarity. In order to let the matching model h(·) obtain the difference between different candidate summaries, the relevant representation information of each candidate summary S i is directly used as additional information to estimate the semantic matching score, and the semantic interaction model t(·) is used to encode the part of the output related to the candidate summary S i to calculate the representation information, and to enhance the final output result;

[0053]

[0054]

[0055] wherein the AvgPooling(·) function divides the candidate-document encoding representation into two groups, corresponding to Si and D respectively, and then performs average pooling on all representation vectors in each group, aiming to obtain the embedding representation information of each group; W2 is another linear layer used to transform the embedding representation information of the candidate summary into a scalar as an additional score; the additional score is reweighted by the hyperparameter a and added to the main score to produce the final score:

[0056] h(S i , D) = a h1(S i , D) + (1-a) h2(S i )

[0057] The second strategy: the candidate summaries generated in the generation stage are highly condensed from the document, so theoretically the semantics of each candidate summary S i and the document D are very similar; based on this, a helper function is designed to enhance the semantic similarity between the candidate summary S i and the document D; the embedding representation information of Si and D obtained by average pooling in the first strategy v D is subjected to semantic correction processing to further enhance their semantic consistency by maximizing their similarity:

[0058]

[0059] where the Similarity(·) function adopts cosine similarity loss function to approximate and v D representations, the semantic representation information calculated by the correction model;

[0060] Step 4.2: Select the candidate summary with the highest score as the final generated summary S of the text summary model:

[0061] S = argmax h1(S i , D)

[0062] Step 5: In the above training process, use the ranking loss function L rank to learn the ranking of the real evaluation indicators:

[0063]

[0064] where reflects the difference between the reference summary and the candidate summary Si, ∑ i ∑ j＞i max(0, h(D, S j )-h(D, S i )+λ ij ) is used to increase the gap between different candidate summaries; candidate summaries S1,..., S n are ranked in descending order according to the ROUGE indicator, and λ ij is a hyperparameter used to adjust the difference between different candidate summaries S i and S j ; add the auxiliary loss function L cos-sim to the main loss function L rank to form the final loss function:

[0065] L(Θ) = L rank + L cos-sim

[0066] By using the above loss function, the training of the matching model h(·) proposed in the matching stage is completed, so that the model can identify the most matching candidate summary.

[0067] The ROUGE indicator is used to evaluate the performance of the model. The F1 scores of ROUGE-1, ROUGE-2 and ROUGE-L are reported. The pre-trained BART model is used as the generation model in the generation stage, because it is the most advanced generative summary model at present and is widely used. The beam width of the generation model is set to 16, so that it generates 16 candidate summaries. The parameters of the semantic interaction model t(·) are initialized using the 'roberta-base' version of RoBERTa in the matching stage. The Adam optimizer is used, and the learning rate is set to:

[0068]

[0069] wherein warmup is set to 10000. Due to the limited length of input, the length of source document is set to 392 in CNN / Daily Mail dataset (candidate summary length limit is 120) and 432 in XSum dataset (candidate summary length limit is 80).

[0070] The candidate summaries are generated using the same generation model as SimCLS on CNN / Daily Mail dataset, but the enhanced matching model is used to score and select the final output summaries. The experimental results are shown in Table 2. Figure 4 As shown in Table 2, the SimCLS model is superior to all other text summarization models because it adopts a two-stage method to solve the inconsistency between the training target and the evaluation indicators; the present application is 0.50, 0.42 and 0.49 better than the SimCLS model on ROUGE-1, ROUGE-2 and ROUGE-L respectively, reaching the state-of-the-art performance, proving that the fine-grained interaction between the source document and the candidate summary helps to better measure their similarity.

[0071] Further experiments are conducted on XSum dataset. XSum is a highly summarized dataset with only one sentence as the summary. The experimental results are shown in Table 3. Figure 5 As shown in Table 3, similar to the results on the CNN / Daily Mail dataset, the pre-training effect of the SimCLS model is better than all other methods, again verifying the effectiveness of the two-stage method. The present application improves the performance of the SimCLS model by 0.24, 0.20 and 0.18 on ROUGE-1, ROUGE-2 and ROUGE-L respectively, also reaching the new state-of-the-art performance.

Claims

1. A text summarization method based on semantic matching and improved cross-encoding model, characterized in that, Specifically comprising the following steps: Step 1: Training the generative model by minimizing the negative log-likelihood of the generated summaries and reference summaries ; Step 2: Pre-training the model with step 1 Generating a plurality of candidate summaries where n is the number of candidate summaries; n takes an integer greater than or equal to 1; Step 3: The generated plurality of candidate summaries are ranked in descending order according to the ROUGE evaluation metric in descending order. Step 4: Design a matching model score different candidate summaries respectively, and then select the candidate summary with the highest score as the final output of the text summary model; Step 4.1: Cross-encode the document and candidate summaries using a cross-encoder model in the semantic matching task. and the candidate summaries are cross-encoded; first the document and the candidate summaries are concatenated into a single long text; Then feed it into the semantic interaction model In the middle, after the model Take the first word after encoding The embedding vector is used as the final representation vector of the candidate document; the semantic interaction model It is a Transformer-based encoder that uses a pre-trained self-attention model, RoBERTa, to initialize its parameters for encoding. and Finally obtained and Similarity score; Step 4.1 is specifically: ； ； ； where the first word is an extra input, aiming to let the semantic interaction model learn the interaction representation information of the candidate-document, is a separator between different texts; the function selects the first vector as the final representation information; linear layer vectors into a scalar, to represent the document and candidate summaries similarity; Considering the essential difference between the candidate summary matching and the semantic matching task, two strategies are proposed to assist the matching model training; The first strategy is specifically: Since the multiple candidate summaries generated in step 2 are obtained from the same document, they have high similarity between each other. In order to let the matching model obtain the difference between different candidate summaries, the relevant representation information of each candidate summary is directly used as additional information to estimate the semantic matching score, and the semantic interaction model is used to encode the part of the output calculation representation information related to the candidate summary to enhance the final output result; ； ； where The function will encode the candidate-document into two groups, respectively corresponding to and Then, average pooling is done for all the representation vectors in each group, aiming to obtain the embedding representation information of each group respectively; is another linear layer to transform the embedding representation information of the candidate summary into a scalar as an additional score; through the hyperparameter The additional score is reweighted and added to the main score to produce the final score: ； The second strategy is specifically: The candidate summaries generated during the generation phase are highly condensed from the documents, so theoretically each candidate summary... semantics and documents The candidate summaries are very similar; based on this, an auxiliary function is designed to enhance the candidate summaries. and documents Semantic similarity; for the first strategy, the results obtained through average pooling. and Each embedded representation information , Semantic correction is performed to further enhance their semantic consistency by maximizing their similarity: ； wherein The function adopts a cosine similarity loss function to approximate and The representation of the correction model calculates semantic representation information; Step 4.2: select the candidate summary with the highest score as the final generated summary S of the text summary model; Step 5: In the above training process, use the ranking loss function Learn the ranking way of the real evaluation index, complete the training of the matching model proposed in step 3 , so that the model can identify the most matching candidate summary. 2.The text summarization method based on semantic matching and improved cross-encoding model according to claim 1, characterized in that, Step 4.2 Final generated summary . 3.The text summarization method based on semantic matching and improved cross-encoding model according to claim 1, characterized in that, Step 5 is specifically: ； wherein reflects the difference between the reference abstract and the candidate abstracts , to increase the gap between different candidate abstracts; candidate summaries are ranked in descending order of ROUGE score, is a hyper-parameter to adjust the difference between different candidate summaries and ; the auxiliary loss function is added to the main loss function to form the final loss function: ； By using the above loss function, the training of the matching model proposed in step 3 is completed, enabling the model to identify the most matching candidate summary. ​

Images