An automated text summarization evaluation method based on pre-trained language model and information theory

Abstract

The present invention provides an automatic text summarization evaluation method based on a pre-trained language model and information theory, including: using a pre-trained language model to calculate the probability of semantic units based on input text and generated summaries; using information theory to calculate information content for semantic units; The unit information is summed to obtain the total information of the text abstract; the mutual information is used to calculate the correlation between the input text and the abstract; the abstract redundancy is modeled by subtracting the total information of the abstract from the maximum information content; the total information, The weighted average of correlation and redundancy is used as a comprehensive evaluation index. At the same time, a corresponding system, terminal and storage medium are provided. The present invention uses the pre-trained language model to assist information theory to more accurately estimate text probability and calculate the amount of text information, and the created three automatic indicators of information amount, relevance, and redundancy are more in line with human evaluation standards and can be used to replace manual evaluation , to reduce the evaluation cost of the automated summarization system.

Description

technical field [0001] The present invention relates to the technical field of natural language processing, in particular to an automatic text summarization evaluation method based on a pre-trained language model and information theory, and provides a corresponding system. Background technique [0002] Text summarization is an important means to quickly acquire knowledge from massive text information, and it is becoming more and more important in the era of information explosion. The design and learning of automated text summarization systems largely depend on the accuracy of evaluation metrics. A good evaluation quality should be able to reflect human's subjective feelings about the abstract. [0003] There are currently two evaluation methods: one is manual evaluation, and the other is automated evaluation that simulates manual evaluation. Human evaluation is the gold standard for summarization evaluation techniques, for example, the recall-oriented pyramid evaluation me...

AI Technical Summary

Problems solved by technology

However, these automated evaluation methods require manually written reference summaries when evaluating system summaries, which greatly limits their usage scenarios.

[0004] In addition, there are great differences between automated evaluation methods and artificial subjective feelings, and there are still great differences in indicators such as information and relevance.

Images