An automatic text abstraction method based on a pre-training language model

Abstract

The invention provides an automatic text abstraction method based on a pre-training language model. According to the method, an ultra-large-scale unsupervised Chinese corpus is used for training a complex deep language model; the low-layer network structure of the model can extract and retain the grammar and structure information of a text, and the high-layer network structure can extract and retain the semantics and context information of the text, so that the richer text features and semantic information are provided for an automatic text summary task; a pre-training language model and an Encoder are combined to realize, the text features and the semantic information in the pre-training language model are fully utilized, so that a better semantic compression effect is provided, and the performance of an automatic text abstract is improved; a pre-training language model and a decoder are combined, not only the semantics in an original text are considered in the text generation process, but also the semantic information of vocabularies is also considered, so that the readability of the generated text and relevance with the original text are improved, and the performance of an automatic text abstract is improved.

Description

technical field [0001] The invention relates to a text processing method, and mainly relates to an automatic text summarization method based on a pre-trained language model. Background technique [0002] The task of automatic text summarization is to take a certain length of text as input and a shorter length of text as output, and the output text can express the core meaning of the input text. Automatic text summarization is an important research direction of natural language processing. There are two main methods. One is extractive automatic summarization. Find the sentences closest to the main idea of ​​the full text in the original text, and extract them as a summary of the full text; the other type is generative automatic summarization, that is, after the system reads the original text, on the basis of understanding the meaning of the whole article, press certain The method generates a piece of text word by word or word by word to express the central idea of ​​the orig...

AI Technical Summary

Problems solved by technology

[0007] (1) Summary often requires a more complex model to learn the meaning in the text and generate a summary. The number of parameters is large, and the relevant large-scale high-quality training corpus is very scarce, making it difficult for the model to converge to a better performance.

[0008] (2) Chinese high-quality automatic text summary training corpus is very scarce. Although there is a Weibo corpus released by Harbin Institute of Technology, the quality is not good. There are a lot of low-quality corpus, and the abstract has nothing to do with the original content.

[0009] (3) The readability of generative abstracts is often not good. Different words often repeat meaninglessly, some words have no connection between them, and some abstracts are even completely unreadable.

Images