A text summary generation method and system

Abstract

The present invention relates to a method and system for generating a text summary. The method comprises: preprocessing the text; extracting keywords from the text; pre-training a class label classifier using the BERTopic method to extract the class label of the text; ranking the sentences in the text according to the keywords and class labels, and extracting the top-ranked key sentences; re-sorting the text in combination with the position information of the key sentences in the text, and outputting a summary according to the sorting result. The technical solution provided by the present invention is based on the information bottleneck technology in information theory, and automatically generates a smooth summary for a long text by integrating keyword extraction, clustering and representation learning of a pre-trained language model. It is not only effective for long documents in any field, including academic articles, government reports, etc., but also does not require manual construction of training corpus, and is independent of the language type of the text.

Description

Technical Field

[0001] The present invention relates to the field of text processing, and in particular to a text summary generation method and system. Background Art

[0002] With the rapid development of natural language processing, automatic text summarization has been applied in various scenarios, such as news title generation, academic abstract writing, etc. A reliable text summarization model requires multiple elements: accurate text representation, context analysis, text structure information extraction, and fluent natural language generation.

[0003] With the development of digital information technology, more and more literature retrieval institutions hope to use computers to automatically summarize the text information in published papers. Existing text technologies are divided into two types: one is extractive text summarization, which ignores the structural information of the input text, extracts the key sentences of the full text after dividing the text into sentences, and the typical algorithm is TextRank; the other is generative summarization, which uses a sequence-to-sequence training method to construct large-scale chapter summary training pairs, relying on the Transformer model for end-to-end training, and the typical models are T5 and BART.

[0004] The extractive method transforms text summarization into an information retrieval task, sorting all the sentences in the input text and selecting the top-ranked key sentences as the final summary. However, the summary output by this method is less fluent, and the number of these key sentences is difficult to control, which depends largely on the length of the input text.

[0005] Generative solutions are highly dependent on sequence-to-sequence training models. There are two main disadvantages to this approach. First, it requires the construction of large-scale training corpora. Currently, most of these corpora come from the news field, and it is difficult and time-consuming to manually construct these corpora. Second, sequence-to-sequence models cannot model long texts well. Currently, the Transformer encoder can only support 4096 words of input at most. It is difficult to train and learn longer text inputs using the Transformer model. Summary of the invention

[0006] In order to overcome the problems existing in the related art, the present invention provides a text summary generation method and system.

[0007] According to a first aspect of an embodiment of the present invention, a method for generating a text summary is provided, comprising:

[0008] Preprocess the text;

[0009] extracting keywords from the text;

[0010] Pre-training a class label classifier using the BERTopic method to extract class labels of the text;

[0011] Ranking the sentences in the text according to the keywords and class labels, and extracting the top-ranked key sentences;

[0012] The key sentences are re-arranged in combination with position information of the key sentences in the text, and a summary is output according to the sorting result.

[0013] Further, the text is preprocessed, specifically including:

[0014] Split all sentences in the input text and keep only those sentences whose length is greater than the preset length.

[0015] Further, the extracting keywords from the text specifically includes:

[0016] The RAKE algorithm is used for keyword extraction, and all sentences are segmented based on stop words and punctuation marks to extract all entity vocabulary phrases.

[0017] Constructing a co-occurrence frequency matrix of the entity vocabulary phrases;

[0018] Calculating the key value of each entity vocabulary phrase according to the co-occurrence frequency matrix;

[0019] The top-ranked keywords are extracted from the entity vocabulary phrases by sorting the key values.

[0020] Furthermore, the BERTopic method is used to pre-train a class label classifier to extract the class label of the text, specifically including:

[0021] Fine-tune the pre-trained language model BERT on the text;

[0022] Use the fine-tuned BERT to vectorize the text;

[0023] Use the UMAP dimensionality reduction algorithm to reduce the dimensionality of all text vectors, and then use the DBSCAN algorithm to cluster the text vectors after dimensionality reduction;

[0024] According to the clustering results, the words whose tfidf values ​​exceed the preset threshold under each cluster are extracted as the class labels of each cluster.

[0025] Further, the ranking of the sentences in the text according to the keywords and class labels, and extracting the top-ranked key sentences, specifically includes:

[0026] Circularly determine the number of keywords contained in each sentence in the text and perform a smoothing operation to obtain a first probability value, input the sentence into the class label classifier to obtain a second probability value corresponding to the correct class label, and multiply the first probability value and the second probability value to obtain a selection value of the sentence;

[0027] Extract the sentences with the highest selection values ​​as key sentences.

[0028] Further, the step of re-arranging the key sentences in the text in combination with the position information of the key sentences, and outputting a summary according to the sorting result, specifically includes:

[0029] Slide from the sentence at the front to the sentence at the back, and use BERT to calculate the sentence coherence value between every two positions. This coherence value is the subsidiary output result of the next sentence prediction strategy in BERT;

[0030] Perform beam search based on all coherent values ​​to obtain different search paths;

[0031] Sort the beam search results and select the top-ranked sentence paths;

[0032] The corresponding sentences are output according to the selected sentence paths as a summary of the text.

[0033] According to a second aspect of an embodiment of the present invention, a text summary generation system is provided, including:

[0034] A text preprocessing module, used for preprocessing text;

[0035] A keyword extraction module, used to extract keywords from the text;

[0036] A class label extraction module, used to pre-train a class label classifier using a BERTopic method to extract the class label of the text;

[0037] A key sentence extraction module, used to rank the sentences in the text according to the keywords and class labels, and extract the top-ranked key sentences;

[0038] The summary output module is used to re-arrange the key sentences in combination with the position information of the key sentences in the text, and output a summary according to the sorting result.

[0039] Furthermore, the class label extraction module is specifically used for:

[0040] Fine-tune the pre-trained language model BERT on the text;

[0041] Use the fine-tuned BERT to vectorize the text;

[0042] Use the UMAP dimensionality reduction algorithm to reduce the dimensionality of all text vectors, and then use the DBSCAN algorithm to cluster the text vectors after dimensionality reduction;

[0043] According to the clustering results, the words whose tfidf values ​​exceed the preset threshold under each cluster are extracted as the class labels of each cluster.

[0044] Furthermore, the key sentence extraction module is specifically used for:

[0045] Circularly determine the number of keywords contained in each sentence in the text and perform a smoothing operation to obtain a first probability value, input the sentence into the class label classifier to obtain a second probability value corresponding to the correct class label, and multiply the first probability value and the second probability value to obtain a selection value of the sentence;

[0046] Extract the sentences with the highest selection values ​​as key sentences.

[0047] Furthermore, the summary output module is specifically used for:

[0048] Slide from the sentence at the front to the sentence at the back, and use BERT to calculate the sentence coherence value between every two positions. This coherence value is the subsidiary output result of the next sentence prediction strategy in BERT;

[0049] Perform beam search based on all coherent values ​​to obtain different search paths;

[0050] Sort the beam search results and select the top-ranked sentence paths;

[0051] The corresponding sentences are output according to the selected sentence paths as a summary of the text.

[0052] The technical solution provided by the embodiments of the present invention may have the following beneficial effects:

[0053] The present invention is based on the information bottleneck technology in information theory, and automatically generates a fluent summary for a long text by integrating keyword extraction, clustering and representation learning of a pre-trained language model. It is effective for long documents in any field, including academic articles and government reports, and does not require manual construction of training corpus, and is independent of the language type of the text.

[0054] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The above and other objects, features and advantages of the present invention will become more apparent through a more detailed description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, wherein like reference numerals generally represent like components throughout the exemplary embodiments of the present invention.

[0056] Figure 1 The figure is a flowchart of a method for generating a text summary according to an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0057] The preferred embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present invention more thorough and complete, and to fully convey the scope of the present invention to those skilled in the art.

[0058] The terms used in the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "the" and "the" used in the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0059] It should be understood that although the terms "first", "second", "third", etc. may be used to describe various information in the present invention, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0060] The present invention aims to automatically generate summaries for long texts (such as scientific research documents, government reports, etc.). In view of the shortcomings of the current extractive summaries with low accuracy and poor reliability of sequence-to-sequence generative summaries, the present invention starts from the perspective of information theory, integrates the four technologies of multi-angle information bottleneck theory, keyword extraction, language model and clustering, and compresses and extracts multiple input long texts to output fluent summaries that conform to the original text.

[0061] The present invention is based on the multi-angle information bottleneck theory. The core of the information bottleneck theory is to squeeze information out of a bottleneck, remove the noise input data containing irrelevant details, and only retain the features most relevant to the general concept. Transferring this principle to long text summarization means removing those irrelevant sentences or phrases and retaining only the sentences most relevant to the core idea of ​​the original text. The present invention is based on the multi-angle information bottleneck theory, and its formula is as follows:

[0062]

[0063] Where D represents the input long text, S represents the summary to be output, t represents the key sentence that can cover the original text information, Y1 and Y2 represent the two angles of the input long text (keywords and article class labels), and I represents mutual information. The main idea conveyed by the formula is: by retrieving the key sentences in the input text, these key sentences can contain enough keywords and effectively predict the class labels of the original text based on these key sentences. The two angles of the original text can be provided in advance or obtained through unsupervised learning methods.

[0064] The technical solutions of the embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0065] Figure 1 The figure is a flowchart of a method for generating a text summary according to an exemplary embodiment of the present invention.

[0066] See also Figure 1 , the method comprising:

[0067] 110. Preprocess the text;

[0068] Specifically, in the text preprocessing stage, all sentences in the input document may be segmented, and the sentence lengths may be screened, for example, only sentences with a length greater than 5 are retained.

[0069] 120. Extracting keywords from the text;

[0070] Specifically, this step belongs to the keyword extraction stage. The RAKE (Rapid Automatic Keyword Extraction) algorithm can be used for keyword extraction. All sentences are segmented based on stop words and punctuation marks to extract all entity vocabulary phrases. Then, a co-occurrence frequency matrix of these entity phrases is constructed. Finally, the key value of each entity vocabulary phrase is calculated based on the co-occurrence frequency matrix, and the top-ranked keywords are extracted by sorting the key values.

[0071] 130. Pre-training a class label classifier using the BERTopic method to extract the class label of the text;

[0072] Specifically, this step is the class label extraction stage, which uses the BERTopic method to extract class labels for all texts. First, the pre-trained language model BERT is fine-tuned on the target text, and then the fine-tuned BERT is used to vectorize each article, and the UMAP dimensionality reduction algorithm is used to reduce the dimension of all text vectors, and then the DBSCAN algorithm is used to cluster the text vectors after dimensionality reduction. According to the clustering results, the words with high frequency tfidf values ​​under each cluster are extracted as the class labels of each cluster.

[0073] 140. Ranking the sentences in the text according to the keywords and class labels, and extracting the top-ranked key sentences;

[0074] Specifically, this step includes two stages: text selection and fluency processing.

[0075] The text selection stage is to initially extract the top-ranked key sentences of the long text. Specifically, each sentence is looped to determine the number of keywords in step 120 in the sentence and perform a smoothing operation to obtain the keyword probability value. At the same time, the sentence is independently input into a pre-trained class label classifier in step 130 to obtain the probability value of the corresponding correct class label. The above two probability values ​​are multiplied to obtain the final sentence selection value. The sentence with a higher selection value is more likely to be used as the final summary.

[0076] The fluency processing stage is to re-rank the top-ranked sentences in the text selection stage in combination with the position information of each sentence in the original text. The BERT model fine-tuned in step 130 is used to perform fluency enhancement processing on these top-ranked sentences. The specific operation method is to slide from the top-ranked sentences to the bottom-ranked sentences in sequence, and calculate the sentence coherence value between each two positions through BERT. The coherence value is the auxiliary output result of the next sentence prediction strategy in BERT, and then perform beam retrieval based on all the coherence values ​​to obtain different retrieval paths.

[0077] 150. Re-arrange the key sentences based on their position information in the text, and output a summary based on the sorting result.

[0078] Specifically, the step is the final summary output stage, specifically, the beam search results in step 140 are sorted, the top-ranked sentence paths are selected, and these sentences are output as the final summary.

[0079] Optionally, in this embodiment, step 110 specifically includes:

[0080] Split all sentences in the input text and keep only those sentences whose length is greater than the preset length.

[0081] Optionally, in this embodiment, step 120 specifically includes:

[0082] 1201. Use the RAKE algorithm to extract keywords, segment all sentences based on stop words and punctuation marks, and then extract all entity vocabulary phrases;

[0083] 1202. Construct a co-occurrence frequency matrix of the entity vocabulary phrases;

[0084] 1203. Calculate the key value of each entity vocabulary phrase according to the co-occurrence frequency matrix;

[0085] 1204. Extract top-ranked keywords from the entity vocabulary phrases by sorting the key values.

[0086] Optionally, in this embodiment, step 130 specifically includes:

[0087] 1301. Fine-tune the pre-trained language model BERT on the text;

[0088] 1302. Perform vector representation on the text using the fine-tuned BERT.

[0089] 1303. Use the UMAP dimensionality reduction algorithm to reduce the dimensionality of all text vectors, and then use the DBSCAN algorithm to cluster the text vectors after dimensionality reduction;

[0090] 1304. According to the clustering result, extract the words whose tfidf values ​​exceed a preset threshold under each cluster as the class label of each cluster.

[0091] Optionally, in this embodiment, step 140 specifically includes:

[0092] 1401, cyclically determining the number of keywords contained in each sentence in the text and performing a smoothing operation to obtain a first probability value, and inputting the sentence into the class label classifier to obtain a second probability value corresponding to a correct class label, and multiplying the first probability value and the second probability value to obtain a selection value of the sentence;

[0093] 1402. Extract sentences with top selection values ​​as key sentences.

[0094] Optionally, in this embodiment, step 150 specifically includes:

[0095] 1501. Slide from the front sentence to the back sentence in sequence, and calculate the sentence coherence value between every two positions through BERT. The coherence value is the subsidiary output result of the next sentence prediction strategy in BERT.

[0096] 1502. Perform beam search based on all the coherent values ​​to obtain different search paths;

[0097] 1503. Sort the beam search results and select the sentence path with the highest ranking;

[0098] 1504. Output corresponding sentences according to the selected sentence paths as a summary of the text.

[0099] Corresponding to the above method embodiment, an embodiment of the present invention provides a text summary generation system, including:

[0100] A text preprocessing module, used for preprocessing text;

[0101] A keyword extraction module, used to extract keywords from the text;

[0102] A class label extraction module, used to pre-train a class label classifier using a BERTopic method to extract the class label of the text;

[0103] A key sentence extraction module, used to rank the sentences in the text according to the keywords and class labels, and extract the top-ranked key sentences;

[0104] The summary output module is used to re-arrange the key sentences in combination with the position information of the key sentences in the text, and output a summary according to the sorting result.

[0105] Furthermore, the class label extraction module is specifically used for:

[0106] Fine-tune the pre-trained language model BERT on the text;

[0107] Use the fine-tuned BERT to vectorize the text;

[0108] Use the UMAP dimensionality reduction algorithm to reduce the dimensionality of all text vectors, and then use the DBSCAN algorithm to cluster the text vectors after dimensionality reduction;

[0109] According to the clustering results, the words whose tfidf values ​​exceed the preset threshold under each cluster are extracted as the class labels of each cluster.

[0110] Furthermore, the key sentence extraction module is specifically used for:

[0111] Circularly determine the number of keywords contained in each sentence in the text and perform a smoothing operation to obtain a first probability value, input the sentence into the class label classifier to obtain a second probability value corresponding to the correct class label, and multiply the first probability value and the second probability value to obtain a selection value of the sentence;

[0112] Extract the sentences with the highest selection values ​​as key sentences.

[0113] Furthermore, the summary output module is specifically used for:

[0114] Slide from the sentence at the front to the sentence at the back, and use BERT to calculate the sentence coherence value between every two positions. This coherence value is the subsidiary output result of the next sentence prediction strategy in BERT;

[0115] Perform beam search based on all coherent values ​​to obtain different search paths;

[0116] Sort the beam search results and select the top-ranked sentence paths;

[0117] The corresponding sentences are output according to the selected sentence paths as a summary of the text.

[0118] Regarding the system in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated again here.

[0119] Example:

[0120] The process of the present invention is mainly divided into 6 steps: text preprocessing, keyword extraction, class label extraction, text selection, fluency processing and summary output.

[0121] The pseudocode flowchart corresponding to each stage is as follows, where the text preprocessing step corresponds to line 1 of the pseudocode flowchart. The keyword extraction step corresponds to line 2 of the pseudocode flowchart. The class label extraction step corresponds to line 3 of the pseudocode flowchart. The text selection step corresponds to lines 4-11 of the pseudocode flowchart. The smooth processing step corresponds to lines 12-18 of the pseudocode flowchart. The summary output step corresponds to lines 19-21 of the pseudocode flowchart.

[0122]

[0123] The present invention integrates the latest natural language processing technology, covering pre-trained language models, information extraction, topic models, classification models and multiple aspects, and generates long text summaries based on multi-angle information bottleneck technology. The main advantage is that the whole process is unsupervised and automated, and no additional manual annotation and corpus collection work is required. At the same time, the functional modules corresponding to each stage are relatively independent, and each functional module has strong internal flexibility, and the algorithm inside the module can be updated in time according to the latest natural language processing technology.

[0124] The scheme of the present invention has been described in detail above with reference to the accompanying drawings. In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments. Those skilled in the art should also be aware that the actions and modules involved in the specification are not necessarily required by the present invention. In addition, it can be understood that the steps in the method of the embodiment of the present invention can be adjusted in order, merged and deleted according to actual needs, and the modules in the device of the embodiment of the present invention can be merged, divided and deleted according to actual needs.

[0125] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or a combination of both.

[0126] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system and method according to multiple embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or code, and the part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0127] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A text summary generation method, characterized in that: include: Preprocess the text; extracting keywords from the text; Pre-training a class label classifier using the BERTopic method to extract class labels of the text; Ranking the sentences in the text according to the keywords and class labels, and extracting the top-ranked key sentences; Re-arrange the key sentences in the text according to the position information of the key sentences, and output a summary according to the sorting result; The method of pre-training a class label classifier using the BERTopic method to extract the class label of the text specifically includes: Fine-tune the pre-trained language model BERT on the text; Use the fine-tuned BERT to vectorize the text; Use the UMAP dimensionality reduction algorithm to reduce the dimensionality of all text vectors, and then use the DBSCAN algorithm to cluster the text vectors after dimensionality reduction; According to the clustering results, the words whose tfidf values ​​exceed the preset threshold under each cluster are extracted as the class labels of each cluster; The ranking of the sentences in the text according to the keywords and the class labels, and extracting the top-ranked key sentences, specifically includes: Circularly determine the number of keywords contained in each sentence in the text and perform a smoothing operation to obtain a first probability value, input the sentence into the class label classifier to obtain a second probability value corresponding to the correct class label, and multiply the first probability value and the second probability value to obtain a selection value of the sentence; Extract sentences with the highest selection values ​​as key sentences; The step of re-arranging the key sentences in the text in combination with the position information of the key sentences, and outputting a summary according to the arrangement result, specifically includes: Slide from the sentence at the front to the sentence at the back, and use BERT to calculate the sentence coherence value between every two positions. This coherence value is the subsidiary output result of the next sentence prediction strategy in BERT; Perform beam search based on all coherent values ​​to obtain different search paths; Sort the beam search results and select the top-ranked sentence paths; The corresponding sentences are output according to the selected sentence paths as a summary of the text.

2. The method according to claim 1, characterized in that The text preprocessing specifically includes: Split all sentences in the input text and keep only those sentences whose length is greater than the preset length.

3. The method according to claim 1, characterized in that The extracting of keywords from the text specifically includes: The RAKE algorithm is used for keyword extraction, and all sentences are segmented based on stop words and punctuation marks to extract all entity vocabulary phrases. Constructing a co-occurrence frequency matrix of the entity vocabulary phrases; Calculating the key value of each entity vocabulary phrase according to the co-occurrence frequency matrix; The top-ranked keywords are extracted from the entity vocabulary phrases by sorting the key values.

4. A text summary generation system, characterized in that: include: A text preprocessing module, used for preprocessing text; A keyword extraction module, used to extract keywords from the text; A class label extraction module, used to pre-train a class label classifier using a BERTopic method to extract the class label of the text; A key sentence extraction module, used to rank the sentences in the text according to the keywords and class labels, and extract the top-ranked key sentences; A summary output module, used to re-arrange the key sentences in combination with the position information of the key sentences in the text, and output a summary according to the sorting result; The class label extraction module is specifically used for: Fine-tune the pre-trained language model BERT on the text; Use the fine-tuned BERT to vectorize the text; Use the UMAP dimensionality reduction algorithm to reduce the dimensionality of all text vectors, and then use the DBSCAN algorithm to cluster the text vectors after dimensionality reduction; According to the clustering results, the words whose tfidf values ​​exceed the preset threshold under each cluster are extracted as the class labels of each cluster; The key sentence extraction module is specifically used for: Circularly determine the number of keywords contained in each sentence in the text and perform a smoothing operation to obtain a first probability value, input the sentence into the class label classifier to obtain a second probability value corresponding to the correct class label, and multiply the first probability value and the second probability value to obtain a selection value of the sentence; Extract sentences with the highest selection values ​​as key sentences; The summary output module is specifically used for: Slide from the sentence at the front to the sentence at the back, and use BERT to calculate the sentence coherence value between every two positions. This coherence value is the subsidiary output result of the next sentence prediction strategy in BERT; Perform beam search based on all coherent values ​​to obtain different search paths; Sort the beam search results and select the top-ranked sentence paths; The corresponding sentences are output according to the selected sentence paths as a summary of the text.

Images