A method for automatically generating an abstract by fusing discourse information
By combining the UniLM model and the BERT vocabulary with a self-attention mechanism, and introducing the prior probability distribution of discourse information, the problem of insufficient accuracy in generative summarization is solved, and more accurate and coherent summarization is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- UNIV OF ELECTRONICS SCI & TECH OF CHINA
- Filing Date
- 2022-07-11
- Publication Date
- 2026-06-09
AI Technical Summary
Existing generative text summarization methods are insufficient in accuracy, especially when integrating textual information.
We use the UniLM pre-trained model combined with the BERT vocabulary and self-attention mechanism for data cleaning and training. We introduce the prior probability distribution of the text information and generate summaries by fusing the probability vectors through Softmax and translation scaling layers.
It improves the accuracy of generative summaries, ensures the coherence and accuracy of summaries, and overcomes the problems of incoherent summaries and difficulty in word count control in existing technologies.
Smart Images
Figure CN115186088B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of automatic abstract generation, and more specifically, to a method and system for automatic Chinese abstract generation based on NLP technology. Background Technology
[0002] Text summarization refers to the process of extracting, summarizing, or refining key information from a text or collection of texts using various techniques to summarize and present the main content or gist of the original text. Because text summarization technology, especially generative text summarization, involves deep levels of natural language processing (natural language understanding, natural language generation, etc.), it has always been a challenging area of research in natural language processing. Currently, commonly used text summarization techniques can be mainly divided into two categories: extractive text summarization and generative text summarization. Extractive text summarization, as the name suggests, extracts one or several sentences from a document or collection of documents to form a summary. This is done by calculating the scores of sentences in the document, with higher scores indicating greater importance. The summary is then composed of several high-scoring sentences, and its length depends on the compression ratio. The advantage of this approach is its simplicity and practicality, and it doesn't completely deviate from the original document. However, despite its advantages, it may also have disadvantages such as incoherent generated summaries, difficulty in controlling word count, and unclear main points of the target sentences. In fact, it could be said that the quality of the summary depends entirely on the original text. Generative text summarization avoids this problem. Instead of simply using words or phrases from the original document to create a summary, it extracts the main ideas and expresses them in different ways. While it can reuse phrases and sentences from the original document to convey its main points, the summary ultimately needs to be written in the author's own words. Generative summarization methods utilize natural language understanding techniques to analyze the syntax and semantics of the original document, then fuse the information and generate a new text summary using natural language generation techniques. Summary of the Invention
[0003] The main objective of this invention is to provide an automatic summary generation method that integrates textual information, so as to improve the accuracy of generative summaries.
[0004] To achieve the above objectives, according to one aspect of the present invention, a method for automatically generating summaries that integrate textual information is provided, comprising the following steps:
[0005] Step 1: Using the Weibo summary dataset, based on the UniLM pre-trained model, perform data cleaning and divide the data into training and test sets;
[0006] Step 2: First, using a conventional method, obtain the original probability vector y = (y1, y2, ..., y|V|), where |V| is the total number of words in the vocabulary. After applying Softmax, the probability of y is obtained, as shown in the following formula:
[0007]
[0008] Step 3: Introduce the prior probability distribution of text information;
[0009] Step 4: Sum the two probability distribution vectors and take the average to obtain the final probability vector;
[0010] Step 5: Use the final probability vector to generate a summary.
[0011] In step 2, the method for obtaining the original probability vector using a conventional approach is as follows:
[0012] First, using the BERT vocabulary and nn.Embedding(), word embeddings are performed on the original text of the training data to obtain the word vector corresponding to each word.
[0013] Secondly, the training data is input into the UniLM model for fine-tuning. Through the self-attention mechanism, the model learns sentence information, the relationship between words, the key points of the sentence, and the relationship between the original text and the summary.
[0014] Finally, the last step of the model outputs the Logits of the position to be predicted. After processing them with Softmax, the probability vector can be obtained.
[0015] In step 3, the scheme of introducing prior distribution is to first perform one-hot encoding of the vocabulary for the chapter to be summarized, and then translate and scale this 0 / 1 vector to obtain the vector of Logits, which is the prior distribution of the information of the chapter. Attached Figure Description
[0016] Figure 1 This is a flowchart illustrating an automatic summary generation method that integrates textual information, provided by the present invention.
[0017] Figure 2 This is a schematic diagram illustrating the fusion of original probability and prior probability provided by the present invention;
[0018] Figure 3 This is a schematic diagram of the data cleaning process provided by the present invention. Detailed Implementation
[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0020] The following is combined Figure 1 This invention describes the method for automatically generating summaries that integrate chapter information.
[0021] This invention provides a method for automatically generating summaries that integrate textual information, including but not limited to the following steps:
[0022] First, the text data is preprocessed, such as... Figure 3 As shown. Specifically:
[0023] The text data used is a Chinese microblog news dataset, totaling 1 million entries. The data mainly consists of two parts: the original text and a summary, in .csv format. Data preprocessing mainly includes: removing spaces, removing emoticons, converting between Traditional and Simplified Chinese, converting between Chinese and English punctuation, and formatting each entry into a uniform format. The primary method used is regular expressions. In the Traditional to Simplified Chinese conversion step, OpenccPy, a Python tool for converting between Traditional and Simplified Chinese, needs to be installed using the command: `pip install openccpy`.
[0024] By following the steps above, the original Chinese data can be cleaned, which will facilitate subsequent word segmentation, pre-trained model learning, and sentence vector generation, and will enable better acquisition of semantic information and avoid semantic interruptions caused by spaces. The different encoding forms of Traditional Chinese and Simplified Chinese will increase the learning workload and degrade the learning effect and the generation of summaries. Emojis and HTML identifiers will increase the difficulty of semantic understanding as noise, and the differences between Chinese and English symbols will cause problems in reading and writing subsequent JSON files.
[0025] Next, the cleaned text data needs to be converted into the standard JSON file format. For the Weibo news dataset, the original text needs to be enclosed in English double quotes and preceded by "src_text", and the summary needs to be enclosed in English double quotes and preceded by "tgt_text". The original text and summary are separated by English commas, and each data entry is identified by "{}".
[0026] For example:
[0027] {"src_text":"Everyone eats vegetables, but most people overlook their nutritional value. Habits such as storing vegetables for too long, not eating cooked dishes immediately, cooking over low heat, and cutting vegetables before washing them can all lead to significant nutrient loss. Here are 9 common mistakes you might have.","tgt_text":"Health Taboos: 9 Common Misconceptions About Eating Vegetables, How Many Do You Know?"},
[0028] After data cleaning, the coding process begins.
[0029] Since the UniLM model used in this scheme is based on the pre-trained language model BERT, with the addition of a mask mechanism, and is obtained using three pre-training tasks (the three pre-training tasks are: bidirectional language model training, unidirectional language model training, and sequence2sequence-based language model training), the word embedding stage still uses the sum of the three word embeddings from BERT, as follows:
[0030] First, each character in the sentence is encoded to obtain a character vector (or word vector), which is the Token embedding layer. The resulting vector is called Word Embedding. Second, the position of each character in the sentence is encoded to obtain a position vector, which is the Position embedding layer. The resulting vector is called Positional Embedding. Third, the source of the sentence is encoded to distinguish whether the sentence is the original text or a summary. If it is the original text, it is encoded as 1, and if it is a summary, it is encoded as 0. This is the Segment embedding layer. The resulting vector is called Segment Embedding.
[0031] After obtaining the three sets of vectors, they are superimposed to obtain the final word vectors of the input UniLM.
[0032] The resulting word vectors will be fed into the Transformer Encoder Block, where attention is calculated to capture key information and relationships between words in the sentence.
[0033] First, the model performs a linear transformation on the input word vectors, generating three vectors: query, key, and value, which are the search vector, content vector, and query target vector, respectively. Then, we calculate the attention vector based on the query, key, and value, using the following formula:
[0034]
[0035] Where Q, K, and V are query, key, and value vectors, respectively, and are in matrix form due to batch computation;
[0036] Softmax is a normalization exponential function. It is an activation function used in multi-class classification problems, where class membership is required if there are more than two class labels. For any real vector of length K, the Softmax function can compress it into a real vector of length K, with values in the range [0,1], and a sum of 1 for all elements. The formula is as follows:
[0037]
[0038] The square root of the dimension of the word embedding.
[0039] Since a single calculation of the attention mechanism cannot fully capture all the information in a sentence, the attention mechanism needs to be calculated many times, which is called a multi-head attention mechanism. The formula is as follows:
[0040] MultiHead(Q,K,V)=Concat(head1,…,head h W O
[0041]
[0042]
[0043] In this invention, h = 8.
[0044] After calculating the attention mechanism, the result is input into the Layer Normalization layer, where a residual connection is added. This is then fed into the Feed Forward layer, and finally back into the Layer Normalization layer, where another residual connection is added. The purpose is to prevent neural network degradation when there are many layers; the residual connections act as "highways," directly transmitting data to the next layer.
[0045] When the data is passed to the last layer, we fuse the original distribution and the prior distribution of the text information, such as... Figure 2 As shown. First, the Logits of the positions to be generated are retained, which is a y = (y1, y2, ..., y|V|) of size |V|, where |V| is the total number of words in the vocabulary.
[0046] Define a new class for One-Hot encoding of documents. After One-Hot encoding, for each document, we obtain a 0 / 1 vector X = (X1, X2, ..., X...) of size |V|.|V| ), where X i =1 means the word appeared in the passage; otherwise, X i =0.
[0047] Then, consider a 0 / 1 vector X = (X1, X2, ..., X...) |V| After passing through a translation and scaling layer, we get:
[0048]
[0049] Where s and t are the training parameters, the vector is then averaged with the original y before performing Softmax, as shown in the following formula:
[0050]
[0051]
[0052] There are two details to note here:
[0053] First, the introduction of prior knowledge cannot incorporate future knowledge. That is, one cannot iterate through the vocabulary of the articles whose summaries are to be generated and then add it to the original vocabulary to enrich it. This would render the test set meaningless, and for this reason, this method still suffers from the OOV (Out of Context) problem.
[0054] Second, translation and scaling can be implemented using fully connected layers, where the parameters are learnable.
[0055] After completing the above steps, the vectors can be trained in the UniLM model.
[0056] After training, Beam Search is used for decoding. The pseudocode is as follows:
[0057]
[0058]
[0059] Each step of Beam Search maintains a k-max heap (B in the pseudocode), then generates a new sequence from the k highest-scoring sequences of the previous step, adds it to the max heap B, and selects the k sequences with the highest scores at the current time. The `score` in the pseudocode is the scoring function, typically the log-likelihood.
[0060]
[0061] After Beam Search decoding, we can obtain k sets of predicted sequences with scores from high to low. The sequence with the highest overall score is the final predicted sequence.
Claims
1. A method for automatically generating summaries that integrate textual information, characterized in that, Specifically, the steps include the following: Step 1: Using the Weibo summary dataset, based on the UniLM pre-trained model, perform data cleaning and divide the data into training and test sets; Step 2: First, obtain the original probability vector of the location to be predicted using a conventional method. The BERT vocabulary and nn.Embedding() are used to embed words into the original text and summary of the training data. The embedded vectors are then input into the UniLM model for fine-tuning. A self-attention mechanism is used to learn semantic relationships within the text. The final step of the model outputs the Logits of the position to be predicted, which is the original vector y. It is the total number of words in the vocabulary. After applying Softmax, the probability is obtained, as shown in the following formula: ； Step 3: Introduce the prior probability distribution of the text information in the following way: Perform one-hot lexical encoding on the text to be summarized, obtaining a 0 / 1 vector X=( This vector is input into a translation scaling layer containing learnable scaling coefficients s and translation coefficients t, and then processed using the formula... The transformation yields a Logits vector, which represents the prior probability distribution of the text information. Step 4: Sum the two probability distribution vectors and take the average to obtain the final probability vector; Step 5: Use the final probability vector to generate a summary.
2. The method as described in claim 1, characterized in that, In step 2, the method for obtaining the original probability vector using a conventional approach is as follows: First, using the BERT vocabulary and nn.Embedding(), word embeddings are performed on the original text of the training data to obtain the word vector corresponding to each word; Secondly, the training data is input into the UniLM model for fine-tuning. Through the self-attention mechanism, the model learns sentence information, the relationship between words, the key points of the sentence, and the relationship between the original text and the summary. Finally, the last step of the model outputs the Logits of the position to be predicted. After processing them with Softmax, the probability vector can be obtained.