A method for automatic generation of summaries based on kernel sampling

Abstract

This invention discloses an automatic summary generation method based on kernel sampling. Utilizing the UniLM model, it enhances semantic comprehension by incorporating contextual semantic information during original text understanding. During summary generation, a unidirectional mask mechanism ensures consistent text continuation logic. Furthermore, the UniLM model undergoes multi-task pre-training, resulting in stronger generalization capabilities. During UniLM decoding, a kernel sampling function is employed to construct a Mask matrix, allowing for random sampling of words within a finite range, primarily addressing the issue of repetitive text generation in generative summarization.

Description

Technical Field

[0001] This invention relates to the field of automatic abstract generation, and more specifically, to a method and system for automatic Chinese abstract generation based on NLP technology. Background Technology

[0002] Text summarization refers to the process of extracting, summarizing, or refining key information from a text or collection of texts using various techniques to summarize and present the main content or gist of the original text. Because text summarization technology, especially generative text summarization, involves deep levels of natural language processing (natural language understanding, natural language generation, etc.), it has always been a challenging area of ​​research in natural language processing. Currently, commonly used text summarization techniques can be mainly divided into two categories: extractive text summarization and generative text summarization. Extractive text summarization, as the name suggests, extracts one or several sentences from a document or collection of documents to form a summary. This is done by calculating the scores of sentences in the document, with higher scores indicating greater importance. The summary is then composed of several high-scoring sentences, and its length depends on the compression ratio. The advantage of this approach is its simplicity and practicality, and it doesn't completely deviate from the original document. However, despite its advantages, it may also have disadvantages such as incoherent generated summaries, difficulty in controlling word count, and unclear main points of the target sentences. In fact, it could be said that the quality of the summary depends entirely on the original text. Generative text summarization avoids this problem. Instead of simply using words or phrases from the original document to create a summary, it extracts the main ideas and expresses them in different ways. While it can reuse phrases and sentences from the original document to convey its main points, the summary ultimately needs to be written in the author's own words. Generative summarization methods utilize natural language understanding techniques to analyze the syntax and semantics of the original document, then fuse the information and generate a new text summary using natural language generation techniques. Summary of the Invention

[0003] The main objective of this invention is to provide an automatic summary generation method based on kernel sampling to solve the problem of generating duplicate text in generative summaries.

[0004] To achieve the above objectives, according to one aspect of the present invention, an automatic summary generation method based on kernel sampling is provided, comprising the following steps:

[0005] Step 1: Using the Weibo summary dataset, based on the UniLM pre-trained model, perform data cleaning and divide the data into training and test sets;

[0006] Step 2, construct the Mask matrix suitable for Seq2Seq;

[0007] Step 3: Fine-tuning the parameters of the UniLM language model;

[0008] Step 4, kernel sampling decoding:

[0009] a) Construct a kernel sampling function based on the input threshold p and the temperature parameter t;

[0010] b) Construct the kernel sampling Mask matrix based on the kernel sampling function;

[0011] c) Based on the Mask matrix, sort the probabilities of the tokens from largest to smallest, and accumulate them one by one until the sum of the probabilities is greater than the threshold p, then stop accumulating to form a set of candidate tokens.

[0012] d) Randomly sample from the set of tokens to obtain the predicted tokens;

[0013] Step 5: For the part of the summary that needs to be predicted to generate the summary, perform cyclic kernel sampling and decoding on a per-token basis to form a summary generation model;

[0014] Step 6: Input the cleaned test set into the summary generation model to obtain the summary results.

[0015] The specific Mask matrix for step 2 is as follows:

[0016] The Mask matrix is ​​a trapezoidal Mask matrix obtained by concatenating a lower triangular Mask matrix and another Mask matrix. This allows BERT to possess both reading comprehension and generative capabilities through a single Mask mechanism, without altering its fundamental architecture. Unlike purely generative models like GPT, it enables the model to simultaneously perceive bidirectional contextual information while reading the original text, resulting in stronger comprehension and inductive abilities than GPT. Furthermore, when generating summaries, it operates unidirectionally from beginning to end, preserving the fundamental logic of the generation process.

[0017] Step 4 specifically involves:

[0018] Given a distribution We have its Top-P vocabulary. , is defined as the smallest set that satisfies The smallest subset that "occupies the vast majority of probability quality" is called the nucleus. Next, the kernel sampling method (NucleusSampling) selects from the words with the highest probabilities, letting... The initial conditional probability distribution is rescaled to a new distribution:

[0019]

[0020] The specific resource method involves reshaping a probability distribution through temperature sampling, as shown in the following formula:

[0021] ,set up This causes the distribution to tilt towards high-probability events, which also implicitly reduces the proportion of the tail distribution.

[0022] Step 5 specifically involves:

[0023] For example, given a text data item "[CLS]+original text+[SEP]+China Women's Volleyball Team won the championship+[SEP]", during the iterative prediction process, each time data is input to the Decoder part of the model, the model receives the following input and output:

[0024] First time: Input: "Original text + [SEP]", Output: "Original text + [SEP] + Chinese";

[0025] Second input: "Original text + [SEP] + Chinese", output "Original text + [SEP] + China";

[0026] Third time: Input: "original text + [SEP] + China", output: "original text + [SEP] + Chinese woman".

[0027] And so on.

[0028] This operation is not achieved by constantly adjusting the input and output. Instead, the entire data "[CLS]+original text+[SEP]+China Women's Volleyball Team won the championship+[SEP]" is input into the model, and the model automatically implements this through the Mask mechanism of Seq2Seq introduced earlier. Attached Figure Description

[0029] Figure 1 This invention provides a Mask method suitable for Sequence2Sequence;

[0030] Figure 2 This is a flowchart illustrating the automatic summary generation method based on kernel sampling provided by the present invention.

[0031] Figure 3 This is a schematic diagram of the data cleaning process provided by the present invention. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this invention clearer, the following will be combined with the present invention. Figure 2The technical solutions of this invention are clearly and completely described herein. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0033] The following is combined with Figure 2 This invention describes the automatic summary generation method based on kernel sampling provided by the present invention.

[0034] This invention provides a Chinese text summarization extraction method based on a BERT pre-trained model, including but not limited to the following steps:

[0035] First, the text data is preprocessed, such as... Figure 3 As shown. Specifically:

[0036] The text data used is a Chinese microblog news dataset, totaling 1 million entries. The data mainly consists of two parts: the original text and a summary, in .csv format. Data preprocessing mainly includes: removing spaces, removing emoticons, converting between Traditional and Simplified Chinese, converting between Chinese and English punctuation, and formatting each entry into a uniform format. The primary method used is regular expressions. In the Traditional to Simplified Chinese conversion step, OpenccPy, a Python tool for converting between Traditional and Simplified Chinese, needs to be installed using the command: `pip install openccpy`.

[0037] By following the steps above, the original Chinese data can be cleaned, which will facilitate subsequent word segmentation, pre-trained model learning, and sentence vector generation, and will enable better acquisition of semantic information and avoid semantic interruptions caused by spaces. The different encoding forms of Traditional Chinese and Simplified Chinese will increase the learning workload and degrade the learning effect and the generation of summaries. Emojis and HTML identifiers will increase the difficulty of semantic understanding as noise, and the differences between Chinese and English symbols will cause problems in reading and writing subsequent JSON files.

[0038] Next, the cleaned text data needs to be converted into the standard JSON file format. For the Weibo news dataset, the original text needs to be enclosed in English double quotes and preceded by "src_text", and the summary needs to be enclosed in English double quotes and preceded by "tgt_text". The original text and summary are separated by English commas, and each data entry is identified by "{}".

[0039] After data cleaning, the coding process begins.

[0040] Because the UniLM model used in this solution adds a masking mechanism to the pre-trained language model BERT, such as... Figure 1 As shown, the word embeddings were obtained using three pre-training tasks (bidirectional language model training, unidirectional language model training, and Sequence2Sequence-based language model training). Therefore, in the word embedding stage, the three word embeddings of BERT are still added together, as follows:

[0041] First, each character in the sentence is encoded to obtain a character vector (or word vector), which is the Token embedding layer. The resulting vector is called Word Embedding. Second, the position of each character in the sentence is encoded to obtain a position vector, which is the Position embedding layer. The resulting vector is called Positional Embedding. Third, the source of the sentence is encoded to distinguish whether the sentence is the original text or a summary. If it is the original text, it is encoded as 1, and if it is a summary, it is encoded as 0. This is the Segment embedding layer. The resulting vector is called Segment Embedding.

[0042] After obtaining the three sets of vectors, they are superimposed to obtain the final word vectors of the input UniLM.

[0043] The resulting word vectors will be fed into the Transformer Encoder Block, where attention is calculated to capture key information and relationships between words in the sentence.

[0044] First, the model performs a linear transformation on the input word vectors, generating three vectors: query, key, and value, which are the search vector, content vector, and query target vector, respectively. Then, we calculate the attention vector based on the query, key, and value, using the following formula:

[0045]

[0046] Where Q, K, and V are query, key, and value vectors, respectively, and are in matrix form due to batch computation;

[0047] is the normalization exponential function. It's an activation function used in multi-class classification problems, where class membership is required if there are more than two class labels. For any real vector of length K, the Softmax function can compress it into a real vector of length K, with values ​​in the range [0,1], and a sum of 1 for all elements. The formula is as follows:

[0048]

[0049] The square root of the dimension of the word embedding.

[0050] Since a single calculation of the attention mechanism cannot fully capture all the information in a sentence, the attention mechanism needs to be calculated many times, which is called a multi-head attention mechanism. The formula is as follows:

[0051] ,

[0052]

[0053]

[0054] In this invention, .

[0055] After calculating the attention mechanism, the result is input into the Layer Normalization layer, where a residual connection is added. This is then fed into the Feed Forward layer, and finally back into the Layer Normalization layer, where another residual connection is added. The purpose is to prevent neural network degradation when there are many layers; the residual connections act as "highways," directly transmitting data to the next layer.

[0056] After completing the above steps, the vectors can be trained in the UniLM model.

[0057] After the model is trained, decoding is performed using the nucleus sampling method, as follows:

[0058] Kernel sampling methods utilize the shape of a probability distribution to determine the set of labels to sample. Given a distribution... We have its Top-P vocabulary. , is defined as the smallest set that satisfies The smallest subset that "occupies the vast majority of probability quality" is called Nucleus.

[0059] Since Nucleus Sampling is designed to solve the problem of repetitive generated text, and the probability of generating a repeated phrase increases with each repetition, it creates a negative feedback loop. This pattern is independent of context and the length of the repeated text.

[0060] make The original probability distribution will then be rescaled to a new distribution, from which the next word is extracted, i.e.:

[0061]

[0062] The test set is fed into the UniLM model to obtain the Logits of the generated data.

[0063] Given a threshold p and a temperature parameter t, construct a kernel sampling function. The temperature sampling formula is as follows:

[0064]

[0065] Construct a Mask matrix for kernel sampling. Based on the Mask matrix, sort the probabilities of the tokens from largest to smallest and accumulate them one by one until the sum of the probabilities exceeds the threshold p. Stop accumulating and mask all subsequent tokens to form an updated set of Tokenlogits.

[0066] The predicted token is obtained by randomly sampling from the set of tokens.

[0067] The experimental parameters are as follows:

[0068] Network setup: 24 Transformer layers, 1024 hidden sizes, 16 attention heads;

[0069] Parameter size: 340M;

[0070] Initialization: Initialize using Bert-Large parameters;

[0071] Activation function: GELU;

[0072] Dropout ratio: 0.1;

[0073] Weight decay factor: 0.01;

[0074] Batch size: 16;

[0075] p: 0.9;

[0076] Dataset: Weibo news dataset, 250,000 articles.

[0077] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for automatic summary generation based on kernel sampling, characterized in that, Specifically, the steps include the following: Step 1: Using the Weibo summary dataset, based on the UniLM pre-trained model, perform data cleaning and divide the dataset into training and testing sets; Step 2, construct the Mask matrix suitable for Seq2Seq; Step 3: Fine-tuning the parameters of the UniLM language model; Step 4, kernel sampling decoding: a) First, based on the input temperature parameter t, using the formula... Reshape the original probability distribution of the token; b) Secondly, based on the input threshold p, define the vocabulary of Top-p. To meet the conditions The smallest vocabulary set; c) Then, let The initial conditional probability distribution is rescaled to a new distribution. , among which when hour, Otherwise, it is 0, thus constructing the kernel sampling function; d) Finally, sort the tokens according to the probability corresponding to the new distribution from largest to smallest and accumulate them one by one to form a set of candidate tokens, and randomly sample from them to obtain the predicted token; Step 5: For the part of the summary that needs to be predicted to generate the summary, perform cyclic kernel sampling and decoding on a per-token basis to form a summary generation model; Step 6: Input the cleaned test set into the summary generation model to obtain the summary results.

2. The method for automatic summary generation based on kernel sampling according to claim 1, characterized in that: The trapezoidal Mask matrix in step 2 is obtained by concatenating a lower triangular Mask matrix and another Mask matrix, so that the model can see bidirectional information of context when reading the original text, and performs unidirectional generation from front to back when generating the summary.

3. The method as described in claim 1, characterized in that, Step 5 specifically involves inputting the entire text data to be processed into the model. The model uses the Mask mechanism of Seq2Seq to utilize the generated tokens as input for subsequent decoding during the cyclic prediction process until a complete summary text is generated.

Images