Text summarization method based on deep learning

Abstract

The present invention discloses a method for obtaining text summarization based on deep learning. The method first extracts keywords from the original document; constructs an Encoder module to extract global semantic information; constructs a graph convolution module to extract local semantic information; and constructs a Decoder module to generate a text summary. The text summary task is to refine and summarize massive text data, and save the time cost of users browsing text data by compressing massive text data into simple and intuitive summaries. The method uses key points as local features and original text as global features to obtain rich semantic representations of the original text; the premise of generating high-quality summaries is to understand the semantics of the original text; graph convolution is used to update the weights between features, further promote the transmission of semantic information, and suppress the transmission of meaningless messages, so that the semantic information of the original text obtained can better reflect the central idea of ​​the original text, thereby ensuring that the generated summary can reflect the center of the original text and avoid generating a summary without a central idea.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and in particular to a method for obtaining text summaries based on deep learning. Background Art

[0002] With the rapid development of the Internet industry, more and more people rely on publishing and obtaining information from Internet platforms. The text information people come into contact with on a daily basis has exploded. Through the Internet platform, a large amount of information can be quickly accessed. However, due to the extremely large and messy information on the Internet, people need to spend more time to filter the key information in the text. Therefore, extracting important content from a large amount of text information has become an urgent need for people. Traditional text summarization mainly relies on manual summarization, which requires huge time and labor costs. At the same time, due to the explosive growth of text information, it is impractical to rely solely on manual summary of text summaries. Therefore, automatic text summarization, as a technology that automatically summarizes text summaries through machines, has become a hot field that is currently being actively studied.

[0003] Automatic text summarization can be divided into two categories according to the output type, namely extractive text summarization and generative text summarization. Extractive text summarization extracts important fragments from the original text and combines them to form a summary. It can not only effectively make the content concise and easy for people to understand, but also is simple to implement. It is currently the most mainstream, most widely used and easiest method. However, this method has a disadvantage that cannot be ignored. In the text summary, adjacent fragments are not necessarily adjacent in content, so it may cause the semantic incoherence of the summary. In contrast, generative text summarization is not just extracting a few existing fragments from the original text, but a compressed interpretation of the main content of the original text. It may produce vocabulary that is not in the original text. Compared with abstract text summarization, it is more flexible and closer to the process of manual summary. Generative text summarization requires understanding the original document and generating a concise summary with strong readability, which makes the task difficult and challenging. Automatic text summarization can be divided into single document summary and multi-document summary according to the document type. Single document summary generates a summary from a given document, and multi-document summary generates a summary from a given set of topic-related documents. With the rapid development of artificial intelligence technology, natural language processing technology based on neural networks and deep learning has also made remarkable progress. As an important field of natural language processing, automatic text summarization has also received widespread attention. More and more researchers are committed to using deep neural networks to achieve automatic text summarization. Generative text summarization technology has made substantial progress to a certain extent. At the same time, extractive text summarization technology has also been greatly improved. Although automatic text summarization technology has made great progress in recent years, it is still far from enough to generate high-quality summaries. For generative text summarization, the model is required to have a stronger ability to represent, understand and generate text. At present, generative text summarization still has problems such as readability, redundancy, amount of information and false information. Summary of the invention

[0004] The task of text summarization is to extract and summarize massive amounts of text data, and to save users the time cost of browsing text data by compressing massive amounts of text data into simple and intuitive summaries. As people are exposed to more and more text information in their daily lives, text summarization has become an urgent need for people. Automatic text summarization is an important field in natural language processing, which aims to automatically summarize concise, coherent, informative and accurate text summaries through machines. With the development of deep learning, text summarization technology has made progress to a certain extent. However, this technology is far from enough to meet people's actual needs. For computers, generating summaries is a very challenging task. When generating summaries, computers are required to understand the content after reading the original text, and to select, cut and splice the content according to the importance, and finally generate a fluent short text.

[0005] In order to solve the quality problem of generated text summaries, the technology used in this invention proposes a method of fusing local information and global features to strengthen the semantic representation of the input text by the model, thereby improving the quality of summary generation. The method includes the following steps:

[0006] Step 1: Extract keywords from the original text.

[0007] For text, the theme of the entire text can often be glimpsed through some keywords. The present invention extracts several keywords representing the semantic content of the article as local information of the text. Since there are no given keywords in the data set related to text summarization, the present invention first needs to extract the keywords of the original document. The method used in the present invention is mainly based on the idea of ​​unsupervised.

[0008] The steps to extract keywords from the original text are as follows:

[0009] Step 1.1 considers the position information of the word. Generally, the words that appear in the first and last sentences are more likely to be keywords. Therefore, we repeat the first and last sentences of the document three times respectively to increase the frequency of keywords in the first and last sentences.

[0010] Step 1.2: Segment the text into words and use the tf-idf statistics of each word to select 20 words as candidate keywords.

[0011] In step 1.3, we hope that the keywords obtained can represent the central idea of ​​the original text as much as possible. Keywords obtained only by using statistical information cannot guarantee this. Therefore, we need to further screen the keywords obtained in step 1.2: use Doc2Vec to obtain the vector representation d of the document, and use Word2Vec to obtain the vector representation w of the candidate keywords. Sort the candidate keywords according to the cosine distance between w and d, and select key phrases close to the document from the initial candidate keywords. The closer the keyword is to the document, the greater the amount of information, thus ensuring that the obtained keywords are more relevant to the document.

[0012] Step 1.4 is to avoid redundancy of the final keywords, that is, the extracted keywords have the same meaning although they have different expressions, so we need to perform a secondary screening on the keywords obtained in step 1.3: Similarly, we sort the candidate keywords according to the cosine distance between them, and for keywords with the same semantics, we only retain one.

[0013] Step 2: Build the Encoder module.

[0014] The purpose of this module is to encode the input text, that is, to represent it in vector form. The Encoder module of the present invention uses the encoder module of the Transformer to finally obtain the semantic representation of the original text with semantic features and context features, which we call global semantic information.

[0015] Step 3: Build a graph convolution module.

[0016] The purpose of this module is to integrate the relationships of local semantic information. In step 1, the semantic information of different keywords was obtained. In order to mine more effective local semantic features, we use the graph convolution method to add local features to relational features, thereby obtaining local semantic information with relational information. In graph convolution, the input includes nodes and adjacency matrices, where nodes are the local semantic information extracted in step 1. There are relationships between nodes, and the adjacency matrix represents the degree of relationship between nodes. Then, graph convolution is used to adaptively learn the relationship weights between each keyword. After obtaining the adjacency matrix between keywords, it is multiplied by the initial semantic information to obtain relational features, and then the relational features are fused with the initial features to obtain a new round of features.

[0017] The following steps are involved:

[0018] Step 3.1 obtains K local semantic information and 1 global semantic information in steps 1 and 2 as nodes of the graph.

[0019] Step 3.2 constructs the adjacency matrix of the graph and initializes it to 1.

[0020] Step 3.3 The greater the difference between the local features and the overall features, the more outliers there are. Therefore, the present invention calculates the difference between the local semantic information and the global semantic information, constructs a difference matrix, and uses the difference to dynamically update the weights of the edges of all nodes in the graph. First, the global semantic information is repeated K times to obtain the difference between K local semantic information and the global semantic information, and finally obtains the difference matrix.

[0021] Step 3.4 uses linear transformation and other operations to convert the difference matrix obtained in 3.3 into a matrix with dimension (K, K), which is called the update matrix.

[0022] Step 3.5 multiplies the update matrix obtained in step 3.4 by the adjacency matrix element by element. The purpose of this operation is to use the update matrix to adaptively learn the adjacency matrix.

[0023] Step 3.6 multiplies the adjacency matrix obtained in 3.5 by the node information to obtain the relational features of the semantic information.

[0024] Step 3.7 concatenates the local relationship features obtained in 3.6 with the local semantic information of the node to obtain local semantic information with relationship information.

[0025] Step 4: Build the Decoder module

[0026] The purpose of this module is to generate a summary of the original text. The pointer generator network is a seq2seq model with a copy mechanism, which predicts words based on the probability distribution of the generator and the pointer. The generator mainly uses the background vector output by the encoder module, the hidden layer of the current step of the decoder, and the output of the previous step prediction of the decoder to predict the vocabulary of the current step. The summary predicted by the generator is the words in the vocabulary, and can predict words outside the original document. The words predicted by the probability distribution of the pointer are the text in the original document pointed to by the pointer. Therefore, the summary generated by the pointer generator network can both generate new vocabulary and copy the text in the original document. The pointer generator network can be regarded as a balance between the extraction method and the abstraction method. By copying words, the accuracy and processing power of unregistered words are improved, while the ability to generate new words is retained. The present invention uses an RNN with an attention mechanism as a decoder to output the summary. The specific steps are as follows:

[0027] Step 4.1: The global semantic information obtained in steps 2 and 3 and the local semantic information with relationship information are fused by summing up and used as the initial hidden vector of the decoder module.

[0028] Step 4.2 calculates the importance of each word in the original text to the Decoder word according to the attention mechanism. For each step of the Decoder, the semantic representation of the original text with different attention information is obtained according to different importance, which is called the background vector.

[0029] Step 4.3 predicts the output of the current time step based on the semantic representation of the original text, the output of the previous time step, and the hidden vector of the current time step, and finally obtains the predicted output of each time step, thereby obtaining the text summary of the original text.

[0030] Compared with the prior art, the present invention has the following advantages:

[0031] (1) This method uses key points as local features and the original text as global features, obtaining a richer semantic representation of the original text. The premise of generating high-quality summaries is to understand the semantics of the original text.

[0032] (2) Meaningful features are closer to global features than meaningless features. This method uses graph convolution to update the weights between features, further promoting the transmission of semantic information and suppressing the transmission of meaningless messages. The semantic information of the original text obtained can better reflect the central idea of ​​the original text, thereby ensuring that the generated summary can reflect the central idea of ​​the original text and avoiding the generation of a summary without a central idea. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a flow chart of a specific implementation mode of the present invention. DETAILED DESCRIPTION

[0034] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0035] The flowchart of the implementation method is as follows Figure 1 As shown, the following steps are included:

[0036] Step S10, extracting keywords from the original document;

[0037] Step S20, constructing an Encoder module to extract global semantic information;

[0038] Step S30, constructing a graph convolution module to extract local semantic information;

[0039] Step S40, constructing a Decoder module to generate a text summary.

[0040] Step 1: Extract keywords from the original text.

[0041] For text, the theme of the entire text can often be glimpsed through some keywords. The present invention extracts several keywords representing the semantic content of the article as local information of the text. Since there are no given keywords in the data set related to text summarization, the present invention first needs to extract the keywords of the original document. The method used in the present invention is mainly based on the idea of ​​unsupervised.

[0042] The steps to extract keywords from the original text are as follows:

[0043] Step 1.1 considers the position information of the word. Generally, the words that appear in the first and last sentences are more likely to be keywords. Therefore, we repeat the first and last sentences of the document three times respectively to increase the frequency of keywords in the first and last sentences.

[0044] Step 1.2: Segment the text into words and use the tf-idf statistics of each word to select 20 words as candidate keywords.

[0045] In step 1.3, we hope that the keywords obtained can represent the central idea of ​​the original text as much as possible. Keywords obtained only by using statistical information cannot guarantee this. Therefore, we need to further screen the keywords obtained in step 1.2: use Doc2Vec to obtain the vector representation d of the document, and use Word2Vec to obtain the vector representation w of the candidate keywords. Sort the candidate keywords according to the cosine distance between w and d, and select key phrases close to the document from the initial candidate keywords. The closer the keyword is to the document, the greater the amount of information, thus ensuring that the obtained keywords are more relevant to the document.

[0046] Step 1.4 is to avoid redundancy of the final keywords, that is, the extracted keywords have the same meaning although they have different expressions, so we need to perform a secondary screening on the keywords obtained in step 1.3: Similarly, we sort the candidate keywords according to the cosine distance between them, and for keywords with the same semantics, we only retain one.

[0047] Step 2: Build the Encoder module.

[0048] The purpose of this module is to encode the input text, that is, to represent it in vector form. The Encoder module of the present invention uses the encoder module of the Transformer to finally obtain the semantic representation of the original text with semantic features and context features, which we call global semantic information.

[0049] Step 3: Build a graph convolution module.

[0050] The purpose of this module is to integrate the relationships of local semantic information. In step 1, the semantic information of different keywords was obtained. In order to mine more effective local semantic features, we use the graph convolution method to add local features to relational features, thereby obtaining local semantic information with relational information. In graph convolution, the input includes nodes and adjacency matrices, where the nodes are the local semantic information extracted in step 1. There are relationships between nodes, and the adjacency matrix represents the degree of relationship between nodes. Then, graph convolution is used to adaptively learn the relationship weights between each keyword. After obtaining the adjacency matrix between keywords, it is multiplied by the initial semantic information to obtain relational features, and then the relational features are fused with the initial features to obtain a new round of features. The following steps are included:

[0051] Step 3.1 obtains K local semantic information and 1 global semantic information in steps 1 and 2 as nodes of the graph.

[0052] Step 3.2 constructs the adjacency matrix of the graph and initializes it to 1.

[0053] Step 3.3 The greater the difference between the local features and the overall features, the more outliers there are. Therefore, the present invention calculates the difference between the local semantic information and the global semantic information, constructs a difference matrix, and uses the difference to dynamically update the weights of the edges of all nodes in the graph. First, the global semantic information is repeated K times to obtain the difference between K local semantic information and the global semantic information, and finally obtains the difference matrix.

[0054] Step 3.4 uses linear transformation and other operations to convert the difference matrix obtained in 3.3 into a matrix with dimension (K, K), which is called the update matrix.

[0055] Step 3.5 multiplies the update matrix obtained in step 3.4 by the adjacency matrix element by element. The purpose of this operation is to use the update matrix to adaptively learn the adjacency matrix.

[0056] Step 3.6 multiplies the adjacency matrix obtained in 3.5 by the node information to obtain the relational features of the semantic information.

[0057] Step 3.7 concatenates the local relationship features obtained in 3.6 with the local semantic information of the node to obtain local semantic information with relationship information.

[0058] Step 4: Build the Decoder module

[0059] The purpose of this module is to generate a summary of the original text. The pointer generator network is a seq2seq model with a copy mechanism, which predicts words based on the probability distribution of the generator and the pointer. The generator mainly uses the background vector output by the encoder module, the hidden layer of the current step of the decoder, and the output of the previous step prediction of the decoder to predict the vocabulary of the current step. The summary predicted by the generator is the words in the vocabulary, and can predict words outside the original document. The words predicted by the probability distribution of the pointer are the text in the original document pointed to by the pointer. Therefore, the summary generated by the pointer generator network can both generate new vocabulary and copy the text in the original document. The pointer generator network can be regarded as a balance between the extraction method and the abstraction method. By copying words, the accuracy and processing power of unregistered words are improved, while the ability to generate new words is retained. The present invention uses an RNN with an attention mechanism as a decoder to output the summary. The specific steps are as follows:

[0060] Step 4.1: The global semantic information obtained in steps 2 and 3 and the local semantic information with relationship information are fused by summing up and used as the initial hidden vector of the decoder module.

[0061] Step 4.2 calculates the importance of each word in the original text to the Decoder word according to the attention mechanism. For each step of the Decoder, the semantic representation of the original text with different attention information is obtained according to different importance, which is called the background vector.

[0062] Step 4.3 predicts the output of the current time step based on the semantic representation of the original text, the output of the previous time step, and the hidden vector of the current time step, and finally obtains the predicted output of each time step, thereby obtaining the text summary of the original text.

Claims

1. A text summary acquisition method based on deep learning, characterized by: The method comprises the following steps: Step 1, extracting keywords from the original text; By extracting several keywords representing the semantic content of the article as local information of the text; extracting the keywords of the original document, the steps of extracting the keywords of the original text based on the unsupervised idea are as follows: Step 1.1 Consider the position information of the word. The words that appear in the first and last sentences have a high probability of being keywords. Repeat the first and last sentences of the document three times respectively to increase the frequency of keywords in the first and last sentences. Step 1.2: Segment the text into words and select 20 words as candidate keywords using the tf-idf statistics of each word; Step 1.3: further screen the keywords obtained in step 1.2: use Doc2Vec to obtain the vector representation d of the document, and use Word2Vec to obtain the vector representation w of the candidate keywords; sort the candidate keywords according to the cosine distance between w and d, and select key phrases close to the document from the initial candidate keywords. The closer the keyword is to the document, the greater the amount of information, thus ensuring that the obtained keyword is more relevant to the document; Step 1.4 To avoid the redundancy of the final keywords, that is, the keywords extracted have the same meaning although they have different expressions, it is necessary to perform a secondary screening on the keywords obtained in step 1.3: sort them according to the cosine distance between the candidate keywords, and only retain one keyword with the same semantics; Step 2: Build the Encoder module; The purpose of the Encoder module is to encode the input text, that is, to represent it in vector form. The Encoder module uses the Transformer encoder module to finally obtain the semantic representation of the original text with semantic features and context features, which becomes the global semantic information. Step 3: Build a graph convolution module; In step 1, semantic information of different keywords is obtained. In order to mine more effective local semantic features, the local features are added to the relational features by using the graph convolution method, thereby obtaining local semantic information with relational information. In the graph convolution, the input includes nodes and adjacency matrices, where the nodes are the local semantic information extracted in step 1. The nodes are related to each other, and the adjacency matrix represents the degree of relationship between the nodes. Then, the graph convolution is used to adaptively learn the relationship weight between each keyword. After obtaining the adjacency matrix between the keywords, it is multiplied by the initial semantic information to obtain the relational features, and then the relational features are fused with the initial features to obtain a new round of features. Step 4: Build the Decoder module; The purpose of the decoder module is to generate a summary of the original text. The pointer generator network is a seq2seq model with a copy mechanism. It predicts words based on the probability distribution of the generator and the pointer. The generator uses the background vector output by the encoder module, the hidden layer of the current step of the decoder, and the output of the previous step prediction of the decoder to predict the vocabulary of the current step. The summary predicted by the generator is the words in the vocabulary, and the words outside the original document are predicted. The words predicted by the probability distribution of the pointer are the text in the original document pointed to by the pointer. Therefore, the summary generated by the pointer generator network can both generate new vocabulary and copy the text in the original document; the pointer generator network is regarded as a balance between the extraction method and the abstraction method. By copying words, the accuracy and processing power of unregistered words are improved, while the ability to generate new words is retained; an RNN with an attention mechanism is used as a decoder to output the summary.

2. The method for obtaining text summarization based on deep learning according to claim 1, characterized in that: Step 3 includes the following steps: Step 3.1: Get K local semantic information and 1 global semantic information in steps 1 and 2 as nodes of the graph; Step 3.2 constructs the adjacency matrix of the graph and initializes it to 1; Step 3.3 The greater the difference between the local features and the overall features, the more outliers there are. Therefore, by calculating the difference between the local semantic information and the global semantic information, a difference matrix is ​​constructed, and the difference is used to dynamically update the weights of the edges of all nodes in the graph. First, the global semantic information is repeated K times to obtain the difference between K local semantic information and the global semantic information, and finally the difference matrix is ​​obtained. Step 3.4 uses a linear transformation operation to convert the difference matrix obtained in step 3.3 into a matrix of dimension (K, K), which is called the update matrix; Step 3.5 multiplies the update matrix obtained in step 3.4 by the adjacency matrix element by element. The purpose of this operation is to use the update matrix to adaptively learn the adjacency matrix. Step 3.6 multiplies the adjacency matrix obtained in 3.5 by the node information to obtain the relational features of the semantic information; Step 3.7 concatenates the local relationship features obtained in 3.6 with the local semantic information of the node to obtain local semantic information with relationship information.

3. The method for obtaining text summarization based on deep learning according to claim 1, characterized in that: Step 4 The specific steps are as follows: Step 4.1: The global semantic information obtained in step 2 and step 3 and the local semantic information with relational information are fused by summing up and used as the initial hidden vector of the decoder module; Step 4.2: Calculate the importance of each word in the original text to the Decoder word according to the attention mechanism. For each step of the Decoder, obtain the semantic representation of the original text with different attention information according to different importance, which is called the background vector. Step 4.3 predicts the output of the current time step based on the semantic representation of the original text, the output of the previous time step, and the hidden vector of the current time step, and finally obtains the predicted output of each time step, thereby obtaining the text summary of the original text.

Images