A News Text Summarization Method Based on Deep Learning

Abstract

A news text summary generation method based on deep learning includes the following steps: 1) crawling news titles and news texts to obtain raw data; 2) preprocessing the data to obtain the data format required by the model; 3) using the Bert model as the core to build an extractive and generative summary generation system; 4) the extractive model first extracts sentences with high relevance to the title from the input text, which are used as guidance signals for the generative model; 5) the input text and the guidance signal are input to the generative model to generate a final summary; 6) the generated summary is compared and evaluated with the reference summary. The present invention combines the extractive and generative summary models to generate a summary of the news text, which can simplify the lengthy text and quickly obtain the key information therein.

Description

Technical Field

[0001] The invention relates to crawler algorithms, news big data and deep learning, and provides a news text summary generation method based on deep learning. Background Art

[0002] With the rapid development of information technology, a large amount of text data has emerged like a tide. How to quickly and effectively obtain key information from a lengthy text is particularly important in this rapidly developing Internet era. Automatic summarization technology is one of the important research contents in the field of natural language processing. It uses computers to automatically extract, summarize or refine key information in texts or text collections, and extract concise and semantically coherent short articles that can summarize the source documents. Automatic summarization technology greatly improves the speed of information acquisition and effectively reduces people's information burden. It has high research value in the fields of document summarization and information retrieval.

[0003] Early text summarization work was mostly done manually. In recent years, the amount of text data has exploded, and automatic summarization technology has received widespread attention and research, and a large number of model algorithms have emerged. The current mainstream methods are extractive summarization and generative summarization. The extractive summarization method simply regards automatic summarization as a binary classification task to determine whether a sentence in a document belongs to the summary content; the generative summarization method is to learn the text-summary pairs of training data and generate summaries according to different algorithms. These working methods have achieved good results to a certain extent and have been applied in fields such as medicine, news, and finance. For example, the use of social media summaries can help users quickly understand the key information. At present, the most studied and most effective automatic summary generation technology based on deep learning is the method that uses computers to encode and vectorize documents, obtains vector representations of document context and semantics, and then uses deep networks for training and learning, achieving many optimal results in text summarization tasks. However, the summaries generated by existing model algorithms are still far from the effect of manual annotation. There is still huge room for improvement in this task and it needs to be explored continuously.

[0004] In this case, the model algorithm that combines extractive and generative methods can greatly improve the quality of the generated summary, alleviating the problems of redundancy and semantic incoherence in the summaries generated by the extractive model, and avoiding the phenomenon that the summaries generated by the generative model have factual errors compared with the original text. Summary of the invention

[0005] In order to overcome the shortcomings of the existing technology and to quickly obtain the approximate content therein, the present invention proposes a news text summary generation method based on deep learning. In massive news texts, an extractive summary model is first used to extract important sentences in the text, and then a generative summary model is used to generate a semantically coherent and concise summary.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] A news text summary generation method based on deep learning, the method comprising the following steps:

[0008] 1) Use the crawler algorithm to crawl news titles and news texts on news websites. Each news text and news title form a story file, and the text and title are separated by the @highlight identifier;

[0009] 2) The above story file is segmented into sentences and words using the Stanford CoreNLP toolkit. The first three sentences with the highest ROUGE-1 / 2 index scores in the news text and the title sentence are selected through a greedy algorithm and labeled. Finally, the labeled text in a certain format is generated as training data and verification data;

[0010] 3) The text summary generation method uses an extraction + generation strategy. First, an extractive model is used to extract sentences with high relevance to the title from the news text. Then, the extracted sentences are used as guidance signals to act on another generative model to guide the generation of the summary.

[0011] 4) The extractive model uses the Bert network model to obtain sentence representation, and the Inter-sentence Transformer layer is used as the summary layer to perform text summary extraction training on the annotated text of a certain format; at the same time, two other Bert network models are used as two encoders, and the decoder uses the Decoder end of the Transformer to perform text summary generation training on the input text and the guidance signal. The guidance signal is the key sentence marked with a label. The implementation process is as follows:

[0012] 4.1) Given a tokenized text X = [cls, x1, x2, …, x n ,sep] where cls and sep are tokens inserted at the beginning and end of each sentence, x t is the token at position t in the text. After inputting the text X into the Bert model, the hidden representation of the text is obtained: H = [h1, h2, ... h m ], and further obtain the hidden representation of the clstoken at the beginning of each sentence T = [T1, T2, ... Tt ];

[0013] 4.2) The hidden representation T is transmitted to the summary layer consisting of two layers of Transformer Encoder layer, and the output is h l , and then pass through a sigmoid layer to get the score of each sentence Y = [y1, y2, ...y n ], according to the label tag L of the annotated text = [l1,l2,…l n ], and then train the extractive model using the binary cross entropy loss function:

[0014]

[0015]

[0016] y i =σ(W o h l +b o ) (3)

[0017]

[0018] l n =-[l n ·logy n +(1-l n )·log(1-y n )] (5)

[0019] Where l represents the number of layers of the Transformer Encoder layer, and MHAtt is the output h of the Transformer Encoder layer. l-1 Perform multi-head attention operations, It is the output after the layer normalization operation LN, and then the FFN feedforward network operation is performed on the output to generate the output vector h of the lth layer l , σ is the sigmoid operation, W o is the weight parameter, b o is the deviation, y i is the prediction score of the i-th sentence, N is the number of sentences, l n is the label of the nth sentence, y n is the prediction score of the nth sentence;

[0020] 4.3) Tokenize the input text X = [x1, x2, ..., x n ] and the tokenized guidance signal G = [g1, g2, …, g m] are input to the two encoder Bert models of the generative model respectively, and the hidden representation H is obtained respectively. x =[h x1 ,h x2 ,…,h xn ], H g =[h g1 ,h g2 ,…,h gm ];

[0021] 4.4) The decoder of the generative model first performs a self-attention operation on the output y of the previous layer, and then uses the Cross Attention mechanism to focus on the guidance signal and generate the corresponding representation y1. Then, it passes through a Cross Attention layer to process the entire input text according to y1 to obtain the representation y2. Finally, it passes through a feedforward network layer to obtain the intermediate output y3. After passing through 6 layers of Decoder layers, the probability distribution of the output word is calculated, the loss is calculated, and the model is trained.

[0022] y=LN(y+Selfatt(y)) (6)

[0023] y1=LN(y+Crossatt(y,g)) (7)

[0024] y2=LN(y1+Crossatt(y1,x)) (8)

[0025] y3=LN(y2+FFN(y2)) (9)

[0026] Among them, LN is the layer normalization operation, Selfatt is the self-attention mechanism, Crossatt is the cross-attention mechanism, FFN is the feedforward network operation, y is the output vector of the previous layer on the decoder side, g is the vector representation of the guidance signal, x is the vector representation of the input document, y1 is the vector representation of the guidance signal g fused on the basis of y, y2 is the vector representation of the input document vector x fused on the basis of y1, and y3 is the output of the decoder layer on the decoder side;

[0027] 5) After training is completed, save the extractive model and the generative model separately;

[0028] 6) When generating a summary, a piece of news text is first input into the extractive model to extract the candidate summary considered by the model, and then the candidate summary and the news text are input into the generative model again to obtain the final summary.

[0029] The beneficial effects of the present invention are as follows: extractive and generative summary generation are combined to make up for their respective shortcomings, and the generated summary is semantically coherent and concise while ensuring factual consistency between the summary and the original text. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a logical flow chart of the news text summary generation method based on deep learning. DETAILED DESCRIPTION

[0031] The present invention will be further described below in conjunction with the accompanying drawings.

[0032] Reference Figure 1 , a news text summary generation method based on deep learning, the method comprising the following steps:

[0033] 1) Use the crawler algorithm to crawl news titles and news texts on news websites. Each news text and news title form a story file, and the text and title are separated by the @highlight identifier;

[0034] 2) The above story file is segmented into sentences and words using the Stanford CoreNLP toolkit. The first three sentences with the highest ROUGE-1 / 2 index scores in the news text and the title sentence are selected through a greedy algorithm and labeled. Finally, the labeled text in a certain format is generated as training data and verification data;

[0035] 3) The text summary generation method uses an extraction + generation strategy. First, an extractive model is used to extract sentences with high relevance to the title from the news text. Then, the extracted sentences are used as guidance signals to act on another generative model to guide the generation of the summary.

[0036] 4) The extractive model uses the Bert network model to obtain sentence representation, and the Inter-sentence Transformer layer is used as the summary layer to perform text summary extraction training on the annotated text of a certain format; at the same time, two other Bert network models are used as two encoders, and the decoder uses the Decoder end of the Transformer to perform text summary generation training on the input text and the guidance signal. The guidance signal is the key sentence marked with a label. The implementation process is as follows:

[0037] 4.1) Given a tokenized text X = [cls, x1, x2, …, x n ,sep] where cls and sep are tokens inserted at the beginning and end of each sentence, x tis the token at position t in the text. After inputting the text X into the Bert model, the hidden representation of the text is obtained: H = [h1, h2, ... h m ], and further obtain the hidden representation of the clstoken at the beginning of each sentence T = [T1, T2, ... T t ];

[0038] 4.2) The hidden representation T is transmitted to the summary layer consisting of two layers of Transformer Encoder layer, and the output is h l , and then pass through a sigmoid layer to get the score of each sentence Y = [y1, y2, ...y n ], according to the label tag L of the annotated text = [l1,l2,…l n ], and then train the extractive model using the binary cross entropy loss function:

[0039]

[0040]

[0041] y i =σ(W o h l +b o ) (3)

[0042]

[0043] l n =-[l n ·logy n +(1-l n )·log(1-y n )] (5)

[0044] Where l represents the number of layers of the Transformer Encoder layer, and MHAtt is the output h of the Transformer Encoder layer. l-1 Perform multi-head attention operations, It is the output after the layer normalization operation LN, and then the FFN feedforward network operation is performed on the output to generate the output vector h of the lth layer l , σ is the sigmoid operation, W o is the weight parameter, b o is the deviation, y i is the prediction score of the i-th sentence, N is the number of sentences, l n is the label of the nth sentence, y n is the prediction score of the nth sentence;

[0045] 4.3) Tokenize the input text X = [x1, x2, ..., x n ] and the tokenized guidance signal G = [g1, g2, …, g m ] are input to the two encoder Bert models of the generative model respectively, and the hidden representation H is obtained respectively. x =[h x1 ,h x2 ,…,h xn ], H g =[h g1 ,h g2 ,…,h gm ];

[0046] 4.4) The decoder of the generative model first performs a self-attention operation on the output y of the previous layer, and then uses the Cross Attention mechanism to focus on the guidance signal and generate the corresponding representation y1. Then, it passes through a Cross Attention layer to process the entire input text according to y1 to obtain the representation y2. Finally, it passes through a feedforward network layer to obtain the intermediate output y3. After passing through 6 layers of Decoder layers, the probability distribution of the output word is calculated, the loss is calculated, and the model is trained.

[0047] y=LN(y+Selfatt(y)) (6)

[0048] y1=LN(y+Crossatt(y,g)) (7)

[0049] y2=LN(y1+Crossatt(y1,x)) (8)

[0050] y3=LN(y2+FFN(y2)) (9)

[0051] Among them, LN is the layer normalization operation, Selfatt is the self-attention mechanism, Crossatt is the cross-attention mechanism, FFN is the feedforward network operation, y is the output vector of the previous layer on the decoder side, g is the vector representation of the guidance signal, x is the vector representation of the input document, y1 is the vector representation of the guidance signal g fused on the basis of y, y2 is the vector representation of the input document vector x fused on the basis of y1, and y3 is the output of the decoder layer on the decoder side;

[0052] 5) After training is completed, save the extractive model and the generative model separately;

[0053] 6) When generating a summary, a piece of news text is first input into the extractive model to extract the candidate summary considered by the model, and then the candidate summary and the news text are input into the generative model again to obtain the final summary.

[0054] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A news text summary generation method based on deep learning, characterized in that: The method comprises the following steps: 1) Use the crawler algorithm to crawl news titles and news texts on news websites. Each news text and news title form a story file, and the text and title are separated by the @highlight identifier; 2) The above story file is segmented into sentences and words using the Stanford CoreNLP toolkit. The first three sentences with the highest ROUGE-1 / 2 index scores in the news text and the title sentence are selected through a greedy algorithm and labeled. Finally, the labeled text in a certain format is generated as training data and verification data; 3) The text summary generation method uses an extraction + generation strategy. First, an extractive model is used to extract sentences with high relevance to the title from the news text. Then, the extracted sentences are used as guidance signals to act on another generative model to guide the generation of the summary. 4) The extractive model uses the Bert network model to obtain sentence representation, and the Inter-sentence Transformer layer is used as the summary layer to perform text summary extraction training on the annotated text of a certain format; at the same time, two other Bert network models are used as two encoders, and the decoder uses the Decoder end of the Transformer to perform text summary generation training on the input text and the guidance signal. The guidance signal is the key sentence marked with a label. The implementation process is as follows: 4.1) Given a tokenized text X = [cls, x1, x2, …, x n ,sep] where cls and sep are tokens inserted at the beginning and end of each sentence, x t is the token at position t in the text. After inputting the text X into the Bert model, the hidden representation of the text is obtained: H = [h1, h2, ... h m ], get the hidden representation of the cls token at the beginning of each sentence T = [T1, T2, ... T t ]; 4.2) The hidden representation T is transmitted to the summary layer consisting of two layers of Transformer Encoder layer, and the output is h l , and then pass through a sigmoid layer to get the score of each sentence Y = [y1, y2, ...y n ], according to the label tag L of the annotated text = [l1,l2,…l n ], and then train the extractive model using the binary cross entropy loss function: l n =-[l n ·logy n +(1-l n )·log(1-y n )] (5) Where l represents the number of layers of the Transformer Encoder layer, and MHAtt is the output h of the Transformer Encoder layer. l-1 Perform multi-head attention operations, It is the output after the layer normalization operation LN, and then the FFN feedforward network operation is performed on the output to generate the output vector h of the lth layer l ,σ is the sigmoid operation, W o is the weight parameter, b o is the deviation, y i is the prediction score of the i-th sentence, N is the number of sentences, l n is the label of the nth sentence, y n is the prediction score of the nth sentence; 4.3) Tokenize the input text X = [x1, x2, ..., x n ] and the tokenized guidance signal G = [g1, g2, …, g m ] are input to the two encoder Bert models of the generative model respectively, and the hidden representation H is obtained respectively. x =[h x1 ,h x2 ,…,h xn ], H g =[h g1 ,h g2 ,…,h gm ]; 4.4) The decoder of the generative model first performs a self-attention operation on the output y of the previous layer, and then uses the Cross Attention mechanism to focus on the guidance signal and generate the corresponding representation y1. Then, it passes through a Cross Attention layer to process the entire input text according to y1 to obtain the representation y2. Finally, it passes through a feedforward network layer to obtain the intermediate output y3. After passing through 6 layers of Decoder layers, the probability distribution of the output word is calculated, the loss is calculated, and the model is trained. y=LN(y+Selfatt(y)) (6) y1=LN(y+Crossatt(y,g)) (7) y2=LN(y1+Crossatt(y1,x)) (8) y3=LN(y2+FFN(y2)) (9) Among them, LN is the layer normalization operation, Selfatt is the self-attention mechanism, Crossatt is the cross-attention mechanism, FFN is the feedforward network operation, y is the output vector of the previous layer on the decoder side, g is the vector representation of the guidance signal, x is the vector representation of the input document, y1 is the vector representation of the guidance signal g fused on the basis of y, y2 is the vector representation of the input document vector x fused on the basis of y1, and y3 is the output of the decoder layer on the decoder side; 5) After training is completed, save the extractive model and the generative model separately; 6) When generating a summary, a piece of news text is first input into the extractive model to extract the candidate summary considered by the model, and then the candidate summary and the news text are input into the generative model again to obtain the final summary.

Images