A text abstract generation method and system fusing global and local semantic features

Abstract

This invention discloses a text summarization method and system that integrates global and local semantic features, belonging to the field of natural language processing technology. The method includes: receiving text to be summarized and preprocessing it; inputting the preprocessed text into a global semantic feature extraction module and a local semantic feature extraction module respectively to obtain a global semantic feature vector and a local semantic feature vector; fusing them through an attention mechanism to obtain a context vector; and inputting the obtained context vector into a pointer-based decoding layer to obtain the final generated text summary. This invention integrates local and global semantic features, enhancing the accuracy of the generated summary and the model's ability to extract semantic information; by using a pointer-based decoder, it simultaneously extracts words from the original text and the vocabulary to form the final generated summary, solving the common out-of-vocabulary (OOV) problem in the field of text summarization.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a method and system for generating text summaries that integrates global and local semantic features. Background Technology

[0002] With the development of the internet age, a vast amount of information, including images, videos, and text, is constantly emerging online, impacting people's lives. However, while people enjoy the convenience that information brings, the problem of information overload is becoming increasingly common, making it crucial to accurately extract the most important content from this massive amount of information. As one of the most common forms of information on the internet, accurately helping users extract the most essential information from massive amounts of text data is a pressing issue that needs to be addressed.

[0003] Automatic text summarization is a branch of natural language processing. After inputting a long string of text, the machine quickly extracts the main content and summarizes it into a summary, helping users save time and improve reading efficiency. Currently, automatic text summarization technology has been widely applied in many fields such as news headline generation, key information retrieval, and public opinion monitoring.

[0004] Based on their generation methods, automatic text summarization can be categorized into extractive text summarization and abstract text summarization. Extractive text summarization directly extracts words and sentences from the original text to form the final summary. This method performs well in terms of sentence readability and fluency. Abstract text summarization extracts words from an established vocabulary to form the final summary. With the development of deep learning in recent years, abstract text summarization has seen significant improvements in both sentence accuracy and readability. However, summarization methods still face many challenges, such as unclear semantic expression and out-of-vocabulary (OOV) words, which urgently need to be addressed. Summary of the Invention

[0005] To address the shortcomings of current technologies, this invention introduces convolutional neural networks to enhance the extraction of global semantic features of text, and extracts local semantic features of text through a Keybert keyword extractor and a multi-head self-attention mechanism, thereby improving the overall capabilities of traditional models and solving technical problems such as unclear semantic expression and missing semantic information in the process of summarizing.

[0006] To achieve the above objectives, this invention provides a text summarization method that integrates global and local semantic features, comprising the following steps:

[0007] (1) Receive the text to be used to generate a summary and perform preprocessing;

[0008] (2) Construct a text summarization generation model, which includes: a global semantic feature extraction module and a local semantic feature extraction module, an attention mechanism and a decoding layer based on a pointer mechanism;

[0009] The preprocessed text is input into the global semantic feature extraction module and the local semantic feature extraction module respectively to obtain the global semantic feature vector and the local semantic feature vector.

[0010] The global semantic feature vector and the local semantic feature vector are input into the attention mechanism for fusion to obtain a context vector that fuses global and local semantic features;

[0011] Based on the obtained context vector, the text is input into a pointer-based decoding layer to obtain the final generated text summary.

[0012] (3) The model is trained based on the loss function to obtain the trained text summarization generation model;

[0013] (4) Input the text to be generated after processing in step (1) into the trained text summarization model to generate a text summary.

[0014] Furthermore, the preprocessing specifically includes:

[0015] Use the jieba word segmenter to segment the text;

[0016] Remove unnecessary tags, special characters, and stop words;

[0017] The text that is too long will be truncated.

[0018] Furthermore, the global semantic feature extraction module consists of a convolutional neural network and a single-layer bidirectional long short-term memory network (Bi-LSTM), which fuses the outputs of the convolutional neural network (CNN) and the bidirectional long short-term memory network (Bi-LSTM) to obtain a global semantic feature vector.

[0019] The local semantic feature extraction module consists of a Keybert keyword extractor and a transformer-based multi-head self-attention mechanism.

[0020] Furthermore, the outputs of the Convolutional Neural Network (CNN) and the Bidirectional Long Short-Term Memory (Bi-LSTM) network are fused as follows:

[0021] (2.1) The preprocessed text is input into the convolutional neural network (CNN), and the calculation formula is as follows:

[0022] m i =f(W c x i:i+h-1 +bc )

[0023] Where h refers to the length of the convolution kernel, and x i:i+h-1 Let f represent the matrix consisting of word vectors from the i-th word vector to the h-th word vector in the preprocessed text, and let b represent the non-linear activation function. c As a bias term, W c Represents the convolution kernel;

[0024] (2.2) The preprocessed text is input into the Bidirectional Long Short-Term Memory (Bi-LSTM) network, and the calculation formula is as follows:

[0025]

[0026]

[0027]

[0028] in, and h represents the forward and reverse states of the word in the Bi-LSTM hidden layer, respectively. i Indicates the hidden layer state of the Bi-LSTM encoder;

[0029] (2.3) The global semantic feature vector is obtained by fusion:

[0030] p i =h i +m i

[0031] Where: p i This is the global semantic feature vector.

[0032] Furthermore, the specific process of the Keybert keyword extractor is as follows:

[0033] First, BERT is used to obtain the vector-level representation of the preprocessed text. Then, the similarity between the n-gram phrases or terms and the preprocessed text vectors is calculated. Based on the results, the phrases or terms with the highest similarity are found and extracted as keywords.

[0034] Furthermore, the specific process of the multi-head self-attention mechanism of the transformer is as follows:

[0035] The keyword is input into the transformer-based multi-head self-attention mechanism, and three initialization matrices Q, K, and V are randomly initialized based on the keyword to obtain the attention weights:

[0036]

[0037] Where: d represents the dimension of matrices Q and K;

[0038] Multiple attention layers are processed in parallel and connected to obtain the keyword vector K:

[0039]

[0040] K=Concat(head1,head2,...,head n W o

[0041] Among them, head i W represents the output of the i-th attention in the multi-head self-attention mechanism. o W Q W K W V For the corresponding linear matrix, K represents the keyword vector after weight allocation.

[0042] Furthermore, the global semantic feature vector and the local semantic feature vector are input into the attention mechanism for fusion, and the process of calculating the context vector that fuses the global and local semantic features is as follows;

[0043] e t,i =softmax(v T tanh(W h p i +W s s t +W k K+b))

[0044]

[0045] α t,i =softmax(e′) ti )

[0046]

[0047] Where: p i Represents the global semantic feature vector. s represents the context vector, where K is the local semantic feature vector; t v represents the state of the LSTM decoding layer at time t. T W h W s W k , b are training parameters, α t,i Here, denoted as attention weights, and tanh represents the activation function.

[0048] Furthermore, the pointer-based decoding layer calculation method is as follows:

[0049]

[0050]

[0051]

[0052] in: s represents the context vector. t Let x be the state of the LSTM decoding layer at time t. t Let denot be the preprocessed text word sequence at time t, σ represent the sigmoid function, and P(w) represent the probability distribution of the final generated words. voc Represents the probability distribution of words drawn from the vocabulary, 1-P gen This represents the probability of extracting a word from the original text. b, V, b ptr All of these are training parameters.

[0053] Furthermore, the loss function employs a maximum likelihood estimation algorithm.

[0054] This invention also provides a text summarization generation system that integrates global and local semantic features, comprising:

[0055] Text acquisition module: Used to receive the text to be used to generate a summary and to perform preprocessing;

[0056] Feature extraction module: used to input the preprocessed text into the global semantic feature extraction module and the local semantic feature extraction module respectively to obtain global semantic feature vector and local semantic feature vector; and input them into the attention mechanism for fusion to obtain a context vector that fuses global and local semantic features;

[0057] The generation module is used to input the obtained context vector into the pointer-based decoding layer to obtain the final generated text summary.

[0058] The beneficial effects of this invention are:

[0059] 1. The global semantic feature extraction module has been enhanced by adding a convolutional neural network (CNN), which improves the model's global semantic feature extraction capability and overall model capability.

[0060] 2. A local semantic feature extraction module is introduced and its output is integrated with that of the global semantic feature extraction module, which enhances the accuracy of the generated summary and the model's ability to extract semantic information.

[0061] 3. A pointer-based decoder is used to simultaneously extract words from the original text and the vocabulary to form the final generated summary, solving the common out-of-vocabulary (OOV) problem in the field of text summarization. Attached Figure Description

[0062] Figure 1 This is a flowchart illustrating the text summarization generation method that integrates global and local semantic features according to an embodiment of the present invention.

[0063] Figure 2 This is a schematic diagram of the text summarization generation model according to an embodiment of the present invention. Detailed Implementation

[0064] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings and preferred embodiments.

[0065] like Figure 1 As shown, this embodiment of the invention provides a text summarization generation method that integrates global and local semantic features, including the following steps:

[0066] S101. Receive the text to be used to generate a summary and perform preprocessing;

[0067] Use the jieba word segmenter to segment the text;

[0068] Then remove useless tags, special characters, and stop words;

[0069] Finally, the original text that is too long is truncated, and the maximum length is set to 100.

[0070] S102. Construct a text summarization generation model, the model including: a global semantic feature extraction module and a local semantic feature extraction module, an attention mechanism and a decoding layer based on a pointer mechanism;

[0071] like Figure 2 As shown, a text summarization generation model is constructed as follows:

[0072] (1) Input the preprocessed text into the global semantic feature extraction module and the local semantic feature extraction module respectively to obtain the global semantic feature vector and the local semantic feature vector;

[0073] (1.1) Global semantic feature extraction module

[0074] The global semantic feature extraction module consists of a single-layer bidirectional long short-term sequence network and a convolutional neural network. The input of the original text is x = {x1, x2, x3…x}. i} represents, where x i This represents the i-th word in the original text, which is embedded using PyTorch's nn.embedding module.

[0075] First, the model is input through a Bi-LSTM long short-term sequence neural network, and each state becomes a hidden state h of the encoder. i ,

[0076]

[0077]

[0078]

[0079] in, and h represents the forward and reverse states of the word in the Bi-LSTM hidden layer, respectively. i This indicates the hidden layer state of the Bi-LSTM encoder.

[0080] Meanwhile, the original text is input into the CNN from the other end, and its calculation formula is as follows:

[0081] m i =f(W c x i:i+h-1 +b c )

[0082] Where h refers to the length of the convolution kernel, and x i:i+h-1 Let f represent the matrix consisting of word vectors from the i-th word vector to the h-th word vector, and let f represent the non-linear activation function. c As a bias term, W c This represents the convolution kernel.

[0083] Combining the outputs of CNN and Bi-LSTM yields the global semantic feature vector p. i =h i +m i .

[0084] (1.2) Local semantic feature extraction module

[0085] The local semantic feature extraction module consists of a Keybert keyword extractor and a transformer-based multi-head self-attention mechanism.

[0086] The Keybert keyword extractor extracts keywords from the original text by first using BERT to extract the vector-level representation of the original text, and then calculating the similarity between the vector representation and the original text based on n-grams. This embodiment uses the cosine similarity calculation formula:

[0087] sim i =cos(x) i ,y)

[0088] Where: x iLet y be the vector of the i-th word in the n-gram, and y be the vector of the original text.

[0089] Subsequently, based on the obtained keyword information, a transformer-based multi-head self-attention mechanism is input. Three initialization matrices, Q, K, and V, are randomly initialized based on the keyword information. After Q and K... T The series of operations following the multiplication yields the attention weights, which are then used... The input matrices Q and K are scaled using the denominator, where d represents the dimension of the input matrices Q and K. After matrix multiplication and scaling, the matrix passes through a softmax layer and is then multiplied by matrix V to obtain the attention weights. The formula for calculating the weights Attention(Q, K, V) is as follows:

[0090]

[0091] In multi-head attention mechanisms, multiple attention layers are typically processed in parallel and connected to obtain the final output. The formulas for calculating the individual attention mechanism and the overall result are as follows:

[0092]

[0093] K=Concat(head1,head2,...,head n W o

[0094] Among them, head i W represents the output of the i-th attention in the multi-head self-attention mechanism. o W Q W K W V To correspond to the linear matrix, the outputs of each attention are concatenated, and then a linear operation is performed to obtain the keyword vector K after weight allocation.

[0095] (2) Input the global semantic feature vector and the local semantic feature vector into the attention mechanism for fusion to obtain a context vector that fuses global and local semantic features;

[0096] The specific attention mechanism that integrates global and local semantic features is calculated as follows:

[0097] e t,i =softmax(v T tanh(W h a i +W s s t +W k K+b))

[0098]

[0099] α t,i =softmax(e′) ti )

[0100]

[0101] in, s represents the context vector. t For the LSTM decoding layer state, v T W h W s W k , b are trainable parameters, and tanh represents the activation function.

[0102] (3) Based on the obtained context vector, input it into the pointer-based decoding layer to obtain the final generated text summary;

[0103] The decoding layer calculation method based on the pointer mechanism is as follows:

[0104]

[0105]

[0106]

[0107] in, s represents the context vector. t Let x be the state of the LSTM decoding layer at time t. t Let denot be the preprocessed text word sequence at time t, σ represent the sigmoid function, and P(w) represent the probability distribution of the final generated words. voc Represents the probability distribution of words drawn from the vocabulary, 1-P gen This represents the probability of extracting a word from the original text. b, V, b ptr All of these are trainable parameters.

[0108] S103. Train the model based on the loss function to obtain the trained text summarization generation model;

[0109] This embodiment employs the Maximum Likelihood Estimate (MLE) algorithm. MLE is obtained by maximizing the probability of a given sample during the sequence-to-sequence training process. Let the original text of an article be x, and the generated summary be y = {y1}. * y2 * y3 * …}, during training, the optimal result is obtained by minimizing the following loss function:

[0110]

[0111] Among them, L ml This represents the loss function based on MLE. The MLE-based method generates words one by one during training by comparing them with the standard summary.

[0112] The public evaluation metrics used were ROUGE and manual evaluation. Based on the existing basic model, the final evaluation results showed that ROUGE-1, ROUGE-2 and ROUGE-L all showed certain improvements, with accuracy rates of 47.87%, 31.12% and 45.71% respectively. After introducing local semantic information extraction modules and global semantic information extraction modules, the abstract generation results had good accuracy.

[0113] S104. Input the text to be summarized after step S101 into the trained text summarization generation model to generate a text summary.

[0114] Another embodiment of the present invention provides a text summarization generation system that integrates global and local semantic features, comprising:

[0115] Text acquisition module: Used to receive the text to be used to generate a summary and to perform preprocessing;

[0116] Feature extraction module: used to input the preprocessed text into the global semantic feature extraction module and the local semantic feature extraction module respectively to obtain global semantic feature vector and local semantic feature vector; and input them into the attention mechanism for fusion to obtain a context vector that fuses global and local semantic features;

[0117] The generation module is used to input the obtained context vector into the pointer-based decoding layer to obtain the final generated text summary.

[0118] In summary, this invention proposes a text summarization method that integrates local and global semantic information, and generates a summary that combines both. The pointer-based decoder extracts words from the original text and the generated vocabulary to form the final summary, thus solving the common OOV (Out-of-Voice) problem. The summary obtained through this method demonstrates good fluency and keyword expression capabilities.

[0119] While specific embodiments of the present invention have been described above, those skilled in the art should understand that these are merely illustrative examples. Various changes or modifications can be made to these embodiments without departing from the principles and essence of the present invention. Therefore, the scope of protection of the present invention is defined by the appended claims.

Claims

1. A text summarization generation method that integrates global and local semantic features, characterized in that, Includes the following steps: (1) Receive the text to be used to generate a summary and perform preprocessing; (2) Construct a text summarization generation model, which includes: a global semantic feature extraction module and a local semantic feature extraction module, an attention mechanism and a decoding layer based on a pointer mechanism; The preprocessed text is input into the global semantic feature extraction module and the local semantic feature extraction module respectively to obtain the global semantic feature vector and the local semantic feature vector. The global semantic feature vector and the local semantic feature vector are input into the attention mechanism for fusion to obtain a context vector that fuses global and local semantic features; Based on the obtained context vector, the text is input into a pointer-based decoding layer to obtain the final generated text summary. The global semantic feature extraction module consists of a convolutional neural network (CNN) and a single-layer bidirectional long short-term memory (Bi-LSTM) network. It fuses the outputs of the CNN and the Bi-LSTM to obtain a global semantic feature vector; specifically: (2.1) The preprocessed text is input into the convolutional neural network (CNN), and the calculation formula is as follows: Where h refers to the length of the convolution kernel. Let f represent the matrix consisting of word vectors from the i-th word vector to the h-th word vector in the preprocessed text, and let f represent the non-linear activation function. For bias terms, Represents the convolution kernel; (2.2) The preprocessed text is input into the Bidirectional Long Short-Term Memory (Bi-LSTM) network, and the calculation formula is as follows: in, and These represent the forward and backward states of the word vector in the Bi-LSTM hidden layer, respectively. Indicates the hidden layer state of the Bi-LSTM encoder; (2.3) The global semantic feature vector is obtained by fusion: in: This is the global semantic feature vector; The local semantic feature extraction module consists of a Keybert keyword extractor and a transformer-based multi-head self-attention mechanism; the specific process of the Keybert keyword extractor is as follows: First, BERT is used to obtain the vector-level representation of the preprocessed text. Then, the similarity between the n-gram phrases or terms and the preprocessed text vectors is calculated. Based on the results, the phrases or terms with the highest similarity are found and extracted as keywords. (3) The model is trained based on the loss function to obtain the trained text summarization generation model; (4) Input the text to be generated after processing in step (1) into the trained text summarization generation model to generate a text summary.

2. The text summarization generation method integrating global and local semantic features according to claim 1, characterized in that, The preprocessing specifically includes: Use the jieba word segmenter to segment the text; Remove unnecessary tags, special characters, and stop words; The text that is too long will be truncated.

3. The text summarization generation method integrating global and local semantic features according to claim 1, characterized in that, The specific process of the multi-head self-attention mechanism of the transformer is as follows: The keyword is input into the transformer-based multi-head self-attention mechanism, and three initialization matrices Q, K, and V are randomly initialized based on the keyword to obtain the attention weights: Where: d represents the dimension of matrices Q and K; Multiple attention layers are processed in parallel and connected to obtain the keyword vector K: in, This represents the output of the i-th attention in the multi-head self-attention mechanism. For the corresponding linear matrix, K represents the keyword vector after weight allocation.

4. The text summarization generation method integrating global and local semantic features according to claim 1, characterized in that, The global semantic feature vector and the local semantic feature vector are input into the attention mechanism for fusion, and the process of calculating the context vector that fuses the global and local semantic features is as follows: in: Represents the global semantic feature vector. Represents the context vector. It is a local semantic feature vector; For LSTM decoding layer t Current state , , , , All of these are training parameters. For attention weights, tanh This represents the activation function.

5. The text summarization generation method integrating global and local semantic features according to claim 1, characterized in that, The pointer-based decoding layer calculation method is as follows: in: Represents the context vector. For LSTM decoding layer t Current state express t The preprocessed text word sequence at time σ represents the sigmoid function. This represents the probability distribution of the final generated words. This represents the probability distribution of words drawn from the vocabulary. This represents the probability of extracting a word from the original text. , , , , , , All of these are training parameters.

6. The text summarization generation method integrating global and local semantic features according to claim 1, characterized in that, The loss function employs the maximum likelihood estimation algorithm.

7. A text summarization generation system that integrates global and local semantic features, employing the text summarization generation method integrating global and local semantic features as described in any one of claims 1-6, characterized in that, include: Text acquisition module: Used to receive the text to be used to generate a summary and to perform preprocessing; Feature extraction module: used to input the preprocessed text into the global semantic feature extraction module and the local semantic feature extraction module respectively to obtain global semantic feature vector and local semantic feature vector; and input them into the attention mechanism for fusion to obtain a context vector that fuses global and local semantic features; The generation module is used to input the obtained context vector into the pointer-based decoding layer to obtain the final generated text summary.

Images