A summary generation method based on knowledge and semantic information enhancement

By combining graph neural networks, the pre-trained language model BERT, and an improved Transformer decoder, and utilizing pointer networks and reinforcement learning algorithms to optimize the generation of summaries, the problems of information loss and incoherent sentences in existing technologies are solved, resulting in more coherent and realistic summaries.

CN116595953BActive Publication Date: 2026-11-17NANJING UNIV OF INFORMATION SCI & TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310596388.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-05-24
Publication Date
2026-11-17
Estimated Expiration
2043-05-24

AI Technical Summary

Technical Problem

Existing automatic text summarization methods suffer from problems such as information loss, incoherent sentences, and false information when generating summaries, resulting in a lack of authenticity and readability.

Method used

We employ graph neural networks and the pre-trained language model BERT combined with an improved Transformer decoder. Through knowledge and semantic information enhancement, we utilize pointer networks and reinforcement learning algorithms to optimize the generation of summaries.

Benefits of technology

It improves the quality and readability of the generated summaries, enhances text comprehension and analysis capabilities, captures structural information of the text, and makes the generated summaries more coherent and authentic.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116595953B_ABST
    Figure CN116595953B_ABST
Patent Text Reader

Abstract

This invention discloses a summarization method based on knowledge and semantic information enhancement, comprising the following steps: S1 Constructing a contextual knowledge graph G; S2 Updating the feature vector of node i in the contextual knowledge graph G to obtain the contextual knowledge feature representation R. g S3 obtains the text document feature representation R through a language model. t S4 uses an improved Transformer decoder to decode and obtain the summary sequence vector S. t S5 represents the abstract sequence vector S t and text document feature representation R t Perform calculations to obtain the weight distribution C. t S6 encodes the location information into a probability distribution P. gen Steps S4 and S5 are repeated to obtain the complete summary sequence y'; in S7, the reward value is obtained through the similarity index, and gradient optimization is performed on the pointer network and the improved Transformer decoder. This invention improves the traditional Transformer decoder to better interact textual knowledge and semantic information.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of natural language processing, and more particularly to a summarization method based on knowledge and semantic information enhancement. Background Technology

[0002] Generally, automatic text summarization methods can be divided into extractive and generative methods. Extractive methods focus on capturing text units with key information from the original text and combining them into a summary. These text units typically include sentences, words, and phrases. Generative methods focus more on the method's comprehension ability, understanding the content of the original text to form corresponding summary information. This method can generate text that does not exist in the original text and has the ability to generate high-quality summaries. In recent years, deep learning models have been widely used in text summarization tasks, and the introduction of pre-trained language models has further improved text representation capabilities, which greatly enhances the ability to generate summaries, surpassing traditional machine learning methods. However, extracting information from text and expressing its features has certain limitations. There are problems such as significant information loss, subject-verb inversion, and object substitution in the generated text, resulting in false information that makes the generated summaries lack authenticity and readability. Summary of the Invention

[0003] Purpose of the invention: To address the problems of insufficient encoding of semantic information and incoherent generated summary sentences in automatic text summarization tasks in natural language processing, this invention provides a summarization generation method based on knowledge and semantic information enhancement to improve the speed at which users capture information and thus reduce the information burden.

[0004] Technical solution:

[0005] A summarization method based on knowledge and semantic information enhancement includes the following steps:

[0006] S1. Use the Stanford OpenIE open-source tool to extract the knowledge triple set of each text document in the publicly available automatic text summarization dataset, and use the triple set to construct the contextual knowledge graph G of the text document;

[0007] S2, using a graph attention network, update the feature vector of node i in the context knowledge graph G one by one to obtain the updated context knowledge feature representation R. g ;

[0008] S3, using the pre-trained language model BERT, obtains the feature representation R of each text document in the automatic text summarization dataset. t ;

[0009] S4, using an improved Transformer decoder to represent contextual knowledge features R gand text document feature representation R t Decoding is performed to obtain the summary sequence vector S. t ;

[0010] S5 introduces the generation probability P through the pointer network model. gen ∈[0,1], for the summary sequence vector S t and text document feature representation R t Perform correlation calculations to obtain the weight distribution C. t ;

[0011] S6. The pointer network model is used to calculate the position of the generated word or phrase in the original text, and the position information is encoded into a probability distribution P. gen Repeat steps S4 and S5 until the length of the generated summary reaches the preset value, and a complete summary sequence y' is obtained.

[0012] S7 utilizes reinforcement learning theory and algorithms to calculate the reward value r(y') by calculating the similarity index ROUGE between the generated summary sequence y' and the reference summary sequence y; then, gradient optimization is performed on the pointer network and the improved Transformer decoder.

[0013] Furthermore, the specific steps of step S2 are as follows:

[0014] S2-1, Calculate the similarity coefficient e between node i and its neighbor node j in the context knowledge graph G one by one. ij The calculation formula is as follows:

[0015] e ij =a([Wh i ||Wh j ]),j∈N i

[0016] Among them, h i h is the initialization vector of i; j The initialization vector for j; W represents the shared parameters of the neural network, Wh i For h i Perform feature mapping; Wh j For h j Perform feature mapping; [·||·] is the concatenation operation; a(·) is the mapping function; e ij Let N be the similarity coefficient between i and j; i Let i be the neighborhood of node i;

[0017] S2-2, the similarity coefficient e between nodes i and j ij Perform a normalization operation to obtain the attention coefficients α for nodes i and j. ij The specific formula is as follows:

[0018]

[0019] Where exp(.) is the exponential function; LeakyReLU() is the activation function that normalizes the coefficients; n represents the neighboring nodes other than i and j; e in Let be the similarity coefficient between i and n;

[0020] S2-3, using the M-head attention mechanism to adjust the attention coefficients α of i and j. ij The update is performed, and a weighted summation operation is applied to the M attention results to output a new feature vector h′ after aggregating information from i and j. i Then, the M-head attention mechanism is used to focus on h′. i Enhancement is performed to obtain vector h′ i (M).

[0021] Furthermore, the new feature vector h′ i The expression is as follows:

[0022]

[0023] Where W represents the shared parameters of the neural network; σ(·) is the activation function;

[0024] Using the M-head attention mechanism to h i 'Enhance the vector h' i (M), the calculation formula is as follows:

[0025]

[0026] Where M represents the attention of the Mth head; || represents the splicing operation.

[0027] Furthermore, the specific calculation steps of step S3 are as follows:

[0028] S3-1, the multi-head self-attention layer performs a linear transformation on the feature vector input to the pre-trained language model BERT, generating three vectors: the lookup vector Q, the content vector K, and the lookup vector V; and calculates the attention vector based on these three vectors, as shown in the following formula:

[0029]

[0030] Where Attention() is the attention module function for a single head; T is the matrix transpose; d k The column numbers of the Q and K matrices are denoted as 'Q' and 'K', respectively; Softmax() is the normalization exponential function.

[0031] The Softmax() function can be used to compress the length L of any real vector into the range [0, 1], with the sum of all elements in the vector being 1. The specific formula is as follows:

[0032]

[0033] Where e is an exponential function;

[0034] S3-2, Repeat step S3-1 multiple times to obtain the multi-head attention vector MultiHead(Q,K,V), the specific formula is as follows:

[0035]

[0036]

[0037] Among them, Multihead() is the multi-head attention function; Concat() is the concatenation function; Attention() is the attention module function for a single head; head m This is the output of the m-th head; Let d be the weight matrix for the m-th head, where d = {Q, K, V}; Let Q be the weight matrix of the search vector Q for the m-th head; Let K be the weight matrix of the content vector K of the m-th header; Let V be the weight matrix of the vector V being searched for in the m-th head;

[0038] S3-3, after the forward feedback layer maps the MultiHead(Q,K,V) vectors, the text document feature representation R is obtained. t .

[0039] Furthermore, in step S4, the improved Transformer decoder comprises four modules from bottom to top: a multi-head attention layer + residual and normalization layer, a knowledge attention layer + a first normalization layer, a semantic attention layer + a second normalization layer, and a feedforward neural network + a third normalization layer; the modules of the improved Transformer decoder are connected in a skip connection.

[0040] Furthermore, the specific steps of step S4 are as follows:

[0041] S4-1, the improved Transformer decoder uses the token T of the Tth summary word. t As input, it passes through a multi-head attention layer; then, the output of the multi-head attention layer serves as the input to the first normalization layer, which transforms it into data with a mean of 0 and a variance of 1.

[0042] S4-2, representing contextual knowledge features in R gAs the lookup vector Q in the knowledge attention mechanism, the output of the first normalization layer is used as the content vector K and the lookup vector V. Attention is calculated and normalized to obtain the output value of the second normalization layer.

[0043] S4-3, representing the text document features R t As the lookup vector Q in the semantic attention mechanism, the output value of the second normalization layer is used as the content vector K and the lookup vector V to realize semantic attention calculation and normalization, and obtain the output value of the third normalization layer.

[0044] S4-4, the output of the third normalization layer is fed into the feedforward neural network for linear mapping and normalization. The decoder output is the summary sequence vector S. t .

[0045] Furthermore, step S5 involves processing the summary sequence vector St and the text document feature representation R. t Perform correlation calculations to obtain the weight distribution C. t The specific calculation formula is as follows:

[0046] α e =softmax(V T ·tanh(W t R t +W s S t +b))

[0047]

[0048] Where tanh() is the activation function; softmax() is the normalization exponential function; V, W t W s b and are the parameters learned during the training of the pointer network model; T is the matrix transpose.

[0049] Furthermore, the probability distribution P in step S6 gen The specific calculation formula is as follows:

[0050]

[0051] in b gen It is the learnable parameter matrix; σ(·) is the activation function; C t For weighted distribution; L t Input the token L of the Lth summary word into the decoder t ;

[0052] Furthermore, the specific implementation steps of step S7 are as follows:

[0053] S7-1, The goal during the training phase is to minimize the maximum likelihood loss L. ML The specific formula is as follows:

[0054]

[0055] Where x is a text document in the automatic text summarization dataset, y is a reference summary in the dataset, D represents the training set; log p() is the conditional probability formula; θ is the neural network parameter;

[0056] S7-2 uses a self-critical policy gradient optimization method for training, sampling the output sequence y from the probability distribution of all time steps in the decoding operation S5. * Its reward value is r(y) * The output sequence y is obtained by selecting the output probability distribution with the largest value across all time steps. a Its reward value is r(y) a );

[0057] S7-3, Minimize the loss function L RL The formula is as follows:

[0058]

[0059] in, This represents the sequence sampled from the summary sequence y' at time step t; T represents the length of the summary sequence.

[0060] The model combines maximum likelihood loss L ML and reinforcement learning training loss function L RL The common learning objective function L is given by the following formula:

[0061] L=βL ML +(1-β)L RL

[0062] Where β is the balance factor.

[0063] Compared with the prior art, the significant advantages of this invention are as follows:

[0064] 1. This invention uses graph neural networks to capture knowledge from text documents and pre-trains the BERT semantic features model. Through knowledge extraction and representation, it enhances the influence of structural information on the text, promoting the understanding and analysis of the semantic feature representation of the text.

[0065] 2. This invention adds two parts to the Transformer decoder: a knowledge attention layer + a first normalization layer and a semantic attention layer + a second normalization layer, to capture knowledge and semantic information, thereby better enabling the interaction between textual knowledge and semantic information. Attached Figure Description

[0066] Figure 1 This is a schematic diagram of the module composition of the present invention;

[0067] Figure 2 This is a flowchart of the present invention;

[0068] Figure 3 Graph of attention instance update process when M=3heads. Detailed Implementation

[0069] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0070] This invention provides a summarization method based on knowledge and semantic information enhancement. This method constructs three important modules: an encoding module, a decoding module, and a generation module, as detailed below. Figure 1 The diagram shown is a schematic representation of the module composition of the present invention.

[0071] Encoding Module: The encoder module is constructed using a BERT pre-trained language model and a graph attention network. The BERT pre-trained language model captures semantic information from the text, while the graph attention network acquires contextual knowledge, fully mining the deep knowledge and semantic features in the text document.

[0072] Decoding Module: Based on the traditional Transformer decoder structure, knowledge attention and semantic attention mechanisms are introduced to integrate contextual semantic information and key knowledge information to enhance text readability and generate summaries that are concise and coherent; a pointer network is introduced to generate generation probabilities from an existing vocabulary to prompt the decoder which words in the original text are more important during the generation process.

[0073] Generation module: In the training phase, reinforcement learning theory and algorithms are introduced to achieve augmented training, and training is carried out using self-criticism policy gradient optimization.

[0074] Figure 2 The flowchart of this invention is as follows:

[0075] S1. For two publicly available automatic text summarization datasets, CNN / Daily Mail and Xsum, the StanfordOpenIE open-source tool is used to extract a set of knowledge triples for each text document in the dataset, and the set of knowledge triples is used to construct a contextual knowledge graph G for the text document.

[0076] Knowledge triples in text documents are typically represented by SRO triples.<Subject,Relation,Object> In this representation, Subject represents the subject, Relation represents the relation, and Object represents the object. The set of knowledge triples for a document is represented as S = {S1, S2, ..., S...} n}, where n represents the number of knowledge triples contained in the document.

[0077] The context knowledge graph G is a multi-relation graph composed of entity nodes and edges representing the relationships between nodes. It uses the Subject and Object from the SRO triple as nodes and Relation as edges representing the relationships between nodes. This leads to the construction of a multi-relation graph, resulting in the contextual knowledge graph G = (V, E) of the text document, where entity nodes V is the set of Subjects and edges E is the set of Relations.

[0078] S2, using a graph attention network, update the feature vector of node i in the context knowledge graph G one by one to obtain the updated context knowledge feature representation R. g Graph attention networks iteratively update the representation of each node by aggregating the representations of its neighboring nodes j, thus achieving adaptive allocation of weights for different neighboring nodes. The specific steps are as follows:

[0079] S2-1, taking any node i in the context knowledge graph G as an example, its neighboring nodes j∈N i Calculate the similarity coefficient e between i and j one by one. ij The specific calculation formula is as follows:

[0080] e ij =a([Wh i ||Wh j ]),j∈N i (1)

[0081] Among them, h i h is the initialization vector of i; j The initialization vector for j; W represents the shared parameters of the neural network, Wh i For h i Perform feature mapping; Wh j For h j Perform feature mapping; [·||·] is the concatenation operation; a(·) is the mapping function; e ij Let N be the similarity coefficient between i and j. i Let i be the neighborhood of node i.

[0082] S2-2, the similarity coefficient e between nodes i and j ijNormalization is performed using the LeakyReLU() activation function to normalize the coefficients, and the attention coefficients α for i and j are output. ij As shown in formula (2):

[0083]

[0084] Where exp(.) is the exponential function; LeakyReLU() is the activation function that normalizes the coefficients; n represents the neighboring nodes other than i and j; N i e is the neighborhood of node i; in Let be the similarity coefficient between i and n.

[0085] S2-3, using the M-head attention mechanism to adjust the attention coefficients α of i and j. ij The algorithm updates the vector and performs a weighted summation operation on the M attention results, outputting a new feature vector h after aggregating information from i and j. i Then, the M-head attention mechanism is used to focus on h. i 'Enhance the vector h' i (M). Details are as follows:

[0086] Based on the attention coefficients α of i and j ij The features of i and j are weighted, summed, and aggregated. The specific calculation formula is as follows:

[0087]

[0088] Where W represents the shared parameters of the neural network; σ(·) is the activation function.

[0089] The new feature vector h′ is obtained by aggregating information from i and j using the M-head attention mechanism. i Enhancement is performed to obtain vector h′ i (M), the specific formula is as follows:

[0090]

[0091] Where M represents the attention of the Mth head; || represents the splicing operation.

[0092] Figure 3 As shown, this is the process of updating the attention coefficients when M=3 heads. Node V1 updates using information from its neighbors V2-V6, connects the three independent attention coefficients, and takes the average value to output V1'.

[0093] S3, using the pre-trained language model BERT, obtain the feature representation R of each text document from the two automatic text summarization datasets, CNN / Daily Mail and Xsum, obtained in step S1. t .

[0094] The pre-trained language model BERT is a language model trained using a large amount of unlabeled text in an unsupervised manner, and it performs excellently in various language understanding tasks such as classification, entity recognition, and machine reading comprehension. This pre-trained language model BERT consists of multiple stacked Transformer decoder layers, which can better capture bidirectional relationships in sentences; each layer contains a multi-head self-attention layer and a feedforward layer, effectively solving the problem of long-term word dependencies.

[0095] The principle of the pre-trained language model BERT is as follows: The pre-trained language model BERT adds two special symbols, [CLS] and [SEP], to the input text. [CLS] is embedded at the beginning of each sentence, and its feature representation is used to aggregate the semantic information of the entire sentence; [SEP] is embedded at the end of the sentence as a marker of sentence boundaries. The resulting sequence is used as the input to the model, and the output of the last layer is used as the text document feature representation R. t .

[0096] Based on the above analysis, the specific steps of S3 are as follows:

[0097] S3-1, the multi-head self-attention layer performs a linear transformation on the feature vector input to the pre-trained language model BERT, generating three vectors: the lookup vector Q, the content vector K, and the lookup vector V; the attention vector is then calculated based on these three vectors, using the following formula:

[0098]

[0099] Where Attention() is the attention module function for a single head; T is the matrix transpose; d k The number of columns in the Q and K matrices is the vector dimension; Softmax() is the normalization exponential function, which is an activation function used for multi-class classification problems.

[0100] The Softmax() function can be used to compress the length L of any real vector into the range [0, 1], with the sum of all elements in the vector being 1. The specific formula is as follows:

[0101]

[0102] Where e is an exponential function.

[0103] S3-2, since a single calculation of the attention mechanism cannot fully capture multiple pieces of information, the attention mechanism needs to be calculated many times, which is called a multi-head attention mechanism. The specific formula is as follows:

[0104]

[0105]

[0106] Among them, Multihead() is the multi-head attention function; Concat() is the concatenation function; Attention() is the attention module function for a single head; head m This is the output of the m-th head; Let d be the weight matrix for the m-th head, where d = {Q, K, V}; Let Q be the weight matrix of the search vector Q for the m-th head; Let K be the weight matrix of the content vector K of the m-th header; Let V be the weight matrix of the vector V being searched for at the m-th head.

[0107] S3-3, after the forward feedback layer maps the MultiHead(Q,K,V) vectors, the text document feature representation R is obtained. t .

[0108] S4, using an improved Transformer decoder to represent contextual knowledge features R g and text document feature representation R t Decoding is performed to obtain the summary sequence vector S. t The improved Transformer decoder consists of four modules from bottom to top: a multi-head attention layer + residual and normalization layer, a knowledge attention layer + first normalization layer, a semantic attention layer + second normalization layer, and a feedforward neural network + third normalization layer. Skip connections are added between the modules to prevent information loss during the process.

[0109] The specific steps are as follows:

[0110] S4-1, Improved Transformer Decoder with token L of the Lth summary word t As input, the data passes through a multi-head attention layer (as in steps S3-1 to S3-2). Then, the output of the multi-head attention layer serves as the input to the first normalization layer, which transforms the data into data with a mean of 0 and a variance of 1 (i.e., the output value of the first normalization layer).

[0111] S4-2, representing contextual knowledge features in R g As the lookup vector Q in the knowledge attention mechanism, the output of the first normalization layer is used as the content vector K and the lookup vector V. Attention is calculated and normalized to obtain the output value of the second normalization layer.

[0112] S4-3, representing the text document features Rt As the lookup vector Q in the semantic attention mechanism, the output value of the second normalization layer is used as the content vector K and the lookup vector V to realize semantic attention calculation and normalization, and obtain the output value of the third normalization layer.

[0113] S4-4, the output of the third normalization layer is fed into the feedforward neural network for linear mapping and normalization. The decoder output is the summary sequence vector S. t .

[0114] S5 introduces the generation probability P through the pointer network model. gen ∈[0,1], for the summary sequence vector S t and text document feature representation R t Perform correlation calculations to obtain the weight distribution C. t .

[0115] During decoding, the model may generate new words. If some of these new words are not in the vocabulary, the out-of-vocabulary (OV) word problem will occur. To address this issue, a pointer network is needed to compute the attention at each step, indicating to the decoder which words in the original text are more important during generation. Each step of the pointer network computation yields a generation probability P. gem ∈[0, 1], which represents the probability of generating a word from the vocabulary. In this embodiment, the input to the pointer network includes the text document feature representation R at time t. t Abstract sequence vector S t And the token L of the Lth summary term t .

[0116] For the summary sequence vector S t and text document feature representation R t Perform correlation calculations to obtain the weight distribution C. t The specific calculation formula is as follows:

[0117] α e =softmax(V T ·tanh(W t R t +W s S t +b)) (8)

[0118]

[0119] Where tanh() is the activation function; softmax() is the normalization exponential function; V, W t W s b and are the parameters learned during the training of the pointer network model; T is the matrix transpose; the weight distribution C tThe probability distribution on the original text document indicates certain words that the decoder should focus on, which are used to extract important semantic information and facilitate the generation of text summaries.

[0120] S6. For each generated word or phrase, the pointer network model will calculate the position of the word or phrase in the original text and encode the position information into a probability distribution P. gen This allows you to select from which position to generate the next word or phrase; repeat steps S4 and S5 until the length of the generated summary reaches the preset value, resulting in a complete summary sequence y'.

[0121] probability distribution P gen The specific calculation formula is as follows:

[0122]

[0123] in b gen It is the learnable parameter matrix; σ(·) is the activation function; C t S represents the weighted distribution; t For the abstract sequence vector; L t Input the token L of the Lth summary word into the decoder t ;

[0124] Step S7: To further optimize the quality of the generated summaries, this invention introduces a reinforcement learning algorithm for enhanced training. In this process, reinforcement learning theory is used to calculate the reward value r(y') by calculating the similarity index ROUGE between the generated summary sequence y' and the reference summary sequence y. Then, gradient optimization is performed on the pointer network and Transformer decoder to improve the generation quality and the summary similarity index ROUGE.

[0125] The overall model is viewed as an agent interacting with the environment. Words in the generated summary are considered actions taken by the agent. The environment refers to the summary sequence generated by the current model. The model's generated complete summary sequence y' is compared with the reference summary y using the ROUGE metric to obtain a reward value r(y'). The entire summary generation environment is adjusted based on this reward value. The specific implementation steps are as follows:

[0126] S7-1, The goal during the training phase is to minimize the maximum likelihood loss L. ML The specific formula is as follows:

[0127]

[0128] Where x represents the text documents in the two publicly available automatic text summarization datasets CNN / Daily Mail and Xsum, y represents the reference summary in the dataset, D is the training set (80% of the dataset is used for training and 20% for testing), log p() is the conditional probability formula, and θ is the neural network parameter.

[0129] S7-2, Gradient Optimization Stage: Training is performed using a self-critical policy gradient optimization method, sampling the output sequence y from the probability distribution of all time steps in the decoding operation S5. * Its reward value is r(y) * The output sequence y is obtained by selecting the output probability distribution with the largest value across all time steps. a Its reward value is r(y) a );

[0130] S7-3, Minimize the loss function L RL The formula is as follows:

[0131]

[0132] Where log p() is the conditional probability formula; This represents the sequence sampled from the summary sequence y' at time step t; T represents the length of the summary sequence.

[0133] Minimize loss function L RL Equivalent to maximizing the sequence y * The conditional likelihood estimate yields a reward higher than the baseline output sequence y. a The reward increases the model's overall expected reward. Reinforcement learning training methods may reduce the readability and smoothness of the generated summaries; the model incorporates maximum likelihood loss L... ML and reinforcement learning training loss function L RL The common learning objective function L is given by the following formula:

[0134] L=βL ML +(1-β)L RL (13)

[0135] Where β is the balance factor, used to adjust the maximum likelihood loss L. ML and reinforcement learning training loss L RL The proportion of losses.

[0136] According to the above steps, this invention generates a probability distribution representing the probability of each word or sub-word appearing in the summary from the input dataset text information, and finally selects the word or sub-word with the highest probability as part of the summary, generating a sequence as the output.

[0137] Table 1 shows the experimental results of this invention on the CNN / Daily Mail dataset. Compared with existing summary generation methods such as Lead-3, SummaRuNNer, POINTGEN+COV, Bottom-up, DCA, BEAR, FASUM, and RL_intra-att, this invention has certain advantages.

[0138] Table 1 Experimental results on the CNN / Daily Mail dataset

[0139]

[0140] ROUGE refers to the machine summarization evaluation metric, which is a test metric for measuring the performance of automatic text summarization and sentence summarization. Rouge-1 and Rouge-2 are actually calculated by splitting the model-generated results and standard results into N-grams and then calculating the recall rate. Rouge-L is calculated using the longest common subsequence, and Rouge-AVG is calculated using the average sequence.

Claims

1. A summarization method based on knowledge and semantic information enhancement, characterized in that, Includes the following steps: S1. Use the Stanford OpenIE open-source tool to extract the knowledge triple set of each text document in the publicly available automatic text summarization dataset, and use the triple set to construct the contextual knowledge graph G of the text document; S2, using a graph attention network, update the feature vector of node i in the context knowledge graph G one by one to obtain the updated context knowledge feature representation. R g ; S3, using the pre-trained language model BERT, obtains the feature representation of each text document in the automatic text summarization dataset. R t ; S4, using an improved Transformer decoder to represent contextual knowledge features. R g and text document feature representation R t Decode the data to obtain the summary sequence vector. S t The improved Transformer decoder comprises four modules from bottom to top: a multi-head attention layer + residual and normalization layer, a knowledge attention layer + first normalization layer, a semantic attention layer + second normalization layer, and a feedforward neural network + third normalization layer; the modules of the improved Transformer decoder are interconnected. S4-1, Improved Transformer Decoder with token of the Tth summary word T t As input, it passes through a multi-head attention layer; then, the output of the multi-head attention layer serves as the input to the first normalization layer, which transforms it into data with a mean of 0 and a variance of 1. S4-2, Representing contextual knowledge features R g As the lookup vector Q in the knowledge attention mechanism, the output of the first normalization layer is used as the content vector K and the lookup vector V. Attention is calculated and normalized to obtain the output value of the second normalization layer. S4-3, Representing Text Document Features R t As the lookup vector Q in the semantic attention mechanism, the output value of the second normalization layer is used as the content vector K and the lookup vector V to realize semantic attention calculation and normalization, and obtain the output value of the third normalization layer. In step S4-4, the output of the third normalization layer is fed into the feedforward neural network for linear mapping and normalization. The decoder output is a summary sequence vector. S t ; S5 introduces generation probability through a pointer network model. For the summary sequence vector S t and text document feature representation R t Perform correlation calculations to obtain the weight distribution. C t ; S6. The pointer network model is used to calculate the position of the generated word or phrase in the original text, and the position information is encoded into a probability distribution. P gen Repeat steps S4 and S5 until the length of the generated abstract reaches the preset value, thus obtaining a complete abstract sequence. y’ ; S7 utilizes reinforcement learning theory and algorithms to calculate the generated summary sequence. y’ The reward value r(y') is calculated using the similarity metric ROUGE between the sequence and the reference summary y; then, gradient optimization is performed on the pointer network and the improved Transformer decoder.

2. The summary generation method based on knowledge and semantic information enhancement as described in claim 1, characterized in that, The specific steps of step S2 are as follows: S2-1, Calculate the similarity coefficient between node i and its neighbor node j in the context knowledge graph G one by one. e ij The calculation formula is as follows: ; in, h i for i The initialized representation vector; h j Let j be the initialized representation vector; W is the shared parameter of the neural network. Wh i To h i Perform feature mapping; Wh j To h j Perform feature mapping; For splicing operations; For mapping functions; e ij for i and j The similarity coefficient; Let i be the neighborhood of node i; S2-2, Similarity coefficients for nodes i and j e ij Perform a normalization operation to obtain the attention coefficients of nodes i and j. The specific formula is as follows: ; in, exp (.) is the exponential function; LeakyReLU() is the activation function that normalizes the coefficients; n represents the neighboring nodes other than i and j; e in Let be the similarity coefficient between i and n; S2-3, using the M-head attention mechanism to adjust the attention coefficients of i and j. Perform an update and a weighted summation of the M attention results, then output the result. i The new feature vector after aggregating j information Then, the M-head attention mechanism is used to... Enhancement yields vectors .

3. The summarization method based on knowledge and semantic information enhancement as described in claim 2, characterized in that, The new feature vector The expression is as follows: ; Where W represents the shared parameters of the neural network; For activation functions; Using the M-head attention mechanism Enhancement yields vectors The calculation formula is as follows: ; Where M represents the Mth head attention, This is for splicing operations.

4. The summarization method based on knowledge and semantic information enhancement as described in claim 1, characterized in that, The specific calculation steps for step S3 are as follows: S3-1, the multi-head self-attention layer performs a linear transformation on the feature vector input to the pre-trained language model BERT, generating three vectors: the lookup vector Q, the content vector K, and the lookup vector V; and calculates the attention vector based on these three vectors, as shown in the following formula: ; Where Attention() is the attention module function for a single head; T is the matrix transpose; d k is the number of columns in the Q and K matrices; Softmax() is the normalization exponential function; The Softmax() function can be used to compress the length L of any real vector into the range [0, 1], with the sum of the elements in the vector being 1. The specific formula is as follows: ; Where e is an exponential function; S3-2, Repeat step S3-1 multiple times to obtain the multi-head attention vector. The specific formula is as follows: ; ; Where Multihead() is the multi-head attention function; Concat() is the concatenation function; and Attention() is the attention module function for a single head. This is the output of the m-th head; Let m be the weight matrix of the m-th head, where d={Q, K, V} ; Let Q be the weight matrix of the search vector Q for the m-th head; Let K be the weight matrix of the content vector K of the m-th header; Let V be the weight matrix of the vector V being searched for in the m-th head; S3-3, after passing through the forward feedback layer... Vector mapping yields the feature representation of the text document. R t .

5. The summarization method based on knowledge and semantic information enhancement as described in claim 1, characterized in that, Step S5 involves processing the summary sequence vector. S t and text document feature representation R t Perform correlation calculations to obtain the weight distribution C. t The specific calculation formula is as follows: ; ; Where tanh() is the activation function; softmax() is the normalization exponential function; V, W t 、W s b and are the parameters learned during the training of the pointer network model; T is the matrix transpose.

6. The summarization method based on knowledge and semantic information enhancement as described in claim 5, characterized in that, The probability distribution of step S6 P gen The specific calculation formula is as follows: ; in It is a learnable parameter matrix; It is an activation function; C t For weighted distribution; L t Input the token of the Lth summary word into the decoder L t .

7. The summarization method based on knowledge and semantic information enhancement as described in claim 1, characterized in that, The specific implementation steps of step S7 are as follows: S7-1, The goal during the training phase is to minimize the maximum likelihood loss. The specific formula is as follows: ; Where x is a text document in the automatic text summarization dataset, y is a reference summary in the dataset, D represents the training set, and log p() is the conditional probability formula; For neural network parameters; S7-2 uses a self-critical policy gradient optimization method for training, and the output sequence is sampled from the probability distribution of all time steps in the decoding operation S5. Its reward value is The output sequence is obtained by selecting the output probability distribution with the largest value across all time steps. Its reward value is ; S7-3, Minimize the loss function L RL The formula is as follows: ; in, This represents the sequence sampled from the summary sequence y' at time step t; T represents the length of the summary sequence. The model combines maximum likelihood loss and reinforcement learning training loss function L RL The common learning objective function L is given by the following formula: ; in, It is a balancing factor.