A text summarization generation method based on enhancing factual consistency

Abstract

The present invention relates to the technical field of natural language processing. It discloses a method for generating text summaries based on enhanced fact consistency, which solves the problem that the prior art ignores the different contributions of different importance levels among fact triples to the final summary result, and improves the credibility of the generated text summaries. The present invention uses the Transformer architecture to construct a sequence-to-sequence text summary generation model, and introduces a fact attention module between the feed-forward network module and the cross-attention module of the decoder, which is used to calculate the influence of each fact triple on the generated word based on the attention vector of each fact triple and the word vector of the generated word output by the cross-attention module, and update the word vector of the generated word accordingly; the attention vector of the fact triple is calculated based on the fact triple encoding vector through the self-attention mechanism.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and particularly relates to a text summary generation method based on enhanced fact consistency. Background Art

[0002] With the wide application of the mobile Internet and the rapid popularization of intelligent terminal devices, people can create and publish content on the Internet and can also conveniently obtain publicly available Internet information, which not only makes people's daily life and work more convenient, but also greatly enriches people's spiritual life. However, network text information often has a long length and is difficult to understand. Compared with short video information, it has the characteristics of long reading time and lack of prominent key points, which is difficult to arouse readers' interest in today's fast-paced life and also reduces readers' reading experience to a certain extent.

[0003] The generative text summary model aims to convert text or a text collection into a short summary containing key information. The emergence of this technology has solved the problem of information overload. The generative summary model usually implements a sequence-to-sequence task based on an encoder-decoder framework and uses the powerful mapping ability of a pre-trained model and network architecture based on Transformer to generate a short and highly fluent summary. However, the way of outputting words based on probability by the generative summary model may still lead to the generation of words or phrases that are inconsistent with the original meaning, resulting in factual errors in the summary result, affecting the credibility of the summary result, thus limiting the popularization and implementation of the generative summary model and further reducing the research and application value of the generative summary model.

[0004] Therefore, how to solve the problem of factual errors that easily occur in the generative summary model and effectively enhance the fact consistency of the summary result is one of the hot topics of concern in this field currently.

[0005] Enhancing fact consistency aims to reduce the probability of factual errors occurring in the generative summary result. The research on the fact consistency of the automatic text summary technology was first started by Ziqiang Cao et al. in 2018. They found that there are approximately 30% of errors or unverifiable facts in the summary results generated by mainstream generative summary models. Subsequently, Ziqiang Cao et al. proposed the FTSum model, which encodes fact triples using an additional encoder and concatenates them with the sentence encodings of the original document, enabling the model to notice the influence of fact triples. However, it ignores the problem that different degrees of importance among fact triples contribute differently to the final summary result.

[0006] Gunel B et al. proposed to construct a knowledge graph from Wikipedia data and introduce entity-level knowledge into the encoding-decoding framework to guide the model to generate correct facts. However, it ignores the problem that the traditional encoding-decoding framework is prone to output words that do not conform to the original text facts during the decoding process, thus generating factual errors. Summary of the Invention

[0007] The technical problem to be solved by the present invention is: to propose a method for generating text summaries enhanced by fact consistency, to solve the problem that the prior art ignores the different contributions of different importance levels among fact triples to the final summary result, and to improve the credibility of the generated text summaries.

[0008] The technical solution adopted by the present invention to solve the above technical problems is:

[0009] A method for generating text summaries enhanced by fact consistency uses a Transformer architecture to construct a sequence-to-sequence text summary generation model. The decoder of the Transformer architecture includes a self-attention module, a cross-attention module, and a feed-forward network module connected in sequence. A fact attention module is introduced between the feed-forward network module and the cross-attention module of the decoder;

[0010] Define fact triples and represent them as F i =<s i ,r i ,o i >>, where s is the subject word, r is the relationship modifier, o is the object word, the subscript i is the serial number of the fact triple, and each word in any fact triple comes from the same sentence in the original text; the fact attention module takes the attention vectors of each fact triple in the original text and the third word vector sequence output by the decoder cross-attention module as inputs, and based on the cross-attention mechanism, obtains the influence coefficients of all fact triples on each generated word, and updates the third word vector sequence based on the influence coefficients of all fact triples on each generated word to obtain a fourth word vector sequence as the input of the decoder feed-forward network module;

[0011] The attention vector of the fact triple is calculated according to the following steps:

[0012] A1. Use natural language processing tools to process the original text, extract the fact triple F i , and construct a fact triple set F; then, for each fact triple F iThe words contained therein construct nodes and connection edges, map the set of fact triples F to a graph, and use a graph neural network to calculate the node vectors of each node in the graph; then, splice the node vectors of the nodes contained in each fact triple to obtain the encoded features of each fact triple Among them and respectively represent the fact triple F i the contained s i 、r i and o i the node vectors of the corresponding nodes;

[0013] A2. Input the encoded features of the fact triples into a recurrent neural network to obtain the vector representation z of each fact triple that fuses the semantic information of its previous and subsequent fact triples i ;

[0014] A3. Use the vector representation z of each fact triple i , and based on the self-attention mechanism, obtain the attention vector of each fact triple.

[0015] Furthermore, in step A2, the recurrent neural network is a Bi-LSTM network; based on the encoded features of the fact triples, use the Bi-LSTM network to respectively obtain the forward hidden layer vector and the backward hidden layer vector of each fact triple. Then, splice the forward hidden layer vector and the backward hidden layer vector to obtain the vector representation of each fact triple that fuses the semantic information of its previous and subsequent fact triples

[0016] Furthermore, in step A1, using a graph neural network to calculate the node vectors of each node in the graph includes:

[0017] First, use a pre-trained model to initialize the node vectors of each node in the graph;

[0018] Then, according to the following formula, use the GCN network to update the node vectors of each node in the graph:

[0019]

[0020] where ReLU represents the activation function, H l and H l+1 respectively represent the outputs of the l-th and (l + 1)-th layers of the GCN network, A represents the adjacency matrix of the graph, represents the degree matrix, and W l represents the weight matrix of the l-th layer of the GCN network.

[0021] Further, in step A1, it further includes:

[0022] First, using the decision tree model of the natural language processing tool, based on the relationship types it contains, classify the relationships between each fact triple F i ; then, based on the relationship classification results, obtain the relationship triples between fact triples F i , and construct a relationship triple set R;

[0023] Then, concatenate the encoding features and relationship vectors of the fact triples included in each relationship triple to obtain the encoding feature R j of the relationship triple = [h 1j , cr j , h 2j , where h 1j and h 2j respectively represent the encoding features of the fact triples included in the j-th relationship triple, and cr j represents the relationship vector obtained based on the pre-trained model for the relationship included in the j-th relationship triple;

[0024] After that, use the encoding features of each relationship triple and the encoding features of each fact triple to perform cross-attention calculation, and update the encoding features of each fact triple based on the calculation results. Take the updated encoding feature h' i of the fact triple as the input of step A2.

[0025] Further, using the encoding features of each relationship triple and the encoding features of each fact triple to perform cross-attention calculation, and updating the encoding features of each fact triple based on the calculation results specifically includes:

[0026] α ij = h i * R j

[0027]

[0028]

[0029] where α ij represents the correlation between the i-th fact triple and the j-th relationship triple, and β ij represents the correlation weight between the i-th fact triple and the j-th relationship triple, and R represents the relationship triple set.

[0030] Further, in step A3, using the vector representation z i of each fact triple, based on the self-attention mechanism, obtain the attention vector of each fact triple; the self-attention mechanism is multi-head self-attention, and its calculation process includes:

[0031] A31. The transformation matrices W k1 , W k2 and W k3 , transform the vector representation z i of the fact triple into Query values Key values and Value values

[0032]

[0033]

[0034]

[0035] A32. Based on and , perform the attention calculation for each attention head:

[0036]

[0037] where d k is the dimension of ;

[0038] A33. Aggregate the attention calculation results of each attention head to obtain the attention vector Z i :

[0039]

[0040] where K is the number of attention heads;

[0041] A34. Use linear transformation to change the dimension of the attention vector Z i to the dimension of the vector representation z i to obtain the attention vector r i of this fact triple;

[0042] r i = Z i * W 0

[0043] where W 0 is the linear transformation matrix.

[0044] Furthermore, at each time step, the decoding process of the decoder is as follows:

[0045] B1. Based on the generated words obtained before the current time step t, use the pre-trained model to obtain their word embeddings, and form the first word vector sequence of the generated words in the generation order according to the word embeddings of each generated word;

[0046] B2. Input the first word vector sequence into the self-attention module of the decoder. First, update the first word vector sequence based on the self-attention mechanism. Then, perform a residual connection on the input first word vector sequence and the updated first word vector sequence, and normalize it to obtain the second word vector sequence of the generated word.

[0047] B3. Input the second word vector sequence into the cross-attention module of the decoder. First, construct the Key value and Value value with the context vector generated by the encoder of the Transformer architecture based on the original text, and output the attention distribution a of each word in the original text t , construct the Query value with the second word vector sequence, and combine the attention distribution a t , update the second word vector sequence. Then, perform a residual connection on the input second word vector sequence and the updated second word vector sequence, and normalize it to obtain the third word vector sequence of the generated word.

[0048] B4. Input the third word vector sequence into the fact attention module of the decoder. First, based on the attention vectors of each fact triple and the third word vector sequence, use the cross-attention mechanism to obtain the influence coefficients of all fact triples on each generated word. Then, perform a residual connection on the input third word vector sequence and the influence coefficients of all fact triples on each generated word according to the word correspondence, and normalize it to obtain the fourth word vector sequence of the generated word.

[0049] B5. Input the fourth word vector sequence into the feed-forward network module of the decoder. First, use the feed-forward network to calculate according to the formula p t = max(0, ln t W1 + b1)W2 + b2. Then, perform a residual connection on the output p t of the feed-forward network layer and the fourth word vector sequence ln t , and normalize it to obtain the output of the decoder where W1, W2, b1, and b2 are all learnable parameters of the feed-forward network.

[0050] Further, based on the attention vectors of each fact triple and the third word vector sequence, use the cross-attention mechanism to obtain the influence coefficients of all fact triples on each generated word, specifically including:

[0051]

[0052]

[0053]

[0054] Among them, Denotes the word vector of the m-th generated word in the third word vector sequence; α im Denotes the relevance between the m-th generated word and the i-th fact triple, β im Denotes the relevance weight between the m-th generated word and the i-th fact triple, u m Denotes the influence coefficient of all fact triples on the m-th generated word, and F denotes the set of fact triples.

[0055] Furthermore, the text summary generation model further includes a pointer network;

[0056] At each time step, the processing process of the pointer network includes:

[0057] First, according to the following formula, using a linear layer, map the output of the decoder at the current time step t to the feature space of the word table:

[0058]

[0059] where, W vocab and b vocab Denote the learnable parameter matrices corresponding to the word table, and the word table is the word table of the pre-trained model;

[0060] Then, based on l t calculate the vocabulary distribution and the pointer probability

[0061]

[0062]

[0063] where, w gen and b gen Denote learnable parameters; Denotes the probability that the generated word at the current time step is generated from the word table, Denotes the output probability distribution of each word in the word table as the generated word at the current time step;

[0064] After that, based on and calculate the final probability distribution P t (w):

[0065]

[0066] where, P t(w) represents the probability that word w in the extended vocabulary is the generated word at the current time step t. The extended vocabulary includes the word list and all words contained in the original text. N represents the number of words in the original text, and n represents the serial number of the word in the original text. It is the attention distribution a of each word vector of the original text generated by the cross-attention module t The attention of the nth word in It represents the sum of the attentions of words in the original text that are w.

[0067] Furthermore, the text summary generation model further includes a beam search algorithm. At each time step, the processing process of the beam search algorithm includes:

[0068] First, use the final probability distribution P t (w) output by the pointer network at the current time step t to screen the top P words with the highest probability to construct candidate words;

[0069] Then, make a judgment. If the candidate words contain words contained in the original text, select the word contained in the original text with the highest output probability as the generated word at the current time step; otherwise, use the P candidate words and the generated words generated before the current time step to form P candidate summaries, and calculate the combined probability of each candidate summary according to the following formula, and select the candidate word corresponding to the candidate summary with the highest combined probability as the generated word at the current time step:

[0070] P Y = argmax(log(Π t p(y t |y1,y2,…,y t-1 )))

[0071] where Y represents the candidate summary, and p represents the output probability of the corresponding word.

[0072] The beneficial effects of the present invention are:

[0073] The present invention constructs a sequence-to-sequence text summary generation model using the Transformer architecture and improves the decoder of the Transformer architecture. A fact attention module is introduced between the feed-forward network module and the cross-attention module of the decoder. This module calculates the influence of each fact triple on the generated word based on the attention vector of each fact triple and the word vector of the generated word output by the cross-attention module in the decoder, and updates the word vector of the generated word accordingly; the calculation of the attention vector of the fact triple is obtained through the self-attention mechanism based on the fact triple encoding vector. Therefore, it can avoid the problem of ignoring the different contributions of different degrees of importance between fact triples to the final summary result, and can improve the fact consistency of text summary generation.

[0074] Furthermore, considering that in the case of long texts, there are also complex relationships between fact triples, and such complex relationships may also lead to different contributions of different fact triples to the final result. Therefore, in the process of calculating the encoding vector of fact triples, the influence of the complex relationships between different fact triples is also incorporated.

[0075] Furthermore, the present invention also introduces a pointer network. Through the pointer network, the output vector of the decoder at the current time step is mapped to the feature space of the vocabulary to obtain a mapping vector representation. According to the mapping vector representation, the probability that the generated word at the current time step is generated from the vocabulary and the output probability distribution of each word in the vocabulary as the generated word at the current time step are calculated, and then the probability that each word in the extended vocabulary composed of the vocabulary and the original text words finally serves as the generated word at the current time step is calculated; this avoids the factual error that may occur in the traditional decoding process of directly mapping the previous decoding information to the vocabulary to obtain the distribution probability of the current generated word, which may introduce words that do not belong to the original text but belong to the vocabulary in the generated summary. That is, by introducing the pointer network, the present invention tries to avoid generating words that do not belong to the original text but belong to the vocabulary, thereby reducing the probability of factual errors and further improving the factual consistency of text summary generation.

[0076] Furthermore, the present invention also introduces a beam search algorithm. Using the beam search algorithm, the probability that each word in the extended vocabulary output by the pointer network at the current time step finally serves as the generated word at the current time step is utilized to screen multiple words with the highest probability to construct candidate words. If the candidate words contain the words included in the original text, then the word included in the original text and with the highest output probability is selected as the generated word at the current time step; otherwise, multiple candidate summaries are constructed using the multiple candidate words and the generated words before the current time step, and the combined probability of each candidate summary is calculated, and the candidate word corresponding to the candidate summary with the highest combined probability is selected as the generated word at the current time step. That is, through the beam search algorithm, when the calculated candidate words do not contain the words in the original text, the candidate words and the generated words before the current time step are used to construct candidate summaries, and the combined probability of each candidate summary is calculated, so as to find a relatively approximate global optimal solution at the phrase level, thereby improving the rationality of the semantic combination of the generated word at the current time step and the generated words, and further improving the factual consistency of text summary generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 It is a principle framework diagram of text summary generation based on enhanced factual consistency in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0078] The present invention aims to propose a text summarization generation method based on enhanced factual consistency, solve the problem that the prior art ignores the different degrees of importance among factual triples and their different contributions to the final summary result, and improve the credibility of the generated text summary.

[0079] The text summarization generation method of the present invention constructs a sequence-to-sequence text summarization generation model using the Transformer architecture. The Transformer architecture is an Encoder-Decoder framework. In the encoding part, first, the input original text is tokenized to obtain the word sequence of the original text; then, through a pre-trained model, the word vectors of each word in the word sequence are obtained to form the word vector sequence of the original text; finally, the word vector sequence is input into the encoder of the Transformer architecture, and the context vector of the original text is obtained through encoding by the encoder. In the decoding part, based on the context vector obtained by encoding, the summary text is gradually generated, that is, the generated words are obtained one by one according to the time steps, and one word is decoded and generated at each time step.

[0080] The key of the present invention lies in the improvement of the decoder of the Transformer architecture. A factual attention module is introduced between the feed-forward network module and the cross-attention module of the existing decoder. That is, the decoder in the present invention includes a self-attention module, a cross-attention module, a factual attention module, and a feed-forward network module connected in sequence. The factual attention module calculates the influence of each factual triple on the generated word based on the attention vector of each factual triple and the word vector of the generated word output by the cross-attention module, and updates the word vector of the generated word accordingly.

[0081] Specifically, in the method of the present invention, at each time step, the decoding process of the decoder is as follows:

[0082] First, all the words generated before the current time step are passed through a pre-trained model to obtain the first word vector sequence of the generated words; then, the self-attention module is used to update the first word vector sequence, and a residual connection and normalization process are performed with the input first word vector sequence to obtain the second word vector sequence of the generated words; then, based on the cross-attention module, the original text context vector obtained by the encoding part is used to update the second word vector sequence, and a residual connection and normalization process are performed with the second word vector sequence to obtain the third word vector sequence of the generated words; then, based on the attention vector of each factual triple and the third word vector sequence, the factual attention module uses the cross-attention mechanism to obtain the influence coefficients of all factual triples on each generated word, and a residual connection and normalization process are performed in a word-by-word correspondence with the third word vector sequence to obtain the fourth word vector sequence of the generated words. The fourth word vector sequence is used as the input of the feed-forward network to obtain the decoding output vector of the current time step.

[0083] In the above decoding process, the distribution probability of the word at the current time step is obtained, which can be directly mapped to the word list to obtain the generated word. However, this method may introduce words that do not belong to the original text but belong to the word list in the generated abstract, and to a certain extent, factual errors may occur due to insufficient calculation accuracy of the distribution probability. To effectively alleviate this problem, the present invention introduces a pointer network. Through the pointer network, the decoding output vector at the current time step is mapped to the feature space of the word list to obtain a mapping vector representation. According to the mapping vector representation, the probability of generating the word at the current time step from the word list and the output probability distribution of each word in the word list as the generated word at the current time step are calculated, and then the probability of each word in the extended word list composed of all the words in the word list and the original text as the generated word at the current time step is calculated.

[0084] Furthermore, the method of the present invention also introduces a beam search method. By calculating the combination probability of the currently generated word and the already generated words, a relatively approximate global optimal solution is searched at the phrase level, improving the rationality of the semantic combination of the generated word at the current time step and the already generated words, and trying to avoid generating words that do not belong to the original text but belong to the word list, thereby further improving the factual consistency and credibility of text abstract generation. The beam search algorithm uses the probability of each word in the extended word list output by the pointer network at the current time step as the generated word at the current time step to screen multiple words with the highest probability to construct candidate words. If the candidate words contain the words included in the original text, the word included in the original text and with the highest output probability is selected as the generated word at the current time step; otherwise, multiple candidate abstracts are formed by using multiple candidate words and the generated words before the current time step, and the combination probability of each candidate abstract is calculated, and the candidate word corresponding to the candidate abstract with the highest combination probability is selected as the generated word at the current time step.

[0085] The method of the present invention will be further described below in conjunction with embodiments.

[0086] Embodiment:

[0087] In the method for generating a text abstract based on enhancing factual consistency in this embodiment, the text abstract generation model, as Figure 1 shown, is based on the Encoder-Decoder framework of the Transformer architecture. Compared with the prior art, the structure of its encoder Encoder remains unchanged, while in the decoder Decoder, a factual attention module is introduced. The improved decoder Decoder includes a self-attention module, a cross-attention module, a factual attention module, and a feed-forward network module.

[0088] Specific elaboration will be carried out from three aspects: generation of factual triple attention vectors, encoding and decoding processes, and model training.

[0089] 1. Generation of Fact Triple Attention Vectors

[0090] The attention vectors of fact triples are used to introduce the weights of fact triples in the subsequent decoding process and calculate the different contributions of each fact triple to the generated words, including the following processes:

[0091] A1. Extracting Fact Triples

[0092] In this step, using natural language processing tools to process the original text, fact triples F are extracted from the original text i , and a set of fact triples F is constructed. In this embodiment, the natural language processing tool is the StanfordNLP toolkit.

[0093] Specifically, first, the original text D is input into the StanfordNLP toolkit for word segmentation to obtain the word sequence {x1, x2,..., x b ,..., x N} of the original text, where x n represents the nth word in the text, n represents the serial number of the word in the original text, and N represents the number of words in the original text.

[0094] Then, using the StanfordNLP toolkit, through processes such as part-of-speech tagging, syntactic constituent analysis, coreference resolution, and relation extraction, the fact triples F extracted from the original text D are output i , and a set of fact triples F is constructed. The set F = {F1, F2,..., F i ,..., F I}, and the fact triple F i = <s i , r i , o i >, where s is the subject word, r is the relation modifier, o is the object word, the subscript i is the serial number of the fact triple, I represents the number of fact triples, and for any fact triple F i in which s i , r i and o i all come from the same sentence. For example, <Kobe, likes, Coke> is a fact triple.

[0095] A2. Extracting Relation Triples

[0096] In this step, applying the Rhetorical Structure Theory, complex relationships between fact triples are extracted based on the fact triples. Further, using the decision tree model of natural language processing tools, the relationships between each fact triple F i are classified based on the relationship types they contain; then, based on the relationship classification results, relationship triples of fact triples F i are obtained, and a relationship triple set R is constructed.

[0097] In this embodiment, the natural language processing tool is the StanfordNLP toolkit. Since the RST tree model in the Rhetorical Structure Theory has been built into the StanfordCoreNLP toolkit, it can be directly called. Specifically, the original text D = {x1, x2, …, x n , …, x N} and the fact triple set F = {F1, F2, …, F i , …, F I} are input into the StanfordCoreNLP toolkit. Using the StanfordCoreNLP toolkit, the relationships between each fact triple F i are classified based on the relationship types included in its RST tree model. Currently, there are 23 relationships included in the RST tree model; then, based on the relationship classification results, relationship triples R i of fact triples F j are obtained, and a relationship triple set R is constructed.

[0098] The set R = {R1, R2 …, R j , … R J}, and the relationship triple R j = <F 1j , CR j , F 2j >, where F 1j and F 2j respectively represent the fact triples included in the j-th relationship triple, and CR j represents the relationship included in the j-th relationship triple. The subscript j is the serial number of the relationship triple, and J represents the number of relationship triples. For example: The fact triples <Xiaoming, was born in, Sichuan> and <Xiaoming, masters, Sichuan dialect> are in a "background" relationship, and <Jack, likes, cola> and <Jack, suffers from, tooth decay> are in a "causal" relationship.

[0099] A3. Vector Encoding

[0100] In this step, the set of factual triples is mapped to a graph structure, and the graph convolutional network is used to encode the nodes of the graph structure and incorporate complex relationship information to obtain the encoded vectors of each factual triple.

[0101] Specifically, it includes the following processing procedures:

[0102] A31. Map the set of factual triples to a graph

[0103] To encode using the graph convolutional neural network, it is necessary to convert the unstructured set of factual triples F into a structured graph. For each factual triple F i =<s i , r i , o i (>, a subject node s i , a relation node r i and an object node o i are created; the subject node and the relation node are connected by an edge, and the object node and the relation node are also connected by an edge, and the edge does not represent any information and only indicates the existence of a connection between the two. That is, nodes are constructed from the words contained in each factual triple F i , and edges are constructed for the connections, mapping the set of factual triples F to a graph.

[0104] A32. Node initialization

[0105] In this step, the pre-trained Bert model is used to initialize the node vectors of each node in the graph.

[0106] A33. Node update

[0107] In this step, the GCN network is used to update the node vectors of each node in the graph:

[0108] For the GCN network, for any node, it will collect the information of its surrounding neighbor nodes and use this information to transform its own characteristic information. This process will be repeated multiple times, and the specific calculation process is as follows:

[0109]

[0110] Among them, ReLU represents the activation function, H l and H l+1 represent the outputs of the l-th and (l + 1)-th layers of the GCN network respectively, A represents the adjacency matrix of the graph, represents the degree matrix, and W l represents the weight matrix of the l-th layer of the GCN network.

[0111] After the aggregation calculation by the GCN network, each node contains information of surrounding nodes and even nodes two or three hops away, effectively capturing the association information and containing the factual relationship to a certain extent. In this embodiment, the GCN network has two layers.

[0112] A34. Construct the encoded features of factual triples

[0113] In this step, the node vectors of the nodes included in each factual triple are concatenated to obtain the encoded features of each factual triple Among them, and respectively represent the node vectors of s i , r i , and o i corresponding to the factual triple F i

[0114] A35. Update the encoded features of factual triples

[0115] To effectively integrate the influence of the complex relationships between factual triples on the information contained in different factual triples, in this step, the encoded features of each relationship triple are first constructed, and then based on the encoded features of the relationship triples, the attention mechanism is used to update the encoded features of the factual triples.

[0116] Specifically, first, the encoded features of the factual triples and the relationship vectors included in each relationship triple are concatenated to obtain the encoded features R j =[h 1j , cr j , h 2j , h 1j and h 2j respectively represent the encoded features of the factual triples included in the j-th relationship triple, and cr j represents the relationship vector obtained based on the pre-trained model for the relationship included in the j-th relationship triple.

[0117] Then, using the encoded features of each relationship triple and the encoded features of each factual triple, cross-attention calculation is performed, and based on the calculation results, the encoded features of each factual triple are updated to obtain the updated encoded features h' i of the factual triples, and the calculation process is as follows:

[0118] α ij =h i *R j

[0119]

[0120] ​

[0121] Among them, α ij represents the correlation between the i-th factual triple and the j-th relational triple, and β ij represents the correlation weight between the i-th factual triple and the j-th relational triple, and R represents the set of relational triples.

[0122] A4. Semantics before and after fusion

[0123] For the encoded features of the factual triples obtained above, although they implicitly contain certain relational structure information among the factual triples, they only contain local structural information and have insufficient ability to capture long-distance factual relationships in the case of long texts. To solve the long-distance dependence problem, in this step, the encoded features of the factual triples are input into a recurrent neural network to obtain a vector representation z that fuses the semantic information of the previous and subsequent factual triples of each factual triple. i .

[0124] Specifically, in this embodiment, the encoded features h′ of the factual triples updated in step A35 i are input into a Bi-LSTM network to obtain a vector representation z that fuses the semantic information of the previous and subsequent factual triples of each factual triple. i . Further, in the Bi-LSTM network, based on the forward LSTM, a forward hidden layer vector is obtained Based on the backward LSTM, a backward hidden layer vector is obtained Then, the forward hidden layer vector is concatenated with the backward hidden layer vector to obtain a vector representation z that fuses the semantic information of the previous and subsequent factual triples. The formula is as follows: i . The formula is as follows:

[0125]

[0126] A5. Self-attention calculation

[0127] In this step, using the vector representation z of each factual triple i , based on the self-attention mechanism, the attention vector of each factual triple is obtained. To improve the ability to capture different information and thus improve the effect of attention calculation, the self-attention mechanism is multi-head self-attention, specifically as follows:

[0128] A51. Based on the transformation matrices W k1 , W k2 and W k3 , the vector representation z of the factual triple i is transformed into a Query value a Key value and a Value value

[0129]

[0130]

[0131]

[0132] A52. Calculate the attention of each attention head based on and :

[0133]

[0134] where d k is 's dimension;

[0135] A53. Aggregate the attention calculation results of each attention head to obtain the attention vector Z i :

[0136]

[0137] where K is the number of attention heads;

[0138] A54. Use linear transformation to change the dimension of the attention vector Z i to the dimension of the vector representation z i to obtain the attention vector r i of this fact triple;

[0139] r i = Z i * W 0

[0140] where W 0 is the linear transformation matrix.

[0141] II. Encoding Process

[0142] For the encoding process, first, tokenize the original text to obtain the word sequence of the original text; then, through the pre-trained Bert model, extract the word vectors of each word in the word sequence of the original text to obtain the word vector sequence of the original text; after that, encode the word vector sequence of the original text through the encoder to obtain the context vector of the original text. Since the encoding process is consistent with the encoding process of the existing Transformer architecture, it will not be elaborated here.

[0143] III. Decoding Process

[0144] In the decoding process, generate words one by one according to the time steps, that is, decode and generate one word at each time step.

[0145] The decoding for each time step includes the following steps:

[0146] B1. Obtain the first word vector sequence of the words generated before the current time step:

[0147] In this step, based on each generated word obtained before the current time step t, use the pre-trained Bert model to obtain its word embedding, and form the first word vector sequence of the generated words in the order of generation with the word embeddings of each generated word.

[0148] B2. Perform self-attention calculation on the first word vector sequence to obtain the second word vector sequence of the generated words:

[0149] In this step, input the first word vector sequence into the self-attention module of the decoder; first, update the first word vector sequence based on the self-attention mechanism; then, to alleviate the problem of vanishing training gradients in deep networks and accelerate network convergence, perform a residual connection on the input first word vector sequence and the updated first word vector sequence, and normalize to obtain the second word vector sequence of the generated words.

[0150] B3. Update the second word vector sequence based on the context vector of the original text using the cross-attention mechanism to obtain the third word vector sequence of the generated words:

[0151] In this step, input the second word vector sequence into the cross-attention module of the decoder; first, construct the Key value and Value value with the context vector (obtained from the encoding part) generated by the encoder of the Transformer architecture based on the original text, and output the attention distribution a of each word in the original text t , construct the Query value with the second word vector sequence, and combine the attention distribution a t , update the second word vector sequence; then, perform a residual connection on the input second word vector sequence and the updated second word vector sequence, and normalize to obtain the third word vector sequence of the generated words.

[0152] B4. Based on the attention vectors of each fact triple and the third word vector sequence, use the cross-attention mechanism to calculate the influence coefficients of all fact triples on each generated word to obtain the fourth word vector sequence of the generated words:

[0153] In this step, input the third word vector sequence into the fact attention module of the decoder; first, based on the attention vectors of each fact triple and the third word vector sequence, use the cross-attention mechanism to obtain the influence coefficients of all fact triples on each generated word; then, perform a residual connection on the input third word vector sequence and the influence coefficients of all fact triples on each generated word according to the word correspondence, and normalize to obtain the fourth word vector sequence of the generated words.

[0154] Among them, the influence coefficient of the fact triple on each generated word is calculated using the cross-attention mechanism, specifically as follows:

[0155] First, calculate the correlation between each fact triple and each generated word that has been generated:

[0156]

[0157] Among them, α im represents the correlation between the m-th generated word and the i-th fact triple, represents the word vector of the m-th generated word in the third word vector sequence;

[0158] Next, calculate the correlation weight between each generated word that has been generated and each fact triple:

[0159]

[0160] Among them, β im represents the correlation weight between the m-th generated word and the i-th fact triple, and F represents the set of fact triples; then, calculate the influence coefficient of the fact triple on each generated word that has been generated:

[0161]

[0162] u m represents the influence coefficient of all fact triples on the m-th generated word.

[0163] B5. Use the fourth word vector sequence as the input of the feed-forward network to obtain the decoded output vector at the current time step:

[0164] In this step, input the fourth word vector sequence into the feed-forward network module of the decoder; first, use the feed-forward network to calculate according to the formula p t = max(0, ln t W1 + b1)W2 + b2; then, perform a residual connection on the output p t of the feed-forward network layer and the fourth word vector sequence ln t and normalize it to obtain the output of the decoder Among them, W1, W2, b1, and b2 are all learnable parameters of the feed-forward network.

[0165] B6. Based on the decoded output vector, use the pointer network to predict the probabilities of the words in the word list and the words in the original text as the generated words at the current time step:

[0166] In order to obtain the distribution probability of the current generated word, the traditional decoding process directly maps the previous text decoding information to the word list, which may introduce factual errors caused by words that do not belong to the original text but belong to the word list in the generated summary. In order to minimize the factual errors caused by this, in this step, a pointer network is introduced to avoid generating words that do not belong to the original text but belong to the word list, thereby reducing the probability of factual errors.

[0167] Specifically, first, according to the following formula, the output of the decoder at the current time step t is converted using a linear layer Mapping to the feature space of the vocabulary:

[0168]

[0169] Among them, W vocab and b vocab represents a learnable parameter matrix corresponding to a vocabulary, wherein the vocabulary is a vocabulary of a pre-trained model;

[0170] Next, based on the mapping vector representation l t Calculate the vocabulary distribution at the current time step t and pointer probability

[0171]

[0172]

[0173] Among them, w gen and b gen represents a learnable parameter; represents the probability of the generated word at the current time step being generated from the vocabulary, Represents the output probability distribution of each word in the word list as the generated word at the current time step;

[0174] Finally, based on and Calculate the final probability distribution P at the current time step t t (w):

[0175]

[0176] Among them, P t (w) represents the probability of word w in the extended vocabulary being the generated word at the current time step t, wherein the extended vocabulary includes all words contained in the word list and the original text, N represents the number of words in the original text, and n represents the ordinal number of the word in the original text, The attention distribution a of each word vector of the original text generated by the cross attention module t The attention of the nth word in Represents the sum of attentions for the word w in the original text.

[0177] B7. Based on the probability of the word in the predicted word list and the word in the original text as the generated word at the current time step, the beam search algorithm is used to obtain the generated word at the current time step:

[0178] In this step, since the words in the word list are foreign words for the original text, and foreign words are more likely to have factual errors in the generation of the abstract compared to the words in the original text. Therefore, when the beam search algorithm really needs to introduce foreign words, it calculates the combined probability between the foreign word to be introduced and the already generated words, so as to judge the rationality of the words.

[0179] Specifically, the processing process includes:

[0180] First, use the final probability distribution P t (w) output by the pointer network at the current time step t to screen the P words with the highest probability to construct candidate words;

[0181] Then, make a judgment. If the candidate words contain the words included in the original text, then select the word included in the original text and with the highest output probability as the generated word at the current time step; otherwise, use the P candidate words and the generated words generated before the current time step to form M candidate abstracts, and calculate the combined probability of each candidate abstract according to the following formula, and select the candidate word corresponding to the candidate abstract with the highest combined probability as the generated word at the current time step:

[0182] P Y =argmax(log(Π t p(y t |y1,y2,…,y t-1 )))

[0183] Among them, Y represents the candidate abstract, and p represents the output probability of the corresponding word.

[0184] The above steps B1 - B7 are the generation process of one time step. One generated word is output in one time step. By repeatedly executing the above generation process until the predetermined number of generated words in the abstract is reached, the generated abstract can be obtained.

[0185] IV. Model Training

[0186] The training process of the text abstract generation model in this embodiment is as follows:

[0187] C1. Select a sample from the training set as the current training sample. The sample includes the original text and its true abstract; specifically, common data sets in the field of text abstract can be used as the basic data, such as the CNN / DailyMail data set as the training set;

[0188] C2. Input the original text of the current training sample into the text summarization generation model;

[0189] Use the encoder of the text summarization generation model to obtain the context vector of the original text;

[0190] The text summarization generation model generates the generated words at each time step step by step according to steps B1 to B7. Finally, the generated summary of the original text of the training sample is formed by the generated words at each time step, and the number of the generated words is consistent with the number of words in the true summary;

[0191] C3. Calculate the loss according to the following loss function, and update the text summarization generation model based on the loss:

[0192]

[0193] where, y m represents the word vector of the m-th word in the true summary, represents the word vector of the m-th generated word in the generated summary, and T represents matrix transpose;

[0194] C4. Repeat steps C1 to C3 until the training of the text summarization generation model is completed.

[0195] Finally, it should be noted that the above embodiments are only preferred embodiments and do not limit the present invention. It should be pointed out that for those of ordinary skill in the art, without departing from the spirit and scope of the present invention as defined by the claims, several modifications, equivalent replacements, improvements, etc. should be included within the protection scope of the present invention.

Claims

1. A method for generating text summaries enhanced by fact consistency, which constructs a sequence-to-sequence text summary generation model using the Transformer architecture. The decoder of the Transformer architecture includes a self-attention module, a cross-attention module, and a feed-forward network module connected in sequence. It is characterized in that: A fact attention module is introduced between the feed-forward network module and the cross-attention module of the decoder; Define a fact triple and denote it as F i =<s i , r i , o i >>, where s is the subject word, r is the relationship modifier, o is the object word, the subscript i is the serial number of the fact triple, and each word in any fact triple comes from the same sentence in the original text; the fact attention module takes the attention vectors of each fact triple in the original text and the third word vector sequence output by the decoder cross-attention module as inputs, and based on the cross-attention mechanism, obtains the influence coefficients of all fact triples on each generated word, and updates the third word vector sequence based on the influence coefficients of all fact triples on each generated word to obtain a fourth word vector sequence as the input of the decoder feed-forward network module; The attention vector of the fact triple is calculated according to the following steps: A1. Use natural language processing tools to process the original text and extract the fact triples F i , and construct a set of fact triples F; then, use the words contained in each fact triple F i to construct nodes and construct edges for the relationships, map the set of fact triples F to a graph, and use a graph neural network to calculate the node vectors of each node in the graph; afterwards, concatenate the node vectors of the nodes contained in each fact triple to obtain the encoded features of each fact triple Among them, and respectively represent the node vectors of the nodes corresponding to s i , r i , and o i contained in the fact triple F i ; A2. Input the encoded features of the fact triples into a recurrent neural network to obtain the vector representation z of each fact triple that fuses the semantic information of its preceding and succeeding fact triples i ; A3. Use the vector representation z of each fact triple i , and based on the self-attention mechanism, obtain the attention vector of each fact triple.

2. The method for generating text summaries enhanced by fact consistency according to claim 1, characterized in that In step A2, the recurrent neural network is a Bi-LSTM network; based on the encoded features of the fact triples, the forward hidden layer vectors of each fact triple are obtained using the Bi-LSTM network and the backward hidden layer vectors Then, the forward hidden layer vectors and the backward hidden layer vectors are vector concatenated to obtain the vector representation of each fact triple that fuses the semantic information of its preceding and succeeding fact triples 3. The method for generating text summaries enhanced by fact consistency according to claim 1, characterized in that In step A1, using a graph neural network, the node vectors of each node in the graph are calculated, including: First, using a pre-trained model, the node vectors of each node in the graph are initialized; Then, according to the following formula, using the GCN network, the node vectors of each node in the graph are updated: Among them, ReLU represents the activation function, and H l and H l+1 respectively represent the outputs of the l-th and (l + 1)-th layers of the GCN network, A represents the adjacency matrix of the graph, represents the degree matrix, and W l represents the weight matrix of the l-th layer of the GCN network.

4. The method for generating text summaries enhanced by fact consistency according to claim 1, characterized in that In step A1, it also includes: First, using the decision tree model of the natural language processing tool, classify the relationships between each fact triple F i based on the relationship types it contains; then, based on the relationship classification results, obtain the relationship triples between the fact triples F i and construct a relationship triple set R; Then, the encoding features of the fact triples and the relation vectors included in each relation triple are concatenated to obtain the encoding feature R of the relation triple j = [h 1j , cr j , h 2j , where h 1j and h 2j respectively represent the encoding features of the fact triples included in the j-th relation triple, and cr j represents the relation vector obtained for the relation included in the j-th relation triple based on the pre-trained model; After that, cross-attention calculation is performed using the encoding features of each relational triple and the encoding features of each factual triple. Based on the calculation results, the encoding features of each factual triple are updated, and the updated encoding features h' of the factual triple are used i as the input of step A2.

5. The method for generating text summaries enhanced by fact consistency according to claim 4, characterized in that Using the encoded features of each relationship triple and the encoded features of each fact triple, cross-attention calculation is performed, and the encoded features of each fact triple are updated based on the calculation results, specifically including: α ij = h i * R j Among them, α ij represents the relevance between the i-th factual triple and the j-th relational triple, and β ij represents the relevance weight between the i-th factual triple and the j-th relational triple, and R represents the set of relational triples.

6. A method for generating a text summary based on enhanced factual consistency according to any one of claims 1 to 5, characterized in that In step A3, use the vector representation z of each factual triple i , and based on the self-attention mechanism, obtain the attention vectors of each factual triple; The self-attention mechanism is multi-head self-attention, and its calculation process includes: A31. Conversion matrix W based on each attention head k1 , W k2 and W k3 , the vector representation z of the fact triple i , is converted into Query value Key value and Value value A32. Based on and perform the attention calculation for each attention head: where d k is the dimension; Aggregate the attention calculation results of each attention head to obtain the attention vector Z i : where K is the number of attention heads; A34. Using linear transformation, change the dimension of the attention vector Z i to the dimension of the vector representation z i to obtain the attention vector r of the fact triple i ; r i = Z i * W 0 Among them, W 0 is a linear transformation matrix.

7. A method for generating a text summary based on enhanced factual consistency as claimed in claim 1, wherein, At each time step, the decoding process of the decoder is: B1. Based on the generated words obtained before the current time step t, using a pre-trained model, obtain their word embeddings, and form a first word vector sequence of the generated words in the generation order; B2. Input the first word vector sequence into the self-attention module of the decoder; first, update the first word vector sequence based on the self-attention mechanism; then, perform a residual connection on the input first word vector sequence and the updated first word vector sequence, and normalize to obtain a second word vector sequence of the generated words; B3. Input the second word vector sequence into the cross-attention module of the decoder; First, construct Key values and Value values based on the context vectors generated by the encoder of the Transformer architecture from the original text, and output the attention distribution a of each word in the original text t , construct Query values from the second word vector sequence, and combine with the attention distribution a t , to update the second word vector sequence; then, perform a residual connection on the input second word vector sequence and the updated second word vector sequence, and normalize to obtain the third word vector sequence of the generated words; B4. Input the third word vector sequence into the fact attention module of the decoder; First, based on the attention vectors of each fact triple and the third word vector sequence, using the cross-attention mechanism, obtain the influence coefficients of all fact triples on each generated word; then, perform a residual connection on the input third word vector sequence and the influence coefficients of all fact triples on each generated word according to the word correspondence, and normalize to obtain a fourth word vector sequence of the generated words; B5. Input the fourth word vector sequence into the feed-forward network module of the decoder; First, use the feed-forward network to calculate according to the formula p t = max(0, ln t W1 + b1)W2 + b2; Then, perform a residual connection between the output p t of the feed-forward network layer and the fourth word vector sequence ln t , and normalize it to obtain the output of the decoder. Among them, W1, W2, b1, and b2 are all learnable parameters of the feed-forward network.

8. The method for generating text summaries enhanced by fact consistency according to claim 1 or 7, characterized in that Based on the attention vectors of each fact triple and the third word vector sequence, using the cross-attention mechanism, obtaining the influence coefficients of all fact triples on each generated word, specifically including: Among them, represents the word vector of the m-th generated word in the third word vector sequence; α im represents the correlation between the m-th generated word and the i-th fact triple, and β im represents the correlation weight between the m-th generated word and the i-th fact triple, and u m represents the influence coefficient of all fact triples on the m-th generated word, and F represents the set of fact triples.

9. A method for generating text summaries based on enhanced factual consistency as claimed in claim 7, wherein: The text summary generation model further includes a pointer network; At each time step, the processing process of the pointer network includes: First, according to the following formula, use a linear layer to map the output of the decoder at the current time step t to the feature space of the vocabulary: Among them, W vocab and b vocab represent learnable parameter matrices corresponding to the vocabulary, and the vocabulary is the vocabulary of the pre-trained model; Then, based on l t , calculate the vocabulary distribution and the pointer probability where w gen and b gen represent learnable parameters; represents the probability that the generated word at the current time step is generated from the vocabulary, represents the output probability distribution of each word in the vocabulary as the generated word at the current time step; After that, based on and calculate the final probability distribution P t (w) at the current time step t: Among them, P t (w) represents the probability that the word w in the extended vocabulary is the generated word at the current time step t. The extended vocabulary includes the word list and all the words contained in the original text. N represents the number of words in the original text, and n represents the serial number of the word in the original text. is the attention distribution a of each word vector of the original text generated by the cross-attention module t for the nth word in indicating the sum of the attentions where the word in the original text is w.

10. A method for generating text summaries based on enhanced factual consistency as claimed in claim 9, wherein: The text summary generation model further includes a beam search algorithm. At each time step, the processing process of the beam search algorithm includes: First, use the final probability distribution P t (w) output by the pointer network at the current time step t to screen the P words with the highest probabilities to construct candidate words; Then, a determination is made. If the candidate word contains a word included in the original text, the word included in the original text with the highest output probability is selected as the generated word at the current time step; otherwise, using the P candidate words and the generated words before the current time step, P candidate summaries are formed, and the combined probabilities of the candidate summaries are calculated according to the following formula, and the candidate word corresponding to the candidate summary with the highest combined probability is selected as the generated word at the current time step: P Y = argmax(log(Π t p(y t | y1, y2,..., y t-1 ))) where Y represents the candidate summary and p represents the output probability of the corresponding word.

Images