A multi-document summarization method based on knowledge graph and BART semantics

Abstract

The application belongs to the technical field of natural language processing, and particularly relates to a multi-document summarization method based on a knowledge graph and BART semantics. The method comprises the following steps: constructing a multi-document summarization training data set; constructing a knowledge graph for multi-document summarization; constructing a multi-document summarization model fusing knowledge and graph attention; training the multi-document summarization model and generating a summary. The semantic knowledge graph fusing external knowledge strengthens the connection of distant entities, the method of fusing knowledge graph and BART semantic information makes the model better combine the attention of the knowledge graph and the text sequence, makes up for the shortcomings of the deep learning model, reduces the dependence of the model on large-scale labeled samples, and generates higher-quality summary content.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of natural language processing, and particularly relates to a multi-document summarization method based on a knowledge graph and BART semantics. BACKGROUND

[0002] In recent years, it has become a major challenge in the field of artificial intelligence for computers to generate summaries from an article or news. Multi-document summarization refers to extracting the most important information from multiple documents and presenting it to users in a concise manner. It is an important problem in the fields of information retrieval and text mining, and has a wide range of applications, such as news summaries, technology literature reviews, and business reports.

[0003] In multi-document summarization, multi-document is usually long, and generative summarization is more complex than general extractive summarization tasks. The semantic association between sentences plays a crucial role in sentence representation learning. The application adopts a summarization method based on the combination of a knowledge graph and BART semantics. First, a method for semantic modeling of long-distance text is introduced, which can encode longer input documents, solving problems such as sentence truncation and information loss caused by long input documents. Second, a semantic knowledge graph that integrates external knowledge is fused, and adding knowledge graph encoding can help the model strengthen the connection of distant entities. Finally, a method of fusing knowledge graph and BART semantic information is adopted, which enables the model to better combine the attention of the knowledge graph and the text sequence, making up for the shortcomings of deep learning models, reducing the dependence of the model on large-scale labeled samples, and generating higher-quality summary content. SUMMARY

[0004] To solve the above problems, the application provides a multi-document summarization method based on a knowledge graph and BART semantics.

[0005] To achieve the above purpose, the application adopts the following technical solutions:

[0006] A multi-document summarization method based on a knowledge graph and BART semantics, comprising the following steps:

[0007] S1, constructing a multi-document summarization training data set;

[0008] S2, constructing a knowledge graph for multi-document summarization;

[0009] S3, constructing a multi-document summarization model that fuses knowledge and graph attention;

[0010] S4, training the multi-document summarization model and generating a summary.

[0011] Further, the S1 of constructing a multi-document summarization training data set comprises the following specific steps:

[0012] S101, Download open-source multi-document digest data;

[0013] S102, preprocesses and cleans the open-source multi-document digest data to obtain the documents. Where 𝑀 is the number of documents, the required documents are split according to the specified maximum document start limit, and so on. Split into the starting document and content documents Two parts, of which This represents the beginning of the document, which is a summary of the document's content; This represents the detailed content of the document, used for extracting important core information later. and It serves as input for subsequent text encoding and graph encoding.

[0014] Furthermore, the specific steps for constructing a knowledge graph for multi-document summarization in S2 are as follows:

[0015] S201, Constructing an entity-relationship diagram for the content document: First, use OpenIE to construct the content document... First, provide common guidance; second, examine the content documents. Sentence-level open information triples are extracted. Each OPENIE triple consists of a subject [ARG0], a predicate [V], and an object [ARG1]. The extracted triple entities are connected according to similarity to construct an entity relationship graph with the subject and object as nodes and the predicate as the node relationship. ,in, The set of nodes in a graph , The set of edges between nodes in the graph , The number of triples;

[0016] S202, Constructing a knowledge graph for multi-document summarization: The triplet data format in the knowledge base ConceptNet is as follows: ,in Indicates the concept of head, Indicates relationship Indicates the concept of tail, This represents the confidence score, based on the entity relationship diagram. Middle node Find entities in the knowledge base , knowledge base entities Candidate tuples Add to the graph to build a new knowledge graph. ,in For a knowledge graph node set , A set of node edges , The number of added matched knowledge base entities.

[0017] Further, the S3 constructs a multi-document summarization model fusing knowledge and graph attention, and the specific steps are:

[0018] S301, knowledge representation of the fusion knowledge graph network: first, initialize the encoding of the knowledge graph , obtain the node feature vector ; second, learn the representation of the graph node using the graph attention network; finally, update the subject node, object node and predicate representation in the knowledge graph node, and use the self-attention mechanism to obtain the influence coefficient of the node on the node , and the representation of the knowledge graph , as shown in formulas (2) and (3):

[0019] (2)

[0020] (3)

[0021] Among them, represents the calculation score of the first multi-head self-attention, is a trainable weight, is the attention weight between nodes , represents the node to be updated in the knowledge graph, , represents the neighbor node associated with , represents the neighbor node associated with node , represents the neighbor node set associated with node ;

[0022] In order to avoid gradient vanishing after multiple iterations, a residual connection is added to obtain the final output , as shown in formula (4):

[0023] (4)

[0024] Among them, represents the neighbor node associated with node , represents a randomly initialized weight, represents the connection of multi-head attention, and the node is the final output feature combined with the multi-head attention mechanism;

[0025] S302, obtaining the BART semantic information of the knowledge graph and the text: according to the knowledge graph of S202 , and the starting document of S102 , first, the knowledge graph is converted into the form of linear text, the knowledge graph is first decomposed into multiple connected subgraphs , the subgraphs are traversed according to the size of the subgraphs, in each subgraph, the connected subgraph is arranged in descending order according to the number of nodes , , , the nodes are arranged in descending order according to the number of nodes, starting from the node with the largest number of nodes, moving down the subgraph in a breadth-first search method, linearizing the traversal result into a string list, and returning the linearized graph text ;

[0026] Secondly, the starting document , the linear graph text are respectively encoded into the BART semantic encoder. Wherein the BART Encoder processes the document , the Linear Graph Encoder processes the linear graph text , the encoding of the starting document and the encoding of the linear graph are as shown in equations (5) and (6):

[0027] (5)

[0028] (6)

[0029] wherein, , are the initial vector embeddings of and ; are the position embeddings of and ;

[0030] the Transform-based multi-head self-attention calculation is performed on respectively, and the representations of the starting document and the linear text are updated, as shown in equation (7):

[0031] (7)

[0032] wherein, and are the semantic representations of the starting document and the linear text after attention interaction;

[0033] S303 involves connecting and fusing vectors with attention mechanisms obtained by contextualizing the knowledge graph with the document text: node attention of the knowledge graph is obtained through S301. And the coding layer for the linear graph obtained by S302 Then, together with the starting document, it represents... The final semantic representation of the multiple documents is obtained by concatenation, as shown in equations (8), (9), and (10):

[0034] (8)

[0035] (9)

[0036] (10)

[0037] in, It is a method for attention fusion and standardization. It is a multi-head attention method. It is a As a feedforward network with hidden activation functions It is the result of integration. yes and The result of splicing.

[0038] Furthermore, in step S301, the graph attention network is used to learn the representation of graph nodes, specifically as follows:

[0039] Use a word embedding layer and a position embedding layer , each vertex Converted into feature vectors, then the nodes The vector representation of is shown in equation (1):

[0040] (1)

[0041] Furthermore, in step S4, the multi-document summarizing model is trained and a summary is generated. The specific steps are as follows:

[0042] S401, Construct the cross-entropy loss function and train the summarizing model: Minimize the cross-entropy loss during training using BART Decoder:

[0043] (11)

[0044] in, yes BART output before the time step, For the actual tags from the input document, For the first BART output generated at time steps;

[0045] S402, Generate Summary: Take a new multi-document dataset as input sample, feed it into the summary generation model trained using S401, and finally generate a summary of the multi-document dataset.

[0046] Compared with the prior art, the present invention has the following advantages:

[0047] (1) This invention incorporates external knowledge into the abstract model, adds external knowledge nodes, and uses entities and relationships as important semantic bridges in the text to improve information coherence and strengthen text structuring.

[0048] (2) The present invention uses a graph attention network-based model to calculate the context relationship of nodes, aggregate and transmit the feature information of nodes in the graph, capture global information in the graph structure, thereby enhancing the generalization of the model.

[0049] (3) This invention considers both graph neural network attention and text sequence attention information to capture semantic information at different levels and structures, thereby improving the model’s representation and learning capabilities.

[0050] (4) The method proposed in this invention can provide companies with high-tech products and services, generating direct practical value and economic benefits. Attached Figure Description

[0051] Figure 1 This is a flowchart of a multi-document summarization method based on knowledge graphs and BART semantics;

[0052] Figure 2 This is a flowchart of the process of constructing a multi-document summarization training dataset using S1;

[0053] Figure 3 This is a flowchart of the S2 knowledge graph unit construction process;

[0054] Figure 4 This is a flowchart of the unit process for building S3 that integrates knowledge and BART semantics;

[0055] Figure 5 This is a flowchart of the S4 model training and summary unit generation process;

[0056] Figure 6 This is a schematic diagram of converting an S302 knowledge graph into linear text;

[0057] Figure 7 This is a structural diagram of the multi-document summarization model constructed by S1-S4. Detailed Implementation

[0058] The following examples are implemented on the basis of the technical solutions of the present application, and detailed implementation modes and specific operation processes are given, but do not limit the protection scope of the present application, and any technical solution obtained by equivalent substitution or equivalent transformation shall fall within the protection scope of the present application.

[0059] Embodiment 1

[0060] As Figure 1 shown, the multi-document summarization method based on the knowledge graph and BART semantics of the present embodiment includes the following steps:

[0061] S1, constructing a multi-document summarization training data set, the specific steps are (as shown in Figure 2 ):

[0062] S101, downloading open source multi-document summarization data;

[0063] S102, preprocessing and cleaning the open source multi-document summarization data to obtain document , where M is the number of multi-documents, and the required multi-documents are cut according to the specified maximum document beginning limit (usually 1000 characters), and is divided into a starting document and a content document , wherein , represents the summary content of the document; , represents the detailed content of the document, which is used for extracting important core information subsequently, and as input for subsequent text encoding and graph encoding.

[0064] S10201, for the document data, first replace irrelevant characters such as "nbsp;" "amp;" NEWLINE_CHAR" and the like into empty characters, remove stop words, paragraph words and the like.

[0065] S10202, use the separator tag (usually |||) for segmentation; S2, constructing a knowledge graph for multi-document summarization, the specific steps are (as shown in Figure 3 ):

[0066] S201, constructing an entity relationship graph for the content document: first, using OPENIE to perform co-reference for the content document , and then extracting sentence-level open information triples from the content document , each OPENIE extracted triple consists of a subject part [ARG0], a predicate part [V], and an object part [ARG1];

[0067] The extracted triplet entities are connected based on similarity to construct an entity relationship graph with subject and object as nodes and predicate as the node relationship. ,in, The set of nodes in a graph , The set of edges between nodes in the graph , The number of triples is specified. Specifically, nodes are created by calculating the TF-IDF score of each word as a vector for each entity or phrase; S202, a knowledge graph for multi-document summarization is constructed: the triple data format in the Concept Net knowledge base is... ,in Indicates the concept of head, Indicates relationship Indicates the concept of tail, This represents the confidence score, based on the entity relationship diagram. Middle node Find entities in the knowledge base , knowledge base entities Candidate tuples Add to the graph to build a new knowledge graph. ,in For a knowledge graph node set , The set of nodes and edges , To increase the number of matching knowledge base entities. The knowledge graph is as follows: Figure 6 As shown in (a).

[0068] For the S102 starting document In part S3, the returned concept triples are appended to the end of the text using the same steps. The specific steps are as follows (e.g., ...). Figure 4 (as shown)

[0069] S301, representing knowledge in a fused knowledge graph network:

[0070] First, let's talk about knowledge graphs. Perform initialization encoding Obtain node feature vectors Secondly, a graph attention network is used to learn the representation of graph nodes, specifically:

[0071] Use a word embedding layer and a position embedding layer , each vertex Converted into feature vectors, then the nodes The vector representation of is shown in equation (1):

[0072] (1).

[0073] Finally, the subject node, object node and predicate representation in the knowledge graph node are updated until the importance of the node no longer changes. According to the characteristics of the input node, the node influence coefficient of the node , the representation of the knowledge graph is obtained by using self-attention mechanism

[0074] (2)

[0075] (3)

[0076] wherein, represents the calculation score of the first multi-head self-attention, is a trainable weight, is the attention weight between nodes , indicates the node to be updated in the knowledge graph, , , , indicates the neighbor node associated with , indicates the neighbor node set associated with the node ;

[0077] In order to avoid gradient vanishing after multiple iterations, a residual connection is added to obtain the final output , as shown in formula (4):

[0078] (4)

[0079] wherein, indicates the neighbor node associated with the node , indicates a randomly initialized weight, indicates a connection of multi-head attention, and the node is the final output feature combined with the multi-head attention mechanism;

[0080] S302, obtaining the BART semantic information of the knowledge graph and the text (as shown in Figure 7 ): according to the knowledge graph of S202 and the starting document of S102, first convert the knowledge graph into linear text form (as shown in Figure 6 (b)), first convert the knowledge graph decomposing into multiple connected subgraphs , traversing the subgraphs according to the size of the subgraphs, in each subgraph, the connected subgraph according to the number of nodes contained , for the number of nodes, arrange them in descending order, starting from the node containing the most nodes, move down the subgraph in a breadth-first search method, linearize the traversal result into a string list, and return the linearized graph text ; wherein 为主语节点， <obj> 为对应宾语节点， <pred> 是相应的谓语边，得到的线性文本。

[0081] 其次对起始文档 、线性图文本 分别编码输入BART语义编码器。其中BARTEncoder处理文档 ，Linear Graph Encoder处理线性图文本 ，起始文档的编码 和线性图的编码 如式（5）（6）所示：

[0082] （5）

[0083] （6）

[0084] 其中， 、 为 和 初始向量嵌入， 为 和 位置嵌入；

[0085] 对 分别进行基于Transform的多头自注意力计算，并更新起始文档 和线型文本 的表示，如式（7）所示：

[0086] （7）

[0087] 其中， 和 为起始文档和线型文本经过注意力交互后得到的语义表示；

[0088] S303，通过将知识图与文档文本的上下文化得到具有注意力机制的向量进行连接融合：通过S301得到知识图的节点注意力 以及S302得到的对于线性图的编码层 ，再与起始文档表示 拼接，获取多文档的最终语义表示，具体实现如式（8）、（9）、（10）所示：

[0089] (8)

[0090] (9)

[0091] (10)

[0092] 其中， 是进行注意力融合与标准化的方法， 是多头注意力方法， 是一个以 作为隐藏激活函数的前馈网络， 是融合的结果， 是 与 拼接的结果。

[0093] S4，训练多文档摘要模型并生成摘要，具体步骤为（如 图5所示）：

[0094] S401，构建交叉熵损失函数，并对摘要模型进行训练：利用BART Decoder最小化训练过程中的交叉熵损失：

[0095] (11)

[0096] 其中， 是 时间步之前的BART输出， 为来自输入文档的真实标签， 为第 的时间步生成的BART输出；

[0097] S402，生成摘要：将一个新的多文档数据组成输入样本，送入利用S4011训练完成的摘要生成模型，最后输出摘要生成示例。 < / pred> < / obj>

Claims

1. A multi-document summarization method based on knowledge graph and BART semantics, characterized in that, The method comprises the following steps: S1, constructing a multi-document summarization training data set; S2, constructing a knowledge graph for multi-document summarization; S3, constructing a multi-document summarization model integrating knowledge and graph attention; S4, training the multi-document summarization model and generating a summary; The specific steps of constructing the multi-document summarization model integrating knowledge and graph attention in S3 are: S301, Knowledge Representation of the Integrated Knowledge Graph Network: First, the knowledge graph... Perform initialization encoding Obtain node feature vectors Secondly, a graph attention network is used to learn the representation of the graph nodes; finally, the representations of the subject, object, and predicate nodes in the knowledge graph are updated, and a self-attention mechanism is used to obtain the nodes based on the features of the input nodes. For nodes Influence coefficient, knowledge graph representation As shown in equations (2) and (3): (2) (3) in, Representing the The calculation score of multi-head self-attention. For trainable weights, For nodes Attention weights between them This represents a node in the knowledge graph that needs to be updated. , Indicates and Associated neighbor nodes, Represents nodes A set of related neighboring nodes; To avoid vanishing gradients after multiple iterations, a residual connection is also added , to get the final output , as shown in equation (4): (4) wherein, represents a node associated with a neighboring node, represents a randomly initialized weight, represents connections of multiple heads of attention, the node is a final output feature incorporating the multi-head attention mechanism; S302, obtaining the BART semantic information of the knowledge graph and the text: according to the knowledge graph of S202 , and the starting document of S102 , first convert the knowledge graph into the form of linear text, first decompose the knowledge graph into multiple connected subgraphs , traverse the subgraphs according to the size of the subgraphs, in each subgraph, the connected subgraph is arranged in descending order according to the number of nodes , , starting from the node with the largest number of contained nodes, move down the subgraph in a breadth-first search method, linearize the traversal result into a string list, and return the linearized graph text ; Second, the start document , linear graph text respectively encoded input BART semantic encoder, wherein the BART Encoder processes the document , Linear Graph Encoder processes linear graph text , encoding of the start document and the encoding of the linear graph As shown in equations (5) (6): （5） （6） wherein, , is and initial vector embedding, is and position embedding; On Respectively, the multi-head self-attention calculation based on Transform is performed, and the representation of the starting document is updated and linear text as shown in equation (7): （7） wherein, and is the semantic representation of the starting document and linear text after attention interaction; S303, obtain the vector with attention mechanism by connecting and fusing the knowledge graph and the contextualization of the document text: obtain the node attention of the knowledge graph by S301 and the encoding layer for the linear graph obtained by S302 , splice with the initial document representation , obtain the final semantic representation of the multi-document, and the specific implementation is shown in formulas (8), (9), and (10): (8) (9) (10) wherein, is a method of attention fusion and standardization, is a multi-head attention method, is a feed-forward network with as a hidden activation function, is the result of fusion, is and is the result of concatenation.

2. The method of claim 1, wherein, The specific steps of constructing the multi-document summarization training data set in S1 are: S101, downloading an open-source multi-document summarization data; S102, preprocessing and cleaning the open source multi-document summarization data to obtain documents where M is the number of multi-documents, cutting the required multi-documents according to the specified maximum document beginning limit, cutting into two parts of the starting document and the content document , where represents the summary content of the document, which is the summary content of the document; represents the detailed content of the document, which is used for subsequent extraction of important core information, and as the input of subsequent text encoding and graph encoding.

3. The method of claim 2, wherein, The specific steps of constructing the knowledge graph for multi-document summarization in S2 are: S201, constructing an entity relationship graph for the content document: first, performing co-reference for the content document using OPENIE, and then extracting sentence-level open information triples from the content document . Each triple extracted by the OPENIE includes a subject part [ARG0], a predicate part [V], and an object part [ARG1]. The extracted triple entities are connected according to similarity to construct an entity relationship graph with the subject and the object as nodes and the predicate as a node relationship wherein,​​​​​​ S202, Constructing a knowledge graph for multi-document summarization: The triplet data format in the knowledge base ConceptNet is as follows: ,in Indicates the concept of head, Indicates relationship Indicates the concept of tail, This represents the confidence score, based on the entity relationship diagram. Middle node Find entities in the knowledge base , knowledge base entities Candidate tuples Add to the graph to build a new knowledge graph. ,in For a knowledge graph node set , The set of nodes and edges , Add the number of matching knowledge base entities.

4. The method of claim 1, wherein, The specific steps of learning the representation of the graph node by using the graph attention network in S301 are: using a word embedding layer and a position embedding layer convert each vertex into a feature vector, then the vector representation of a node is given by equation (1): (1)。 5. The method of claim 1, wherein, The specific steps of training the multi-document summarization model and generating a summary in S4 are: S401, constructing a cross-entropy loss function and training the summary model: minimizing the cross-entropy loss in the training process by using a BART Decoder: (11) wherein, is BART output at a time step, is a true label from the input document, is the BART output generated for the time step. S402, generating a summary: inputting a new multi-document data into the summary generation model trained in S401, and finally generating a multi-document summary.

Images