Paper summary generation method based on global graph attention and local graph attention
Through the global and local graph attention methods, the knowledge graph structure is transformed and combined with the pointer-generate network, the problem of difficulty in logical relationship capture in the generation of paper abstracts of complex structures is solved, and the generation quality is significantly improved.
Patent Information
- Application Number
- CN202110952568.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-08-19
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2041-08-19
AI Technical Summary
The prior art is difficult to effectively generate paper abstracts for complex structures, especially to effectively capture the internal logical relationship between multiple entities, resulting in low generation quality.
Using a method based on global graph attention and local graph attention, the paper abstract is generated by transforming the knowledge graph structure, global and local features are extracted, and the pointer-generate network and Coverage loss mechanism are combined.
It improves the generation quality of paper abstracts, can better capture the importance and relationships of nodes in the knowledge graph, adapt to complex multi-entity scenarios, and generates better than using global or local attention alone.
Smart Images

Figure CN113656598B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of natural language processing technology and relates to a method for generating paper abstracts based on a knowledge graph structure. The graph structure features are extracted through global graph attention and local graph attention to obtain the relationship between multiple entities, so as to solve the problem of generating paper abstracts with complex structures and multiple description entities. Background Art
[0002] With the advent of the era of artificial intelligence, there is an increasing demand for interaction between humans and computers and intelligent machines. How to enable machines to understand complex natural languages, have the same perspective as humans, observe the world as humans observe, and perform corresponding operations according to human intentions has become a hot topic of concern in the industry. Text generation is a traditional research direction in the field of natural language processing and has always attracted people's attention. In recent years, it has been widely used in various fields such as news, finance, and sports. Therefore, natural language generation technology can be applied to specific business needs to improve work efficiency.
[0003] The paper abstract is a summary of the full text, but the document summarization technology cannot be used to generate the paper abstract in the full text of the paper. The paper abstract usually consists of the research purpose, method, results and conclusion. There is a certain logical relationship between the parts, which is similar to the short article in form. Among the mainstream text generation tasks, machine translation tasks belong to long text generation, and the input and output information volume is consistent; document summarization tasks belong to long text generation, and the input information volume is greater than the output information volume; short article generation tasks belong to short text generation. Long text, lack of input information, and the input sequence cannot reflect the internal logical relationship.
[0004] In order to solve the above problem of lack of input information, we need additional input information that can express the internal logical relationship to support the paper abstract generation task. According to the idea of text generation task with fusion knowledge enhancement, we introduce prior knowledge such as external labels and graph structure to improve the quality of text generation. Therefore, we use graph structure to construct the knowledge graph representation of paper abstracts and use graph structure as additional input information to assist paper abstract generation. The input of traditional text generation tasks is word sequence, but graph structure cannot be directly converted into the same sequence for processing. Summary of the invention
[0005] The present invention provides a method for generating paper summaries based on global graph attention and local graph attention, which captures the global structural features and local detail features in the knowledge graph and uses them to guide the generation of paper summaries. The method can cope with complex knowledge graph structures with multiple entities, and can effectively improve the generation of paper summaries under the premise of complex knowledge graph input with multiple entity descriptions.
[0006] The paper summary generation method based on global graph attention and local graph attention includes the following steps:
[0007] (1) Graph structure conversion
[0008] The organization of the input knowledge graph structure is transformed. The entities are regarded as nodes in the graph, and the relationships between entities are regarded as edges in the graph, thereby forming a directed non-connected knowledge graph corresponding to the paper abstract. Each edge in the graph is then replaced with two nodes representing the direction of the relationship: one for the forward relationship and the other for the reverse relationship. The relationship direction nodes are connected to the initial head and tail entities to maintain the original directionality. At this time, the edges in the graph no longer have directionality, but are represented by the relationship nodes in the graph. At the same time, since the knowledge graph at this time is a non-connected undirected graph, in order to enhance the representation of global graph structure information, an additional global node is added, and the global node is connected to all nodes in the graph, converting the knowledge graph into a connected undirected graph.
[0009] (2) Feature extraction
[0010] First, the global graph structure features are extracted. Through global encoding, the attention weight distribution of node V in the graph on all nodes in the graph is calculated. Here, a multi-head self-attention mechanism is adopted, and finally the concatenation of the attention distribution matrices of multiple heads is used as the output result of the attention layer, namely global attention.
[0011] The global encoding vector captures the relationship between node V and all other nodes in the graph, but this vector representation lacks the local details of the node and its neighboring nodes, and also dilutes the relationship information between nodes. In order to make up for the above shortcomings, a local graph attention algorithm based on relationship weights is designed, and local encoding is constructed based on this. The calculation method of local attention is similar to that of global attention, but only the attention with adjacent nodes in the graph is calculated. For each node, its global attention is first calculated, and then the node vector updated by the global attention is passed into the local attention, and finally the feature representation of the node is obtained.
[0012] (3) Decoding generation
[0013] The decoding end designs a pointer-generate network structure based on the attention mechanism. This structure was originally proposed in the document summarization task. Using the copy mechanism, the model can copy the input information from the knowledge graph as output. In the copy mechanism, the probability of the model copying from the input at each moment is determined by the hidden layer state h tThe final generated probability distribution consists of a copy part and a vocabulary generation part. At each time step, there is a probability of p to copy from the input and a probability of 1-p to generate from the vocabulary. In order to prevent the model from focusing too much on a single entity or word during generation, the Coverage loss mechanism in the document summarization task is introduced this time to maintain a cumulative attention parameter ct, and the attention distribution α of each previous step is copy In order to suppress the generated words at the current step, ct is added to the loss function and copy For comparison, when ct is too high, the loss will be penalized.
[0014] The present invention can be applied to various fields such as human-computer interaction, and can enable the knowledge graph-based paper abstract generation method to adapt to complex scenarios with multiple descriptive entities, effectively capture the importance of each node in the input knowledge graph and the relationship between them, and better generate paper abstracts. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 This is the overall model structure diagram of the present invention. DETAILED DESCRIPTION
[0016] The embodiments of the present invention are described in detail below with reference to the accompanying drawings and examples.
[0017] Figure 1 The framework design of the text-based image segmentation method of the present invention is given, and the main process is as follows:
[0018] First, replace each edge in the input knowledge graph with two nodes representing the direction of the relationship: one for the forward relationship and the other for the reverse relationship. Connect the relationship direction nodes to the initial head and tail entities to maintain the original directionality. At this time, the edges in the graph no longer have directionality, but the directionality is represented by the relationship nodes in the graph. At the same time, since the knowledge graph at this time is a non-connected undirected graph, in order to enhance the representation of global graph structure information, an additional global node is added, and the global node is connected to all nodes in the graph, converting the knowledge graph into a connected undirected graph. For the node V in the graph, its feature vector is updated by calculating multi-head self-attention with all vertices in the graph. The specific calculation process is shown in the formula:
[0019]
[0020] e ij =(W k k j ) T W Q q i
[0021] where i and j are nodes in the graph, is the updated node vector, v i is the initial vector of node i, N represents multi-head attention, N i represents other nodes in the graph, || represents matrix concatenation operations, Represents the attention parameters of nodes i and j in the nth head. a n is the multi-head attention parameter, and z belongs to other nodes in the graph. k and W Q is the independent transformation matrix in each head, e ij is the dot product similarity between nodes i and j, k j and q i is the word vector representation of the node.
[0022] The updated node vector is then passed to the local encoding module, where the attention distribution between it and its immediate neighbors is calculated to update the feature vector. The specific calculation method is similar to that of the global attention, but the calculation method of the parameter e′ is different, as follows:
[0023] e′ ij =f(a T [W i v′ i ||W r v′ j ])
[0024] where i and j are adjacent nodes, e′ ij is the sum similarity of two nodes, v′ i and v′ j is its global encoding representation, || represents the concatenation operation, a T is the transposed matrix, W i and W i are the parameters learned by the model, and f is the transformation matrix.
[0025] Then, the feature vectors of all vertices are passed into the model decoding end as the input sequence. At the decoding end of the model, the copy mechanism is used. At each decoding moment, the probability of the model copying from the input is determined by the hidden layer state h t and the context vector c t The calculation method is as follows:
[0026] p=σ(W copy [h t ||c t ]+b copy )
[0027] p is the probability of replication, σ is the conversion matrix, W copyis the weight matrix and b copy is the bias parameter.
[0028] The final generation probability distribution consists of a copy part and a vocabulary generation part. At each time step, there is a probability of p to copy from the input and a probability of 1-p to generate from the vocabulary. The calculation method is as follows:
[0029] P gen =p*a copy +(1-p)*a vocab
[0030] a vocab =softmax(W(U[h t , c t ]+b 1 )+b 2 )
[0031] α copy =a([h t ||c t ], x j )
[0032] P gen is the distribution of the final generated results, a copy is the probability distribution over entities and titles, a vocab The parameter is the probability distribution of generated words in the vocabulary, h t is the hidden layer state, c t is the up and down vector, W and U are weight matrices, b 1 and b 2 is the bias parameter, x j It is a concatenation of entity code and title code.
[0033] In order to prevent the model from focusing too much on a single entity or word during generation, the Coverage loss mechanism in the document summarization task is introduced this time to maintain a cumulative attention parameter c t , for each previous step, the attention distribution a copy To suppress the generated words at the current step, c t Add to the loss function, and copy For comparison, when c t When it is too high, the loss will be penalized. The specific calculation process is as follows:
[0034]
[0035]
[0036] a copy′ is the copied attention distribution of the previous step, losst is the loss function of the model, is the direct generation part of the model, λ is the model parameter, is the copy attention distribution of the i-th step, is the context state at step i.
[0037] Example
[0038] In this embodiment, the deep learning framework PyTorch is used on the GTX 1080Ti12G graphics card.
[0039] Dataset: The experimental evaluation was conducted on the standard public dataset AGENDA. The dataset includes 40,000 papers from 12 top AI conferences, including standard entities and relations for paper abstracts. The average length of the abstract is 140 words, and each abstract corresponds to 12 entities and 7 relations on average.
[0040] Experiment: To prove the effectiveness of global attention and local attention in the paper summary generation method based on global graph attention and local graph attention, the BLEU and METEIR indicators are tested on the AGENDA dataset. The results are shown in Table 1. The experiment proves that the present invention can effectively improve the results.
[0041] Table 1 Generation results of experiments with different attention combinations
[0042] Method BLEU METEOR Using global attention alone 15.31 19.08 Using local attention alone 14.74 18.86 Using two types of attention in parallel 17.35 21.92 Method of the present invention 18.10 22.21
[0043] It can be seen from the experimental results that compared with the prior art, the present invention has a more accurate understanding of the knowledge graph and a better generation effect.
Claims
1. A paper summary generation method based on global graph attention and local graph attention, characterized in that: Here are the steps: (1) Graph structure conversion Replace each edge in the input knowledge graph with two nodes representing the direction of the relationship: one for the forward relationship and the other for the reverse relationship; connect the relationship direction nodes to the initial head and tail entities to maintain the original directionality. At this time, the edges in the graph no longer have directionality, but are represented by the relationship nodes in the graph; at the same time, add an additional global node and connect the global node to all nodes in the graph to convert the knowledge graph into a connected undirected graph; (2) Feature extraction For the node V in the graph, its feature vector is updated by calculating multi-head self-attention with all vertices in the graph. The specific calculation process is shown in the formula: e ij =(W k k j ) T W Q q i where i and j are nodes in the graph, is the updated node vector, v i is the initial vector of node i, N represents multi-head attention, N i represents other nodes in the graph, || represents matrix concatenation operations, represents the attention parameters of nodes i and j in the nth head; a n is the multi-head attention parameter, z belongs to other nodes in the graph; W k and W Q is the independent transformation matrix in each head, e ij is the dot product similarity between nodes i and j, k j and q i is the word vector representation of the node; The updated node vector is passed to the local encoding module, where the attention distribution between it and its immediate neighbors is calculated to update the feature vector. The specific calculation method is similar to the global attention, but the calculation method of the parameter e is different, as follows: e i ′ j =f(a T [W i v i ′∥W r v j ′]) where i and j are adjacent nodes, e i ' j is the sum similarity of two nodes, v i ′ and v j ′ is its global encoding representation, || represents the concatenation operation, a T is the transposed matrix, W i and W i is the parameter learned by the model, and f is the transformation matrix; (3) Decoding generation The feature vectors of all vertices are passed as input sequences to the model decoding end; at the model decoding end, the copy mechanism is used, and the probability of the model copying from the input at each decoding moment is determined by the hidden layer state h t and the context vector c t The calculation method is as follows: p=σ(W copy [h t ∥c t ]+b copy ) p is the probability of replication, σ is the conversion matrix, W copy is the weight matrix and b copy is the bias parameter; The final generation probability distribution consists of a copy part and a vocabulary generation part. At each time step, there is a probability of p to copy from the input and a probability of 1-p to generate from the vocabulary. The calculation method is as follows: P gen =p*a copy +(1-p)*a vocab a vocab =softmax(W(U[h t ,c t ]+b1)+b2) a copy =a([h t ∥c t ],x j ) P gen is the distribution of the final generated results, a copy is the probability distribution over entities and titles, a vocab The parameter is the probability distribution of generated words in the vocabulary, h t is the hidden layer state, c t is the context vector, W and U are weight matrices, b1 and b2 are bias parameters, and x j It is a concatenation of entity code and title code; Introduce the Coverage loss mechanism in the document summarization task and maintain a cumulative attention parameter c t , for each previous step, the attention distribution a copy Accumulate; c t Add to the loss function, and copy For comparison, when c t When it is too high, the loss will be penalized; the specific calculation process is as follows: a copy′ is the copied attention distribution of the previous step, loss t is the loss function of the model, is the direct generation part of the model, λ is the model parameter, is the copy attention distribution of the i-th step, is the context state at step i.
Citation Information
Patent Citations
Image title generation method based on global and local attention mechanisms
CN111444367A
Abstract generation method based on planning mechanism and knowledge graph guidance
CN112597296A
