Hierarchical planning abstract generation method based on topic-question theory

Abstract

The application discloses a hierarchical planning paper abstract generation method based on a topic-thesis theory, (1) a paper title and a knowledge graph are encoded by using a Transform encoder and a Graph Transform encoder respectively to obtain representations of the paper title and the knowledge graph; (2) a topic word is planned; (3) a thesis subgraph is planned; (4) each sentence is generated according to the topic word and the thesis subgraph; and (5) the processes of (3-4) are repeated until the traversal of all topic words is completed, and all sentences are spliced together to obtain a corresponding paper abstract.

Description

Technical Field

[0001] This invention relates to natural language generation, content planning, and knowledge graph applications, specifically a hierarchical planning method for generating paper abstracts based on topic-topic theory. Background Technology

[0002] Abstract generation is a crucial first step in scientific paper collaboration. It aims to produce coherent and rich abstracts that match the given paper title. With the rapid development of natural language generation technologies, automatic abstract generation has also made remarkable progress. However, previous work has relied on sequential neural networks and limited input from only a few article titles. Due to the length gap between the title and abstract, as well as model performance limitations, these models tend to generate safe but uninteresting text.

[0003] While injecting external knowledge can mitigate the limitations imposed by limited input, paper abstracts involve the generation of multiple sentences. Abstract writing is not simply about generating and combining these sentences; each sentence in a paper abstract should have its own functional meaning, such as describing the paper's improvements, methods, or content. We believe that the function of each sentence in a paper abstract can be determined by the topic-response structure. The topic-response structure posits that a sentence can be divided into two parts: "topic words" and "response words." The "topic words" describe what the sentence aims to achieve, while the "response words" describe how to achieve it. Previous planning models have the potential for different sentences to acquire the same topic words, leading to semantic redundancy. The topic-response structure avoids this problem. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by providing a hierarchical planning method for generating academic abstracts based on topic-response theory. Combining external knowledge graphs has been proven to effectively improve the quality of generated abstracts. This method, based on knowledge graphs, further fills the gap in current research on the internal structure of academic abstracts. This invention addresses the problems of monotonous content and repetitive sentences in generated abstracts by proposing a hierarchical planning method for generating academic abstracts based on topic-response theory. This method can plan a series of topic words on a complete knowledge graph, or plan smaller-scale subgraphs as recourse subgraphs for sentence generation. The method mainly consists of two parts: 1) Topic planning: used to plan a series of topic words as the first stage of planning, topic... k It is the topic word of the kth sentence. 2) Topic planning: Topic subgraph planning is to plan the topic using a scorer. kThe corresponding topic subgraphs are generated. Different topic terms have different topic subgraphs. The combination of topic terms and topic subgraphs determines the meaning of the k-th sentence. Furthermore, a scoring mechanism is designed to obtain the corresponding topic subgraphs. Given topic terms, vertices in the knowledge graph are scored, and only nodes with scores greater than a certain value are retained in the corresponding topic subgraphs.

[0005] The objective of this invention is achieved through the following technical solution:

[0006] A hierarchical planning method for generating paper abstracts based on topic-response theory includes the following steps:

[0007] (1) The paper title and knowledge graph are encoded using a Transformer encoder and a Graph Transformer encoder respectively to obtain the representation of the paper title and knowledge graph; the knowledge graph is the knowledge graph corresponding to the paper abstract;

[0008] (2) Keyword planning: The paper title and the representations of nodes in the knowledge graph are concatenated and the latent semantic information is learned. The resulting latent variable is denoted as z. topic , z topic It contains latent semantic information from the paper title and knowledge graph; topic Used to initialize the topic term decoder; the topic term decoder is trained to generate a topic term sequence T, which contains several topic terms: T = {topic...} k |1≤k≤|Y|}, where |Y| represents the number of sentences in the abstract; the first subject term in the subject term sequence, topic1, is the subject term of the first sentence in the abstract, topic2 is the subject term of the second sentence in the abstract, and so on. k It is the keyword of the k-th sentence in the paper abstract, k = 1, 2, ... |Y|; the keyword decoder will select a node from the knowledge graph as the keyword at each step of the generation process;

[0009] (3) Subgraph planning: For each keyword in the keyword sequence T, there is a corresponding subgraph. The set of subgraphs is defined as g = {g k |1≤k≤|Y|}; Taking the construction of the topic subgraph corresponding to the kth topic as an example: taking the topic as k Based on this, each node in the knowledge graph is scored, with the score ranging from [0,1]. Nodes with a score greater than 0.4 are considered relevant, while those with a score less than 0.4 are considered irrelevant. Finally, only nodes with scores greater than 0.4 are retained, while irrelevant nodes are discarded. The resulting subgraph of the knowledge graph is the topic subgraph, denoted as g. k ;

[0010] (4) Based on the topic k and the subgraph g k To generate the k-th sentence: Since the sentences in the paper abstract influence each other, a latent variable is set. Indicates the semantic state of the current sentence. We combine the states of k-1 sentences with topic words and topic subgraphs to learn the semantic state that the kth sentence should have; Used to initialize the initial state of the sentence decoder; a supervised semaphore is used to monitor the probability that the sentence decoder copies nodes from the comment subgraph when generating each word, thereby avoiding repeated copying of nodes from the comment subgraph.

[0011] (5) Repeat process (3-4) until all keywords have been traversed, and then concatenate all the sentences to obtain the corresponding paper abstract.

[0012] Furthermore, step (1) specifically includes:

[0013] (101) Encode the paper title: Let the representation vector of the input paper title be title={x1,x2,…,x m}, x i Let represent the embedding vector of the i-th word in the paper title, where i = 1, 2, ..., m. The Transformer encoder is used to encode the input paper title vector; the final representation vector is: in It represents the i-th word in the paper title, where i = 1, 2, ..., m; define h title It is the average of all the words, that is, the overall representation of the paper title;

[0014] (102) Encoding the knowledge graph: Let the input knowledge graph be G={V,E}, where V= {v i |1≤i≤n} is the set of nodes in the knowledge graph, where n is the total number of nodes in the knowledge graph, and v i It is the embedding vector of the i-th node in the knowledge graph; E = {e ij |1≤i≤n,1≤j≤n} is the set of edges in the knowledge graph, where e iEj It is a connection to v i and v j The edge embedding vectors; the knowledge graph G is encoded by a Graph Transformer encoder; finally, the representation is obtained. It represents the i-th node; definition The average value is represented by all nodes.

[0015] 9. The method for generating paper abstracts based on topic-response theory according to claim 1, characterized in that step (2) specifically includes:

[0016] (201) Sampling of latent variables: Based on the VAE (Variational Autoencoder) framework, the paper title representation and the knowledge graph representation are first concatenated and then fed into the fully connected layer to obtain the mean μ of the Gaussian distribution. topic and variance σ topic Latent variable z is constructed based on mean and variance. topic ;where z topic The sampling conforms to the formula Indicate z topic The sampling process and h title , Related, z topic The sampling process is equivalent to μ topic ,σ topic The resampling process under the Gaussian function N(·); where h title It is a general representation of the paper's title; The average value represented by all nodes;

[0017] (202) Generation of the topic term sequence: The topic term sequence is generated by the topic term decoder. The topic term decoder generates one topic term per decoding operation, therefore the topic term decoder can decode |Y| times; the initial state of the topic term decoder. By z topic initialization: This is the initial state of the topic term decoder; different decoder states will yield different topic terms to generate topics. k For example, generating a topic k The decoder state at that time is According to formula (1):

[0018]

[0019] Bi-GRU topic (·,·) represents a bidirectional GRU decoder applied to topic generation; topic k-1 There are k-1 keywords; The decoder state when generating k-1 topic words; then topic k Determined by formula (2):

[0020]

[0021] Where Tanh(·) is the activation function, softmax(·) is the probability distribution function, and W topic ,btopic These are trainable parameters; Formula (2) can calculate the number of nodes selected as topics in the knowledge graph. k The probability will be calculated, and the node with the highest probability will ultimately be selected as the topic. k The topic decoder repeatedly executes formulas (1) and (2) to finally obtain the topic sequence T = {topic}. k |1≤k≤|Y|}, where |Y| represents the abstract of the paper containing |Y| sentences, and each keyword corresponds to the main theme of one sentence;

[0022] (203) The loss function L1 for keyword planning is:

[0023]

[0024] q(z topic ) is z topic The posterior distribution, p(z) topic ) is z topic The prior distribution;

[0025] It is a reconstruction loss function based on the posterior distribution;

[0026] D kl (q(z topic )|p(z topic ))] is the KL divergence loss function between the prior and posterior distributions.

[0027] Furthermore, step (3) specifically includes:

[0028] (301) Using the keyword k Based on this, each node in the knowledge graph is scored, with the score ranging from [0, 1]. Nodes with a score greater than 0.4 are considered relevant, while those with a score less than 0.4 are considered irrelevant. Finally, only nodes with scores greater than 0.4 are retained, while irrelevant nodes are discarded. The resulting subgraph of the knowledge graph is the topic subgraph, denoted as g. k The formula for the above workflow is expressed as follows:

[0029]

[0030]

[0031] g k ={v i |score i >β,i=1,2,…n} #(6)

[0032] Where σ(·) represents the sigmoid probability distribution function, Tanh(·) is the activation function, and W comm b comm For trainable parameters; Bi-GRU topic (·,·) represents a bidirectional GRU network used for predicate subgraph programming. This indicates that the scorer generates g. k The scoring status at that time, g k Which nodes will be included in the middle? The decision is made regarding the nodes in the knowledge graph. In the scoring state, if the score exceeds 0.4, the node is considered to be retained in the problem subgraph g. k If it is in the middle, it will be discarded; score i For v in knowledge graph i Node at The score below; Formula (6) describes g k It only retains nodes whose scores are greater than β under the current conditions regarding the topic. k The title subgraph; It is a predicate subgraph g k The set represented by each node in the set;

[0033] (302) The loss function of subgraph programming. for:

[0034]

[0035] in Indicates whether a node in the knowledge graph should be stored in g. k In the code, 1 indicates that the node should be saved, and 0 indicates that the node should not be saved.

[0036] Furthermore, step (4) specifically includes:

[0037] Abstract of the paper Y={Y k |k=1,2,...|Y|}, where |Y| represents the number of sentences in the paper's abstract, where Y... k Let |Y| represent the k-th (k = 1, 2, ..., |Y|) sentence in the abstract Y of the paper. k | represents the length of the k-th sentence, Y k ={w kt |t=1,2,...|Y k |}, w kt Sentence Y k The t-th word in |Y i The details are as follows:

[0038] (401) Latent semantic state of sentences: To ensure the contextual coherence of the paper abstract, latent variables are set. Used to represent the k-th sentence Y in the paper abstract Y k The latent semantic state of Y is used to guide Y. k The generation; respectively obtain Mean of Gaussian distribution and variance Construct latent variables based on mean and variance in The sampling conforms to the formula express The sampling process and Related, among which It is the average of the nodes represented in the subgraph of the problem; The sampling process is equivalent to The resampling process under the Gaussian function N(·) is shown in formulas (8) and (9).

[0039]

[0040]

[0041] LSTM(·,·) is an LSTM (Long Short-Term Memory) network applied to sentence state representation; It is sentence Y k The sentence state is influenced only by the latent semantic states of the first k-1 sentences; however, the generation of a sentence is affected not only by the context but also by the topic words and the statement subgraph, therefore... from topick, Of the three, learning is used to guide Y. k The potential semantic state;

[0042] (402)Y k The generation process: the state of the sentence decoder Depend on initialization: Indicates when decoding generates Y k The sentence decoder state at the t-th word will affect Y. k Word generation results;

[0043]

[0044]

[0045]

[0046] Among them, Bi-GRU topic (·,·) represents a bidirectional GRU network applied to word generation; Attention(·,·) is the attention function. The topic pick is the form calculated by the attention mechanism. This is the form of the predicate subgraph after computation using the attention mechanism;

[0047] w kt The formula generated by decoding is represented by formula (13):

[0048]

[0049]

[0050] p kt There is a supervisory signal used to control the generation of w. kt Whether to copy relevant nodes from the subgraph in the context of the problem; To generate w kt The probability distribution of time in the lexicon To generate w kt The probability distribution of time at each node of the subgraph; w kt Ultimately, the output will be selected from the two weighted probability distributions, with the highest probability being chosen as the final output.

[0051] (403) Sentence Y k Loss function of the generation process It consists of a generation loss function and a supervised copy loss function:

[0052]

[0053] yes The posterior distribution, yes The prior distribution; Based on The reconstruction loss function of the posterior distribution; yes The KL divergence loss functions for the prior and posterior distributions; where For supervised copying signals, A value of 1 indicates the probability of selecting a node from the vocabulary, while 0 indicates the probability of selecting a node from the topic subgraph.

[0054] Furthermore, in step (5), until all keywords have been traversed, the generated Y... k By concatenating these elements, we obtain the corresponding paper abstract, k = 1, 2, ... |Y|; the final complete loss function is... L1 is the loss function for topic word planning. For the topic subgraph g k The loss function for planning; For sentence Y k The loss function of the generation process.

[0055] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the program to implement the steps of the hierarchical planning paper abstract generation method based on topic-state theory.

[0056] The present invention also provides a computer-readable storage medium having a computer program stored thereon, characterized in that, when the computer program is executed by a processor, it implements the steps of the hierarchical planning paper abstract generation method based on topic-state theory.

[0057] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0058] (1) Using a joint knowledge graph as input: Previous work has mostly used only the paper title as input. However, due to the length difference between the paper abstract and the paper title, using only the paper title as input cannot achieve good results. The method of this invention uses a knowledge graph to provide additional data information. The nodes in the knowledge graph can compensate for the problem of limited input length.

[0059] (2) Hierarchical Planning Structure: Abstracts place great emphasis on contextual consistency, but existing methods lack specific planning tools to ensure this consistency. This invention follows linguistic sentence structure analysis and proposes a hierarchical planning method based on topic-response theory. The first stage plans the topic word sequence, a text-level plan that primarily plans the topic of each sentence in the abstract. The second stage plans the subgraphs corresponding to the topic words, a sentence-level plan that addresses how each sentence in the abstract should be presented. The first-stage planning method ensures contextual consistency, thus guaranteeing rich content in the abstract; the second-stage planning ensures coherence within sentences, thus guaranteeing the final quality of the abstract.

[0060] (3) Scoring Mechanism for Knowledge Graph: Unlike previous works that used knowledge graphs indiscriminately, this invention proposes a scoring mechanism that divides the knowledge graph into different subgraphs under different keywords. The subgraphs contain only the most core node information, thus generating more accurate sentence content. Simultaneously, the existence of subgraphs significantly reduces the size of the candidate set, improving model speed.

[0061] (4) Supervised copying mechanism: In order to avoid frequent copying of nodes in the pointer network and cause repetition within the sentence, the method of this invention will use a supervised copying mechanism to construct the sentence and punish the phenomenon of continuously copying nodes from the topic subgraph.

[0062] (5) Improved experimental performance: This invention was compared with similar generative models on a dataset related to paper abstracts. Ultimately, the overall results obtained by this invention were superior to those of similar generative models on different metrics. In addition, different ablation experiments also proved the rationality of the hierarchical planning model architecture proposed in this invention. Attached Figure Description

[0063] Figure 1 This is a flowchart illustrating the method of the present invention.

[0064] Figure 2 This is a flowchart illustrating the planning phase of the subject headings for this invention.

[0065] Figure 3 This is a schematic diagram of the planning stage of the topic of this invention. Detailed Implementation

[0066] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only for explaining the present invention and are not intended to limit the present invention.

[0067] This embodiment uses the AGENDA dataset as an example to illustrate the specific implementation method of the present invention. The AGENDA dataset comes from the Semantic Scholar corpus. AGENDA contains 40k paper titles and their abstracts, and provides corresponding knowledge graphs.

[0068] 1. Encode the paper title and knowledge graph to obtain the keyword planning for the paper title and knowledge graph; the knowledge graph is the knowledge graph corresponding to the paper abstract.

[0069] (101) Encode the paper title: Let the input paper title be title={x1,x2,...,x m}, x i Let represent the i-th word in the paper title, where i = 1, 2, ..., m. The vector representing the paper title is initialized as follows:

[0070]

[0071] Where embedding(title) represents the word embedding of the input title, and PE(title) represents the position embedding of the input title under the cosine algorithm; the Transformer Encoder has 6 layers, and the output representation of the first layer is denoted as:

[0072]

[0073] Where FFN(·) represents a feedforward neural network, and MHA(·,·,·) represents a multi-head attention mechanism; the input paper title vector is encoded using a Transformer encoder; the final representation vector is as follows. in It represents the i-th word in the paper title, where i = 1, 2, ..., m; define h title The average of all the words represents the overall representation of the paper title;

[0074] (102) The knowledge graph is G = {V, E}, where V = {v i |1≤i≤n} is the set of nodes in the knowledge graph, where n is the total number of nodes in the knowledge graph, and v i It is the embedding vector of the i-th node in the knowledge graph; E = {e ij |1≤i≤n,1≤j≤n} is the set of edges in the knowledge graph, where e ij It is a connection to v i and v j The edge embedding vector;

[0075] The knowledge graph G is encoded by a Graph Transformer encoder. This is the initial representation of the node. The initial representation of the edge set:

[0076]

[0077]

[0078] embedding(V) represents the word embedding of a node, and lapPE(V) is the positional embedding under the Laplacian operator;

[0079] embedding(E) represents the word embedding of the edge; in the Graph Transformer encoder, the first layer graph node representation and edge The update formula can be written as:

[0080]

[0081]

[0082] Since the previous GraphTransformer encoder ignored the relationships between nodes when calculating node representations, this paper introduces a node representation that incorporates edge relationships. The main improvement lies in incorporating the edge relationships between any two nodes when calculating the attention score in the multi-head attention mechanism. This section will demonstrate how the k-th head calculates v in the (l-1)-th layer of the GraphTransformer encoder's multi-head attention mechanism. i v j The attention score between nodes in the remaining layers and heads is calculated in the same way:

[0083]

[0084] in This represents the value of v computed in the k-th head of layer l-1 in the Graph Transformer encoder. i v j Attention score between Will participate Update; for Variations in multi-headed attention for Variations in multi-headed attention for The variant form in multi-head attention; the rest of the calculation method remains unchanged;

[0085] The Graph Transformer encoder ultimately yields a set of representations for each node in the knowledge graph. It represents the i-th node; definition The average value represented by all nodes;

[0086] 2. Keyword Planning

[0087] Based on the conceptual structure of VAE (variational autoencoder), the paper title representation and the knowledge graph representation are first concatenated and then fed into a fully connected layer to obtain the mean μ. topic and variance σ topic Latent variable z is constructed based on mean and variance. topic ;where z topic The sampling conforms to the prior distribution Posterior distribution During the testing phase, only the prior distribution is used; mean μ topic and variance σ topic The calculation process is as follows:

[0088]

[0089]

[0090] μ topic , σ topic Let the mean and variance be the prior distribution. h represents the mean and variance of the posterior distribution. title The title represents the entirety of the article. h is the average of the node representations in the knowledge graph. topic For the overall representation of the keywords, h in the posterior distribution calculation topic Derived from the dataset; MLP(·,·) represents a fully connected layer; the initial state of the topic word decoder. Initialized by ztopic: This is the initial state of the topic term decoder; different decoder states will yield different topic terms to generate topics. k For example, generating a topic k The decoder state at that time is According to formula (10), we obtain:

[0091]

[0092] Bi-GRU topic (·,·) represents a bidirectional GRU decoder applied to topic generation; topic k-1 There are k-1 keywords; The decoder state when generating k-1 topic words; then topic k Determined by formula (11):

[0093]

[0094] Where Tanh(·) is the activation function, softmax(·) is the probability distribution function, and W topic b topic These are trainable parameters; Formula (11) can calculate the number of nodes selected as topics in the knowledge graph. k The probability will be calculated, and the node with the highest probability will ultimately be selected as the topic. k The topic decoder repeatedly executes formulas (10) and (11) to finally obtain the topic sequence T = {topic}. k |1≤k≤|Y|}, where |Y| represents the abstract of the paper containing |Y| sentences, and each keyword corresponds to the main theme of one sentence;

[0095] The loss function L1 for keyword planning is

[0096]

[0097] q(z topic ) is z topic The posterior distribution, p(z) topic ) is z topic The prior distribution;

[0098] It is a reconstruction loss function based on the posterior distribution;

[0099] D kl (q(z topic )|p(z topic [)] is the KL divergence loss function between the prior and posterior distributions;

[0100] 3. Problem Subgraph Programming

[0101] This embodiment will demonstrate how to plan the topic subgraph corresponding to the k-th topic term (topic subgraph: in linguistics, a sentence can be divided into two parts: topic and statement. The topic is the main content discussed in the sentence, and the statement is a supplementary statement to the topic); using the topic term... k Based on this, each node in the knowledge graph is scored, with the score ranging from [0, 1]. Nodes with a score greater than 0.4 are considered relevant, while those with a score less than 0.4 are considered irrelevant. Finally, only nodes with scores greater than 0.4 are retained, while irrelevant nodes are discarded. The resulting subgraph of the knowledge graph is the topic subgraph, denoted as g. k The formula for the above workflow is expressed as follows:

[0102]

[0103]

[0104] g k ={v i |score i >β,i=1,2,…n} #(15)

[0105] Where σ(·) represents the sigmoid probability distribution function, Tanh(·) is the activation function, and W comm b comm For trainable parameters; Bi-GRU comm (·,·) represents a bidirectional GRU network used for predicate subgraph programming. This indicates that the scorer generates g. k The scoring status at that time, g k Which nodes will be included in the middle? The decision is made regarding the nodes in the knowledge graph. In the scoring state, if the score exceeds 0.4, the node is considered to be retained in the problem subgraph g. k If it is in the middle, it will be discarded; score i For v in knowledge graph i Node at The score below; Formula (15) describes g k It is a topic subgraph that only retains nodes whose scores are greater than β (β = 0.4) under the current conditions; It is a predicate subgraph g k The set represented by each node in the set;

[0106] The loss function of subgraph programming for:

[0107]

[0108] in Indicates whether a node in the knowledge graph should be stored in g. k In the code, 1 indicates that the node should be saved, and 0 indicates that the node should not be saved.

[0109] 4. Sentence generation process

[0110] Abstract of the paper Y={Y k |k=1,2,...|Y|}, where |Y| represents the number of sentences in the paper's abstract, where Y... k Let |Y| represent the k-th (k = 1, 2, ..., |Y|) sentence in the abstract of the paper. k | represents the length of the k-th sentence, Y k ={w kt |t= 1,2,...|Y k |},w kt Sentence Y k The t-th word in |Y i The details are as follows:

[0111] Latent semantic state of sentences: To ensure the contextual coherence of the paper abstract, latent variables are set. Used to represent the k-th sentence Y in the paper abstract Y k The latent semantic state of Y is used to guide Y. k The generation; respectively obtain Mean under Gaussian distribution and variance Construct latent variables based on mean and variance in Sampling during the training phase prior distribution in It is the average of the representations of each node in the subgraph, sampled from the posterior distribution during the testing phase.

[0112]

[0113]

[0114]

[0115] LSTM(·,·) is an LSTM network applied to sentence state representation. Let be the mean and variance of the posterior distribution. Let be the mean and variance of the prior distribution, where For sentence Y k The latent representation of the state is used only in the posterior distribution to train the model parameters; It is sentence Y k The sentence state is influenced only by the latent semantic states of the first k-1 sentences; however, the generation of a sentence is affected not only by the context but also by the topic words and the statement subgraph, therefore... from topic k , Of the three, learning is used to guide Y. k The potential semantic state;

[0116] Y k The generation process: the state of the sentence decoder Depend on initialization: Indicates when decoding generates Y k The sentence decoder state at the t-th word will affect Y. k Word generation results;

[0117]

[0118]

[0119]

[0120] Among them, Bi-GRU word (·,·) represents a bidirectional GRU network applied to word generation; Attention(·,·) is the attention function. The topic pick is the form calculated by the attention mechanism. This is the form of the predicate subgraph after computation using the attention mechanism;

[0121] w kt The formula generated by decoding is represented by formula (13):

[0122]

[0123]

[0124]

[0125]

[0126] W vocab b vocab W copy b copy W pointer b pointer p are trainable parameters kt There is a supervisory signal used to control the generation of w. kt Whether to copy relevant nodes from the subgraph in the context of the problem; To generate w kt The probability distribution of time in the lexicon To generate w kt The probability distribution of time at each node of the subgraph; w kt Ultimately, the output will be selected from the two weighted probability distributions, with the highest probability being chosen as the final output.

[0127] Sentence Y k Loss function of the generation process It consists of a generation loss function and a supervised copy loss function:

[0128]

[0129] Repeat steps 3 and 4 until all keywords have been traversed, then concatenate all statements to obtain the corresponding paper abstract; the complete loss function of the model corresponding to the method of this invention is...

[0130] Specifically, in the implementation, both the Transformer encoder and the Graph Transformer encoder are set to 6 layers. The β value for constructing the title subgraph is 0.4. All decoders in all modules use a bidirectional GRU model. The word vector size is 512, the hidden states in each layer are set to 200, and the latent variable z... topic , The size is 30, the learning rate is 0.0005, and to avoid posterior collapse, all KL divergence terms are multiplied by 0.1. To avoid overfitting, a dropout strategy is used, and the optimizer strategy is Adam optimization.

[0131] To verify the effectiveness of the method of the present invention, the method of the present invention (IIP) is compared with the current advanced and representative method (seq2seq). [1] rewriter [2] CVAE [3] ml-cvae [4] GAT [5] GraphWriter [6] BART [7] Stylized [8] The invention is compared with three variants of the method of the present invention (w / o graph, w / o intrnsic, w / o inner).

[0132] Seq2Seq is a typical encoder-decoder structure model that can be applied to most text generation tasks.

[0133] Rewriter: is a multi-pass decoding model based on seq2seq that iteratively checks, improves, and optimizes the generated content.

[0134] CVAE: It is a typical variational autoencoder model with generative capabilities and strong paper abstract generation ability.

[0135] ml-cvae is a multi-layer encoder-decoder structure model based on CVAE, which takes into account the structural relationships between sentences.

[0136] GAT stands for Knowledge Augmentation Model. It uses a knowledge graph as input to generate paper summaries.

[0137] The graphwriter uses a Graph Transformer as the encoder and employs a copying mechanism in the decoder to generate as much relevant content as possible.

[0138] BART is one of the mainstream and high-performing pre-trained models for generative tasks.

[0139] Stylized: A BART-based planning model that only considers the structure between different sentences.

[0140] w / o graph: The method of this invention after removing the external knowledge graph. Due to the lack of a knowledge graph, the modeling within and between sentences is also removed.

[0141] w / o TC: The method of the present invention for removing the topic word planning and topic subgraph planning modules.

[0142] w / o comment: The method of the present invention is a method for removing the topic subgraph programming.

[0143] Evaluation indicators:

[0144] (1)BLEU: The overlap ratio of n-grams between standard output and model output is used as an indicator.

[0145] (2) Rouge-L: The longest common subsequence length between the standard output and the model output is used as the indicator.

[0146] (3)Distict: The diversity of words in n-grams in the model output is used as an indicator.

[0147] (4) Distance: The length difference between the standard output and the model output is used as an indicator. Here, outputs represents the length of the text output by the model, golden represents the length of the text output by the standard in the dataset, and abs represents the absolute value.

[0148]

[0149] Table 1. Performance comparison of the method of this invention and the comparison method on the AGENDA dataset.

[0150] Models BLEU-1 BLEU-2 BLEU-4 Rouge-L Dist-4 Distance Seq2seq 18.184 7.920 1.525 16.642 82.62 62.61 Rewriter 22.492 9.091 1.773 15.908 86.95 31.19 Cvae 20.462 11.988 2.543 14.681 52.83 5.20 ml-cvae 27.013 16.694 5.360 18.380 61.83 4.92 GAT 36.042 22.093 12.741 24.375 70.13 10.48 GraphWriter 38.495 25.690 13.023 27.640 73.72 27.33 BART 24.728 10.571 2.136 16.742 88.71 34.41 Stylized 25.961 18.076 4.554 19.203 92.72 52.02 Method of the present invention 39.823 25.932 10.511 29.253 91.51 4.14

[0151] Table 2 Performance comparison of the method of the present invention and its variants on the AGENDA dataset.

[0152] Models BLEU-1 Rouge-L Dist-1 Method of the present invention 39.823 24.074 63.95 -w / o graph 19.582 13.236 22.47 -w / o TC 29.744 17.857 28.18 -w / o comment 35.054 20.135 23.92

[0153] As can be clearly seen from the results of the various indicators in Table 1, the hierarchical planning model on the knowledge graph can effectively improve the overall quality of the generated paper abstract, making the generated text closer to the input title content and more coherent. To further confirm the impact of knowledge graphs and structural planning, Table 2 shows the performance of the variants of the method of this invention on the AGENDA dataset. It can be clearly seen from the table that any variant of the method of this invention performs worse than the complete method of this invention, proving the role of knowledge graphs and hierarchical structural planning.

[0154] The above description is intended to illustrate the technical solution of the present invention, and the present invention is not limited to the embodiments described above. Without departing from the spirit and scope of the claims, those skilled in the art can make many specific modifications based on the teachings of the present invention, and these modifications all fall within the scope of protection of the present invention.

[0155] Finally, it should be noted that the above examples are only used to illustrate the calculation process of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing examples, those skilled in the art should understand that modifications can still be made to the calculation process described in the foregoing examples, or equivalent substitutions can be made to some of the parameters. Such modifications or substitutions do not cause the essence of the corresponding calculation method to deviate from the spirit and scope of the calculation method of the present invention.

[0156] This invention is not limited to the embodiments described above. The above description of specific embodiments is intended to illustrate and explain the technical solutions of this invention. The specific embodiments described above are merely illustrative and not restrictive. Without departing from the spirit and scope of the claims, those skilled in the art can make many specific modifications based on the teachings of this invention, and these modifications all fall within the scope of protection of this invention.

[0157] References:

[0158] [1]Michael Alexander Kirkwood Halliday,Christian MIM Matthiessen,Michael Halliday,and Christian Matthiessen.An introduction to functionalgrammar.Routledge,2014.

[0159] [2] Qingyun Wang, Zhihao Zhou, Lifu Huang, Spencer Whitehead, BoliangZhang, Heng Ji, and Kevin Knight. Paper abstract writing through editing mechanism. In 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, pages 260–265. Association for Computational Linguistics (ACL), 2018.

[0160] [3]Tiancheng Zhao,Ran Zhao,and Maxine Esk′enazi.Learning Discourse-Level Diversity for Neural Dialog Models Using Conditional VariationAutoencoders.In ACL(1),2017.

[0161] [4] Dinghan Shen, Asli C, Elikyilmaz, Yizhe Zhang, Liqun Chen, Xin Wang, Jianfeng Gao, and Lawrence Carin.Towards generating long and coherent textwith multi-level latent variable models. In ACL(1),2019.

[0162] [5]Petar Veliˇckovi′c,William Cucurull,Arantxa Casanova,Adriana Romero,Pietro Li`o,and Joshua Bengio.Graph attention networks.InInternational Conference on Learning Representations,

[0163] [6] Rick Koncel-Kedziorski, Dhanush Bekal, Yi Luan, Mirella Lapata, and Hannaneh Hajishirzi. In Proceedings of the NAACL-HLT, pages 2284–2293,2

[0164] [7]Mike Lewis,Yinhan Liu,Naman Goyal,Marjan Ghazvininejad,AbdelrahmanMohamed, Omer Levy,Veselin Stoyanov,and Luke Zettlemoyer.Bart:Denoisingsequence-to-sequence pre-training for natural language generation,translation,and comprehension.In Proceedings of the 58th Annual Meeting ofthe Association for Computational Linguistics,pages 7871–7880,2020.

[0165] [8]Xiangzhe Kong,Jialiang Huang,Ziquan Tung,Jian Guan,and MinlieHuang.Stylized story generation with style-guided planning.In Findings ofthe Association for Computational Linguistics:ACL-IJCNLP 2021,pages 2430–2436,Online,August 2021.Association for Computational Linguistics.

Claims

1. A hierarchical planning method for generating paper abstracts based on topic-response theory, characterized in that, Includes the following steps: (1) The paper title and knowledge graph are encoded using a Transformer encoder and a Graph Transformer encoder respectively to obtain the representation of the paper title and knowledge graph; the knowledge graph is the knowledge graph corresponding to the paper abstract; (2) Keyword Planning: The paper title and the representations of nodes in the knowledge graph are concatenated, and the latent semantic information is learned. The resulting latent variables are denoted as... , It contains potential semantic information from paper titles and knowledge graphs; Used to initialize the topic term decoder; The topic term decoder is trained to generate topic term sequences. Keyword sequence It contains several keywords: , This indicates that the abstract of the paper contains One sentence; the first keyword in the keyword sequence. These are the keywords in the first sentence of the paper's abstract. These are the keywords in the second sentence of the paper's abstract, and so on. It is the first one in the paper abstract The keywords of each sentence The keyword decoder selects a node from the knowledge graph as a keyword at each step of the generation process. (3) Subgraph planning: For the keyword sequence Each keyword in the dataset has a corresponding title subgraph, and the set of title subgraphs is defined as follows: ; to construct the first Taking the title subgraph corresponding to each keyword as an example: (Taking keywords) Based on this, each node in the knowledge graph is scored, with the score ranging from [0,1]. Nodes with a score greater than 0.4 are considered relevant, while those with a score less than 0.4 are considered irrelevant. Finally, only nodes with scores greater than 0.4 are retained, while irrelevant nodes are discarded. The resulting subgraph of the knowledge graph is the topic subgraph, denoted as […]. ; (4) Based on keywords and the topic subgraph Proceed to the first Sentence generation: Because the sentences in the paper abstract influence each other, latent variables are set. Indicates the semantic state of the current sentence. Combination k -1 sentence state, as well as keywords and topic subgraphs, to learn the first k The semantic state that a sentence should have; Used to initialize the initial state of the sentence decoder; A supervised semaphore is used to monitor the probability that the sentence decoder copies nodes from the comment subgraph when generating each word, thereby avoiding repeated copying of nodes from the comment subgraph. (5) Repeat (3-4) until all keywords have been traversed, and then concatenate all the sentences to obtain the corresponding paper abstract.

2. The method for generating paper abstracts based on topic-response theory according to claim 1, characterized in that, Step (1) specifically includes: (101) Encode the paper title: Let the representation vector of the input paper title be denoted as . , Indicates the first in the paper title Embedding vectors of words, The Transformer encoder is used to encode the input paper title vector; the final representation vector is as follows: ,in It is the first in the title of the paper The representation of a word, ;definition It is the average of all the words, that is, the overall representation of the paper title; (102) Encode and represent the knowledge graph: Let the input knowledge graph be denoted as . ,in For a knowledge graph, it is a set of nodes. n It is the total number of nodes in the knowledge graph. It is the embedding vector of the i-th node in the knowledge graph; It is the set of edges in the knowledge graph, where It is a connection and The embedding vector of the edge; knowledge graph Encoding is performed by a Graph Transformer encoder; the final representation is obtained. , It is the first Each node represents; definition The average value is represented by all nodes.

3. The method for generating paper abstracts based on topic-response theory according to claim 1, characterized in that, Step (2) specifically includes: (201) Sampling of latent variables: Based on the conceptual structure of VAE, the paper title representation and the knowledge graph representation are first concatenated and then fed into the fully connected layer to obtain the mean of the Gaussian distribution. Latent variables are constructed based on the mean and variance. ;in The sampling conforms to the formula ; express The sampling process and Related, The sampling process is equivalent to In Gaussian function The resampling process below; where It is a general representation of the paper's title; ; (202) Generation of topic term sequences: The topic term sequence is generated by the topic term decoder. The topic term decoder generates one topic term per decoding operation. Therefore, the topic term decoder can decode... Next; the initial state of the keyword decoder Depend on initialization: , This is the initial state of the keyword decoder; different decoder states will yield different keywords; in order to generate For example, generate The decoder state at that time is According to formula (1), we obtain: A bidirectional GRU decoder applied to keyword generation; for k -1 keyword; To generate k The decoder state when -1 keyword; then Determined by formula (2): in For activation function, Let be the probability distribution function. For trainable parameters; Formula (2) can calculate the number of nodes selected in the knowledge graph. The probability will be determined by the number of nodes, and the node with the highest probability will ultimately be selected as the node with the highest probability. The keyword decoder repeatedly executes formulas (1) and (2) to finally obtain the keyword sequence. , This indicates that the abstract of the paper contains Each sentence contains 10 keywords, and each keyword corresponds to the main theme of a sentence. (203) Loss function for keyword planning for yes The posterior distribution, yes The prior distribution; It is a reconstruction loss function based on the posterior distribution; It is the KL divergence loss function between the prior and posterior distributions.

4. The method for generating paper abstracts based on topic-response theory according to claim 1, characterized in that, Step (3) specifically includes: (301) Using keywords Based on this, each node in the knowledge graph is scored, with the score ranging from [0,1]. Nodes with a score greater than 0.4 are considered relevant, while those with a score less than 0.4 are considered irrelevant. Finally, only nodes with scores greater than 0.4 are retained, while irrelevant nodes are discarded. The resulting subgraph of the knowledge graph is the topic subgraph, denoted as […]. The formula for obtaining the subgraph is expressed as follows: in This represents the sigmoid probability distribution function. For activation function, These are trainable parameters; For a bidirectional GRU network used in predicate subgraph programming, Indicates the scoring device is in The scoring status at that time Which nodes will be included in the middle? The decision is made regarding the nodes in the knowledge graph. In the scoring mode, if the score exceeds 0.4, the node is considered to be retained in the problem subgraph. Otherwise, it will be discarded; For knowledge graphs Node at The score below; Formula (6) describes It only retains scores greater than the current score. Node about The title subgraph; It is a topic subgraph The set represented by each node in the set; (302) The loss function of subgraph programming. for: in Indicates whether a node in a knowledge graph should be stored. In the code, 1 indicates that the node should be saved, and 0 indicates that the node should not be saved.

5. The method for generating paper abstracts based on topic-response theory according to claim 1, characterized in that, Step (4) specifically includes: Abstract , This indicates that the abstract of the paper contains 1 sentence, of which Abstract of the paper Y The Middle k (k = 1,2,…,|Y|) One sentence. , Sentence The first in t One word, The details are as follows: (401) Latent semantic state of sentences: To ensure the contextual coherence of the paper abstract, latent variables are set. Used to represent The latent semantic state is used to guide The generation; respectively obtain Mean of Gaussian distribution and variance Latent variables are constructed based on the mean and variance. ;in The sampling conforms to the formula ; express The sampling process and Related, among which The sampling process is equivalent to In Gaussian function The resampling process below; in detail This is an LSTM (Long Short-Term Memory) network applied to sentence state representation; It is a sentence Only subject to the former k -1 sentence state influenced by the latent semantic state of a sentence; however, the generation of a sentence is influenced not only by the context, but also by the topic words and the statement subgraph, therefore from Learning from these three is used for guidance The potential semantic state; (402) The generation process: The state of the sentence decoder Depend on , Indicates when decoding generates The sentence decoder state at the t-th word will affect... Word generation results; in A bidirectional GRU network applied to word generation; Let be the attention function. Keywords The form calculated by the attention mechanism This is the form of the predicate subgraph after computation using the attention mechanism; The formula generated by decoding is represented by formula (13): There is a supervisory signal used to control the generation process. Whether to copy relevant nodes from the subgraph in the context of the problem; To generate The probability distribution of time in the lexicon To generate The probability distribution of time at each node of the subgraph of the problem statement; Ultimately, the output will be selected from the two weighted probability distributions, with the highest probability being chosen as the final output. (403) Sentence Loss function of the generation process It consists of a generation loss function and a supervised copy loss function: yes The posterior distribution, yes The prior distribution; Based on The reconstruction loss function of the posterior distribution; yes The KL divergence loss functions for the prior and posterior distributions; where For supervised copying signals, A value of 1 indicates the probability of selecting a node from the vocabulary, while 0 indicates the probability of selecting a node from the topic subgraph.

6. The method for generating paper abstracts based on topic-response theory according to claim 1, characterized in that, In step (5), the generated keywords will be processed until all keywords have been traversed. By piecing them together, we obtain the corresponding paper abstract. The final complete loss function is: ; The loss function for keyword planning, Title Subgraph The loss function for planning; sentence The loss function of the generation process.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the hierarchical planning paper abstract generation method based on topic-state theory as described in any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the hierarchical planning paper abstract generation method based on topic-state theory as described in any one of claims 1 to 6.

Images