Multi-document summarization method and system incorporating explicit and implicit variational augmentation

Abstract

The application discloses a kind of multi-document literature abstract methods and systems combined with explicit and implicit variation enhancement, the method of the application includes: input document is captured using neural topic model explicit language topic representation, with initial abstract sentence or output last sentence abstract sentence fusion to obtain explicit fusion features, then using implicit variable model to obtain implicit upper feature, implicit upper feature is used Sentence decoder predicts the sentence planning of next sentence;Initial abstract sentence or output last sentence abstract sentence is input word decoder, combined with the sentence planning of next sentence to obtain next word until the sentence planning of next sentence is the terminator.The application can realize sentence planning control abstract generation, overcome the problems of internal decoding error and incoherent sentence context in decoding process, realize accurate and coherent abstract, improve the quality of abstract generation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-document scientific literature summarization, and particularly relates to a multi-document literature summarization method and system combining explicit and implicit variational enhancement. BACKGROUND

[0002] With the development of science and technology and the advent of the information age, people's access to information has become increasingly fast. However, at the same time, the exponential growth of massive information on the Internet has consumed people's limited time and energy, which is particularly evident in the field of science and technology. According to the statistics of literature statisticians, the number of scientific literature publications will double every nine years. A large number of published scientific literature often makes it difficult for researchers to quickly grasp the latest research achievements in the field or understand the research dynamics of cross-disciplines in a short period of time. The emergence of multi-document scientific literature summarization technology is to solve this demand. Its purpose is to generate a concise, condensed and accurate summary for a given single or multiple scientific literature, aiming to reduce the information overload pressure of researchers. According to the different ways of generating summaries, multi-document scientific literature summaries can be divided into extractive summaries and generative summaries. Extractive summaries are composed of existing sentences extracted from the original text. Since only the sentences of the original text need to be scored, and then sorted according to the score, combined with post-processing steps such as redundancy removal, reference replacement, and tense transformation, the summary can be generated. The advantage is simple and smooth. However, the generated summary is harsh, the sentences before and after are not coherent, and it does not conform to the pattern of human-generated summaries, often with poor readability. On the contrary, generative summaries aim to produce summaries in new ways, which can produce new words and sentences that do not exist in the original text, have coherent context, and strong readability. Therefore, in recent years, it has become the mainstream.

[0003] Early generative summarization mainly focuses on sentence compression. Researchers found that professional summarizers usually generate summaries by compressing the original sentences. This finding is based on the Ziff-Davis corpus, which is designed to match the sentences in the artificial summary with the original text. The results show that 78% of the summary sentences are generated by editing the original text, and more than half of these sentences are obtained by deleting part of the text. Sentence compression research includes rule-based methods and machine learning-based methods. Rule-based methods use information such as syntactic knowledge, context information, and word frequency to determine which parts of the sentence can be deleted; machine learning-based methods use data-driven learning to learn which nodes in the syntactic tree can be deleted, so they can get rid of the imperfections of rule-based methods. In recent years, with the explosive development of deep learning, text generation technology has gradually matured and begun to be applied in the field of multi-document scientific literature summarization. The generative summarization method based on deep learning uses an encoder-decoder framework. The encoder is responsible for encoding the input text sequence into a low-dimensional dense real-valued vector to extract text features, and the decoder decodes the corresponding output sequence from the vector. Initially, researchers used recurrent neural networks (RNN) or convolutional neural networks (CNN) as encoders, and recurrent neural networks as decoders. Later, with the introduction of the Transformer, its multi-head self-attention mechanism overcomes the local dependence of recurrent neural networks and achieves superior performance in various tasks. Current research on multi-document scientific literature summarization is also based on the Transformer-based encoder-decoder structure. Previous research on multi-document scientific literature summarization based on the Transformer can accurately model the content of scientific literature and the cross-document relationship, but the single-layer decoder structure makes it prone to errors during the decoding of long sequences, resulting in internal decoding errors and incoherent context problems. Therefore, how to improve the decoder structure to overcome the problems of internal decoding errors and incoherent context during the decoding process and generate more accurate and coherent summaries is a technical problem that needs to be solved in the task of multi-document scientific literature summarization. SUMMARY

[0004] The technical problem to be solved by the present application: In view of the above problems of the prior art, a multi-document literature summarization method and system combining explicit and implicit variational enhancement are provided. The present application can realize sentence planning control summary generation, overcome the problems of internal decoding errors and incoherent context during the decoding process, realize accurate and coherent summaries, and improve the quality of summary generation.

[0005] To solve the above technical problems, the technical scheme adopted by the present application is:

[0006] A multi-document literature abstract method combining explicit and implicit variational enhancement, comprising:

[0007] S101, capturing the explicit sentence topic representation of the input document by using a neural topic model;

[0008] S102, obtaining the abstract sentence representation of the initial abstract sentence or the output last abstract sentence through an abstract encoder, and fusing the explicit abstract sentence representation with the projected representation of the projection layer to obtain the explicit fusion feature Wherein the superscript S represents the abstract;

[0009] S103, inputting the explicit fusion feature into an implicit variable model and combining it with the input document encoding obtained by the input document encoder to obtain the implicit upper feature Wherein the subscript n represents the number of words contained in the current abstract sentence;

[0010] S104, inputting the implicit upper feature into a sentence decoder and combining it with the input document encoding obtained by the input document encoder to predict the sentence planning of the next sentence

[0011] S105, inputting the initial abstract sentence or the output last abstract sentence into a word decoder, and combining it with the sentence planning of the next sentence to obtain the next word w nj+1 until the sentence planning of the next sentence is the end symbol, and finally the word w n1 ~ word w nj+1 constitutes the abstract sentence of the input document.

[0012] Optionally, the function expression of the explicit fusion feature obtained by fusing the explicit abstract sentence representation with the projected representation of the projection layer in step S102 is:

[0013]

[0014]

[0015]

[0016] In the above formula, represents the topic representation of the i-th abstract sentence, represents the feature after the projection transformation of , and ηi a ratio representing feature fusion, representing fused features, Tanh and Sigmoid are activation functions, W p and W f · are weight matrices, b p and b f are bias quantities.

[0017] Optionally, the summary encoder and the document encoder each comprise a Transformer encoding layer and a multi-head pooling layer connected in sequence.

[0018] Optionally, the latent variable model in step S103 comprises a Transformer encoding layer, a feedforward neural network layer, a feedforward layer and an addition and normalization module connected in sequence, the Transformer encoding layer comprises a multi-head self-attention module, an addition and normalization module and a multi-head cross-attention module connected in sequence, and the explicit fusion feature input into the latent variable model is first input into the multi-head self-attention module of the Transformer encoding layer, and then combined with the input document encoding obtained by inputting the input document into the document encoder in the multi-head cross-attention module of the Transformer encoding layer, and then the prior distribution and the posterior distribution are calculated respectively through the feedforward neural network layer, and then the implicit upper-level features are captured through the subsequent feedforward layer and the addition and normalization module wherein subscript n represents the number of words currently contained in the summary sentence; wherein the function expression for calculating the prior distribution and the posterior distribution by the feedforward neural network layer is:

[0019]

[0020] In the above formula, p θ ( i |,{s ＜i}) is the prior distribution, z i is the latent variable, D is the input document, s ＜i is the semantic representation of the sentence before the i-th summary sentence, μ i and σ i are learnable parameters, and I is an identity matrix;

[0021]

[0022] In the above formula, p θ ( i |,S) is the posterior distribution, S is a given complete summary, μ i ′ and σ i ′ are learnable parameters.

[0023] Optionally, the word decoder in step S104 comprises 6 Transformer decoding layers, a linear layer, a copy layer and a combination layer, the 6 Transformer decoding layers are sequentially cascaded, each Transformer decoding layer comprises a multi-head self-attention module, an addition and normalization module, a multi-head cross-attention module, a feed-forward layer and an addition and normalization module connected in sequence, the implicit upper-layer features are input from the first Transformer decoding layer, and the multi-head cross-attention fusion is performed with the input document encoding obtained by the input document encoder in the multi-head cross-attention module of each Transformer decoding layer, and the output of the last Transformer decoding layer is input into the linear layer to obtain the probability mapping p of the vector representation to the word table g , the copy layer extracts the probability p of the word copied from the input scientific literature c , the combination layer combines the probability mapping p of the vector representation to the word table g and the probability p of the word copied from the input scientific literature c to obtain the final word generation probability p; the function expression of the multi-head cross-attention fusion is:

[0024]

[0025] In the above formula, represents the output of the multi-head cross-attention fusion, LN represents the layer normalization operation, CrossAtt represents the multi-head cross-attention operation, represents the word decoder self-attention layer output, l represents the current l-th layer, dw represents the word decoder, represents the word representation of the input document, N e is the total number of layers of the encoder, D represents the input document, e represents the encoder, represents the sentence planning that fuses the explicit and implicit features, N ds represents the total number of layers of the sentence decoder, S represents the summary, ds is the sentence decoder; the function expression of the probability mapping p of the vector representation to the word table g is:

[0026]

[0027] In the above formula, softmax represents the softmax activation function, W g represents the weight matrix, b g represents the bias, represents the output of the last Transformer decoding layer.

[0028] Optionally, the word decoder in step S104 comprises 6 Transformer decoding layers which are sequentially cascaded, each of the 6 Transformer decoding layers comprises a multi-head self-attention module, an addition and normalization module, a multi-head cross-attention module, a feed-forward layer and an addition and normalization module which are sequentially connected, the initial summary sentence or the output last summary sentence is input from the first Transformer decoding layer, and in the multi-head cross-attention module of each Transformer decoding layer, the multi-head cross-attention fusion is performed between the input document and the input document encoding obtained by the input document encoder, and the sentence planning of the next sentence is fused in the feed-forward layer and the next word w is obtained after the last addition and normalization module. until the sentence planning of the next sentence is obtained. nj+1 until the sentence planning of the next sentence is obtained. is a terminal symbol, and finally the word w is obtained. n1 is a terminal symbol, and finally the word w is obtained. nj+1 to constitute the summary sentence of the input document.

[0029] Optionally, before step S101, the neural topic model, the latent variable model, the sentence decoder and the word decoder are trained, and when training, the initial summary sentence or the output last summary sentence is a standard summary from the input document.

[0030] Optionally, when training the neural topic model, the latent variable model, the sentence decoder and the word decoder, the function expression of the loss function is as follows:

[0031]

[0032] In the above formula, represents the loss function, represents the likelihood function of summary generation, represents the loss function of the neural topic model, represents the loss of bag-of-words reconstruction, and λ and η are adjustment factors, and have:

[0033]

[0034]

[0035]

[0036]

[0037]

[0038] In the above formula, p(S|D) represents the probability of generating a summary S from an input document D, p(d) represents the probability of generating a bag-of-words representation d from the document, and |S| represents the number of words in the summary. This means finding the given z. i Mathematical expectation under the given conditions Represents a given latent variable z i s ＜i And the bag-of-words representation for generating the i-th summary statement when inputting document D. The probability, q φ (z i |S,D) represents the latent variable z generated given a complete summary S and an input document D. i The posterior distribution of the probability, The bag-of-words representation of the i-th summary statement, s <i This represents the semantic representation of the statements preceding the i-th summary statement, logp(s) i |s <i D) represents a given s <i The log-likelihood of the probability that _ and D generate the semantic representation of the i-th statement, p(s i |s <i ,z i D) represents a given s <i _Hidden variable z i The probability of generating the semantic representation of the i-th statement when the document representation is D, KL(q) φ (z i |S,D)|p θ (z i |s <i ,D)) represents the latent variable z i The KL distance between the prior and posterior distributions, p θ (z i |s <i D) represents the latent variable z i The prior distribution of is determined by the statements preceding the i-th sentence and the document representation D. Let represent the expected value of a given topic distribution θ, and let logp(d|β,θ) represent the distribution θ from a given document to a topic and the distribution β from a topic to a word in the topic model. Let logp(d|β,θ) represent the log-likelihood of generating the bag-of-words representation d of a document, where β represents the distribution from a topic to a word, θ represents the document topic distribution, KL(q(θ|d)|p(θ)) represent the KL distance between the prior and posterior distributions of the document topic distribution θ, q(θ|d) represent the posterior probability distribution of generating the document topic distribution θ from the bag-of-words representation d of the document, and p(θ) represent the prior probability distribution of the document topic distribution θ. Represents the estimated document topic distribution Based on the topic and word distribution β, generate the nth word w. nthe probability of w n denotes the n-th word, denotes the estimated document topic distribution, denotes the sampled topic latent variable the KL distance between the prior and posterior distribution of denotes the bag-of-words representation d of a given document, generating the topic latent variable the posterior distribution of denotes the sampled topic latent variable, denotes the topic latent variable the prior distribution of

[0039] Furthermore, the present application also provides a multi-document literature summarization system combining explicit and implicit variational enhancement, comprising a microprocessor and a memory connected to each other, the microprocessor being programmed or configured to execute the multi-document literature summarization method combining explicit and implicit variational enhancement.

[0040] Furthermore, the present application also provides a computer readable storage medium, wherein a computer program is stored, the computer program being used for programming or configuring a microprocessor to execute the multi-document literature summarization method combining explicit and implicit variational enhancement.

[0041] Compared with the prior art, the present application mainly has the following advantages:

[0042] 1. The present application realizes the upper control of the multi-document scientific literature summary word decoding by using the summary sentence planning, which helps to generate a summary with correct grammar and more coherent context semantics.

[0043] 2. The present application can effectively enrich the summary sentence representation by using the sentence topic information provided by the neural topic model and the implicit feature information provided by the control sentence generation of the latent variable model, which helps to further improve the summary performance. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 is the basic flowchart of the embodiment method of the present application.

[0045] Figure 2 is the principle schematic diagram of the network structure in the use state in the embodiment of the present application.

[0046] Figure 3 is the principle schematic diagram of the network structure in the training state in the embodiment of the present application.

[0047] Figure 4 is the training and testing flowchart of the network structure in the embodiment of the present application. DETAILED DESCRIPTION

[0048] As Figure 1 andFigure 2 As shown, the embodiment of the multi-document literature summary method combining explicit and implicit variational enhancement includes:

[0049] S101, capturing the explicit sentence topic representation of the input document by using the neural topic model;

[0050] S102, obtaining the summary sentence representation of the initial summary sentence or the output last summary sentence through the summary encoder, and fusing the summary sentence representation with the projected representation of the explicit summary sentence representation through the projection layer to obtain the explicit fusion feature Wherein the superscript S represents summary;

[0051] S103, inputting the explicit fusion feature into the implicit variable model, and combining the input document code obtained by the input document encoder to obtain the implicit upper layer feature Wherein the subscript n represents the number of words contained in the summary sentence;

[0052] S104, inputting the implicit upper layer feature into the sentence decoder, and combining the input document code obtained by the input document encoder to predict the sentence planning of the next sentence

[0053] S105, inputting the initial summary sentence or the output last summary sentence into the word decoder, and combining the sentence planning of the next sentence to obtain the next word w nj+1 until the sentence planning of the next sentence is the terminal symbol, and finally the word w n1 ~ word w nj+1 is obtained to constitute the summary sentence of the input document.

[0054] In the embodiment, the neural topic model specifically adopts Neural Topic Model (abbreviated as NTM), which is a network structure. For details, please refer to the literature: “Miao Y, Grefenstette E, Blunsom P. Discovering discrete latent topics with neural variational inference [C] / / International Conference on Machine Learning. PMLR, 2017: 2410-2419.”. The neural topic model takes the bag-of-words representation as input, and the bag-of-words representation of the sentence is where |V| denotes the size of the vocabulary, and each dimension corresponds to the number of times the corresponding word appears in the sentence. After training the neural topic model, we use the bag-of-words representation of the summary sentence as input to infer the topic distribution of the summary sentence using the neural topic model The topic representation of the summary sentence is obtained by combining the topic vector representation φ obtained by model training

[0055] In this embodiment, the explicit summary sentence representation is fused with the projected representation of the projection layer in step S102 to obtain the explicit fusion feature The function expression is:

[0056]

[0057]

[0058]

[0059] In the above formula, denotes the topic representation of the i-th summary sentence, denotes the feature after projection transformation of η i denotes the proportion of feature fusion, denotes the fused feature, Tanh and Sigmoid are activation functions, W p and W f · are weight matrices, and b p and b f are bias quantities.

[0060] In this embodiment, the summary encoder and the document encoder each include a Transformer encoding layer and a multi-head pooling layer connected in sequence. The Transformer encoding layer is a conventional network structure, and details can be found in the literature: “Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need [J]. Advances in neural information processing systems, 2017, 30.”. The Transformer encoder contains N e layers, each of which includes a multi-head self-attention sublayer (SelfAtt) and a feedforward neural network sublayer (FFN) for encoding each word of the input. In this embodiment, the encoding of the input document and the standard summary is treated differently because the encoding of the input scientific literature will be used to calculate cross-attention during subsequent decoding, while the encoding of the standard summary is to obtain the initialized summary sentence vector representation. In this embodiment, the word representation obtained after encoding the input document is where i, j, k represent the k-th word of the j-th sentence of the i-th document, respectively. The word representation after encoding the standard abstract is where i, j represent the k-th word of the i-th sentence, respectively. The multi-head pooling layer is a network structure, which can be referred to in the document “Liu Y, Lapata M. Hierarchical Transformers for Multi-Document Summarization [C] / / Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019: 5070-5081.”. The function expression is:

[0061]

[0062] In the above formula, is the abstract sentence representation obtained by using the word representation of the i-th sentence, and MHPool represents the multi-head pooling layer, respectively, the word representation of the j+1-th word of the i-th sentence.

[0063] Capture the implicit upper-level features of the control abstract sentence generation, such as argumentation function, sentiment polarity, rhetorical role, etc. See Figure 2 , the hidden variable model in step S103 of the embodiment includes a Transformer encoding layer, a feedforward neural network layer, a feedforward layer, and an addition and normalization module connected in sequence, the Transformer encoding layer includes a multi-head self-attention module, an addition and normalization module, and a multi-head cross-attention module connected in sequence, and the explicit fusion features input into the hidden variable model First, input the multi-head self-attention module of the Transformer encoding layer, and combine the input document encoding obtained by the input document encoder with the multi-head cross-attention module of the Transformer encoding layer, and then calculate the prior distribution and the posterior distribution through the feedforward neural network layer respectively, and then capture the implicit upper-level features through the subsequent feedforward layer and the addition and normalization module where subscript n represents the number of words currently contained in the abstract sentence; wherein, the function expression for calculating the prior distribution and the posterior distribution by the feedforward neural network layer is:

[0064]

[0065] In the above formula, p θ (z i |D,{s ＜i}) is the prior distribution, z i Let s be a latent variable, D be the input document, and s be the input document. ＜i μ represents the semantic representation of the statements preceding the i-th summary statement. i and σ i Here, I represents the learnable parameters and the identity matrix;

[0066]

[0067] In the above formula, p θ (z i |D,S) represents the posterior distribution, S is the given complete summary, and μ′ i and σ′ i These are learnable parameters.

[0068] In this embodiment, the latent variable model borrows from the Conditional Variational Auto-Encoder (CVAE) structure, as described in the literature "Sohn K, Lee H, Yan X. Learning structured output representation using deep conditional generative models[J]. Advances in neural information processing systems, 2015, 28.". The goal is to learn the latent variable z to generate the output Y under the joint control of the input X and z.

[0069] p θ (Y|X)=∫ z p θ (Y|X,z)·p θ (x|X)dz,

[0070] In the above formula, p θ (Y|X) represents the probabilities of input X and output Y given network parameters θ, p θ (Y|X,z) represents the probability of output Y being jointly controlled by inputs X and z given network parameters θ, ·p θ (z|X) represents the probability of input X and latent variable z given network parameters θ; the optimization objective function is a variational objective with a lower bound of ELBO.

[0071]

[0072] In the above formula, q represents the expected value, KL represents the KL distance, and q represents the expected value. φ(z|X, Y) represents the prior distribution of the latent variable z relative to the input X and the output Y. Therefore, in this embodiment, the prior distribution and the posterior distribution are calculated by the feedforward neural network layer respectively, and the parameters μ i , σ i , μ' i and σ' i of the Gaussian distribution are learnable parameters.

[0073] The Transformer encoding layer takes the abstract sentence representation as the query, and sequentially passes through the multi-head self-attention sublayer with the rest of the abstract sentence representations and the multi-head cross-attention sublayer with the input scientific literature D to obtain to realize the modeling of the input information and the sentence representation context information. Then, a feedforward neural network FFN is used to realize the prediction of the parameters μ i , σ i of the Gaussian distribution, respectively The prediction method of the parameters μ' i , σ' i of the Gaussian distribution is the same. Through the above steps, the parameters of the Gaussian distribution are obtained, and the latent variable z i of the abstract sentence s i can be sampled. Then, another feedforward neural network (FFN) is used to realize the fusion of the abstract sentence representation and z i , specifically: After the above steps, the abstract sentence representation is obtained, which integrates the neural topic model information and the latent variable information.

[0074] The above steps use the neural topic model and the latent variable model to capture the explicit topic-level features and the implicit upper-level features that control the generation of the abstract sentence, enriching the representation of the abstract sentence. However, to realize the sentence-level planning generation, a sentence decoder needs to be designed to generate the sentence planning of the next sentence to be decoded in advance according to the previously decoded sentence. In this embodiment, the word decoder in step S104 includes 6 Transformer decoding layers, a linear layer, a copy layer and a combination layer, and the 6 Transformer decoding layers are connected in cascade. Each Transformer decoding layer includes a multi-head self-attention module, an addition and normalization module, a multi-head cross-attention module, a feedforward layer and an addition and normalization module connected in sequence, and the implicit upper-level features from the first Transformer decoding layer input, and multi-head cross attention fusion with the input document encoding obtained by the input document encoder in each multi-head cross attention module of each Transformer decoding layer, and the output of the last Transformer decoding layer is fed into a linear layer to obtain a vector representation of a probability mapping p to a word vocabulary g , the copy layer extracts a probability p of a word being copied from the input scientific literature c , the combination layer combines the probability mapping p of the vector representation to the word vocabulary g and the probability p of the word being copied from the input scientific literature c to obtain a final word generation probability p; the function expression for the multi-head cross attention fusion is:

[0075]

[0076] In the above formula, represents the output of the multi-head cross attention fusion, LN represents the layer normalization operation, CrossAtt represents the multi-head cross attention operation, represents the word decoder self-attention layer output, l represents the current l-th layer, and dw represents the word decoder, represents the word representation of the input document, N e is the total number of layers of the encoder, D represents the input document, and e represents the encoder, represents the sentence planning that fuses explicit and implicit features, N ds represents the total number of layers of the sentence decoder, S represents the summary, and ds is the sentence decoder; the function expression of the probability mapping p of the vector representation to the word vocabulary g is:

[0077]

[0078] In the above formula, softmax represents the softmax activation function, W g represents the weight matrix, b g represents the bias, represents the output of the last Transformer decoding layer.

[0079] In order to alleviate the Out-Of-Vocabulary (OOV) problem of words in the summary that are not in the vocabulary during decoding, the copy layer in this embodiment uses the Copy mechanism to extract a probability p of a word being copied from the input scientific literature cThe method is the existing method, and details can be found in "See A, Liu P J, Manning C D. Get To The Point: Summarization with Pointer-Generator Networks [C] / / Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2017: 1073-1083."

[0080] The combined layer maps the probability p of representing the vector to the word vocabulary g And the probability p of copying the word from the input scientific literature c The function expression of the final word generation probability p is obtained by combining the two:

[0081] p = λ·p g +(1-λ)·p c ,

[0082] In the above formula, λ is a learning parameter.

[0083] As Figure 2 shown, the word decoder in step S104 of the embodiment includes 6 Transformer decoding layers, which are connected in cascade, each Transformer decoding layer includes a multi-head self-attention module, an addition and normalization module, a multi-head cross-attention module, a feedforward layer and an addition and normalization module connected in sequence, the initial summary sentence or the last output summary sentence is input from the first Transformer decoding layer, and the multi-head cross-attention module in each Transformer decoding layer is fused with the input document encoding obtained by the input document encoder through the input document, and the sentence planning of the next sentence is fused in the feedforward layer and the next word w nj+1 is obtained through the last addition and normalization module until the sentence planning of the next sentence is a terminal symbol, and finally the summary sentence of the input document composed of the words w n1 ~ w nj+1 is obtained.

[0084] Referring to Figure 3 and Figure 4, the embodiment further includes training the neural topic model, the latent variable model, the sentence decoder and the word decoder before step S101, and the initial summary sentence or the last output summary sentence is a standard summary from the input document. In the model training stage of the latent variable model, since the standard summary is available, the latent variable z i is calculated by the posterior distribution, and the KL divergence between the prior distribution and the posterior distribution is optimized to make the prior distribution and the posterior distribution close enough; in the model inference stage, since the summary sentence is generated sentence by sentence, the latent variable z i of the sentence to be generated is obtained by the prior distribution.

[0085] As shown in Figure 4 , the process of training the neural topic model, the latent variable model, the sentence decoder and the word decoder includes: first, encoding the input multiple scientific literatures and the standard summary respectively. Second, training the neural topic model, training the neural topic model using the input scientific literature sentences, and then inferring the topic representation of the summary sentence. Third, training the latent variable model. Fourth, generating the sentence plan of the next sentence using the sentence decoder. Fifth, generating the complete summary using the word decoder. Sixth, overall model training. Optimize the summary generation likelihood, the neural topic model objective function and the bag-of-words reconstruction objective function. Seventh, model verification. Save the model node every 4000 training steps, test the model performance on the validation set, and find the optimal performance node for testing. In this embodiment, we set the learning rate warmup when training the model, so that the learning rate gradually increases in the first N steps, and when the training step number exceeds N, the learning rate gradually decays. The total number of training steps of the model is N S , and the model is trained every S steps, i.e. the training intermediate node is saved for subsequent verification and testing. In addition, since the number of rounds required for training the neural topic model is more than that of the summary model, we pre-train the neural topic model N T steps and save the neural topic model training node, and then load the training together with the summary model.

[0086] In this embodiment, when training the neural topic model, the latent variable model, the sentence decoder and the word decoder, the function expression of the loss function used is:

[0087]

[0088] In the above formula, represents the loss function, represents the summary generation likelihood function, represents the loss function of the neural topic model, represents the loss of bag-of-words reconstruction, and λ and η are adjustment factors, and have:

[0089]

[0090]

[0091]

[0092]

[0093]

[0094] In the above formula, p(S|D) represents the probability of generating a summary S from an input document D, p(d) represents the probability of generating a bag-of-words representation d from the document, and |S| represents the number of words in the summary. This means finding the given z. i Mathematical expectation under the given conditions Represents a given latent variable z i s ＜i And the bag-of-words representation for generating the i-th summary statement when inputting document D. The probability, q φ (z i |S,D) represents the latent variable z generated given a complete summary S and an input document D. i The posterior distribution of the probability, The bag-of-words representation of the i-th summary statement, s <i This represents the semantic representation of the statements preceding the i-th summary statement, logp(s) i |s <i D) represents a given s <i The log-likelihood of the probability that _ and D generate the semantic representation of the i-th statement, p(s i |s <i ,z i D) represents a given s <i _Hidden variable z i The probability of generating the semantic representation of the i-th statement when the document representation is D, KL(q) φ (z i |S,D)|p θ (z i |s <i ,D)) represents the latent variable z i The KL distance between the prior and posterior distributions, p θ (z i |s <i D) represents the latent variable z i The prior distribution of is determined by the statements preceding the i-th sentence and the document representation D. Let represent the expected value of a given topic distribution θ, and let logp(d|β,θ) represent the distribution θ from a given document to a topic and the distribution β from a topic to a word in the topic model. Let logp(d|β,θ) represent the log-likelihood of generating the bag-of-words representation d of a document, where β represents the distribution from a topic to a word, θ represents the document topic distribution, KL(q(θ|d)|p(θ)) represent the KL distance between the prior and posterior distributions of the document topic distribution θ, q(θ|d) represent the posterior probability distribution of generating the document topic distribution θ from the bag-of-words representation d of the document, and p(θ) represent the prior probability distribution of the document topic distribution θ. Represents the estimated document topic distribution Based on the topic and word distribution β, generate the nth word w. n The probability, w n This represents the nth word. This represents the estimated distribution of document topics. Topic latent variables representing sampling The KL distance between the prior and posterior distributions. Let d be the bag-of-words representation of a given document, and generate topic latent variables. The posterior distribution, This represents the latent variable for the sampled topic. Representing topic-hidden variables The prior distribution of .

[0095] like Figure 4As shown, the process tested in the embodiment includes: first, encoding the inputted multiple scientific literature and the generated previous sentence summary statement. Second, neural topic model inference, using the bag-of-words representation of the previous generated sentence to infer the topic representation. Third, latent variable model inference. Using the prior network to infer the latent variable of the previous sentence. Fourth, using the sentence decoder to generate the sentence plan of the next sentence. Fifth, using the word decoder to generate the complete summary. Sixth, model performance testing. The model performance is evaluated by using the ROUGE evaluation index. In the embodiment, the performance of the optimal node is selected from the trained several model nodes to test by using the validation set data. In the model testing, the model parameters are fixed and no longer updated, the beam search width of the decoding is set to B, that is, the top five candidates with the highest scores are saved when the model predicts each word, which to some extent avoids the shortcomings of the greedy sampling method and improves the quality of sentence generation; the minimum generation length of the summary is set to L words; the N-gram blocking strategy is also set in the embodiment to reduce the repetition rate of the model N-gram and reduce the redundancy of the summary. The model effect evaluation in the embodiment uses the recognized evaluation index ROUGE in the summary generation (for details, refer to "Lin C Y, Hovy E. Manual and automatic evaluation of summaries [C] / / Proceedings of the ACL-02 Workshop on Automatic Summarization. 2002: 45-51." which can be translated as "Manual and automatic evaluation of summaries [C] / / Proceedings of the ACL-02 Workshop on Automatic Summarization. 2002: 45-51." published in the 2002 ACL Conference on Automatic Summarization), wherein ROUGE-1 and ROUGE-2 are used to evaluate the information amount of the summary generation, and ROUGE-L is used to evaluate the fluency of the summary.

[0096] In the embodiment, the experimental conditions use one NVIDIA GeForce TITAN X GPU workstation, the operating system is Ubuntu 16.04, and the platform is based on PyTorch. The data set uses the Multi_Xscience multi-document scientific literature summary data set, the training set of which contains 30369 samples, each sample includes the summary section of several scientific literature as input and the standard summary as output; the validation set and the test set contain 5066 and 5093 samples respectively. The evaluation results of the method of the embodiment are compared with the evaluation results of the benchmark method as shown in Table 1.

[0097] Method category ROUGE-1 ROUGE-2 ROUGE-L Baseline method 34.11 6.76 30.63 Example method 37.23 8.00 32.82

[0098] As can be seen from Table 1, the evaluation results of the benchmark method on ROUGE-1 / 2 / L are 34.11 / 6.76 / 30.63 respectively. The evaluation results of the method of the embodiment on ROUGE-1 / 2 / L are 37.23 / 8.00 / 32.82, which are all better than the results of the benchmark method.

[0099] In summary, the method of the embodiment encodes the inputted multiple scientific literatures and standard abstracts by two parameter-independent encoders respectively to obtain vector representations of the scientific literatures and the standard abstract sentences; trains a neural topic model by using the bag-of-words representation of the scientific literature sentences and infers the abstract sentence topic representation by using the bag-of-words representation of the abstract sentences as input; simultaneously trains a latent variable model to integrate the upper-layer features controlling the generation of the abstract sentences into the latent variables; combines the topic representation and the latent variable representation into the abstract sentence representation and inputs the abstract sentence representation into a sentence decoder to obtain the sentence planning of the next abstract sentence for the final word decoding, realizes the abstract generation controlled by the sentence planning, can realize the abstract generation controlled by the sentence planning, overcomes the problems of the internal decoding error of the sentence and the incoherence of the context of the sentence in the decoding process, realizes the accurate and coherent abstract, and improves the quality of the abstract generation.

[0100] In addition, the embodiment also provides a multi-document literature abstract system combining explicit and implicit variational enhancement, which comprises a microprocessor and a memory connected with each other, the microprocessor is programmed or configured to execute the multi-document literature abstract method combining explicit and implicit variational enhancement. In addition, the embodiment also provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program is used for programming or configuring the microprocessor to execute the multi-document literature abstract method combining explicit and implicit variational enhancement.

[0101] Those skilled in the art will appreciate that embodiments of the application can be readily used as software, hardware, or a combination of software and hardware. In a software embodiment, various software modules in accordance with embodiments of the application are stored in a memory such as a computer program product (e.g., a disk storage, memory, etc.) and executed by a processor. As such, various program modules in accordance with embodiments of the application can include program instructions (e.g., machine-readable instructions) that, when executed by a processor, cause the processor to perform operations in accordance with embodiments of the application. The program instructions can be stored in a computer readable medium, which can include a memory, a disk storage, etc. The program instructions can be executed by a processor to cause the processor to perform operations in accordance with embodiments of the application. Figure 1 one or more functions specified in the flow or flows and / or blocks Figure 1 one or more functions specified in the flow or flows and / or blocks Figure 1 one or more functions specified in the flow or flows and / or blocks Figure 1 one or more functions specified in the flow or flows and / or blocks Figure 1 one or more functions specified in the flow or flows and / or blocks Figure 1 one or more functions specified in the flow or flows and / or blocks

[0102] The above description is only preferred embodiments of the application. The protection scope of the application is not limited to the above-mentioned embodiments. Any technical scheme falling within the concept of the application should be considered as falling within the protection scope of the application. It should be noted that some improvements and refinements made by those skilled in the art without departing from the principles of the application should also be considered as falling within the protection scope of the application.

Claims

1. A multi-document literature summarization method incorporating both explicit and implicit variational augmentation, characterized in that, Comprising: S101, capturing explicit sentence topic representation of input document by using neural topic model; S102, obtaining the summary sentence representation of the initial summary sentence or the output last summary sentence by a summary encoder, and fusing the explicit summary sentence representation with the projected representation of the explicit summary sentence representation by a projection layer to obtain explicit fusion features wherein the superscript S represents summary; S103, obtaining the explicit fusion feature The input latent variable model and the input document code obtained by combining the input document through the input document encoder capture the implicit upper layer feature Wherein subscript n represents the number of words currently contained in the summary sentence; S104, the implicit upper layer features The input sentence decoder, in combination with the input document encoding obtained by the input document encoder, predicts the sentence plan of the next sentence ; S105, inputting the initial summary sentence or the output last summary sentence into a word decoder, combining the next sentence plan to obtain the next word w nj+1 until the next sentence plan as a terminal symbol, finally obtaining the word w n1 ~ the word w nj+1 to constitute a summary sentence of the input document; The hidden variable model in step S103 includes a Transformer encoding layer, a feedforward neural network layer and a feedforward layer connected in sequence, and an addition and normalization module, the Transformer encoding layer includes a multi-head self-attention module, an addition and normalization module and a multi-head cross-attention module connected in sequence, and the explicit fusion feature after inputting the hidden variable model First, the multi-head self-attention module of the Transformer encoding layer is input, and then the multi-head cross-attention module of the Transformer encoding layer is combined with the input document code obtained by inputting the input document through the input document encoder, and then the prior distribution and the posterior distribution are calculated through the feedforward neural network layer respectively, and then the prior distribution and the posterior distribution are captured to obtain the implicit upper layer feature through the subsequent feedforward layer and the addition and normalization module Wherein subscript n represents the number of words currently contained in the summary sentence; wherein the function expression for calculating the prior distribution and the posterior distribution by the feedforward neural network layer is: ， In the above formula, is a prior distribution, is a latent variable, is an input document, is a semantic representation of the sentence preceding the i-th summary sentence, and are learnable parameters, is an identity matrix; ， In the above formula, is the posterior distribution, is the given complete summary, and are the learnable parameters.

2. The method for multi-document summarization with combined explicit and implicit variational augmentation according to claim 1, wherein, In step S102, the projection representation of the explicit summary sentence representation and the projection representation of the projected layer are fused to obtain an explicit fusion feature The function expression is: In the above formula, represents the first The topic of the abstract sentence is represented by, represents the feature after the projection transformation, represents the proportion of feature fusion, and is an activation function, and is a weight matrix, and is a bias.​​ 3. The method for multi-document summarization with combined explicit and implicit variational augmentation according to claim 1, wherein, The summary encoder and the document encoder each comprise a Transformer encoding layer and a multi-head pooling layer connected in sequence.

4. The method for multi-document summarization with combined explicit and implicit variational augmentation according to claim 1, wherein, The word decoder in step S104 includes 6 Transformer decoding layers, a linear layer, a copying layer and a combination layer, the 6 Transformer decoding layers are sequentially cascaded, each of the Transformer decoding layers includes a multi-head self-attention module, an addition and normalization module, a multi-head cross-attention module, a feedforward layer and an addition and normalization module connected in sequence, the implicit upper layer features are input from the first Transformer decoding layer, and the multi-head cross-attention fusion is performed with the input document encoding obtained by the input document encoder in the multi-head cross-attention module of each Transformer decoding layer, and the output of the last Transformer decoding layer is input into the linear layer to obtain a vector representation of a probability mapping to a word table , the copying layer extracts a probability of a word copied from the input scientific literature , the combination layer combines the vector representation of the probability mapping to the word table and the probability of the word copied from the input scientific literature to obtain a final word generation probability ; and the function expression for performing the multi-head cross-attention fusion is: In the above formula, denotes the output of multi-head cross-attention fusion, LN denotes the layer normalization operation, CrossAtt denotes the multi-head cross-attention operation, denotes the output of the word decoder self-attention layer, denotes the current is the layer, dw denotes the word decoder, denotes the word representation of the input document, is the total number of layers of the encoder, D denotes the input document, e denotes the encoder, denotes the sentence planning that fuses explicit and implicit features, denotes the total number of layers of the sentence decoder, S denotes the summary, ds is the sentence decoder; the vector represents a probability mapping to a word vocabulary The functional expression is: In the above formula, softmax represents a softmax activation function, represents a weight matrix, represents a bias, represents the output of the last Transformer decoding layer.

5. The method for multi-document summarization with explicit and implicit variational augmentation according to claim 1, wherein, The word decoder in step S104 includes 6 Transformer decoding layers which are sequentially connected in cascade, each of which includes a multi-head self-attention module, an addition and normalization module, a multi-head cross-attention module, a feedforward layer and an addition and normalization module which are sequentially connected, the initial summary sentence or the last output summary sentence is input from the first Transformer decoding layer, and in the multi-head cross-attention module of each Transformer decoding layer, the multi-head cross-attention fusion is performed with the input document encoding obtained by the input document encoder from the input document, and the sentence planning of the next sentence is fused in the feedforward layer and the next word w is obtained after the last addition and normalization module The final summary sentence of the input document is composed of the word w nj+1 until the sentence planning of the next sentence is a terminal symbol, and finally the word w n1 is obtained nj+1 The final summary sentence of the input document is composed of the word w 6. The method for multi-document summarization with explicit and implicit variational augmentation according to claim 1, wherein, Before step S101, the neural topic model, the latent variable model, the sentence decoder and the word decoder are trained, and the initial summary sentence or the last output summary sentence is a standard summary from the input document.

7. The method for multi-document summarization with explicit and implicit variational augmentation according to claim 6, wherein, When training the neural topic model, the latent variable model, the sentence decoder and the word decoder, the function expression of the loss function is: In the above formula, denotes a loss function, denotes a likelihood function of summary generation, denotes a loss function of neural topic model, denotes a loss of bag-of-words reconstruction, and is a tuning factor, and has: ， ， ， ， ， In the above formula, This represents the probability that an input document D will generate a summary S. The bag-of-words representation of a document represents the probability of generating d, and |S| represents the number of words in the summary. Indicates finding a given... Mathematical expectation under the given conditions Represents a given latent variable , And when inputting document D, the first... Bag-of-words representation of summary statements The probability, This indicates that given a complete summary S and an input document D, latent variables are generated. The posterior distribution of the probability, Indicates the first Bag-of-words representation of summary statements, Indicates the first The semantic representation of the statements preceding the summary statement. Indicates a given And D generates the first The log-likelihood of the semantic representation of each statement. Indicates a given Latent variables When generating the document D, the first... The probability of the semantic representation of a statement. Representing latent variables The KL distance between the prior and posterior distributions. Representing latent variables The prior distribution of is given by the first The preceding statements and the document representation are jointly determined by D. Represents the distribution of a given topic The mathematical expectation, This represents the distribution of a given document to a topic in the topic model. Distribution of topics to words The log-likelihood of generating the bag-of-words representation of document d is given. This indicates the distribution of topics to words. Indicates the distribution of document topics. Indicates the distribution of document topics The KL distance between the prior and posterior distributions. This represents the distribution of document topics generated by the bag-of-words representation d of the document. The posterior probability distribution, Indicates the distribution of document topics The prior probability distribution, Represents the estimated document topic distribution From topic to word distribution Generate the nth word the probability of the nth word, denotes the nth word, denotes the estimated document topic distribution, denotes the sampled topic latent variable the KL distance between the prior and posterior distribution of the nth word, denotes the bag-of-words representation d of a given document, generating the topic latent variable the posterior distribution of the nth word, denotes the sampled topic latent variable, denotes the topic latent variable the prior distribution of the nth word.

8. A multi-document literature summarization system incorporating both explicit and implicit variational augmentation, comprising interconnected microprocessors and memory, characterized by, The microprocessor is programmed or configured to perform the multi-document literature summarization method with explicit and implicit variational augmentation according to any one of claims 1-7.

9. A computer-readable storage medium having stored therein a computer program, characterized in that, The computer program is used for programming or configuring the microprocessor to perform the multi-document literature summarization method with explicit and implicit variational augmentation according to any one of claims 1-7.

Images