Sequence-to-sequence text summarization generation method and system based on causality

Abstract

The application provides a sequence-to-sequence text abstract generation method and system based on causality, and belongs to the field of natural language processing and automatic text abstract generation. The method is inspired by the causality theory, and studies the causality of various elements in the abstract task from the perspective of data generation. The method first introduces two unobservable variables to obtain a structural causal model of the abstract task; then, a corresponding sequence-to-sequence generation framework is obtained according to the structural causal model, which is used to model the generation process of the original text and the abstract. The framework includes three core modules: a double latent variable variational encoder, an original text reconstruction decoder and an abstract prediction decoder. The method not only has stronger explainability than existing end-to-end deep text abstract methods, but also has better abstract performance and stronger generalization ability. The method is a sequence-to-sequence framework with strong applicability, and therefore can be migrated to more model subjects, generation tasks and different data sets.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of sequence-to-sequence text summarization in the natural language processing text summarization technology, and particularly relates to a sequence-to-sequence text summarization generation method and system based on causal relationship. BACKGROUND

[0002] Automatic text summarization technology aims to automatically identify important topics and key information from input documents and generate accurate, concise and fluent text as a summary. Early summarization techniques mainly revolve around manually designed heuristic rules and templates; nowadays, deep text summarization technology based on neural networks has become mainstream. It learns the matching pattern between the original text and the summary through supervised training in an end-to-end manner, which can be divided into extractive and generative summarization methods.

[0003] Deep extractive summarization methods extract a subset of sentences from the original text as summary text. It usually converts the summarization task into a sequence labeling or ranking task, and selects key sentences to form a summary by performing binary classification labeling on each candidate sentence in the document or ranking candidate sentences according to their importance. Deep generative summarization methods treat the summarization task as a sequence-to-sequence generation task, taking words or phrases as the basic generation unit, and generate summaries from scratch. Since the summary text is not limited to the expression of the original text, the summary generated by this method has higher flexibility and diversity, and is more suitable.

[0004] In recent years, pre-training models based on Transformer have further improved the performance of deep text summarization methods. The generative method inherits excellent language expression ability from the pre-training model. With its strong applicability, the generative summarization method based on pre-training models has gradually become a mainstream research topic.

[0005] The generative summarization method based on pre-training models has the following shortcomings.

[0006] (1) First, the data-driven method in the prior art lacks interpretability. This is because in the end-to-end training method, the user only needs to feed the model with samples and their labels, and the model automatically learns the matching pattern between the original text and the summary. This "black box model" is not understandable to the user.

[0007] (2) Then, the summary generated by the prior art often contains redundant information. These redundant information are not the core information in the original text. This is because the prior art tends to use all the relevant relationships in the data set when learning the matching pattern of the original text and the summary, which contains a large number of pseudo-relevant relationships, and inherits the statistical bias in the training corpus. At this time, the model is easily misled by the surface features that are easy to learn in the data set, such as high-frequency co-occurrence but not necessarily related text pairs. For example, "red maple leaves" as a high-frequency co-occurrence text pair, "red" and "maple leaves" have a high statistical correlation, but do not have a causal relationship, and "maple leaves" can also be "green". SUMMARY

[0008] The purpose of the present application is to solve the problem of excessive dependence on relevant relationships and easy to be misled by pseudo-relevant relationships in the prior art, and to propose a sequence-to-sequence framework inspired by causal theory.

[0009] The present application also proposes a sequence-to-sequence text summary generation method based on causal relationship, which comprises:

[0010] Step 1, input the original document into a neural network-based double latent variable variational encoder, which extracts summary-related features and summary-irrelevant features in the original document through multiple sampling;

[0011] Step 2, splice the summary-related features and the summary-irrelevant features to obtain summary comprehensive features, reconstruct the original document based on the summary comprehensive features to obtain a reconstructed document, and train the double latent variable variational encoder based on the reconstructed document and the training target to construct a loss function;

[0012] Step 3, use the trained double latent variable variational encoder to extract the summary-related features of the original document as target features, and based on the target features, obtain the text summary of the original document.

[0013] The sequence-to-sequence text summary generation method based on causal relationship, wherein step 1 comprises:

[0014] Obtain the encoding representation vector of the document through the document encoder, and the document encoder in the double latent variable variational encoder encodes the original document X to obtain the encoding representation vector h doc of the document;

[0015] The variational encoder in the double latent variable variational encoder module encodes and samples the encoding representation vector h doc of the document to obtain latent variables h c and h nc ;

[0016] h c and hnc inputting the original text reconstruction decoder to obtain the output o at each position i ; superimposing h c and h nc and o i , inputting the language model output layer to generate the reconstructed original document X'; calculating the reconstruction loss L according to X' and X R ; storing L R corresponding to the minimum hidden variable representation h c and h nc as the summary-related feature and the summary-independent feature, respectively.

[0017] The sequence-to-sequence text summary generation method based on causality, wherein step 3 comprises:

[0018] The summary prediction decoder calculates the output r at each position according to the target feature j ; superimposes the target feature and r j , inputs the language model output layer to obtain the word table distribution probability, and generates the corresponding words to form the text summary.

[0019] The sequence-to-sequence text summary generation method based on causality, wherein the training method of the triple of the double-hidden variable variational encoder, the summary prediction decoder, and the original text reconstruction decoder is:

[0020] Obtain the training document and the corresponding reference summary, input the training document into the double-hidden variable variational encoder based on neural network to obtain the hidden variable representation h c and h nc ; simultaneously obtain the Gaussian distribution of h c and h nc and

[0021] Use the hidden variable representation h c and h nc to generate the reconstructed original text; and use the hidden variable representation h c to generate the summary result;

[0022] Construct the original text reconstruction loss L R based on the training document and the reconstructed original text; construct the summary prediction loss L P based on the reference summary and the summary result; calculate the KL divergence based on the standard normal distribution and the Gaussian distribution

[0023] The final training loss function is L = L R + L P + λL KL ​​wherein lambda is used to adjust the degree of distribution normalization constraint, and based on the training loss function L, the dual latent variable variational encoder, the abstract prediction decoder and the original text reconstruction decoder are trained.

[0024] The application further provides a sequence-to-sequence text abstract generation system based on a causal relationship, comprising the following steps:

[0025] a sampling module configured to input an original document into a neural network-based dual latent variable variational encoder, and the dual latent variable variational encoder is configured to sample the original document multiple times to extract abstract-related features and abstract-irrelevant features from the original document;

[0026] a splicing module configured to splice the abstract-related features and the abstract-irrelevant features to obtain abstract comprehensive features, reconstruct the original document based on the abstract comprehensive features to obtain a reconstructed document, and construct a loss function based on the reconstructed document and a training target of the original document, and train the dual latent variable variational encoder;

[0027] an extraction module configured to extract the abstract-related features of the original document as target features by using the dual latent variable variational encoder after training, and obtain a text abstract of the original document based on the target features.

[0028] The sequence-to-sequence text abstract generation system based on a causal relationship, wherein the sampling module is configured to:

[0029] obtain a document encoding representation vector by using a document encoder, the document encoder in the dual latent variable variational encoder is configured to encode the original document X to obtain a document encoding representation vector h doc ;

[0030] the variational encoder in the dual latent variable variational encoder module is configured to encode and sample the document encoding representation vector h doc to obtain latent variables h c and h nc , respectively;

[0031] input h c and h nc into an original text reconstruction decoder to obtain an output o i at each position; superimpose h c , h nc and o i , and input a language model output layer to generate a reconstructed original document X'; calculate a reconstruction loss L R according to X' and X; store the minimum value of L R , and take the corresponding latent variables h c and h nc as the abstract-related features and the abstract-irrelevant features, respectively.

[0032] The aforementioned sequence-to-sequence text summarization system based on causal relationships, wherein the extraction module is used for:

[0033] The summary prediction decoder calculates the output r at each position based on the target feature. j ; Superimpose the target feature and r j The input is the output layer of the language model, which obtains the vocabulary distribution probability to generate the corresponding words to form the text summary.

[0034] The sequence-to-sequence text summarization system based on causal relationships, wherein the training method for the dual latent variable variational encoder, the summarization prediction decoder, and the original text reconstruction decoder is as follows:

[0035] Obtain the training documents and corresponding reference summaries, input the training documents into a neural network-based dual latent variable variational encoder, and obtain the latent variable representation h. c and h nc At the same time, h was obtained respectively. c and h nc Gaussian distribution and

[0036] h is represented using implicit variables. c and h nc Generate the reconstructed original text; and use implicit variables to represent h. c Generate summary results;

[0037] Based on the training documents and the reconstructed original text, construct the original text reconstruction loss L. R Based on the reference summary and the summary results, a summary prediction loss L is constructed. P Based on the standard normal distribution and the Gaussian distribution Calculate the KL divergence;

[0038] The final training loss function is L = L R +L P +λL KL λ is used to adjust the degree of distribution normalization constraint. The dual latent variable variational encoder, the summary prediction decoder, and the original text reconstruction decoder are trained based on the training loss function L.

[0039] The present invention also proposes a storage medium for storing a program that executes any of the causal-based sequence-to-sequence text summarization methods.

[0040] The present invention also proposes a client for any of the causal-based sequence-to-sequence text summarization systems.

[0041] From the above scheme, the advantages of the CI-Seq2Seq framework of the present application compared with the prior art are:

[0042] (1) Stronger interpretability. Compared with the prior art which learns the representation of the original text generally, the method proposed in the present patent can effectively distinguish and extract the hidden variable representations h c and h nc of SC and SNC in the original text. We verify the distribution of the learned hidden variable representations by t-SNE visualization analysis, as shown in Figure 1 From Figure 1 it can be seen that the distribution spaces of h c and h nc representations learned are obviously different: the distribution space of h c is more concentrated, while the distribution space of h nc is more dispersed. This proves that SC and SNC have learned the core information and the information of representation diversity, respectively.

[0043] (2) Better summary performance. Compared with the prior art, the method proposed in the present patent can effectively improve the quality of summary generation. We conduct experiments on the public summary datasets CNN / DM and XSUM, and compare CI-Seq2Seq with other baseline models. The compared baseline models include two Unified VAE-PGN and VHTM which are also based on variational autoencoder, three classic general pre-training models T5, BART and GLM, and CLIFF, Debiased-Ext and PtLAAM which are implemented using contrastive learning, adversarial learning and length controllable methods. The evaluation indexes are Rouge-1, Rouge-2 and Rouge-L, which measure the recall rates of words, bigrams and longest common subsequences, respectively. The comparison results are shown in Table 1. According to Table 1, it can be seen that not only compared with other models based on variational autoencoder, general powerful pre-training models, but also compared with models using other training techniques, the method proposed in the present patent can obtain obvious improvement in all indexes on the two datasets.

[0044] Table 1: Summary performance comparison on CNN / DM and XSUM datasets

[0045]

[0046] (3) stronger generalization ability. Compared with the prior art, the method proposed in the patent can effectively improve the generalization ability of the model. We use the model trained on CNN / DM on the XSUM test set, and use the model trained on XSUM on the CNN / DM test set, so as to test the performance of different models on the unobserved data set, and the experimental results are shown in Table 2. By comparing with the original framework BART, our method can obtain certain improvement, which proves that our model has stronger generalization ability.

[0047] Table 2: Comparison of generalization performance of cross-experiments on CNN / DM and XSUM data sets

[0048] BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 is a schematic diagram for t-SNE visualization analysis;

[0050] Figure 2 is a general flowchart for generating summary text using CI-Seq2Seq;

[0051] Figure 3 is a flowchart for obtaining initial latent variable representation and by n times sampling when generating summary text using CI-Seq2Seq;

[0052] Figure 4 is a flowchart for obtaining optimal latent variable representation and by k rounds of iteration update when generating summary text using CI-Seq2Seq;

[0053] Figure 5 is a flowchart for generating summaries using optimal latent variable representation when generating summary text using CI-Seq2Seq;

[0054] Figure 6 is a training process diagram of the method of the present application. DETAILED DESCRIPTION

[0055] When the inventors studied the text summarization task in combination with the causal theory, it was found that the defect of the prior art to generate redundant summaries was caused by excessive dependence on the correlation in the data set. The prior art excessively depends on the correlation of each component in the summary task, without considering the more essential causal relationship. The causal relationship describes the potential data generation process, and it is difficult for the prior art to depict the natural causal relationship behind the data only by relying on the observable variables.

[0056] The present application introduces two unobservable variables and uses a causal perception model to solve the problem of over-reliance on correlation and susceptibility to pseudo-correlation in the prior art. Specifically, the present application designs a structural causal model for the abstract task, which involves two unobservable variables in addition to the observable original text and abstract, representing the information that determines the content of the abstract (summary-causal factors, SC), such as the core information in the original text, and other information in the corpus that does not determine the content of the abstract (summary-non-causal factors, SNC), such as the marginal information that leads to the diversity of the original text. The fundamental difference between the two unobservable variables is that only the former SC is the cause of the abstract generation. By explicitly distinguishing between the two types of information, the present application designs a sequence-to-sequence framework inspired by causality (Causality Inspired Sequence-to-Sequence model, CI-Seq2Seq) to characterize the generation process of abstracts and texts, using only SC when generating abstracts and both when generating original texts. In terms of model structure, the present application improves the encoder-decoder architecture in the traditional sequence-to-sequence framework as follows: single encoder, single decoder is improved to double hidden variable variational encoder, original text reconstruction decoder and abstract prediction decoder. Specifically, the present application includes the following key technical points:

[0057] Key point 1: Design a structural causal model for text summarization from the perspective of data generation process; technical effect: the text summarization method designed according to the structural causal model can explicitly distinguish between the two types of information, learn the representation of SC and SNC, and have stronger interpretability;

[0058] Key point 2: Causality Inspired Sequence-to-Sequence framework (CI-Seq2Seq); technical effect: this framework can effectively distinguish between SC and SNC and obtain their respective representations, where the representation of SC is used to generate abstracts, and SC and SNC are used to generate original texts; this framework is implemented based on supervised variational autoencoder, which can effectively improve the quality of generated abstracts. The core modules include: double hidden variable variational encoder: using neural networks to encode the original text into hidden variable representations of SC and SNC; original text reconstruction decoder: using neural networks to decode the hidden variable representations of SC and SNC for reconstructing the original text; abstract prediction decoder: using neural networks to decode the hidden variable representation of SC for predicting the abstract.

[0059] To make the above features and effects of the present application more explicit and easy to understand, the following embodiments are described in detail below, with reference to the accompanying drawings.

[0060] As Figure 2The input document X is input into the neural network-based dual latent variable variational encoder module.

[0061] Step S1: input the original document X, and input the input original document X into the neural network-based dual latent variable variational encoder module.

[0062] Step S2: obtain the initial latent variable representation h c0 and h nc0 by n times sampling. The specific process is shown in Figure 3 , including the following sub-steps:

[0063] Step S201: obtain the encoding representation vector h doc of the document by the document encoder. The document encoder in the dual latent variable variational encoder module encodes the input original document X to obtain the encoding representation vector h doc of the document.

[0064] Step S202: obtain the Gaussian distribution of the latent variables SC and SNC by the variational encoder. The variational encoder in the dual latent variable variational encoder module further encodes the encoding representation vector h doc of the document to obtain the Gaussian distribution (distribution parameters are μ c , μ nc and ) of the latent variables SC and SNC.

[0065] Step S203: obtain the latent variable representation h c and h nc by the latent variable sampler. The latent variable sampler samples according to the Gaussian distribution obtained in the previous step to obtain the representation h c and h nc of the latent variables SC and SNC, respectively. The sampling process is completed by the classical reparametrization technique. Specifically, first, the variables ε c and ε nc are sampled from the standard normal distribution, i.e. Then, the required latent variable representations h c and h nc are obtained according to the following formulas, respectively:

[0066] h c = μ c + σ c * ε c ,

[0067] h nc = μ nc + σ nc * ε nc .

[0068] Step S204: Represent the latent variable h c and h nc h obtained by splicing cnc Input the original text to the reconstruction decoder. Then, use the latent variable representation h obtained in the previous step... c and h nc By splicing, we get h. cnc , which serves as the initial input to the original text reconstruction decoder.

[0069] Step S205: Obtain o through the original text reconstruction decoder i The original text reconstruction decoder calculates the output o at each position based on the initial input. i The position refers to a word in the original word sequence, arranged from left to right.

[0070] Step S206: Superimpose h cnc and o i Input the language model to the output layer. Then, take the h obtained in step S204... cnc The o obtained by superimposing it on the previous step i The input to the language model output layer is used as the input. The language model output layer is a current technology, implemented based on linear transformation and beam search.

[0071] Step S207: Generate the reconstructed original document X′ through the language model output layer. The language model output layer maps the result obtained in the previous step onto the vocabulary through a linear transformation to obtain the vocabulary distribution probability, and generates the corresponding words to form the original document X′ according to the beam search method.

[0072] Step S208: Calculate the reconstruction loss L based on X′ and X. R The reconstructed original document X′ is compared with the input original document X, and the reconstruction loss L is calculated. R .

[0073] The purpose of generating the original text is to obtain the optimal SC representation h for the current original text. c SNC indicates h nc , where h c This will be used for abstract generation. Since the original text is given during both training and testing, this invention uses the original text as a supervision signal: after reconstructing the original text, the generated original text X′ is compared with the given original text X, and the optimal latent variable representation h is determined based on the reconstruction error. c and h nc , where h cThe generation of the summary. That is, the accurate SC representation can not only work together with the SNC to well reconstruct the original information, but also work alone to generate the summary. When an SC representation can well reconstruct the original text, it is also suitable for generating a summary.

[0074] Step S209: store the current minimum L R The corresponding latent variable representation h c And h nc . Compare the size of L R In this cycle and the current minimum L R , the minimum L R After this cycle is stored. The corresponding latent variable representation h c And h nc .

[0075] Step S210: get the initial latent variable representation h c0 And h nc0 . Repeat the above steps (S203-S209) n times, return the latent variable representation h R And h c That minimizes the reconstruction loss L nc As the initial latent variable representation h c0 And h nc0 .

[0076] Step S3: update the optimal latent variable representation h c And h nc Through k rounds of iteration. As shown in Figure 4 , the specific process includes the following sub-steps:

[0077] Step S301: initialize the latent variable representation h c0 And h nc0 h c And h nc . According to the initial latent variable representation h c0 And h nc0 Obtained in step S2, initialize the latent variable representation h c And h nc Of SC and SNC.

[0078] Step S302: splice the latent variable representation h c And h nc h cnc Input the original text reconstruction decoder. Splice the latent variable representation h c And h nc Obtained in the last step to get h cnc , as the initial input of the original text reconstruction decoder.

[0079] Step S303: Obtain o by the original text reconstruction decoder i . The original text reconstruction decoder calculates the output o i at each position according to the initial input.

[0080] Step S304: Superimpose h cnc and o i , and input the language model output layer. Superimpose h cnc obtained in step S302 on o i obtained in the previous step as input of the language model output layer.

[0081] Step S305: Generate the reconstructed original document X' by the language model output layer. The language model output layer maps the result obtained in the previous step to the word table to obtain the word table distribution probability and generates the corresponding word composition original document X'.

[0082] Step S306: Calculate the reconstruction loss L R according to X' and X. Compare the reconstructed original document X' with the input original document X to calculate the reconstruction loss L R .

[0083] Step S307: Optimize to obtain new hidden variable representations h c and h nc . Optimize the hidden variable representations h R and h c according to the reconstruction loss L nc and the Adam optimization algorithm. The specific optimization process is as follows: first, set the hidden variable representation as a learnable parameter, and then set a special Adam optimizer for it; for the reconstruction loss L R calculated in the previous step, perform gradient backpropagation through the backpropagation algorithm, and finally directly update the learnable hidden variable representations h c and h nc according to the Adam algorithm.

[0084] Step S308: Obtain optimal hidden variable representations h c and h nc . Repeat the above steps (S302-S307) k times, and return the optimal hidden variable representations h c and h nc through multiple optimizations.

[0085] In summary, the purposes of steps S2 and S3 are to obtain accurate latent variable representations, and the evaluation basis of both is reconstruction loss. S2 is a pre-step of S3 and is used for initialization of S3. Technically, it obtains the latent variable representation by multiple sampling. S3 is a subsequent optimization operation based on the optimal initialization vector obtained by S2. The specific optimization process is as follows: first, set the latent variable representation as a learnable parameter, and then set a special Adam optimizer for it; for the reconstruction loss L R calculated in the last step, perform gradient backpropagation through the backpropagation algorithm, and finally directly update the learnable latent variable representation h c and h nc according to the Adam algorithm. Good initialization is crucial for the optimization process, and the time-consuming of the optimization operation is longer. Therefore, in S2, the current best representation is selected by multiple sampling, which is more efficient.

[0086] Step S4: generate the summary Y' using the optimal latent variable representation h c . As shown in Figure 5 , the specific process includes the following sub-steps:

[0087] Step S401: input the optimal latent variable representation h c into the summary prediction decoder. The optimal latent variable representation h c obtained in step S3 is used as the initial input of the summary prediction decoder.

[0088] Step S402: obtain r j through the summary prediction decoder. The summary prediction decoder calculates the output r j at each position according to the initial input.

[0089] Step S403: superimpose h c and r j , and input into the language model output layer. Superimpose h c of step S401 on r j obtained in the last step as the input of the language model output layer.

[0090] Step S404: generate the predicted summary Y' through the language model output layer. The language model output layer maps the result obtained in the last step to the word table to obtain the word table distribution probability and generate the corresponding words to form the summary Y'.

[0091] The above is the use process of the present application, and the training process of the method of the present application is described as follows. The corresponding training flowchart is shown in Figure 6 .

[0092] Step S1: input the original document X and the reference summary Y. Input the original document X into the neural network-based double latent variable variational encoder module.

[0093] Step S2: Obtain the latent variable representation h by sampling c and h nc . Refer to S201-S203 of the usage process for specific steps. This step will obtain the Gaussian distribution of the latent variables SC and SNC and (distribution parameters μ c , μ nc are the mean, is the variance).

[0094] Step S3: Generate the original text X' using the latent variable representation h c and h nc . Refer to S204-S207 of the usage process for specific steps.

[0095] Step S4: Generate the summary Y' using the latent variable representation h c . Refer to S4 of the usage process for specific steps.

[0096] Step S5: Calculate the training loss and optimize the model.

[0097] The training loss consists of three parts: the original text reconstruction loss L R , the summary prediction loss L P , and the distribution constraint loss L KL .

[0098] The original text reconstruction loss L R is the cross-entropy calculated based on the input original document X and the reconstructed X', with the purpose of training the dual latent variable variational encoder and the original text reconstruction decoder.

[0099] The summary prediction loss L P is the cross-entropy calculated based on the input reference summary Y and the predicted Y', with the purpose of training the dual latent variable variational encoder and the summary prediction decoder.

[0100] The distribution constraint loss L KL is the KL divergence calculated based on the standard normal distribution (0, I) and the Gaussian distribution of the latent variables SC / SNC (obtained by step S2), with the purpose of training the dual latent variable variational encoder to make the predicted variable distribution more normalized.

[0101] The final training loss function is L = L R + L P + λL KL , where λ is used to adjust the degree of distribution normalization constraint.

[0102] According to the above training loss function, the model is trained using the Adam optimizer.

[0103] The following are system embodiments corresponding to the above method embodiments. This embodiment can be implemented in conjunction with the above embodiments. The relevant technical details mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.

[0104] This invention also proposes a sequence-to-sequence text summarization system based on causal relationships, which includes:

[0105] The sampling module is used to input the original document into the neural network-based dual latent variable variational encoder, which performs multiple samplings on the original document to extract summary-related features and summary-irrelevant features from the original document;

[0106] The concatenation module is used to concatenate the summary-related features and summary-irrelevant features to obtain the summary comprehensive features. Based on the summary comprehensive features, the original document is reconstructed to obtain the reconstructed document. The original document is used as the training target. Based on the reconstructed document and the training target, a loss function is constructed to train the dual latent variable variational encoder.

[0107] The extraction module is used to extract the summary-related features of the original document as target features using the trained dual latent variable variational encoder, and to obtain the text summary of the original document based on the target features.

[0108] The aforementioned sequence-to-sequence text summarization system based on causal relationships, wherein the sampling module is used for:

[0109] The document encoder obtains the encoded representation vector of the document. The document encoder in this dual latent variable variational encoder encodes the original document X to obtain the encoded representation vector h of the document. doc ;

[0110] The variational encoder in the dual implicit variable variational encoder module encodes the document's representation vector h. doc Encoding and sampling are performed to obtain the latent variable h. c and h nc ;

[0111] h c and h nc Input the original text to the reconstruction decoder and obtain the output o at each position. i ; superimposed h c and h nc and o i The language model output layer generates the reconstructed original document X′; the reconstruction loss L is calculated based on X′ and X. R Storage L R The hidden variable h corresponding to the minimum valuec and h nc respectively as the summary-related features and the summary-unrelated features.

[0112] The sequence-to-sequence text summarization system based on causality, wherein the extraction module is configured to:

[0113] The summary prediction decoder calculates an output r j for each position according to the target features; j superimposes the target features and r c inputs a language model output layer to obtain a word distribution probability, so as to generate a corresponding word to form the text summary.

[0114] The sequence-to-sequence text summarization system based on causality, wherein the training method of the triple of the double-latent variable variational encoder, the summary prediction decoder and the original text reconstruction decoder is:

[0115] Obtain training documents and corresponding reference summaries, input the training documents into the double-latent variable variational encoder based on a neural network to obtain latent variable representations h nc and h c ; simultaneously obtain Gaussian distributions nc and and

[0116] Use the latent variable representations h c and L nc to generate a reconstructed original text; and use the latent variable representations h c to generate a summary result.

[0117] Construct an original text reconstruction loss L R based on the training documents and the reconstructed original text; construct a summary prediction loss L P based on the reference summaries and the summary result; calculate a KL divergence based on a standard normal distribution and the Gaussian distribution .

[0118] The final training loss function is L=L R +L P +λL KL , wherein λ is used to adjust the degree of distribution normalization constraint, and the double-latent variable variational encoder, the summary prediction decoder and the original text reconstruction decoder are trained based on the training loss function L.

[0119] The application further provides a storage medium for storing a program for executing any one of the sequence-to-sequence text summarization methods based on causality.

[0120] The application further provides a client for the any one of the causality-based sequence-to-sequence text abstract generation systems.

Claims

1. A cause-effect based sequence-to-sequence text summarization method, characterized in that, Comprise: Step 1, input the original document into a neural network-based double latent variable variational encoder, the double latent variable variational encoder extracts the summary-related features and the summary-independent features in the original document through multiple sampling, comprising: obtaining an encoded representation vector of the document by a document encoder, the document encoder in the dual latent variable variational encoder encoding the original document to obtain an encoded representation vector of the document ; a variational encoder in a dual-latent variational encoder module encodes a document into a latent representation vector performing encoding and sampling to obtain latent variable representations and ; will be described in detail below. and input the original text reconstruction decoder to obtain the output at each position ; superimposed and and , input the language model output layer to generate the reconstructed original document ; calculate the reconstruction loss according to and ; store the hidden variable representation corresponding to the minimum time and as the summary-related features and the summary-independent features, respectively;​ Step 2, splice the summary-related features and the summary-independent features to obtain the summary comprehensive features, reconstruct the original document based on the summary comprehensive features to obtain the reconstructed document, and take the original document as the training target, construct the loss function based on the reconstructed document and the training target, and train the double latent variable variational encoder; Step 3, use the trained double latent variable variational encoder to extract the summary-related features of the original document as the target features, and based on the target features, obtain the text summary of the original document, comprising: The abstract prediction decoder calculates the output at each position according to the target feature ; superimposes the target feature and , inputting the language model output layer to obtain the vocabulary distribution probability, so as to generate the corresponding words to constitute the text abstract.

2. The cause-effect based sequence-to-sequence text summarization generation method of claim 1, wherein, The training method of the double latent variable variational encoder, the summary prediction decoder and the original text reconstruction decoder is: Obtaining training documents and corresponding reference abstracts, inputting the training documents into a neural network-based double latent variable variational encoder to obtain latent variable representations and ; while obtaining Gaussian distributions of and and ;​ Using latent variable representation Generating the reconstructed original text; and using latent variable representation generating summary results; wherein and are the means of the latent variable representations and and are the variances of the latent variable representations and ​​ Building a reconstructed text loss based on training documents and reconstructed original texts ; constructing an abstract prediction loss based on the reference abstract and the abstract result ; based on a standard normal distribution and the Gaussian distribution / calculating a KL divergence; The final training loss function is where is used to adjust the degree of distributional regularization constraints, based on which the training loss function The dual latent variable variational encoder, the summary prediction decoder, and the original reconstruction decoder are trained.

3. A cause-effect based sequence-to-sequence text summarization system, characterized in that, Comprise: The sampling module is used for inputting the original document into the neural network-based double latent variable variational encoder, and the double latent variable variational encoder extracts the summary-related features and the summary-independent features in the original document through multiple sampling, comprising: obtaining an encoded representation vector of the document by a document encoder, the document encoder in the dual latent variable variational encoder encoding the original document to obtain an encoded representation vector of the document ; a variational encoder in a dual-latent variational encoder module encodes a document into a latent representation vector performing encoding and sampling to obtain latent variable representations and ; will be described in detail below. and input the original text reconstruction decoder to obtain the output at each position ; superimposed and and , input the language model output layer to generate the reconstructed original document ; calculate the reconstruction loss according to and ; store the hidden variable representation corresponding to the minimum time and as the summary-related features and the summary-independent features, respectively;​ The splicing module is used for splicing the summary-related features and the summary-independent features to obtain the summary comprehensive features, reconstructing the original document based on the summary comprehensive features to obtain the reconstructed document, and taking the original document as the training target, constructing the loss function based on the reconstructed document and the training target, and training the double latent variable variational encoder; The extraction module is used for extracting the summary-related features of the original document as the target features by using the trained double latent variable variational encoder, and based on the target features, obtaining the text summary of the original document, comprising: The abstract prediction decoder calculates the output at each position according to the target feature ; superimposes the target feature and , input language model output layer, to obtain the vocabulary distribution probability, to generate the corresponding words to form the text abstract.

4. The cause-effect based sequence-to-sequence text summarization generation system of claim 3, wherein, The training method of the double latent variable variational encoder, the summary prediction decoder and the original text reconstruction decoder is: Obtaining training documents and corresponding reference abstracts, inputting the training documents into a neural network-based double latent variable variational encoder to obtain latent variable representations and ; while obtaining Gaussian distributions of and and ;​ Using latent variable representation Generating the reconstructed original text; and using latent variable representation generating summary results; wherein and are the means of the latent variable representations and and are the variances of the latent variable representations and ​​ Constructing a reconstruction loss of the original text based on the training document and the reconstructed original text ; constructing an abstract prediction loss based on the reference abstract and the abstract result ; based on a standard normal distribution and the Gaussian distribution / calculating the KL divergence; The final training loss function is where is used to adjust the degree of distributional regularization constraints, based on which the training loss function is The dual latent variable variational encoder, the summary prediction decoder, and the original reconstruction decoder are trained.

5. A storage medium for storing a program for executing the causal relationship-based sequence-to-sequence text summary generation method according to claim 1 or 2.

Images