Systems and methods for controllable text summarization - Patents.com

Abstract

The embodiments described herein provide a flexible and controllable summarization system that allows users to control the generation of summaries without manually editing or writing the summaries, e.g., without the user actually adding or removing specific information at various levels of granularity. Specifically, the summarization system performs controllable summarization through keyword manipulation. A neural network model is trained to generate summaries conditioned on both keywords and source documents, so that, at test time, users can interact with the neural network model through a keyword interface, potentially enabling multi-factor control.

Description

[Technical field]

[0001] [Cross reference] This disclosure claims priority to U.S. Patent Application No. 17 / 125,468, filed December 17, 2020, and U.S. Provisional Application No. 63 / 071,571, filed August 28, 2020, which are expressly incorporated by reference in their entireties herein.

[0002] [Technical field] The present disclosure relates generally to machine learning models and neural networks, and more specifically to a controllable text summarization framework. [Background technology]

[0003] Text summarization compresses a document into a short paragraph or sentence as a "summary" of the document, which is expected to retain key information from the document. Some existing summarization systems create summaries by extracting important sentences from the document, while others create summaries from scratch by combining sentences selected by themselves. These summarization systems generate summaries based only on the input document, which often results in one version of the summary for the input document. This universal version of the summary may sometimes fail to capture the different interests of users requesting summaries. [Brief description of the drawings]

[0004] [Figure 1A] FIG. 2 shows a block diagram illustrating an example neural network model diagram of a summarization system according to embodiments described herein. [Figure 1B] FIG. 2 shows a block diagram illustrating a control workflow for a summarization system during testing according to an embodiment described herein. [Diagram 2] FIG. 2 shows an exemplary diagram illustrating the inference stage of a keyword-based model of controllable summarization according to embodiments described herein. [Diagram 3]1 is a simplified diagram of a computing device for implementing a summarization system according to some embodiments. [Figure 4] 3 is a simplified logic flow diagram illustrating a method for training the keyword-based summarization model shown in FIG. 2 according to certain embodiments described herein. [Diagram 5] FIG. 3 is a simplified logic flow diagram illustrating a method for generating a controlled summary using the keyword-based model shown in FIG. 2 during the inference stage, according to some embodiments described herein. [Figure 6] 13 provides qualitative examples illustrating resultant summaries generated based on different user-controlled configuration parameters, according to one embodiment described herein. [Figure 7] 1 provides an example data chart illustrating example test performance of a keyword-based model described herein compared to existing models, according to some embodiments described herein. [Figure 8] 1 provides an example data chart illustrating example test performance of a keyword-based model described herein compared to existing models, according to some embodiments described herein. [Figure 9] 1 provides an example data chart illustrating example test performance of a keyword-based model described herein compared to existing models, according to some embodiments described herein. [Figure 10] 1 provides an example data chart illustrating example test performance of a keyword-based model described herein compared to existing models, according to some embodiments described herein. [Figure 11] 1 provides an example data chart illustrating example test performance of a keyword-based model described herein compared to existing models, according to some embodiments described herein. [Figure 12]1 provides an example data chart illustrating example test performance of a keyword-based model described herein compared to existing models, according to some embodiments described herein. [Figure 13] 1 provides an example data chart illustrating example test performance of a keyword-based model described herein compared to existing models, according to some embodiments described herein. [Figure 14] 1 provides an example data chart illustrating example test performance of a keyword-based model described herein compared to existing models, according to some embodiments described herein.

[0005] In the figures and appendices, elements having the same name have the same or similar functions. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006] Existing summarization systems often generate summaries based only on the input document, which often results in one version of the summary for the input document. This universal version of the summary may sometimes fail to capture the different interests of users requesting summaries. For example, if a document contains news articles about sports news, users may want summaries that focus on specific players, or summaries of different lengths considering the user's interests or available time. User preferences for different versions of the summary may also be extended to other control factors, such as topics or specific sections (when summarizing academic papers or books).

[0007] In view of the need to generate customized summaries of documents that reflect user preferences, embodiments described herein provide a flexible and controllable summarization system that allows users to control the generation of summaries without manually editing or writing the summaries, e.g., without the user actually having to add or remove specific information under various levels of granularity. Specifically, the summarization system performs controllable summarization through keyword manipulation. A neural network model is trained to generate summaries that are conditioned on both keywords and source documents, so that, at test time, users can interact with the neural network model through a keyword interface, potentially enabling multi-factor control.

[0008] For example, a controllable summarization system allows users to control and manipulate summaries from a model. Through a user interface, a user may input control tokens in the form of a set of keywords or descriptive prompts, which can be used to generate customized summaries that reflect the user's preferences for source articles. During training, the model learns to predict summaries conditional on both the source documents and the keywords that serve as external guidance. During inference, the keywords, which are target prefixes to constrain decoding, and optional prompts (e.g., entered by the user) are combined as control tokens to convey the user's preferences in summary generation.

[0009] In one embodiment, the keywords and user prompts can be complementary. For example, the user may enter or select entity names as keywords, or change the number of keywords to control the entities and length, respectively. The model can be trained using only keywords as additional inputs that can be identified from training summaries. This process requires neither extra human annotations nor predefined control aspects for training, but is quite flexible to achieve a wide range of text manipulations. In contrast, most existing summarization systems either do not allow user input to control the summarization process or require predefined "control codes" (see Fan et al., Controllable abstractive summarization, in Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, 2018; Liu et al., Controlling length in abstractive summarization using a convolutional neural network, in Proceedings of EMNLP, 2018; Keskar et al., Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858, 2019, all of which are expressly incorporated by reference in their entireties), which requires the system to collect annotations for training and cannot generalize to unknown control aspects, such as different types of articles or different types of control commands, at test time.

[0010] As used herein, the term "prompt" is used to refer to a predefined text sequence that is used as a target prefix to constrain the decoding of a summarization system. For example, the prompt "the main contributions of this paper are:(1)" may be used to constrain the decoding for summarizing the list of contributions of a journal paper.

[0011] As used herein, the term "network" may include any hardware or software-based framework, including any artificial intelligence network or system, neural network or system, and / or any training or learning model implemented thereon or in conjunction therewith.

[0012] As used herein, the term "module" may comprise a hardware or software-based framework that performs one or more functions. In some embodiments, a module may be implemented on one or more neural networks.

[0013] Controllable Summarization Overview FIG. 1A shows a block diagram illustrating an exemplary neural network model diagram 100a of a summarization system according to an embodiment described herein. Traditionally, unconstrained neural summarization methods are trained to learn a conditional distribution p(y|x), where x and y represent source documents, such as articles and summaries, respectively. The generated summaries depend only on document x, without human intervention. To control the output summaries, additional control tokens, such as keywords z, may be used to represent user preferences. Thus, diagram 100a shows that in the training phase, a neural network model for controllable summarization is trained to model a probability p(y|x,z), where x represents the source document article 110, y represents the generated summary 130, and z represents the keywords 120 extracted from the article 110. Thus, the probability p(y|x,z) represents a summary distribution conditional on both the source document 110 and the keywords 120.

[0014] FIG. 1B shows a block diagram 100b illustrating a control workflow for a summarization system at test time according to an embodiment described herein. FIG. 100b illustrates the use of a keyword manipulation mechanism to bridge a user and a keyword interface called a "control center" 140 during the inference phase. Specifically, keywords 120a can be automatically extracted from source documents, e.g., articles 110, and (optionally) presented to a user 150 in a user interface of the control center 140. The user 150 can then interact with the auto-keywords 120a through the control center 140 to insert control tokens z.

[0015] In one embodiment, the control tokens z may include keywords as additional inputs during training and inference. The control tokens may optionally include prompts during testing to further constrain the decoding process. The control tokens z (in the form of keywords, prompts, or a combination of both) may act as an interface between the user and other black-box neural models, providing a flexible way for the user to explicitly control the automatic summarization.

[0016] For example, user 150 may configure a target length (word limit) for the summary and prompt control center 140 to remove some auto-keywords to generate a shorter summary. Alternatively, control center 140 may choose to retain only certain entity-related keywords if user 150 indicates interest in a particular entity name. In addition, user 150 may also edit customized keywords, allowing for more flexible customized summarization without the user having to directly manually edit the summary.

[0017] 2 shows an example diagram illustrating the inference stage of the controllable summarization keyword-based model 230 according to an embodiment described herein. FIG. 2 shows an NBA basketball news article 210 with a reference summary 215 listing the results of several games. However, fans of certain star players of these basketball teams, such as LeBron James or Stephen Curry, may only be interested in the games they played and may also want to know their scores. Thus, a user 150 may provide such user preferences through control tokens 232 that may control the summary generation in the keyword-based model 230.

[0018] Specifically, during training, the keyword-based model may learn to predict summaries conditional on both the source documents and the keywords that serve as external guidance. For example, ground-truth summaries may be used to identify keywords in the source documents. In this example, reference summaries 215 may be used for training. In another example, ground-truth summaries customized to a user's preferences for the names of players of interest to the user, such as "Dwyane Wade," "James," or "Stephen Curry," may be used for training along with the source documents 210.

[0019] During inference, keywords, which are target prefixes for constraining decoding, and optional prompts are combined as control tokens 232 to convey user preferences. Specifically, the keywords provide a generic interface for controlling multiple aspects of the summary, allowing the user to optionally rely on automatically extracted keywords, user-provided keywords, or a combination of both. This approach provides a clear separation between user control at test time and the training process, including pre-training. As a result, the keyword-based model 230 can adapt to new use cases without changing model parameters. This is true, for example, even if the keyword-based model 230 is not trained to specifically focus on controlling entities or lengths during training.

[0020] For example, keywords 225 may be input to the keyword-based model 230 during training and testing, and prompts 227 are optionally used at testing time. The dashed lines represent optional paths by which control tokens 232 may come from the source article 210, the user 150, or both. The keyword-based model 230 may then generate different versions of summaries 235a-c in response to different keywords 225 or prompts 227 during inference time.

[0021] Computer environment 3 is a simplified diagram of a computing device for implementing a summarization system, according to some embodiments. As shown in FIG. 3, the computing device 300 includes a processor 310 coupled to a memory 320. The operation of the computing device 300 is controlled by the processor 310. Also, while the computing device 300 is shown with only one processor 310, it is understood that the processor 310 may represent one or more central processing units, multi-core processors, microprocessors, microcontrollers, digital signal processors, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), graphics processing units (GPUs), and the like within the computing device 300. The computing device 300 may be implemented as a standalone subsystem, as a board added to a computing device, and / or as a virtual machine.

[0022] The memory 320 may be used to store software executed by the computing device 300 and / or one or more data structures used during operation of the computing device 300. The memory 320 may include one or more types of machine-readable media. Some common forms of machine-readable media may include a floppy disk, a flexible disk, a hard disk, a magnetic tape, any other magnetic medium, a CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with a pattern of holes, a RAM, a PROM, an EPROM, a FLASH-EPROM, any other memory chip or cartridge, and / or any other medium adapted to be read by a processor or a computer.

[0023] The processor 310 and / or memory 320 may be arranged in any suitable physical arrangement. In some embodiments, the processor 310 and / or memory 320 may be implemented on the same board, in the same package (e.g., system-in-package), on the same chip (e.g., system-on-chip), etc. In some embodiments, the processor 310 and / or memory 320 may comprise distributed, virtualized, and / or containerized computing resources. Consistent with such embodiments, the processor 310 and / or memory 320 may be located in one or more data centers and / or cloud computing facilities.

[0024] In some examples, the memory 320 may include a non-transitory tangible machine-readable medium including executable code that, when executed by one or more processors (e.g., processor 310), may cause the one or more processors to perform methods described in further detail herein. For example, as shown, the memory 320 includes instructions for a control summarization module 330 that may be used to implement and / or emulate the systems and models and / or to implement any of the methods described further herein. In some examples, the controllable summarization module 330 may receive an input 340, e.g., a source document. The data interface 315 may be either a user interface that receives input entered by a user or a communication interface that may receive or retrieve documents from a database. The controllable summarization module 330 may generate an output 350, e.g., a summary.

[0025] In some embodiments, the controllable summarization module 330 includes a keyword-based module 331 and a control center 332. For example, the keyword-based model 331 may be similar to the model 230 of FIG. 2, which may employ different keyword extraction mechanisms during training and testing. In one implementation, the control center 332 may communicate with the data interface 315 and act as a filter to select customized keywords from the auto-keywords based on user preferences obtained from the data interface 315.

[0026] In some examples, the controllable summarization module 330 and sub-modules 331-232 may be implemented using hardware, software, and / or a combination of hardware and software.

[0027] Controllable Summarization Workflow 4 is a simplified logic flow diagram illustrating a method for training the keyword-based summarization model shown in FIG. 2 according to some embodiments described herein. One or more of the processes 402-412 of the method 400 may be implemented, at least in part, in the form of executable code stored on a non-transitory, tangible, machine-readable medium that, when executed by one or more processors, may cause the one or more processors to perform one or more of the processes 402-412. In some embodiments, the method 400 may correspond to a method used by the module 330.

[0028] In step 402, input documents (eg, 210) and ground truth summaries (eg, 215) from a training dataset may be received, for example, via a data interface 315.

[0029] In step 404, sentences that maximize the ROUGE score with the reference summary 215 may be greedily selected from the document 210. The ROUGE score may be defined in (Lin, 2004), which is expressly incorporated herein by reference in its entirety. In this step, the keywords are constrained to those contained in the important sentences.

[0030] In step 406, all longest subsequences in the extracted sentence that matched a subsequence in the ground truth summary are identified. This matching step may be similar to the duplicate word recognition method described in Gehrmann et al., Bottom-up abstractive summarization, in Proceedings of EMNLP, 2018, which is expressly incorporated herein by reference in its entirety.

[0031] In step 408, redundant words and stop words are removed from the sentence, and the remaining tokens are retained as keywords. Thus, compared to other existing keyword extraction methods that output only a few salient words, keyword extraction retains most of the content words contained in the summary. This encourages reliance on a given keyword by building a reliable correlation between the presence of the given keyword in the input (e.g., source article 210) and the presence of the given keyword in the target (e.g., ground truth summary). And it ensures that user-provided keywords are not ignored by the model during testing.

[0032] Then, in step 410, the generated keyword sequence is separated with a special token, prepended to the source document, and fed to the summarization model. In one embodiment, the keyword sequence maintains the order of the keywords as they were in the source document. In another embodiment, the keyword sequence may employ a different order of keywords, as this ordering may frequently differ between the source document and the target summary. Keywords may also be separated from different source sentences using a special token ("|"). In applications where sentence boundaries are unknown, for example when a user suggests their own keywords, the "|" token can be ignored.

[0033] In step 412, the keyword-based model generates a probability distribution p(y|x,z) of summaries conditional on the input document x and keyword z. The summarization model is then trained to maximize p(y|x,z) in an end-to-end manner. For example, the conditional probability distribution p(y|x,z) of summaries generated from the summarization model is compared to the ground truth summaries to calculate a cross-entropy loss, which can be used to update the summarization model via backpropagation.

[0034] In one embodiment, the keyword extraction strategy described in steps 404-408 may retain most words from the summary that are included in the source document. Without regularization, the dependency on such keywords may be strong enough that the keyword-based summarization model 230 may rarely generate new words in the summary. To remedy this, keywords may be randomly dropped during training such that the keyword-based summarization model 230 may learn to rely on keywords that are present in the keyword sequences that are part of the model's input while still retaining important information from the source document that is not present in the keyword sequence. Note that keyword dropout may only be applied at training time.

[0035] 5 is a simplified logic flow diagram illustrating a method for generating a controlled summary using the keyword-based model shown in FIG. 2 during an inference stage according to some embodiments described herein. One or more of the processes 502-514 of the method 500 may be implemented, at least in part, in the form of executable code stored on a non-transitory, tangible, machine-readable medium that, when executed by one or more processors, may cause the one or more processors to perform one or more of the processes 502-514. In some embodiments, the method 500 may correspond to a method used by the module 330.

[0036] An input document (e.g., document 210) may be received in step 502. For example, an input summary may be received at data interface 315 of FIG.

[0037] In step 504, a set of keywords may be extracted from the input documents, for example, by sequence labeling the keywords. For example, keyword extraction during inference may be formulated as a sequence labeling task. Specifically, a BERT-based sequence tagger (e.g., 220) may have been optionally trained on keywords and documents from a training dataset. The BERT-based sequence tagger 220 may be similar to the BERT model described in Devlin et al., BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018, which is expressly incorporated herein by reference in its entirety. The tagger then calculates a selection probability q for each token in the test document. j Similar to keyword extraction during training (described in steps 404-408 of FIG. 4), we can calculate the n s A set of sentences is selected from the input source document. Among these sentences, qj > ε words can be used as keywords up to m max The three hyperparameters n s , ε,m max can be selected based on uncontrolled summarization performance on the validation dataset. The results are reasonably robust to different settings.

[0038] In step 506, user input of one or more control parameters and / or control token sequences related to characteristics of the summary to be generated may be received, for example, via the control center 232 of Figure 2, to modify the set of keywords into a customized set of keywords. During inference time, the user 150 may provide different configurations of control tokens 232 reflecting keyword control over the length and entities of the summary.

[0039] In step 508, the set of keywords is modified based on the received control token sequence.

[0040] In step 510, a summary of the input document may be generated based on a customized set of keywords according to one or more control parameters. For example, entity control may generate summaries that focus on entities of interest. The example summaries 235a-c provide different versions of the summary that focus on different players if their names are included as keywords that directly influence the respective summaries.

[0041] As another example, users may have different preferences regarding the length of the summary, which may be controlled by a user-specified length parameter. Specifically, the training data may be separated into five buckets represented by different summary lengths, such that each bucket has the same number of examples. Then, the average number of keywords K for each bucket on the training data may be calculated. l At test time, the user 150 selects the K sequences with the highest selection probabilities calculated by the sequence tagger 220. lWe can specify a length parameter l∈{0,1,2,3,4} to include keywords.

[0042] In one embodiment, prompts (e.g., 227 in FIG. 2) are utilized to perform multipurpose text generation with a single unified model. Specifically, prompts may take the form of controlled token sequences, where such tokens may be used as both target prefixes and keywords. For example, prompts can be used to summarize the contributions of scientific papers. Existing datasets on academic papers, such as arXiv, collect abstracts of papers as summaries, which often contain extra background context and lack detailed contribution descriptions of the associated papers. In many cases, readers would benefit from an explicit list of contributions to understand the novelty and value of the paper. In these cases, a controlled token prompt, such as “the main contributions of this paper are: (1)”, may trigger the generation of a contribution-focused summary. As another example, prompts may be used to summarize the purpose of an invention in a patent document. Abstracts of patent articles in existing datasets, such as BIG-PATENT, are often too complex and often cover even core methodological details. However, for non-expert readers, it would be preferable to provide a one-sentence summary stating the purpose of the invention, ignoring technical details. The prompt "the purpose of the present invention is" may be used to trigger a concise summary of the input patent document focusing on the purpose of the patent. As another example, question-guided summarization may be used to enhance the user's reading comprehension. Human summarization may be constrained by questions that require the answer to be included in the summary. This shows the important relationship between summarization and reading comprehension. The summarization model, if properly guided, may directly answer some questions about the article. This suggests the possibility of subsummarization of reading comprehension as a form of summarization. To test this hypothesis, the control token prompt "Q: question [text]? A:" triggers the reading comprehension behavior in generating a summary.

[0043] Working Example 6 provides qualitative examples illustrating resultant summaries generated based on different user-controlled configuration parameters according to one embodiment described herein. Example 601 shows a source document summarized into different versions of the summary based on different control aspects such as entity name (e.g., "ISIS", "Hasd Al-Shaabi" as keywords), length, question and answer format, and / or the like. Example 602 shows a reference summary re-summarized by a prompt with a contribution format, e.g., "the main contribution of this paper are ...". Example 603 shows a reference summary re-summarized by a prompt for the purpose of the invention.

[0044] Additional examples showing the performance of the keyword-based model can be performed on the following separate domain summarization datasets: CNN / Dailymail (CNNDM) news articles, arXiv academic papers (described in Cohan et al., A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of NAACL (Short Papers), 2018), and BIGPATENT patent articles. For all datasets, the source documents are truncated to 1024 tokens, and then the target summarization is truncated to 256 tokens. The conditional distribution p(y|x,z) in the keyword-based model is a fine-tuned version of the pre-trained BARTLARGE model, which achieves comparable performance on several summarization benchmarks. The automatic keyword tagger at test time is based on the pre-trained BERTLARGE model, fine-tuned as described in connection with Figure 2. The summarization model implementation is based on the fairseq toolkit, and the automatic keyword extraction model is based on the HuggingFace Transformers library.

[0045] For evaluation, the ROUGE score and the recently proposed BERTScore (see Zhang et al., BERTScore: Evaluating text generation with BERT, in Proceedings of ICLR, 2020) are used when ground truth is available. For control-related evaluation where no reference summary is available, we (1) collect ground truth summaries if possible, (2) inspect the summaries for respecting the control signals, or (3) resort to human evaluation.

[0046] To test the performance of entity control, user preferences are first simulated by providing the model with oracle entities extracted from ground truth targets, and then compared to a model using auto-keywords in an uncontrolled setting to show the effect of oracle entities. To examine whether the decoded summaries respect the entity changes, 100 documents are sampled and all entities in the documents are iteratively retrieved to generate summaries. The success rate, the percentage of requested entities that actually appear in the output summaries, is then calculated. Results are reported distinguishing whether the entities are from the first three sentences or from the whole article. To test whether summaries from different entity inputs effectively match the documents, another 100 documents are sampled and for each one, a summary is generated by randomly sampling one "important" entity that appears in the references and one "unimportant" entity that does not appear in either the references or the first three source sentences. For each (article, abstract) pair, three annotators from Amazon Mechanical Turk are recruited to make a binary decision on whether the abstract can be associated with the article. The majority vote is then taken as the result, reporting the percentage of factually correct summaries. Since many examples in arXiv and BIGPATENT do not have identifiable entities, the evaluation is performed only on CNNDM.

[0047] Figure 7 shows an exemplary performance of a keyword-based model (called CTRLsum) compared to BART. It is observed that the use of oracle entities helps boost the ROUGE-2 score by 3.6 points compared to using auto-keywords, implying that CTRLsum can take advantage of the given entities. Figure 8 shows the success rate and factual correctness evaluation. Notably, the keyword-based model described herein (CTRLsum) achieves a high success rate reaching about 95% for both lead-3 and full-article entities. However, other systems struggle to include given entities, especially entities that do not appear at the beginning of the article. Factual correctness scores from human annotators suggest that CTRLsum can generate virtually consistent summaries regardless of whether the entities of interest are important or not, and are comparable to the unconstrained BART baseline.

[0048] Similar to entity control, we first investigate the effect of oracle length signals from the reference to simulate user preferences. In addition to ROUGE and BERTScore, we then measure the length distance between the decoded summary and the reference (Liu et al., 2018). Specifically, we measure the actual length bucket code l of the decoded summary. sys The mean absolute deviation (MAD) of the ground truth control code ref It is calculated from the following formula:

number

[0049] In Fig. 7, CTRLsum with oracle length signal shows relatively small gain over the auto-CTRLsum baseline. This means that oracle length conveys only limited additional information to assist in generating reference summaries. The LengthCode baseline is performed based on BART, where ground truth length bucket codes are prepended to articles at both training and testing time. However, LengthCode cannot consistently improve over BART with oracle length signal. Moreover, the BART model fine-tuned with the LengthCode method almost ignores length signals with PCC close to 0, as shown in Fig. 9. This is not very surprising, since as the summarizer gets stronger, it can already learn a good length predictor implicitly, and length codes become less useful. In contrast, CTRLsum with length-guided keywords can achieve high positive PCC between the control signal and the actual output length, and reduce the length deviation MAD compared to the auto-baseline.

[0050] There is no existing dataset to evaluate the summarization of contributions in academic papers, which poses a challenge to the evaluation of this application. However, researchers often summarize the bullet contributions of papers in the introduction, which gives the idea of ​​extracting such contribution claims as reference summaries. Thus, the entire arXiv database2 is downloaded, which contains 67,000 papers, all of which were first submitted within the first six months of 20193. The introduction and bullet contributions are extracted using regular expressions to filter out the non-applicable ones. The contributions are used as references, and the introduction after removing the contribution claims is used as the source article to predict the contributions from the rest of the introduction. This procedure results in 1018 test examples. The model is trained and tested on arXiv.

[0051] For the purpose summarization setup, 1000 test examples are sampled from BIGPATENT to collect a test dataset featuring one-sentence invention purpose summaries, and their reference summaries are presented to human annotators from Amazon Mechanical Turk. For each example, one annotator is asked to select a sentence that conveys the invention purpose. An option is also provided for annotators who are unable to identify the invention purpose. After filtering out invalid examples, 763 examples are collected as test data.

[0052] Figure 11 shows the results of summarizing the contributions of academic papers and summarizing the invention objectives of patent applications. By using prompt texts as both decoder prefixes and keywords, CTRLsum outperforms the BART baseline in most cases. Furthermore, we report precision (P) and recall (R) scores in BERTScore in addition to F1. It is observed that the BART baseline tends to over-generate complete summaries with low precision scores, while CTRLsum is able to focus on keyword-related content.

[0053] Question-guided summarization is tested on reading comprehension benchmarks in a zero-shot setting. Specifically, the CNNDM summarization models are evaluated on the in-domain NewsQA and out-of-domain SQuAD1.1, respectively. Several NewsQA test articles are present in the CNNDM summarization training dataset. This is because it is still a reasonable unsupervised setting, as the keyword-based model never sees either the question or the answer during training. In addition to a comparison with the vanilla BART model, zero-shot performance from the GPT2 language model (without fine-tuning) is included as a reference point. The largest GPT2 model with 1.5 billion parameters is omitted, as it cannot be evaluated on a single GPU device due to memory limitations. F1 scores are reported for the two benchmarks.

[0054] BART is pre-trained with a denoising task to predict a denoised version of the source, and performs poorly for out-of-the-box zero-shot reading comprehension, as shown in Figure 10. However, interestingly, BART fine-tuned on the summarization task, without seeing question-answer pairs in the training data, can improve the F1 score by 24.4 and 25.9 points on NewsQA and SQuAD, respectively. Moreover, CTRLsum with question keywords can boost the performance by another 15.6 and 17.9 points, approaching the supervised MatchLSTM score on NewsQA. Such results suggest that summarization may be a suitable transfer task for abstract reading comprehension.

[0055] Figure 12 shows the uncontrolled summarization performance without user input using automatically extracted keywords as described with respect to Figure 2. On the CNNDM and arXiv datasets, CTRLsum significantly outperforms the strong BART and PEGASUS baselines, yielding new state-of-the-art performance for CNNDM. It also performs as well as the BART baseline on BIGPATENT in terms of BERTScore, although with a worse ROUGE-2 score. However, there is a large performance gap between the BART-based model and PEGASUS on BIGPATENT. The reasons for this could be the different dataset processing, the four suboptimal training schedules, or the inherent differences between BART and PEGASUS.

[0056] In the case of controlled summarization, we evaluate the "control" directly by informing the annotator of the intended control signal as a result of further human evaluation. Experiments are conducted for entity and objective control. Specifically, the annotator is informed of the intent (to obtain a summary focused on a specific entity or patent objective), and then the annotator provides a score on a scale of 1-5 across two dimensions: (1) Control Accuracy (CA): whether the summary contains key information that is accurate with respect to the intent, and (2) Control Relevance (CR): how relevant the summary is to the overall control intent - summaries containing redundant content that is not relevant to the intent are penalized. Results including significance tests are shown in Figure 13. Control accuracy for important entity control and objective control are comparable with no significant difference between BART and CTRLsum (p-value > 0.05), while CTRLsum shows significantly better control relevance overall by focusing on the desired information. Also, unconstrained BART fails to generate summaries relevant to non-important entities, thus scoring lower on both dimensions.

[0057] For uncontrolled summarization, human annotators from Amazon Mechanical Turk score summaries across four dimensions (scale 1–5): (1) Factual Consistency (FAC): the summary should only contain statements that the source document can contain, (2) Relevance (REL): the summary should only contain important information of the source document, (3) Fluency (FLU): each sentence in the summary should be fluent, and (4) Coherence (COH): the summary should be well-structured and well-organized. The results including significance tests are shown in Figure 14. The quality of summaries from all systems on all dimensions is generally good, with scores mostly higher than 4.0. However, most scores do not show significant differences from CTRLsum (automated keywords) with large p-values ​​(e.g., CTRLsum with oracle keywords) despite their very different similarity to the reference summaries in terms of ROUGE / BERTScore. This means that the quality of summaries from different systems with strong pre-trained models like BART becomes difficult to clearly distinguish by non-expert MTurkers. Also, note that non-expert human judgments of summarizations may be unreliable and poorly correlated with expert judgments.

[0058] Some examples of computing devices, such as computing device 200, may include a non-transitory, tangible, machine-readable medium that includes executable code that, when executed by one or more processors (e.g., processor 210), may cause the one or more processors to perform the process of method 400. Some common forms of machine-readable media that may include the process of method 400 are, for example, a floppy disk, a flexible disk, a hard disk, a magnetic tape, any other magnetic medium, a CD-ROM, any other optical medium, a punch card, a paper tape, any other physical medium having a pattern of holes, a RAM, a PROM, an EPROM, a FLASH-EPROM, any other memory chip or cartridge, and / or any other medium configured to be read by a processor or a computer.

[0059] This description and the accompanying drawings illustrating aspects, embodiments, implementations, or applications of the present invention should not be construed as limiting. Various mechanical, compositional, structural, electrical, and operational changes may be made without departing from the spirit and scope of this description and claims. In some instances, well-known circuits, structures, or techniques have not been shown or described in detail so as not to obscure the embodiments of the present disclosure. Similar numbers in two or more figures represent the same or similar elements.

[0060] In this description, specific details are described that describe some embodiments consistent with the present disclosure. Numerous specific details are described to provide a thorough understanding of the embodiments. However, it will be apparent to one skilled in the art that some embodiments can be practiced without some or all of these specific details. The specific embodiments disclosed herein are meant to be illustrative, not limiting. One skilled in the art may recognize other elements that are not specifically described herein, but are within the scope and spirit of the present disclosure. In addition, to avoid unnecessary repetition, one or more features shown and described in connection with one embodiment may be incorporated into other embodiments, unless specifically described otherwise, or if one or more features would render the embodiment non-functional.

[0061] Although exemplary embodiments have been shown and described, the foregoing disclosure contemplates a wide range of modifications, changes, and substitutions, and in some cases, some features of the embodiments may be employed without the corresponding use of other features. Those skilled in the art will recognize many variations, alternatives, and modifications. Accordingly, the scope of the present invention should be limited only by the claims that follow, which claims are appropriately construed broadly and in a manner consistent with the scope of the embodiments disclosed herein.

Claims

1. 1. A method for controllable text summarization of a document, performed by a text summarization system, comprising: Receiving an input text document at a communications interface; extracting one or more keywords from the input text documents by sequence labeling the one or more keywords through a language model trained with a training dataset of a plurality of documents and a plurality of corresponding keywords; receiving, via a user interface, a sequence of control tokens and one or more control parameters relating to characteristics of a summary to be generated, the characteristics of the summary including any of entities mentioned in the input text document, a target length of the summary, and a type of the input text document, the one or more control parameters including a prompt corresponding to the type of the input text document; modifying the one or more keywords based on the received control token sequence; generating, by the language model, the summary for the input text document based on the modified one or more keywords in accordance with the one or more control parameters; the language model comprises: prepending keyword sequences to training source documents separated by special tokens; Randomly dropping a subset of keywords from the keyword sequence during training. A method for training by performing the steps.

2. The prompt may include: A first abstract prefix summarizing the contributions of the research paper; A second abstract prefix summarizing the object of the invention of the patent document; and A third summarization prefix for summarizing the input text document in a guided question-and-answer format. The method of claim 1 , wherein the first and second inputs are connected to the first and second inputs.

3. generating a first set of modified keywords and a first control parameter from the received sequence of control tokens; generating, by the language model, a first version of the summary for the input text document based on the first set of modified keywords in accordance with the first control parameters; generating a second set of modified keywords and second control parameters from the received sequence of control tokens; generating a second version of the summary for the input text document based on the second set of modified keywords in accordance with the second control parameters using the language model; The method of claim 1 further comprising:

4. The language model further comprises: inputting the training source documents having the keyword sequences into the language model; generating an output summary using said language model; updating the language model by maximizing the conditional probability of an output summary conditional on the training source documents and the keyword sequences; The method of claim 1 , wherein the training is performed by performing

5. 1. A system for controllable text summarization of documents, comprising: a communications interface for receiving an input text document; a memory for storing a language model trained with a training data set of a plurality of documents and a plurality of corresponding keywords; one or more hardware processors; wherein the one or more hardware processors are extracting one or more keywords from the input text document by sequence labeling the one or more keywords through the language model; receiving, via the communications interface, a sequence of control tokens and one or more control parameters relating to characteristics of a summary to be generated, the characteristics of the summary including any of entities mentioned in the input text document, a target length of the summary, and a type of the input text document, the one or more control parameters including a prompt corresponding to the type of the input text document; modifying the one or more keywords based on the received control token sequence; generating, by the language model, the summary for the input text document based on the modified one or more keywords in accordance with the one or more control parameters; and the language model is prepending keyword sequences to training source documents separated by special tokens; Randomly dropping a subset of keywords from the keyword sequence during training. The system is trained by performing

6. 1. A non-transitory processor-readable medium storing a plurality of processor-executable instructions for controllable text summarization of documents, the plurality of processor-executable instructions comprising: Receiving an input text document at a communications interface; extracting one or more keywords from the input text documents by sequence labeling the one or more keywords through a language model trained with a training dataset of a plurality of documents and a plurality of corresponding keywords; receiving, via a user interface, a sequence of control tokens and one or more control parameters relating to characteristics of a summary to be generated, said characteristics of said summary comprising: the one or more control parameters include any of entities mentioned in the input text document, a target length of the summary, and a type of the input text document, the one or more control parameters including a prompt corresponding to the type of the input text document; modifying the one or more keywords based on the received control token sequence; generating, by the language model, the summary for the input text document based on the modified one or more keywords in accordance with the one or more control parameters; and wherein the language model is executed by one or more processors to perform operations including: prepending keyword sequences to training source documents separated by special tokens; Randomly dropping a subset of keywords from the keyword sequence during training. A non-transitory processor-readable medium that is trained by performing

Images