Generative text summarization system and method

Abstract

Generative text summarization system and method. A generative automatic text summarization system and method are disclosed, which can employ a search and re-ranking strategy to improve the performance of the summarization task. The system and method can employ a transformer neural model to assist the summarization task. The transformer neural model can be trained to learn human abstractions and then can be operable to generate abstractive summaries. In the case of generating multiple summary hypotheses, a best-first search algorithm and a re-ranking algorithm can be employed to select the best candidate summary as part of the output summary.

Description

Technical Field

[0001] The present disclosure relates to generative text summarization systems and methods. Background Art

[0002] Text summarization strategies tend to employ machine learning algorithms to generate concise summaries of larger texts. For example, text summarization can be used to generate shorter paragraph summaries for longer news articles or text articles that may be tens to hundreds of pages long. The machine learning employed needs to filter out redundant or unimportant information and generate summaries that accurately convey the meaning of the larger text. Summary of the invention

[0003] A system and method for a generative text summarization model is disclosed. The model may receive an input text dataset and expand a search space for one or more candidate words to be selected for inclusion in a text summary. The model may include ranking one or more candidate words within the search space using a best-first search algorithm. The model may also re-rank one or more candidate words to be included in the text summary using a soft-bound word-reward (SBWR) algorithm. It is contemplated that when the text summary exceeds a predicted length threshold, the SBWR algorithm may apply a decreasing reward value to the one or more candidate words. The SBWR algorithm may also apply an increased reward value to the one or more candidate words when the text summary is below the predicted length threshold. The SBWR algorithm may further select one or more candidate words when the text summary is equal to the predicted length threshold.

[0004] The model can further smooth the decreasing reward values ​​and increasing reward values ​​using a sigmoid function. The decreasing reward values ​​and increasing reward values ​​can be scaled using the value trained to select one or more candidate words to be included in the text summary. In addition, one or more candidate words can be re-ranked when the input text dataset exceeds a predefined length threshold.

[0005] The BP normalization can be calculated to apply a penalty to one or more candidate words that do not meet the prediction length threshold. The BP normalization can be calculated by adding the logarithm of the brevity penalty to the length normalization scoring function. In addition, the brevity penalty can be designed so that the generative text summarization model does not produce short translations from the input text dataset. The brevity penalty can also include a duplication rate value that reduces the brevity penalty to zero.

[0006] A generative text summarization model may also be trained using a transformer neural model that includes an encoder machine learning algorithm and a decoder machine learning algorithm. During a training sequence, an input text dataset may be input to the encoder machine learning algorithm; and a target summary text dataset may be input to the decoder machine learning algorithm. The transformer neural model may also use one or more source tokens to determine probability values ​​for one or more target summary tokens. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 is an exemplary system that employs a neural model for generative text summarization.

[0008] Figure 2 is an exemplary flowchart using a neural model for generative text summarization.

[0009] Figure 3 is an exemplary source code section for implementing a best-first search strategy.

[0010] Figure 4 is an exemplary embodiment of a transformer neural model for training a generative text summarization neural model. DETAILED DESCRIPTION

[0011] Embodiments of the present disclosure are described herein. However, it will be understood that the disclosed embodiments are merely examples, and other embodiments may take various forms and alternative forms. The figures are not necessarily to scale; some features may be enlarged or minimized to show the details of a particular component. Therefore, the specific structural and functional details disclosed herein should not be interpreted as restrictive, but only as a representative basis for teaching those skilled in the art to adopt the embodiments in various ways. As will be understood by those of ordinary skill in the art, the various features illustrated and described with reference to any one of the accompanying drawings may be combined with the features illustrated in one or more other figures to produce embodiments that are not explicitly illustrated or described. The combination of illustrated features provides representative embodiments for typical applications. However, various combinations and modifications of features consistent with the teachings of the present disclosure may be desirable for specific applications or implementations.

[0012] Text summarization is generally the process of reducing a larger text (e.g., a long news article) to generate a summary that includes the contextually accurate content of the original input text data set. In order to generate an accurate summary, various algorithms can attempt to take into account the length, writing style, and grammar of the original text. Two known methods for performing text summarization include extractive summarization and generative (i.e., abstractive) summarization. Extractive summarization typically operates by selecting and using sentences from the original text as part of the summary.

[0013] Alternatively, generative summarization can build an internal semantic representation and use natural language generation techniques to create an abstract target summary from the original text. Therefore, a generative summarization system can create an abstract target summary that is more accurate than an extractive summarization system. In addition, generative summarization can create a more abstract summary and express a meaning that is more closely similar to the original text.

[0014] It is contemplated that a Transformer Neural Framework employing word embeddings and an encoder-decoder structure can be used to improve the output summaries of a generative summarization system. During the decoding phase, multiple summary hypotheses can be generated as candidates for the system to select as the summary output. If the search strategy employs a known "beam search" algorithm, the possible output candidates may look like each other, with only minor variations on a given word. Therefore, it is also contemplated that a different strategy can be employed, in which the search space of summary candidates can be first expanded.

[0015] For example, a best-first search algorithm may be employed to expand the search space, thereby generating more diverse candidates. Once diversified, candidate summaries may be selected that have different styles or different emphasis on information. A re-ranking method may then be employed to select the best candidate as output. The re-ranking method may employ a soft bound word reward (SBWR) algorithm that selects the best candidate as output for the summary.

[0016] Figure 1 An exemplary system 100 that can be used to employ a generative text summarization neural model is illustrated. System 100 may include at least one computing device 102. Computing system 102 may include at least one processor 104 operably connected to a memory unit 108. Processor 104 may be one or more integrated circuits that implement the functionality of a processing unit (PU) 106. PU 106 may be a commercially available central processing unit (CPU) that implements instructions such as one of the x86, ARM, Power, or MIPS instruction set families. Alternatively, processing unit 106 may be a commercially available graphics processing unit (GPU) that consists of hundreds of cores that are operable to simultaneously process a large number of parallel tasks (i.e., parallel computations).

[0017] During operation, the PU 106 may execute stored program instructions retrieved from the memory unit 108. The stored program instructions may include software that controls the operation of the PU 106 to perform the operations described herein. In some examples, the processor 104 may be a system on a chip (SoC) that integrates the functionality of the PU 106, the memory unit 108, a network interface, and an input / output interface into a single integrated device. The computing system 102 may implement an operating system for managing various aspects of operation.

[0018] The memory unit 108 may include volatile memory and non-volatile memory for storing instructions and data. Non-volatile memory may include solid-state memory such as NAND flash memory, magnetic and optical storage media, or any other suitable data storage device that retains data when the computing system 102 is disabled or loses power. Volatile memory may include static and dynamic random access memory (RAM) that stores program instructions and data. For example, the memory unit 108 may store a machine learning model 110 or algorithm, a training data set 112 for the machine learning model 110, and / or raw source data 115.

[0019] The computing system 102 may include a network interface device 122 configured to provide communications with external systems and devices. For example, the network interface device 122 may include a wired and / or wireless Ethernet interface as defined by the Institute of Electrical and Electronics Engineers (IEEE) 802.11 family of standards. The network interface device 122 may include a cellular communication interface for communicating with a cellular network (e.g., 3G, 4G, 5G). The network interface device 122 may further be configured to provide a communication interface to an external network 124 or a cloud.

[0020] The external network 124 may be referred to as the World Wide Web or the Internet. The external network 124 may establish a standard communication protocol between computing devices. The external network 124 may allow information and data to be easily exchanged between computing devices and the network. One or more servers 130 may communicate with the external network 124.

[0021] Computing system 102 may include input / output (I / O) interface 120, which may be configured to provide digital and / or analog input and output. I / O interface 120 may include an additional serial interface (eg, a universal serial bus (USB) interface) for communicating with external devices.

[0022] The computing system 102 may include a human-machine interface (HMI) device 118, which may include any device that enables the system 100 to receive control inputs. Examples of input devices may include human-machine interface inputs such as keyboards, mice, touch screens, voice input devices, and other similar devices. The computing system 102 may include a display device 132. The computing system 102 may include hardware and software for outputting graphics and text information to the display device 132. The display device 132 may include an electronic display screen, a projector, a printer, or other suitable devices for displaying information to a user or operator. The computing system 102 may be further configured to allow interaction with a remote HMI and a remote display device via a network interface device 122.

[0023] System 100 may be implemented using one or more computing systems. Although this example depicts a single computing system 102 implementing the described features, it is intended that the various features and functions may be separated and implemented by multiple computing units communicating with each other. The architecture selected may depend on a variety of factors.

[0024] The system 100 may implement a machine learning algorithm 110 configured to analyze raw source data 115 (or data sets). The raw source data 115 may include raw or unprocessed sensor data, which may represent an input data set for a machine learning system. The raw source data 115 may include videos, video clips, images, and raw or partially processed sensor data (e.g., data from a digital camera or LiDAR sensor). In some examples, the machine learning algorithm 110 may be a neural network algorithm (e.g., a Transformer, CNN, RNN, or DNN) that may be designed to perform a predetermined function.

[0025] Figure 2 An exemplary flowchart 200 employing a generative text summarization neural model is illustrated. Flowchart 200 may begin at block 202, where a plain text dataset may be provided as input to a generative summarization system. The dataset may be text provided from a keyboard, or the text may be provided from one or more documents stored in memory 118. The text may also be a web page or document provided from an external network 124.

[0026] The flowchart may then proceed to block 204 where a decoding phase may be employed to determine an optimal output summary based on the input text data set. It is contemplated that a "beam search" algorithm may be employed to determine a near-optimal solution from the sequence tag decoding process. Preferably, a best-first search strategy (e.g., a greedy best-first search or a pure heuristic search) that favors a given candidate (i.e., a possible choice) may be employed, and then the best score may be assigned to the candidate.

[0027] Figure 3 An exemplary source code portion for implementing a best-first search strategy is illustrated. As illustrated, the best-first search strategy can employ a priority heap that maintains partial summaries of the input text. The partial summaries can be scored according to a heuristic function. The best-first search algorithm can iteratively take the highest-scoring partial summaries and extend the partial summaries by one word. The newly extended summary sequence can then be placed (i.e., pushed) back into the priority heap. The best-first search strategy can also generate the top k number of candidates for the new summary sequence. It is envisioned that the top k number of candidates can be generated by selecting the word that generates the highest probability score (P) and iteratively appending the selected word to the partial summary ( y ) to generate the top k. The highest probability can be generated using equation 1 below:

[0028] .

[0029] The highest probability score ( P ) can be calculated using the logarithmic value based on the input text ( x ) bitwise OR the partial summary ( ) to the current section summary ( y j ).

[0030] The flowchart may then proceed to block 206 where a re-ranking process is applied to the summary candidates and the candidate that produces the best result is selected. It is contemplated that in addition to a best-first search process that expands the search space to provide one or more diverse candidates, a re-ranking process may still be necessary to rank the candidates.

[0031] For example, an important aspect to consider during text summarization is the length of the input text data (e.g., the length of the input text sentence or string). A best-first search strategy will generally provide enhanced results (i.e., higher output scores) for candidates of shorter length. However, short summaries may be too abstract and may lose key information from the original text. In fact, in some applications, an overly short summary that includes only a few words may not be an informative summary, even though a best-first search strategy can generate a high log score using Equation 1 above.

[0032] Length normalization, which adjusts term frequency or relevance scores, can be employed to normalize the effect of text length on document ranking. Length normalization can be employed specifically so that longer text strings or sentences are considered for re-ranking. It is generally understood that length normalization can provide better results than beam search algorithms. A brevity penalty (BP) normalization value ( ) to ensure that the input text is suitable for the summarization task. The BP-norm value can also apply a penalty to summaries that do not meet a predefined expected length. The BP-norm algorithm can be used to calculate the summaries by dividing the logarithm of the brevity penalty ( ) and the length normalized scoring function ( ) are added together, as shown in Equation 2 below:

[0033] [Equation 2]

[0034] where x can be defined as The input sequence, and y can be defined as Assume that the shortness penalty (bp) that can be used to penalize short translations can be calculated using Equation 3 below:

[0035] [Equation 3]

[0036] Where r is the duplication rate, which may include the percentage of summary tokens seen in the source text scaled by a factor c. It is contemplated that when the duplication rate r is set to 1, the penalty may be reduced to a value close to or equal to 0. The penalty term may be further modified to prefer summaries with more duplication from the source text, as shown in Equations 4A and 4B below:

[0037] [Equation 4a]

[0038] [Equation 4b]

[0039] The calculated penalty term can be directly converted into a coefficient to be multiplied by the log-likelihood score. Next, the soft bound word reward (SBWR) algorithm can be used to re-rank the candidates, as shown in Equation 5 below:

[0040] [Equation 5]

[0041] The SWBR algorithm can assign a reward to each word in the summary. If the decoded summary length is greater than the predicted length threshold (i.e., ), then the SWBR algorithm will apply a decreasing reward to the added words. The decreasing reward can be defined as When the decoded digest length is shorter than the expected threshold (i.e., ), the SWBR algorithm will reward each word. It is expected that the SWBR algorithm may prefer the word closest to the predicted length ( ). In addition, a sigmoid function may be used to smooth the reward value, and a coefficient (r) may be used to scale the total reward tuned according to the validation data. The flowchart may then proceed to block 208 where an output text summary is generated based on the word that received the highest reward through the SWBR algorithm.

[0042] Figure 4 An exemplary embodiment of a transformer neural model 400 that can be used to train a generative text summarization neural model is illustrated. The transformer neural model 400 can include an encoder structure 404 and a decoder structure 408. To train the system, an input source text 402, which can include a series of tokens, can be input into the encoder module 404. In addition, a target summary text 406, which can also include a series of text or text strings, can be input into the decoder structure 408. It is contemplated that given a sequence of source tokens as shown in Equation 6 below, the transformer neural model 400 can determine the probability of a target summary token:

[0043] [Equation 6]

[0044] in y can be defined as The target summary tag, and x can be defined as The source tag sequence.

[0045] Figure 4 It is also illustrated that during the training phase, both the input (source) text 402 and the target summary text 406 can be given as training examples to maximize the loss function or the maximum likelihood of observing a given set of training examples. In the decoding phase, given the parameters learned by the transformer neural model 400, the generative summary system can determine the output (y) using Equation 7 below:

[0046] [Equation 7].

[0047] The process, method or algorithm disclosed herein may be delivered to / implemented by a processing device, controller or computer, which may include any existing programmable electronic control unit or dedicated electronic control unit. Similarly, the process, method or algorithm may be stored in a variety of forms as data and instructions that can be executed by a controller or computer, including but not limited to information permanently stored on a non-writable storage medium such as a ROM device and information that can be replaced and stored on a writable storage medium such as a floppy disk, a tape, a CD, a RAM device, and other magnetic and optical media. The process, method or algorithm may also be implemented in a software executable object. Alternatively, a suitable hardware component, such as an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a state machine, a controller, or other hardware components or devices, or a combination of hardware, software, and firmware components, may be used to embody the process, method or algorithm in whole or in part.

[0048] Although exemplary embodiments are described above, it is not intended that these embodiments describe all possible forms included in the claims. The words used in the specification are descriptive words, not restrictive words, and it should be understood that various changes can be made without departing from the spirit and scope of the present disclosure. As previously described, the features of the various embodiments can be combined to form other embodiments of the present invention that may not be explicitly described or illustrated. Although various embodiments may have been described as providing advantages or being preferred over other embodiments or prior art implementations in terms of one or more desired characteristics, it is recognized by those of ordinary skill in the art that one or more features or characteristics can be compromised to achieve the desired overall system properties, depending on the specific application and implementation. These properties may include, but are not limited to, cost, strength, durability, life cycle cost, marketability, appearance, packaging, size, applicability, weight, manufacturability, ease of assembly, etc. As such, in terms of one or more characteristics, to the extent that any embodiment is described as not as desirable as other embodiments or prior art implementations, these embodiments are not outside the scope of the present disclosure and may be desirable for specific applications.

Claims

1. A method for a generative text summarization model, comprising: Receive an input text dataset; expanding a search space for one or more candidate words to be selected for inclusion in the text summary, wherein the one or more candidate words included in the search space are ranked using a best-first search algorithm; and The one or more candidate words to be included in the text summary are re-ranked using a soft bound word reward (SBWR) algorithm, wherein when the text summary exceeds a prediction length threshold, the SBWR algorithm applies a decreasing reward value to the one or more candidate words, and wherein when the text summary is below the prediction length threshold, the SBWR algorithm applies an increasing reward value to the one or more candidate words.

2. The method of claim 1, wherein the SBWR algorithm selects the one or more candidate words when the text summary is equal to a predicted length threshold.

3. The method of claim 1, wherein the SBWR algorithm operates using the following formula: Where x is the input sequence and y is the output hypothesis, is the soft bound word reward SBWR algorithm adopted to rerank the candidates, S(x, y) is the scoring function, r is the duplication rate, i is the decoded digest length, is the predicted length threshold, and It's a diminishing reward.

4. The method according to claim 1, further comprising: Use the sigmoid function to smooth out decreasing and increasing reward values.

5. The method according to claim 1, further comprising: The decreasing reward value and the increasing reward value are scaled using the value trained to select the one or more candidate words to be included in the text summary.

6. The method according to claim 1, when the input text data set exceeds a predefined length threshold, the one or more candidate words are re-ranked.

7. The method according to claim 1, further comprising: A brevity penalty BP normalization is calculated, the BP normalization applying a penalty to the one or more candidate words that do not meet a prediction length threshold. 8 . The method of claim 7 , wherein the brevity penalty BP normalization is calculated by adding a logarithmic value of the brevity penalty to a length normalization scoring function.

9. The method of claim 8, wherein the brevity penalty is designed so that the generative text summarization model does not produce short translations from the input text dataset.

10. The method according to claim 8, wherein: The brevity penalty includes a replication rate value that reduces the brevity penalty to zero.

11. The method according to claim 1, further comprising: Use the Transformer neural model to train a generative text summarization model.

12. The method of claim 11, wherein the transformer neural model comprises an encoder machine learning algorithm and a decoder machine learning algorithm.

13. The method according to claim 12, further comprising: Feed the input text dataset to the encoder machine learning algorithm; And input the target summary text dataset to the decoder machine learning algorithm.

14. The method of claim 13, wherein the transformer neural model uses one or more source tags to determine probability values ​​for one or more target summary tags.

15. The method of claim 14, wherein the transformer neural model determines probability values ​​for the one or more target summary tags using the one or more source tags based on the following equation: Where x is the source token sequence, y is the target summary token, and y j is the current section summary, and y ＜j It is a partial summary.

16. A system operable to employ a generative text summarization model, comprising: a memory operable to store an input text dataset; A processor operable to: expanding a search space for one or more candidate words to be selected for inclusion in the text summary, wherein the one or more candidate words included in the search space are ranked using a best-first search algorithm; and The one or more candidate words to be included in the text summary are re-ranked using a soft boundary word reward (SBWR) algorithm, wherein when the text summary exceeds a prediction length threshold, the SBWR algorithm applies a decreasing reward value to the one or more candidate words, and wherein when the text summary is below the prediction length threshold, the SBWR algorithm applies an increasing reward value to the one or more candidate words.

17. The system of claim 16, wherein the SBWR algorithm selects the one or more candidate words when the text summary is equal to a predicted length threshold.

18. The system of claim 16, wherein: The processor is further operable to smooth decreasing reward values ​​and increasing reward values ​​using a sigmoid function.

19. The system of claim 16, wherein the processor is further operable to scale the decreasing reward value and the increasing reward value using weighted values ​​trained to select the one or more candidate words to be included in the text summary.

20. A non-transitory computer-readable medium operable to employ a generative text summarization model, the non-transitory computer-readable medium having computer-readable instructions stored thereon, the computer-readable instructions operable to be executed to perform the following functions: Receive an input text dataset; expanding a search space for one or more candidate words to be selected for inclusion in the text summary, wherein the one or more candidate words included in the search space are ranked using a best-first search algorithm; and The one or more candidate words to be included in the text summary are re-ranked using a soft boundary word reward (SBWR) algorithm, wherein when the text summary exceeds a prediction length threshold, the SBWR algorithm applies a decreasing reward value to the one or more candidate words, and wherein when the text summary is below the prediction length threshold, the SBWR algorithm applies an increasing reward value to the one or more candidate words.

Images