Machine learning methods, computer systems, and programs

Abstract

A machine learning method implemented on a computer to generate explainable text summaries extracts a subset of sentences from at least one input document as an extracted summary, and generates prompts by adding context to the sentences in the extracted summary. A fluent summary is generated by feeding the prompts to a generative language model. The source information of the sentences from the fluent summary is determined by mapping the sentences from the fluent summary to at least one sentence in the extracted summary, and mapping at least one sentence from the extracted summary to at least one sentence from at least one input document. A transparent summary view displays the sentences from the fluent summary, along with the source information from the extracted summary and at least one input document, in a user interface. This method can be used in, but is not limited to, medical AI, public safety, and other machine learning applications for stable and explainable document summarization.

Description

Technical Field

[0004] , , , , , , , , , ,

[0003]

[0001] This application claims priority based on U.S. Provisional Application No. 63 / 522,470, filed on June 22, 2023, which is incorporated herein by reference.

[0002] The present invention relates to artificial intelligence (AI) and machine learning, and more particularly, to an explainable and efficient method, system, and computer-readable medium for text summarization.

Background Art

[0003] Large language models (LLMs) such as ChatGPT have shown high performance in many natural language processing (NLP) tasks, including text summarization. However, in the generated summaries, such generative LLMs may generate incorrect information. In particular, the generated text may contain information contrary to facts, also known as hallucination, and may appear to be stated with confidence, leading to a lack of reliability and stability of the LLM system and making its use in many high-risk scenarios dangerous. Furthermore, the use of such LLM systems is inefficient in terms of computing resources and computing time.

Summary of the Invention

Means for Solving the Problems

[0004] In one embodiment, the present invention provides a computer-implemented machine learning method for generating explainable text summaries. A subset of sentences is extracted from at least one input document as an extracted summary. Context is added to the extracted sentences to generate prompts. A fluent summary is generated by using these prompts as input to a generative language model. Source information for the sentences in the fluent summary is determined by mapping the sentences in the fluent summary to at least one sentence in the extracted summary, and further mapping the sentences in the extracted summary to sentences in at least one input document. A transparent summary view is generated, displaying the sentences in the fluent summary, the extracted summary, and the source information obtained from at least one input document on a user interface. The method is applicable to, but is not limited to, medical AI, public safety, and other machine learning applications for stable and explainable document summarization. [Brief explanation of the drawing]

[0005] [Figure 1] Figure 1 schematically shows a text summary generation method and overall system structure according to one embodiment of the present invention. [Figure 2] Figure 2 schematically shows the overall structure of a preprocessor according to one embodiment of the present invention and the steps performed by the preprocessor. [Figure 3] Figure 3 schematically shows the overall structure of a contextualizer according to one embodiment of the present invention and the steps performed by the contextualizer. [Figure 4] Figure 4 schematically shows the overall structure of an explanatory device according to one embodiment of the present invention and the steps performed by the explanatory device. [Figure 5] Figure 5 schematically shows a potential implementation of the tracer of the demonstrator according to one embodiment of the present invention. [Figure 6] Figure 6 shows an example of a transparent view according to one embodiment of the present invention. [Figure 7]Figure 7 is a block diagram of an exemplary processing system capable of performing any or all of the operations disclosed herein. [Modes for carrying out the invention]

[0006] One embodiment of the present invention makes the use of a Large-Scale Language Model (LLM) for summarization more reliable, secure, stable, and efficient in terms of either or both the computational resources and computation time required, by modifying the input in a transparent and explainable manner. Another embodiment of the present invention reduces the input length by reducing the size of the input document before it is fed into the LLM, thereby reducing the computational load in terms of either or both the computational resources and computation time required, and significantly reducing the cost of the LLM. This makes it possible to allocate computational resources to other tasks (e.g., other query processing), enabling secure, stable, and reliable processing of increased queries.

[0007] According to existing technologies, LLM systems have a technical drawback called hallucination, which can lead to the generation of incorrect information in summaries (i.e., information that is factually inaccurate in light of the information contained in the original document). This is particularly dangerous in high-risk applications and use cases. For example, if a physician makes decisions about a patient based on a summary automatically obtained from an LLM, and that summary contains incorrect information, the physician may make an incorrect judgment based on that incorrect information, potentially endangering the patient's life. For instance, if the original document states that the patient smokes, the LLM may generate a summary that incorrectly states the patient does not smoke. In this case, the physician may not realize that smoking may be the cause of the symptoms. One embodiment of the present invention improves such LLM systems by suppressing hallucination and improving the factual accuracy of summaries, thereby enhancing the stability and reliability of the LLM system.

[0008] Furthermore, according to existing technologies, querying an LLM system is computationally intensive and therefore costly in terms of computational resources and computation time. For example, in a call center, if an employee needs to summarize the content of the previous call, the cost of the employee's salary to create this single summary may be lower than the cost of querying an LLM system like ChatGPT to perform the same task. The computational cost of a query depends on the length of the input provided to the LLM system. Naturally, in summarization, LLMs are often given very long documents (or multiple documents) as input. This is particularly useful in reducing the reading time and cognitive load that would be very large for a human to read a large amount of text. One embodiment of the present invention can reduce this cognitive load by using improved AI technology. Another embodiment of the present invention can also significantly reduce the cost of using an LLM system by reducing the computational load, and can save either or both computational resources and computation time. In particular, one embodiment of the present invention can shorten the input length in queries to LLMs as much as possible.

[0009] One embodiment of the present invention provides a solution that simultaneously addresses both of these shortcomings of the existing technology, and allows for the secure and reliable reduction of the input length before the LLM generates a fluent summary.

[0010] In a first embodiment, the present invention provides a machine learning method for generating a computer-implemented, explainable text summary, comprising extracting a subset of sentences from at least one input document as an extracted summary, and generating prompts by adding context to the extracted sentences. The fluent summary is generated by feeding the prompts to a generative language model. Source information for sentences from the fluent summary is determined by mapping sentences from the fluent summary to at least one sentence in the extracted summary, and mapping at least one sentence from the extracted summary to at least one sentence from at least one input document. A transparent summary view is generated for display in a user interface, showing sentences from the fluent summary together with source information from the extracted summary and at least one input document.

[0011] In a second embodiment, the present invention provides a method relating to the first embodiment, wherein one or both of the following are performed using a natural language inference model that predicts whether each of the sentences is implied to the other one for the mapping: mapping sentences from a fluent summary to at least one sentence of an extracted summary, and mapping the at least one sentence from the extracted summary to at least one sentence from at least one input document.

[0012] In a third embodiment, the present invention provides a method according to the first or second embodiment, wherein mapping the sentences from the fluent summary to at least one sentence of the extracted summary is performed by embedding each of the sentences from the fluent summary and the extracted sentences as numerical vectors using a sentence embedding model, and selecting the k extracted sentences that are closest to the sentences from the fluent summary as evidence in the extracted summary.

[0013] In a fourth embodiment, the present invention provides a method according to any of the first to third embodiments, wherein mapping the at least one sentence from the extracted summary to at least one sentence from at least one input document is performed by embedding each of the at least one sentences from the at least one input document as a numerical vector using a sentence embedding model, and selecting k of the at least one sentences from the at least one input document that are closest to the k extracted sentences in the evidence in the extracted summary as evidence in the input document.

[0014] In a fifth embodiment, the present invention provides a method according to any of the first to fourth embodiments, further comprising removing meaningless words and phrases from the extracted sentence prior to generating a prompt.

[0015] In a sixth embodiment, the present invention provides a method according to any of the first to fifth embodiments, wherein meaningless words and phrases are determined by comparing the extracted sentence with a database containing words and phrases that have been previously classified as meaningless.

[0016] In a seventh embodiment, the present invention provides a method according to any of the first to sixth embodiments, further comprising determining a subset of sentences using a neural network that receives at least one input document and outputs an informality score for each sentence contained in at least one document.

[0017] In an eighth embodiment, the present invention provides a method according to any of the first to seventh embodiments, wherein providing context to extracted sentences includes resolving ambiguity in each of the extracted sentences by performing coreference resolution and entity linking based on the at least one input document.

[0018] In a ninth aspect, the present invention provides a method according to any one of the first to eighth aspects, further comprising checking whether one or more of the extracted sentences overlap by semantically comparing the embeddings of the sentences extracted using a similarity threshold, and excluding one or more of the extracted sentences from the prompt based on a determination that one or more of the extracted sentences are within the similarity threshold with respect to another one of the extracted sentences.

[0019] In a tenth aspect, the present invention provides a method according to any one of the first to ninth aspects, wherein the prompt includes a list of the extracted sentences, and for each of the extracted sentences having an assigned context, the assigned context is concatenated to each of the extracted sentences, and the prompt further includes instructions for causing a generative language model to summarize, paraphrase, or rewrite the extracted sentences and output them as a fluent summary.

[0020] In an eleventh aspect, the present invention provides a method according to any one of the first to tenth aspects, wherein a transparency summary view highlights in a user interface source information including the at least one sentence from and at least one sentence from at least one input document, and sentences from the fluent summary.

[0021] In a twelfth aspect, the present invention provides a method according to any one of the first to eleventh aspects, wherein the at least one input document includes patient data, and the transparency summary view is used to assist decision-making in a use case of medical artificial intelligence (AI) or automated medicine.

[0022] In a thirteenth aspect, the present invention provides a method according to any one of the first to twelfth aspects, wherein the at least one input document includes a criminal investigation report, and the transparency summary view is used for one or both of assisting decision-making in a use case of public safety and activating forensic tools.

[0023] In a 14th aspect, the present invention provides a computer system for generating a text summary having one or more processors, either alone or combined, that execute a machine learning method for generating an explainable text summary according to any one of the 1st to 13th aspects, either alone or in combination.

[0024] In a 15th aspect, the present invention provides a tangible non-transitory computer-readable medium storing instructions that, when executed by one or more hardware processors, execute a machine learning method according to any one of the 1st to 13th aspects to generate an explainable text summary.

[0025] FIG. 1 schematically shows a method for generating a text summary and an overall system structure 100 according to an embodiment of the present invention. At least one document is obtained as (a) input 102. Then, it is passed to (1) an extractive summarizer 104. Next, (1) the extractive summarizer 104 selects a subset of sentences from (a) input 102. And (3) a preprocessor 108 generates a prompt for an abstractive summarizer as (c) a preprocessed summary 110 by imparting context to the extracted sentences and removing meaningless words and phrases. Then, (2) an abstractive summarizer 112 (which may be a large language model) receives (c) the preprocessed summary 110 as input and generates (d) a fluent summary 114 as output. Finally, (4) an explainer 120 receives three different summaries ((b) an extractive summary 106, (c) the preprocessed summary 110, (d) the fluent summary 114) for (a) input 102 and generates a transparent summary view for one or both of another AI system and a user.

[0026] Therefore, as a whole, the system according to one embodiment of the present invention generates three different summaries for the input 102: (a) an extracted summary 106, (c) a pre-processed summary 110, and (d) a fluent summary 114. Then, (e) a transparent summary view 125 links these four texts ((a) input 102, (b) extracted summary 106, (c) pre-processed summary 110, and (d) fluent summary 114) in a transparent, and therefore secure, stable, and reliable manner. The advantages of different summaries and inputs are shown in Table 1 below: [Table 1] Extract summarizer:

[0027] (1) The extracting summarizer 104 is configured to (a) extract useful text units contained in the input 102, so (2) the abstracting summarizer 112 can process with fewer tokens while maintaining the same level of usefulness in the final (d) fluent summary 114. Furthermore, providing only the extracted subset as input to the abstract summarizer 112 (2) results in a more factual and consistent summary of the document in input 102 (a).

[0028] (1) The extracting and summarizing machine 104 receives (a) the document of input 102 and outputs (b) an extracting summary 106 which is a portion of the data contained in (a) the document of input 102. For example, it is (a) a collection of sentences, phrases, words, and subwords contained in the document of input 102. The order of these units (e.g., sentences) can be defined so that it can be read as a natural document.

[0029] (1) One implementation of the extracting summarizer 104 is, for example, a script that searches for the first k sentences in each document from the documents of input 102, where k is a parameter of the extracting summarizer 104. Another implementation is to use a neural network that takes the documents of input 102 as input and outputs a usefulness score for each sentence in the documents of input 102 as input 102. If the usefulness score of a sentence is greater than a threshold m, that sentence is included in the extracting summary 106 as input 104, where m is a parameter of the extracting summarizer 104. The neural network can be trained using training data (D, L). The label L represents the usefulness of each sentence s in the document D of input 102 as input 102. Thus, given a document D consisting of text, the trained neural network attempts to predict the label L. preprocessor:

[0030] Figure 2 schematically shows the overall structure of a preprocessor 200 according to one embodiment of the present invention and the steps performed by the preprocessor 200. The preprocessor 200 is configured to generate prompts for an abstract summarizer (e.g., an LLM system such as ChatGPT) to (1) reduce the number of prompt tokens to save computational cost and inference time, and (2) provide context to the extracted sentences to reduce hallucination in the final summary. Instead of using the entire original text as a prompt to the abstract summarizer, the preprocessor 200 (b) takes an extracted summary 206 (essentially a list of sentences) as input and then returns a final prompt to pass to the abstract summarizer to generate an abstract and fluent summary.

[0031] The preprocessor 200 receives (b) Extracted Summary 206 as input. (b) Extracted Summary 206 consists of a list of sentences selected and ranked by the Extracted Summary Module (see Figure 1). These sentences are the original sentences from the original (a) Input 202. Each sentence is then examined individually according to embodiments of the present invention. The parts of the preprocessor according to embodiments of the present invention are numbered in the figures as follows: (1) A module according to one embodiment of the present invention pops sentence 214 from the list of sentences 212 in the extracted summary and adds it to the database of previous sentences. This is done for each sentence in the extracted summary. (2) The contextualizer 216 receives (a) the document(s) of input 202 and (b) the sentence 214 from the extracted summary 206 and generates a context to attach to the sentence 218 so that the LLM can correctly understand the meaning of the sentence even in isolation. For example, the context can be used to resolve ambiguity of personal pronouns (e.g., "he" refers to "John F. Kennedy"). The component also converts sentence 214 into a contextualized sentence 220 that can be interpreted without any context. A method and system structure 300 of the contextualizer 216 according to one embodiment of the present invention is shown in Figure 3 and includes the parts indicated by the reference numerals in the figure as follows. (α) Contexts are connected to a knowledge graph (KG) database 308, obtained from pre-trained coreference resolution modules 304 and entity links 306, respectively. These steps are performed "offline" against the entire original document (or, if the input consists of multiple documents, the entire collection of documents). Since many of the sentences selected from the summary contain ambiguous terms, their contexts are used to resolve the ambiguity. Such ambiguous terms may be, for example, personal pronouns, place names, or names of people where it is not clear what they refer to (e.g., "John F. Kennedy" could refer to both the former US president and the airport in New York). For the coreference resolution unit, a pre-trained coreference resolution model is used to explain which string this pronoun refers to (e.g., in "He lived in Washington," "he" refers to "John F. Kennedy"). For entity link 306, the entity description is retrieved from the knowledge graph in the KG database 308 and provided as an attribution context (for example, in "John F. Kennedy was busy today," "John F. Kennedy" is the airport in New York). Entity linker 306 is pre-trained to link to the knowledge graph to provide the attribution context. All personal pronouns, as well as nouns and entity names, can be considered ambiguous. Other ambiguous terms can be determined by looking up the string database. A string can be considered ambiguous if it has multiple meanings or entries (for example, "bank" could be an institution, a building, or a riverbank). Once an entity is resolved against the reference knowledge graph, the entity description can be easily retrieved from the knowledge graph (as this is the information most knowledge graphs today possess). (β) The context retriever 312 receives the sentence from the extracted summary 310 and the previously generated context. It then searches for and outputs the context 318 of the sentence (for example, searching for coreference resolution phrases for pronouns in the sentence). This step is performed "online" only for sentences from the extracted summary. (γ) Finally, the sentence rewriter 314 receives (a) the input 302, the extracted sentence in the extracted summary 310, and the sentence context 318 document, and transforms them into sentence 320 (contextualized sentence 220 in Figure 2). The sentence rewriter 314 can be implemented by concatenating the context to the original sentence (for example, in "He lived in Washington," "he" refers to "John F. Kennedy"). Another implementation example is to use a neural network trained to paraphrase sentences. For example, this implementation can use a neural network that takes (s1, s2) pairs as input data, where s1 is the original sentence and s2 is the paraphrased sentence. Given the input sentence s1, the neural network is trained by the model to generate s2 which has the same meaning as s1, although different in terms of the words used.

[0032] (3) The contextualized sentence 220 is passed to a drop module according to one embodiment of the present invention. The drop module generates a simplified sentence 226 by dropping words or phrases 222 that are considered meaningless, in particular words or phrases whose removal does not change the meaning of the original sentence. The module may be, for example, a trained model or neural network that has been trained to drop such meaningless words. Similarly, the module may obtain information from, for example, an external database 224 of words and phrases. In particular, whenever the module encounters a particular word or phrase (e.g., "still", "on the other hand", etc.) that has been marked as meaningless by the database 224, the module drops them. The database 224 may include a list of words and phrases that can be dropped, including a predefined dictionary, including conjunctive adverbs ("still", "on the other hand", "however", "to sum up", "in other words", etc.), articles (e.g., "a"), and irrelevant conjunctive expressions (e.g., "The iPhone costs $1,000, but it's not worth it"). For convenience, one embodiment of the present invention does not depend on the source of these words or phrases, which can be obtained from a database, a recommender system, etc. The abbreviated sentence 226 is also added to the database 230 of the previous sentence. (4) Next, a module according to one embodiment of the present invention checks for semantic duplication 228 between the current sentence and a previously processed sentence, and if there is duplication, it discards the sentence 226. This is achieved by semantically comparing the two sentences using sentence embedding, according to one embodiment of the present invention. If the similarity falls below a certain threshold, the sentences are considered equivalent. If they are equivalent, the sentence currently under consideration is dropped. Otherwise, processing continues and the sentence is added 232 to the database 230 of previous sentences. A consistency filter may also be applied to recognize whether there is inconsistency in the original sentence, and if there is inconsistency, the sentence or inconsistency may be discarded as well. (5) An instruction prompt 236 is generated for each sentence, and a sentence-level prompt 238 is provided. If the abstract summarizer is tuned to follow instructions (e.g., ChatGPT), the prompt consists of (a) instructions to the abstract summarizer, and (b) each preprocessed sentence and its relevant context. The instructions to the abstract summarizer are static and can be any prompt that prompts the abstract summarizer to summarize the given text. For example, such a prompt could be, "Rewrite the following sentences more fluently." Each sentence is written on a new line. Each sentence is written in two columns, followed by tab-separated context to avoid ambiguity. Example: "He lived in Washington" → "'He' refers to 'John F. Kennedy'" If the abstract summarizer is not tuned to follow instructions (e.g., BART (Bidirectional Auto-Regressive Transformers)), the prompt consists only of each preprocessed sentence, and their concatenation is used as the input document to the model. (6) General instructions for the LLM are concatenated together with each statement and the context attached thereto to generate (c) a final prompt 240 for generating a preprocessed summary 210. Abstract summarizer:

[0033] Referring again to Figure 1, (2) the abstract summarizer 112 receives the reduced (c) preprocessed summary 110 and outputs (d) a fluent summary 114. The fluent summary 114 is the information contained in the reduced (c) preprocessed summary 110, which has been transformed into a fluent, coherent document that is easy for the user to read. The contents of the fluent summary 114, including entities (e.g., names of people, organizations, places), claims, and facts, must be included in the contents of the reduced (c) preprocessed summary 110.

[0034] One implementation involves using a neural network that takes a reduced (c) preprocessed summary 110 and outputs a (d) fluent summary 114 that paraphrases the content it contains. For example, a summarizer can be obtained by fine-tuning a trained language model with a summary dataset. Alternatively, a language model trained for a general-purpose task (e.g., ChatGPT) can be used. Given a reduced summary (c) and a task instruction (e.g., "Summarize the document"), the abstract summarizer generates an abstract summary as (d) fluent summary 114. (2) The abstract summarizer 112 can be executed via a web API (Application Processing Interface) or a local computing system. Explanator:

[0035] (4) The explainer 120 is configured to trace the sentences in the final (d) fluent summary 114 to (a) the original document of input 102 and (b) the sentences in the extracted summary 106. This component improves the explainability of the summarization system by allowing the user to easily verify the source of the summary sentences and whether the summary contains any information that contradicts the facts.

[0036] Figure 4 schematically shows the overall structure of an explainer 400 according to one embodiment of the present invention and the steps performed by the explainer 400 according to one embodiment of the present invention. The explainer receives (a) an input 402, (b) an extracted summary 406, (d) a fluent summary 414, and (f) documents of each sentence 408 in the fluent summary. The explainer 400 outputs (e) a transparent summary view 425. The explainer 400 includes (5) a first evidence retriever 430 having a tracer 410 for sentences in the fluent summary, and (6) a second evidence retriever 440 having a tracer 416 for evidence in the extracted summary. The explainer 400 operates as follows: (1) (5) The tracer 410 for sentences in the fluent summary receives (f) each sentence 408 from the fluent summary and (b) the extracted summary 406, and outputs (g) evidence 412 in the extracted summary. This evidence is the source information of the sentence, which is the text unit (e.g., sentence) contained in (b) the extracted summary 406. Figure 5 shows two possible implementations 500a and 500b of the tracer 410 for sentences in the fluent summary, as follows. First implementation 500a: (5) Tracers 410 for sentences in the fluent summary are implemented by a natural language inference (NLI) model 510. An NLI model predicts whether the meaning of one text (hypothesis) is implied by the meaning of another text (premise), and is implemented, for example, by using an existing trained NLI model. (b) For each sentence in the extracted summary 506a, it is checked whether a sentence in the fluent summary 508a implies each sentence. Only sentences predicted to "imply" are filtered 512 and defined as (g) evidence 514a in the extracted summary. In NLI, the task is to determine whether a "hypothesis" is true (implies), false (contradictory), or unknown (neutral) given a "premise," so the NLI model is trained to determine whether the hypothesis is implied. For example, consider the premise p = "a soccer game in which multiple men are playing." In this case, the following hypothesis is true: h = "some men are playing sports." In this case, h is implied by p. Second implementation 500b: Alternatively or additionally, (5) the tracer 410 for sentences in the fluent summary can be implemented using a sentence embedding model (dense retriever) such as the SentenceBERT model, where each input sentence is represented as an n-dimensional vector. (f) Each sentence in the fluent summary 508b and (b) the extracted summary 506b are converted into numerical vectors which are embeddings 520 in the latent space. The distance between the vector of the fluent summary and the vector of the extracted summary is calculated for each case using a k-nearest neighbors (kNN) retriever 522. The sentences are ranked in ascending order of distance, and the top k closest sentences are retrieved from the fluent summary and provided as evidence 514b of the extracted summary, where k may be a learned parameter or a predetermined parameter. This process is repeated for each sentence 508b in the fluent summary (f), providing embeddings for each sentence 508b in the fluent summary (f) and the extracted summary 506b. (2) (6) Tracer 416 for evidence in extracted summary receives (g) evidence 412 in extracted summary and (a) document in input 402, and outputs (h) evidence 418 of input document, which is the source information of evidence 412 in extracted summary, which is the text unit (e.g., sentence) contained in (a) document in input 402. (6) Tracer 416 for evidence in extracted summary can be implemented in the same way as used for (5) Tracer 410 for sentences in fluent summary. Thus, in the method for (6) Tracer 416 for evidence in extracted summary, compared to the method for (5) Tracer 410 for sentences in fluent summary, the sentences of the extracted summary output from (5) Tracer 410 for sentences in fluent summary as (g) evidence 412 in extracted summary replace each sentence 402 from (f) fluent summary, and (a) input document 402 replaces (b) extracted summary 406. Therefore, (a) the embedding of each sentence in input document 402 is determined. (3) (7) The summary viewer 422 receives (d) the fluent summary 414, (f) the sentence 408 in the fluent summary, (b) the extracted summary 406, (g) the evidence 412 in the extracted summary, (a) the document of input 402, and (h) the evidence 418 in the input document. (7) The summary viewer 422 outputs (e) a transparent summary view 425 that highlights the sentences in the fluent summary and the source information in the extracted summary and the input document. Figure 6 shows an example of (e) a transparent summary view 425 that includes (d) the fluent summary with each sentence in the fluent summary, (b) the extracted summary with the evidence text in the extracted summary, and (a) the input document with the evidence text in the input document. Thus, the user can select individual sentences from the fluent summary, and the evidence for that sentence is,

number

[0037] Embodiments of the present invention can be practically applied to bring about further improvements in many technological fields, such as medical AI, automated medicine, AI-assisted drug or material development, resource allocation, and forensic medicine.

[0038] In one embodiment, the present invention can be applied to summarizing a patient's diagnosis. Here, a use case is given where physician records (from one or more physicians) can be summarized along with the patient's final diagnosis. The data source (input) includes at least one document of physician notes (from at least one physician) concerning a patient with specific symptoms. The document includes information on the diagnosis and specific methods for treating the particular patient. By applying the method according to the embodiment of the present invention, the input document can be shortened to create three levels of summaries: extracted summaries, brief summaries, and fluent summaries. These summaries include information on the patient's diagnosis and methods for treating the particular patient. These summaries are automatically highlighted for physician review from a safety perspective.

[0039] In another embodiment, the present invention can be applied, for example, to AI-assisted drug development, contact center or consulting support, cyber threat intelligence (CTI (e.g., CTI report summaries)), etc. Here, a use case is to generate three summaries with different levels of safety and fluency given at least one input document, thereby reducing the cognitive burden on humans. The data source (input) includes at least one document. By applying the method according to one embodiment of the present invention, the input document can be shortened to create three levels of summaries: an extracted summary, a simplified summary, and a fluent summary. The relationship between the summaries and the original document is traced. The output is three different summaries and a user interface that traces and highlights how the different summaries relate to each other and to the original document. Processing can be performed automatically or semi-automatically based on the summaries used to support decision-making. In addition, the s1, s2 highlighting pairs of sentences can also be used to calculate factual accuracy scores, so that the fidelity of sentences generated in the fluent summary can be automatically estimated. This provides the user with further information and can further increase confidence in the AI ​​system.

[0040] In another embodiment, the present invention can be applied to the summarization of patient histories for medical AI or automated medical care. Here, as a use case, given at least one patient's medical history report (e.g., electronic health records (EHR)), a physician-safe summary and explanatory interface can be generated. The data source (input) includes at least one patient report. By applying the method according to one embodiment of the present invention, the input document can be shortened and three levels of summarization can be created: extracted summary, brief summary, and fluent summary. The relationship between the summaries and the original document is traced. The output is the three different summaries and a user interface that provides a final summary of the patient's medical history that allows for diagnosis, tracing and highlighting how the different summaries relate to each other and to the original document. Based on the report and / or diagnosis, potential drugs or treatments may be generated automatically or semi-automatically.

[0041] In another embodiment, the present invention can be applied to summarizing citizen reports. Here, as a use case, given at least one citizen report, a secure summarization and explanation interface can be generated for government officials (for example, when a citizen is seeking employment at an employment agency and the government official is the citizen's caseworker). This saves government officials time and allows for the extraction of insights that might otherwise be overlooked. The data source (input) includes at least one citizen report. By applying the method according to one embodiment of the present invention, the input document can be shortened and three levels of summarization can be created: extracted summaries, brief summaries, and fluent summaries. The relationship between the summaries and the original document is traced. The output is the three different summaries and a user interface that traces and highlights how the different summaries relate to each other and to the original document, providing a final summary of the citizen report for government officials to make decisions. Based on the report, employment opportunity forecasts can be generated automatically or semi-automatically.

[0042] In another embodiment, the present invention can be applied to patent abstracts. Here, the use case is that given at least one patent or patent application, a secure abstract and descriptive interface can be generated. This saves patent readers time and allows them to extract insights that might otherwise be missed. The data source (input) includes at least one patent or patent application. By applying the method according to one embodiment of the present invention, the input document can be shortened to create three levels of abstracts: an extracted abstract, a brief abstract, and a fluent abstract. The relationship between the abstracts and the original document is traced. The output is the three different abstracts and a user interface that traces and highlights how the different abstracts relate to each other and to the original document, and provides the user with a final abstract of the patent or patent application to make a final decision (e.g., whether the patent or patent application relates to other patents).

[0043] In another embodiment, the present invention can be applied to an automated forensic tool or to the summarization of suspect reports for public safety. Here, a use case is that given at least one suspect report, a secure summary and explanatory interface for police officers can be generated. This saves police officers time and allows them to extract insights that might otherwise be missed. The data source (input) includes at least one suspect report. By applying the method according to one embodiment of the present invention, the input document can be shortened and three levels of summarization can be created: extracted summary, brief summary, and fluent summary. The relationship between the summaries and the original document is traced. The output is the three different summaries and a user interface that traces and highlights how the different summaries relate to each other and to the original document, providing a final summary of the suspect report that serves as a basis for police officers to make decisions. Based on the report, a forensic tool can be activated in an automatically or semi-automatic manner.

[0044] In one embodiment, the present invention provides a method for preparing a summary in a transparent, explainable, and cost-effective manner. This method includes the following steps: 1) Select a method for extraction and summarization, and implement it. 2) The extractor / summarizer receives at least one document as an input document (a) and selects a subset of sentences from it. 3) The preprocessor adds context to the extracted sentence (b), removes meaningless words and phrases, and generates a prompt for an abstract summarizer (e.g., ChatGPT) (c). 4) The abstract summarizer takes prompt (c) as input and generates a fluent summary (d). 5) The explainer maps sentences from (d) to (b) and sentences from (b) to (a). 6) The user interface highlights how (a), (b), and (d) are related to each other.

[0045] One embodiment of the present invention enables the following improvements over existing technologies. 1) An extractive summarizer is used first, and only the extracted subset is provided as input to an abstract summarizer (e.g., LLM) to generate an abstract summary (e.g., generated by LLM) in a more secure, transparent, and stable manner. 2) Reduce the cost and computational resources required to generate the abstract summary by further reducing unnecessary words in the extracted summary. 3) By using contextualizers (e.g., coreference resolution), the efficiency, security, stability, and reliability of the generated summaries are improved, and hallucination is avoided. In contrast, feeding raw extracted text to an abstract summarizer may cause hallucination. 4) Improve the efficiency, security, stability, and reliability of the generated summaries by using an explainer that can trace from the final summary sentence back to the original input sentence and the intermediate expression (the sentences in steps 2) and 3)) of the method described above.

[0046] Using current LLM systems such as ChatGPT is unsafe and costly, especially when the input query length is long. One embodiment of the present invention improves the safety of summaries, which is particularly important in high-risk areas such as medicine and healthcare. While computational costs using LLM increase with input length, one embodiment of the present invention also achieves cost reduction. Furthermore, one embodiment of the present invention can be used to safely shorten the input length in the first place.

[0047] Zhang, Haopeng, et. al., “Extractive Summarization via ChatGPT for Faithful Summary Generation”, arXiv:2304.04193 (2023) describes a summarization method using LLM and proposes the adoption of a post-extraction abstraction strategy to improve the factual accuracy of the summaries. However, in contrast to one embodiment of the present invention, the summarization method (1) does not consider providing context to the extracted sentences for hallucination reduction, and (2) does not have a system for tracing back from the summaries to the original documents.

[0048] Norkute, Milda, et al., “Towards Explainable AI: Assessing the Usefulness and Impact of Added Explainability Features in Legal Document Summarization,” CHI EA '21: Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, Article No.: 53, Pages 1-7 (May 2021), describes a traceback system that directly traces back from a summary to sentences in the original document. However, this system is not memory-efficient because it requires retraining the similarity score for each pair of summary and original document sentences. In contrast, one embodiment of the present invention improves the calculation by retaining only the similarity score between the summary and the sentences extracted from the original document. This feature reduces memory usage and is particularly beneficial when the original document length is long.

[0049] Choi, Eunsol, et, “Decontextualization: Making Sentences Stand-Alone,” arXiv:2102.05169 (2021), describes a technique for rewriting sentences to allow for out-of-context interpretations while maintaining their meaning. However, this technique is not applicable to sentences extracted in post-extraction summarization systems, and therefore does not mitigate hallucination.

[0050] Referring to Figure 7, the processing system 700 may include one or more processors 702, a memory 704, one or more input / output devices 706, one or more sensors 708, one or more user interfaces 710, and one or more actuators 712. The processing system 700 can represent each of the computing systems disclosed herein.

[0051] The processor 702 may include one or more separate processors, each having one or more cores. Each of the separate processors may have the same or different structure. The processor 702 may include one or more central processing units (CPUs), one or more graphics processing units (GPUs), circuits (e.g., application-specific integrated circuits (ASICs)), digital signal processors (DSPs), etc. The processor 702 may be mounted on a common board or on multiple different boards.

[0052] Processor 702 is configured to perform a particular function, method, or operation (for example, configured to implement the execution of a function, method, or operation) if at least one of several separate processors is capable of performing an operation that embodies the function, method, or operation. Processor 702 may perform an operation that embodies the function, method, or operation, for example, by executing code stored in memory 704 (e.g., interpreting a script) and / or by trafficking data through one or more ASICs. Processor 702, i.e., processing system 700, may be configured to automatically perform any of the functions, methods, and operations disclosed herein. Accordingly, processing system 700 may be configured to perform any (e.g., all) of the protocols, devices, mechanisms, systems, and methods described herein.

[0053] For example, where this disclosure states that a method or apparatus performs task "X" (or that task "X" is performed), such statement should be understood to disclose that the processing system 700 may be configured to perform task "X". The processing system 700 is configured such that at least the processor 702 performs a function, method or operation.

[0054] Memory 704 may include volatile memory, non-volatile memory, and any other medium capable of storing data. Each of the volatile memory, non-volatile memory, and any other type of memory may include multiple different memory devices located in multiple different locations, each having a different structure. Memory 704 may include remotely hosted (e.g., cloud) storage.

[0055] Examples of memory 704 include non-temporary computer-readable media such as RAM, ROM, flash memory, and EEPROM; any type of optical storage disk such as DVDs, Blu-ray discs, magnetic storage devices, holographic storage, HDDs, and SSDs; and any medium that can be used to store program code in the form of instructions or data structures. Any and all methods, functions, and operations described herein can be fully executed in the form of tangible and / or non-temporary machine-readable code (e.g., interpretable scripts) stored in memory 704.

[0056] The input / output device 706 may include any components for trafficking data, such as ports, antennas (i.e., transceivers), and printed conductive paths. The input / output device 706 may enable wired communication via USB®, DisplayPort®, HDMI®, Ethernet, etc. The input / output device 706 may enable electronic, optical, magnetic, and holographic communication with appropriate memory 706. The input / output device 706 may enable wireless communication via WiFi®, Bluetooth®, cellular communication (e.g., LTE®, CDMA®, GSM®, WiMAX®, NFC®, GPS, etc.). The input / output device 706 may include wired and / or wireless communication paths.

[0057] Sensor 708 may capture physical measurements of the environment and report them to processor 702. User interface 710 may include a display, physical buttons, a speaker, a microphone, and a keyboard. Actuator 712 may enable mechanical force control by processor 702.

[0058] The processing system 700 may be distributed. For example, some components of the processing system 700 may reside in a remote host network service (e.g., a cloud computing environment), while other components of the processing system 700 may reside in a local computing system. The processing system 700 may have a modular design in which a particular module includes multiple features / functions as shown in Figure 7. For example, an I / O module may include volatile memory and one or more processors. As another example, individual processor modules may include either or both ROM (read-only memory) and a local cache.

[0059] The subject matter of this disclosure has been illustrated and described in detail in the drawings and the foregoing description, but such illustrations and descriptions should be interpreted as illustrative and not limiting. Furthermore, since the present invention is defined by its claims, the descriptions herein that characterize the invention should also be interpreted as illustrative and not limiting. Changes and modifications may be made within the scope of the following claims, which may include any combination of features from the different embodiments described above.

[0060] The terms used in the claims should be interpreted as having the broadest and most reasonable meaning consistent with the foregoing description. For example, the use of the article "a" or "the" when introducing an element should not be interpreted as excluding multiple elements. Similarly, unless it is clear from the context or the foregoing description that only one of A and B is intended, the wording "or" should be interpreted as inclusive, so as the wording "A or B" does not exclude "A and B". Furthermore, the wording "at least one of A, B and C" should be interpreted as one or more of a group of elements consisting of A, B and C, and should not be interpreted as requiring at least one of each of the enumerated elements A, B and C, regardless of whether A, B and C are related as categories or not. Furthermore, the wording "A, B and / or C" or "at least one of A, B or C" should be interpreted as including any one entity from the enumerated elements, e.g., A, any subset from the enumerated elements, e.g., A and B, or the entire list of elements A, B and C.

Claims

1. A machine learning method for generating explainable text summaries, implemented on a computer, Extract a subset of sentences from at least one input document as an extracted summary. A prompt is generated by adding context to the sentence of the extracted summary. By using the aforementioned prompt as input to a generative language model, a fluent summary is generated. The source information of the sentence from the fluent summary is determined by mapping the sentence from the fluent summary to at least one sentence from the extracted summary, and mapping the at least one sentence from the extracted summary to at least one sentence from the at least one input document. To generate a transparent summary view for display in a user interface, which together shows the sentences from the fluent summary, the source information from the extracted summary, and the at least one input document. method.

2. Mapping sentences from the fluent summary to at least one sentence from the extracted summary, and mapping at least one sentence from the extracted summary to at least one sentence from at least one input document, or both, are performed using a natural language inference model that predicts whether each of the sentences is implied to one of the other sentences. The method according to claim 1.

3. Mapping the sentences from the fluent summary to at least one sentence of the extracted summary is performed by embedding each of the sentences from the fluent summary and the extracted sentences as numerical vectors using a sentence embedding model, and selecting the k extracted sentences that are closest to the sentences from the fluent summary as evidence in the extracted summary. The method according to claim 1.

4. Mapping the at least one sentence from the extracted summary to the at least one sentence from the at least one input document is performed by embedding each of the at least one sentences from the at least one input document as a numerical vector using a sentence embedding model, and selecting the k of the at least one sentences from the at least one input document that are closest to the k extracted sentences in the evidence within the extracted summary as evidence within the input document. The method according to claim 3.

5. Prior to generating the prompt, the process further includes removing meaningless words and phrases from the extracted sentence, The method according to claim 1.

6. The aforementioned meaningless words and phrases are determined by comparing the extracted sentence with a database containing words and phrases previously classified as meaningless. The method according to claim 5.

7. The process further includes determining a subset of the sentences using a neural network that receives at least one input document and outputs an informationality score for each sentence contained in the at least one document. The method according to claim 1.

8. Assigning the context to the extracted sentences includes resolving ambiguities in each of the extracted sentences by performing coreference resolution and entity linking based on the at least one input document. The method according to claim 1.

9. By semantically comparing the embeddings of the extracted sentences using a similarity threshold, it is checked whether one or more of the extracted sentences are duplicates. The further step includes excluding one or more of the extracted sentences from the prompt based on the determination that one or more of the extracted sentences fall within the similarity threshold with respect to another of the extracted sentences, The method according to claim 1.

10. The prompt includes a list of extracted sentences, wherein for each of the extracted sentences having an assigned context, the assigned context is concatenated to each of the extracted sentences, and the prompt further includes instructions to cause a generative language model to summarize, paraphrase, or rewrite the extracted sentences to output them as fluent summaries. The method according to claim 1.

11. The transparent summary view highlights in the user interface the source information, which includes the at least one sentence from the extracted summary and the at least one sentence from the at least one input document, and the sentence from the fluent summary. The method according to claim 1.

12. The at least one input document includes patient data, and the transparent summary view is used to support decision-making in use cases of artificial intelligence (AI) or automated healthcare. The method according to claim 1.

13. The at least one input document includes a criminal investigation report, and the transparent summary view is used for either or both to support decision-making and / or to activate forensic tools in public safety use cases. The method according to claim 1.

14. A computer system for generating text summaries, comprising one or more processors configured to perform, alone or in combination, machine learning methods for generating explainable text summaries, wherein the machine learning methods are: Extract a subset of sentences from at least one input document as an extracted summary. A prompt is generated by adding context to the sentence of the extracted summary. By using the aforementioned prompt as input to a generative language model, a fluent summary is generated. The source information of the sentence from the fluent summary is determined by mapping the sentence from the fluent summary to at least one sentence from the extracted summary, and mapping the at least one sentence from the extracted summary to at least one sentence from the at least one input document. To generate a transparent summary view for display in a user interface, which together shows the sentences from the fluent summary, the source information from the extracted summary, and the at least one input document. Computer system.

15. A tangible, non-temporary, computer-readable medium for generating an explainable text summary, wherein the medium stores instructions that, when executed by one or more hardware processors, enable the execution of a machine learning method, and the machine learning method is Extract a subset of sentences from at least one input document as an extracted summary. A prompt is generated by adding context to the sentence of the extracted summary. By using the aforementioned prompt as input to a generative language model, a fluent summary is generated. The source information of the sentence from the fluent summary is determined by mapping the sentence from the fluent summary to at least one sentence from the extracted summary, and mapping the at least one sentence from the extracted summary to at least one sentence from the at least one input document. To generate a transparent summary view for display in a user interface, which together shows the sentences from the fluent summary, the source information from the extracted summary, and the at least one input document. A tangible, non-temporary, computer-readable medium.

Images