Time-series-based text summarization method for a continuous process

Abstract

Objects from text documents are extracted the objects are associated with a time factor. The objects are transformed to an object oriented data structure that includes time-series based text pieces sorted by the time factor from the text documents. An intermediate output is generated for the objects by identifying changes in the time-series based text pieces and labeling the changes as contradictions or updates. A text summary of the text documents is generated based on a received prompt and the intermediate output. Fact-based contradictions are detected in the text summary based on comparing the text summary to the intermediate output. An updated text summary is generated based on the one or more fact-based contradictions detected in the text summary. The method has applications including, but not limited to, use cases in computational biology, medical AI and healthcare, and cyber threat security for optimizing machine learning processes or supporting decision making.

Description

TIME-SERIES-BASED TEXT SUMMARIZATION METHOD FOR A CONTINUOUSPROCESSCROSS-REFERENCE TO PRIOR APPLICATION

[0001] Priority is claimed to U.S. Provisional Application Serial No. 63 / 647,123 filed on May 14, 2024, the entire contents of which is hereby incorporated by reference herein.FIELD

[0002] The present disclosure relates to Artificial Intelligence (Al) and machine learning (ML), and in particular to a method, system, data structure, computer program product and computer-readable medium for text summarization.BACKGROUND

[0003] Although many organizations and companies are using enterprise resource planning (ERP) software to support the daily business handling and documentation, they still need to invest a large amount of time and personnel resources to track individual cases to gain a process overview of their business. For example, to decide a further treatment direction of a critical disease such as a brain tumor, doctors need to review the whole treatment process to see which interventions have been tested, and the corresponding response, and how the physical status of the patient has varied. Todays, doctors have to click through their system to manually distill information from complex testing results and long medical records.

[0004] Most of the recent summarization research such as pretrained encode-decoder models and large autoregressive language models focus on single text summarization (see Fabian Retkowski, “The Current State of Summarization,” arXiv:2305.04853v2 (2023), which is hereby incorporated by reference herein, hereinafter “Retkowski”). Few of the current approaches work on multi-documents summarization, such as Michihiro Y asunaga, Rui Zhang, Kshitijh Meelu, Ayush Pareek, Krishnan Srinivasan, and Dragomir Radev, “Graph-based Neural MultiDocument Summarization,” In Proceedings of the 21stConference on Computational Natural Language Learning (CoNLL 2017), pages 452-462, Vancouver, Canada, Association for Computational Linguistics (2017), which is hereby incorporated by reference herein, hereinafter ‘Yasunaga et al.”, but they aim to improve the summarization quality by leveraging information from multiple document sources. None of the current approaches can be directly used to generate descriptive summarization for a process due to the lack of time-series awareness. There is little research related to time-series-based summarization, such as Alireza Ghods, Trong Nghia Hoang, Diane Cook, “Time-to-Pattem: Information-Theoretic Unsupervised Learning for Scalable Time Series Summarization,” arXiv:2308.13722vl (2023), which is hereby incorporated by reference herein, hereinafter “Ghods et al.”, but this work focuses onsummarizing certain data patterns from time-series data instead of text summarization for documents which have a time factor.

[0005] Compared to standard text summarization in natural language processing (NLP) domain, process summarization naturally has the following technical challenges: 1) it usually involves multiple documents at each step in the process and multiple documents usually involve more than one objects; and 2) contradiction handling for time-series information is a technical problem to be overcome for an accurate and reliable summarization.SUMMARY

[0006] In an embodiment, the present disclosure provides a computer-implemented method for text summarization of text documents using a machine learning process. Objects from each of the text documents are extracted and each of the objects are associated with a time factor. Each of the objects are transformed to an object oriented data structure that includes time-series based text pieces sorted by the time factor from the text documents. An intermediate output is generated for each of the objects by identifying changes in the time-series based text pieces and labeling the changes as contradictions or updates. The contradictions and updates are resolved in the intermediate output. A text summary of the text documents is generated based on a received prompt and the intermediate output. One or more fact-based contradictions are detected in the text summary based on comparing the text summary to the intermediate output. An updated text summary is generated based on the one or more fact-based contradictions detected in the text summary. The method has applications including, but not limited to, use cases in computational biology, medical Al and healthcare, and cyber threat security for optimizing machine learning processes or supporting decision making.BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Embodiments of the present disclosure will be described in even greater detail below based on the exemplary figures. The present disclosure is not limited to the exemplary embodiments. All features described and / or illustrated herein can be used alone or combined in different combinations in embodiments of the present disclosure. The features and advantages of various embodiments of the present disclosure will become apparent by reading the following detailed description with reference to the attached drawings which illustrate the following:

[0008] FIG. 1 schematically illustrates a method and system for time-series-based text summarization for a continuous process according to an embodiment of the present disclosure;

[0009] FIG. 2 illustrates an example of input process data;

[0010] FIG. 3 illustrates an example output of an object modeling module and a document modeling module according to an embodiment of the present disclosure;

[0011] FIG. 4 schematically illustrates a two-phase contradiction detector and a process implemented by the two-phase contradiction detector according to an embodiment of the present disclosure;

[0012] FIG. 5 illustrates two methods to feed the time-series-based contradiction recognizer with input data; and

[0013] FIG. 6 is a block diagram of an exemplary processing system, which can be configured to perform any and all operations disclosed herein.DETAILED DESCRIPTION

[0014] Embodiments of the present disclosure provide solutions to the technical challenges of process summarization discussed above by summarizing time-series-based process documents with a two-phase contradiction handling module.

[0015] In a first aspect, the present disclosure provides a computer-implemented method for text summarization of text documents using a machine learning process. Objects from each of the text documents are extracted. Each of the objects are associated with a time factor. Each of the objects are transformed to an object oriented data structure that includes time-series based text pieces sorted by the time factor from the text documents. An intermediate output is generated for each of the objects by identifying changes in the time-series based text pieces and labeling the changes as contradictions or updates, wherein the contradictions and updates are resolved in the intermediate output. A text summary of the text documents is generated based on a received prompt and the intermediate output. One or more fact-based contradictions are detected in the text-summary based on comparing the text summary to the intermediate output. An updated text summary is generated based on the one or more fact-based contradictions detected in the text summary.

[0016] In a second aspect, the present disclosure provides the method according to the first aspect, further comprising detecting fact-based contradictions in the updated text summary based on comparing the updated text summary to the intermediate output.

[0017] In a third aspect, the present disclosure provides the method according to the first aspect or the second aspect, wherein detecting the fact-based contradictions in the updated text summary and generating the updated text summary steps are iteratively repeated until the updated text summary meets a predefined condition, and wherein the predefined condition includes no contradictions detected in the updated text summary or a threshold of contradictions present in the updated text summary.

[0018] In a fourth aspect, the present disclosure provides the method according to any of the first to third aspects, wherein identifying the changes and labeling the changes is implemented by a neural network model, wherein the neural network model is a transformer-based languagemodel, and wherein generating the intermediate output further comprises extracting subjects from the time-series based text pieces, and wherein extracting the subjects is implemented by a neural network language model or a dependency-based information extractor.

[0019] In a fifth aspect, the present disclosure provides the method according to any of the first to fourth aspects, wherein generating the updated text summary includes correcting the detected one or more fact-based contradictions in the text summary using the intermediate output or re-generating the text summary using the detected one or more fact-based contradictions in the text summary.

[0020] In a sixth aspect, the present disclosure provides the method according to any of the first to fifth aspects, further comprising, prior to detecting the one or more fact-based contradictions in the text summary, clustering the objects of each text document based on textual similarity of the objects by computing a vectorized semantic embedding of the time-series based text pieces of the objects and using at least one of cosine similarity, Euclidean distance, or Jaccard similarity to evaluate a similarity of text embeddings of the vectorized semantic embedding of the time-series based text pieces.

[0021] In a seventh aspect, the present disclosure provides the method according to any of the first to sixth aspects, wherein generating the updated text summary is further based on using the clustered objects of each text document.

[0022] In an eighth aspect, the present disclosure provides the method according to any of the first to seventh aspects, wherein identifying the changes and labeling the changes is implemented by a neural network model, wherein the neural network model is trained using a dataset having a subject, text pair, contradiction label, and the time factor, wherein detecting the one or more fact-based contradictions includes using a fact-based contradiction detector that is trained using the dataset and data including the text pair and the contradiction label, wherein the time factor includes a value for the subject that changes at a different time factor.

[0023] In a ninth aspect, the present disclosure provides the method according to any of the first to eighth aspects, wherein identifying the changes and labeling the changes is implemented by a neural network model, wherein the neural network model has been trained by: sorting the time-series based text pieces in chronological order for a subject; and feeding the subject and the sorted time-series based text pieces into the neural network model such that the neural network model learns to identify text changes as a contradiction or an update by learning how the text changes are distributed in the sorted time-series based text pieces, how frequency the text changes occur, and a magnitude of the text changes.

[0024] In a tenth aspect, the present disclosure provides the method according to any of the first to ninth aspects, wherein identifying the changes and labeling the changes is implementedby a neural network model, wherein the neural network model has been trained by: sorting the time-series based text pieces in chronological order for a subject; and feeding, for each time factor, the subject, historical text sequences, and a previous identification of a change in the sorted time-series based text pieces such that the neural network model learns to identify text changes as a contradiction or an update in a recurrent manner from a historical contradiction distribution and a historical update distribution.

[0025] In an eleventh aspect, the present disclosure provides the method according to any of the first to tenth aspects, wherein extracting the objects includes using a pre-trained neural network model that identifies the objects from each of the text documents, the pre-trained neural network model including a name entity recognition model.

[0026] In a twelfth aspect, the present disclosure provides the method according to any of the first to eleventh aspects, wherein the pre-trained neural network model is configured to extract the time factor based on the text including a date or a timestamp, or based on a creation date of a respective text document.

[0027] In a thirteenth aspect, the present disclosure provides the method according to any of the first to twelfth aspects, wherein the text documents are health records from patients and treatment documents from doctors, and wherein the updated text summary includes a long-term disease diagnosis based on the received prompt corresponding to a summary of a treatment process for a particular patient of the patients.

[0028] In a fourteenth aspect, the present disclosure provides a computer system for text summarization of text documents using a machine learning process comprising one or more processors, which, alone or in combination, are configured to perform a machine learning method for text summarization of text documents using a machine learning process according to any of the first to thirteenth aspects.

[0029] In a fifteenth aspect, the present disclosure provides a tangible, non-transitory computer-readable medium for text summarization of text documents using a machine learning process which, upon being executed by one or more hardware processors, provide for execution of a machine learning method according to any of the first to thirteenth aspects.

[0030] Although organizations have their own internal processes to handle their business, to obtain a clear process summary, users usually need to click through their system and manually distill information from a large amount of data. Embodiments of the present disclosure provide an automatic process summarization system with a focus on resolving contradictions that occur in time-series documents, thereby not only overcoming the technical challenges discussed above, but also resulting in more accurate summarization.

[0031] Embodiments of the present disclosure provide a method and system to generate descriptive (textual) summary for a continuous process. From a high-level perspective, the method and system provide following features: 1) modeling a process by objects and documents with a time factor (e.g. in chronological order); 2) linking related objects across the entire process; 3) generating a customizable textual summary of the process; and 4) detecting both time-series-based and fact-based contradictions.

[0032] FIG. 1 shows a high-level overview of the system architecture and process flow according to an embodiment of the present disclosure. In FIG. 1, system components are numbered, such as (1) and (2), inputs and (intermediate) outputs are tagged as (a) and (b), and the dotted line indicates an additional procedure performed in some embodiments. The system is firstly triggered by the of input process data (a). An object modeling module (object modeler) (1.1) then identifies objects for each of the documents included in the process data (a). For each detected object, the document modeling module (document modeler) (1.2) gathers all related text from all object-mentioned corresponding documents and sorts them in chronological order with a corresponding time factor. For each object, based on the time-series texts, the two-phase contradiction detector (1.3) implements a first-phase operation to resolve contradictions recognized from the text. Then, in some embodiments, an object linker (1.4) links related objects based on their content similarity. Given the contradiction-free text of the process, a promptbased summarizer (summarizer) (1.5) generates the textual summary based on user interests. The two-phase contradiction detector (1.3) then applies a second-phase operation to check if the generated summary (b) content has a contradiction to the source process text. If a contradiction is detected, the summarizer (1.5) regenerates the summary (b) until the quality is satisfactory.

[0033] The following will explain each components in details: Process Data (a):

[0034] To enable process summarization, the first step is to find a feasible way to reasonably represent a continuous process in a dataset, which should contain all static information in the process as well as the dynamic flow of the process steps. Here, the process data (a) is designed to represent a process. In practice, process data (a) is a collection of documents with a time factor (ti - tn). As shown in FIG. 2, the process data (a) represents the time-series sequence of a workflow 200, where each step in the workflow involves one or multiple documents 202.Object Modeling Module (1.1) and Document Modeling Module (1.2):

[0035] Due to a process usually consisting of various steps, where different steps might handle different perspectives of the case, the documents involved in different steps may by related but differ a great deal with respect to topics. The object modeling module (1. 1) is designed to analyze process documents 302 from an object perspective (for instance, one topiccould be one object), and is responsible to identify objects 304 from each of the documents 302 (as shown in FIG. 3). The document modeling module (1.2) is designed to transform the process documents into a time-series-based and object-oriented representation to be prepared for a better summarization.

[0036] Pseudocode 1 below explains how both the object modeling module (1.1) and the document modeling module (1.2) work. From an implementation perspective, they can be realized together (e.g., they do not have to be separate models). For each step / phrase ti in the process (represented by workflow 200), the system will walk through all documents. For each document in each step ti, the object modeling module (1.1) identifies objects from the text. For instance, the object modeling module (1.1) can be a name entity recognition model such as Applicant’s NSP (see Carolin Lawrence, Bhushan Kotnis, Mathias Niepert, “Attending to Future Tokens For Bidirectional Sequence Generation,” arXiv: 1908.05915v2 (2019), which is hereby incorporated by reference herein, which is a pretrained neural network model. Meanwhile, the object modeling module (1.1) extracts the time factor of the object-related text pieces. For example, the time factor can be recognized from the text itself if it mentions a certain date or time stamp, or it can take the creation date of the document. The document modeling module (1.2) formulates a data structure (e.g., a dictionary in Python) to store detected objects and corresponding object-related text pieces sorted on time factor 306 (as shown in FIG. 3). In the implementation of an embodiment, the document modeling module (1.2) is not an Al model, but rather a software program with a developer predefined data structure.1. Begin2. Obj_sequences = {}3. For each phrase ti in process do4. For each document in phrase do5. Identify objects from document6. Extract time factor of object-related text7. Add detected unseen object and related text with timestamp into Obj_sequences8. End9. Output: a dictionary Obj sequencesPseudocode 1: Implementations of Object Modeling Module (1.1) and Document Modeling Module (1.2)Two-phase Contradiction Detector (1.3):

[0037] FIG. 4 schematically illustrates a system architecture of the two-phase contradiction detector (1.3) for a single object example (the dotted line from the summary (b) to the fact-based contradiction detector (2.2) indicates the flow will stop based on certain conditions). In practice,the two-phase contradiction detector (1.3) processes all objects of the process and the input of the prompt-based summarizer (1.5) can be a collection of golden text (e) of a group of linked objects. Due to information usually being modified frequently along the workflow, process summarization has significant technical challenges for contradiction handling compare to normal text summarization. The first technical challenge is to identify the self-contradiction of the object to decide if the change is a contradiction or a real update. The second technical challenge is to validate if the generated summary content contradicts the input text source. The two-phase contradiction detector (1.3) solves these technical challenges by addressing both of the two types of contradiction issues phase by phase.1) Time-series-based Contradiction Recognizer (2. 1):

[0038] The first phase uses a time-series-based contradiction recognizer (2.1), a new component introduced by embodiments of the present disclosure, to identify the self- contradiction of each object. Preferably, it is a neural network model, for instance, implemented as a transformer-based language model. For each object, the system firstly extracts all subjects of each sentence text, for example, the extraction can be achieved by either a neural network language model such as MillE (see Bhushan Kotnis, Kiril Gashteovski, Daniel Onoro Rubio, Vanesa Rodriguez-Tembras, Ammar Shaker, Makoto Takamoto, Mathias Niepert, Carolin Lawrence, “millE: Modular & Iterative Multilingual Open Information Extraction,” arXiv:2110.08144 (2022), which is hereby incorporated by reference herein) or a dependencybased information extractor, such as CoreNLP. For each subject, the time-series-based contradiction recognizer (2. 1) then recognizes if the text change is a “contradiction” or an “update”.

[0039] To enable the model learning the classification with time-series awareness, the model input is adapted to be able to represent the time sequence. FIG. 5 illustrates two possible methods to feed the time-series-based contradiction recognizer. Method 1 sorts all related text in chronological sequence for a subject 502, and feeds both the subject and text sequences (ti - tn) into the model. In this way, the model is able to learn the contradiction recognition by looking into how the text changes distributed in the time sequences (e.g., in an early stage of the process or in a later stage), how frequent the changes occur, what is the magnitude of the changes, etc. Method 2 feeds the same content into the model in a sequential way. For each time stamp, it is provided to feed always <subject 502, the historical text sequences (ti - tn), recognition of previous step 504> into the model, so that the model will generate the final recognition result in a recurrent way. The sequential method of Method 2 could additionally allow the model to learn from the historical contradiction / update distribution (e.g., if the last three changes are updates, the fourth one is more likely to be an update than a contradiction). In implementation, thedecision of which method is used to train the model, or whether a combination of both methods are used, could depend on the specific dataset and use case. For example, if an object is a person’s age and a contradiction is detected Method 1, described above, would work better than Method 2 because the change frequency of an age and the change magnitude are good indicators for identifying whether an age change is a contradiction. To continue the example if a person’s age changes from 20 to 22 this is more likely a reasonable update, however changing from 20 to 80 is more likely a contradiction. In scenarios where the object is something else, like blood sugar, Method 2 would be more appropriate to detect contradictions. For example, if a given patient has blood sugar issues their blood sugar usually changes overtime with treatment and so if a last three changes were determined to be updates it is more likely that the fourth detected change is also an update rather than a contradiction.2) Training Data (d):

[0040] The system is provided with a set of training data (d) to train the two contradiction detectors (2. 1), (2.2). Each data in the training dataset involves <subject, text pair, contradiction label, time factor>. Regarding the time factor, the detailed implementation can be flexible as long as it presents the value of the subject in text pair changes at different time stamps. Although the two contradiction detectors (2.1) and (2.2) are trained in different ways, the set of training data (d) fulfills both their data requirements. To train the time-series-based contradiction recognizer (2. 1), the information of <subject, text pair, contradiction label, time factor> is used. To train the fact-based contradiction detector (2.2), at least the information of <text pair, contradiction label> is used.3) Golden Text (e):

[0041] The golden text (e) is the intermediate output generated by the time-series-based contradiction recognizer (2.1), where all detected contradictions have been resolved. In addition to the contradiction recognition, the module (time-series-based contradiction recognizer 2.1) will further refine the time sequential text sequences based on the recognition results. For instance, if “age” changes of subject A are detected as an “update”, the module will take the last change as the golden value. If the “amount” changes of subject B are identified as a “contradiction”, the module will keep the changes in the golden text (e) so that human users have the chance to notify the change history. The golden text (e) is then used as the data source to generate the process summary and as the source to validate if the generated summary content has contradictions.4) Fact-based Contradiction Detector (2.2):

[0042] The fact-based contradiction detector (2.2) conducts the second phase of the two- phase contradiction detector (1.3). It is responsible to validate if the generated summary (b)contradicts the fact stated in data source (i.e. golden text (e)), for instance, the detector can be a neural network model such as SUMMAC (see Philippe Laban, Tobias Schnabel, Paul N.Bennett, Marti A. Hearst, “SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization,” arXiv:2111.09525 (2021), which is hereby incorporated by reference herein, hereinafter “Laban et al.”). Once any contradiction is detected, the system can improve the summary (b) consistency in different ways. For instance, the system could ask the summarizer (1.5) to regenerate the summary until no contradictions exist or until an acceptable threshold is achieved, or the system could implement hard correction by directly correcting the summary content. For regenerating the summary a user may provide a different prompt or instruct the system to generate a different summary as described below with reference to the Prompt-based Summarizer (1.5). A user may also add detected contradictions into the prompt instruction, for example a prompt could include “your last summary had X issue, please generate a new summary to avoid X issue.” In embodiments, the threshold could be an acceptable level of contradictions in summary (b). For example, the threshold could be an amount of contradictions the user can accept.Object Linker (1.4):

[0043] Referring back to FIG. 1, the object linker (1.4) is configured to connect related objects in the process. It links objects by evaluating how similar the text content of the objects are and clusters them based on the textual similarity. In embodiments, the object linker performs this process prior to contradiction identification. For instance, the object linker (1.4) can be a transformer-based language model which firstly computes the vectorized semantic embedding of the text and then evaluates the similarity of the text embeddings by a measure such as Cosine similarity, Euclidean distance or Jaccard similarity. As an example, the model can be based on Sentence-BERT (see Nils Reimers, Iryna Gurevych, “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks,” arXiv: 1908.10084 (2019), which is hereby incorporated by reference herein). As another example, the textual similarity can be based on a simple determination of whether objects belong to the same document. There are numerous ways to determine textual similarity of objects but an example approach would include using a language model to understand a semantic meaning of the text and then compute a textual similarity score. Other approaches may include determining whether objects come from a same document and if they do, the textual similarity is high, otherwise it would be low. The object linker (1.4) is not required in all embodiments, but offers additional advantages in some embodiments. One advantage of activating the object linker (1.4) is that the later summarizer (1.5) is enabled to generate a more comprehensive summary, which involves related objects.Prompt-based Summarizer (1.5):

[0044] The prompt-based summarizer (1.5) is configured to generate the abstractive summary of the given long text. By using different prompts (an instruction text), the summarizer can generate a specific summary which fits best to the user interests. An example prompt may include a request or question such as “Please generate a short summary of the text.” For instance, if the user is interested in knowing the status of an object in the process, the summary will only summarize status-related information of that object excluding unnecessary information such as description text about the object background. In implementation, the system could use any pretrained prompt-based summarizer such as ChatGPT or could fine tune its own summarizer using an annotated training dataset.Summary (b):

[0045] The summary (b) is the final output of the system. It summarizes the textual documents of the process in an abstract way, meanwhile fulfilling user interests. For instance, if the user wants to know a specific aspect of a specific object, the summary (b) will contain only the aspect-related content. Without specifications, the summary will be a comprehensive process summary by default, which contains all aspects of involved objects.

[0046] Embodiments of the present disclosure thus provide for general improvements to computers in machine learning systems by enhancing computer functionality therein to enable time-series-based process summarization that corrects for contradictions. Moreover, embodiments of the present disclosure can be practically applied to use cases to effect further improvements in a number of technical fields including, but not limited to, medical and healthcare (e.g., digital medicine, personalized healthcare, Al-assisted drug or vaccine development, therapy, treatment, etc.), material development, cyberthreat security, public safety and smart cities (e.g., automated traffic or vehicle control, smart districts, smart buildings, smart industrial plants, smart agriculture, energy management, etc.), smart manufacturing (e.g., quality control).

[0047] One embodiment can be practically applied in the medical domain for diagnosis and / or treatment recommendation and planning, for example, through treatment summarization of long-term disease for doctors. Here, a use case is that, for critical illnesses, such as malignant tumors or cardiovascular disease, doctors usually need to spend a long period (in some cases, years) to follow one patient case. Due to the complex treatment process, each patient has a large amount of medical data stored in databases, but the doctor cannot easily gain a clear treatment overview to continue the diagnosis. Embodiments of the present disclosure enable to generate an accurate and time-series-based treatment summary to speed up the diagnosis. The data source and input for this use case includes health records and treatment documents provided by bothpatients and doctors. For instance, an example quality control inspection process might have following steps and documents:

[0048] Application of the method according to an embodiment of the present disclosure represents the patient documents on an object basis in chronological sequence and resolves the time-series data contradictions to identify if the change of an object value (e.g., patient age, blood pressure) in the process is an “update” or a “conflict.” Based on resolved data, the system generates a summary. If the generated summary contains inconsistent facts with original data, the system will redo the generation until the summary quality is satisfactory. The output includes a textual treatment summary of a long-term disease diagnosis. For example, the doctor can receive “an overall summary of the treatment process of patient A” or “a specific summary about the status of sentinel node in last 6 months for patient B with breast cancer.” The output can be used for diagnosis, prescriptions, or to recommend or plan resources for treatment in an automated or semi-automated manner.

[0049] Another embodiment can be practically applied in the domain of smart manufacturing, for example, for automating quality control inspection. Here, a use case is that quality control inspection is a necessary process in factories to monitor the quality of each piece of manufactured goods. The process usually consists of multiple stages of inspection from raw material until the shipment. Although there are many software tools such as ERP systems to help workers record the large amount of inspection activities, they still need to click through their internal system hundreds of times to check the progress of certain products every day and manually write the progress report of the inspections. A smart solution such as that provided by embodiments of the present disclosure, which can automatically generate textual process summaries for each product, would therefore be a useful computer tool that would save time and manufacturing resources. The data source and input for this use case includes quality checking- related documents, for example, provided by regulations or manufacturers. Nowadays, manufactures usually use ERP software to record their quality inspection activities and data along the process, so the documents can be retrieved from this kind of software. For instance, an example quality control inspection process might have following steps and documents:

[0050] Application of the method according to an embodiment of the present disclosure firstly recognizes objects from all documents. For each object, it extracts related textual sentences from all documents and sorts them in a chronological sequence. In some embodiments, the system can also use the object linker to group related objects together, for example, it can group all of the related objects of a specific product together, such as color, weight and measurement. For each object, the system can further resolve time-series-based contradictions by classifying if the value change is a “contradiction” or an “update.” For example, it can resolve a color contradiction if the color of the same product is maintained differently in different document. Afterwards, the system generates the textual summary based on object-related text and detects further if the content in the generated summary contradicts to the input text and, if so, it will redo the summarization until there is no contradiction occurring in summary or the summary is of sufficient quality relative to a threshold. The output includes a textual process summary of the quality control specification of one or more products. The output can be used to automate the quality control processes and / or to control manufacturing processes for the products.

[0051] A further embodiment can be practically applied in the case management domain, for example, to assist handling child support applications. Here, a use case is that the child support department in a citizen’s office has a mature internal process to handle child support applications from the entire city. To handle each case, the process involves a long workflow across several internal teams. The office personnel need to click through their internal system hundreds of times to check the progress of certain cases every day and manually write the progress reports. A smart solution such as that provided by embodiments of the present disclosure, which can automatically generate textual process summary, would therefore be a useful computer tool that would help automate the process and make it more efficient andaccurate. The data source and input for this use case includes application documents, such as forms and certificates provided by applicants. The citizen’s office then handles received documents step by step by following a predefined process, and supplementary documents might be provided along the process. For instance, the process work on different documents at each step could be as below:

[0052] Application of the method according to an embodiment of the present disclosure firstly recognizes objects from all documents. For each object, it extracts related textual sentences from all documents and sorts them in a chronological sequence. In some embodiments, the system can also use the object linker to group related objects together, for example, it can group all father related objects such as age, job and salary together. For each object, the system further resolves time-series-based contradictions by classifying if the value change is a “contradiction” or an “update.” For example, it resolves an age contradiction if the age of the same person is maintained differently in different documents. Afterwards, the system generates the textual summary based on object-related text and detects further if the content in the generated summary contradicts the input text and, if so, it will redo the summarization until there is no contradiction occurring in summary or until the summary is of sufficient quality compared to a threshold. The output includes a textual process summary of the child support application case. The output thereby enables to partly automate the application process and saves time and resources.

[0053] In an embodiment, the present disclosure provides a method for time-series-based process summarization, the method comprising the steps of: Setup a new system (cold start):1. Build an object modeling module (1.1) which extracts objects from the textual documents of a continuous process (see FIG. 3). For instance, the extractor can be a name entity recognizer such as Applicant’s NSP model.2. Build a document modeling module ( 1.2) which transforms the textual documents of a continuous process in an object-oriented way, where texts are extracted for all documents on an object basis and sorted in chronological sequences (see FIG. 3). For instance, the modeling module can be a software tool with a predefined data schema. For example, a data scheme may be of a format as depicted in FIG. 3 at 1.2. An example implementation includes the below pseudocode:If store the documents in an object-oriented database such as MongoDB, the data scheme can be:{First name: xxx,Surname: xxx,Disease 1: [ {disease name: xxx,Medical data: xxx.}}3. Prepare a set of training data (d), in which each data involves information of <subject, text pair, contradiction label, time factor>. The dataset is used to train both the time-series-based contradiction recognizer (2.1) and the fact-based contradiction detector (2.2).4. Build a time-series-based contradiction recognizer (2. 1) which recognizes if the text changes are a “contradiction” or an “update”. For instance, it can be a neural network-based model, which can be trained by one of the two different methods, or a combination thereof, shown in FIG. 5.5. Build a fact-based contradiction detector (2.2) which detects if the generated summary content contradicts to the input data source. For instance, it can be a neural network model such as SUMMAC. For example, if the fact-based contradiction detector (2.2) is a SUMMAC model it will first fragment both the generated summary and input document into sentences and then use a natural language inference model to determine whether a summary sentence logically follows from a document sentence. If the summary sentence does not follow any document sentence it is a contradiction.6. Build a prompt-based summarize (1.5) which generates the abstractive summary of a long text source. By using different prompts, the model can automatically adjust the summary content to fulfill user interests. For instance, it can be based on any pretrained prompt-based summarizer or a large language model such as ChatGPT.When new data comes in:1. The system is triggered by providing it with the process data (a).2. The system generates the descriptive process summary via: a. The object modeling module (1.1) extracts objects from all textual document involved in the process. b. The document modeling module (1.2) transforms the document text in an object- oriented way where each object has a sequence of text in chronological order. c. In the two-phase contradiction detector (1.3), the time-series-based contradiction recognizer (2. 1) recognizes for each object if the text change is a “contradiction” or an “update.” As a result, it outputs a golden text (e) for each object, in which all contradictions and updates are resolved. The time-series-based contradiction recognizer (2.1) resolves recognized contradictions and outputs the contradiction-free result as the “golden text”. In embodiments, the time-series-based contradiction recognizer (2. 1) may further refine time sequential text sequences based on recognition results. For instance, if an age object of a given subject A is detected as an update the module 2. 1 will take the last change as the golden value for inclusion in the golden text. d. The prompt-based summarizer (1.5) generates the abstractive summary of the input golden text. e. In the two-phase contradiction detector (1.3), the fact-based contradiction detector (2.2) detects if the generated summary (b) from the previous step contradicts to the facts stated in the golden text (e). If fact-based contradiction is detected, the system improves the summary by regeneration or contradiction correction.3. Once the quality of the generated summary (b) satisfies predefined conditions (e.g., a quality threshold or no contradictions), the process ends and the system outputs the final process summary (b).

[0054] Embodiments of the present disclosure provide for the following improvements and technical advantages over existing technology:1) Providing enhanced computer functionality in a method to generate a descriptive summary of a continuous process by summarizing multiple time-series-based documents. In particular, the system according to embodiments of the present disclosure is the first that is capable to generate a process-level summary with time-series-based documents.2) Introducing a two-phase contradiction detector that provides an inventive method to detect a contradiction in summarization, and thereby improve the accuracy of the system. The first phase recognizes self-contradiction from the time-series-based text to classify “contradiction” and “update”, which guarantees the consistency of the source data for the summarization. The second phase detects fact-based contradictions by comparing the generatedsummary with the input data source, which guarantees the consistency of the final summary output.

[0055] Retkowski is a survey of existing technology in abstractive text summarization, the most popular summarization topic in the natural language processing (NLP) domain. As shown in the work, most of the advanced summarization models according to existing technology, from a pretrained encoder-decoder model such as BART and T5 to a large language model such as GPT-3 and LLaMA, are focusing on summarizing a single long text. In contrast, embodiments of the present disclosure enable the text summarization for a continuous process by working on multiple time-series-based documents, which overcomes the technical challenges unique to process summarization discussed herein.

[0056] In contrast to Yasunaga et al., which present a graph-based approach for normal multi -document summarization, embodiments of the present disclosure enable to handle time- series-based information by constructing time sequential text sequences and recognizing contradictions / updates on a time-series basis.

[0057] In contrast to Ghods et al., which focus on mining one or a set of patterns which encode the most salient information from the dataset and the data (e.g., the pattern can be a recurring subsequence occurs in the data), the input is not a text. In contrast, embodiments of the present disclosure enable the summarization of time-series-based textual documents.

[0058] In contrast to Laban et al., embodiments of the present disclosure enable to handle contradictions in a two-phase manner to enhance accuracy, which firstly recognizes the self- contradiction is based on time-series text sequences before the summarization, then detects whether there is a fact-based contradiction in the generated summary with respect to input text source, and further also provides methods to resolve detected contradictions.

[0059] Satoko Suzuki and Ichiro Kobayashi, “On-line Summarization of Time-series Documents using a Graph-based Algorithm,” In Proceedings of the 28th Pacific Asia Conference on Language, Information and Computing, pages 470-478, Phuket, Thailand, Department of Linguistics, Chulalongkorn University (2014), which is hereby incorporated by reference herein, present a graph-based algorithm to generate an on-line summarization, which means, given a time line, the model can generate a summary at a point on the time line. In contrast, embodiments of the present disclosure provide to generate an accurate summary from time-series document with a new contradiction detection mechanism that improves accuracy, and thereby trust, security and reliability in the summarization system. As a further difference, embodiments of the present disclosure provide a predictive model rather than a graph-based method.

[0060] Manling Li, Tengfei Ma, Mo Yu, Lingfei Wu, Tian Gao, Heng Ji, and Kathleen McKeown, “Timeline Summarization based on Event Graph Compression via Time-Aware Optimal Transport,” In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6443-6456, Online and Punta Cana, Dominican Republic, Association for Computational Linguistics (2021), which is hereby incorporated by reference herein, also present a graph-based method to generate a timeline summarization from multiple documents, focusing on identifying coherence of events across the timeline to construct a global event picture from a collection of new articles. However, unlike news articles, whose description can be everything, embodiments of the present disclosure can be practically applied to a number of use cases where the coherence of objects among different documents are given, enabling flexibility to use cases in specialized domains, such as healthcare or manufacturing, as the same object name can be maintained consistently as domain-specific terminologies (e.g., disease name, medicine name, component name). Thus, rather than focusing on object linking, embodiments of the present disclosure provide a mechanism for how to handle the value changes of each object across time. As a further difference, embodiments of the present disclosure provide a predictive model rather than a graph-based method.

[0061] Referring to FIG. 6, a processing system 600 can include one or more processors 602, memory 604, one or more input / output devices 606, one or more sensors 608, one or more user interfaces 610, and one or more actuators 612. Processing system 600 can be representative of each computing system disclosed herein.

[0062] Processors 602 can include one or more distinct processors, each having one or more cores. Each of the distinct processors can have the same or different structure. Processors 602 can include one or more central processing units (CPUs), one or more graphics processing units (GPUs), circuitry (e.g., application specific integrated circuits (ASICs)), digital signal processors (DSPs), and the like. Processors 602 can be mounted to a common substrate or to multiple different substrates.

[0063] Processors 602 are configured to perform a certain function, method, or operation (e.g., are configured to provide for performance of a function, method, or operation) at least when one of the one or more of the distinct processors is capable of performing operations embodying the function, method, or operation. Processors 602 can perform operations embodying the function, method, or operation by, for example, executing code (e.g., interpreting scripts) stored on memory 604 and / or trafficking data through one or more ASICs. Processors 602, and thus processing system 600, can be configured to perform, automatically, any and all functions, methods, and operations disclosed herein. Therefore, processing system 600 can beconfigured to implement any of (e.g., all of) the protocols, devices, mechanisms, systems, and methods described herein.

[0064] For example, when the present disclosure states that a method or device performs task “X” (or that task “X” is performed), such a statement should be understood to disclose that processing system 600 can be configured to perform task “X”. Processing system 600 is configured to perform a function, method, or operation at least when processors 602 are configured to do the same.

[0065] Memory 604 can include volatile memory, non-volatile memory, and any other medium capable of storing data. Each of the volatile memory, non-volatile memory, and any other type of memory can include multiple different memory devices, located at multiple distinct locations and each having a different structure. Memory 604 can include remotely hosted (e.g., cloud) storage.

[0066] Examples of memory 604 include a non-transitory computer-readable media such as RAM, ROM, flash memory, EEPROM, any kind of optical storage disk such as a DVD, a Blu- Ray® disc, magnetic storage, holographic storage, a HDD, a SSD, any medium that can be used to store program code in the form of instructions or data structures, and the like. Any and all of the methods, functions, and operations described herein can be fully embodied in the form of tangible and / or non-transitory machine-readable code (e.g., interpretable scripts) saved in memory 604.

[0067] Input-output devices 606 can include any component for trafficking data such as ports, antennas (i.e., transceivers), printed conductive paths, and the like. Input-output devices 606 can enable wired communication via USB®, DisplayPort®, HDMI®, Ethernet, and the like. Input-output devices 606 can enable electronic, optical, magnetic, and holographic, communication with suitable memory 606. Input-output devices 606 can enable wireless communication via WiFi®, Bluetooth®, cellular (e.g., LTE®, CDMA®, GSM®, WiMax®, NFC®), GPS, and the like. Input-output devices 606 can include wired and / or wireless communication pathways.

[0068] Sensors 608 can capture physical measurements of environment and report the same to processors 602. User interface 610 can include displays, physical buttons, speakers, microphones, keyboards, and the like. Actuators 612 can enable processors 602 to control mechanical forces.

[0069] Processing system 600 can be distributed. For example, some components of processing system 600 can reside in a remote hosted network service (e.g., a cloud computing environment) while other components of processing system 600 can reside in a local computing system. Processing system 600 can have a modular design where certain modules include aplurality of the features / functions shown in FIG. 6. For example, I / O modules can include volatile memory and one or more processors. As another example, individual processor modules can include read-only-memory and / or local caches.

[0070] While subject matter of the present disclosure has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive. Any statement made herein characterizing the disclosure is also to be considered illustrative or exemplary and not restrictive as the present disclosure is defined by the claims. It will be understood that changes and modifications may be made, by those of ordinary skill in the art, within the scope of the following claims, which may include any combination of features from different embodiments described above.

[0071] The terms used in the claims should be construed to have the broadest reasonable interpretation consistent with the foregoing description. For example, the use of the article “a” or “the” in introducing an element should not be interpreted as being exclusive of a plurality of elements. Likewise, the recitation of “or” should be interpreted as being inclusive, such that the recitation of “A or B” is not exclusive of “A and B,” unless it is clear from the context or the foregoing description that only one of A and B is intended. Further, the recitation of “at least one of A, B and C” should be interpreted as one or more of a group of elements consisting of A, B and C, and should not be interpreted as requiring at least one of each of the listed elements A, B and C, regardless of whether A, B and C are related as categories or otherwise. Moreover, the recitation of “A, B and / or C” or “at least one of A, B or C” should be interpreted as including any singular entity from the listed elements, e.g., A, any subset from the listed elements, e.g., A and B, or the entire list of elements A, B and C.

Claims

CLAIMSWhat is claimed is:

1. A computer-implemented method for text summarization of text documents using a machine learning process, the computer-implemented method comprising: extracting objects from each of the text documents, each of the objects being associated with a time factor; transforming each of the objects to an object oriented data structure that includes timeseries based text pieces sorted by the time factor from the text documents; generating an intermediate output for each of the objects by identifying changes in the time-series based text pieces and labeling the changes as contradictions or updates, wherein the contradictions and updates are resolved in the intermediate output; generating a text summary of the text documents based on a received prompt and the intermediate output; detecting one or more fact-based contradictions in the text summary based on comparing the text summary to the intermediate output; and generating an updated text summary based on the one or more fact-based contradictions detected in the text summary.

2. The computer-implemented method according to claim 1, further comprising detecting fact-based contradictions in the updated text summary based on comparing the updated text summary to the intermediate output.

3. The computer-implemented method according to claims 1 or 2, wherein detecting the fact-based contradictions in the updated text summary and generating the updated text summary steps are iteratively repeated until the updated text summary meets a predefined condition, and wherein the predefined condition includes no contradictions detected in the updated text summary or a threshold of contradictions present in the updated text summary.

4. The computer-implemented method according to any of the preceding claims, wherein identifying the changes and labeling the changes is implemented by a neural network model, wherein the neural network model is a transformer-based language model, and wherein generating the intermediate output further comprises extracting subjects from the time-series based text pieces, and wherein extracting the subjects is implemented by a neural network language model or a dependency-based information extractor.

5. The computer-implemented method according to any of the preceding claims, wherein generating the updated text summary includes correcting the detected one or more fact-based contradictions in the text summary using the intermediate output or re-generating the text summary using the detected one or more fact-based contradictions in the text summary.

6. The computer-implemented method according to any of the preceding claims, further comprising, prior to detecting the one or more fact-based contradictions in the text summary, clustering the objects of each text document based on textual similarity of the objects by computing a vectorized semantic embedding of the time-series based text pieces of the objects and using at least one of cosine similarity, Euclidean distance, or Jaccard similarity to evaluate a similarity of text embeddings of the vectorized semantic embedding of the time-series based text pieces.

7. The computer-implemented method according to claim 6, wherein generating the updated text summary is further based on using the clustered objects of each text document.

8. The computer-implemented method according to any of the preceding claims, wherein identifying the changes and labeling the changes is implemented by a neural network model, wherein the neural network model is trained using a dataset having a subject, text pair, contradiction label, and the time factor, wherein detecting the one or more fact-based contradictions includes using a fact-based contradiction detector that is trained using the dataset and data including the text pair and the contradiction label, wherein the time factor includes a value for the subject that changes at a different time factor.

9. The computer-implemented method according to any of the preceding claims, wherein identifying the changes and labeling the changes is implemented by a neural network model, wherein the neural network model has been trained by: sorting the time-series based text pieces in chronological order for a subject; and feeding the subject and the sorted time-series based text pieces into the neural network model such that the neural network model learns to identify text changes as a contradiction or an update by learning how the text changes are distributed in the sorted time-series based text pieces, how frequency the text changes occur, and a magnitude of the text changes.

10. The computer-implemented method according to any of the preceding claims, wherein identifying the changes and labeling the changes is implemented by a neural network model, wherein the neural network model has been trained by: sorting the time-series based text pieces in chronological order for a subject; andfeeding, for each time factor, the subject, historical text sequences, and a previous identification of a change in the sorted time-series based text pieces such that the neural network model learns to identify text changes as a contradiction or an update in a recurrent manner from a historical contradiction distribution and a historical update distribution.

11. The computer-implemented method according to any of the preceding claims, wherein extracting the objects includes using a pre-trained neural network model that identifies the objects from each of the text documents, the pre-trained neural network model including a name entity recognition model.

12. The computer-implemented method according to claim 11, wherein the pre-trained neural network model is configured to extract the time factor based on the text including a date or a timestamp, or based on a creation date of a respective text document.

13. The computer-implemented method according to any of the preceding claims, wherein the text documents are health records from patients and treatment documents from doctors, and wherein the updated text summary includes a long-term disease diagnosis based on the received prompt corresponding to a summary of a treatment process for a particular patient of the patients.

14. A computer system for text summarization of text documents using a machine learning process, the computer system comprising one or more hardware processors which, alone or in combination, are configured to provide for execution of the following steps: extracting objects from each of the text documents, each of the objects being associated with a time factor; transforming each of the objects to an object oriented data structure that includes timeseries based text pieces sorted by the time factor from the text documents; generating an intermediate output for each of the objects by identifying changes in the time-series based text pieces and labeling the changes as contradictions or updates, wherein the contradictions and updates are resolved in the intermediate output; generating a text summary of the text documents based on a received prompt and the intermediate output; detecting one or more fact-based contradictions in the text summary based on comparing the text summary to the intermediate output; and generating an updated text summary based on the one or more fact-based contradictions detected in the text summary.

15. A tangible, non-transitory computer-readable medium having instructions thereon which, upon being executed by one or more processors, provide for text summarization of text documents using a machine learning process by execution of the following steps: extracting objects from each of the text documents, each of the objects being associated with a time factor; transforming each of the objects to an object oriented data structure that includes timeseries based text pieces sorted by the time factor from the text documents; generating an intermediate output for each of the objects by identifying changes in the time-series based text pieces and labeling the changes as contradictions or updates, wherein the contradictions and updates are resolved in the intermediate output; generating a text summary of the text documents based on a received prompt and the intermediate output; detecting one or more fact-based contradictions in the text summary based on comparing the text summary to the intermediate output; and generating an updated text summary based on the one or more fact-based contradictions detected in the text summary.

Images