SYSTEM AND METHOD FOR ANALYZING AND IDENTIFYING RELATIONSHIPS FROM DIFFERENT DATA SOURCES - Patent application

Abstract

The present invention relates to a computer-implemented system and method for analyzing data from various data sources. Embodiments of the system and method further provide for generating answers to specific questions based on the analyzed data, where generating includes retrieving relevant documents associated with the analyzed data, determining what information should be reported from which of the retrieved relevant documents, and providing an answer based on the determination and a graph schema associated with the relevant documents.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is a continuation-in-part of U.S. application Ser. No. 16 / 159,088, filed Oct. 12, 2018, which claims the benefit of the filing date of U.S. Provisional Application No. 62 / 572,266, filed Oct. 13, 2017, which is incorporated herein by reference in its entirety.

[0002] The present invention relates to systems and methods for analyzing data from a variety of data sources and generating answers to specific questions based on the analyzed data. [Background technology]

[0003]

[0003] The digitization of work continues to progress as advances in machine learning, natural language processing, data analytics, mobile computing, and cloud computing are used in various combinations to replace some processes and functions. Basic process automation can be implemented without significant IT investments, as solutions can be designed, tested, and implemented at relatively low cost. Enhanced process automation incorporates more advanced techniques that allow the use of data to support elements of machine learning. Machine learning tools can be used to discover naturally occurring patterns in data and predict outcomes. And natural language processing tools are used to analyze text in context and extract the desired information.

[0004]

[0004] However, such digital tools are typically found in a variety of formats and coding languages, and are therefore difficult to integrate and are often not customized. As a result, such systems are unable to provide automated solutions or answers to specific questions that require analysis and process various types of input data, e.g., structured, semi-structured, unstructured, as well as images and audio. For example, such systems currently cannot efficiently address questions such as "Which of these 500 contracts fail to comply with the new banking regulation XYZ?" Summary of the Invention [Problem to be solved by the invention]

[0005]

[0005] It would therefore be desirable to have a system and method that could overcome the aforementioned shortcomings of known systems and that could apply automated and customized analysis to analyze documents, correspondence, text files, websites, and other structured and unstructured input files to generate output in the form of answers to specific questions and other supporting information. [Means for solving the problem]

[0006] According to one embodiment, the present invention relates to a computer-implemented system and method for analyzing data from various data sources. The method may include receiving data from the various data sources as input, converting the received data from each of the various data sources into a common data structure, identifying keywords in the received data, generating sentence or word embeddings based on a document corpus, receiving a selection of one or more labels based on the generated sentence or word embeddings, adding the selected one or more labels to a model, training the model over the common data structure based on a configuration file, and generating results responsive to a user's question based on the model, the generating step including searching for relevant documents from the received data, determining which information should be reported from which of the searched relevant documents, and providing the results based on the determination and a graph schema associated with the relevant documents.

[0007] The present invention also relates to a computer-implemented system for analyzing data from various data sources.

[0008]

[0008] The exemplary document management workflow seamlessly integrates the key tasks for document ingestion, prediction, integration, and analysis. The workflow enables users to answer specific questions about documents (e.g., contracts) and model relationships with other documents to build a knowledge base. Specifically, each step (e.g., ingestion, prediction, integration, and analysis) is integrated into an end-to-end workflow that is configurable with minimal effort or modification required by the user. Each step builds on the previous step to enable analysis and extraction of information from documents. In this regard, other document management frameworks typically require a significant amount of "glue code" (e.g., code custom-made for a specific project) to put together the entire workflow. In contrast, in the present invention, users can configure each step to create an exemplary process that is easily reusable for various projects without the need to rewrite code.

[0009]

[0009] Furthermore, according to one embodiment, the exemplary workflow can address various types of document analysis problems, for example, by mapping the process to specific problems / use cases. The problems can originate from various domains, for example, clause / regulatory compliance, procurement contracts, commercial leakage, contract risk analysis, etc. Furthermore, the exemplary framework is flexible, allowing users to customize business logic rules, post-processing, and quality assessment tasks and adapt them to the business use cases and specific needs of a particular user. In other words, the exemplary process is to fit the document analysis problem rather than trying to fit the problem into a standard, inflexible framework. Furthermore, each part of the exemplary workflow (e.g., document processing, feature generation, model architecture, quality assessment, post-processing, and contract integration) can be associated with a default configuration that can cover many specific problems. However, these default configurations can be easily modified to address new or unique questions. In addition, the Lume data structure persists data and metadata throughout the exemplary process, thereby enabling a unified learning model. Moreover, because the process is fully integrated, documents (e.g., contracts) and their corresponding companion documents can be processed to extract knowledge and answer specific questions about content within the documents. Furthermore, the exemplary process can use a graph-based reasoning framework to decompose information across multiple documents. For example, the business logic layer can enable subject matter experts to specify how document families are joined. Furthermore, the graph-based reasoning framework can specify disposition of conflicting clauses. Additionally, reasoning can also be done at the document family level or at the individual document level.

[0010]

[0010] Furthermore, the present invention also provides interchangeable model architectures, which can be switched to find the best model framework for extracting a particular clause with minimal human interaction. Framework-specific language can be included in the default or customized configuration. Furthermore, framework-specific features can be made available via a knowledge base. In addition, highly effective default options for a particular problem minimize user configuration. Furthermore, the interchangeable model architecture provides support for sequence labeling, classification, and deep learning models that can be replaced via customized configuration files that can be used by non-experts.

[0011]

[0011] Furthermore, according to one embodiment, the present invention allows subject matter expertise to be encoded into the overall solution. For example, the present invention can digitize the completion of complex manual tasks to enhance the output of machine learning. Furthermore, post-processing may be applied to clean or reformat the high-confidence answers based on customer specifications. In addition, post-processing can also leverage subject matter expertise to generate downstream answers to questions that rely on multiple pieces of information from documents. Furthermore, a quality assessment step is added to ensure that the high-confidence answers comply with customer specifications.

[0012]

[0012] Furthermore, according to one embodiment, the present invention also provides for the development of rich, high-quality training and testing datasets. For example, the present invention provides curated subject matter expertise in labeling data. Furthermore, the present invention leverages search, text similarity, and clustering techniques to obtain labeled, representative, and diverse datasets that are more efficient and effective in producing high-performance models. In addition, the datasets can also incorporate information from framework-specific knowledge bases. Furthermore, the present invention also provides for the creation of custom word embeddings to better represent specific domains of interest. Furthermore, at least one of the unified learning model or the active learning model can be leveraged to label specific document information. Finally, the specific models and results from the exemplary framework can be stored in a third-party storage device.

[0013]

[0013] These and other advantages are more fully described in the detailed description that follows.

[0014]

[0014] In order to facilitate a more complete understanding of the present invention, reference is now made to the accompanying drawings, which should not be construed as limiting the invention, but are intended only to illustrate different aspects and embodiments of the invention, in which: [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 is a functional block diagram of an analysis system in accordance with an exemplary embodiment of the present invention. [Diagram 2] FIG. 1 is a diagram of an architecture of an analysis system according to an exemplary embodiment of the present invention. [Diagram 3] FIG. 2 is a diagram of a standard data format for a transformation file, referred to herein as Lume, in accordance with an exemplary embodiment of the present invention. [Figure 4A] 1 is a diagram depicting an example Lume structure and example levels, according to an exemplary embodiment of the present invention. [Figure 4B] FIG. 4B is a close-up view of the document with metadata depicted in FIG. 4A. [Diagram 5] 1 is a diagram depicting a Lume creation process from a Microsoft Word® document, according to an exemplary embodiment of the present invention. [Figure 6] 1 is a diagram depicting a dataset creation process from Microsoft Word and a directory of text files in accordance with an exemplary embodiment of the present invention. [Figure 7] FIG. 2 is a flow diagram for an analysis system in accordance with an exemplary embodiment of the present invention. [Figure 8] FIG. 2 illustrates an example of a document to be captured and analyzed by the analysis system according to an exemplary embodiment of the present invention. [Figure 9] FIG. 11 is a diagram of an example of an expression presented as a tabular expression string according to an exemplary embodiment of the present invention. [Figure 10] 1 is an example of an output from an Intellectual Domain Engine in the form of predicted answers, according to an exemplary embodiment of the present invention. [Figure 11] 11 is an example of output from an intellectual domain engine in the form of support and reasons for an answer, according to an exemplary embodiment of the present invention. [Figure 12] 1 is a system diagram of an analysis system in accordance with an exemplary embodiment of the present invention. [Figure 13] FIG. 2 is a flow diagram for an analysis system in accordance with an exemplary embodiment of the present invention. [Figure 14] FIG. 14 is a flow diagram of the annotation steps depicted in FIG. 13 according to an exemplary embodiment of the present invention. [Figure 15A] FIG. 14 is an architecture diagram for the active learning steps depicted in FIG. 13 according to an exemplary embodiment of the present invention. [Figure 15B] FIG. 14 is a workflow diagram for the active learning steps depicted in FIG. 13, in accordance with an exemplary embodiment of the present invention. [Figure 16] FIG. 14 is a diagram of the machine learning steps depicted in FIG. 13 in accordance with an exemplary embodiment of the present invention. [Figure 17] FIG. 14 is a diagram of the integration steps depicted in FIG. 13 in accordance with an exemplary embodiment of the present invention. [Figure 18] FIG. 2 depicts a graph schema representing multiple documents according to an exemplary embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016]

[0035] Exemplary embodiments of the invention will now be described to illustrate various features of the invention. The embodiments described herein are not intended to be limiting on the scope of the invention, but rather to provide examples of the components, uses, and operation of the invention.

[0017]

[0036] According to one embodiment, the present invention relates to an automated system and method for the analysis of structured and unstructured data. The analysis system (sometimes referred to herein as the "system") may include a portfolio of artificial intelligence capabilities, including components of artificial intelligence domain expertise and related technologies. The system may include basic capabilities such as document capture and optical character recognition (OCR), e.g., the ability to capture documents and convert them into a machine-readable format for analysis. According to a preferred embodiment, the system also includes a machine learning component that gives the system the ability to learn without being explicitly programmed (supervised and unsupervised), a deep learning component that models high-level abstractions in the data, and natural language processing (NLP) and generation, e.g., the ability to understand human words or text and generate text or words.

[0018]

[0037] The system can also be designed to ingest and process various types of input data, including structured data (e.g., data organized in columns and rows, such as trading system data and Microsoft Excel® files), semi-structured data (e.g., text that is not stored in a recognized data structure and still contains some type of tabs or formatting, such as forms), unstructured data (e.g., text that is not stored in a recognized data structure, such as contracts, tweets, and policy documents), and images and audio (e.g., photographs or other visual depictions of physical objects and human audio data).

[0019]

[0038] The system can be deployed to ingest, understand, and analyze documents, correspondence, and websites that create a rapidly growing body of structured and unstructured data. According to one embodiment, the system may be designed to (a) read scripts, tax returns, correspondence, accounting reports, and similar documents and input files, (b) extract information and ingest the information into structured files, (c) evaluate the information in the context of policies, rules, regulations, and / or business objectives, and (d) answer questions, provide insights, and identify patterns and exceptions within the information. The system captures and stores subject matter expertise, ingests, extracts, and classifies documents using natural language processing (NLP), incorporates advanced machine learning and artificial intelligence methods, and utilizes collaborative, iterative refinement with advisory and client stakeholders.

[0020]

[0039] Examples of questions the system can answer may include, for example, which documents comply with a particular policy or regulation, which assets are most at risk, which claims warrant intervention, which counterparties are likely / unlikely to undergo contraction, which clients have growing / shrinking financial resources and market share, and which documents are experiencing trends or changes in meaning. Examples of policies or rules the system can analyze may include, for example, new regulations, accounting standards, revenue targets, identifying expanding versus dilutive projects, credit risk assessments, asset selection, portfolio rebalancing, or settlement outcomes, to name a few. Examples of documents the system can analyze may include, for example, legal contracts, loan documents, securities prospectuses, corporate financial statements, derivative endorsements and principals, insurance policies, insurance claims, customer service records, and email exchanges.

[0021]

[0040] FIG. 1 is a functional block diagram of a system for automated analysis of structured and unstructured data, according to an exemplary embodiment of the present invention. As shown in FIG. 1, the system integrates various data sources, domain knowledge, and human interaction, in addition to algorithms that ingest and structure the content. The system includes a scanning component 10 that ingests multiple documents 5, such as contracts, loan documents, and / or text files, and extracts relevant data 6. During the ingest process, the system can incorporate OCR technology that converts images (e.g., PDF images) into searchable text, and NLP pre-processing that converts scanned images into raw documents 11 and essential content 12. In addition, appropriate ingest techniques are used to convert and preserve document metadata and formatting information. In many instances, the input unstructured data resides in multiple documents that together form a corpus 15 of documents stored in a dataset.

[0022]

[0041] The example of FIG. 1 depicts a "control rule set" that is implemented in a particular business situation. One example of a control rule set may be a new or modified financial regulation, and a financial institution or company may need to ensure that its contracts comply with the new regulation. Manual review of contracts to evaluate compliance with the new regulation is one alternative, but that approach may require significant time and significant costs for experts to review the contracts. Alternatively, a system may be configured to read contracts, extract information, populate a structured file, evaluate the information in the context of the modified regulation, and / or business objectives, answer questions, provide insights, and identify patterns and exceptions within the contracts. An exemplary embodiment of the present invention may thus automate the analysis of complex documents, which may provide the benefits of enabling 100% coverage over traditional sampling approaches, reducing the cost and development time required to provide insights, enabling humans to achieve and manage accurate consistency, leveraging the knowledge and expertise of subject matter experts (SMEs), and automatically creating audit logs that describe how the data was processed.

[0023]

[0042] Referring to FIG. 1, the control rule set is used by the subject matter expert in manual investigation and is translated into the relevant semantics 21 and decision strategy 22 in manual investigation. The semantics 21 comprises domain knowledge embodied in an ontology or knowledge base composed of entities, relationships, and facts. The decision strategy 22 comprises business rules applied to the relevant semantics 21 to answer specific questions. This includes document level assessment (such as compliance vs. non-compliance), feature level extraction (end date, key entities), inferred facts (such as making inferences utilizing extracted facts and ontologies), or identifying risks (such as identifying parts of the document that require further scrutiny). The machine learning investigation 25a analyzes the features 26a that determine the direction of specified contract clauses, dates, entities, facts, etc., and undertakes the automated document analysis evaluation 27a using an intelligent domain engine (sometimes referred to herein as an "IDE"). The machine learning investigation 25a assists the machine compliance determination 28a by providing a confidence score. In parallel, manual review 25b of selected documents, for example, performed by a subject matter expert, analyzes directional features 26b and undertakes document analysis evaluation 27b and manual compliance determination 28b on the sample of contracts. The parallel manual and machine evaluations are used to determine accuracy and confidence scores 29, which are then used as feedback 30 for the manual and machine reviews. The feedback 30 allows for refinement of the machine review, such that each iteration can result in improved accuracy in the automated analysis and a corresponding increase in confidence score. To reduce the number of iterations required to achieve a given accuracy, active learning methods are used.

[0024]

[0043] Referring to Figure 2, the architecture of a system according to an exemplary embodiment of the present invention is depicted. As previously described, the system can support information extraction and data analysis on structured and unstructured data. Input data 210 can take the form of various files or information of different types and formats, such as documents, text, video, audio, tables, and databases. As shown in Figure 2, the data to be analyzed can be input into a core document management system 220.

[0025]

[0044] According to a preferred embodiment of the present invention, input data 210 is converted into a common data format 230, called "Lume" in FIG. 2. Lume may preferably be the common format for all components and data storage. As shown in FIG. 2, the core document management system includes a document conversion system 240 (which converts documents into Lume format 230) and a document and corpus repository 220. The document conversion system provides utilities to extract document data and metadata and store it in a format 240 used to perform natural language processing. The standardized Lume format 230 makes it easy to process and analyze data in Lume, since a large number of components can then be easily applied to Lume and utilize upstream information for extended processing. In one application, processing workflows can be strung together to identify sentences, tokens, and annotations to other document structures, entity identities, taxonomies, and ontologies, and an intelligent domain engine 251 can utilize this information to create derived and inferred features. Each of these components utilizes Lume 240 as an input and Lume 240 as an output, and metadata can be incrementally inserted into Lume. Other examples of components may include, for example, different engines, a natural language processing (NLP) component 255, an indexing component, as well as other types of components (e.g., optical character recognition (OCR) 252, machine learning 253, and image processing 254).

[0026]

[0045] Component 250 reads Lume 240 and generates Lume elements. The Lume elements are then stored in a stand-off annotation format (described by database 220, parent class definitions in base data format 230, and specific instances of the format in application specific data format 240). As an example, NLP component 255 processes Lume 240 and adds additional Lume elements to indicate human language specific constructions in the underlying data, including word tokens, parts of speech, semantic role labels, named entities, coreferences, etc. These elements can be indexed to provide users with the ability to quickly search for sets (or single) Lume 240 or Lume elements via a query language.

[0027]

[0046] The Lume technology is further described below with reference to Figures 3-6.

[0028]

[0047] FIG. 2 also shows that several machine learning (ML) components 253 can be incorporated into the system. For example, the system may include an ML transformation component, a classification component, a clustering component, and a deep learning component. The ML transformation component converts the basic Lume representation into machine-readable vectors for rapid analytical processing. The classification component maps a given set of inputs to a learned set of outputs (categories or numbers) based on initial training and configuration. The clustering component generates groups of vectors based on a pre-determined similarity criterion. The deep learning component is a unique type of machine learning component 253 that utilizes a multi-layer network representation of nodes and connections to learn the output (category or number).

[0029]

[0048] FIG. 2 shows that the system may include several user interfaces 270 that allow different types of users to interact with the system. An IDE manager 273 allows users to modify, delete, and add expressions to the system. A model manager 274 allows users to select machine learning models for execution in the pipeline. A search interface 272 (i.e., data exploration) allows users to find data loaded within the platform. A document and corpus annotator 271 (i.e., annotation manager) and editor allows users to manually create and modify annotations on Lume and group Lume into corpora for training and testing the system. A visual workflow interface 275 (i.e., workbench) provides visual capabilities for building workflows and can be used to create histograms and other statistical charts of data stored within the platform.

[0030]

[0049] Figure 3 illustrates properties and features of Lume, in accordance with an exemplary embodiment of the present invention. As shown in Figure 3, "name" is a string containing a non-qualified name of the document. "data" is a string or binary representation of the document (e.g., serialized data representing the original data). "elements" is an array of Lume elements.

[0031]

[0050] As shown in FIG. 3, each Lume element includes an element ID and an element type. According to a preferred embodiment of the present invention, only the element ID and element type are required to define and create a Lume element. The element ID is a string that contains a unique identifier for the element. The element type is a string that identifies the type of the Lume element. Examples of types of Lume elements include parts of speech (POS), such as noun, verb, adjective, and named entity recognition (NER), such as person, place, and organization. Additionally, file path and file type information can be stored as elements. The file path is a string that contains the full source file path of the document. The file type is a string that contains the file type of the original document.

[0032]

[0051] Although not required, a Lume element may also include one or more attributes. An attribute is an object that is composed of key-value pairs. An example of a key-value pair may be, for example, {"name":"Wilbur", "age":27}. This creates a simple yet powerful format that allows developer flexibility. The reason that only the ID and type of the element are required according to an exemplary embodiment of the present invention is that it provides developers with the flexibility to store information about Lume within the element while also ensuring that the element is accessible by ID or type. This flexibility allows users to determine how they want to store the relationships and hierarchies between elements according to their domain expertise. For example, an element can contain the necessary information about complex language structures, store relationships between elements, or reference other elements.

[0033]

[0052] According to an exemplary embodiment of the present invention, Lume elements are used to store a stand-off annotation format, i.e., the elements are stored as annotations separate from the document text, rather than embedded within the text. This embodiment allows the system to recover the original data without modifying it.

[0034]

[0053] According to a preferred embodiment, Lume elements are not stored in a hierarchical relationship with other Lume elements, and document data and metadata are stored in a non-hierarchical manner. Well-known formats (other than Lume) are hierarchical, making them difficult to manipulate and transform. Lume's non-hierarchical format allows easy access to any element of the document data or its metadata, either at the document level or at the text level. In addition, editing, adding, or parsing the data structure can be done via operations on the elements, without the need for conflict resolution, hierarchy management, or other operations that may or may not be required for the application. According to this embodiment, since it is a stand-off annotation format, the system can store exact copies of the original data and support overlapping annotations. In addition, this allows annotations of multiple formats, such as audio, images, and videos.

[0035]

[0054] Lume technology can provide a universal format for document data and metadata. Once a Lume is created, it can be used in each tool of the natural language processing pipeline without the need for write format transformations to be built into the tool in the pipeline. This is because the Lume format establishes the basic transformations that need to be passed to the data and metadata. The system provides utilities for extracting document data and metadata from several formats, including plain text and Microsoft Word. Format-specific parsers convert data and metadata from these formats into Lume and write correspondingly modified Lume back to the format. The system can use Lume technology to store information related to families of words to prepare them for natural language processing, such as preprocessing and stemming. In addition, the system can use Lume technology to store information related to relationships and graph structures within documents.

[0036]

[0055] According to an exemplary embodiment of the present invention, the system includes other components in addition to Lume and Lume elements. Specifically, the system may be configured to include a dataset, a Lume data frame, an Ignite component, and an element index. A dataset is a collection of Lume objects with a unique identifier. A dataset is typically used to specify training and testing sets for machine learning, and can be used to perform large volume operations on many documents. A Lume data frame is a dedicated matrix representation of Lume. Many machine learning and arithmetic components in the system can leverage this optimized format. The system may also include an Ignite component that reads Lume (or Lume corpus) data and returns Lume (or Lume corpus) data, typically by processing existing Lume elements or original source data and adding new Lume element objects. An element index is a computer object representation of a set or element, as well as a representation typically leveraged within Ignite for efficiency in searching Lume data and metadata. For example, some components may be optimized to reshape character offsets, and thus indexes on character offsets can accelerate operations on those components.

[0037]

[0056] In accordance with an exemplary embodiment of the present invention, the main functions of the system include data representation, data modeling, discovery and composition, and service interoperability, which are described as follows.

[0038]

[0057] Data Representation: Lume is a common data format used to store and communicate analyses on the system. Lume takes a stand-off approach to data representation, e.g., the results of an analysis are stored as annotations independent of the original data. According to one embodiment, Lume is implemented in Python, has computer object representation as Python objects, and is serialized as JavaScript Object Symbols ("JSON") for inter-process communication. Lume may be designed for use with web-based specifications such as JSON, Swagger (YAML), RESTful, etc., and interfaces with the Python ecosystem, but it can also be implemented in and support components written in Java and other languages.

[0039]

[0058] Data Modeling: Lume can be designed to be simple and only enforce basic requirements on users of the system. Rather than requiring a declarative representation of both data and processes, interpretation and business logic are left to users of the system. The system can be designed to leave the modeling informal and leave the implementation details in the processing components. This allows Lume to maintain a very simple specification and allows Lume to be extended for specific applications without interfering with other applications. For example, when it is important to search Lume, it is integrated with a module that indexes on top of the Lume structure. When it is important to work with a Document Object Model (DOM), the DOM parser stores additional information in Lume in the form of Lume elements and attributes and translates this information back into the DOM model.

[0040]

[0059] Discovery and composition: Lume may also have further design features regarding provenance of the analysis process. System workflows may require provenance information to facilitate repeatability and discovery of components. This provenance information is stored in Lume and can be enriched via provenance enrichment workflows. For example, this may implement checks on each output Lume to ensure that the correct processing steps have been completed. In the validation phase, it may provide a means to track the provenance of Lume elements that created correct or incorrect metadata. Additionally, it may also track to ensure that all inputs have been received as outputs.

[0041]

[0060] Service Interoperability: Services provided by the system may require a Swagger (YAML markup language) specification, according to one embodiment of the present invention. There may be many assumptions about the business logic, operation order, and other data interpretations utilized to implement the system components. Identifying which components are interoperable may be achieved through analysis of example workflows rather than a specification of inputs and outputs. In the system, components can simply operate against Lume and, in case of errors, return the correct error code and write appropriate logging information.

[0042]

[0061] 4A shows an example of a Lume structure and the initial conversion of different types of files into Lume. As shown in FIG. 4A, a dataset 410 refers to a body of different types of files or documents. These documents may initially be in different formats, such as, for example, Adobe® Portable Document Format (PDF), unstructured text files, Microsoft Word® files, and HTML files.

[0043]

[0062] Figure 4A also shows an example of defined elements for Lume. For example, a first element 411 can correspond to a study director with contact information, a second element can correspond to a protocol manager with contact information 412, a third element can correspond to a contract research organization (CRO) with contact information 413, a fourth element can correspond to a research and development company 414, and a fifth element 415 can correspond to a confidentiality notice for the document. Figure 4B shows an expanded view of a document with metadata depicted in Figure 4A.

[0044]

[0063] Exemplary levels of element types are also shown in Figure 4A. For example, the system may implement functionality that allows a user to identify individual paragraphs, tokens, or entities, each of which may be extracted from Lume.

[0045]

[0064] FIG. 5 provides further details of an example of Lume creation from a Microsoft Word document. As shown in FIG. 5, the first step, step 501, is to initialize the original document. Initialization involves storing the original data in a Lume object. The second step, step 502, is to parse the document into Lume formatted elements. This step may include a loop 502a in which elements corresponding to metadata from the source document are created. This is performed by a document-specific component that captures the native format. Specifically, during the capture, (i) the original file is opened, (ii) the DOCX format is unpacked into an XML file, and then (iii) the XML file is read into a data structure for parsing. Parsing separates the data in the document from the metadata, then stores the data in the Lume "data" field and the metadata in the Lume element. This is then output as Lume text. Examples of metadata stored are author, page, paragraph, and font information.

[0046]

[0065] Upon completion of the process illustrated in FIG. 5, the input document has been converted to Lume and the desired elements have been generated and stored.

[0047]

[0066] FIG. 6 illustrates an example of applying the functionality of FIG. 5 to a corpus of documents. The first step in FIG. 6, i.e., step 601, involves initializing a dataset. The next step in FIG. 6 involves applying the process illustrated in FIG. 5 to each document in the dataset. As the Lumes in the dataset are converted to Lume format in step 602, the results are stored in the dataset. The conversion includes creating a Lume data structure (i.e., loop 602b), converting format-specific metadata into Lume elements (i.e., step 602a), and any additional annotations required, such as semantic annotation, natural language processing, creating domain-specific features, or vectorization into quantitative fingerprints. More specifically, in step 601, documents in the dataset are identified in their URLs, and then Lumes containing the file data are passed to 602. Next, in 602b, the Lumes are passed to an appropriate parser, which creates a data structure suitable for parsing. In 602a, the parser works through the documents, parsing the data into Lume "data" fields and the metadata into Lume elements. This is then output as Lume Text.

[0048]

[0067] FIG. 7 is a process diagram illustrating an example of a process for analyzing structured and unstructured data according to an exemplary embodiment of the present invention. In step 710, a document, such as a text, Microsoft Word®, and / or Adobe® PDF document, is brought into the system. Then, in step 712, the document is converted to Lume format, as described above. In step 714, an OCR process may be used to convert image files to characters. In step 716, the document is collected into a dataset. In step 718, the system identifies and annotates structural Lume elements (see, for example, FIG. 6). Once the document has been converted to Lume format and the Lume elements have been generated, in step 720, natural language processing (NLP) routines or components can be applied to the Lume formatted information.

[0049]

[0068] In step 722, a user of the system creates and inputs an ontology that includes a list of entities. According to one example, the ontology can describe people and which businesses they are employees of. The ontology can, for example, help extract people and businesses from documents in the platform. Alternatively, the ontology can describe different products of a company, the categories they belong to, and any dependencies between them. Step 724 includes entity decomposition and semantic annotation. Entity decomposition determines which entities referenced in the data are in fact the same real-world entity. This decomposition is accomplished using the extracted data, the ontology, and further machine learning models. Semantic annotation relates terms in the data to formally defined concepts defined on the ontology. In the business employee example above, occurrences of the word "John Doe" are identified and connected in the ontology with employee John Doe. This allows downstream components to utilize additional information about John Doe, for example, his title and job within the company.

[0050]

[0069] In step 726, a user of the system creates representations to be applied to documents stored in the dataset. Representations may be, for example, comma-separated value (CSV) files that specify patterns to search for or other distinguishing features of documents. Representations may incorporate the expertise and know-how of subject matter experts. For example, a representation may identify various unique words and relationships between words, or patterns that identify a particular contract clause or clause within a tax document. These representations are used to search for and identify specific aspects, clauses, or other identifying features of the document. Representations may also leverage machine learning operators, pre-trained sequence labeling components, or algorithmic parsers that serve as one of the operators to the IDE.

[0051]

[0070] In step 728, the expressions are input into the IDE, which reads the expressions and applies them to the data set. According to one embodiment, the output may include a predicted answer, as well as support and reasons for the answer. The IDE is further described below in connection with Figures 8-12.

[0052]

[0071] In step 730, the output of the IDE can be used to engineer further features. This utilizes previously created Lume elements and creates new Lume elements corresponding to further features. This feature engineering can be thought of abstractly as indicator functions across a set of Lume elements to create unique signal-related features for learning and inference tasks. In the general case, feature engineering can generate further categorical or descriptive text features required for sequence labeling or sequence learning tasks. For example, engineering can prepare features for custom entity tagging, identify relationships, or target a subset of elements for downstream learning.

[0053]

[0072] In step 732, machine learning algorithms or routines are applied to generate results from the Lume elements created upstream. Machine learning can also be replaced by sequence labeling or Bayesian network analysis. This produces machine learning scores, or accuracy of previous annotations, probabilistic information about relationships between elements, or even metadata for new annotations or classifications. In step 734, the results are analyzed, where they are provided to an analyst for review, either via a UI to inspect the annotations, or via a workbench to perform further analysis on the results. In step 736, one or more iterations are performed to improve the prediction accuracy. The steps of applying representations 728, designing features 730, applying machine learning 732, and investigating results 734 may be repeated to improve accuracy. Once accuracy is improved to achieve a desired level, the results may be stored in a database in step 738. It should be noted that entity and semantic decomposition 724, feature design 730, and machine learning 734 are also utilized within the intelligent domain engine, but are separated for large-scale processing pipelines.

[0054]

[0073] According to an exemplary embodiment of the present invention, an IDE comprises a platform for systematically classifying and analyzing a corpus of documents leveraging natural language processing, custom-built annotation components, and manually coded representations. The IDE can provide a platform for combining a company's cognitive / AI capabilities with industry domain knowledge. The classification of each document can be represented by a set of representations, which may include features to be utilized, patterns of features to be identified, and reference location or range information that focuses the classification task. Representations can be composed of and work with Lume elements and data contained in Lume. The IDE can be designed to generate specified results as well as annotated text that supports classification decisions to systematically evaluate the representations for each document in the corpus. In this example, the IDE is utilized for natural language processing and text mining, but it should be noted that the IDE framework applies to all Lume formats such as images, audio, and video.

[0055]

[0074] IDEs can provide several advantages. For example, IDEs can output text that is annotated to support classification decisions in addition to answers to specific questions. Annotations can be used to audit results and provide transparency. Additionally, training accurate machine learning models typically requires a large number of labeled documents. Integrating domain knowledge with machine learning using IDEs can reduce the number of documents required to train accurate models by an order of magnitude by utilizing expert-derived features. This is because machine learning problems involving unstructured data are globally overdetermined, and the ability to select accurate and interpretable features requires more data than is generally available. For example, in a document, there can be tens of thousands of features, including a dictionary of words, orthographic features, document structure, syntactic features, and semantic features. Furthermore, according to an exemplary embodiment of the present invention, the representations can be created using a domain-specific language that can be codified in a non-code environment, such as in a spreadsheet (CSV or XLSX) or via an IDE user interface, so that an individual, such as a subject matter expert (SME), who enters the representations does not need computer coding skills. The SME can thereby create domain-relevant features that can be leveraged in the machine training process. The IDE UI allows users to modify, remove, and add representations to the system and visualize the elements created by running the IDE. Additionally, representations can be designed to be interchangeable. They can be created for reuse in use cases across industries or problem sets. Furthermore, the IDE can be designed to leverage the Lume format for storing and working with documents. This design allows annotations and metadata to be input to the representations in addition to the text features present in the documents.

[0056]

[0075] According to an exemplary embodiment of the present invention, the process for creating and using representations includes (1) manually examining documents, (2) creating custom-built code that can capture patterns through the representations and leverage machine learning or statistical extraction, (3) loading the representations into an IDE and running the IDE, (4) building a confusion matrix and accuracy statistics (i.e., by comparing current results to an unseen set of documents, this creates an estimate of how well the representation generalizes and determines whether the system meets performance requirements), (5) iterating and refining the aforementioned steps, and (6) generating output such as predicted answers and a section providing support and reasons for the answers.

[0057]

[0076] According to one particular example, the IDE may be used to automatically determine answers to legal questions by analyzing documents such as investment management agreements or other legal documents. For illustrative purposes, in this particular example, assume that a company has eight legal questions to answer related to 500 investment management agreements. An example question may be, "Does the agreement require notice related to the identified personnel changes?" FIG. 8 depicts one example of a section of an investment management agreement related to legal questions.

[0058]

[0077] FIG. 9 illustrates an example of an expression, according to an embodiment of the present invention. As illustrated in FIG. 9, expressions may be detailed in a table format (such as CSV) rather than code. In the example of FIG. 9, each expression has a "name" that may be useful when referencing other expressions. The name may also be used by the output file to create features. Each expression may also include a "scope" to focus and restrict the expression to which it should be applied. The scope itself is evaluated as an expression, and the result is used to restrict the scope of the parent expression. For example, a scope expression may point to a Lume element (either pre-specified in the conversion to Lume format or created by another expression), or it may be the result of an operator that identifies the appropriate clause in a contract. An expression also includes a "string" field, which is where the expression is contained. The string field has a pre-determined syntax. The string field may specify a pattern to look for in the document or in a logical operation. FIG. 9 illustrates an example of a string field.

[0059]

[0078] Expressions may also contain a "condition" field that is used to determine whether a particular expression should be evaluated. This is useful in enabling or disabling expressions for computational efficiency, or to implement control logic to enable or disable certain types of processing.

[0060]

[0079] Expressions may be used to search for patterns within documents, and expressions can encapsulate those patterns. Examples of such patterns include, for example, different ways of expressing filing requirements and personnel changes. For example, there are many words for "employee," such as "key people," "investment team," "professional staff," "senior staff," "senior officers," "portfolio manager," "portfolio managers," "investment managers," "key decision makers," "key employees," and "investment manager." In some cases, case is important. For example, "investment manager" may refer to an employee, but "investment manager" may refer to a client's investment organization. In some cases, the order of words (indicating a master-slave relationship) is important. For example, an investment manager informing a client is not the same as a client informing an investment manager. All of these types of patterns can be encapsulated within expressions. Subject matter experts (SMEs) can encapsulate their know-how within expressions when analyzing certain types of specialized document types.

[0061]

[0080] FIG. 10 shows an example of one form of output from the IDE: expected answers, which includes the answers to each question for each document. For example, as shown in FIG. 10, the output may include a table listing the file names of the input files, answers to four questions that provide decisions regarding the characteristics of the contract. According to one embodiment, there may be many more questions or characteristics that are output from the IDE.

[0062]

[0081] FIG. 11 shows an example of another form of output from the IDE: support and justification for a predicted answer. In FIG. 11, the user interface displays the actual contract language used by the IDE to support and justify its given answer. The actual contract language is presented so that the user can evaluate whether the IDE is correct or not. The system can utilize the information stored in the Lume elements to highlight certain words in the text that specifically form the basis for the answer provided by the IDE. In this way, the IDE allows a human user to easily verify whether the answer is correct. It also facilitates the user's ability to understand any errors and refine the representation to correct such errors.

[0063]

[0082] FIG. 12 is a system diagram of a system according to an exemplary embodiment of the present invention. As shown in FIG. 12, the system may include a server 120 and an associated database 122 along with software and data used to run the system. The system may also include a scanner 126 used to scan and import original documents into the system. The server 120 and database 122 may be used to store the imported documents as well as to store the IDE, Lume, and Lume Elements, and other software and data used by the system. A user 125, such as a subject matter expert (e.g., a tax professional), may access and use the server 120, scanner 126, and database 122 via a personal computing device 124, such as, for example, a laptop computer, desktop computer, or tablet computer.

[0064]

[0083] The system may also be configured to allow one or more customers or other users to access the system. For example, as shown in FIG. 12, a customer 135 may use a personal computing device 134 and a company server 130 to access the server 120 via the network 110. The customer may also transmit customer-specific data stored in a customer database 132 (e.g., a set of contracts to be analyzed) to the system to be analyzed by the server 120 and incorporated into a data set document to be stored in the database 122. The server 120 shown in FIG. 12 may receive other documents, spreadsheets, pdf files, text files, audio files, video files, and other structured and unstructured data from other customers or users, generally represented by servers 140 and 150.

[0065]

[0084] Also shown in Figure 12 is a network 110. The network 110 may include, for example, any one or more of the Internet, an intranet, a local area network (LAN), a wide area network (WAN), an Ethernet connection, a WiFi network, a Global System for Mobile Communications (GSM) link, a cellular network, a Global Positioning System (GPS) link, a satellite communication network, or other networks. Other computing devices, such as servers, desktop computers, laptop computers, and mobile computers, may be operated, for example, by different individuals or groups, and may transmit data, such as contracts or insurance policies, to the server 120 and database 122 via the network 110. In addition, cloud-based architectures, along with containerized or microservices-based architectures, may also be used to deploy the system.

[0066]

[0085] 13 is a flow diagram for an analysis system according to an exemplary embodiment of the present invention. As depicted in the figure, the flow diagram 1300 includes a document ingestion step 1310, a pre-processing step 1320, an annotation step 1330, an ML framework step 1340, a post-processing step 1350, and a multi-document integration step 1360. As a result of these steps, the flow diagram 1300 can provide extracted document knowledge.

[0067]

[0086] According to one embodiment, during step 1310, data is captured (i.e., input) from various data sources, e.g., machine-readable and / or non-machine-readable PDFs, Word® documents, Excel® spreadsheets, images, HTML, etc. In particular, raw data from various data sources is converted to and stored in the same Lume data structure, thereby achieving consistency across different data types.

[0068]

[0087] Furthermore, according to one embodiment, during the pre-processing step 1320, several tasks are performed to enhance downstream modeling steps. For example, optical character recognition (OCR) can be performed to convert text from non-machine-readable PDFs or images to machine-readable text, if necessary. Further Lume elements may be added to incorporate image-related features that can also be exploited downstream. In addition, natural language processing tasks are also performed on the document text. For example, words and sentences in the document text can be tokenized and / or lemmatized. Furthermore, any information such as part of part-of-speech tagging or named entity recognition can also be included during this step to enhance the available information for the next modeling. Custom word embeddings can also be added to token elements, in which word embeddings are retrained across a domain-specific document set and added to tokenized word and / or sentence elements. According to one embodiment, word embeddings may be retrained on a large number of documents, for example more than 50. Furthermore, according to one embodiment, the added word embeddings can simplify annotation and gloss over OCR errors in feature creation and modeling. Furthermore, in situations where documents are compiled within a single file (e.g., a master service agreement and corresponding amendments that are typically stored within a single PDF), it may be necessary to split the file into component documents. In these cases, heuristic or trained models are utilized to split the documents into their component parts. According to one embodiment, document splitting is useful when a unification logic is applied to a set of document families. In these situations, each document needs to be analyzed and considered separately to properly apply the logic to the set of documents. For example, assuming a master service agreement has three amendments, information across these related documents, e.g., payment terms for the agreement, can be consolidated after pre-processing and model predictions are performed. However, only one of the documents, e.g., the most recent amendment, may contain the most suitable information, e.g., payment terms for the agreement.Contract integration can therefore be used to apply logic across a set of documents and extract the most pertinent information.

[0069]

[0088] Additionally, according to one embodiment, during the annotation step 1330, human knowledge and expertise can be incorporated into the process 1300, where SMEs can label unique information within the document. This information can be unique phrases and / or text to extract, or unique clauses and / or paragraphs can be labeled as unique types, e.g., Type A, Type B, etc. According to one embodiment, such SME knowledge can be incorporated in a variety of ways, e.g., in a web or Excel®-based user interface. These annotations can then be added directly to the Lume data structure.

[0070]

[0089] FIG. 14 is a flow diagram of the annotation step 1330. After preprocessing is completed, the Lume data structure is ready for annotation. The data in Lume includes elements that describe the text of the document as well as its words, sentences, etc. The information in Lume is then exploited during the annotation step. Specifically, annotations are added as elements that directly point to the data (e.g., text) contained in Lume. As depicted in the figure, in step 1331, keywords / phrases and representatives of the document language are identified. According to one embodiment, the identification may be performed by the SME via a user interface. Furthermore, the identified keywords / phrases and representatives may be provided to a knowledge base 1334. In addition, the identified keywords / phrases and representatives may also be used to compute embeddings of example sentences as depicted in step 1332. Then, in step 1333, based on the computed embeddings and SME knowledge, custom word embeddings are trained, which may also be provided to the knowledge base 1334. Furthermore, as depicted in the figure, an active learning step may also be performed.

[0071]

[0090] During active learning, strategies are developed to identify and simplify the data annotation and training set creation process. According to one embodiment, active learning can leverage word embeddings, sentence embeddings, and keywords to find possible candidates for text within a broader dataset. Specifically, a set of logical keyword searches as well as several examples of the target text (e.g., exemplary sentences where the target information appears) are input for analysis. For example, when searching for candidates to annotate contract clauses, keywords may include language such as "clause," "term," "year," or "month." Furthermore, embeddings of sentences such as "the contract shall last for a term of 10 years" may be leveraged to find similar contextual language. This particular active learning strategy narrows down the search for annotations that are similar but not identical, with a high probability. The user then explores these results and uses these candidate annotations to directly add labels to the Lume dataset of documents. Furthermore, according to one embodiment, this active learning strategy also helps to balance the training set with rare information, e.g., rare fields. Moreover, active learning allows the generation of diverse annotations, which can be used to develop representative datasets in a simplified manner and stored in Lume together with other metadata. In this way, annotations can be leveraged together with complementary information stored in Lume.

[0072]

[0091] According to one embodiment, certain active learning strategies (e.g., increasing the diversity of the data, improving the informativeness of the model, etc.) can be applied as depicted in the figure. For example, the similarity of the sentence embeddings can be compared to the average. Then, in step 1336, the user can explore the results of the strategy, for example, by confirming or rejecting certain labels. The results are then incorporated into the Lume metadata. Furthermore, the user may also refine the search or annotations, or add new data as needed. The confirmed labels are then added to the model, as depicted by step 1337.

[0073]

[0092] According to one embodiment, the exemplary framework combines both implicit and explicit knowledge transfer in a complementary manner. For example, implicit knowledge transfer, such as feature engineering in the form of IDE representations, is used to support explicit knowledge transfer, i.e., annotation via active learning. In other words, IDE representations can be used to provide active learning algorithms with the ability to supply candidates for SMEs to label / investigate. Furthermore, according to one embodiment, in the process of investigating candidates, the designed features are also updated / improved based on the observations of the SMEs. This cycle (e.g., IDE representation features ("explicit")->investigating candidates ("implicit")->improving features based on observations ("explicit")->investigating more candidates ("implicit")) is repeated until the model meets the expected performance.

[0074]

[0093] 15A and 15B show interactions between components in the active learning step depicted in FIG. 13. According to one embodiment, the active learning step can utilize a user interface 1410, an active learning application programming interface (API) 1420, a database 1430, a module management module 1440, an Ignite platform 1450, and a local platform 1460. The API 1420 communicates with a model management module 1440, which allows a user to run any number of experiments (e.g., varying hyperparameters or feature sets) on a given dataset. Additionally, the API 1420 tracks performance metrics for the unique settings of that experiment. Additionally, the API 1420 can also interact with either the Ignite platform (e.g., a cloud server running Ignite software to execute the workflow) or a local platform (e.g., a local server or personal computing device running Ignite software to execute the workflow) to interpret instructions for active learning. For example, if an SME was attempting to create a model to predict “Supplier Name” from multiple contracts, the SME could indicate to the model, for example, via the user interface 1410, that the Supplier Name may typically be located somewhere around the words “by”, “between”, “contract”, “(stock)”, etc. According to one embodiment, the SME could provide this information to the model in the form of an IDE expression. The active learning strategy then uses the API 1420 to select annotation candidates, e.g., automatic annotations (“Auto Annotations”), that best fit the description of the IDE expression. These candidates can be refined by the SME using the user interface 1410, thus providing the model with implicit knowledge about “Supplier Name”. For example, an initial model, e.g., Model 1 in FIG. 15B, can be trained against the investigated examples (candidates manually confirmed by the user) as well as further auto-annotated examples from the active learning strategy. Model performance can then be evaluated on a test set.According to one embodiment, the manually inspected examples may be retained for future training, while the automatically annotated examples are not propagated through further model iterations. During this candidate inspection process, the SME may refine the IDE representation based on observed results (e.g., removing the word "by" and adding the word "company"). Once this refinement is completed, the active learning strategy of Model 2 may be configured from the refinements of the IDE representation, which may be provided by the SME via the user interface 1410. The user may then manually inspect the examples from this updated active learning strategy. As a first iteration, a new model is trained from both the manually inspected annotations (provided by the SME via the user interface 1410) and the automatic annotations (provided directly by the active learning prediction framework). This results in a new model version (e.g., from Model 1 to Model 2 in FIG. 15B), which is then leveraged within the active learning prediction framework to create and explore new candidates based on these refinements. The cycle continues until the model has enough implicit and explicit knowledge to make predictions with an acceptable level of performance.

[0075]

[0094] According to one embodiment, after the SME annotations are incorporated into the Lume data structure, model training can begin with the ML framework 1340. According to one embodiment, the ML framework 1340 is composed of several components that work together to train or apply algorithms across the Lume data structure. For example, the information extraction component 1349 serves as an interactive layer with the machine learning component 1346. Furthermore, according to one embodiment, a user can create a configuration file 1341 that can be interpreted by the information extraction component 1349 before sending instructions to the machine learning component 1346. According to one embodiment, the instructions in the configuration file 1341 include the task type (e.g., training, validation, prediction, etc.), the algorithm type and package (e.g., regression algorithms such as Sklearn logistic regression, recursive algorithms such as keras LSTM, etc.), and features (e.g., custom features, word embeddings, etc.). The machine learning component 1346 acts on the information passed to it from the configuration file 1341 by performing training or prediction as instructed and / or by sending instructions to the regression or recursive algorithms. The machine learning component 1346 can also apply any labeling techniques that may be required, such as BIO labeling, sliding window, etc., as well as save or load trained models. According to one embodiment, a regression or recursive algorithm receives data input from the machine learning component 1346, performs training or prediction as instructed via the configuration file 1341, and returns results (trained models or predictions) to the machine learning component 1346. Furthermore, according to one embodiment, the process builder 1345 can enable all of the above tasks by acting as an API to build and interpret instructions that may be provided in YAML format. For example, if a user wants to use different modeling packages for training and prediction, the user can provide the package and model type names in the YAML configuration to the framework 1347 of the process builder 1345.Users can also customize any default modeling algorithms using module 1348. Additionally, in ML framework 1340, minimal, if any, changes to YAML files are required to modify feature engineering and inclusion / exclusion from model training. Additionally, behavioral differences across models are isolated to configuration YAML files and not mixed with the common code base. This allows the code base to remain "stable" while still allowing users the flexibility to make targeted modifications (e.g., fine-grained and / or coarse-grained modifications) at any point and to any extent to the workflow behavior of a particular model instance. Additionally, because these modifications reside in configuration file 1341 (and not in the code), they can be safely passed to the platform without the need to install additional code in the deployment. For example, a user can modify the model inputs to ignore punctuation, stop words, or add additional features such as word embeddings, as well as determine if a word is capitalized. These modifications can be performed by modifying the configuration YAML file rather than modifying the source code. The configuration file can then retrieve the referenced features and generate a feature matrix from the training dataset.

[0076]

[0095] FIG. 16 is a diagram of the machine learning steps depicted in FIG. 13 according to an exemplary embodiment of the present invention. According to one embodiment, training of a model as well as prediction from an existing model are performed using the same configuration file, e.g., configuration file 1341. As depicted in the figure, during training mode, the true labels of the target may be extracted from a training dataset, e.g., Lume dataset, and then provided to the initialized model. Furthermore, features may also be extracted from the training dataset and then provided to the initialized model. Then, a selected model architecture, e.g., a third-party modeling package 1440 (e.g., sklearn, keras, etc.), performs a model training step, and then the trained model is saved in database 1430. Then, during prediction mode, the trained model may be loaded from database 1430 and run against the feature matrix settings from the configuration file as well as the features extracted from the test dataset to predict results from the test dataset. According to one embodiment, the training dataset is specifically data used to develop a model but never used to test model performance, and conversely, the test dataset is used to test model performance but never used to train a model. However, both datasets must be labeled.

[0077]

[0096] According to one embodiment, many questions that can be asked about a document involve explicit extraction of raw information from the text itself. However, for non-machine readable documents where spelling errors are common and / or formatting is inconsistent, further processing is required. For example, a date may be written in many different ways in a document (e.g., 4 / 5 / 2010, 4.5.10, April 5th,2010, the fifth of April 2010, etc.), but the information must still be formatted consistently when reported for analysis. Thus, post-processing is required. In this regard, during the post-processing step 1350, the user can customize specific tasks and functions to perform on the model results. Furthermore, the post-processing step 1350 can also be used to impose certain business logic and conditioning on the model results. For example, certain business logic can be imposed where one field may depend on another field - if the model predicts that there should be no auto-renewal in the contract, then there should be no result on the length of the auto-renewal clause. Thus, in the post-processing step 1350, the data can be provided in a format required by the user. Furthermore, business logic can be imposed across various model predictions when the results involve independent fields.

[0078]

[0097] Further, according to one embodiment, during the consolidation step 1360, relevant documents are input and then business logic is executed by the graph consolidation engine 1361 (see FIG. 17) to determine which information should be reported from which document. For example, in the case of a master service agreement with multiple amendments, information about the contract clause should be derived from the latest amendment. According to one embodiment, this logic can be coded into the graph consolidation engine 1361 by a user. Furthermore, the consolidation task can be implemented by a graph database 1370 (e.g., JanusGraph) to model the relationships between the documents. For example, as depicted in FIG. 17, multiple documents 1362 and 1363 (or versions of the same document, i.e., “Document 1”) can be input into the graph consolidation engine 1361 along with updated or conflicting facts (e.g., Facts A and B). For example, for document 1362, Fact A = “True” and Fact B = “1”. Meanwhile, in document 1363, fact A = "False" and fact B = "2." In this regard, to resolve the conflict between document 1362 and document 1363, graph consolidation engine 1361 uses other model outputs found in the documents, which may be retrieved from graph database 1370. Graph consolidation engine 1361 may then provide a consolidated output 1364 that reflects the current true facts for document 1.

[0079]

[0098] FIG. 18 is a diagram depicting a graph schema representing multiple documents, according to an exemplary embodiment of the present invention. For example, as depicted in the diagram, documents 1366 (i.e., Doc1, Doc2, Doc3, and Doc4) can be represented in either graph schema 1367 (i.e., graph schema A) or graph schema 1368 (i.e., graph schema B). According to one embodiment, graph schemas 1367 and 1368 can be based on a custom model for a business case defined by an SME. Graph schemas 1367 and 1368 can be generated via a configuration file, in which an SME can specify what information in a document can be used to determine connections between documents 1366 in the graph. This graph model can then be loaded into a graph database, and all data loaded into the graph will attach to this graph model. Furthermore, graph edges can be automatically and dynamically established based on the processed model. In this regard, in graph schema 1367, documents 1366 are connected by a shared document ID, e.g., "contract family 1". Additionally, in the graph schema 1368, Lume is connected to the document root through the associated customer name.

[0080]

[0099] Furthermore, according to one embodiment, the exemplary framework can answer questions about document families using graph queries custom to the dynamic schema. For example, assuming the question was "Find update periods and prioritize amendments from most recent," the exemplary framework would find only amendments, order them by the "effective date model," convert the query to a graph query, and perform a graph traversal. Results are returned to the user without the need to understand the underlying graph model, along with a complete explanation of how the integration is performed. For example, the result could be "Find X amendments, which have the following dates: They have the following update periods: Y. The best answer is Z." Furthermore, if the question was "What is the lowest price valid price," the exemplary framework would find any documents with a price, then find the lowest price, convert the query to a graph query, and perform a graph traversal. In this regard, the result could be "Found X documents with price, value is [...]. The lowest value is Y."

[0081]

[0100] 13, the exemplary framework, e.g., flow 1300, also includes a quality assessment (QA) component that performs QA checks after every step in flow 1300 to enforce high quality and consistency. These checks can include (i) whether certain Lume elements were created and added to the Lume data structure as expected, (ii) whether all Lumes were successfully passed from step to step, and (iii) whether the correct attribute keys and counts were included at each step. Additionally, users can also configure and add their own custom quality assessment checks as needed.

[0082]

[0101] It will be appreciated by those skilled in the art that the various embodiments described herein are capable of broad utility and application. Thus, while various embodiments have been described in detail herein in connection with exemplary embodiments, it should be understood that the present disclosure is an explanation and example of the various embodiments, and is made to provide an enabling disclosure. Thus, the present disclosure is not to be construed as limiting the embodiments or otherwise excluding any other such embodiments, adaptations, variations, modifications, and equivalent arrangements.

[0083]

[0102] The foregoing description provides examples of different configurations and features of embodiments of the present invention. Although certain names and types of applications / hardware have been described, other names and applications / hardware may be used, and the names are provided by way of non-limiting example only. Additionally, although specific embodiments have been described, it should be appreciated that the features and functions of each embodiment may be combined in any combination as is within the ability of one of ordinary skill in the art. The figures provide further exemplary details regarding various embodiments.

[0084]

[0103] Various exemplary methods are provided herein by way of example. The described methods may be performed or otherwise implemented by one or a combination of various systems and modules.

[0085]

[0104] Use of the term computer system in this disclosure can refer to a single computer or multiple computers. In various embodiments, multiple computers can be networked. The networking can be any type of network, including, but not limited to, wired and wireless networks, local area networks, wide area networks, and the Internet.

[0086]

[0105] According to an exemplary embodiment, the system software may be implemented as one or more computer program products, e.g., one or more modules of computer program instructions encoded on a computer-readable medium, for execution by or to control the operation of a data processing apparatus. The implementation may include single or distributed processing of an algorithm. The computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a memory device, or a combination of one or more of them. The term "processor" encompasses all apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, an apparatus may include software code that creates an execution environment for the computer program in question, e.g., code constituting a processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0087]

[0106] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program can be stored in a single file dedicated to the program, or in multiple linked files (e.g., files that store one or more modules, subprograms, or portions of code), as a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document). A computer program can be deployed to run on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communications network.

[0088]

[0107] A computer may encompass all apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, computer, or multiple processors or computers, which may include, in addition to hardware, code that creates an execution environment for the computer program in question, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0089]

[0108] The processes and logic flows described herein may be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by, and an apparatus may be implemented as, special purpose logic circuitry, for example, an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0090]

[0109] Computer-readable media suitable for storing computer program instructions and data can include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0091]

[0110] Although the embodiments are illustrated and described within a framework specifically for performing the analysis, it will be appreciated that variations and modifications may be effected by those skilled in the art without departing from the scope of the various embodiments. Moreover, those skilled in the art will recognize that such processes and systems need not be limited to the specific embodiments described herein. From consideration of the specification and practice of the embodiments disclosed herein, other embodiments, combinations of the embodiments, and their uses and advantages will be apparent to those skilled in the art. The specification and examples should be considered as illustrative.

Claims

1. 1. A computer-implemented method for analyzing data from various data sources, comprising: receiving data from said various data sources as input; converting the received data from each of the various data sources into a common data structure; identifying keywords within the received data; generating sentence or word embeddings based on the identified keywords; receiving a selection of one or more labels based on the generated sentence or word embeddings; adding the selected one or more labels to the common data structure; training a model over the common data structure based on a configuration file; generating results based on the model in response to a user's query; wherein the generating step comprises: retrieving a plurality of relevant documents from the received data; determining what information should be reported from which of the retrieved relevant documents via a graph schema, the graph schema comprising: linking the plurality of relevant documents by one or more connections; configured to compare one or more data facts from each of the associated related documents to resolve conflicts among the associated related documents. A determining step; providing said result based on said determining; A computer-implemented method comprising:

2. The method of claim 1 , wherein the various data sources include at least one of: machine-readable documents, non-machine-readable documents, spreadsheets, images, and hypertext markup language files.

3. Segmenting the received data into component documents, the received data being segmented based on one of a heuristic model and a trained model. The method of claim 1 further comprising:

4. tokenizing at least one of word elements and sentence elements in the received data; adding default word embeddings to at least one of the tokenized word elements and sentence elements; The method of claim 1 further comprising:

5. The method of claim 1 , wherein the configuration file includes instructions regarding at least one of a task type, an algorithm type, and a feature.

6. 6. The method of claim 5, wherein (i) the task type is one of training, validation, and prediction, (ii) the algorithm type is one of a regression algorithm and a recursive algorithm, and (iii) the features comprise word embeddings.

7. performing at least one quality assessment check The method of claim 1 further comprising:

8. receiving at least one representation via a user interface; providing said at least one representation to said model; selecting, using an application programming interface, candidate annotations associated with the at least one representation; training the model based on the selected annotation candidates; The method of claim 1 further comprising:

9. The method of claim 1 , wherein during training, target true labels and features are extracted from a training dataset and then provided to the model.

10. 1. A computer-implemented system for analyzing data from various data sources, comprising: Processor wherein the processor: receiving data from said various data sources as input; converting the received data from each of the various data sources into a common data structure; Identifying keywords within the received data; generating word or sentence embeddings based on the identified keywords; receiving a selection of one or more labels based on the generated word or sentence embeddings; adding the selected one or more labels to the common data structure; training a model over the common data structure based on a configuration file; Generate results based on the model that respond to user queries. The generating step is configured to: retrieving a plurality of relevant documents from the received data; determining which information should be reported from which of the retrieved relevant documents via a graph schema, the graph schema comprising: linking the plurality of relevant documents by one or more connections; configured to compare one or more data facts from each of the associated related documents to resolve conflicts among the associated related documents. To judge, providing said result based on said determination; 1. A computer-implemented system comprising:

11. The system of claim 10 , wherein the various data sources include at least one of a machine-readable document, a non-machine-readable document, a spreadsheet, an image, and a hypertext markup language file.

12. The processor, Dividing the received data into component documents. wherein the received data is segmented based on one of a heuristic model and a trained model. The system of claim 10.

13. The processor, tokenizing at least one of word elements and sentence elements in the received data; Adding default word embeddings to at least one of the tokenized word elements and sentence elements The system of claim 10 further configured to:

14. The system of claim 10 , wherein the configuration file includes instructions regarding at least one of a task type, an algorithm type, and a feature.

15. 15. The system of claim 14, wherein (i) the task type is one of training, validation, and prediction, (ii) the algorithm type is one of a regression algorithm and a recursive algorithm, and (iii) the features comprise word embeddings.

16. The processor, Run at least one quality assessment check The system of claim 10 further configured to:

17. The processor, receiving at least one representation via a user interface; providing said at least one representation to said model; selecting, using an application programming interface, a candidate annotation associated with the at least one representation; Training the model based on the selected annotation candidates. The system of claim 10 further configured to:

18. The system of claim 10 , wherein during training, target true labels and features are extracted from a training dataset and then provided to the model.

19. 1. A computer-implemented system for analyzing data from various data sources, comprising: An application programming interface; Processor and wherein the processor: Generate results based on machine learning models to answer user questions The generating step is configured to: retrieving a plurality of relevant documents from the data from the various data sources; determining which information should be reported from which of the retrieved relevant documents via a graph schema, the graph schema comprising: linking the plurality of relevant documents by one or more connections; configured to compare one or more data facts from each of the associated related documents to resolve conflicts among the associated related documents. To judge, providing said result based on said determination; Including, A computer-implemented system, wherein the machine learning model is trained on annotation candidates provided by the application programming interface.

Images