Methods, computer systems, and non-transitory computer-readable storage media for analyzing and improving documents

Abstract

Machine learning, artificial intelligence, and other computer-implemented methods are used to identify various semantically important chunks in documents, automatically tag them with appropriate data types and semantic roles, and use this augmented information to assist authors as well as support downstream processes. Chunk location, data type, and semantic role can often be determined automatically from what is referred to herein as "context," i.e., their formatting, structure, and content in combination; those in adjacent or nearby content; overall patterns of occurrence in a document; and similarity of all these things across documents (mainly but not exclusively among documents in the same document set). Similarity is not limited to exact or fuzzy string or attribute comparisons, but can include similarity of natural language grammatical structures, ML (machine learning) techniques such as word, chunk, and other embeddings measuring similarity, and similarity of previously identified chunks' data types and semantic roles.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims priority under 35 U.S.C. § 119(e) to U.S. Provisional Patent Application Serial No. 62 / 900,793, “Cross-Document Intelligent Authoring and Processing Assistant,” filed September 16, 2019. The subject matter of all of the above is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0003] The present disclosure relates generally to methods and apparatus for AI self-supervised creation of hierarchically semantically tagged documents and / or for assisting in authoring and processing such documents. BACKGROUND

[0004] Many businesses create multiple documents that are very similar, even though they are each customized. For example, an insurance office can produce many proposals for a specific kind of insurance, but each proposal must be tailored to the needs of a specific customer. These documents can be considered to be of the same “type” because they have similar textual (and possibly image) content (reflecting similar purposes and themes), similar selection and arrangement of large units such as sections, and often even similar geometric layout and formatting characteristics.

[0005] Some types of documents are widely known and used, but many are not. Many are specific to a particular business, market, or application, and new documents are created for new situations. Users, who can be called “authors” or “editors,” typically create new documents of a specific type (sometimes called “target documents”) by copying an earlier document of the same type and then modifying it as needed (e.g., by manual editing or replacing certain blocks of content).

[0006] In current practice, word processing typically identifies blocks only when formatting needs to be implemented: for example, headings, footnotes, and numbers can be explicitly marked out for special formatting; but names, addresses, or dates are rarely explicitly marked out. Even when identified, blocks are typically only associated with formatting effects such as margins, fonts, etc., which are useful information, but do not directly provide any indication of their data type or semantic role. Similarly, word processing programs often only visually represent hierarchical inclusion: often there is no explicit identification of nested sections themselves, but only explicit representation of headings that are differently formatted.

[0007] When creating a new document of the same general kind as a previous document, in many cases most of the work is text editing, replacing, removing or inserting certain blocks, taking care not to confuse blocks with different semantic roles (such as swapping the buyer and seller addresses). This often requires human intervention, as writing systems usually know nothing about these blocks, in particular their data types or semantic roles, and therefore cannot help very effectively.

[0008] In some simple cases, "forms" and "templates" can be used, providing explicit places to fill in the content of particular blocks. However, forms usually only solve simple cases where essentially all the blocks needed can be enumerated in advance, and where there are few large, repeatable or highly structured blocks. Creating forms also requires skilled work, is difficult to adjust to changes in the situation, and does not actively help the author. BRIEF DESCRIPTION OF DRAWINGS

[0009] The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

[0010] Embodiments of the disclosure have other advantages and features that will be more readily apparent from the following detailed description when taken in conjunction with the examples illustrated in the drawings, of which:

[0011] Figure 1 is a block diagram of one implementation of a system and process for creating hierarchically semantically tagged documents using machine learning and artificial intelligence.

[0012] Figure 2 is a screenshot of a dashboard illustrating the tracking of the processing of different document sets by Figure 1

[0013] Figure 3 is a screenshot of a user interface for receiving feedback from a user.

[0014] Figure 4 is a screenshot of integration with other software applications.

[0015] Figure 5 is a block diagram of one embodiment of a computer system that can be used with the present invention. DETAILED DESCRIPTION

[0016] SUMMARY

[0017] ​A group of documents that are determined to be of the same type constitute a "document set" or "document cluster." For example, recommendations by an insurance company for a certain insurance for a certain class of customers can be considered to be of the same type, and form a document set. Recommendations by the same company for a different kind of insurance, or for customers they consider to be different, can be considered to be of different types belonging to different document sets. Lease agreements, clinical notes for a certain kind of patient, sales recommendations, schedules, meeting minutes, etc. are other potential document types, with subtypes that share unique patterns of content, structure, and / or formatting.

[0018] Creating and editing new target documents within a document set often involves editing or replacing "semantically significant" "chunks": such chunks are particular portions of a document, usually but not necessarily contiguous spans of text, that have particular data types and semantic roles, and that have meaning and significance to a business or other process.

[0019] These chunks have various data types, which are here more refined than the atomic data types in many computer systems. For example, a given chunk can represent not only a string, but: a person or organization name; a date; a duration (quite different from a date); a monetary amount. Larger chunks can include lists of drugs or other substances, itineraries, procedures to be followed, information packets such as medical prescriptions; and countless other kinds of information.

[0020] Furthermore, chunks can have semantic roles related to the document in which they appear. For example, a person name can be a "tenant" in a lease agreement, or a "seller" in a sales recommendation, or an "agent" of another person. A date can represent the beginning or end of a certain responsibility or activity. A dollar amount can be a periodic payment amount, or a penalty or bonus related to certain conditions, etc. Such semantic roles are important for proper use of the information in the chunk. The names of semantic roles are called "semantic role labels," or simply "labels."

[0021] Chunks are typically represented as packages, including their location, data type, semantic role, and / or other data / metadata. Location is typically represented as a start and end point, which can be represented in several ways such as inserted markers or byte, character, or token offsets (either globally for the document, or relative to a given ID, marker, or other object). Semantic roles are represented by labels or other identifiers. Chunks can be of any size, and some chunks can contain other chunks as "subchunks." Chunks can contain not only text, but also non-text data such as images or other media, as well as "structure" (such as tables, lists, sections, etc.).

[0022] The technology disclosed herein uses machine learning, artificial intelligence, and other computer-implemented methods to identify various semantically important chunks in a document, automatically provide them with appropriate data types and semantic roles, and use this enhanced information to assist authors and support downstream processes. The location, data type, and semantic role of a chunk can often be determined automatically from what is referred to herein as the "context" of their formatting, structure, and content; those in adjacent or nearby content; overall patterns of occurrence in the document; and similarities across documents (mainly but not limited to between documents in the same document set) of all these things. "Nearby content" includes content that is close in the horizontal direction, such as before and after in the text reading order; and also content that is close in the vertical direction, such as within the same container structure, like lists and sections, and their respective markers, headings, levels, etc. Similarity is not limited to exact or fuzzy string or attribute comparisons, but can include similarity of natural language grammatical structures, ML (machine learning) techniques such as word, chunk, and other embedding similarity, and similarity of data types and semantic roles of previously identified chunks.

[0023] For example, a personal or organizational name can often be identified as having a semantic role such as "seller" because the document says so, often using a sentence or sentences in some human language, but often involving larger context as well. As another example, one or more words can often be easily identified as representing a data type such as "drug name"; but context is needed to determine that it carries the semantic role of an allergy, rather than a prescription. Important evidence for a semantic role is often not in the same sentence as the chunk, but is expressed in various other ways, such as by the chunk's occurrence in a larger chunk such as a "known allergies" section. The flexibility and variety of grammatical and document structure organization, not to mention typos, transcription errors, etc., make it less difficult to identify data types, but very difficult to identify semantic roles, especially those that range over more than a single sentence.

[0024] A given semantic role can relate a chunk to the entire document or other chunks. For example, a flight's departure time is tied to a particular "leg" of a journey, and is only indirectly related to other "legs." Often, the hierarchical structure of chunks groups such items together appropriately, such as co-located within sections, table sections, etc.

[0025] In more detail, examples of semantically significant chunks include: the name, address, and other characteristics of a particular party to a contract; the prescribed medications and procedures in a medical record; the requirements (or exclusions) in a real estate offer; the date and flight number in a travel itinerary; and so on. These can all be considered as the semantic roles of the chunks. There are also larger chunks with various types and roles, such as entire sections and subsections. These are often inserted or removed as a whole, perhaps with smaller chunks changed within. Chunks can be hierarchical; that is, larger "containing" chunks can contain other "child" chunks to any number of levels.

[0026] A chunk is typically a contiguous sequence of text in a document, such as "John Doe". However, a chunk can include partial text. "John Doe's house" includes the name, but the name ends before the apostrophe (in the middle of the text). A chunk can even be non-contiguous, for example, the same name in "John (also known as 'Bill') Doe". Formatting can also make a chunk non-contiguous, for example, a page break can occur in the middle of a chunk (perhaps with a header, footer, or footnote, which can be ignored for some purposes); a figure, table, chart, sidebar, or other display in the middle of a chunk; and so on.

[0027] The actual location and context of a chunk can also be important - a chunk is not just an isolated string, it can occur multiple times, with different (or no) semantic roles for different instances. More modern systems often support in-line or standoff tagging, sometimes called "annotation", which can persistently associate various labels and other information with a chunk of text. For example, HTML provides tags for manually marking the boundaries of general structural chunks ("div", "ol", etc.), as well as several widely-used types or roles for (usually) smaller chunks ("acronym", "kbd", "dfn", "cite", etc.). Other XML schemas provide many other tags, and word processors allow somewhat similar tagging via "styles".

[0028] Some chunks can represent what are often called "fields". These are often small chunks, and often occur in many or all documents in a given set with similar context and formatting, but usually with different textual content in each document. They can also occur multiple times in a single document, with the same or very similar content. Such chunks can be called "field chunks". In template-based systems, they are often manually discovered and treated as "fields", but here they are discovered by their patterns of context and occurrence within and across documents, and are assigned data types and semantic roles in much the same way as other chunks. They can or can not represent named entities, such as personal names, addresses, dates, and so on.

[0029] Another common type of block can be called a "structural" or "structural" block. Such blocks are typically large, and often contain many other blocks (some of which can also be structural blocks). They frequently have a "title" or "heading" that provides a name, number, description, and / or other information about the structural block. Examples of structural blocks include chapters, sections, tables, figures, sidebars, and many others. The type and semantic role of a structural block are often important to determining the type and semantic role of nearby or contained blocks.

[0030] Not only data types, but also specific semantic roles, are important for properly authoring and utilizing documents: it matters whether a certain name represents a buyer versus a seller, or a patient versus a doctor; whether a given date is a required start or end, or a flight's departure versus arrival time; whether a number specifies principal, interest, dosage, temperature, penalty, or something else. For larger blocks, roles include things like "Limitation of Liability" statements versus "Governing Law" specifications and "definitions", as well as countless others. The semantic role of a block is often specific to a particular domain or transaction, and can be considered among the most important features of a document. Blocks with specific data types and semantic roles are necessary, or at least very common, in many kinds of documents, and when blocks correspond across different documents, they are called "counterparts". Counterpart blocks can occur in similar order and pattern, especially for documents by the same author or organization, and often belong to the same document set. Counterpart blocks have the same or very similar roles, and often have similar context and / or formatting. Thus, the distribution of block data types and semantic roles provides valuable information to distinguish types of documents, as well as to help identify counterpart blocks in other documents.

[0031] Many counterpart blocks have similar content, but others do not. For example, the same party (semantic role) in different documents is often a different individual, despite appearing in very similar context and usage patterns. This can be especially common for "field blocks", but is not exclusive to them.

[0032] Once discovered, the hierarchical semantic blocks in a business document, with their data types and semantic roles, can be used for downstream business processes. For example, if a back-end database is given a specific party name, a specific date, and numbers such as term and interest rate, it can properly record a new mortgage. Semantic roles are especially important for such uses: putting the correct data type into the wrong database field (such as swapping a seller and buyer's name or address) is a big problem, especially when moving information to downstream databases, processes, or reports.

[0033] Some features and benefits

[0034] The technology described herein can have various features and benefits, including any of the following.

[0035] Some implementations can provide an easier, more efficient, and more accurate way to produce documents with hierarchically organized chunks that have semantic labels useful to business processes. This can be accomplished using various techniques to identify such chunks of varying sizes, discover their data types and semantic roles that they play in the document, and learn their usage patterns, characteristic contexts, etc. This learning can come from analysis of the content, structure, and formatting of current and previous documents; from feedback from authors and editors; and from comparisons of multiple documents, particularly documents in the same document set. With this knowledge, the system can provide valuable assistance to users, e.g., to more easily create higher quality new documents, and to extract desired information for downstream use, such as with other software applications, in background databases, exported reports, compliance checks, etc. Such learning can be accomplished with unsupervised and self-supervised learning techniques that do not require large amounts of pre-labeled or pre-analyzed data, but rather infer patterns from unlabeled or minimally labeled data.

[0036] Some implementations can enable a computer to help writers avoid many of these errors by discovering and using patterns within and across business documents, and thus reduce the time needed to achieve a given level of quality, assisting in the writing process.

[0037] Today, typical document systems do not identify chunks, or particularly their data types or semantic roles. This increases the time and overhead for authors and editors, and increases the time and overhead to import data from the document into backend databases, dashboards, or other downstream business processes. For example, it is common to manually find and copy data from a contract (chunk by chunk) into a spreadsheet or data entry form.

[0038] Some implementations can help to label such hierarchical semantic chunks during the authoring process, and to explicitly represent them, thus making them easy for people and / or computers to extract, and saving time and overhead in connecting to other various business processes.

[0039] Current techniques often do not take full advantage of similarities between multiple documents created by the same author or group, and / or the same type of document (here represented by membership in a particular document set), to more reliably identify blocks in a new document, or flag possible significant differences for attention. Explicit rules, such as requiring sections titled "Divisibility," only cover similarities that analysts are readily able to notice and describe; are static, and often constraining (e.g., missing cases of paraphrasing or reorganization, or failing to respond to counteracting conditions); and quickly become out of date. Small companies often lack the resources necessary to develop more responsive techniques, and often have little documentation to justify the overhead. Smaller companies, on the other hand, often have less diverse documents that are more amenable to automated analysis, as described herein.

[0040] Some implementations can use extracted information about blocks and their content, context, formatting, and usage across documents to assist authors in creating new documents. Examples include at least suggesting: specific content to change, reformat, or move; clauses missing in the new document, although they are typically present in similar documents (blocks or content called "missing" or "possibly omitted"); clauses present, although they are typically absent in similar documents (blocks or content called "unusual"); changes, such as swapping names or roles of different parties in a particular location; and so on.

[0041] Some implementations can accept and retain user feedback, such as when a user indicates that a block: is labeled with incorrect scope, data type, or semantic role; is not of interest to them; or is not labeled at all. Some implementations can use specific user corrections to improve machine learning and neural models, and to remember not to repeat earlier suggestions in cases where the user has rejected them (even if additional learning fails to prevent the particular error instance). In particular, some implementations can avoid requiring extensive review steps or corrections, which benefit from small-sample learning techniques and judicious selection of required feedback, in order to minimize the amount of user action required. Some current techniques learn very specific things, for example, when a user tells a spell checker to add a word to its dictionary. However, this involves merely a mechanical list to remember, rather than iterative training or fine-tuning of the model used to determine complex later behavior, and thus does not take full advantage of capabilities such as described herein.

[0042] Some implementations can use a small number of user corrections to learn and improve their behavior, while avoiding annoying users with repeated suggestions when reapplying an improved but still imperfect model.

[0043] Many businesses record specific information obtained from documents in various databases that support their processes. For example, a company that owns many rental properties often uses a back-end system to help manage not only payments by tenants, but also specific information that arises from their lease agreements, such as approved pets, prior damage for which the tenant is not responsible, or other information. Car or tool renters, mortgage companies, health care providers, municipal authorities, and other organizations use other information. Many goods and services have many mixed and matched options, and executives review statistics about their acceptance, combinations, pricing, and other factors. Business information systems often provide analysis, check for consistency or compliance, draw reports, and / or support other business processes, all of which can be facilitated by using the block information described herein.

[0044] In general, blocks and the information they provide are scattered throughout the prose text, extracted manually, and entered into spreadsheets, databases, or other systems manually. Manual effort was previously required because important blocks can be expressed in an infinite number of different ways, because of the flexibility of natural human language in which agreements, emails, and the like are written, and similarly variable typography and notation conventions. Negotiation content underlying such documents is also often scattered across multiple documents, including emails, transcripts of conversations, slide presentations, and the like. This information can also be useful, but is often handled manually. Some systems can process such sources of information as documents, obtaining the same benefits that have been described.

[0045] Some implementations can provide means for a document computer to begin performing a particular document once it has been transformed into a hierarchically semantically tagged document as described herein. By combining the hierarchical tagged structure of a document with tools that provide vector-semantic representations of text, certain blocks can be identified as requiring certain actions. For example, a contract can specify a transfer of funds, a notice, or other action, and the conditions that enable or trigger them. These can be identified and used to begin performing the contract.

[0046] Some implementations can provide a simple way to review and summarize information from a set of documents in an interface such as a "dashboard," and to move identified information into a customer's back-end database or similar system, making the flow of business data more efficient and less costly, and enhancing quality assurance, consistency, and reporting. Once blocks are semantically tagged, it becomes easier to generate summary reports on a set of documents that contain corresponding blocks. Some implementations can provide users with a very simple way to create such reports, by clicking on one or more examples of blocks to be included, and then locating and extracting those blocks across all documents in the set by role or context. Some implementations can also assist users in finding documents that lack expected corresponding blocks, and to correct them to include or identify such blocks, or to confirm that they correctly do not include such blocks.

[0047] In another aspect, performance for a given group, such as a company or department, can be improved by incorporating information such as chunk semantics, patterns of occurrence, and other characteristics of their documents, and their user feedback, into the system's learning process, and using the resulting improved models to augment and / or check future documents. However, many customers do not want to share such information with other customers, and many customers have restrictive confidentiality requirements. On the other hand, general information and learning from public, non-confidential sources can be used and shared freely.

[0048] Some implementations can provide the benefits of feedback and learning while keeping each customer's data and any model information derived from it, individually and private to each customer, while still sharing general learning based on non-confidential public data. Keeping those data processes separate ensures that information does not "leak" from one customer to another, even statistically.

[0049] Introduction to example implementations

[0050] The following is a description of an example system. See Figure 1 . The system generally relates to methods and apparatus for AI to create semantically labeled documents hierarchically, and / or to assist in authoring and working with such documents. This includes processes such as authoring, structuring, annotating, modifying, reviewing, extracting data from documents, and / or using such data in downstream business processes. More specifically, it focuses on documents similar to previous documents, discovering detailed hierarchical structures of the documents, consisting of many semantically meaningful chunks, associated with their roles, by primarily using unsupervised and self-supervised machine learning techniques across a set of documents, including relatively small sets of documents; and on using such highly augmented documents in business processes.

[0051] The operation of this example system uses the following processes, which are described in more detail in the following sections. This is just an example. Other implementations can use different combinations of steps, including omitting steps, adding other steps, and changing the order of some steps. They can also use different implementations of the steps listed below, including different combinations of the techniques described under each step. In Figure 1 In the following, steps are preceded by "S", so step 1 below is labeled "S01", and so on.

[0052] 1) Import : Bring a set of user documents into a data store 110.

[0053] 2) Organization : Divide the documents into document sets by type, such as leases versus sales agreements, or medical histories versus current clinical records.

[0054] 3) Visual extraction : Extract linear text stream(s) from each document based at least on their content and visual layout, including limited information about distinct text and other regions, their start and end positions, formatting, and content. The extracted data can be organized as "visual lines" or "visual boxes" (also known as "superlines" or "visual" blocks), such as paragraphs distinguished by geometric layout.

[0055] 4) Structure : Identify headings, list items, and other large categories of structural blocks in the document.

[0056] 5) Re-nesting : Determine the nested relationships of sections and lists, and the text ranges of each section and list.

[0057] 6) Topic chunking : Analyze the subject matter content of each document and produce blocks (topic-level blocks) containing regions of similar subject matter.

[0058] 7) Topic tagging :

[0059] i) Use embeddings and clustering to produce candidate data type and semantic role labels for each heading in the corpus.

[0060] ii) Use key phrase extraction techniques to produce candidate data type and semantic role labels for blocks.

[0061] 8) Chunk tagging : Use a variety of methods, such as neural networks, word and character embeddings, syntactic analysis and pattern matching, regular expressions, similarity metrics, and / or other methods, to identify and assign (possibly multiple) data type and semantic role candidates to other blocks in the entire document. Of particular interest for certain embodiments are:

[0062] i) Syntactic parsing and pattern matching of the resulting structures.

[0063] ii) Use question answering techniques to link small blocks to the specific semantic roles they play in the document.

[0064] iii) Combine XPath tree matching with word embedding techniques to match patterns in the structure and syntax trees, despite wide variations in wording and phrasing.

[0065] 9) Named entity recognition (NER): Identify data types and assign to blocks detected as named entities in the entire document.

[0066] 10) Role tagging, extractive tagging: Assigning semantic role labels to chunks, such as indicating that the "Seller" party constitutes a contract, or that a drug is mentioned as an allergy, rather than as a prescription.

[0067] 11) Anomalies : Identifying semantic roles that are typically present in documents of the document set under consideration but not in the current document, or vice versa.

[0068] 12) Arbitration : Making adjustments and / or selections among alternative ranges, data types, and semantic role labels for chunks, resulting in well-formed structures that are easily expressed in formats such as XML format.

[0069] 13) DGML : Creating an enhanced version of the document that contains explicit identification of chunk locations, data types, and semantic role labels, and possibly additional information such as a confidence level for each identified chunk, expected data types in similar chunks (such as dates, date ranges, personal names, etc.), and the like. The enhanced version is created using an XML-based markup language called DGML.

[0070] 14) Feedback : Displaying the enhanced version to the user(s) and selecting chunks (and potentially the underlying locations of segments that were ignored) to show the user, thereby collecting the user's selections to confirm, reject, or otherwise change. The user is also free to select their own reading and review order. Feedback can also be applied to any other interpretations made by the system, such as the organization of the document into a document set described in step (2).

[0071] i) In the case of potentially ignored chunks, providing priority examples from other documents that can be inspected and / or copied into the current document, and automatically customized by applying target document values for smaller nested chunks, as desired.

[0072] 15) Feedback response : Tracking the user's responses to these interactions, and using that information to fine-tune the model 120, as well as to prevent the same or similar errors from being repeated later.

[0073] 16) Downstream communication, transmission : Selecting chunks by type and / or role, and using them to generate reports on the document set, and / or exporting them to downstream systems that add functionality such as back-end contract databases, regulatory compliance checkers, management report generators, and the like.

[0074] Figure 2is a screenshot showing a dashboard that tracks the processing of different document sets One through Seven by the above process. In this dashboard, the process is divided into the following stages:

[0075] • Upload

[0076] • Preprocessing

[0077] • Reviewing Large Blocks

[0078] • Reviewing Small Blocks

[0079] • Ready for Use

[0080] Color coding shows the degree of completion. Green stages are already completed, red stages are in progress, and black stages have not yet started.

[0081] Each of the steps listed above is described in more detail below.

[0082] Further description of example implementations

[0083] The numbering here reflects the general analysis order for this particular example. However, not every step depends on every previous step, and thus, many of the elements can be reordered or parallelized in other implementations. Elements can also be shifted, or even repeated, in order to exchange additional information with other elements, or elements can run independently, such as in separate processes or machines.

[0084] 1) Import

[0085] The system accepts typical word processing program documents (such as MS Word) and typeset documents (such as PDF or.png files). In each case, visually contiguous regions, such as headings, paragraphs, table cells, tables, images, etc., are identified and represented as blocks using a combination of their relative position, surrounding whitespace, font, and typesetting characteristics, etc. These features are chosen in part by the designer and in part learned through image and pattern analysis on a large number of documents. For incoming documents that do not yet have machine-readable text content, OCR is also applied.

[0086] Those blocks, along with selected typesetting information, are submitted to subsequent modules in the system.

[0087] 2) Organization

[0088] Users do not have to organize the documents they examine into the system. The system uses clustering methods operating on textual content, layout information, and already detected structural information such as identification of some headings, to group documents into "sets" of particular types of documents, e.g. leases with rentals vs. sales. The particular document sets found can be examined with the user, and named automatically or by the user. Once established, these document sets facilitate later machine learning and reasoning about format, content, semantic roles, and differences therein. For example, the system can find that almost all documents in a given set have a particular section with three particular sub-blocks of data types of specific roles and personal names, one of which repeats in five different sections. Such patterns are used to help identify similar (and dissimilar) parts of other documents, suggest review or changes to the user, and provide example text for reuse in other documents in the same (or possibly different) set.

[0089] Clustering documents into document sets can use features from document structure (order and containment relationships between blocks of different sizes, data types, and roles), and layout, as well as textual content. Once some blocks and / or roles are identified in at least some documents, this information can also be used to improve clustering, either by completely re-clustering, or by smaller adjustments. For example, if the particular content of blocks with the same role is ignored, such as the names, addresses, etc. of the seller and buyer, similar documents can become almost or even completely identical; or the pattern of occurrence of different blocks is examined to see if it is the same, e.g. one name (like the seller's) appears in certain locations, while another (like the buyer's) appears in certain other locations.

[0090] The system both preserves the original organization of the uploaded files into directories, if any, and also preserves its own organization of them into document sets. Thus, users can view both organizations, and the learning algorithm can use both organizations as information. For example, some users name documents according to various conventions, and / or organize documents by customer, document type, or other features, which is almost always useful for understanding patterns of similarity, such as having common block locations and roles, and relationships between documents.

[0091] 3) Visual extraction

[0092] i) Region finding

[0093] The system uses heuristic methods and machine learning to identify regions in documents based on geometric patterns. For example, in many documents, meaningful blocks have special layout, such as signature blocks, abstracts, definition lists, tables, etc. Such patterns can be learned automatically by considering geometric and / or layout features, uniqueness or rarity, and / or correspondence within the same document or across documents, especially within the same document set.

[0094] The choice of method depends on the format of the incoming document. For example, word processing program documents usually provide explicit information about paragraph boundaries, but PDFs or scanned pages require the system to assemble them from visual lines, or even to analyze white space dimensions to assign characters into visual lines, such as in a multi-column document.

[0095] ii) Signature finding

[0096] The system creates signatures (also called "digests") for document parts, and uses these signatures to identify and classify "interesting" additional blocks and find their boundaries. The signatures are based not only on the text content, but also on various aspects of the context, and can ignore small contained blocks whose content varies in the corresponding material block (e.g., field blocks whose content varies in the corresponding material block).

[0097] The signatures can use even pixel representations of the blocks. The bitmap images of the text layout are divided into tiles, preferably of size about 24 pixels square (adjusted according to the scan resolution), and the tiles are clustered. The processing of these, including their neighborhood relations, by autoencoders and neural networks, reveals similar visual events, such as the boundaries between text and rules, edges and corners of text blocks, and even changes in indentation and substantial font / style changes. Further neural networks then use this clustering to jointly identify similar layout objects, which frequently indicate or characterize important blocks.

[0098] The methods here can use unsupervised methods to generate document block embeddings based on the pixels in the document blocks, as well as the size of the characters, blocks, their location in the document, etc. (As mentioned earlier, the images can also be blocks). Clustering and comparison techniques can then be used on these embeddings for many downstream tasks.

[0099] iii) Extraction

[0100] This aspect takes a laid-out document (e.g., a PDF or scanned printed pages) and transforms the recognized character images ("glyphs") in the document into a text stream that represents the correct document order of the glyphs (this stream can also contain figures or image objects as appropriate, and there can be multiple streams, such as footnotes or headers, that do not have a typical position in the reading order). In some documents, the reading order is not an entirely explicit representation. One well-known example is that there is typically no indication of the validity of multi-column layout at any given point, and thus the first "line" extends only halfway (or less), not all the way. However, there are many additional examples in which the order of the text can be complex or not obvious. For example, some layout programs draw each character separately, making the text boundaries not obvious. Table cells, sidebars, figures, footnotes, and other displayed content can not have an obvious position in the text order. Some text, such as in page headers and footers (as well as end-of-line hyphens) can not require a position in the text order at all. Many formats do not provide explicit indication of what belongs in such special categories.

[0101] The system solves this task by combining the visual information of the glyphs (position, style, etc.) with a deep neural network that understands the characteristics of the written language used in the document to establish the text stream. In addition, it detects many basic text boundaries, such as lines, blocks, columns, images, in-line font changes, and text boundaries of header / footer objects.

[0102] iv) Representation

[0103] After the text sequence and some hypothesized structural blocks are extracted, the system creates a representation of the document (called "DGML" in one example) that includes those as well as information about the visual characteristics (font, color, size, etc.). The representation of the blocks, including information such as their position, type, and role, is called "annotations." This combined data can then be used by natural language processing (NLP) and deep neural networks (DNNs). The deep neural networks incorporate this visual information to assist in structuring the document into a hierarchical system to represent the document structure, which includes blocks such as headers / body, lists / list items, etc.

[0104] It can include enough information so that later aspects can construct an editable word processor document that is very similar to the original source. This can be included in the DGML or similar representation along with other structural, content, and block information. In many cases, sections of the document that have distinctive formatting and layout are also useful blocks. However, formatting characteristics that are not consistent with the blocks that are otherwise required (and vice versa) can still be represented via special block types, via isolated annotations, or via other methods.

[0105] 4)Structure

[0106] The structural pipeline converts the flat text files into a hierarchical structure, where the sections, sub-sections and other parts of the document form an ordered hierarchy of content-based objects, a structure known to those skilled in the art. This conversion is done using unsupervised machine learning techniques. The method has several stages:

[0107] i) Hyperlining

[0108] This involves splitting the text into "superlines", which are larger groups than visual lines, and which include (relative to visuals) more meaningful logical units, such as paragraphs, headings, etc. This is preferably done using a pre-trained neural network that takes into account features such as the "letter shapes" of the word pieces (especially the leading and trailing word pieces), the typography information such as font and spacing characteristics, and similar features. Some superlines can also already be provided by earlier steps (depending on the format of the input document).

[0109] ii) Document language model

[0110] This preferably uses a document language model that also includes information about the text content, formatting, and any structure that has been found so far, rather than just a language model based on the text. This enables better detection of blocks and their hierarchy (such as headings / body, lists / list items, etc.), as learning from formatted pages identifies meaningful blocks and patterns in which they appear.

[0111] This creates a representation of the document that includes both the text content and the visual characteristics (geometric shape, font, color, size, etc.). A deep neural network and NLP processes then utilize such information in the task of structuring the document into a hierarchy of blocks with data types and semantic role labels by finding the extents and / or boundaries of blocks of various sizes that represent the structure of the document. At this stage, the blocks discovered are mainly headings, sections, lists and items, tables, figures, and other relatively large units.

[0112] iii) Hyperline clustering

[0113] This uses an autoencoder to cluster the superlines across the set of documents based on the letter shape structure, assigning each superline to a cluster of superlines that are similar in terms of typography, start and end content, and other characteristics, where each cluster is identified by a "cluster ID" (this should not be confused with the creation or identification of the set of documents).

[0114] iv) Inline heading

[0115] A special case of particular interest is the "inline title", i.e. the title of a block (which sometimes provides the semantic role of the block) itself does not appear on a separate visual line, but on the same line as the start of the subsequent text. Typically, inline titles are distinguished in the layout, such as bold, underlined, different font, followed by a colon or other effects. Separate heuristic and neural algorithms identify these blocks.

[0116] v) Few-shot structure learning

[0117] Despite the above advanced structuring methods, it can be expected that the generated structure has certain imperfections, or does not conform to the user's a priori expectations. As described in steps (14)-(15), few-shot structure learning is responsible for creating a machine learning model that depends on user-provided feedback. This model is then used to generate a structure that combines the user's feedback on the structure and the structure that has already been produced by the system (and perhaps iteratively enhanced by previous feedback).

[0118] The main principle applied in this case derives from machine translation (MT) methods, where a sequence is transformed into another sequence. In this case, one sequence describing a superline is transformed into another sequence that also contains start / end markers encoding the hierarchical system.

[0119] The process proceeds in different phases or steps:

[0120] (a) First, the machine translation model is pre-trained using publicly available datasets.

[0121] (b) The "scheduler" (see description in the "SCHEDULER" section) filters the user feedback. Feedback response

[0122] (c) A new structure file is generated from the user feedback, and a fine-tuned machine translation dataset is generated.

[0123] (d) The pre-trained model is further trained using few-shot learning principles.

[0124] 5) Re-nesting

[0125] This aspect uses a "corpus re-nesting" algorithm that, given a flattened list of cluster IDs, preferably from the superline clustering step, iteratively creates nested structures using pushdown automata. By comparing the signatures of adjacent superlines, the system can determine whether a given title or list item belongs to more, equal or less nesting levels. This allows to reconstruct the multiple nested hierarchical structure of many documents (such as chapters, sections, subsections, clauses, lists, etc.).

[0126] ​Features considered in the re-nesting include: "shape" of the word units in the superline (as known in NLP techniques), especially considering the first and last; special punctuation (if any) ending the previous line; capitalization; formatting information such as leading white space, indentation, bold and underline; presence and form of enumerated strings at the beginning of the line (e.g. patterns like "IV(A)(1)" or "iv)" such as specific bold bullets or other printed symbol characters; value of the enumerated symbol; presence, level and value of previous same kind of enumerated symbol; and so on.

[0127] 6) Topic chunking

[0128] This aspect uses lexical statistics and other learning techniques on consecutive document chunks to find locations of topic shifts. This enhances the identification of boundaries of large chunks, such as entire sections on a given topic, as sections (of whatever level) typically have more unified topic, vocabulary and style within them than they do with adjacent sections.

[0129] 7) Topic tagging

[0130] i) Topic tagger

[0131] For each title in the corpus, as shown in Figure 1 this step

[0132] • creates a numerical representation called "embedding" for each title.

[0133] • clusters the titles based at least on those embeddings.

[0134] • filters out "bad" clusters based at least on measures such as density, number of elements and level of similarity.

[0135] • propagates the most common semantic role label in each remaining cluster to all titles in said cluster.

[0136] ii) Key phrase tagger

[0137] For each chunk, this step uses a collection of key phrase extraction techniques (such as rule-based linguistics, ML, statistics, Bayesian, etc.) to produce candidate semantic role labels for the text.

[0138] 8) Chunk tagging

[0139] i) Grammar

[0140] This aspect of the system begins with a linguistic analysis of the text, such as natural language processing tasks, including part-of-speech tagging, dependency parsing, constituent parsing, and the like. The system then applies tree matching mechanisms from another domain to locate grammatical and other structures within the trees or tree-like structures discovered via NLP. These include document structuring methods such as tree grammars and tree pattern matching, as exemplified by tools such as XPath, GATE.

[0141] Using such patterns to identify grammatical phenomena in sentences enables the system to extract semantic role labels from the text itself, which are then used to annotate nearby blocks. For example, a search pattern can be constructed that matches "The following are the terms of our agreement" (and other sentences with similar grammatical structure) based on the constituent structure of the sentence; and then extracts the noun phrase (in this example, "the terms") and uses it as a semantic role label for one or more blocks in the content that follows this sentence and contains such "terms."

[0142] ii) Question-answer

[0143] Question answering techniques, including BERT for question answering, are specifically adapted to identify semantic role labels for candidate blocks (e.g., dates, person names, dollar amounts). In contrast, most traditional question answering models are designed to answer questions such as "What is the effective date?" The system instead trains models to answer questions like "What is July 8, 2018?" and aims to predict "effective date" or "effective date of X," where X represents another block in the text (not just "date," which is a data type rather than a semantic role).

[0144] The system also discovers synthetic questions that, when answered, can point to relevant information in the text. This provides the ability to automatically formulate questions that will be used by question-answering.

[0145] iii) XPath-like rules integrated with embeddings

[0146] Here, the tools in the domain discussed under "Grammar" are integrated with tools that provide vector-semantic representations of text, such as word2vec, char2vec, and many related methods. The system enables analysts to express and query patterns that include both structural information (which can include block data expressed in XML or DOM-compatible form) well-handled by XPath and similar tools; and fuzzy or "semantic" similarity information well-handled by vector models.

[0147] 9) NER (unlabeled nuggets)

[0148] Technology can identify some chunks by data type, such as personal or corporate names, addresses, etc. (this is called “named entity recognition” or “NER”). However, NER is far from identifying the semantic roles of those entities in the document. Current technology also fails to identify larger chunks, such as entire clauses or sections, or groups of chunks that include larger chunks of meaning or utility.

[0149] This aspect of the system detects small chunks of interest, but does not necessarily assign roles to them either. There are many methods and tools for identifying NER in text. This system uses multiple methods, examples of which are listed below. These innovations are primarily unsupervised:

[0150] i) Established NER methods

[0151] ii) Anticipated text

[0152] By training a language model of n-grams using a broad corpus of general text (such as Wikipedia), a model of “what words are expected in English in context” is built. When looking at a particular document, the system provides a means of identifying n-grams that do not conform to this general model, and thus tend to be special to the document being processed.

[0153] iv) TF-IDF

[0154] This is a TF-IDF based method (“term frequency and inverse document frequency”), and is used in conjunction with label propagation and contextual semantic labeling.

[0155] v) Sequence clustering

[0156] Small sequences of words or characters, such as n-grams, are extracted and clustered using contextual embeddings (e.g. BERT’s embeddings). The expected result is that n-grams that share semantic meaning will start to cluster together. The cost of this combinatorial explosion is addressed by filtering out some n-grams prior to clustering using heuristic methods (including on the syntactic tree). A variety of clustering algorithms can be applied. In this example, the hdbscan algorithm achieves effective clustering while assigning random noise to a “none” cluster.

[0157] vi) Few-shot NER

[0158] The system uses few-shot learning techniques to induce more broadly applicable rules or adjustments to learned parameters from a small number of labeled instances (e.g. selective user feedback). This greatly reduces the number of times user feedback must be solicited, and improves the performance of the system more rapidly.

[0159] 10) Extractive tagging

[0160] This aspect of the system finds semantic role labels for small pieces, which directly appear in the sentence(s) surrounding the piece. Meaningful pieces often have their roles prescribed in some form by the context. For example:

[0161] Jane Doe (the "Seller"), residing at...

[0162] Must pay $1000 rent by the last business day of each month.

[0163] i) Contextual semantic tagging (CSL)

[0164] The process uses a neural network operating on previously established structures (including sentence parsing) to learn what parts of the text can be semantic role labels for various pieces. Many pieces can already have such labels, with different sources and confidence levels, but this provides additional evidence for or against those labels as well as new labels. Here, some of the patterns involve grammar. For example, in "Jane Doe (the "Seller"), residing at... the title verb explicitly states the role of the monetary amount: that is, it is the amount of rent to be paid. Other patterns are automatically learned by supervised and / or unsupervised methods using features of the structure, pieces, tokens, and content available in the context. Formatting (such as parentheses, table formatting, key phrases, and literals), as well as other features, also provide features for the neural network.

[0165] Useful information often resides in contained pieces, such as sections or subsections, or their titles. For example, whether a given drug is a prescription drug or an allergen is relevant, perhaps only detectable by looking at the title of the containing section (which is another example of why it is important to detect the correct hierarchical nesting of sections). There are many other clues that exist that can be learned by machine learning techniques and applied to find applicable roles for various pieces. Similarity across documents can also be used, especially with documents in the same document set, to correlate semantic roles discovered for similar contexts but perhaps not discoverable for isolated documents.

[0166] ii) Label propagation

[0167] This process normalizes the labels of similar text blocks across the corpus of documents. It applies to both the labels extracted from context and the labels available from the previous step. The algorithm clusters the blocks based on their embeddings using a merge clustering approach, ranks the candidate labels for each cluster of blocks using a weighted PageRank algorithm that uses the frequency / confidence of the labels as initial node weights, and determines how similar the labels are to each other using co-occurrence and embedding similarity. It then assigns a label to a block based on the cluster level score of the label and how similar the block we are labeling is to the blocks the label came from (in terms of content, embedding, structure, data type, semantic roles, and / or context). The merge clustering and PageRank algorithms are applied to propagate labels across similar contexts and make labels more consistent across the set of documents.

[0168] 11) Anomalies

[0169] This aspect of the system examines the multiple documents within the set of documents (such as the documents produced in step (2)) and identifies blocks that occur in the current document but generally do not have counterparts in other documents of the same set of documents, or vice versa. The counterpart blocks need not have exactly the same content, structure, formatting, context, data type, and semantic roles, but can vary from one document to another. However, they can be recognized as substantially similar in those ways to other identified blocks.

[0170] When the new document includes blocks that are generally not present in other documents of the same set of documents, some or all of them can be queried of the user to confirm that they are in fact intended. In this example system, such queries will be more prevalent when the blocks in question are common to the new document and the document it is based on, if any, but rare in other documents.

[0171] When the new document lacks counterpart blocks that are generally present in other documents of the same set of documents, or even in a particular related external source (e.g., a house style manual, a compliance requirement, etc.), examples of some or all such blocks are suggested to the user, with content taken from other documents. Such suggestions can be ranked for the user depending on factors such as frequency of use, being the most typical (centroid) of available alternatives, or having a high probability of co-occurrence with other blocks present in the new document. The block suggestions can be automatically updated, e.g., with names, dates, and other sub-blocks specific to the document the example was taken from replaced with values taken from the new document.

[0172] Furthermore, the selection of blocks that are suggested to be added or deleted can usefully depend on the practices of different authors, editors, or other staff. For example, if the current author's document frequently differs from another author's document in a particular way, this can indicate that the difference is a considered choice, rather than an error. On the other hand, if all authors working under the same supervisor do things one way, but the current author does things differently, this can indicate a greater need for review, at least on first notice.

[0173] The modeling of anomalies takes into account structure and block data types and semantic roles, as well as context, content, and format. For example, patterns of what data types and semantic roles of blocks appear inside, adjacent to, or otherwise proximate to other blocks are modeled. Violations of established patterns can be classified as anomalies, and presented to user feedback like any other anomaly.

[0174] 12) Arbitration

[0175] Many of the preceding steps create and / or manipulate blocks of the document, which are defined as (usually but not necessarily contiguous) ranges of characters, tokens, and / or non-textual objects within the linear sequence(s) produced in step (3).

[0176] Blocks under consideration at any point can be represented by "in-line" meta-information such as markup, or can be represented by "isolated" notation, which refers to locations in the text by various pointers. In this example, isolated notation is used for most processing, but in-line notation is used for some purposes, such as communication with external tools, which often prefer in-line notation. These and other notations are functionally interchangeable, and the choice between them can be determined by considerations of performance, convenience, and the like.

[0177] The representation of blocks includes information about what steps or implementations created them, how certain they are ("confidence level"), and their specific data type and / or semantic role labels. It can frequently occur that blocks are redundant, uncertain, conflicting, or partially overlapping, which we call "non-optimal" here. For example, two or more different processes can attach semantic role labels to the same span of text (or nearly the same span, e.g., one including "Dr." before the name and one not). Blocks can be nested, sometimes deeply, but also arbitrarily overlapping (that is, where each of the overlapping blocks contains some content that is also in another overlapping block, and contains some content that is not). Throughout the above steps, the system can maintain a representation that can represent a large number of annotations, including overlapping or co-located annotations.

[0178] Such non-optimal chunks are generally undesirable, at least when the document is presented to a user. Furthermore, many existing NLP tools prefer non-overlapping structures, as are many document tools and methods familiar to those skilled in the art, such as XML, JSON, SQL, and other representation systems. The more restricted structures that are generally preferred are often referred to as "hierarchical" or "well-structured," and avoid partially overlapping chunks.

[0179] This aspect of the system modifies the set of chunks to be strictly hierarchical, and avoids non-optimal chunks. This can be accomplished in a number of ways. First, chunks can be deleted entirely (that is, the chunks themselves; the document content they identify is not deleted). Second, chunk ranges can be modified (for example, by including or excluding one or more characters or tokens from either end) to prevent overlap with another chunk(s). Third, chunks can be determined to be redundant and merged. Fourth, chunks can be found to be contradictory (for example, if one tool thinks "Essex" is a region, and one tool thinks it is a person), and a choice is made.

[0180] The process includes the following means: quickly finding partially and / or completely overlapping cases; comparing chunks by type, role, and confidence; and resolving non-optimal cases by modifying chunks and their associated data. The choice of what chunks to modify, merge, or delete takes into account a number of factors, such as: confidence level; prior probability of a given chunk data type, semantic role, and content; hierarchical relationships between semantic role labels; conditional probability of occurrence given a context; number, roles, and distribution of other chunks in the current and other similar documents; priority of the process at the time; customer feedback on similar cases; and / or other methods.

[0181] Modifications can also change chunk confidence levels. For example, multiple aspects of the system can apply similar or identical semantic role labels to the same or nearly identical portions of a document. In this case, the labels will generally be merged, and the resulting chunk assigned a higher confidence than the individual chunks it contains. In some other cases, a choice is made between mutually contradictory chunk assignments, but the chosen chunk can end up with a reduced confidence to reflect that there is some level of disproof.

[0182] This process improves the quality and consistency of block identification and labeling, enables information to be interoperable with a wide variety of tools, and enables results to be more easily and reliably analyzed. The operations just described can be applied at any time(s), not just at the end. For example, if a previous step uses an external tool for a certain subtask, it can request reduction to well-formedness. Blocks that are removed or modified can instead be "suspended," which means that they no longer affect processing, but can be reintroduced on demand; this enables such use of tools that do not support overlap without having to recreate previous work from scratch afterward, and improves processing flexibility and speed.

[0183] In one approach, all overlaps and / or all non-optimal blocks are resolved before the document is generated for presentation to the user, so that the result can easily be encoded in a hierarchical format such as the XML format used by many modern word processing programs and other tools. However, it is also possible to maintain multiple possible overlaps at a particular location (even in XML) for potential later resolution, such as by user feedback or improved algorithmic learning.

[0184] 13) DGML (DocuGami markup language)

[0185] An enhanced version of the document represents the document structure, format, content, and identified blocks, and can identify which steps of the process identified which blocks, and with what level of confidence. Some embodiments use XML as the syntax for this representation, although a wide variety of representations can contain essentially the same information, such as other XML schemas, JSON, various databases, custom text or binary formats, and so on.

[0186] In this step, the document and information about the blocks discovered for it are converted (or "serialized") into an XML form that can easily be passed to other processes, most notably a front-end user interface for feedback, editing, and review; and a format that can be used for "dashboard" applications that provide overview, statistics, and compliance information to other users, such as group managers, quality control personnel, and so on.

[0187] DGML, the Docugami markup language, is a special XML schema for this purpose that contains all the described information in one package. Most previous schemas can handle structure, content, and sometimes also typography, but do not annotate "blocks" in the abstract as described here. Many previous schemas also do not provide a general mechanism in which blocks can be detected and represented automatically in the run, especially with confidence levels and provenance information.

[0188] For some word processing programs and other tools' file formats, it can also be possible to "tunnel" the information by representing the same information in a form that is transparent to the format. For example, if the tool supports embedded comments or metadata, "invisible" text, ignorable attributes, or other similar features, the information described herein can be hidden within them, allowing use and possibly modification of the resulting document in the tool; and the tunneled information can still be available when returned to the system.

[0189] 14) Feedback mode frontend

[0190] The extensive annotations and analysis attached to the documents and their discovered blocks by the methods already described make it feasible to guide the user in producing a new document that is similar but customized for the current need, by editing a sample, template, or previous document. For example, the system will typically identify: the parties and properties of a contract; the medications or conditions listed under the history, current findings, and other specific sections of a clinical record; relevant dates; and so on. By also examining other documents of the same document set, the system learns which things are unusual, common, or required, and can therefore make more useful recommendations to the user of what to review and / or update. For example, an effective date can be present in nearly every contract in a document set, but its value can differ in each contract. Similarly, the parties change, but the kinds of parties are much more consistent.

[0191] i) Unguided feedback

[0192] In the interaction with the user, the system first requests feedback about blocks that were found (or possibly not found) in several documents. The first few documents presented for feedback will be the "cluster centroids" of the document set. The last few documents will be the "outliers" in the document set.

[0193] ii) Guided feedback

[0194] After that, the system guides the user to provide feedback by showing selected portions of the documents and asking for their current or potential labels, their scope, and so on.

[0195] a. "Labels of interest" are determined by a PageRank-based algorithm with syntax and structure models. Among those labels, a set of low-confidence instances is selected for review.

[0196] b.When there are no more low-confidence labels in the current document, the same process can be repeated for additional documents. In some embodiments, the model is continuously updated according to user-provided feedback. However, feedback can be accumulated and applied later, in batches and / or offline. In turn, adjustments to the model can affect the selection of blocks and labels presented for feedback thereafter, and can trigger re-analysis of some documents.

[0197] c.The system uses essentially the same mechanism to solicit feedback for field and structural blocks. In one approach, all block detectors provide an estimate of confidence, which can be used with other information to select candidates for feedback.

[0198] Feedback can be requested in different passes for smaller blocks versus larger blocks, fields versus structural blocks, or in other orders. See Figure 3 This is an example user interface for user feedback. It displays some or all of the blocks, and allows the user to select a particular block for inspection, see the assigned type and / or role, and optionally alternatives. The user can move block boundaries, select or edit labels, etc. Preferably, the user can also request that a particular change, such as to a label, be applied to all corresponding or same-type blocks.

[0199] 15) Feedback response

[0200] i) Cohort querying is a method that allows the system to query both private and public data based on user feedback, typically from multiple users. The selected examples are similar to previous failures both semantically and syntactically, which increases the value of the feedback.

[0201] ii) Scheduler. The scheduler is a method for connecting user feedback on the combined output of multiple ML models and non-ML algorithms back to the specific learning model 120 that can learn from the feedback.

[0202] The system allows the model to improve from user feedback on its output, as well as from user feedback on the output of other learning and non-learning models. This is achieved by using the feedback as incremental (also known as "fine-tuning") training data for the multiple numerical and neural models described. After feedback is used to improve the model, not only is the specific document re-evaluated, but all documents in the document set, or even all documents of the user, are re-evaluated. Thus, feedback on each document can improve block identification, role assignment, structure discovery for all documents, and thus improve user assistance. This re-training is represented by the dashed line from step (15) to step (3) in Figure 1

[0203] ​The documents and all associated information help to learn and analyze the document set (especially but not only within the particular document set), and thus improve performance on future documents. For example, once a new block is added to one or more documents in the document set, the block can be used in future documents (or revisions of older documents), and can be suggested for use in future documents. At some point, the absence of a recently-introduced block role or the presence of a recently-less-used block role can become anomalous. This point can be chosen by the user, either proactively or in response to a feedback question, or automatically based on usage curves of corresponding blocks over time. For example, if few or no documents in a document set created prior to a certain time have a given role and / or context block (such as the "exclusion" section), but most or all later-created documents do, then the absence of the corresponding block in a new document can be anomalous, and can be usefully suggested to the user as such.

[0204] 16) Downstream communication

[0205] After the documents are annotated with block information such as described earlier, the selected information is converted to a particular format required by external business information systems (such as databases, analytics tools, etc.), and passed to those systems, either directly or through an automatic and / or human review step. For example, the name and address of a particular party can be copied into the correct fields in a database, which can not be done automatically if they are only identified as "name" and "address" themselves. See Figure 4 , which is an example of integration with a downstream software application. In this example, blocks representing terms to which a party is expected to agree have been extracted, and they are passed to a downstream application like Docusign to be filled out and signed.

[0206] Figure 5 is a block diagram of one embodiment of a computer system 510 that can be used with the present application. The steps described above can be implemented by software executing on such a computer system. The computer system 510 typically includes at least one computer or processor 514 which communicates with a bus subsystem 512 that interconnects elements of the computer system 510. Generally, a computer can include, or the processor can be, any of the following: a microprocessor, a microcontrolle, a graphics processing unit, or a digital signal processor, as well as their electronic processing equivalents, such as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA). These peripheral devices can include a storage subsystem 524 (which includes a memory subsystem 526 and a file storage subsystem 528), user interface input devices 522, user interface output devices 520, and a network interface subsystem 516. Input and output devices allow user interaction with the computer system 510.

[0207] The computer system can be a server computer, a client computer, a workstation, a mainframe, a personal computer (PC), a tablet computer, a rack-mounted "blade", or any data processing machine that is capable of executing instructions (sequential or otherwise) that specify actions to be taken by that machine. The computer system can be a server computer, a client computer, a workstation, a mainframe, a personal computer (PC), a tablet computer, a rack-mounted "blade", or any data processing machine that is capable of executing instructions (sequential or otherwise) that specify actions to be taken by that machine.

[0208] The computer system typically includes an operating system, such as Microsoft's Windows, Sun Microsystems' Solaris, Apple Computer's MacOs, Linux, or Unix. The computer system can also typically include a basic input / output system (BIOS) and processor firmware. The operating system, BIOS, and firmware are used by the processor to control the subsystems and interfaces connected to the processor. Typical processors that are compatible with these operating systems include Pentium and Itanium from Intel, Opteron and Athlon from AMD, and ARM processors from Anm.

[0209] The innovations, embodiments, and / or examples of the claimed invention are not limited to conventional computer applications, nor are they limited to programmable devices that run them. For example, the claimed innovations, embodiments, and / or examples can include optical computers, quantum computers, analog computers, etc. The computer system can be a multi-processor or multi-core system, and can use or be implemented in a distributed or remote system. The term "processor" is used in its broadest context to include single-processors and multi-core or multi-processor arrays, including graphics processing units, digital signal processors, digital processors, and combinations of these devices. Moreover, although only a single computer system or single machine can be shown, the use of the singular forms of these terms should also be taken to include a collection of computer systems or machines that individually or jointly execute instructions to perform any one or more of the operations discussed herein. Because of the ever-changing nature of computers and networks, Figure 5 The description of the computer system 510 depicted in the preceding is intended only as an example for the purposes of illustrating preferred embodiments. Many other configurations of the computer system 510 are possible having more or less components than the computer system depicted in the preceding. Figure 5 The computer system depicted in the preceding can have more or less components than the computer system depicted in the preceding.

[0210] The network interface subsystem 516 provides an interface to an external network (which includes an interface to the communication network 518) and is coupled via the communication network 518 to corresponding interface devices in other computer systems or machines. The communication network 518 can include many interconnected computer systems, machines, and communication links. These communication links can be wireline links, optical links, wireless links, or any other devices suitable for communicating information. The communication network 518 can be any suitable computer network, such as a wide-area network (such as the Internet), and / or a local-area network (such as an Ethernet). The communication network can be wired and / or wireless, and the communication network can use encryption and decryption methods, such as can be used with virtual private networks. The communication network uses one or more communication interfaces, which can receive data from and transmit data to other systems. Embodiments of communication interfaces typically include an Ethernet card, a modem (e.g., telephone, satellite, cable or ISDN), a (digital) subscriber line (DSL) unit, a firewire interface, a USB interface, and so on. One or more communication protocols can be used, such as HTTP, TCP / IP, RTP / RTSP, IPX, and / or UDP.

[0211] The user interface input devices 522 can include alphanumeric keyboards, keypads, pointing devices (such as mice, trackballs, touchpads, stylus, or graphics tablets), scanners, touchscreens incorporated into the display, audio input devices (such as voice recognition systems or microphones), eye gaze recognition, brainwave pattern recognition, and other types of input devices. Such devices can be connected by wired or wireless means to the computer system. In general, use of the term input device is intended to include all possible types of devices and ways to input information to the computer system 510 or to the communication network 518. User interface input devices are usually under control of a user and the user uses the input devices to control the operation of a computer system in some way.

[0212] The user interface output devices 520 can include display subsystems, printers, or non-visual displays such as audio output devices. Display subsystems can include flat-panel devices (such as liquid crystal displays (LCDs)), projection devices, or some other device for creating a visible image, such as a virtual reality system. Display subsystems can also provide non-visual display, such as via audio output or haptic output (e.g., vibration) devices. In general, use of the term output device is intended to include all possible types of devices and ways to output information from the computer system 510 to the user or to another machine or computer system.

[0213] Memory subsystem 526 generally includes a number of memories including a main random access memory (RAM) 530 (or other volatile memory device) for storage of instructions and data during program execution and a read only memory (ROM) 532 in which fixed instructions are stored. File storage subsystem 528 provides persistent storage for program and data files, and can include a hard disk drive, a floppy disk drive along with associated removable media, CD-ROM drive, optical drive, or removable media cartridges. The databases and modules implementing the functionality of certain embodiments can be stored by file storage subsystem 528.

[0214] Bus subsystem 512 provides a mechanism for letting the various components and subsystems of computer system 510 communicate with each other as intended. Although bus subsystem 512 is shown schematically as a single bus, alternative embodiments of the bus subsystem can use multiple buses. For example, the main memory can be directly accessed by the file storage system using a direct memory access (DMA) system.

[0215] While the DETAILED DESCRIPTION contains many specifics, these should not be construed as limiting the scope of the application but merely as illustrating different examples. It should be appreciated that the scope of the disclosure includes other embodiments not discussed in detail above. Numerous other modifications, changes and variations of the methods and apparatus disclosed herein can be made in light of the above disclosures without departing from the spirit and scope of the appended claims. Accordingly, the scope of the application should be determined not with reference to the above description, but with reference to the appended claims and equivalents thereof.

Claims

1. A method implemented on a computer system executing instructions for analyzing and improving documents, the method comprising: accessing a document set containing a plurality of documents, wherein the document set also identifies blocks within individual documents of the document set; automatically assigning semantic role labels to a plurality of the blocks, wherein the semantic role labels describe semantic roles played by the blocks in transactions described by the documents; and automatically assigning semantic role labels to the blocks includes using machine learning and / or natural language processing methods to determine semantic roles of blocks; and automatically assigning semantic role labels to the blocks is also based on patterns of occurrence of counterpart blocks in different documents of the document set, wherein the counterpart blocks are different blocks in different documents that play the same semantic role within their respective documents; and using the blocks and their semantic role labels in further processing of documents in the document set.

2. The method of claim 1, wherein the plurality of documents in the document set are all of the same document type.

3. The method of claim 1, wherein the blocks in the document set include: field blocks containing content within the documents that are suitable for use as fields in a document template, wherein some of the field blocks are hierarchical and contain other blocks as sub-blocks; and structure blocks containing content that includes structure within the typography of the documents.

4. The method of claim 1, wherein the document set contains legal documents; and the semantic roles include roles played by parties to the legal documents, and roles played by dates, time periods, or other temporal expressions.

5. The method of claim 1, wherein automatically assigning semantic role labels to blocks includes: automatically extracting some of the semantic role labels from blocks; and assigning the extracted semantic role labels to blocks.

6. The method of claim 1, wherein automatically assigning semantic role labels to blocks includes: automatically extracting semantic role labels from blocks using machine learning by: based on content, typography, and context in individual documents; based on patterns across content, typography, and context in the documents in the document set; and based on data types of blocks; and assigning the extracted semantic role labels to blocks.

7. The method of claim 1, wherein automatically assigning semantic role labels to blocks includes: automatically extracting some of the semantic role labels using autoencoder machine learning techniques; and assigning the extracted semantic role labels to blocks.

8. The method of claim 1, wherein automatically assigning semantic role labels to blocks includes: automatically extracting candidate semantic role labels from the blocks; refining the candidate semantic role labels using machine learning; and assigning the extracted semantic role labels to blocks.

9. The method of claim 1, wherein automatically assigning semantic role labels to blocks includes: automatically extracting some of the semantic role labels from the chunks based on similarity of content, typography, and / or context of the chunks from different documents in the document set; and assigning the extracted semantic role labels to the chunks.

10. The method of claim 1, wherein automatically assigning semantic role labels to chunks comprises: assigning candidate semantic role labels to chunks; grouping the chunks into clusters based on similarity of the semantic roles played by the chunks; standardizing the candidate semantic role labels among the chunks in a cluster; and assigning the standardized semantic role labels to the chunks.

11. The method of claim 1, wherein automatically assigning semantic role labels to chunks comprises: assigning candidate semantic role labels to chunks; grouping the chunks into chunk clusters based on similarity of size and text embeddings of the chunks; grouping the candidate semantic role labels into label clusters based on similarity of text embeddings of the candidate semantic role labels; standardizing the candidate semantic role labels based on the chunk clusters and the label clusters; and assigning the standardized semantic role labels to the chunks.

12. The method of claim 1, wherein automatically assigning semantic role labels to chunks comprises: assigning candidate semantic role labels to chunks comprising sections of documents, wherein the candidate semantic role labels are based on headings of the sections; grouping the chunks into clusters based on similarity of content in the sections; standardizing the candidate semantic role labels by selecting the most frequent candidate semantic role label as the semantic role label for all chunks in a cluster; and assigning the standardized semantic role labels to the chunks.

13. The method of claim 1, wherein the semantic role labels are selected from a predetermined set of semantic role labels.

14. The method of claim 1, wherein the semantic role labels comprise: labels identified by a software application for further processing of documents in the document set.

15. The method of claim 1, wherein automatically assigning semantic role labels to chunks comprises at least one of using machine learning to determine semantic roles of chunks based on other chunks nearby or based on containing chunks containing the chunks, or using natural language processing methods to determine semantic roles of chunks based on syntactic structure of nearby chunks.

16. The method of claim 1, wherein some of the chunks are named entity references, such chunks are labeled with semantic role labels of the semantic roles played by those chunks in the document, and such chunks are also labeled with data types of the chunks.

17. The method of claim 1, wherein some of the chunks are multi-paragraph structures in the document, and such chunks are labeled with semantic role labels of the semantic roles played by those chunks in the document.

18. The method of claim 1, further comprising: estimating a confidence level of the automatically assigned semantic role labels; presenting some of the assignments to a user for confirmation based on the estimated confidence levels; receiving user feedback for the automatically assigned semantic role labels; and improve the machine learning and / or the natural language processing method in response to the user feedback.

19. A non-transitory computer-readable storage medium storing executable computer program instructions for analyzing and improving documents, the instructions executable by a computer system and causing the computer system to perform a method comprising: accessing a document set containing a plurality of documents, wherein the document set also identifies blocks within individual documents of the document set; automatically assigning semantic role labels to a plurality of the blocks, wherein the semantic role labels describe semantic roles played by the blocks in transactions described by the documents; and automatically assigning semantic role labels to the blocks includes using a machine learning and / or natural language processing method to determine semantic roles of blocks; and automatically assigning semantic role labels to the blocks is also based on patterns of occurrence of counterpart blocks in different documents of the document set, wherein the counterpart blocks are different blocks in different documents that play the same semantic role within their respective documents; and enabling the blocks and their semantic role labels to be used for further processing of documents in the document set.

20. A computer system for analyzing and improving documents, the computer system comprising: a storage medium for receiving and storing a document set containing a plurality of documents, wherein the document set also identifies blocks within individual documents of the document set; and a processor system having access to the storage medium and executing an application program for analyzing and improving documents, wherein the processor system performs the application program: automatically assigning semantic role labels to a plurality of the blocks, wherein the semantic role labels describe semantic roles played by the blocks in transactions described by the documents; and automatically assigning semantic role labels to the blocks includes using a machine learning and / or natural language processing method to determine semantic roles of blocks; and automatically assigning semantic role labels to the blocks is also based on patterns of occurrence of counterpart blocks in different documents of the document set, wherein the counterpart blocks are different blocks in different documents that play the same semantic role within their respective documents; and enabling the blocks and their semantic role labels to be used for further processing of documents in the document set.

Images