Cross-document intelligent authoring and processing assistant

Abstract

To analyze a document to annotate it.SOLUTION: Machine learning, artificial intelligence, and other computer-implemented methods are used to identify semantically important chunks in documents, the semantically important chunks are automatically labeled with appropriate datatypes and semantic roles, and this enhanced information is used to assist authors and to support downstream processes. Chunk locations, datatypes, and semantic roles can be automatically determined from what is called "context", to wit, combination of their formatting, structure, and content, those of adjacent or nearby content, overall patterns of occurrence in a document, and similarities of all these things across documents. Similarity may include similarity of natural language grammatical structure, machine learning techniques such as measuring similarity of word, chunk, and other embeddings, and the datatypes and semantic roles of the previously-identified chunks.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure generally relates to methods and apparatus for AI self-supervised creation of hierarchically semantically labeled documents and / or for assisted authoring and processing of such documents.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority under 35 USC § 119(e) to U.S. Provisional Patent Application No. 62 / 900,793, entitled "Cross-Document Intelligent Authoring and Processing Assistant," filed September 16, 2019. All of the subject matter set forth above is incorporated herein by reference in its entirety. [Background technology]

[0003] Description of related technology Many businesses produce multiple documents that are very similar, albeit customized each time. For example, an insurance office may have many proposals for a particular type of insurance, but each must be tailored to a specific customer's needs. These documents can be considered to be of the same "type" because they have similar textual (and sometimes image) content (reflecting similar purposes and topics), similar selection and arrangement of macrounits such as sections, and often similar geometric layout and formatting characteristics.

[0004] While some types of documents are widely known and used, many are not. Many are specific to particular businesses, markets, or applications, and new ones are created to fit new situations. A user, who may be called an "author" or an "editor," typically creates a new document of a particular type (sometimes called a "target document") by copying a previous document of the same type and modifying it as needed, e.g., manually editing or replacing particular chunks of content.

[0005] In current practice, word processors typically identify chunks only when necessary to achieve formatting; for example, headings, footnotes, and figures may be explicitly marked to obtain special formatting, but names, addresses, or dates are rarely explicitly marked. Even when identified, chunks are generally associated only with formatting effects (margins, fonts, etc.) that are useful information, but no indication of either these data types or semantic roles is directly provided. Similarly, word processors often only represent hierarchical containment visually, and often there is no explicit representation of the nested sections themselves, only differently formatted headings.

[0006] When creating a new document of the same general type as a previous document, in many cases the bulk of the work will be text editing, replacing, deleting, or inserting specific chunks, being careful not to confuse those with different semantic roles (such as swapping the buyer and seller addresses.) This typically requires human intervention, as the authoring system will usually know nothing about these chunks, especially their data types or semantic roles, and will not be able to help very effectively.

[0007] In some simple cases, "forms" and "templates" may be used that provide explicit locations for filling in the content of a particular chunk. However, forms typically only address simple cases where substantially all necessary chunks can be enumerated up front and few large, repeatable, or highly structured chunks exist. Also, creating forms requires skilled effort, is difficult to adapt to changing circumstances, and does not actively assist the writer. [Brief description of the drawings]

[0008] The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication and color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

[0009] Embodiments of the present disclosure have other advantages and features that will become more readily apparent from the following detailed description and the appended claims, taken in conjunction with the accompanying drawing examples.

[0010] [Figure 1] FIG. 1 is a block diagram of one implementation of a system and process for creating hierarchically semantically labeled documents using machine learning and artificial intelligence. [Diagram 2] FIG. 2 is a screenshot showing a dashboard that tracks the processing of different document sets through the system of FIG. [Diagram 3] FIG. 3 is a screenshot of a user interface for receiving feedback from a user. [Figure 4] Figure 4 is a screenshot of the integration with other software applications. [Diagram 5] FIG. 5 is a block diagram of one embodiment of a computer system that may be used with the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Detailed Description of the Preferred Embodiments overview A group of documents that are determined to be of the same type constitute a "document set" or "document cluster." For example, an insurance company's offerings for a particular type of insurance to a particular class of customers may be considered the same type and form a document set. Different types of insurance offerings for the same company, or offerings to customers that they deem different, may be considered different types that belong in different document sets. Rental agreements, clinical notes for particular types of patients, sales proposals, schedules, meeting minutes, etc. are other potential types of documents, as well as subtypes that share characteristic patterns of content, structure, and / or layout.

[0012] Creating and editing new target documents within a document set very often involves editing or replacing "chunks" that are "semantically significant"; such chunks are typically specific parts of documents, not necessarily contiguous spans of text, but that have particular data types and semantic roles and that have meaning and significance to a business or other process.

[0013] These chunks are of various data types, and in many computer systems are finer than atomic data types. For example, a given chunk may represent not just a string of characters, but also a person or organization name, a date, a time period (which is not exactly the same thing as a date), a currency amount. Larger chunks may contain bundles of information such as lists of drugs or other substances, travel itineraries, procedures to follow, medical prescriptions, and countless other things.

[0014] Additionally, chunks may have semantic roles relative to the documents in which they occur. For example, a person's name may be the "tenant" in a lease, or the "seller" in a sales proposal, or the "agent" of another person. A date may represent the start or end of some responsibility or activity. A dollar amount may be a periodic payment amount, or a penalty or bonus associated with certain conditions, and so on. Such semantic roles are important to enforce the appropriate use of the information in the chunks. The names of semantic roles are called "semantic role labels" or simply "labels".

[0015] A chunk is typically represented as a bundle that includes its location, data type, semantic role, and / or other data / metadata. Location is commonly represented as a start and end point and can be represented in several ways, such as an inserted marker or a byte, character, or token offset (either global to the document or relative to an established ID, marker, or other object). Semantic roles are represented by labels or other identifiers. Chunks can be of any size, and some can contain other chunks as "sub-chunks". Chunks can contain not only text, but also non-text data such as images or other media, and "structure" such as tables, lists, sections, etc.

[0016] The techniques disclosed herein use machine learning, artificial intelligence, and other computer-implemented methods to identify various semantically significant chunks in a document, automatically provide them with appropriate data types and semantic roles, and use this enhanced information to assist authors and support downstream processes. The location, data type, and semantic role of chunks can be automatically determined from what is often called "context", i.e., their combination of format, structure, and content, that of adjacent or nearby content, the overall pattern of occurrence within the document, and the similarity of all these things across documents (primarily, but not exclusively, among documents in the same document set). "Nearby content" includes horizontally nearby content, such as preceding and succeeding in the reading sequence of the text, but also vertically nearby content, such as within the same container structure, such as lists and sections, with their respective markers, headings, levels, etc. Similarity is not limited to exact or fuzzy string or characteristic comparisons, but may include ML (machine learning) techniques such as measuring the similarity of natural language grammatical structures, the similarity of words, chunks, and other embeddings, as well as the data types and semantic roles of previously identified chunks.

[0017] For example, a person or organization name can be commonly identified as having a semantic role such as "seller" because the document states so, very often using a sentence in some human languages, but often also including a larger context. In another example, one or more words can often be easily identified as representing a data type such as "medication name", but context is necessary to determine that it takes on the semantic role of allergy rather than prescription. Often, important evidence of a semantic role is not expressed in the same sentence, but in various other ways, such as chunks that occur in larger chunks (such as a "known allergies" section). The flexibility and variety of grammatical and document structural constructs (not to mention typos, transcription errors, etc.) does not make data type identification any more difficult, but it does make identifying semantic roles, especially those that have a scope larger than a single sentence, very difficult.

[0018] A given semantic role may relate a chunk to the document as a whole, or to other chunks (for example, a flight departure time may be tied to a particular "leg" in a journey and only indirectly related to others). Typically, the chunk hierarchy groups such items appropriately, such as within a section, as table parts, etc.

[0019] More specifically, examples of semantically significant chunks include the names, addresses, and other features of specific parties to a contract, prescribed medications and prohibited procedures in a medical record, requirements (or exclusions) in a real estate proposal, dates and flight numbers in a travel itinerary, etc. All of these can be considered semantic roles of chunks. There are also larger chunks with various types and roles, such as entire sections and subsections. These are often inserted or removed as a whole, possibly with changes to smaller chunks within as well. Chunks can be hierarchical, i.e., a larger "containing" chunk can contain other "sub-chunks" up to any number of levels.

[0020] A chunk is generally a contiguous sequence of words in a document, such as "John Doe." However, chunks can contain partial words: "John Doe's House" contains a name, but the name ends before an apostrophe (in the middle of a word). Chunks can even be discontinuous, for example the same name of "John (also called "Bill") Doe." Layout can also make chunks discontinuous, for example a page break can occur in the middle of a chunk (perhaps with a page header, footer, or footnote that may be ignored for some purposes), or there may be an intervening figure, table, chart, sidebar, or other display.

[0021] The actual location and context of a chunk may also be important: a chunk is not just an isolated string, but may occur many times with different (or none at all) semantic roles for different instances. More modern systems usually support inline or stand-off markup, sometimes called "annotations", that can persistently associate various labels and other information with chunks. For example, HTML provides tags for manually labeling the boundaries of common structural chunks (e.g., "div", "ol"), and several broad types or roles for (typically) smaller chunks (e.g., "initials", "kbd", "dfn", "cite", etc.). Other XML schemas provide many other labels, and word processors allow somewhat similar labeling via "styles".

[0022] Some chunks may represent what are commonly called "fields." These are often small chunks that often occur in similar contexts and layouts in many or all documents in a given set, but usually with different text content in each. They may also occur multiple times in a single document with the same or very similar content. Such chunks may be called "field chunks." They are often manually discovered and treated as "fields" in template-based systems, but here they are discovered by context and patterns of occurrence in and across documents, and assigned data types and semantic roles in much the same way as other chunks. They may or may not represent named entities, such as personal names, addresses, dates, etc.

[0023] Another common type of chunk may be called a "structure" or "structural" chunk. Such chunks are typically larger and often contain many other chunks, some of which may also be structural chunks. They frequently have a "title" or "heading" that provides a name, number, description, and / or other information about the structural chunk. Examples of structural chunks include chapters, sections, tables, figures, sidebars, and many more. The type and semantic role of a structural chunk is often important for determining the types and semantic roles of nearby or contained chunks.

[0024] Not only data types, but also specific semantic roles are important to properly create and utilize documents. Whether a particular name represents a buyer versus a seller or a patient versus a doctor, whether a given date is the start or end date of a requirement or the departure versus arrival time of a flight, whether a numerical value specifies a principal, interest, dosage, temperature, penalty, or something else, is very important. For larger chunks, roles include "limitation of liability" statements versus "governing law" specifications versus "definitions," and countless others. The semantic role of a chunk is often specific to a particular domain or transaction, and is perhaps one of the most important features of a document. For many kinds of documents, chunks with specific data types and semantic roles are necessary, or at least very common, and chunks are called "counterparts" when they correspond to the entire document. Counterpart chunks may occur in similar orders and patterns, especially in documents by the same author or organization, and typically for the same set of documents. Counterpart chunks have the same or very similar roles, and generally have similar contexts and / or formats. Thus, the distribution of chunks' data types and semantic roles provides valuable information for distinguishing types of documents, as well as helping to identify counterpart chunks in other documents.

[0025] While many counterpart chunks have similar content, there are others that do not. For example, the same parties (semantic roles) in different documents are usually different individuals, but appear in very similar contexts and patterns of use. This can be particularly common with, but not limited to, "field chunks".

[0026] Once discovered, the hierarchical semantic chunks with data types and semantic roles within a business document can be used in downstream business processes. For example, a back office database can properly record a new mortgage given the names of certain parties, a certain date, term and numerical values ​​such as interest rate. In particular applications such as these, semantic roles are very important and entering the correct data type into the wrong database field (such as swapping the seller and buyer names or addresses) can be a major problem, especially when moving information to downstream databases, processes or reports.

[0027] Some Features and Benefits The techniques described herein may have a variety of features and advantages, including any of the following.

[0028] Some implementations may provide easier, more efficient, and more accurate ways to generate documents with hierarchically organized chunks with semantic labeling useful for business processes. This may be accomplished using various techniques to identify such chunks of various sizes, discover the data types and semantic roles they play in the document, and learn patterns of their use, characteristic contexts, and the like. Learning may come from analysis of the content, structure, and format of current and previous documents, feedback from authors and editors, and comparison of multiple documents, especially those within the same document set. With this knowledge, the system can provide valuable assistance to users, such as, for example, facilitating the creation of new documents of higher quality, extracting desired information for downstream use in other software applications, in back-office databases, derived reports, compliance checks, and the like. Such learning may be performed using unsupervised and self-supervised learning techniques that do not require large amounts of pre-labeled or pre-analyzed data, but instead infer patterns from unlabeled or minimally labeled data.

[0029] Some implementations may enable computers to assist the writing process by discovering and using patterns within and across business documents to help writers avoid many of these errors, thus reducing the time required to achieve a given level of quality.

[0030] Today, typical document systems do not identify chunks, or specifically their data types or semantic roles. This adds time and expense for authors and editors, and for importing data from documents into back-end databases, dashboards, or other downstream business processes. For example, it is common to manually search for data (chunk by chunk) from contracts and copy it into spreadsheets or data entry forms.

[0031] Some implementations may help to label such hierarchical semantic chunks during the authoring process and represent them explicitly, thus making it easier for people and / or computers to extract and connect them to various kinds of other business processes, saving time and money.

[0032] Current techniques typically do not fully exploit similarities between multiple documents created by the same writer or group and / or of the same type (here indicated by membership in a particular document set) to more reliably identify new chunks of documents or to draw attention to differences that may be important. Explicit rules, such as requiring a section headed "Separability," only cover similarities that analysts quickly notice and explain, are static and often constrained (e.g., cases of omissions with rephrasing or reorganization, or failure to address opposing conditions), and quickly become outdated. Smaller companies often lack the resources necessary to develop more responsive techniques, and often have too few documents to justify the expense. On the other hand, smaller companies often have a less diverse range of documents, making them more suitable for automated analysis such as that described herein.

[0033] Some implementations may use the extracted information about chunks and their content, context, layout, and patterns of use across documents to help writers create new documents. Examples include, at a minimum, suggestions such as changing, reformatting, or moving specific content, phrases that are commonly present in similar documents but are missing in the new document (referred to as "missing" or "potentially omitted" chunks or content), phrases that are not commonly present in similar documents but are present (referred to as "unusual" chunks or content), and changes such as swapping names or roles of different parties in specific places.

[0034] Some implementations may accept and retain user feedback, such as when a user indicates that a chunk is labeled with the wrong range, data type, or semantic role, that they are not interested in them, or fail to label them altogether. Some implementations may use specific user corrections to improve machine learning and neural models, and remember not to repeat previous suggestions in cases where the user rejects them (even if additional learning fails to prevent instances of a particular mistake). In particular, some implementations may avoid requiring a large amount of review steps or corrections in favor of few-shot learning techniques and careful selection of feedback to request, to minimize the amount of user action required. Some current techniques learn very specific things, for example, when a user instructs a spell checker to add a word to its dictionary. However, this involves just a rote list, rather than iterative training or fine-tuning of the model used to determine the behavior after refinement, and thus does not fully utilize the capabilities as described herein.

[0035] Some implementations may use small amounts of user correction to learn and improve their behavior while avoiding annoying the user with repeated suggestions when reapplying an improved but still imperfect model.

[0036] Many businesses record certain information derived from documents in various types of databases that support their processes. For example, a company that owns many rental properties typically uses a back-end system to help manage certain information derived from those rental agreements, such as renter payments, approved pets, pre-paid damages for which the renter is not responsible, or other information. Auto or tool renters, mortgage companies, health care providers, local governments, and other organizations use other information. Many goods and services have numerous mix-and-match options, and supervisors review statistics on their acceptance, mix, pricing, and other factors. Business information systems generally provide analytics, check for consistency or compliance, derive reports, and / or support other business processes, all of which can be facilitated by the use of chunked information as described herein.

[0037] Typically, chunks and the information they provide are scattered throughout prose text, manually extracted, and manually entered into spreadsheets, databases, or other systems. Previously, manual work was necessary because of the flexibility of natural human language, as written in contracts, emails, etc., and the similarly variable layout and presentation conventions, which allow important chunks to be expressed in a myriad of different ways. The negotiations underlying such documents are also often scattered across multiple document types, including emails, notes from conversations, slide presentations, etc. That information can also be useful, but is typically addressed manually. Some systems may treat such information sources as documents and reap the same benefits already described.

[0038] Some implementations, once transformed as described herein into a hierarchically semantically labeled document, may provide a means for a computer to initiate execution of a particular document. By combining the hierarchically labeled structure of a document with tools that provide a vector-semantic representation of the text, specific chunks can be identified as requiring specific actions. For example, a contract may specify remittance, notification, or other actions, and conditions that enable or trigger them. These can be identified and used to initiate execution of the contract.

[0039] Some implementations may provide an easy way to review and summarize information from a set of documents in an interface such as a "dashboard" and move the identified information to a customer's back-end database or similar system, enabling a more efficient and less expensive flow of business data and enhancing quality assurance, consistency, and reporting. Once the chunks are semantically labeled, it becomes easy to generate summary reports across a set of documents that contain counterpart chunks. Some implementations may provide a very easy way for a user to create such reports by simply clicking on one or more examples of chunks that should be included, which are then located and extracted by role or context across all documents in the set. Some implementations may also assist the user in discovering documents that lack expected counterpart chunks and revising them to include or identify such chunks, or verifying that they do not include them correctly.

[0040] In another aspect, the performance of a given group, such as a company or department, can be enhanced by incorporating information such as the semantic roles of chunks, patterns of occurrence, and other characteristics of their documents and the feedback of their users into the system's learning process and using the resulting improved models to enhance and / or validate future documents. However, many customers do not want to share such information with other customers, and many have binding confidentiality requirements. On the other hand, general information and learnings derived from public, non-confidential sources can be freely used and shared.

[0041] Some implementations may provide the benefits of feedback and learning by keeping each customer's data and any model information derived therefrom separate and private to each customer, while still sharing common learnings based on non-private public data. Keeping these data processes separate also ensures that there is no statistically likely to be information "leaking" from one customer to another.

[0042] Introduction to Exemplary Embodiments The following is a description of an exemplary system. See FIG. 1. The system generally relates to a method and apparatus for AI self-supervised creation of hierarchically semantically labeled documents and / or for assisted authoring and processing of such documents. This includes processes such as composing, structuring, annotating, modifying, reviewing, extracting data from documents, and / or using such data in downstream business processes. More specifically, the system focuses on discovering detailed hierarchical structures of documents consisting of many semantically meaningful chunks associated with their roles, primarily using unsupervised and self-supervised machine learning techniques over a set of documents including a relatively small set, to documents similar to previous documents, and the use of such highly enriched documents in business processes.

[0043] The operation of this exemplary system uses the following processes, which are described in more detail in the following sections. This is merely an example. Other implementations may use different combinations of steps, including omitting steps, adding other steps, and changing the order of some steps. They may also use different implementations of the steps listed below, including different combinations of the techniques described under each step. In FIG. 1, the steps are preceded by an "S" and step 1 below is labeled "S01." 1) Import: Bring a group of a user's documents into the data store 110. 2) Organization: Separating documents into document sets by type, such as rental vs. sales contracts, or medical history vs. current clinical notes. 3) Visual Extraction: Extract a linear text stream from each document based at least on its content and visual layout, including limited information about individual text and other areas, their start and end positions, format, and content. The extracted data may be organized as "visual lines" or as "visual blocks" (also called "hyperlines" or "visual" chunks), such as paragraphs differentiated by their geometric layout. 4) Structure: Identify broad classes of headings, list items, and other structural chunks in the document. 5) Re-nesting: Determine the nesting relationships of sections and lists and the scope of text for each. 6) Topical Chunking: Analyze the topical content of each document and create chunks that enclose areas of similar topics (topic-level chunks). 7) Topic Labeling: i) Using embedding and clustering, we create candidate data type and semantic role labels for each heading in the corpus. ii) Using keyphrase extraction techniques, we create candidate data type and semantic role labels for the chunks. 8) Chunk Labeling: Identifying and assigning data type and semantic role candidates to other chunks throughout the document (possibly multiple) using multiple methods, e.g., neural networks, word and character embeddings, grammar analysis and pattern matching, regular expressions, similarity metrics, and / or other methods. Of particular interest to certain embodiments are the following: i) Syntax analysis and pattern matching on the results structure ii) Using question-answering techniques to link small chunks with the specific semantic roles they play in the document. iii) Combining XPath tree matching and word embedding techniques to match patterns in structure and grammar trees despite potentially significant differences in phrasing and word choice. 9) Named Entity Recognition (NER): Identify data types that are detected as named entities throughout the document and assign them to chunks. 10) Role Labeling, Extraction Labeling: Assigning semantic role labels to chunks, such as a name representing that it constitutes the “seller” party in a contract, or a drug being mentioned as an allergy remedy. 11) Anomaly: Identify semantic roles that are typically present or absent in documents in the document set under consideration but not in the current document (or vice versa). 12) Arbitration: Reconcile and / or select alternative scopes, data types, and semantic role labels for chunks to produce a well-formed structure that is easily representable in a format such as XML. 13) DGML: An augmented version of the document is created that includes explicit identification of chunk locations, data types, and semantic role labels, and possibly additional information such as the confidence level of each identified chunk, data types expected in similar chunks (dates, date ranges, personal names, etc.). The augmented version is created using an XML-based markup language called DGML. 14) Feedback: Display the enhanced version to the user, prompt the user with selected chunks (and possibly the potential locations of omitted chunks), and collect the user's choice to confirm, reject, or make other changes. Users are also free to choose their own reading and review order. Feedback can also be applied to any other interpretations made by the system, such as organizing documents into document sets, as described in step (2). i) In the case of possibly omitted chunks, prioritized examples from other documents can be provided, which can be automatically customized by inspecting and / or copying the current document as necessary and applying target document values ​​to smaller nested chunks. 15) Feedback Response: Tracking the user's responses to these interactions and using that information to fine-tune the model 120 as well as to prevent repeating the same or similar errors later. 16) Downstream Communication, Transmission: Select chunks by type and / or role and use them to generate reports via document sets and / or export them to downstream systems that add functionality such as back-end contract databases, regulatory compliance checkers, management report generators, etc.

[0044] Figure 2 is a screenshot showing a dashboard that tracks the processing of the different document sets 1 to 7 through the above process. In this dashboard, the process is divided into the following stages: Uploading Preprocessing Review Large Chunks Review Small Chunks Ready to Use Color coding indicates the degree of completion: green stages are completed, red stages are in progress, and black stages have not yet started.

[0045] Each of the steps listed above is described in more detail below.

[0046] Further Description of Exemplary Implementations The numbering here reflects the general order of analysis of this particular example. However, not every step is dependent on every previous step, and as a result, many elements can be reordered or parallelized in other implementations. Elements can also be shifted or repeated to exchange additional information with other elements, or elements can be performed independently, in separate processes or machines, etc.

[0047] 1) Import The system accepts typical word processor documents (e.g., MS Word) and page layout documents (e.g., PDF or png files). In each case, visually adjacent regions such as headings, paragraphs, table cells, tables, images, etc. are identified and represented as chunks using a combination of their relative positions, surrounding whitespace, font and layout features, etc. These features are partly selected by the designer and partly learned by image and pattern analysis of a large number of documents. For input documents that do not already have machine-readable text content, OCR is also applied.

[0048] These chunks, along with selected layout information, are presented to subsequent modules in the system.

[0049] 2) Organization Users do not need to organize the documents they check into the system. The system uses clustering methods that operate on text content, layout information, and already detected structural information (such as identifying some headings) to group documents into "sets" of specific types of documents, e.g., rental contracts vs. leases vs. sales. The specific document sets found can be confirmed by the user and are named either automatically or by the user. Once established, these document sets facilitate later machine learning and reasoning about format, content, semantic roles, and their differences. For example, the system may discover that almost all documents in a given set have a specific section with three specific subchunks of a specific role and a data type of a person's name, one of which is reproduced in five different sections. Such patterns are used to help identify similar (and dissimilar) parts of other documents, suggesting changes or modifications to the user, and providing examples of text to reuse in other documents in the same (or possibly different) set.

[0050] Clustering documents into document sets can use features from the document's structure (order and containment relationships between chunks of various sizes, data types, and roles) and layout, as well as textual content. Once some chunks and / or roles have been identified in at least some documents, that information can also be used to improve the clustering, either by fully reclustering or by smaller adjustments. For example, ignoring certain content of chunks with the same role, such as seller and buyer names, addresses, etc., or seeing that the occurrence patterns of different chunks are the same, e.g., one name (e.g., the seller's name) appears in certain places while another name (e.g., the buyer's name) appears in certain other places, can result in similar documents being nearly or even completely identical.

[0051] The system maintains both the original organization of uploaded files into directories (if any) and their own organization into sets. Thus, users can see both organization and learning algorithms can use both as information. For example, some users name documents according to different rules and / or organize documents by customer, document type, or other characteristics, which is almost always useful for understanding similar patterns (such as having common chunk positions and roles) and relationships between documents.

[0052] 3) Visual Extraction i) Area detection The system uses heuristics and machine learning to identify regions in documents based on geometric patterns. For example, in many documents, meaningful chunks have special layouts like signature blocks, abstracts, lists of definitions, tables, etc. Such patterns can be learned automatically by considering geometric and / or layout features, uniqueness or rarity, and / or correspondence either within the same document or between documents, especially within the same set of documents.

[0053] The approach is chosen depending on the format of the input document: for example, word processor documents typically provide explicit information about paragraph boundaries, while PDFs or scanned pages require the system to assemble them from visual lines, or even analyze white space dimensions and assign characters to visual lines (e.g., in multi-column documents).

[0054] ii) Signature detection The system creates signatures (also known as "digests") of document parts and uses these to identify and classify additional "interesting" chunks and to detect their boundaries. The signatures are based not just on textual content but also on various aspects of the context, and may ignore the content of smaller contained chunks (e.g., field chunks whose content changes within their counterparts).

[0055] The signature may even use a pixel representation of the chunks. A bitmap image of the text layout is divided into tiles, preferably on the order of 24 pixels square (adjusted for the scan resolution), and the tiles are clustered. These neural network processes, including the autoencoder and its adjacencies, reveal similar visual events such as boundaries between text and rules, edges and corners of text blocks, and even indentation changes and substantial font / style changes. Further neural networks then use this clustering to jointly identify similar layout objects, which frequently indicate or characterize important chunks.

[0056] The approach here may use an unsupervised approach to generate document chunk embeddings based on the pixels and characters in the document chunk, the size of the chunk, its location in the document, etc. (As noted, images can also be chunks.) Clustering and comparison techniques can then be used on these embeddings for many downstream tasks.

[0057] iii) Extraction This aspect takes a post-layout document (e.g., a PDF or scanned print page) and converts recognized character images ("glyphs") in the document into a text stream that represents the correct document order of the glyphs (the stream may also contain figures or image objects, if appropriate, and there may be multiple streams, such as footnotes or page headers, that do not have a typical place in the reading order). In some documents, there is an incomplete explicit representation of the reading order. A well-known example is that there is usually no indication that a multi-column layout is in effect at any given point, and thus the first "line" extends only halfway (or less) across, rather than the whole way through. However, there are many additional examples where the text order may be complex or unclear. For example, some layout programs draw each character individually, and word boundaries are unclear. Table cells, sidebars, figures, footnotes, and other indications may not have a clear location in the text order. Some text, such as in page headers and footers (as well as hyphens at the end of lines), may not require a place in the text order at all. Many formats do not provide a clear indication that something is in such a special category.

[0058] The system addresses this task by combining visual information of glyphs (position, style, etc.) with a deep neural network that understands the characteristics of the written language used in the document to construct a text stream. Additionally, it detects many basic text boundaries such as lines, blocks, columns, images, inline font changes, and header / footer objects.

[0059] iv) Expression Having extracted the text sequence and some hypothesized structural chunks, the system creates a representation of the document (known as "DGML" in one example) that includes information about them as well as their visual features (font, color, size, etc.). The representation of the chunks, including information such as their location, type, role, etc., is called annotation. The combined data can then be used by natural language processing (NLP) and deep neural networks (DNNs). Deep neural networks incorporate this visual information and help structure the document into a hierarchy that represents the document structure, including chunks such as header / body, lists / list items, etc.

[0060] Sufficient information may be included so that subsequent aspects can construct an editable word processor document that closely resembles the original source. This, along with other structure, content, and chunk information, may be included in a DGML or similar representation. In many cases, parts of a document that have a clear format and layout are also useful chunks. However, formatting characteristics that do not match otherwise required chunks (and vice versa) may still be expressed via special types of chunks, via standoff annotations, or via other means.

[0061] 4) Structure The structural pipeline converts flat text files into a hierarchical structure, where sections, subsections, and other parts of documents form an ordered hierarchy of content-based objects, a structure known to those skilled in the art. This conversion is done using unsupervised machine learning techniques. The method has several stages.

[0062] i) Hyperlining This involves segmenting the text into "hyperlines", which are larger groupings than visual lines and comprise more meaningful logical (as opposed to visual) units such as paragraphs, headings, or the like. This is preferably accomplished using a pre-trained neural network that takes into account features such as "word shape" of tokens (especially leading and trailing tokens), layout information such as font and spacing characteristics, and similar features. Some hyperlines may also be provided in previous steps (depending on the format of the input document).

[0063] ii) Document Language Model Instead of a language model based solely on the text, it preferably uses a document language model that also contains information on the text content, formatting, and previously discovered structure, allowing better detection of chunks and their hierarchies (header / body, list / list items, etc.) by learning to recognize meaningful chunks and patterns of their occurrence from formatted pages.

[0064] This creates a representation of the document that includes both textual content and visual features (geometry, font, color, size, etc.). Deep neural networks and NLP processes then exploit such information in the task of structuring the document into a hierarchy of chunks with data type and semantic role labels by discovering the extent and / or boundaries of chunks of various sizes that represent the document structure. At this stage, the discovered chunks are primarily headings, sections, lists and items, tables, figures, and other relatively large units.

[0065] iii) Hyperline Clustering It uses an autoencoder to cluster hyperlines across a document set based on word-shape structure, and assigns each hyperline to a cluster of hyperlines that are similar in terms of layout, start and end content, and other characteristics, with each cluster identified by a "cluster ID" (this should not be confused with creating or identifying a document set).

[0066] iv) Inline headings A special case of particular interest is an "inline heading", where the heading for a chunk (which may provide a semantic role for the chunk) is on the same line as the start of the text following, rather than being on a separate visual line by itself. Typically, inline headings are distinguished typographically, such as by bolding, underlining, a different font, a trailing colon, or other effects. Separate heuristic and neural algorithms identify these chunks.

[0067] v) Small-shot structure learning Despite the above advanced structuring methods, it can be expected that the generated structures will have certain imperfections or will not meet the user's prior expectations. Few-shot structure learning deals with creating a machine learning model relying on feedback provided by the user, as described in steps (14)-(15). This model is then used to generate structures that combine the user feedback on structures with those already generated by the system (possibly iteratively reinforced by previous feedback).

[0068] The main principles applied in this case are derived from machine translation (MT) methods, where a sequence is transformed into another sequence: in this case, one sequence describing a hyperline is transformed into another sequence that also contains start / end markers that encode the hierarchy.

[0069] This process takes place in various phases or steps. (a) First, a machine translation model is pre-trained using a publicly available dataset. (b) The "Dispatcher" (see the "Feedback Response" section for explanation) filters user feedback. (c) New structure files are generated from user feedback to generate a fine-tuned machine translation dataset. (d) The pre-trained model is further trained using few-shot learning principles.

[0070] 5) Re-nesting This embodiment preferably uses a "corpus re-nesting" algorithm that uses a push-down automaton to iteratively create nested structures given a flat list of cluster IDs from the hyperline clustering step. By comparing the signatures of adjacent hyperlines, the system can determine whether a given heading or list item belongs to a more, equal, or less nested level. This allows the multiply nested hierarchical structure of many documents (chapters, sections, subsections, phrases, lists, etc.) to be reconstructed.

[0071] Features considered in renesting include the "shape" (as known from NLP techniques) of the tokens in the hyperline, considering especially the first and last, the particular class of punctuation that terminated the previous line (if any), formatting information such as capitalization, leading whitespace, indentation, bolding, underlining, enumeration strings at the beginning of a line (e.g., patterns such as "IV(A)(1)" or "iv)") or the presence and form of particular bullets or other emojis, the value of that enumerator, the presence, level, and value of preceding enumerators of the same type, etc.

[0072] 6) Topic chunking This aspect uses lexical statistics and other learning techniques across consecutive chunks of a document to detect where topics shift, which enhances the identification of boundaries of large chunks, such as entire sections on a given topic, because sections (at any level) generally have greater topic, vocabulary, and style homogeneity within them than adjacent sections.

[0073] 7) Topic labeling i) Heading Labeller As shown in Figure 1, for each header in the corpus, this step Create a numerical representation, known as an "embedding," for each heading, Clustering headlines based at least on their embeddings; Filter out "bad" clusters, based at least on measures such as density, articulation, and level of similarity; · Propagate the most common semantic role label in each remaining cluster to all headings within said cluster.

[0074] ii) Keyphrase Labeler For each chunk, this step uses an ensemble of keyphrase extraction techniques (such as rule-based linguistic techniques, ML, statistical, Bayesian, and / or others) to generate candidate semantic role labels for the text.

[0075] 8) Chunk Clubbing i) Grammar This aspect of the system begins with a linguistic analysis of the text, such as part of speech tagging, natural language processing tasks including dependency parsing, constituent parsing, and others. The system then applies tree matching mechanisms from other domains to locate grammatical and other structures within the trees or tree-like structures discovered via NLP. These include document structuring methods such as tree grammars and tree pattern matching, as exemplified by XPath, GATE, and other tools.

[0076] Using such patterns to identify grammatical phenomena in a sentence allows the system to extract semantic role labels from the text itself, which are then used to annotate nearby chunks. For example, based on the sentence's constituent structure, we can build a search pattern that matches the sentence "The following are our terms of agreement" (and other sentences with similar grammatical structures), then extract the noun phrase (in this example, "term") and use it as the semantic role label for one or more chunks in the content that follow this sentence and contain such a "term."

[0077] ii) Question Answering Question answering techniques, including BERT for question answering, are specifically tuned to identify semantic role labels (e.g., date, person's name, dollar amount) for candidate chunks. In contrast, most traditional question answering models aim to answer questions like "What is an effective date?" This system instead trains a model to answer questions like "What is July 8, 2018?" and aims to predict "effective dates" or "effective date of X", where X represents another chunk in the text (rather than just "date", which is a data type rather than a semantic role).

[0078] The system also discovers synthetic questions that, when answered, can point to relevant information in the text, providing the ability to automatically suggest questions to be used by question answerers.

[0079] iii) XPath-like rules integrated with embedding Here, tools in the domain described under "grammar" are integrated with tools that provide vector-semantic representations of text, such as word2vec, char2vec, and many related methods. This system allows analysts to represent and query patterns, including both structural information (which can include chunks of data represented in XML or DOM-compatible formats), which is well handled by XPath and similar tools, and fuzzy or "semantic" similarity information, which is well handled by vector models.

[0080] 9) NER (small unlabeled chunks) Techniques can identify some chunks by data type, such as a person or business name, address, etc. (This is known as "Named Entity Recognition" or "NER"). However, NER falls quite short in identifying the semantic roles of these entities within a document. Current techniques also fail to identify larger chunks, such as entire phrases or sections, or groups of chunks that contain larger chunks that are meaningful or useful.

[0081] This aspect of the system detects interesting small chunks without necessarily assigning roles to them. There are many methods and tools for identifying NER in text. This system uses multiple methods, examples of which are listed below. These inventions are largely unsupervised.

[0082] i) Established NER methods ii) Expected words By training an n-gram language model using a wide range of common texts, such as Wikipedia, we build a model of "expected words in context for normal English." When looking at a particular document, the system provides a means to identify n-grams that do not fit that common model and therefore tend to be specific to the document being processed.

[0083] iv) TF-IDF This is a TF-IDF based approach ("term frequency vs. inverse document frequency"), used in conjunction with label propagation and contextual semantic labeling.

[0084] v) Sequence clustering Extract small words or character sequences such as n-grams and cluster them using context embeddings (e.g., those of BERT). The expected result is that n-grams that share semantic meaning start to cluster together. The cost of combinatorial explosion is addressed by using heuristics (including on the syntax tree) to filter out some n-grams before clustering. A wide variety of clustering algorithms can be applied. In this example, the hdbscan algorithm achieves effective clustering while assigning random noise to the "none" cluster.

[0085] vi) Small shot NER The system uses few-shot learning techniques to generalize from a small number of labeled instances (e.g., selective user feedback) to more broadly applicable rules or adjustments of learned parameters. This significantly reduces the number of times the user must be asked for feedback, improving the system's performance more quickly.

[0086] 10) Extraction labeling This aspect of the system detects semantic role labels for smaller chunks that appear directly in the sentences that surround them. Meaningful chunks often have a role that is somehow specified by the context. For example: John Doe ("the Seller") lives in… The $999 rent is due by the end of each month.

[0087] i) Contextual Semantic Labeling (CSL) This process uses a neural network operating on previously built structures including sentence parsing to learn which parts of the text are likely semantic role labels for various chunks. Many chunks may already have such labels with various sources and confidence levels, but this provides additional evidence for or against them, as well as new labels. Some of the patterns here involve grammar. For example, in "Do shall pay $1000 rent by the last business day of each month," the main verb reveals what the role of the monetary amount is, i.e., that it is rent to be paid. Other patterns are learned automatically by supervised and / or unsupervised methods using the structure, chunking, labeling, and content features available in the context. Formatting such as parentheses, table layouts, key phrases and words, and other features also provide features for the neural network.

[0088] Useful information is often present in the chunks that contain the sections or subsections, or their headings, etc. For example, whether a given drug is relevant as a prescription or as an allergy may only be detectable by looking at the headings of the contained sections (this is another example of why it is important to detect the correct hierarchical nesting of sections). There are many other clues that can be learned by machine learning techniques and applied to discover roles applicable to various chunks. Cross-document similarity can also be used to associate semantic roles to documents, especially within the same document set, that are found in similar contexts but may not be discoverable in isolated documents.

[0089] ii) Label Propagation This process standardizes labels across similar chunks of text in a corpus of documents. It applies to both labels extracted from context and labels available from the previous step. The algorithm clusters chunks based on their embeddings using agglomerative clustering, ranks candidate labels for each cluster of chunks using a weighted PageRank algorithm (using label frequency / confidence as initial node weights), and determines how similar the labels are to each other using co-occurrence and embedding similarity. It then assigns labels to chunks based on their cluster-level scores and how similar the chunks we are labeling are to the chunks they were labeled from (in terms of content, embedding, structure, data type, semantic role, and / or context). It applies agglomerative clustering and PageRank algorithms to propagate the labels across similar contexts, making the labels more consistent across a set of documents.

[0090] 11) Abnormality This aspect of the system examines multiple documents in a document set such as that generated in step (2) and identifies chunks that occur in the current document but that generally do not have counterpart chunks in other documents of the same set, or vice versa. Counterpart chunks need not have identical content, structure, format, context, data types, and semantic roles, but may have variations from one document to another. They can nevertheless be recognized as being substantially similar in their manner to other identified chunks.

[0091] If the new document contains chunks that are not typically present in other documents of the same set, the user may query for some or all of them to ensure that they were indeed intended. In this exemplary system, such queries are more common when the chunks in question are common to the new document and those it was based on (if any), but rarely common to the others.

[0092] If the new document lacks counterpart chunks that are typically present in other documents of the same set, or even in particularly relevant external sources (e.g., house style manuals, compliance requirements, etc.), examples of some or all of such chunks are suggested to the user, along with content drawn from the other documents. The suggestions may be ranked to the user according to factors such as frequency of use, being the most typical (centroid) of available alternatives, or having a high likelihood of co-occurrence with other chunks present in the new document. Chunk suggestions may be automatically updated, for example, to replace names, dates, and other subchunks specific to the document from which the example was drawn with values ​​drawn from the new document.

[0093] Moreover, the choice of chunks to be proposed for addition or deletion can usefully depend on the practices of different authors, editors, or other staff. For example, if the current author's documents frequently differ from another author's on a particular measure, that may indicate that the difference is a considered choice and not an error. On the other hand, if all authors working under the same supervisor do something in some way but the current author does it differently, that may indicate a greater need for review, at least when first noticed.

[0094] Anomaly modeling considers the structure and data types and semantic roles of chunks as well as the context, content, and format. For example, modeling patterns of what chunks' data types and semantic roles occur within, adjacent to, or near others. Violations of well-established patterns can be classified as anomalies and presented for user feedback like any other anomaly.

[0095] 12) Arbitration Many previous steps create and / or operate on chunks of the document, defined (typically, but not necessarily contiguous) as ranges of characters, tokens, and / or non-text objects in the linear sequence produced in step (3).

[0096] The chunk being considered at any time can be represented either by "inline" meta-information such as markup, or by a "stand-off" representation that points to its location in the text by various kinds of pointers. In this example, the stand-off representation is used for most processing, while the inline representation is used for some purposes such as communication with external tools, which often prefer it. These and other representations are functionally interchangeable, and the choice between them can be governed by concerns of performance, convenience, etc.

[0097] Representations of chunks include information about the steps or implementations that created them, how certain they are ("confidence level"), and their specific data type and / or semantic role labels. Redundant, uncertain, conflicting, or partially overlapping chunks may frequently arise, which we refer to as "non-optimal" herein. For example, two or more different processes may have semantic role labels attached to the same span of text (or nearly the same span, e.g., one that contains "Dr." before a name and one that does not). Chunks may be nested, sometimes deeply, but may also overlap arbitrarily (i.e., each of the overlapping chunks contains some content that is also in the other, and some that is not). Through the above steps, the system may maintain a representation that can represent a large number of annotations, including those that are overlapping or co-located.

[0098] Such non-optimal chunks are usually undesirable, at least when the document is presented to a user. Moreover, many state-of-the-art NLP tools prefer non-overlapping structures, as do many document tools and methods familiar to those skilled in the art, such as XML, JSON, SQL, and other representation systems. A more restricted structure is usually preferred, often referred to as "hierarchical" or "well-formed," and avoids partially overlapping chunks.

[0099] This aspect of the system modifies the collection of chunks to strictly hierarchical and to avoid non-optimal chunks. This can be accomplished in several ways. First, chunks can be removed entirely (i.e., the chunks themselves, not the document content they identified, are removed). Second, chunk scope can be modified (e.g., by including or excluding one or more characters or tokens from either end) to prevent overlap with another chunk. Third, chunks can be determined to be redundant and merged. Fourth, chunks can be detected as being inconsistent (e.g., if one tool considers "Essex" a place and one a person) and a selection can be made.

[0100] The process includes means to quickly detect cases of partial and / or complete duplication, compare chunks by type, role, and confidence, and resolve non-optimal cases by modifying chunks and their associated data. Selecting which chunks to modify, merge, or remove takes into account several factors, such as confidence level, prior probability of a given chunk's data type, semantic role, and content, hyponyms between semantic role labels, conditional probability of occurrence in a given context, number, role, and distribution of other chunks in the current and other similar documents, current process priority, customer feedback about similar cases, and / or other methods.

[0101] Modifications may also change the confidence level of a chunk. For example, some aspects of the system may apply similar or identical semantic role labels to the same or nearly the same parts of a document. In that case, the labels are typically merged, and the resulting chunk is assigned a higher confidence than the individual chunks it contains. In other cases, a choice is made between conflicting chunk assignments, but the selected chunk may end up with a reduced confidence that reflects that there has been some level of counterevidence.

[0102] This process improves the quality and consistency of chunk identification and labeling, allows the information to interoperate with a wide range of tools, and allows the results to be analyzed more easily and reliably. The actions described above can be applied at any time, not just at the end. For example, if the previous step uses an external tool for some subtask, it may require reduction to well-formedness. Deleted or modified chunks can instead be "paused", meaning that they no longer affect the processing, but can be reintroduced on request, which allows such use of non-overlapping supporting tools without the need to later reproduce the previous work from scratch, increasing the flexibility and speed of the processing.

[0103] In one approach, all overlaps and / or all non-optimal chunks are resolved before generating the document that is shown to the user, so that the results can be easily encoded in a hierarchical format, such as the XML format used by many modern word processors and other tools. However, it is also possible (even in XML) to maintain multiple potentially overlapping alternatives at a particular location for potential later resolution, such as by user feedback or improved algorithmic learning.

[0104] 13)DGML (DocuGami Markup Language) The enriched version of the document represents the document structure, format, content, and identified chunks, and may identify which chunks were identified at which step of the process and with what level of confidence. Some embodiments use XML as the syntax for this representation, although a wide range of representations can contain substantially the same information in other XML schemas, JSON, various databases, custom text or binary formats, etc.

[0105] In this step, information about the document and detected chunks is converted (or "serialized") into XML format so that it can be more easily passed to other processes, in particular front-end user interfaces used for feedback, editing, and review, and in a format useful for "dashboard" applications that provide overviews, statistics, and compliance information to other users such as group managers, quality control staff, etc.

[0106] DGML (Docugami Markup Language) is a specific XML schema for this use, which houses all of the described information in one package. Most previous schemas may address structure, content, and sometimes layout, but do not annotate "chunks" abstractly as described here. Many previous schemas also do not provide a generalized mechanism that can automatically detect and represent chunks on the fly, especially with trust levels and provenance information.

[0107] It is also possible to "tunnel" the same information with the file formats of some word processors and other tools by representing it in a format transparent to the format. For example, if the tool supports embedded comments or metadata, "invisible" text, ignorable attributes, or other similar features, the information described herein can be hidden within them, allowing the resulting document to be used, and possibly modified, by the tool, and passed back to the system with the tunneled information still available.

[0108] 14) Feedback mode front end Extensive annotations and analysis attached to the document and its discovered chunks in the manner already described make it feasible to guide the user through the editing of samples, templates, or previous documents to create similar but new documents customized to the current needs. For example, the system will typically identify the parties and property covered by the agreement, the medical history, current findings, and medications or conditions mentioned in other specific sections of the clinical notes, relevant dates, etc. By also inspecting other documents in the same document set, the system will learn which things are incommon, common, or necessary, and thus can make more useful recommendations to the user about what to review and / or update. For example, an effective date may be present in almost every agreement in the document set, but its value may vary from one to the next. Similarly, the parties will vary, but the party types will be much more consistent.

[0109] i) Unguided Feedback In the interaction with the user, the system first requests feedback on chunks that have been (or may not have been) detected in some documents. The first few documents presented for feedback become the "cluster centroids" of the document set. The last few become the "outliers" of the document set.

[0110] ii) Inductive Feedback The system then presents the user with selected portions of the document and prompts the user to provide feedback by asking about their current or potential labels, their scope, etc. a. "Interesting labels" are determined by a PageRank-based algorithm and grammar and structure models. Of these labels, a set of low-confidence instances are selected for review. b. If the low-confidence label is no longer present for the current document, the same process may be repeated for additional documents. In some embodiments, the model is continuously updated based on the feedback that the user is providing. However, instead, the feedback may be accumulated and applied later, in batches, and / or offline. Adjustments to the model may then affect the selection of chunks and labels subsequently presented for feedback, and may trigger a reanalysis of some documents. c. The system uses essentially the same mechanism to solicit feedback on field and structure chunks. In one approach, every chunk detector provides a reliability estimate, which can be used along with other information to select candidates for feedback.

[0111] Feedback may be requested in different passes, for small vs. large chunks, fields vs. structure chunks, or in other orders. See FIG. 3 for an example of a user interface for user feedback. Some or all chunks are displayed, allowing the user to select and inspect particular ones to see their assigned types and / or roles, and optionally alternatives. The user can move chunk boundaries, select or edit labels, etc. Preferably, the user can also request that a particular change (to a label, etc.) be applied to all corresponding or same type chunks. 15) Feedback Response i) Fleet Query is a method that allows a system to query both private and public data based on user feedback, typically from multiple users. The selected examples are semantically and syntactically similar to previous fault cases, which increases the value of the feedback. ii) Dispatcher: A dispatcher is a methodology for connecting user feedback on the combined output of several ML models and non-ML algorithms back to a particular learning model 120 that can learn from the feedback.

[0112] The system allows the model to be improved from user feedback on its output, and from user feedback on the output of other trained and untrained models. This is achieved by using the feedback as incremental (also called "fine-tuning") training data for several of the described numerical and neural models. After using the feedback to improve the model, not only the specific document but all documents in the set, or even all documents of the user, are re-evaluated. Thus, feedback on each document can improve chunk identification, role assignment, structure discovery, and therefore user assistance, for all documents. This re-training is represented by the dotted connector from step (15) to step (3) in Figure 1.

[0113] The document and all associated information contributes to the learning and analysis of the set of documents (particularly, but not exclusively, within a particular document set), thus improving the performance of future documents. For example, when a new chunk is added to one or more documents in the set, it becomes available for use in future documents (or revisions of old ones) and can be suggested for future documents. At some point, the absence of a recently introduced chunk role, or the presence of a chunk role that has not been used much recently, can be an anomaly. This point can be chosen spontaneously, by the user in response to a feedback question, or automatically based on the usage curve of the counterpart chunk over time. For example, if few documents in one set created before a certain time contain a chunk of a given role and / or context (e.g., an "exclusion" section), but most or all of those created later do have it, then the absence of the counterpart chunk is likely an anomaly in the new document and can be usefully presented to the user as such.

[0114] 16) Downstream Communications After annotating the document with chunk information as described, the selected information is converted into the specific format required by external business information systems, such as databases, analytical tools, etc., and passed to those systems, either directly or through an automated and / or manual review step. For example, the names and addresses of specific parties can be copied into the correct fields in a database, which cannot be done automatically if they were identified only as "name" and "address" themselves. See FIG. 4 for an example of integration with downstream software applications. In this example, chunks representing terms that the parties are expected to agree on are extracted, and they are passed to a downstream application, similar to Docusign, to be filled in and signed.

[0115] FIG. 5 is a block diagram of one embodiment of a computer system 510 that may be used with the present invention. The steps described above may be implemented by software executing on such a computer system. The computer system 510 typically includes at least one computer or processor 514 that communicates with peripheral devices via a bus subsystem 512. Typically, the computer may include or be any of a microprocessor, a graphics processing unit, or a digital signal processor and their electronic processing equivalents such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). These peripheral devices may include a storage subsystem 524 with a memory subsystem 526 and a file storage subsystem 528, a user interface input device 522, a user interface output device 520, and a network interface subsystem 516. The input and output devices enable user interaction with the computer system 510.

[0116] The computer system may be a server computer, a client computer, a workstation, a mainframe, a personal computer (PC), a tablet PC, a rack-mounted "blade," or any data processing machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by that machine.

[0117] A computer system typically includes an operating system, such as Microsoft's Windows, Sun Microsystems' Solaris, Apple Computer's MacOs, Linux, or Unix. A computer system may also typically include a basic input / output system (BIOS) and processor firmware. The operating system, BIOS, and firmware are used by the processor to control the subsystems and interfaces connected to the processor. Typical processors compatible with these operating systems include Intel's Pentium and Itanium, Advanced Micro Devices' Opteron and Athlon, and ARM Holdings' ARM processors.

[0118] The inventions, embodiments, and / or examples of the claimed subject matter are not limited to conventional computer applications, nor to programmable devices that execute them. For example, the inventions, embodiments, and / or examples of the claimed subject matter may include optical computers, quantum computers, analog computers, and the like. The computer system may be a multi-processor or multi-core system, and may be used or implemented in a distributed or remote system. The term "processor" is used herein in its broadest sense to include single processors and multi-core or multi-processor arrays, including graphic processing units, digital signal processors, digital processors, and combinations of these devices. Furthermore, while only a single computer system or single machine may be illustrated, the use of the singular form of such terms shall also mean any collection of computer systems or machines that individually or jointly execute instructions to perform any one or more of the operations discussed herein. Due to the ever-changing nature of computers and networks, the description of computer system 510 shown in FIG. 5 is intended only as one example for purposes of illustrating a preferred embodiment. Many other configurations of computer system 510 are possible with more or fewer components than the computer system shown in FIG. 5.

[0119] The network interface subsystem 516 provides an interface to external networks, including an interface to a communications network 518, through which it is coupled to corresponding interface devices in other computer systems or machines. The communications network 518 may include many interconnected computer systems, machines, and communications links. These communications links may be wired links, optical links, wireless links, or any other devices for communication of information. The communications network 518 may be any suitable computer network, for example, a wide area network such as the Internet, and / or a local area network such as an Ethernet. The communications network may be wired and / or wireless, and the communications network may use encryption and decryption methods, such as those available in virtual private networks. The communications network employs one or more communications interfaces that can receive data from and transmit data to other systems. Embodiments of the communications interface typically include Ethernet cards, modems (e.g., telephone, satellite, cable, or ISDN), (asynchronous) digital subscriber line (DSL) units, Firewire interfaces, USB interfaces, and the like. One or more communication protocols may be used, such as HTTP, TCP / IP, RTP / RTSP, IPX and / or UDP.

[0120] The user interface input devices 522 may include pointing devices such as alphanumeric keyboards, keypads, mice, trackballs, touchpads, styluses, or graphic tablets, scanners, touch screens integrated into displays, audio input devices such as voice recognition systems or microphones, eye gaze recognition, brainwave pattern recognition, and other types of input devices. Such devices may be connected to the computer system by wires or wirelessly. In general, use of the term "input device" is intended to include all possible types of devices and means for inputting information into the computer system 510 or onto the communications network 518. User interface input devices typically allow a user to select objects, icons, text, etc. to be displayed on some type of user interface output device, e.g., a display subsystem.

[0121] The user interface output devices 520 may include a display subsystem, a printer, or a non-visual display such as an audio output device. The display subsystem may include a flat panel device such as a liquid crystal display (LCD), a projection device, or some other device for creating a visible image such as a virtual reality system. The display subsystem may also provide a non-visual display such as through audio output or haptic output (e.g., vibration) devices. In general, use of the term "output device" is intended to include all possible types of devices and means for outputting information from the computer system 510 to a user or to another machine or computer system.

[0122] The memory subsystem 526 typically includes several memories including a main random access memory (RAM) 530 (or other volatile storage device) for storing instructions and data during program execution, and a read-only memory (ROM) 532 in which fixed instructions are stored. The file storage subsystem 528 provides persistent storage for program and data files and may include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, a flash memory, or a removable media cartridge. Databases and modules implementing the functionality of certain embodiments may be stored by the file storage subsystem 528.

[0123] The bus subsystem 512 provides a device for allowing the various components and subsystems of the computer system 510 to communicate with each other as intended. Although the bus subsystem 512 is shown diagrammatically as a single bus, alternative embodiments of the bus subsystem may use multiple buses. For example, a RAM-based main memory may communicate directly with a file storage system using a direct memory access (DMA) system.

[0124] Although the detailed description contains many details, these should not be interpreted as limiting the scope of the present invention, but merely as illustrating different examples. It should be understood that the scope of the present disclosure includes other embodiments not described in detail above. Various other modifications, changes, and variations that are apparent to those skilled in the art may be made in the arrangement, operation, and details of the methods and apparatus disclosed herein without departing from the spirit and scope, as defined by the appended claims. Therefore, the scope of the present invention should be determined by the appended claims and their legal equivalents.

Claims

1. 1. A computer-implemented method performed by a computer system executing instructions for analyzing and annotating a document, comprising: Importing documents from the document set; performing a visual extraction of the imported document, including generating signatures of document portions; (a) automatically identifying a hierarchical structure of chunks within individual documents of the document set based on visual extractions, content, and context of the individual documents, and (b) based on patterns of visual extractions and content across documents of the document set, the hierarchical structure including smaller chunks that contain sequences of words within individual sentences; for at least some of the small chunks, in a process separate from identifying the small chunks, automatically selecting text from sentences surrounding the small chunks as a label for the semantic role played by the small chunk in the transaction described by the respective document; standardizing labels of semantic roles across documents of the document set; annotating documents of the document set, the annotations including locations of the identified small chunks and standardized labels of the semantic roles played by the identified small chunks at those locations; 4. A computer-implemented method comprising:

2. The computer-implemented method of claim 1 , further comprising assembling the document sets by clustering documents into document sets based on content and / or geometric layout similarity.

3. 2. The computer-implemented method of claim 1 , wherein automatically identifying the hierarchical structure of chunks within individual documents of the document set is further based on (c) identifying counterpart chunks in different documents of the document set, the counterpart chunks being different chunks in different documents that play the same semantic role within their respective documents.

4. 4. The computer-implemented method of claim 3, wherein identifying counterpart chunks in the different documents comprises identifying content that is different in the different documents but occurs within substantially similar contexts within the different documents.

5. The computer-implemented method of claim 3 , wherein identifying counterpart chunks in the different documents comprises identifying substantially identical content in the different documents.

6. 2. The computer-implemented method of claim 1, annotating some of the identified chunks with metadata that describes the chunks, and identifying counterpart chunks in different documents is based on similarities in the metadata.

7. Identifying chunks based on patterns across the documents of the document set includes:

2. The computer-implemented method of claim 1, comprising identifying chunks in an individual document that commonly occur in documents of the document set but that do not occur in the individual document.

8. The identified chunks include: field chunks, which contain content in the document suitable for use as fields in a document template; a structural chunk containing content that includes a structure within a geometric layout of the document; 2. The computer-implemented method of claim 1, comprising:

9. 9. The computer-implemented method of claim 8, wherein some of the field chunks are hierarchical and include other chunks as sub-chunks.

10. The computer-implemented method of claim 1 , wherein some of the identified chunks include content that describes semantic roles played by other chunks.

11. The computer-implemented method of claim 1 , wherein the annotation further includes a data type of the identified chunk.

12. The computer-implemented method of claim 1 , wherein identifying chunks based on visual extraction comprises identifying spatial boundaries of structural chunks using machine learning inference trained on tiles of the page image.

13. The computer-implemented method of claim 1 , wherein identifying chunks based on visual extractions comprises identifying spatial boundaries of structural chunks using artificial intelligence based visual recognition of geometric patterns of the visual extractions.

14. 2. The computer-implemented method of claim 1, wherein identifying chunks based on visual extraction comprises identifying structural chunks based on visual extraction of non-textual structural features, the non-textual structural features comprising at least one of a figure, a table, a sidebar, a footnote, and a page header or footer.

15. 2. The computer-implemented method of claim 1, wherein identifying chunks based on content includes identifying chunks using AI techniques for topic inference.

16. 2. The computer-implemented method of claim 1, wherein identifying chunks based on content comprises identifying chunks within the document set using a few-shot named entity recognition technique.

17. receiving a user correction for the incorrectly identified chunk; improving automatic identification of chunk hierarchical structure in response to said user modifications; and The computer-implemented method of claim 1 , further comprising:

18. 1. A non-transitory computer readable storage medium storing executable computer program instructions for analyzing and enhancing documents, the computer program instructions being executable by a computer system, the computer system comprising: Importing a document set that contains multiple documents; performing a visual extraction of the imported document, including generating signatures of document portions; (a) automatically identifying a hierarchical structure of chunks within individual documents of the document set based on visual extractions, content, and context of the individual documents, and (b) based on patterns of visual extractions and content across documents of the document set, the hierarchical structure including smaller chunks that contain sequences of words within individual sentences; for at least some of the small chunks, in a process separate from identifying the small chunks, automatically selecting text from sentences surrounding the small chunks as a label for the semantic role played by the small chunk in the transaction described by the respective document; standardizing labels of semantic roles across documents of the document set; annotating documents of the document set, the annotations including locations of the identified small chunks and standardized labels of the semantic roles played by the identified small chunks at those locations; A non-transitory computer-readable storage medium for causing a method to be performed, comprising:

19. 1. A computer system for analyzing and improving a document, the computer system comprising: a storage medium for receiving and storing a document set including a plurality of documents; a processor system that executes an application program that accesses the storage medium and analyzes and annotates documents; Equipped with The processor system for executing the application program comprises: Importing documents from the document set; performing a visual extraction of the imported document, including generating signatures of document portions; (a) automatically identifying a hierarchical structure of chunks within individual documents of the document set based on visual extractions, content, and context of the individual documents, and (b) based on patterns of visual extractions and content across documents of the document set, the hierarchical structure including smaller chunks that contain sequences of words within individual sentences; for at least some of the small chunks, in a process separate from identifying the small chunks, automatically selecting text from sentences surrounding the small chunks as a label for the semantic role played by the small chunk in the transaction described by the respective document; standardizing labels of semantic roles across documents of the document set; annotating documents of the document set, the annotations including locations of the identified small chunks and standardized labels of the semantic roles played by the identified small chunks at those locations; A computer system that executes

20. The computer-implemented method of claim 1 , wherein importing the document includes importing a word processor document and importing a page layout document.

Images