Cross-document intelligent authoring and processing assistant

Abstract

To analyze a document to annotate it.SOLUTION: Machine learning, artificial intelligence, and other computer-implemented methods are used to identify semantically important chunks in documents, the semantically important chunks are automatically labeled with appropriate datatypes and semantic roles, and this enhanced information is used to assist authors and to support downstream processes. Chunk locations, datatypes, and semantic roles can be automatically determined from what is called "context", to wit, combination of their formatting, structure, and content, those of adjacent or nearby content, overall patterns of occurrence in a document, and similarities of all these things across documents. Similarity may include similarity of natural language grammatical structure, machine learning techniques such as measuring similarity of word, chunk, and other embeddings, and the datatypes and semantic roles of the previously-identified chunks.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure generally relates to methods and apparatus for AI self-supervised creation of hierarchically semantically labeled documents and / or for assisted authoring and processing of such documents.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority under 35 U.S.C. § 119(e) to U.S. Provisional Patent Application No. 62 / 900,793, entitled "Cross-Document Intelligent Authoring and Processing Assistant," filed September 16, 2019. All of the foregoing subject matter is incorporated herein by reference in its entirety. [Background technology]

[0003] Description of related technologies Many businesses produce multiple documents that are very similar, albeit customized each time. For example, an insurance firm may produce many proposals for a particular type of insurance, but each must be tailored to the needs of a specific client. These documents can be considered to be of the same "type" because they have similar textual (and sometimes graphical) content (reflecting similar purposes and topics), similar selection and arrangement of sections and other major units, and often similar geometric layout and formatting characteristics.

[0004] While some types of documents are widely known and used, many are not. Many are specific to particular businesses, markets, or applications, and new ones are created to fit new situations. A user, who may be called an "author" or an "editor," typically creates a new document of a particular type (sometimes called a "target document") by copying a previous document of the same type and modifying it as needed, for example, by manually editing or replacing particular chunks of content.

[0005] In current practice, word processors typically identify chunks only when necessary to achieve formatting; for example, headings, footnotes, and figures may be explicitly marked to receive special formatting, but names, addresses, or dates are rarely explicitly marked. Even when identified, chunks are generally associated only with formatting effects (margins, fonts, etc.) that are useful information, but no direct indication of either these data types or semantic roles is provided. Similarly, word processors often only represent hierarchical storage visually, often with no explicit representation of the nested sections themselves, only differently formatted headings.

[0006] When creating a new document of the same general type as a previous document, in many cases the bulk of the work will be text editing, replacing, deleting, or inserting specific chunks, taking care not to mix up those with different semantic roles (such as swapping buyer and seller addresses). This typically requires human intervention, as the authoring system will usually know nothing about these chunks, especially their data types or semantic roles, and will not be able to help very effectively.

[0007] In some simple cases, "forms" and "templates" may be used, providing explicit locations for filling in the content of specific chunks. However, forms typically address only simple cases where virtually all necessary chunks can be pre-enumerated and where there are few large, repeatable, or highly structured chunks. Forms also require skilled effort to create, are difficult to adapt to changing circumstances, and do not actively support the writer. [Brief explanation of the drawings]

[0008] The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication and color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

[0009] Embodiments of the present disclosure have other advantages and features that will become more readily apparent from the following detailed description and the appended claims, taken in conjunction with the accompanying drawing examples.

[0010] [Figure 1] FIG. 1 is a block diagram of one implementation of a system and process for creating hierarchically semantically labeled documents using machine learning and artificial intelligence. [Figure 2] FIG. 2 is a screenshot showing a dashboard tracking the processing of different document sets through the system of FIG. [Figure 3] FIG. 3 is a screenshot of a user interface for receiving feedback from a user. [Figure 4] Figure 4 shows a screenshot of the integration with other software applications. [Figure 5] FIG. 5 is a block diagram of one embodiment of a computer system that may be used with the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0011] Detailed Description of the Preferred Embodiments overview Groups of documents determined to be of the same type constitute a "document set" or "document cluster." For example, an insurance company's offerings for a particular type of insurance to a particular class of customers may be considered the same type and form a document set. Different types of insurance offers from the same company, or offers to customers they deem different, may be considered different types that belong to different document sets. Rental agreements, clinical notes for particular types of patients, sales proposals, schedules, meeting minutes, etc. are other potential types of documents, as well as subtypes that share characteristic patterns of content, structure, and / or layout.

[0012] Creating and editing new target documents within a document set very often involves editing or replacing "chunks" that are "semantically significant"; such chunks are typically not necessarily contiguous spans of text, but rather specific portions of documents that have particular data types and semantic roles and that have meaning and significance to a business or other process.

[0013] These chunks are of various data types, and in many computer systems are finer than atomic data types. For example, a given chunk may represent not just a string of characters, but also a person or organization name, a date, a time period (which is not exactly the same as a date), a currency amount, etc. Larger chunks may contain bundles of information such as lists of drugs or other substances, travel itineraries, procedures to follow, medical prescriptions, and countless others.

[0014] Additionally, chunks may have semantic roles relative to the documents in which they occur. For example, a person's name may be the "tenant" in a lease agreement, or the "seller" in a sales proposal, or the "agent" of another person. A date may represent the start or end of some responsibility or activity. A dollar amount may be a periodic payment amount, or a penalty or bonus associated with certain conditions, and so on. Such semantic roles are important for enforcing the appropriate use of the information in the chunks. The names of semantic roles are called "semantic role labels" or simply "labels."

[0015] A chunk is typically represented as a bundle that includes its location, data type, semantic role, and / or other data / metadata. Location is commonly represented as a start and end point and can be represented in several ways, such as an inserted marker or a byte, character, or token offset (either global to the document or relative to an established ID, marker, or other object). Semantic roles are represented by a label or other identifier. Chunks can be of any size, and some can contain other chunks as "sub-chunks." Chunks can contain not only text, but also non-text data such as images or other media, and "structure" such as tables, lists, and sections.

[0016] The techniques disclosed herein use machine learning, artificial intelligence, and other computer-implemented methods to identify various semantically significant chunks within a document, automatically provide them with appropriate data types and semantic roles, and use this enriched information to assist authors and support downstream processes. A chunk's location, data type, and semantic role can be automatically determined from what is often referred to as its "context," i.e., its combination of format, structure, and content, its neighboring or nearby content, its overall pattern of occurrence within the document, and the similarity of all these things across documents (primarily, but not exclusively, between documents in the same document set). "Nearby content" includes horizontally nearby content, such as preceding and succeeding content in the reading sequence of text, but also vertically nearby content, such as within the same container structure like lists and sections, along with their respective markers, headings, levels, etc. Similarity is not limited to exact or fuzzy string or property comparisons, but can also include ML (machine learning) techniques such as measuring the similarity of natural language grammatical structures, words, chunks, and other embeddings, as well as the data types and semantic roles of previously identified chunks.

[0017] For example, a person or organization name can generally be identified as having a semantic role such as "seller" because the document states so, very often using a sentence in some human languages, but often also including a larger context. In another example, one or more words can often be easily identified as representing a data type such as "drug name," but context is necessary to determine that it carries the semantic role of allergy rather than prescription. Often, important evidence of a semantic role is expressed not in the same sentence, but in various other ways, such as chunks occurring in larger chunks (e.g., a "known allergies" section). While the flexibility and variety of grammatical and document structural constructs (not to mention typos, transcription errors, etc.) make identifying data types less difficult, identifying semantic roles, especially those with a scope larger than a single sentence, can be very challenging.

[0018] A given semantic role may relate a chunk to the document as a whole or to other chunks. For example, a flight departure time may be tied to a particular "leg" in an itinerary and only indirectly related to others. Typically, a chunk hierarchy groups such items appropriately, such as collocation within a section, table parts, etc.

[0019] More specifically, examples of semantically significant chunks include the names, addresses, and other characteristics of specific parties in a contract, prescribed medications and prohibited procedures in a medical record, requirements (or exclusions) in a real estate proposal, dates and flight numbers in a travel itinerary, etc. All of these can be considered semantic roles of chunks. There are also larger chunks with various types and roles, such as entire sections and subsections. These are often inserted or removed as a whole, possibly with changes to smaller chunks within. Chunks can be hierarchical; that is, larger "containing" chunks can contain other "sub-chunks" up to any number of levels.

[0020] A chunk is generally a contiguous sequence of words in a document, such as "John Doe." However, chunks can also contain partial words: "John Doe's House" includes a name, but the name ends before an apostrophe (in the middle of a word). Chunks can even be discontinuous, for example, the same name for "John (also called "Bill") Doe." Layout can also make chunks discontinuous, for example, a page break can occur in the middle of a chunk (perhaps because there is a page header, footer, or footnote, which can be ignored for some purposes), or because of intervening figures, tables, charts, sidebars, or other displays.

[0021] The actual location and context of a chunk can also be important; a chunk is not just an isolated string of characters, but can occur multiple times with different (or no) semantic roles for different instances. More modern systems usually support inline or standoff markup, sometimes called "annotations," that can persistently associate various labels and other information with chunks. For example, HTML provides tags for manually labeling the boundaries of common structural chunks (e.g., "div," "ol"), and several broad types or roles for (typically) smaller chunks (e.g., "initial," "kbd," "dfn," "cite," etc.). Other XML schemas provide many other labels, and word processors allow somewhat similar labeling via "styles."

[0022] Some chunks may represent what are commonly called "fields." These are often small chunks that often occur in a similar context and layout in many or all documents in a given set, but typically have different text content. They may also occur multiple times in a single document with the same or very similar content. Such chunks may be called "field chunks." They are often manually discovered and treated as "fields" in template-based systems, but here they are discovered by their context and patterns of occurrence within and across documents and are assigned data types and semantic roles in much the same way as other chunks. They may or may not represent named entities such as personal names, addresses, dates, etc.

[0023] Another common type of chunk can be called a "structure" or "structural" chunk. Such chunks are typically larger and often contain many other chunks, some of which may also be structural chunks. They frequently have a "title" or "heading" that provides a name, number, description, and / or other information about the structural chunk. Examples of structural chunks include chapters, sections, tables, figures, sidebars, and many more. The type and semantic role of a structural chunk is often important for determining the types and semantic roles of nearby or contained chunks.

[0024] Not only data types, but also specific semantic roles are important for properly creating and utilizing documents. Whether a particular name represents a buyer versus a seller or a patient versus a doctor, whether a given date is the start or end date of a requirement or the departure or arrival time of a flight, whether a numeric value specifies a principal, interest, dosage, temperature, penalty, or something else, is crucial. For larger chunks, roles include "limitation of liability" statements versus "governing law" specifications versus "definitions," and countless others. A chunk's semantic role is often specific to a particular domain or transaction and is perhaps one of the most important features of a document. Many types of documents require, or at least are very common, chunks with specific data types and semantic roles, and chunks are called "counterparts" when they correspond to the entire document. Counterpart chunks may occur in similar orders and patterns, especially in documents by the same author or organization, and typically across the same document set. Counterpart chunks have the same or very similar roles and generally have similar contexts and / or formats. Thus, the distribution of chunk data types and semantic roles provides valuable information for distinguishing types of documents as well as helping to identify counterpart chunks in other documents.

[0025] While many counterpart chunks have similar content, others do not. For example, the same party (semantic role) in different documents is usually a different individual, but appears in very similar contexts and patterns of use. This can be particularly common with, but not limited to, "field chunks."

[0026] Once discovered, the hierarchical semantic chunks with data types and semantic roles within a business document can be used in downstream business processes. For example, a back-office database can properly record a new mortgage loan given specific party names, a specific date, term, and interest rate values. In applications such as these, semantic roles are crucial, and entering the correct data type into the wrong database field (such as swapping the seller and buyer names or addresses) can be a major problem, especially when moving information to downstream databases, processes, or reports.

[0027] Some Features and Benefits The techniques described herein may have various features and advantages, including any of the following.

[0028] Some implementations may provide easier, more efficient, and more accurate ways to generate documents with hierarchically organized chunks with semantic labeling useful for business processes. This may be achieved using various techniques to identify such chunks of various sizes, discover the data types and semantic roles they play within the document, and learn patterns of their use, characteristic contexts, and the like. Learning may be gained from analyzing the content, structure, and format of current and previous documents, feedback from authors and editors, and comparing multiple documents, particularly those within the same document set. With this knowledge, the system can provide valuable assistance to users, for example, facilitating the creation of new, higher-quality documents and extracting desired information for downstream use in other software applications, in back-office databases, for derived reports, compliance checks, and the like. Such learning may be performed using unsupervised and self-supervised learning techniques that do not require large amounts of pre-labeled or pre-analyzed data, but instead infer patterns from unlabeled or minimally labeled data.

[0029] Some implementations may enable computers to assist the writing process by discovering and using patterns within and across business documents to help writers avoid many of these errors, thus reducing the time required to achieve a given level of quality.

[0030] Today, typical document systems do not identify chunks, or in particular their data types or semantic roles. This adds time and expense for authors and editors and for importing data from documents into back-end databases, dashboards, or other downstream business processes. For example, it is common to manually find data (chunk by chunk) from a contract and copy it into a spreadsheet or data entry form.

[0031] Some implementations may help to label such hierarchical semantic chunks during the authoring process and represent them explicitly, thus making them easier for people and / or computers to extract and connect them to various kinds of other business processes, saving time and money.

[0032] Current techniques typically do not fully exploit similarities between multiple documents created by the same author or group and / or of the same type (here, indicated by membership in a specific document set) to more reliably identify new chunks of documents or to draw attention to potentially important differences. Explicit rules, such as requiring a section headed "Separability," only cover similarities that analysts quickly notice and explain; they are static and often constrained (e.g., cases of omissions that require rephrasing or reorganization, or failure to address opposing conditions), and quickly become outdated. Smaller companies often lack the resources necessary to develop more responsive techniques and often have too few documents to justify the expense. On the other hand, smaller companies often have a less diverse range of documents, making them more suitable for automated analysis such as that described herein.

[0033] Some implementations may use the extracted information about chunks and their content, context, layout, and patterns of use across documents to assist writers in creating new documents. Examples include, at a minimum, suggestions such as changing, reformatting, or moving specific content, phrases that are commonly present in similar documents but are missing from the new document (referred to as "missing" or "potentially omitted" chunks or content), phrases that are commonly absent from similar documents but are present (referred to as "unusual" chunks or content), and changes such as swapping names or roles of different parties in specific places.

[0034] Some implementations may accept and retain user feedback, such as when a user indicates that a chunk is labeled with the wrong range, data type, or semantic role, that they are not interested in it, or fail to label it altogether. Some implementations may use specific user corrections to improve machine learning and neural models and remember not to repeat previous suggestions in cases where the user rejects them (even if additional learning fails to prevent a particular instance of the mistake). In particular, some implementations may avoid requiring extensive review steps or corrections in favor of small-shot learning techniques and careful selection of requested feedback to minimize the amount of user action required. Some current techniques learn very specific things, for example, when a user instructs a spell checker to add a word to its dictionary. However, this involves a mere rote list rather than iterative training or fine-tuning of the model used to determine subsequent behavior after refinement, and therefore does not fully utilize the capabilities described herein.

[0035] Some implementations may use small amounts of user correction to learn and improve their behavior while avoiding annoying users with repeated suggestions when reapplying an improved but still imperfect model.

[0036] Many businesses record specific information derived from documents in various types of databases that support their processes. For example, a company that owns many rental properties typically uses a back-end system to help manage specific information derived from those rental agreements, such as renter payments, approved pets, pre-existing damages for which the renter is not responsible, or other information. Auto or tool renters, mortgage companies, health care providers, local governments, and other organizations use other information. Many goods and services have numerous mix-and-match options, and supervisors review statistics on their acceptance, combinations, pricing, and other factors. Business information systems typically provide analytics, check for consistency or compliance, derive reports, and / or support other business processes, all of which can be facilitated by the use of chunked information described herein.

[0037] Typically, chunks and the information they provide are scattered throughout prose text, manually extracted, and manually entered into spreadsheets, databases, or other systems. Previously, manual effort was necessary because the flexibility of natural human language, as found in contracts, emails, and the like, and its similarly variable layout and presentation conventions, allow important chunks to be expressed in countless different ways. The negotiations underlying such documents are also often scattered across multiple document types, including emails, notes from conversations, slide presentations, and the like. That information can also be useful, but is typically addressed manually. Some systems may treat such information sources as documents and reap the same benefits already described.

[0038] Some implementations may provide a means for a computer to initiate execution of a particular document once it has been transformed as described herein into a hierarchically semantically labeled document. By combining the hierarchically labeled structure of a document with tools that provide a vector-semantic representation of the text, specific chunks can be identified as requiring specific actions. For example, a contract may specify remittance, notification, or other actions and the conditions that enable or trigger them. These can be identified and used to initiate execution of the contract.

[0039] Some implementations may provide an easy way to review and summarize information from a set of documents in an interface such as a "dashboard" and move identified information to a customer's back-end database or similar system, enabling a more efficient and less expensive flow of business data and enhancing quality assurance, consistency, and reporting. Once chunks are semantically labeled, it becomes easy to generate summary reports across a set of documents that contain counterpart chunks. Some implementations may provide a very easy way for users to create such reports by simply clicking on one or more examples of chunks to be included, which are then located and extracted by role or context across all documents in the set. Some implementations may also assist users in discovering documents that lack expected counterpart chunks and modifying them to include or identify such chunks, or verifying that they correctly do not include them.

[0040] In another aspect, the performance of a given group, such as a company or department, can be enhanced by incorporating information such as the semantic roles of chunks, patterns of occurrence, and other characteristics of their documents and their users' feedback into the system's learning process, and using the resulting improved model to enhance and / or verify future documents. However, many customers do not want to share such information with other customers, and many have binding confidentiality requirements. On the other hand, general information and learnings derived from public, non-confidential sources can be freely used and shared.

[0041] Some implementations may provide the benefits of feedback and learning by keeping each customer's data and any model information derived therefrom separate and private to each customer, while still sharing common learnings based on non-confidential public data. Keeping these data processes separate also ensures that there is no statistical possibility of information "leaking" from one customer to another.

[0042] Introduction to Exemplary Embodiments The following is a description of an exemplary system. See FIG. 1. The system generally relates to a method and apparatus for AI self-supervised creation of hierarchically semantically labeled documents and / or for assisted authoring and processing of such documents. This includes processes such as organizing, structuring, annotating, modifying, reviewing, extracting data from the documents, and / or using such data in downstream business processes. More specifically, the system focuses on using primarily unsupervised and self-supervised machine learning techniques across a relatively small set of documents to discover a detailed hierarchical structure of the documents, consisting of many semantically meaningful chunks associated with their roles, to identify documents similar to previous documents and the use of such highly enriched documents in business processes.

[0043] The operation of this exemplary system uses the following processes, which are described in more detail in the following sections. This is merely an example. Other embodiments may use different combinations of steps, including omitting steps, adding other steps, and changing the order of some steps. They may also use different implementations of the steps listed below, including different combinations of the techniques described under each step. In Figure 1, the steps are preceded by an "S," and step 1 below is labeled "S01." 1) Import: Bring a group of a user's documents into the data store 110. 2) Organization: Separating documents into document sets by type, such as rental vs. sales contracts, or medical history vs. current clinical notes. 3) Visual Extraction: Extract a linear text stream from each document based at least on its content and visual layout, including limited information about individual text and other areas, their start and end positions, format, and content. The extracted data may be organized as "visual lines," such as paragraphs distinguished by their geometric layout, or as "visual blocks" (also called "hyperlines" or "visual" chunks). 4) Structure: Identify broad classes of headings, list items, and other structural chunks in the document. 5) Re-nesting: Determine the nesting relationships of sections and lists and the scope of their respective text. 6) Topic Chunking: Analyze the topic content of each document and create chunks that enclose areas of similar topics (topic-level chunks). 7) Topic Labeling: i) Using embedding and clustering, we create candidate data type and semantic role labels for each heading in the corpus. ii) Using keyphrase extraction techniques, we create candidate data type and semantic role labels for the chunks. 8) Chunk Labeling: Identifying and assigning potential data types and semantic roles to other chunks throughout the document using multiple methods, e.g., neural networks, word and character embeddings, grammar analysis and pattern matching, regular expressions, similarity metrics, and / or other methods. Of particular interest to certain embodiments are the following: i) Syntax analysis and pattern matching on the results structure ii) the use of question-answering techniques to connect small chunks with the specific semantic roles they play in the document; iii) Combining XPath tree matching and word embedding techniques to match patterns in structure and grammar trees despite potentially significant differences in phrasing and word choice. 9) Named Entity Recognition (NER): Identifies data types that are detected as named entities throughout the document and assigns them to chunks. 10) Role Labeling, Extraction Labeling: Assigning semantic role labels to chunks, such as indicating that a name constitutes the "seller" party in a contract, or that a drug is mentioned as an allergy remedy. 11) Anomalies: Identify semantic roles that are typically present or absent in documents in the document set under consideration but not in the current document (or vice versa). 12) Arbitration: Reconcile and / or select alternative scopes, data types, and semantic role labels for chunks to produce a well-formed structure that is easily representable in a format such as XML. 13) DGML: Create an augmented version of the document that includes explicit identification of chunk locations, data types, and semantic role labels, and possibly additional information such as the confidence level of each identified chunk and the data types expected in similar chunks (dates, date ranges, personal names, etc.). The augmented version is created using an XML-based markup language called DGML. 14) Feedback: Present the enhanced version to the user, select chunks (and possibly the potential locations of omitted chunks) and collect the user's choice to confirm, reject, or make other changes. The user is also free to choose their own reading and review order. Feedback can also be applied to any other interpretations made by the system, such as organizing the document into a document set, as described in step (2). i) In the case of possibly omitted chunks, prioritized examples from other documents are provided, which can be automatically customized by inspecting and / or copying the current document as needed and applying target document values ​​to smaller nested chunks. 15) Feedback Response: Tracking the user's responses to these interactions and using that information to fine-tune the model 120 as well as to prevent repeating the same or similar errors later. 16) Downstream Communication, Transmission: Selecting chunks by type and / or role and using them to generate reports via document sets and / or exporting them to downstream systems that add functionality such as back-end contract databases, regulatory compliance checkers, management report generators, etc.

[0044] Figure 2 is a screenshot showing a dashboard that tracks the processing of different document sets 1 to 7 through the process described above. In this dashboard, the process is divided into the following stages: Uploading Preprocessing Review Large Chunks Review Small Chunks Ready to Use Color coding indicates the degree of completion: green stages are complete, red stages are in progress, and black stages have not yet started.

[0045] Each of the steps listed above is described in more detail below.

[0046] Further Description of Exemplary Implementations The numbering here reflects the general order of analysis for this particular example. However, not every step is dependent on every previous step, and as a result, many elements can be reordered or parallelized in other implementations. Elements can also be shifted or repeated, exchanging additional information with other elements, or elements can be performed independently, such as in separate processes or machines.

[0047] 1) Import The system accepts typical word processor documents (e.g., MS Word) and page layout documents (e.g., PDF or PNG files). In each case, visually adjacent regions such as headings, paragraphs, table cells, tables, and images are identified and represented as chunks using a combination of their relative positions, surrounding white space, font and layout features, etc. These features are partly selected by the designer and partly learned through image and pattern analysis of numerous documents. For input documents that do not already have machine-readable text content, OCR is also applied.

[0048] These chunks, along with selected layout information, are presented to subsequent modules in the system.

[0049] 2) Organization Users are not required to organize the documents they check into the system. The system uses clustering methods that operate on text content, layout information, and already-detected structural information (such as identifying some headings) to group documents into "sets" of specific types of documents—for example, rental agreements versus leases versus sales. The specific document sets found can be identified by the user and named either automatically or by the user. Once established, these document sets facilitate subsequent machine learning and inference about format, content, semantic roles, and their differences. For example, the system may discover that almost all documents in a given set have a particular section with three specific subchunks of a specific role and a personal name data type, one of which is repeated in five different sections. Such patterns are used to help identify similar (and dissimilar) parts of other documents, suggesting changes or modifications to the user, and providing examples of text to reuse in other documents in the same (or perhaps different) sets.

[0050] Clustering documents into document sets can use features from the document's structure (order and containment relationships between chunks of various sizes, data types, and roles) and layout, as well as textual content. Once several chunks and / or roles have been identified in at least some documents, that information can also be used to improve the clustering, either by full reclustering or by smaller adjustments. For example, ignoring certain content in chunks with the same role, such as seller and buyer names and addresses, or noting that different chunks have the same occurrence pattern, e.g., one name (e.g., the seller's name) appears in certain places while another name (e.g., the buyer's name) appears in certain other places, can result in similar documents being nearly or even completely identical.

[0051] The system maintains both the original organization of uploaded files into directories (if any) and their own organization into sets. Thus, users can see both organizations and learning algorithms can use both as information. For example, some users name documents according to various conventions and / or organize documents by customer, document type, or other characteristics, which is useful for understanding similar patterns (such as having common chunk locations and roles) and relationships between documents.

[0052] 3) Visual Extraction i) Area detection The system uses heuristics and machine learning to identify regions within documents based on geometric patterns. For example, in many documents, meaningful chunks have special layouts such as signature blocks, abstracts, lists of definitions, tables, etc. Such patterns can be learned automatically by considering geometric and / or layout features, uniqueness or rarity, and / or correspondence either within the same document or between documents, especially within the same document set.

[0053] The approach is chosen depending on the format of the input document: for example, word processor documents generally provide explicit information about paragraph boundaries, while PDFs or scanned pages require the system to assemble them from visual lines, or even analyze white space dimensions and assign characters to visual lines (e.g., in multi-column documents).

[0054] ii) Signature detection The system creates signatures (also known as "digests") of document parts and uses these to identify and classify additional "interesting" chunks and detect their boundaries. The signatures are based not just on textual content but also on various aspects of the context, and may ignore the content of smaller contained chunks (e.g., field chunks whose content varies within their counterparts).

[0055] The signature may even use a pixel representation of the chunk. A bitmap image of the text layout is divided into tiles, preferably on the order of 24 pixels by 24 pixels (adjusted for the scan resolution), and the tiles are clustered. These neural network processes, including autoencoders and their adjacencies, reveal similar visual events such as boundaries between text and rules, edges and corners of text blocks, and even indentation changes and substantial font / style changes. Further neural networks then use this clustering to jointly identify similar layout objects that frequently indicate or characterize important chunks.

[0056] The approach here may use an unsupervised approach to generate document chunk embeddings based on the pixels and characters within the document chunk, the size of the chunk, its location within the document, etc. (As noted, images can also be chunks.) Clustering and comparison techniques can then be used on these embeddings for many downstream tasks.

[0057] iii) Extraction This aspect takes a layout document (e.g., a PDF or scanned printed page) and converts recognized character images (“glyphs”) in the document into a text stream that represents the correct document order of the glyphs (the stream may also include figures or image objects, if appropriate, and there may be multiple streams, such as footnotes or page headers, that do not have a typical place in the reading order). Some documents have an incomplete explicit representation of the reading order. A well-known example is that there is usually no indication that a multi-column layout is in effect at any given point, so the first “line” extends only halfway (or less) across, rather than the entirety. However, there are many additional examples where the text order may be complex or unclear. For example, some layout programs draw each character individually, making word boundaries unclear. Table cells, sidebars, figures, footnotes, and other displays may not have a clear position in the text order. Some text, such as in page headers and footers (as well as hyphens at the end of lines), may not require a place in the text order at all. Many formats do not provide a clear indication that something is in such a special category.

[0058] The system addresses this task by combining visual information about glyphs (position, style, etc.) with a deep neural network that understands the characteristics of the written language used in the document to construct a text stream. Additionally, it detects many basic text boundaries such as lines, blocks, columns, images, inline font changes, and header / footer objects.

[0059] iv) Expression Once the text sequence and some hypothesized structural chunks are extracted, the system creates a representation of the document (known in one example as "DGML") that includes information about them as well as their visual features (font, color, size, etc.). The representation of the chunks, including information about their location, type, role, etc., is called annotation. The combined data can then be used by natural language processing (NLP) and deep neural networks (DNNs). Deep neural networks incorporate this visual information and help structure the document into a hierarchy that represents the document structure, including chunks such as header / body, lists / list items, etc.

[0060] Enough information can be included so that subsequent aspects can construct an editable word processor document that closely resembles the original source. This, along with other structure, content, and chunk information, can be included in a DGML or similar representation. In many cases, portions of a document that have a clear format and layout are also useful chunks. However, formatting characteristics that do not match otherwise required chunks (and vice versa) can still be expressed via special types of chunks, via standoff annotations, or via other means.

[0061] 4) Structure The structural pipeline converts flat text files into a hierarchical structure, where sections, subsections, and other parts of a document form an ordered hierarchy of content-based objects, a structure known to those skilled in the art. This conversion is performed using unsupervised machine learning techniques. The method has several stages.

[0062] i) Hyperlining This involves segmenting the text into "hyperlines," which are larger groupings than visual lines and comprise more meaningful, logical (as opposed to visual) units such as paragraphs, headings, or the like. This is preferably achieved using a pre-trained neural network that considers features such as "word shape" of tokens (especially leading and trailing tokens), layout information such as font and spacing characteristics, and similar features. Some hyperlines may also be provided in previous steps (depending on the format of the input document).

[0063] ii) Document Language Model Instead of a language model based solely on the text, it is preferable to use a document language model that also includes information on the text content, formatting, and previously discovered structure. This allows for better detection of chunks and their hierarchy (header / body, list / list items, etc.) by learning to recognize meaningful chunks and patterns of their occurrence from formatted pages.

[0064] This creates a representation of the document that includes both textual content and visual features (geometry, font, color, size, etc.). Deep neural networks and NLP processes then utilize such information in the task of structuring the document into a hierarchy of chunks with data type and semantic role labels by discovering the extent and / or boundaries of chunks of various sizes that represent the document structure. At this stage, the discovered chunks are primarily headings, sections, lists and items, tables, figures, and other relatively large units.

[0065] iii) Hyperline Clustering It uses an autoencoder to cluster hyperlines across a document set based on word-shape structure, assigning each hyperline to a cluster of hyperlines that are similar in terms of layout, start and end content, and other characteristics, with each cluster identified by a "cluster ID" (this should not be confused with creating or identifying a document set).

[0066] iv) Inline headings A particularly interesting special case is an "inline heading", where the chunk's heading (which may provide the chunk's semantic role) is on the same line as the start of the text that follows, rather than being on a separate visual line by itself. Typically, inline headings are distinguished typographically by bolding, underlining, a different font, a trailing colon, or other effects. Separate heuristic and neural algorithms identify these chunks.

[0067] v) Small-shot structural learning Despite the above advanced structuring methods, it is expected that the generated structures will have certain imperfections or will not meet the user's prior expectations. Small-shot structure learning, as described in steps (14)-(15), deals with creating a machine learning model that relies on feedback provided by the user. This model is then used to generate structures that combine the user's feedback on structures with those already generated by the system (possibly iteratively reinforced by previous feedback).

[0068] The main principles applied in this case are derived from machine translation (MT) methods, where a sequence is transformed into another sequence: in this case, one sequence describing a hyperline is transformed into another sequence that also contains start / end markers that encode the hierarchy.

[0069] This process takes place in various phases or steps. (a) First, a machine translation model is pre-trained using a publicly available dataset. (b) The "Dispatcher" (see the "Feedback Response" section for an explanation) filters user feedback. (c) A new structure file is generated from user feedback to generate a fine-tuned machine translation dataset. (d) The pre-trained model is further trained using few-shot learning principles.

[0070] 5) Re-nesting This embodiment preferably uses a "corpus re-nesting" algorithm that, given a flat list of cluster IDs from the hyperline clustering step, uses a pushdown automaton to iteratively create nested structures. By comparing the signatures of adjacent hyperlines, the system can determine whether a given heading or list item belongs to a greater, equal, or lesser nested level. This allows for the reconstruction of the multiply nested hierarchical structure of many documents (chapters, sections, subsections, phrases, lists, etc.).

[0071] The features considered in re-nesting include the "shape" (as known in NLP techniques) of the tokens in the hyperline, considering in particular the beginning and end; the particular class of punctuation that terminated the previous line (if any); formatting information such as capitalization, leading whitespace, indentation, bolding, underlining, etc.; enumeration strings at the beginning of a line (e.g., patterns such as "IV(A)(1)" or "iv)"), or the presence and format of particular bullets or other emojis, the value of that enumerator, the presence, level, and value of preceding enumerators of the same type, etc.

[0072] 6) Topic chunking This aspect uses lexical statistics and other learning techniques across successive chunks of a document to detect where topics shift, which enhances the identification of boundaries of large chunks, such as entire sections on a given topic, because sections (at any level) generally have greater topic, vocabulary, and style uniformity within them than adjacent sections.

[0073] 7) Topic labeling i) Heading Labeler As shown in Figure 1, for each header in the corpus, this step Create a numerical representation, known as an "embedding," for each heading, Clustering headlines based at least on their embeddings, Filter out "bad" clusters based at least on measures such as density, articulation, and level of similarity; · Propagate the most common semantic role label in each remaining cluster to all headings in said cluster.

[0074] ii) Keyphrase Labeler For each chunk, this step uses an ensemble of keyphrase extraction techniques (such as rule-based linguistic techniques, ML, statistical, Bayesian, and / or others) to generate candidate semantic role labels for the text.

[0075] 8) Chunk Clubbing i) Grammar This aspect of the system begins with a linguistic analysis of text, such as natural language processing tasks including parts of speech tagging, dependency parsing, constituent analysis, and others. The system then applies tree-matching mechanisms from other domains to locate grammatical and other structures within the trees or tree-like structures discovered through NLP. These include document structuring methods such as tree grammars and tree pattern matching, as exemplified by XPath, GATE, and other tools.

[0076] Using such patterns to identify grammatical phenomena in a sentence allows the system to extract semantic role labels from the text itself, which are then used to annotate nearby chunks. For example, based on the sentence's constituent structure, a search pattern can be constructed that matches the sentence "The following are our agreed-upon terms" (and other sentences with similar grammatical structures), and then the noun phrase ("term" in this example) can be extracted and used as the semantic role label for one or more chunks in the content that follow this sentence and contain such a "term."

[0077] ii) Question and Answering Question answering techniques, including BERT for question answering, are specifically tuned to identify semantic role labels (e.g., date, person's name, dollar amount) for candidate chunks. In contrast, most traditional question answering models aim to answer questions like "What is an effective date?" This system instead trains a model to answer questions like "What is July 8, 2018?" and aims to predict "effective date" or "effective date of X," where X represents another chunk in text (rather than just "date," which is a data type rather than a semantic role).

[0078] The system also discovers synthetic questions that, when answered, can point to relevant information in the text, providing the ability to automatically suggest questions to be used by question answerers.

[0079] iii) XPath-like rules integrated with embedding Here, tools in the domain described under "grammar" are integrated with tools that provide vector-semantic representations of text, such as word2vec, char2vec, and many related methods. This system allows analysts to express and query patterns, including both structural information (which can include chunks of data represented in XML or DOM-compatible formats), which is well handled by XPath and similar tools, and fuzzy or "semantic" similarity information, which is well handled by vector models.

[0080] 9) NER (small unlabeled chunks) Techniques can identify some chunks by data type, such as person or company names, addresses, etc. (This is known as "Named Entity Recognition" or "NER"). However, NER falls quite short in identifying the semantic roles of these entities within a document. Current techniques also fail to identify larger chunks, such as entire phrases or sections, or groups of chunks that contain larger meaningful or useful chunks.

[0081] This aspect of the system detects interesting small chunks without necessarily assigning them roles. There are many methods and tools for identifying NERs in text. This system uses multiple methods, examples of which are listed below. These inventions are primarily unsupervised.

[0082] i) Established NER methods ii) predicted words By training an n-gram language model using a wide range of general text, such as Wikipedia, the system builds a model of "expected words in context for normal English." When looking at a particular document, the system provides a means to identify n-grams that do not fit that general model and therefore tend to be specific to the document being processed.

[0083] iv) TF-IDF This is a TF-IDF based approach ("term frequency vs. inverse document frequency"), used in conjunction with label propagation and contextual semantic labeling.

[0084] v) Sequence clustering Extract small word or character sequences such as n-grams and cluster them using contextual embeddings (e.g., those from BERT). The expected result is that n-grams that share semantic meaning will start clustering together. The cost of combinatorial explosion is addressed by using heuristics (including on the syntax tree) to filter out some n-grams before clustering. A wide variety of clustering algorithms can be applied. In this example, the hdbscan algorithm achieves effective clustering while assigning random noise to the "none" cluster.

[0085] vi) Small Shot NER The system uses few-shot learning techniques to generalize from a small number of labeled instances (e.g., selective user feedback) to more broadly applicable rules or adjustments of learned parameters. This significantly reduces the number of times the user must be asked for feedback, improving system performance more quickly.

[0086] 10) Extraction labeling This aspect of the system detects semantic role labels for smaller chunks that appear directly in the sentences that surround them. Meaningful chunks often have a role that is somehow specified by the context. For example: John Doe ("the seller") lives in... The $999 rent must be paid by the end of each month.

[0087] i) Contextual Semantic Labeling (CSL) This process uses a neural network operating on previously constructed structures, including sentence parsing, to learn which parts of the text are likely semantic role labels for various chunks. Many chunks may already have such labels, with various sources and confidence levels, but this provides additional evidence for or against them, as well as new labels. Some of the patterns here involve grammar. For example, in "Do shall pay $1,000 rent by the last business day of each month," the main verb reveals what the role of the monetary amount is—that it is rent due. Other patterns are learned automatically through supervised and / or unsupervised methods, using structure, chunking, labeling, and content features available in the context. Formatting such as parentheses, table layout, key phrases and words, and other features also provide features for the neural network.

[0088] Useful information is often present in chunks that contain sections or subsections, or their headings, etc. For example, whether a given drug is relevant as a prescription or as an allergy may only be detectable by looking at the headings of the sections it contains (this is another example of why detecting the correct hierarchical nesting of sections is important). Many other clues exist that can be learned by machine learning techniques and applied to discover roles applicable to various chunks. Cross-document similarity can also be used to associate semantic roles to documents, especially within the same document set, that are found in similar contexts but would not be discoverable in isolated documents.

[0089] ii) Label propagation This process standardizes labels across similar chunks of text in a corpus of documents. It applies both to labels extracted from context and to labels available from the previous step. The algorithm uses agglomerative clustering to cluster chunks based on their embeddings, a weighted PageRank algorithm (using label frequency / confidence as initial node weights) to rank candidate labels for each cluster of chunks, and co-occurrence and embedding similarity to determine how similar the labels are to each other. It then assigns labels to chunks based on their cluster-level scores and how similar the chunk we are labeling is to the original chunk for the label (in terms of content, embedding, structure, data type, semantic role, and / or context). It applies agglomerative clustering and PageRank algorithms to propagate labels across similar contexts, making the labels more consistent across the set of documents.

[0090] 11) Abnormality This aspect of the system examines multiple documents in a document set such as that generated in step (2) and identifies chunks that occur in the current document but generally do not have counterpart chunks in other documents in the same set, or vice versa. Counterpart chunks need not have identical content, structure, format, context, data type, and semantic role, but may have variations from one document to another. They can nevertheless be recognized as being substantially similar in their manner to other identified chunks.

[0091] If the new document contains chunks that are not typically present in other documents in the same set, the user may query for some or all of them to confirm that they were indeed intended. In this exemplary system, such queries are more common when the chunks in question are common to the new document and those it was based on (if any), but rarely common to the others.

[0092] If the new document lacks counterpart chunks that are typically present in other documents in the same set, or even in particularly relevant external sources (e.g., house style manuals, compliance requirements, etc.), examples of some or all of such chunks are suggested to the user, along with content drawn from the other documents. Suggestions may be ranked to the user according to factors such as frequency of use, being most typical (centroid) of available alternatives, or having a high likelihood of co-occurrence with other chunks present in the new document. Chunk suggestions may be automatically updated, for example, to replace names, dates, and other subchunks specific to the document from which the example was drawn with values ​​drawn from the new document.

[0093] Furthermore, the selection of chunks to be proposed for addition or deletion can usefully depend on the practices of different authors, editors, or other staff. For example, if the current author's documents frequently differ from those of another author on a particular measure, that difference may indicate a considered choice and not an error. On the other hand, if all authors working under the same supervisor do something in a way that the current author does differently, that may indicate a greater need for review, at least when first noticed.

[0094] Anomaly modeling considers the structure and data types and semantic roles of chunks as well as the context, content, and format. For example, modeling patterns of what chunk data types and semantic roles occur within, adjacent to, or near others. Violations of well-established patterns can be classified as anomalies and presented for user feedback like any other anomaly.

[0095] 12) Arbitration Many previous steps create and / or operate on chunks of the document, defined (typically, but not necessarily contiguous) as ranges of characters, tokens, and / or non-text objects within the linear sequence produced in step (3).

[0096] The chunk being considered at any time can be represented either by "inline" meta-information such as markup, or by a "standoff" representation that points to its location in the text by various kinds of pointers. In this example, the standoff representation is used for most processing, while the inline representation is used for some purposes such as communication with external tools, which often prefer it. These and other representations are functionally interchangeable, and the choice between them can be governed by concerns such as performance, convenience, etc.

[0097] Representations of chunks include information about the steps or implementations that created them, how certain they are ("confidence level"), and their specific data type and / or semantic role labels. Redundant, uncertain, conflicting, or partially overlapping chunks can frequently occur, which we refer to as "non-optimal." For example, two or more different processes may attach semantic role labels to the same span of text (or nearly the same span, e.g., one with "Dr." before a name and one that does not). Chunks may be nested, sometimes deeply, but may also arbitrarily overlap (i.e., each overlapping chunk contains some content that is also in the other and some that is not). Through the above steps, the system may maintain representations that can represent numerous annotations, including overlapping or co-located ones.

[0098] Such non-optimal chunks are typically undesirable, at least when the document is presented to a user. Furthermore, many state-of-the-art NLP tools prefer non-overlapping structures, as do many document tools and methods familiar to those skilled in the art, such as XML, JSON, SQL, and other representation systems. A more constrained structure, often referred to as "hierarchical" or "well-formed," is typically preferred, avoiding partially overlapping chunks.

[0099] This aspect of the system modifies the collection of chunks to strictly stratify and avoid non-optimal chunks. This can be achieved in several ways. First, chunks can be removed entirely (i.e., the chunks themselves, not the document content they identified, are removed). Second, chunk scope can be modified (e.g., by including or excluding one or more characters or tokens from either end) to prevent overlap with another chunk. Third, chunks can be determined to be redundant and merged. Fourth, chunks can be detected as being inconsistent (e.g., if one tool considers "Essex" a place and one a person), and a selection can be made.

[0100] This process includes means to quickly detect cases of partial and / or complete duplication, compare chunks by type, role, and confidence, and resolve non-optimal cases by modifying chunks and their associated data. Selecting which chunks to modify, merge, or remove takes into account several factors, such as confidence level, prior probability of a given chunk's data type, semantic role, and content, hyponyms between semantic role labels, conditional probability of occurrence in a given context, the number, role, and distribution of other chunks in the current and other similar documents, current process priority, customer feedback about similar cases, and / or other methods.

[0101] Modifications may also change the confidence level of chunks. For example, some aspects of the system may apply similar or identical semantic role labels to the same or nearly the same parts of a document. In that case, the labels are typically merged, and the resulting chunk is assigned a higher confidence than the individual chunks it contains. In other cases, a choice is made between conflicting chunk assignments, but the selected chunk may end up with a reduced confidence that reflects some level of counter-evidence.

[0102] This process improves the quality and consistency of chunk identification and labeling, allows the information to interoperate with a wide range of tools, and makes the results easier and more reliable to analyze. The operations described above can be applied at any time, not just at the end. For example, if a previous step uses an external tool for some subtask, it may require reduction to well-formedness. Deleted or modified chunks can instead be "paused," meaning that they no longer affect processing but can be reintroduced on demand. This allows for the use of non-overlapping supporting tools without the need to later reproduce the previous work from scratch, increasing the flexibility and speed of processing.

[0103] In one approach, all duplicates and / or all non-optimal chunks are resolved before generating the document shown to the user, so that the results can be easily encoded in a hierarchical format, such as the XML format used by many modern word processors and other tools. However, it is also possible (even in XML) to maintain multiple potentially duplicate alternatives at a particular location for potential later resolution, such as through user feedback or improved algorithmic learning.

[0104] 13)DGML(DocuGami Markup Language) The enhanced version of the document represents the document structure, format, content, and identified chunks, and may identify which chunks were identified at which step of the process and with what level of confidence. Some embodiments use XML as the syntax for this representation, but a wide range of representations can contain substantially the same information, such as other XML schemas, JSON, various databases, custom text or binary formats, etc.

[0105] In this step, information about the document and detected chunks is converted (or "serialized") into XML format so that it can be more easily passed to other processes, particularly front-end user interfaces used for feedback, editing, and review, and in a format useful for "dashboard" applications that provide overviews, statistics, and compliance information to other users such as group managers, quality control staff, etc.

[0106] DGML (Docugami Markup Language) is a specific XML schema for this use, which houses all of the described information in one package. Most previous schemas may address structure, content, and sometimes layout, but do not annotate "chunks" abstractly as described here. Many previous schemas also do not provide a generalized mechanism that can automatically detect and represent chunks on the fly, especially with confidence levels and provenance information.

[0107] It is also possible to "tunnel" the same information with the file formats of some word processors and other tools by representing it in a format that is transparent to the format. For example, if the tool supports embedded comments or metadata, "invisible" text, ignorable attributes, or other similar features, the information described herein can be hidden therein, allowing the resulting document to be used, and possibly modified, by the tool, and returned to the system with the tunneled information still available.

[0108] 14) Feedback mode front end The extensive annotations and analysis attached to documents and their discovered chunks in the manner described above guide users through editing of samples, templates, or previous documents, making it feasible to create similar but new documents customized to current needs. For example, the system typically identifies the parties and assets covered by a contract, medical history, current findings, and medications or conditions mentioned in other specific sections of clinical notes, relevant dates, etc. By examining other documents in the same document set, the system learns which aspects are incommon, common, or necessary, and can therefore make more useful recommendations to the user about what to review and / or update. For example, an effective date may be present in nearly every contract in a document set, but its value may vary. Similarly, the parties vary, but the type of party is much more consistent.

[0109] i) Non-guided feedback In its interaction with the user, the system first requests feedback on chunks that have been (or may not have been) detected in some documents. The first few documents presented for feedback become the "cluster centroids" of the document set. The last few become the "outliers" of the document set.

[0110] ii) Inductive Feedback The system then shows the user selected portions of the document and prompts the user to provide feedback by asking about their current or potential labels, their scope, etc. a. "Interesting labels" are determined by a PageRank-based algorithm and a grammar and structure model. Of these labels, a set of low-confidence instances are selected for review. b. If low-confidence labels are no longer present in the current document, the same process may be repeated for additional documents. In some embodiments, the model is continuously updated based on user-provided feedback. However, feedback may alternatively be accumulated and applied later, in batches, and / or offline. Adjustments to the model may then affect the selection of chunks and labels subsequently presented for feedback and may trigger a re-analysis of some documents. c. The system uses essentially the same mechanism to solicit feedback on field and structure chunks. In one approach, all chunk detectors provide a reliability estimate, which can be used along with other information to select candidates for feedback.

[0111] Feedback may be requested in different passes, for small vs. large chunks, fields vs. structure chunks, or in other orders. See FIG. 3 for an example user interface for user feedback. Some or all chunks are displayed, allowing the user to select and inspect specific ones to see their assigned types and / or roles, and optionally alternatives. The user can move chunk boundaries, select or edit labels, etc. Preferably, the user can also request that a particular change (to a label, etc.) be applied to all corresponding or identically typed chunks. 15) Feedback Response i) Fleet Query is a method that allows a system to query both private and public data based on user feedback, typically from multiple users. The selected examples are semantically and syntactically similar to previous failure cases, which increases the value of the feedback. ii) Dispatcher: A dispatcher is a methodology for connecting user feedback on the combined output of several ML models and non-ML algorithms back to a specific learning model 120 that can learn from the feedback.

[0112] The system allows the model to be improved from user feedback on its output and on the output of other trained and untrained models. This is achieved by using the feedback as incremental (also called "fine-tuning") training data for several of the described numerical and neural models. After using the feedback to improve the model, not only the specific document but all documents in the set, or even all of the user's documents, are re-evaluated. Thus, feedback on each document can improve chunk identification, role assignment, structure discovery, and therefore user assistance, for all documents. This re-training is represented by the dotted connector from step (15) to step (3) in Figure 1.

[0113] Documents and all associated information contribute to learning and analysis of the set of documents (particularly, but not exclusively, within a particular document set), thus improving the performance of future documents. For example, when a new chunk is added to one or more documents in a set, it becomes available for use in future documents (or revisions of old ones) and can be suggested for future documents. At some point, the absence of a recently introduced chunk role or the presence of a chunk role that has not been used much recently can become an anomaly. This point can be selected spontaneously, by the user in response to a feedback question, or automatically based on the usage curve of the counterpart chunk over time. For example, if few documents in one set created before a certain time contain a chunk of a given role and / or context (e.g., an "exclusions" section), but most or all documents created later do, the absence of the counterpart chunk is likely an anomaly in the new document and can be usefully presented to the user as such.

[0114] 16) Downstream Communication After annotating a document with chunk information as described, the selected information is converted into the specific format required by external business information systems, such as databases, analytical tools, etc., and passed to those systems either directly or through automated and / or manual review steps. For example, the names and addresses of specific parties can be copied into the correct fields in a database, which cannot be done automatically if they were identified only as "name" and "address" themselves. See Figure 4 for an example of integration with a downstream software application. In this example, chunks representing terms on which the parties are expected to agree are extracted, and they are passed to a downstream application, similar to Docusign, to be completed and signed.

[0115] FIG. 5 is a block diagram of one embodiment of a computer system 510 that can be used with the present invention. The steps described above can be implemented by software executing on such a computer system. The computer system 510 typically includes at least one computer or processor 514 that communicates with peripheral devices via a bus subsystem 512. Typically, the computer can include or be any of a microprocessor, graphics processing unit, or digital signal processor, and their electronic processing equivalents, such as an application-specific integrated circuit (ASIC) or field-programmable gate array (FPGA). These peripheral devices can include a storage subsystem 524 comprising a memory subsystem 526 and a file storage subsystem 528, user interface input devices 522, user interface output devices 520, and a network interface subsystem 516. The input and output devices enable user interaction with the computer system 510.

[0116] A computer system may be a server computer, a client computer, a workstation, a mainframe, a personal computer (PC), a tablet PC, a rack-mounted "blade," or any data processing machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by that machine.

[0117] A computer system typically includes an operating system, such as Microsoft's Windows®, Sun Microsystems' Solaris®, Apple Computer's MacOs®, Linux®, or Unix®. A computer system may also typically include a basic input / output system (BIOS) and processor firmware. The operating system, BIOS, and firmware are used by the processor to control the subsystems and interfaces connected to the processor. Typical processors compatible with these operating systems include Intel's Pentium® and Itanium®, Advanced Micro Devices' Opteron® and Athlon®, and ARM Holdings' ARM® processors.

[0118] The inventions, embodiments, and / or examples of the claimed inventions are not limited to traditional computer applications or programmable devices that execute them. For example, the inventions, embodiments, and / or examples of the claimed inventions may include optical computers, quantum computers, analog computers, etc. The computer system may be a multiprocessor or multicore system and may be used or implemented in a distributed or remote system. The term "processor" is used herein in its broadest sense to include single processors and multicore or multiprocessor arrays, including graphic processing units, digital signal processors, digital processors, and combinations of these devices. Furthermore, while only a single computer system or single machine may be illustrated, the use of the singular form of such terms also refers to any collection of computer systems or machines that individually or jointly execute instructions to perform any one or more of the operations discussed herein. Due to the ever-changing nature of computers and networks, the description of computer system 510 shown in FIG. 5 is intended only as one example for purposes of illustrating a preferred embodiment. Many other configurations of computer system 510 may have more or fewer components than the computer system shown in FIG. 5.

[0119] The network interface subsystem 516 provides an interface to external networks, including an interface to a communications network 518, through which it is coupled to corresponding interface devices in other computer systems or machines. The communications network 518 may include many interconnected computer systems, machines, and communications links. These communications links may be wired, optical, wireless, or any other device for communicating information. The communications network 518 may be any suitable computer network, for example, a wide area network such as the Internet and / or a local area network such as Ethernet. The communications network may be wired and / or wireless, and may use encryption and decryption methods, such as those available in virtual private networks. The communications network uses one or more communications interfaces capable of receiving data from other systems and transmitting data to other systems. Embodiments of the communications interface typically include an Ethernet card, a modem (e.g., telephone, satellite, cable, or ISDN), an (asynchronous) digital subscriber line (DSL) unit, a Firewire interface, a USB interface, etc. One or more communication protocols may be used, such as HTTP, TCP / IP, RTP / RTSP, IPX and / or UDP.

[0120] User interface input devices 522 may include pointing devices such as alphanumeric keyboards, keypads, mice, trackballs, touchpads, styluses, or graphics tablets, scanners, touchscreens integrated into displays, audio input devices such as voice recognition systems or microphones, eye gaze recognition, brainwave pattern recognition, and other types of input devices. Such devices may be connected to the computer system via wires or wirelessly. In general, use of the term "input device" is intended to include all possible types of devices and means for inputting information into the computer system 510 or over the communications network 518. User interface input devices typically allow a user to select objects, icons, text, etc. to be displayed on some type of user interface output device, e.g., a display subsystem.

[0121] The user interface output devices 520 may include a display subsystem, a printer, or a non-visual display such as an audio output device. The display subsystem may include a flat panel device such as a liquid crystal display (LCD), a projection device, or some other device for creating a visible image, such as a virtual reality system. The display subsystem may also provide a non-visual display, such as through audio output or haptic output (e.g., vibration) devices. In general, use of the term "output device" is intended to include all possible types of devices and means for outputting information from the computer system 510 to a user or to another machine or computer system.

[0122] The memory subsystem 526 typically includes several memories, including a main random access memory (RAM) 530 (or other volatile storage device) for storing instructions and data during program execution, and a read-only memory (ROM) 532 in which fixed instructions are stored. The file storage subsystem 528 provides persistent storage for program and data files and may include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, a flash memory, or a removable media cartridge. Databases and modules implementing the functionality of particular embodiments may be stored by the file storage subsystem 528.

[0123] The bus subsystem 512 provides devices for allowing the various components and subsystems of the computer system 510 to communicate with each other as intended. Although the bus subsystem 512 is shown schematically as a single bus, alternative embodiments of the bus subsystem may use multiple buses. For example, a RAM-based main memory may communicate directly with a file storage system using a direct memory access (DMA) system.

[0124] Although the detailed description contains many details, these should not be construed as limiting the scope of the invention, but merely as illustrating different examples. It should be understood that the scope of the present disclosure includes other embodiments not described in detail above. Various other modifications, changes, and variations apparent to those skilled in the art may be made in the arrangement, operation, and details of the methods and apparatus disclosed herein without departing from the spirit and scope, as defined by the appended claims. Therefore, the scope of the present invention should be determined by the appended claims and their legal equivalents.

Claims

1. 1. A computer-implemented method performed by a computer system executing instructions for assisting a user in developing a target document belonging to a document set, comprising: accessing a document set including a plurality of documents, the document set identifying chunks within the individual documents of the document set and including data type and semantic role labels for some of the chunks, the semantic role labels describing the semantic roles played by the chunks within each document; (a) deriving patterns of occurrence of semantic roles played by chunks across the documents in the document set, and (b) patterns of occurrence of counterpart chunks in different documents across the document set, where counterpart chunks play the same semantic role in different documents; providing a user interface for a user to develop a target document belonging to the document set; automatically generating suggestions for developing the target document based on the derived patterns of occurrence across the document set and displaying the suggestions within the user interface; 10. A computer-implemented method comprising:

2. The computer-implemented method of claim 1 , wherein deriving the pattern of occurrence comprises using machine learning and / or artificial intelligence to derive the pattern of occurrence.

3. The computer-implemented method of claim 1 , wherein the documents in the document set are all of the same document type.

4. deriving patterns of occurrence of chunks within individual documents in the document set, wherein automatically generating suggestions is further based on such derived patterns. The computer-implemented method of claim 1 .

5. The chunks in the document set are: field chunks containing content within the document suitable for use as fields in a document template, some of the field chunks being hierarchical and containing other chunks as sub-chunks; and structural chunks containing content from structures within the document's layout, the semantic role labels including semantic role labels for some of the structural chunks; a chunk containing an image or video; The computer-implemented method of claim 1 , comprising:

6. 2. The computer-implemented method of claim 1, further comprising: comparing chunks in the target document to the derived patterns of occurrence of semantic roles and / or counterpart chunks across the document set; and wherein a number of suggestions are automatically generated based on the comparison.

7. identifying anomalies in occurrences of semantic roles in the target document compared to the derived patterns of occurrence of semantic roles and / or counterpart chunks across the document set, wherein at least one suggestion is automatically generated based on the identified anomalies; The computer-implemented method of claim 1 , further comprising:

8. the identified anomaly includes identifying a semantic role that is missing in the target document but commonly occurs in the document set; The computer-implemented method of claim 7 , wherein the automatically generated suggestions include adding content for the missing semantic roles.

9. the identified anomaly includes identifying an extra semantic role that occurs in the target document but that does not commonly occur in the document set; The computer-implemented method of claim 7 , wherein the automatically generated suggestions include removing or modifying chunks corresponding to the extra semantic roles.

10. the identified anomaly includes identifying a semantic role that occurs in the target document and that is also commonly occurring in the document set, but the content of a corresponding chunk in the target document does not match the content of the corresponding chunk in the document set; The computer-implemented method of claim 7 , wherein the automatically generated suggestions include removing or modifying the mismatched content in the target document.

11. the identified chunks include field chunks containing content within the document suitable for use as fields in a document template; the derived pattern includes, for one of the field chunks, a pattern in which the counterpart chunks all contain substantially the same content; The computer-implemented method of claim 1 , wherein at least one automatically generated suggestion includes, for the field chunk, entering the same content into the target document.

12. The computer-implemented method of claim 1 , wherein displaying suggestions to the user includes displaying several suggestions with the user having the option to accept or reject the suggestions.

13. (a) in response to the user accepting an individual suggestion for the target document, repeating the same suggestion for a second target document exhibiting the same pattern in the target document that led to the generation of the accepted suggestion in the target document; (b) in response to the user rejecting an individual suggestion for the target document, not repeating the same suggestion for a third target document that exhibits the same pattern in the target document that led to the generation of the rejected suggestion in the target document; The computer-implemented method of claim 1 , further comprising at least one of:

14. 2. The computer-implemented method of claim 1, further comprising automatically applying some suggestions, and wherein displaying the suggestions to the user includes displaying an option for the user to confirm the automatically applied suggestions.

15. The computer-implemented method of claim 1 , wherein displaying the suggestions comprises displaying the suggestions in the user interface in a ranked order by confidence in the suggestions.

16. 2. The computer-implemented method of claim 1, further comprising automatically generating additional suggestions for the target document based on patterns within the target document itself and / or based on patterns in documents outside the document set, wherein the suggestions based on patterns in the target document, based on patterns in the document set, and based on patterns in documents outside the document set are displayed in the user interface with different priorities.

17. The computer-implemented method of claim 1 , wherein the automatically generated suggestions are further dependent on the user and / or the user's affiliation.

18. 2. The computer-implemented method of claim 1, wherein at least one of: (a) the target document is an existing document being edited by the user and the automatically-generated suggestions include suggestions for editing the existing document; and (b) the target document is a new document being created by the user and the automatically-generated suggestions include suggestions for creating the new document.

19. 1. A non-transitory computer-readable storage medium storing executable computer program instructions for assisting a user in developing a target document belonging to a document set, the computer program instructions being executable by a computer system, the computer system comprising: accessing a document set including a plurality of documents, the document set identifying chunks within the individual documents of the document set and including data type and semantic role labels for some of the chunks, the semantic role labels describing the semantic roles played by the chunks within each document; (a) deriving patterns of occurrence of semantic roles played by chunks across the documents in the document set, and (b) patterns of occurrence of counterpart chunks in different documents across the document set, where counterpart chunks play the same semantic role in different documents; providing a user interface for a user to develop a target document belonging to the document set; automatically generating suggestions for developing the target document based on the derived patterns of occurrence across the document set and displaying the suggestions within the user interface; A non-transitory computer-readable storage medium for causing a method to be performed, comprising:

20. 1. A computer system for assisting a user in developing a target document belonging to a document set, the computer system comprising: a storage medium that receives and stores a document set including a plurality of documents, the document set identifying chunks within the individual documents of the document set and including data type and semantic role labels for some of the chunks, the semantic role labels describing the semantic roles played by the chunks within their respective documents; a processor system having access to said storage medium and executing an application program for developing said target document; Equipped with The processor system that executes the application program includes: (a) deriving patterns of occurrence of semantic roles played by chunks across the documents in the document set, and (b) patterns of occurrence of counterpart chunks in different documents across the document set, where counterpart chunks play the same semantic role in different documents; providing a user interface for a user to develop a target document belonging to the document set; automatically generating suggestions for developing the target document based on the derived patterns of occurrence across the document set and displaying the suggestions within the user interface; A computer system that executes

Images