Cross-document intelligent authoring and processing assistant

Abstract

To analyze a document to annotate it.SOLUTION: Machine learning, artificial intelligence, and other computer-implemented methods are used to identify semantically important chunks in documents, the semantically important chunks are automatically labeled with appropriate datatypes and semantic roles, and this enhanced information is used to assist authors and to support downstream processes. Chunk locations, datatypes, and semantic roles can be automatically determined from what is called "context", to wit, combination of their formatting, structure, and content, those of adjacent or nearby content, overall patterns of occurrence in a document, and similarities of all these things across documents. Similarity may include similarity of natural language grammatical structure, machine learning techniques such as measuring similarity of word, chunk, and other embeddings, and the datatypes and semantic roles of the previously-identified chunks.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure generally relates to methods and apparatus for AI self-governed creation of hierarchically semantically labeled documents and / or for assisted authoring and processing of such documents.

[0002] Cross-reference to Related Applications This application claims priority under 35 USC § 119(e) to U.S. Provisional Patent Application No. 62 / 900,793, filed September 16, 2019, entitled "Cross-Document Intelligent Authoring and Processing Assistant". All of the foregoing subject matter is hereby incorporated by reference in its entirety.

Background Art

[0003] Description of Related Art Many companies create multiple documents that are very similar, even though they are customized each time. For example, an insurance office may make many proposals for a particular type of insurance, but each must be tailored to the needs of a particular customer. These documents can be considered the same "type" because they have similar text (and sometimes images) content that reflects similar purposes and topics, similar selections and arrangements of large units such as sections, and often similar geometric layouts and formatting characteristics.

[0004] While some document types are widely known and used, many are not. Many are specific to a particular business, market, or application, and new ones are created to suit new circumstances. Users, who may be called “authors” or “editors,” typically create a new document of a particular type (sometimes called a “target document”) by copying a previous document of the same type and modifying it as needed, for example, by manually editing or replacing specific chunks of content.

[0005] In current practice, word processing typically identifies chunks only when necessary to achieve formatting; for example, headings, footnotes, and figures may be explicitly marked to obtain special formatting, while names, addresses, or dates are rarely explicitly marked. Even when identified, chunks are generally associated only with formatting effects (margins, fonts, etc.) that are useful information, but no direct representation of any of these data types or semantic roles is provided. Similarly, word processors often represent hierarchical storage only visually, often without explicit representation of the nested sections themselves, only headings with different formats.

[0006] When creating a new document of the same general type as a previous document, in many cases, the majority of the work involves editing, replacing, deleting, or inserting text in specific chunks, taking care not to confuse those with different semantic roles (such as swapping buyer and seller addresses). This typically requires human intervention because authoring systems usually have no knowledge of these chunks, especially their data types or semantic roles, and cannot serve them very effectively.

[0007] In some simple cases, "forms" and "templates" can be used to provide an explicit location for populating the content of a particular chunk. However, forms typically only address simple cases where virtually all necessary chunks can be pre-enumerated and there are few large, repeatable, or highly structured chunks. Furthermore, creating forms requires skilled effort, is difficult to adapt to changing circumstances, and does not actively assist the writer. [Brief explanation of the drawing]

[0008] This patent or application file includes at least one drawing made in color. Copies of the publication of this patent or patent application and the color drawings will be provided by the Office upon request and payment of the required fees.

[0009] Embodiments of this disclosure, in conjunction with the embodiments in the accompanying drawings, have other advantages and features that will become more readily apparent from the following detailed description and the accompanying claims.

[0010] [Figure 1] Figure 1 is a block diagram of one implementation of a system and process for creating hierarchically semantically labeled documents using machine learning and artificial intelligence. [Figure 2] Figure 2 is a screenshot showing a dashboard that tracks the processing of different document sets through the system in Figure 1. [Figure 3] Figure 3 is a screenshot of the user interface for receiving user feedback. [Figure 4] Figure 4 is a screenshot of the integration with other software applications. [Figure 5] Figure 5 is a block diagram of one embodiment of a computer system that may be used in conjunction with the present invention. [Modes for carrying out the invention]

[0011] Detailed description of preferred embodiments overview Groups of documents determined to be the same type constitute a “document set” or “document cluster.” For example, a specific type of insurance offer to a particular class of customers of an insurance company may be considered the same type and form a document set. Different types of insurance offers from the same company, or offers to customers they consider to be different, may be considered different types belonging to different document sets. Rental agreements, clinical notes for a specific type of patient, sales proposals, schedules, meeting minutes, etc., are other potential types of documents, as well as subtypes that share characteristic patterns of content, structure, and / or layout.

[0012] Creating and editing new target documents within a set of documents very often involves editing or replacing "chunks" that are "semantically important," and such chunks are typically not necessarily a continuous span of text, but rather specific parts of a document that have a particular data type and semantic role and are meaningful and significant to the business or other processes.

[0013] These chunks represent various data types and are finer than the smallest data types in many computer systems. For example, a given chunk could represent not just a string, but also a person's or organization's name, a date, a period (not necessarily the same as a date), or a currency amount. Larger chunks could include bundles of information such as a list of drugs or other substances, an itinerary, instructions to follow, a medical prescription, and countless other things.

[0014] Furthermore, chunks may have semantic roles in relation to the documents in which they occur. For example, a person's name could be the "tenant" in a lease agreement, the "seller" in a sales proposal, or the "agent" of another person. A date could represent the start or end of a particular responsibility or activity. Amounts in dollars could represent a periodic payment, or a penalty or bonus associated with specific conditions, etc. Such semantic roles are important for ensuring the proper use of the information within a chunk. The names of these semantic roles are called "semantic role labels" or simply "labels."

[0015] A chunk is typically represented as a bundle containing its location, data type, semantic role, and / or other data / metadata. Location is generally represented as a start and end point and can be expressed in several ways, such as an inserted marker or byte, character, or token offset (either global to the document or relative to an established ID, marker, or other object). The semantic role is represented by a label or other identifier. Chunks can be of any size and some may contain other chunks as “subchunks.” Chunks can contain not only text but also non-text data such as images or other media, and “structures” such as tables, lists, and sections.

[0016] The techniques disclosed herein use machine learning, artificial intelligence, and other computer implementations to identify various semantically significant chunks within a document, automatically assigning them appropriate data types and semantic roles, and using this enhanced information to assist authors and support downstream processes. The location, data type, and semantic role of chunks can often be automatically determined from what is called the "context," i.e., their format, structure, and content combination, adjacent or nearby content, overall occurrence patterns within the document, and the similarity of all these things across documents (primarily, but not exclusively, between documents within the same set of documents). "Nearby content" includes horizontally close content such as preceding and succeeding content in a text reading sequence, as well as vertically close content such as within the same container structure, such as lists and sections, along with their respective markers, headings, levels, etc. Similarity is not limited to the comparison of exact or ambiguous strings or characteristics, but may include ML (machine learning) techniques such as measuring the similarity of natural language grammatical structures, words, chunks, and other embeddings, as well as the data types and semantic roles of previously identified chunks.

[0017] For example, a person's or organization's name can generally be identified as having a semantic role, such as "seller," because the document states so, and very often uses a sentence in some human languages, but often also includes a larger context. Another example is one or more words, which can often be easily identified as representing a data type, such as "drug name," but context is needed to determine that it carries the semantic role of allergy rather than prescription. Often, crucial evidence of a semantic role is expressed not in the same sentence, but in various other ways, such as in chunks occurring in larger chunks (e.g., the "Known Allergies" section). The flexibility and diversity of grammatical and document structure (not to mention typos, transcription errors, etc.) do not make identifying data types more difficult, but they do make identifying semantic roles, especially those that have a larger scope than a single sentence, much more difficult.

[0018] A given semantic role can relate a chunk to the entire document or to other chunks. For example, a plane's departure time is tied to a specific "leg" in the itinerary and only indirectly related to others. Generally, the hierarchical structure of chunks appropriately groups such items, such as co-location within sections or table parts.

[0019] More specifically, examples of semantically important chunks include the names, addresses, and other characteristics of specific parties to a contract, prescribed medications and prohibited procedures in medical records, requirements (or exclusions) in a real estate proposal, and dates and flight numbers in an itinerary. All of these can be considered the semantic roles of chunks. There are also larger chunks with various types and roles, such as entire sections and subsections. These are often inserted or removed as a whole, possibly accompanied by changes to smaller chunks within them. Chunks can be hierarchical; that is, a larger "containing" chunk can contain any number of other "subchunks."

[0020] A chunk is generally a continuous sequence of words within a document, such as "John Doe." However, chunks can also contain partial words. "John Doe's House" includes the name, but the name ends before the apostrophe (in the middle of a word). Chunks can even be discontinuous, for example, the same name "John (also called "Bill") Doe." Layout can also make chunks discontinuous; for example, page breaks can occur in the middle of a chunk (perhaps with a page header, footer, or footnote, which may be ignored depending on the purpose), or with intervening figures, tables, charts, sidebars, or other displays.

[0021] The actual location and context of a chunk can also be important, as chunks are not merely isolated strings but can occur multiple times with different (or none at all) semantic roles for different instances. More modern systems typically support inline or standoff markup, sometimes called “annotations,” which can persistently associate various labels and other information with chunks. HTML, for example, provides tags for manually labeling the boundaries of common structural chunks (such as “div,” “ol,” etc.) and several broad types or roles for (typically) smaller chunks (such as “initials,” “kbd,” “dfn,” “cite,” etc.). Other XML schemas provide many other labels, and word processors allow for somewhat similar labeling through “styles.”

[0022] Some chunks may represent what is generally referred to as a "field". These are often small chunks and often occur in the same context and layout in many or all of the documents within a given set, but usually have different text contents respectively. This can also occur multiple times in a single document with the same or very similar content. Such chunks may be called "field chunks". These are often discovered manually and treated as "fields" in a template-based system, but here they are discovered by the context and pattern of occurrence within and across documents, and data types and semantic roles are assigned in much the same way as to other chunks. They may or may not represent named entities such as personal names, addresses, dates, etc.

[0023] Another common type of chunk may be called a "structure" or "structural" chunk. Such chunks are typically larger and often contain many other chunks (some of which may also be structure chunks). They often have a "title" or "heading" that frequently provides a name, number, description, and / or other information about the structure chunk. Examples of structure chunks include chapters, sections, tables, figures, sidebars, and many more. The type and semantic role of a structure chunk are often important for determining the type and semantic role of nearby or contained chunks.

[0024] Not only data types, but also specific semantic roles are important for properly creating and utilizing documents. Whether a given name represents a buyer to a seller, or a patient to a doctor, whether a given date is the start or end date of a requirement, or the departure vs. arrival time of a flight, whether a numerical value specifies principal, interest, dosage, temperature, penalty, or something else, is very important. In the case of larger chunks, the roles include "limitation of liability" statements vs. "governing law" specifications vs. "definitions", and countless others. The semantic roles of chunks are often specific to a particular domain or transaction, and are perhaps one of the most important functions of a document. In many types of documents, chunks with specific data types and semantic roles are required, or at least very common, and the chunks are called "counterparts" when corresponding across the document. Counterpart chunks can occur in a similar order and pattern, especially for documents by the same author or organization, and usually for the same document set. Counterpart chunks have the same or very similar roles, and generally have a similar context and / or format. Thus, the distribution of the data types and semantic roles of chunks provides valuable information for distinguishing document types, as well as for identifying counterpart chunks in other documents.

[0025] Many counterpart chunks have similar content, but there are others that do not. For example, the same party (semantic role) within different documents is usually a different individual, but appears within a very similar context and pattern of use. This can be particularly common, but not limited to, "field chunks".

[0026] When discovered, hierarchical semantic chunks with data types and semantic roles within business documents can be used in downstream business processes. For example, a back-office database can properly record a new mortgage if given numerical values ​​such as the names of specific parties, specific dates, periods, and interest rates. Semantic roles are particularly important in such applications, and entering the correct data type into the wrong database field (such as swapping the names or addresses of sellers and buyers) can cause significant problems, especially when moving information to downstream databases, processes, or reports.

[0027] Some features and benefits The technologies described herein may have a variety of features and advantages, including any of the following:

[0028] Several implementations may offer a simpler, more efficient, and more accurate method for generating documents with hierarchically organized chunks that have semantic labeling useful for business processes. This can be achieved using various techniques to identify such chunks of varying sizes, discover the data types and semantic roles they play within documents, and learn their usage patterns, characteristic contexts, etc. Learning can be derived from analyzing the content, structure, and format of current and previous documents, feedback from authors and editors, as well as comparing multiple documents, especially those within the same set. This knowledge allows the system to provide users with valuable assistance, such as facilitating the creation of higher-quality new documents and extracting desired information for downstream use in other software applications, such as in derived reports and compliance checks, within back-office databases. Such learning can be performed using unsupervised and self-managed learning techniques, which do not require large amounts of pre-labeled or pre-analyzed data, but instead infer patterns from unlabeled or minimally labeled data.

[0029] Some implementations may allow computers to assist the writing process by discovering and using patterns within and across business documents to help writers avoid many of these errors, thus reducing the time required to achieve a given level of quality.

[0030] Today, typical document systems do not identify chunks, or in particular, their data type or semantic role. This adds time and expense for authors and editors, and for importing data from documents into backend databases, dashboards, or other downstream business processes. For example, it is common to manually search for data (chunk by chunk) from contracts and copy it into spreadsheets or data entry forms.

[0031] Some implementations can help label such hierarchical semantic chunks during the authoring process and express them explicitly, thus saving time and costs for people and / or computers to extract them and connect them to various kinds of other business processes.

[0032] Current technologies typically fail to fully leverage the similarities between multiple documents created by the same writer or group and / or of the same type (indicated here by membership in a particular set of documents) to more reliably identify chunks of new documents or draw attention to differences that seem significant. Explicit rules, such as requiring a section with the heading "Severability," only cover similarities that analysts would readily notice and explain, making them static, often constrained (e.g., cases of omissions involving paraphrasing or reorganization, or failure to address counter-conditions), and quickly becoming outdated. Smaller companies often lack the resources needed to develop more responsive technologies and often have too little documentation to justify the costs. On the other hand, even smaller companies often have a less diverse range of documents, making them more suitable for automated analysis such as that described herein.

[0033] Several implementations may use extracted information about chunks, along with their content, context, layout, and patterns of use across documents, to help writers create new documents. Examples include, at a minimum, specific content to be changed, reformatted, or moved; phrases that are commonly present in similar documents but are missing in the new document (referred to as “missing” or “possibly omitted” chunks or content); phrases that are present but not commonly present in similar documents (referred to as “unusual” chunks or content); and suggestions for changes such as swapping names or roles of different parties in specific locations.

[0034] Some implementations may accept and retain user feedback, such as when a user indicates that chunks are labeled with the wrong range, data type, or semantic role, that they are of no interest to the user, or that the labeling has failed altogether. Some implementations may use specific user corrections to improve machine learning and neural models, and remember not to repeat previous suggestions in cases where the user has rejected them (even if additional learning fails to prevent instances of the particular error). In particular, some implementations may avoid requiring a large number of review steps or corrections, favoring small-shot learning techniques and careful selection of the feedback requested, in order to minimize the amount of user action required. Some current techniques learn very specific things, for example, when a user instructs a spell checker to add a word to a dictionary. However, this involves merely a round list of iterations rather than iterative training or fine-tuning of the model used to determine the refined behavior, and therefore does not fully utilize the capabilities described herein.

[0035] Some implementations may learn and improve those behaviors using small amounts of user corrections when reapplying improved but still imperfect models, while avoiding making the user uncomfortable by repeatedly suggesting them.

[0036] Many businesses record specific information derived from documents in various types of databases that support their processes. For example, a company that owns many rental properties typically uses backend systems to help manage specific information derived from those rental agreements, such as tenant payments, approved pets, prior damages for which the tenant is not responsible, or other information. Automobile or tool rental companies, mortgage companies, healthcare providers, local governments, and other organizations use other types of information. Many goods and services have numerous mix-and-match options, and supervisors review statistics on their acceptance, combinations, pricing, and other factors. Business information systems generally provide analytics, ensure consistency or compliance, derive reports, and / or support other business processes, all of which can be facilitated by the use of chunked information as described herein.

[0037] Generally, chunks and the information they provide are scattered throughout prose text, manually extracted, and manually entered into spreadsheets, databases, or other systems. Previously, manual work was necessary because the flexibility of natural human language, as well as its variable layout and expression rules, allowed important chunks to be expressed in countless different ways, as is often the case with contracts, emails, etc. The negotiations underlying such documents are also often scattered across multiple types of documents, including emails, notes from conversations, and slide presentations. While this information can also be useful, it is usually handled manually. Some systems can treat such information sources as documents and achieve the same advantages already described.

[0038] Some implementations, once transformed and hierarchically semantically labeled as described herein, can provide a means for a computer to initiate the execution of a particular document. By combining the hierarchically labeled structure of a document with tools that provide a vector-semantic representation of the text, specific chunks can be identified as requiring specific actions. For example, a contract may specify remittances, notices, or other actions, and the conditions that enable or trigger them. These can be identified and used to initiate the execution of the contract.

[0039] Several implementations can review and summarize information from a set of documents within an interface such as a "dashboard," move the identified information to the customer's backend database or similar system, enabling a more efficient and lower-cost flow of business data and providing a simple way to enhance quality assurance, consistency, and reporting. When chunks are semantically labeled, it becomes easier to generate summary reports across sets of documents containing counterpart chunks. Several implementations can provide a very simple way for users to create such reports by simply clicking on one or more examples of chunks that should be included, which are then positioned and extracted by role or context across all documents in the set. Some implementations can also assist users in discovering documents that lack expected counterpart chunks and correcting them to include or identify such chunks, or in ensuring they correctly do not include them.

[0040] In another context, the performance of a given group, such as a company or department, can be enhanced by incorporating information such as the semantic roles of chunks, patterns of occurrence, and other characteristics of those documents and their user feedback into the system's learning process, and then using the resulting improved model to refine and / or verify future documents. However, many customers do not wish to share such information with other customers, and many have binding confidentiality requirements. On the other hand, general information and learning derived from public, non-confidential sources can be freely used and shared.

[0041] Several implementations can provide feedback and learning benefits while still sharing general learning based on non-confidential, publicly available data, while maintaining separate and private data and any model information derived from it for each customer. Maintaining these data processes separately ensures, statistically speaking, that there is no possibility of information "leaking" from one customer to another.

[0042] Introduction of Exemplary Embodiments The following is a description of an exemplary system. See Figure 1. This system relates, in general, to methods and apparatus for AI self-managed creation of hierarchically semantically labeled documents, and / or for the assisted authoring and processing of such documents. This includes processes such as composition, structuring, annotation, modification, review, extracting data from documents, and / or using such data in downstream business processes. More specifically, it focuses on documents similar to previous documents and the use of such highly enhanced documents in business processes by discovering the detailed hierarchical structure of documents, consisting of many semantically meaningful chunks associated with their roles, across sets of documents, including relatively small sets, primarily using unsupervised and self-managed machine learning techniques.

[0043] The operation of this exemplary system uses the following processes, which are described in more detail in the following sections. This is merely an example. Other embodiments may use different combinations of steps, including omitting steps, adding other steps, and changing the order of some steps. These may also use different implementations of the steps listed below, including different combinations of techniques described under each step. In Figure 1, there is an "S" before each step, and step 1 below is labeled "S01". 1) Import: Bring the user's document group to datastore 110. 2) Organization: Divide documents into document sets by type, such as rental vs. sales agreements, or medical records vs. current clinical notes. 3) Visual Extraction: A linear text stream is extracted from each document, based at least on its content and visual layout, which includes limited information about individual texts and other areas, their start and end positions, formatting, and content. The extracted data can be organized as “visual lines” or “visual blocks” (also called “hyperlines” or “visual” chunks), such as paragraphs distinguished by their geometric layout. 4) Structure: Identify the broad classes of headings, list items, and other structural chunks within the document. 5) Re-nesting: Determine the nesting relationships between sections and lists, and the scope of their respective texts. 6) Topic Chunking: Analyze the topic content of each document and create chunks (topic-level chunks) that enclose areas of similar topics. 7) Topic labeling: i) Use embedding and clustering to generate candidate data types and semantic role labels for each headword in the corpus. ii) Use key phrase extraction techniques to generate candidate data types and semantic role labels for the chunks. 8) Chunk labeling: Using multiple methods, e.g., neural networks, word and character embeddings, grammatical analysis and pattern matching, regular expressions, similarity metrics, and / or other methods, to identify and assign data types and semantic role candidates to other chunks throughout the document. Of particular interest for certain embodiments are the following: i) Structural grammar analysis and pattern matching of the results ii) Using question-answering techniques to link small chunks of information to the specific semantic roles they play in the document. iii) Combine XPath tree matching with word embedding techniques to match structural and grammatical tree patterns, even though phrasing and word choices may differ significantly. 9) Named Entity Recognition (NER): Identifies data types that are found as named entities throughout the document and assigns them to chunks. 10) Role labeling, extract labeling: Assign semantic role labels to chunks, such as indicating that the name constitutes a “seller” party in the contract, or that the drug is referred to as an allergy treatment. 11) Anomaly: Identify semantic roles that are normally present in the documents of the document set under review, or that are not present in the current document but are not present in the current document (or vice versa). 12) Arbitration: Adjust and / or select alternative scopes, data types, and semantic role labels for chunks to generate a well-structured structure that is easily representable in formats such as XML. 13) DGML: Create an enhanced version of the document that includes explicit identification information for chunk locations, data types, and semantic role labels, as well as additional information such as the confidence level of each identified chunk and the data types expected in similar chunks (dates, date ranges, personal names, etc.). The enhanced version is created using an XML-based markup language called DGML. 14) Feedback: Present the user with an enhanced version, allow them to select chunks (and potentially the location of omitted chunks), and collect the user's selection to confirm, reject, or make other changes. The user is also free to choose their own reading and review order. The feedback can also be applied to any other interpretations the system has made, such as organizing the document set, as described in step (2). i) In the case of presumably omitted chunks, prioritized examples from other documents are provided, which can be automatically customized by inspecting and / or copying the current document as needed and applying the target document values ​​to smaller nested chunks. 15) Feedback response: Track user responses to these interactions and use that information to fine-tune Model 120, as well as prevent the same or similar errors from being repeated later. 16) Downstream communications, transmission: Select chunks by type and / or role, use them to generate reports via document sets, and / or export them to downstream systems that add functionality such as backend contract databases, regulatory compliance checkers, and administrative report generators.

[0044] Figure 2 is a screenshot showing a dashboard that tracks the processing of different document sets 1 through 7 throughout the process described above. In this dashboard, the process is divided into the following stages: Uploading • Preprocessing Review Large Chunks Review Small Chunks Ready to Use The color coding indicates the degree of completion. Green stages are complete, red stages are in progress, and black stages have not yet started.

[0045] Each of the steps listed above is explained in more detail below.

[0046] Further explanation of the exemplary implementation The numbering here reflects the general order of the analysis in this particular example. However, not every step depends on every previous step, and as a result, many elements can be rearranged or parallelized in other implementations. Elements can also be shifted or repeated to exchange additional information with other elements, or elements can be executed independently in separate processes or on different machines.

[0047] 1) Import The system accepts typical word processor documents (such as MS Word) and page layout documents (such as PDF or PNG files). In each case, visually adjacent areas such as headings, paragraphs, table cells, tables, and images are identified and represented as chunks using a combination of their relative positions, surrounding whitespace, fonts, and layout features. These features are partially selected by designers and partially learned through image and pattern analysis of numerous documents. OCR is also applied to input documents that no longer contain machine-readable text content.

[0048] These chunks, along with the selected layout information, are presented to subsequent modules within the system.

[0049] 2) Formation Users do not need to organize the documents they check into the system. The system uses a clustering method that works with text content, layout information, and already detected structural information (such as the identification of several headings) to group documents into "sets" of specific document types, e.g., rental agreements vs. leases vs. sales. Specific document sets found can be reviewed by the user and named either automatically or by the user. Once established, these document sets facilitate subsequent machine learning and inference about format, content, semantic roles, and their differences. For example, the system might discover that almost every document in a given set has a specific section with three specific subchunks of a particular role and a data type of personal name, one of which is reproduced in five different sections. Using such patterns, it helps identify similar (and dissimilar) parts of other documents, suggests reviews or changes to the user, and provides examples of text to reuse in other documents within the same (or possibly different) set.

[0050] Clustering documents into document sets can utilize features from the document structure (various sizes, data types, and order and inclusion relationships between role chunks) and layout, as well as from the text content. Once some chunks and / or roles are identified in at least some documents, that information can be used to improve the clustering, either by completely re-clustering or by making smaller adjustments. Similar documents can become nearly or even completely identical if certain content of chunks with the same role is ignored, such as seller and buyer names and addresses, or if different chunks have the same occurrence pattern, for example, if one name (e.g., seller's name) appears in a certain location while another name (e.g., buyer's name) appears in a certain other location.

[0051] The system maintains both the original organization of uploaded files to a directory (if any) and their own organization within a set. Therefore, users can view both organization types, and learning algorithms can use both as information. For example, some users name documents according to various rules and / or organize them by customer, document type, or other characteristics, which is almost always useful for understanding similar patterns (such as having common chunk locations and roles) and relationships between documents.

[0052] 3) Visual extraction i) Area detection This system uses heuristics and machine learning to identify regions within a document based on geometric patterns. For example, in many documents, meaningful chunks have specific layouts such as signature blocks, abstracts, lists of definitions, and tables. Such patterns can be automatically learned by considering geometric and / or layout features, uniqueness or rarity, and / or correspondences within the same document or between documents, especially within the same set of documents.

[0053] The approach is chosen depending on the format of the input document. For example, word processor documents generally provide explicit information about paragraph boundaries, while PDFs or scanned pages require the system to assemble them from visual lines, or even analyze the dimensions of whitespace to assign characters (in multi-column documents, for example) to visual lines.

[0054] ii) Signature detection The system creates signatures (also known as "digests") of document parts and uses them to identify and classify additional "interesting" chunks and detect their boundaries. The signatures are based not only on text content but also on various aspects of the context, and may ignore the content of smaller included chunks (e.g., field chunks where the content within the counterpart changes).

[0055] The signature may even use a pixel representation of the chunks. The bitmap image of the text layout is divided into tiles, preferably on the order of 24 pixels square (adjusted for scan resolution), and the tiles are clustered. These neural network processing, including autoencoders and their adjacencies, reveal similar visual events such as boundaries between text and rules, edges and corners of text blocks, and even indentation changes and substantial font / style changes. Further neural networks then use this clustering to co-identify similar layout objects, which frequently indicate or characterize important chunks.

[0056] The approach here can use an unsupervised approach to generate document chunk embeddings based on pixels and characters within the document chunk, the size of the chunk, its position within the document, etc. (As noted, images can also be chunks). Clustering and comparison techniques can then be used on these embeddings for a variety of downstream tasks.

[0057] iii) Extraction This embodiment takes a laid-out document (e.g., a PDF or scanned printed page) and converts the recognized character images ("glyphs") within the document into text streams representing the correct document order of the glyphs (the streams may also include figures or image objects where appropriate, and there may be multiple streams such as footnotes or page headers that do not have a typical place in the reading order). Some documents have an incomplete explicit representation of the reading order. A well-known example is that there is usually no indication that a multi-column layout is valid at any given point, and therefore the first "line" extends only over half (or less) of it, not the whole. However, there are many additional examples where the order of text can be complex or ambiguous. For example, some layout programs draw each character individually, making word boundaries ambiguous. Table cells, sidebars, figures, footnotes, and other displays may not have a clear place in the order of text. Some text, such as in page headers and footers (as well as hyphens at the end of lines), may not require a place in the order of text at all. Many formats do not provide a clear indication that something belongs to such a special category.

[0058] The system addresses this task by combining the visual information of glyphs (position, style, etc.) with a deep neural network that understands the characteristics of the written language used within the document to construct the text stream. Furthermore, it detects many basic text boundaries, such as lines, blocks, columns, images, inline font changes, and header / footer objects.

[0059] iv) Expression After extracting text sequences and several hypothetical structural chunks, the system creates a representation of the document (known in some examples as "DGML") that includes information about them as well as visual features (font, color, size, etc.). The chunk representations containing information such as their location, type, and role are called "annotations." The combined data can then be used by natural language processing (NLP) and deep neural networks (DNNs). Deep neural networks incorporate this visual information and help structure the document into a hierarchy representing the document structure, including chunks such as headers / body, lists / list items, etc.

[0060] Sufficient information can be included so that a later version can construct an editable word processor document that closely resembles the original source. This can be included in DGML or a similar representation, along with other structural, content, and chunk information. In many cases, parts of a document with a clear format and layout are also useful chunks. However, formatting characteristics that do not match (or vice versa) the chunks that are otherwise required can still be represented through special types of chunks, through standoff annotations, or otherwise.

[0061] 4) Structure The structure pipeline transforms a flat text file into a hierarchical structure, where sections, subsections, and other parts of the document form an ordered hierarchy of content-based objects, a structure known to those skilled in the art. This transformation is performed using unsupervised machine learning techniques. This method has several stages.

[0062] i) Hyperlining This involves segmenting the text into “hyperlines,” which are larger groups than visual lines and comprise more meaningful, logical units (as opposed to visuals), such as paragraphs, headings, or similar elements. This is preferably achieved using a pre-trained neural network that considers features such as the “word shape” of tokens (especially leading and trailing tokens), layout information such as font and spacing characteristics, and similar features. Some hyperlines may also be provided in previous steps (depending on the format of the input document).

[0063] ii) Document language model It is preferable to use a document language model that includes information about the text content, formatting, and structure discovered so far, rather than a language model that is simply text-based. This allows for better detection of chunks and their hierarchies (headers / body, lists / list items, etc.) by learning to recognize meaningful chunks and patterns of their occurrence from formatted pages.

[0064] This creates a representation of the document that includes both text content and visual features (geometry, font, color, size, etc.). The deep neural network and NLP processes then utilize this information in the task of structuring the document within a hierarchy of chunks with data type and semantic role labels by detecting the extent and / or boundaries of chunks of varying sizes that represent the document structure. At this stage, the discovered chunks are primarily headings, sections, lists and items, tables, figures, and other relatively large units.

[0065] iii) Hyperline clustering This uses an autoencoder to cluster hyperlines across a document set based on word shape structure, assigning each hyperline to a cluster of hyperlines similar in layout, start and end content, and other characteristics, with each cluster identified by a "cluster ID" (which should not be confused with the creation or identification of the document set).

[0066] iv) Inline headings A particularly interesting special case is the "inline heading," where the heading of a chunk (which may provide a semantic role for the chunk) is not on a separate visual line, but on the same line as the beginning of the following text. Generally, inline headings are typographically distinguished by bolding, underlining, different fonts, trailing colons, or other effects. Separate heuristic and neural algorithms identify these chunks.

[0067] v) Small-shot structural learning Despite the advanced structuring methods described above, the generated structures can be expected to have certain imperfections or fail to meet the user's prior expectations. Short-shot structure learning addresses the need to create a machine learning model that relies on user-provided feedback, as described in steps (14) and (15). This model is then used to generate structures that combine user feedback on structures with those already generated by the system (and perhaps iteratively reinforced by previous feedback).

[0068] The main principle applicable to this case is derived from machine translation (MT) methods where one sequence is transformed into another. In this case, one sequence describing a hyperline is transformed into another sequence that also includes start / end markers that encode the hierarchy.

[0069] This process is carried out in various phases or steps. (a) First, the machine translation model is pre-trained using publicly available datasets. (b) The “Dispatcher” (see the “Feedback Response” section for a description) filters user feedback. (c) A new structure file is generated from user feedback, and a fine-tuned machine translation dataset is generated. (d) The pre-trained model is further trained using the small-shot learning principle.

[0070] 5) Re-nesting This embodiment preferably employs a “corpus re-nesting” algorithm that, given a flat list of cluster IDs from a hyperline clustering step, iteratively constructs nested structures using a pushdown automaton. By comparing the signatures of adjacent hyperlines, the system can determine whether a given heading or list item belongs to more, equal to, or fewer nested levels. This makes it possible to reconstruct multi-nested hierarchical structures of many documents (chapters, sections, subsections, clauses, lists, etc.).

[0071] Features considered in re-nesting include the "shape" of tokens within a hyperline (known in NLP techniques), especially considering the first and last; the specific class of punctuation ending the previous line (if any); formatting information such as capitalization, leading whitespace, indentation, bold, and underline; enumeration strings at the beginning of a line (e.g., patterns such as "IV(A)(1)" or "iv)"); or the presence and format of specific bullets or other emojis, the values ​​of their enumerators, the presence, level, and values ​​of preceding enumerators of the same kind.

[0072] 6) Topic Chunking This approach uses lexical statistics and other learning techniques across consecutive chunks of a document to detect where topics shift. This enhances the identification of boundaries of larger chunks, such as entire sections on a given topic, because sections (at any level) generally have greater uniformity of topic, vocabulary, and style within themselves than adjacent sections.

[0073] 7) Topic labeling i) Heading labeler As shown in Figure 1, for each header in the corpus, this step is • Create a numerical representation known as an "embed" for each heading. • Cluster the headings based at least on those embeddings, • Based at least on measures such as density, articulation, and similarity level, we will exclude "bad" clusters. • Propagate the most common semantic role label within each remaining cluster to all headings within that cluster.

[0074] ii) Key phrase labeler For each chunk, this step generates candidate semantic role labels for the text using an ensemble of key phrase extraction techniques (such as rule-based language techniques, ML, statistics, Bayesian, and / or others).

[0075] 8) Chunk labeling i) Grammar This aspect of the system begins with linguistic analysis of text, including natural language processing tasks such as speech tagging, dependency analysis, component analysis, and others. The system then applies tree matching mechanisms from other domains to locate grammatical and other structures within trees or tree-like structures discovered via NLP. These include document structuring methods such as tree grammar and tree pattern matching, as exemplified by XPath, GATE, and other tools.

[0076] Identifying grammatical phenomena in a sentence using such patterns allows the system to extract semantic role labels from the text itself, which are then used to annotate nearby chunks. For example, based on the constituent structure of a sentence, a search pattern can be constructed to match the sentence "The following are the terms of our agreement" (and other sentences with similar grammatical structures), and then noun phrases (in this example, "terms") can be extracted and used as semantic role labels for one or more chunks in content that follow this sentence and contain such "terms".

[0077] ii) Question and Answer Question answering techniques, including BERT for question answering, are specifically tuned to identify the semantic role labels (e.g., date, person's name, dollar amount) of candidate chunks. In contrast, most conventional question answering models aim to answer questions like "What is the effective date?" This system, instead, trains the model to answer questions like "What is July 8, 2018?" and aims to predict the "effective date" or "effective date of X," where X represents another chunk in the text (not just "date," which is a data type, not a semantic role).

[0078] This system also discovers comprehensive questions that, when answered, can point to relevant information in the text. This provides the ability to automatically suggest questions that should be used in question-and-answer sessions.

[0079] iii) Embedding and integrated XPath-like rules Here, the tools within the domain described under "Grammar" are integrated with tools that provide vector-semantic representations of text, such as word2vec, char2vec, and many related methods. This system enables analysts to represent and query patterns that include both structural information (which can include chunk data represented in XML or DOM-compatible format) that is well handled by XPath and similar tools, and fuzzy or "semantic" similarity information that is well handled by vector models.

[0080] 9) NER (Unlabeled small chunks) The technology can identify several chunks by data type, such as personal or company names or addresses (this is known as "named entity recognition" or "NER"). However, NER is quite lacking in identifying the semantic roles of these entities within a document. Current technology also fails to identify larger chunks, such as entire clauses or sections, or groups of chunks that contain larger, more meaningful or useful chunks.

[0081] This aspect of the system detects interesting small chunks without necessarily assigning them roles. Many methods and tools exist for identifying NERs within text. This system employs multiple methods, examples of which are listed below. These inventions are primarily unsupervised.

[0082] i) Established NER method ii) Expected words By training a language model of n-grams using a wide range of general texts such as Wikipedia, a model of "words expected in context for normal English" is constructed. When looking at a specific document, the system provides a means to identify n-grams that do not fit its general model and therefore tend to be specific to the document being processed.

[0083] iv) TF-IDF This is a TF-IDF-based approach ("term frequency vs. inverse document frequency") and is used in conjunction with label propagation and contextual semantic labeling.

[0084] v) Sequence clustering Small words or character sequences, such as n-grams, are extracted and clustered using contextual embeddings (e.g., those of BERT). The expected result is that n-grams sharing semantic meaning will begin clustering together. The cost of combinatorial explosion is addressed by filtering out some n-grams before clustering using heuristics (including on the syntactic tree). A wide variety of clustering algorithms can be applied. In this example, the hdbscan algorithm achieves effective clustering while assigning random noise to "none" clusters.

[0085] vi) Small shot NER The system uses short-shot learning techniques to generalize from a small number of labeled instances (e.g., selective user feedback) to more broadly applicable rules or tunings of learned parameters. This significantly reduces the number of times user feedback needs to be sought, leading to faster improvements in system performance.

[0086] 10) Extraction labeling This aspect of the system detects the semantic role labels of smaller chunks that appear directly in the sentences surrounding them. Meaningful chunks often have a role that is somehow designated by the context. For example: John Doe ("the seller") lives in... I have to pay $999 in rent by the end of each month.

[0087] i) Contextual Semantic Labeling (CSL) This process uses a neural network operating on a previously constructed structure, including sentence analysis, to learn which parts of the text could be semantic role labels for various chunks. Many chunks may already have such labels with varying sources and confidence levels, but this provides additional evidence about or against them, as well as new labels. Some of the patterns here involve grammar. For example, in "Do shall pay a rent of $1,000 by the last business day of each month," the main verb reveals what role the monetary amount plays, i.e., it is the rent to be paid. Other patterns are learned automatically in supervised and / or unsupervised ways using features of structure, chunking, labeling, and content available within the context. Formatting such as parentheses, table layouts, key phrases and words, as well as other features, also provide features for the neural network.

[0088] Useful information often resides in chunks containing sections, subsections, or their headings. For example, whether a given drug is related as a prescription or as an allergy may only be detectable by looking at the heading of the section it contains (this is another example of why detecting the correct hierarchical nesting of sections is important). There are many other cues that can be learned by machine learning techniques and applied to discover roles applicable to various chunks. Cross-document similarity can also be used to associate semantic roles found in similar contexts but not discoverable in isolated documents, particularly between documents within the same set of documents.

[0089] ii) Label propagation This process standardizes labels across similar chunks of text within a corpus of documents. This applies to both labels extracted from context and those available from the previous step. The algorithm clusters chunks based on their embeddings using cohesive clustering, ranks candidate labels for each cluster of chunks using a weighted PageRank algorithm (using label frequency / confidence as initial node weights), and determines how similar labels are to each other using concurrency and embedding similarity. We then assign labels to chunks based on their cluster-level scores and how similar the chunk we are labeling is to the original chunk of the label (in terms of content, embedding, structure, data type, semantic role, and / or context). We apply the cohesive clustering and PageRank algorithms to propagate labels across similar contexts, making labels more consistent across sets of documents.

[0090] 11) Abnormality This aspect of the system examines multiple documents within a set of documents, such as those generated in step (2), and identifies chunks that occur in the current document but generally do not have counterpart chunks in other documents in the same set, and vice versa. Counterpart chunks do not need to have identical content, structure, format, context, data type, and semantic role, but can be variations from one document to another. Nevertheless, they can be recognized as substantially similar in their manner to other identified chunks.

[0091] If a new document contains chunks that are not typically found in other documents of the same set, a user may query some or all of them to confirm what they were actually intended to be. In this exemplary system, such queries become more common when the chunks in question are common to the new document and what it was based on (if any), but are little to common to others.

[0092] If a new document lacks counterpart chunks that typically exist in other documents within the same set, or even in particularly relevant external sources (e.g., house-style manuals, compliance requirements, etc.), some or all examples of such chunks, along with content extracted from other documents, will be suggested to the user. These suggestions may be ranked for the user based on factors such as frequency of use, the most typical (central) of available alternatives, or the likelihood of co-occurrence with other chunks present in the new document. Suggested chunks may be automatically updated, for example, to replace names, dates, and other subchunks specific to the document from which the examples were extracted with values ​​extracted from the new document.

[0093] Furthermore, the selection of chunks to be proposed for addition or deletion can usefully rely on the practices of different authors, editors, or other staff. For example, if the current author's document frequently differs from another author's in a particular way, it may indicate that the difference is a considered choice and not an error. On the other hand, if all authors working under the same supervisor do so in some way, but the current author does not, it may indicate a greater need for review, at least when it is first noticed.

[0094] Anomaly modeling considers the structure and chunk data types and semantic roles, as well as context, content, and format. For example, it models patterns of which chunk data types and semantic roles occur within, adjacent to, or nearby other chunks. Violations of well-established patterns are classified as anomalies and may be presented for user feedback, as with any other anomalies.

[0095] 12) Arbitration Many of the previous steps create and / or manipulate chunks of the document (typically not necessarily contiguous) defined as ranges of characters, tokens, and / or nontext objects within a linear sequence generated in step (3).

[0096] A chunk being considered at any given time can be represented either by "inline" metadata such as markup, or by "standoff" representations that point to its location within the text using various types of pointers. In this example, standoff representations are used for most processing, while inline representations are used for some purposes, such as communication with external tools that often prefer them. These and other representations are functionally interchangeable, and the choice between them can be managed by relationships such as performance and convenience.

[0097] A chunk's representation includes information about the steps or implementation that created them, how reliable they are ("confidence level"), and their specific data type and / or semantic role labels. Redundant, uncertain, conflicting, or partially overlapping chunks can frequently occur, which we will refer to here as "suboptimal." For example, two or more different processes might cause semantic role labels to be pasted into the same span (or nearly the same span, e.g., one containing "Dr." before a name and one not). Chunks can sometimes be deeply nested, but they can also be arbitrarily overlapping (i.e., each overlapping chunk contains some content that is also in the other, and some that is not). Through the steps described above, the system can maintain representations that can represent a large number of annotations, including those that are overlapping or in the same place.

[0098] Such suboptimal chunks are generally undesirable, at least when the document is presented to the user. Furthermore, many state-of-the-art NLP tools prefer a non-repeating structure, as do many document tools and methods well known to those skilled in the art, such as XML, JSON, SQL, and other representation systems. A more restricted structure is generally preferred, often referred to as "hierarchical" or "well-structured," and avoids partially overlapping chunks.

[0099] This aspect of the system strictly hierarchizes the set of chunks and modifies them to avoid suboptimal chunks. This can be achieved in several ways. Firstly, chunks can be deleted completely (i.e., the chunks themselves and the document content they identified are not deleted). Secondly, the chunk scope can be changed (e.g., by including or excluding one or more characters or tokens from either end) to prevent duplication with other chunks. Thirdly, chunks can be determined to be redundant and merged. Fourthly, chunks can be detected to be inconsistent (e.g., one tool considers "Essex" to be a place and another to be a person) and a selection can be made.

[0100] This process includes means for quickly detecting cases of partial and / or complete duplication, comparing chunks by type, role, and reliability, and resolving suboptimal cases by correcting chunks and their associated data. The selection of which chunks to correct, merge, or delete takes into account several factors, including confidence level, the data type, semantic role, and prior probability of the content of a given chunk, sub-words between semantic role labels, conditional occurrence probability in a given context, the number, role, and distribution of other chunks in the current and other similar documents, the priority of the process at that time, customer feedback on similar cases, and / or other methods.

[0101] The modifications may also change the confidence level of the chunks. For example, some aspects of the system may apply similar or identical semantic role labels to the same or nearly identical parts of a document. In that case, the labels are typically merged, and the resulting chunk is assigned a higher confidence level than the individual chunks it comprises. In other cases, a choice is made between conflicting chunk assignments, but the chosen chunk may end up with a lower confidence level, reflecting that there was some level of disproven evidence.

[0102] This process improves the quality and consistency of chunk identification and labeling, enables information to interoperate with a wide range of tools, and allows for easier and more reliable analysis of results. The behavior described above can be applied at any time, not just at the end. For example, if a previous step uses an external tool for some subtask, it may require reduction to well-formedness. Deleted or modified chunks can instead be "paused," meaning they no longer affect processing but can be reintroduced upon request, which allows for such use of non-duplicate-supporting tools without later having to recreate previous work from the beginning, increasing processing flexibility and speed.

[0103] One approach is to resolve all duplicates and / or suboptimal chunks before generating the document presented to the user, thereby allowing the result to be easily encoded within a hierarchical format such as XML, which is used by many modern word processors and other tools. However, it is also possible to maintain multiple potentially duplicate alternatives in certain places for potential later resolution, such as through user feedback or improved algorithm learning (even in XML).

[0104] 13)DGML(DocuGami Markup Language) An enhanced version of a document represents the document structure, format, content, and identified chunks, and can identify which chunks were identified at which step of the process and with what level of confidence. While some embodiments use XML as the syntax for this representation, a broader representation can contain substantially the same information in other XML schemas, JSON, various databases, custom text, or binary formats.

[0105] In this step, information about the documents and detected chunks is converted into XML format (or "serialized"), making it easier to pass to other processes, particularly front-end user interfaces used for feedback, editing, and review, and to a "dashboard" application that provides overviews, statistics, and compliance information to other users such as group managers and quality control staff.

[0106] DGML (Docugami Markup Language) is a specific XML schema for this use, which packages all the information being described into a single package. Most previous schemas can handle structure, content, and sometimes layout, but they do not annotate "chunks" abstractly as described here. Many previous schemas also do not provide a generalized mechanism that can automatically detect and represent chunks on the fly, especially with respect to confidence levels and proofance information.

[0107] Furthermore, it is possible to "tunnel" the same information by representing it in a format transparent to the file formats of certain word processors and other tools. For example, if a tool supports embedded comments or metadata, "invisible" text, negligible attributes, or other similar features, the information described herein can be hidden within them, allowing the resulting document to be used, and possibly modified, by that tool, while the tunneled information is returned to the system still available.

[0108] 14) Feedback Mode Frontend The extensive annotations and analyses attached to documents and their detected chunks, as already described, guide users through editing samples, templates, or previous documents, enabling them to create similar but new documents customized to their current needs. For example, the system typically identifies the parties and property covered by a contract, medical history, current findings, and medications or conditions mentioned in other specific sections of clinical notes, as well as relevant dates. By examining other documents in the same document set, the system learns what is common, what is not, and what is necessary, and can therefore provide more useful recommendations to the user on what to review and / or update. For example, while an effective date may be present in almost every contract within a document set, its value can vary. Similarly, while the parties change, the types of parties are far more consistent.

[0109] i) Uninductive feedback In user interaction, the system first requests feedback on chunks that were detected (or potentially not detected) in some documents. The first few documents presented for feedback become the "cluster centroids" of the document set. The last few become the "outliers" of the document set.

[0110] ii) Inductive feedback The system then shows the user selected portions of the document and guides them to provide feedback by asking about current or potential labels for those portions, their scope, etc. a. "Interesting labels" are determined by a PageRank-based algorithm and a grammatical and structural model. Of these labels, a set of less reliable instances is selected for review. b. If there are no longer any low-confidence labels in the current document, the same process may be repeated for additional documents. In some embodiments, the model is continuously updated based on user-provided feedback. Alternatively, the feedback can be accumulated and applied later, in batches, and / or offline. Adjustments to the model can then influence the selection of chunks and labels subsequently presented for feedback, which may trigger a reanalysis of some documents. c. This system seeks feedback about field and structural chunks using essentially the same mechanism. One approach is that all chunk detectors provide confidence estimates, which can be used along with other information to select feedback candidates.

[0111] Feedback may be requested in different paths, such as small versus large chunks, field versus structure chunks, or in other orders. See Figure 3 for an example of a user interface for user feedback. It displays some or all chunks, allowing the user to select and inspect specific ones, and to see their assigned type and / or role, as well as optionally alternatives. The user can move chunk boundaries, select or edit labels, etc. Preferably, the user can also request that specific changes (to labels, etc.) be applied to all corresponding or identical type chunks. 15) Feedback Response i) Fleet queries are a way for a system to query both private and public data, typically based on user feedback from multiple users. The selected example is semantically and syntactically similar to the previous failure case, which increases the value of the feedback. ii) Dispatcher. A dispatcher is a methodology for connecting user feedback on the combined output of several ML models and non-ML algorithms, which can be learned from the feedback to a specific learning model 120.

[0112] This system allows the model to be improved from user feedback on its output, and from user feedback on the outputs of other trained and untrained models. This is achieved by using the feedback as incremental (also called "fine-tuning") training data for several described numerical and neural models. After improving the model using the feedback, not only a specific document, but all documents in the set, or even all of the user's documents, are re-evaluated. Thus, feedback on each document can improve chunk identification, role assignment, structure discovery, and therefore user assistance for all documents. This retraining is represented by the dotted connectors from step (15) to step (3) in Figure 1.

[0113] Documents and all associated information contribute to the learning and analysis of a set of documents (particularly within a specific set, though not exclusively) and thus improve the performance of future documents. For example, when a new chunk is added to one or more documents in a set, it can be used in future documents (or used to revise older ones) and can be suggested for future documents. At some point, the absence of a recently introduced chunk role, or the presence of a chunk role that has not been used much recently, can be anomalous. This point can be selected spontaneously by the user, in response to feedback questions, or automatically based on the usage curve of counterpart chunks over time. For example, if documents in a set created before a certain time have few documents containing a given role and / or contextual chunk (e.g., an "exclusion" section), but most or all of those created later have it, the absence of the counterpart chunk is likely to be anomalous in newer documents and can thus be presented usefully to the user.

[0114] 16) Downstream Communication After annotating a document containing the chunked information described, the selected information is converted into a specific format required by external business information systems, such as databases and analytical tools, and passed to those systems directly or through automated and / or manual review steps. For example, the names and addresses of specific parties can be copied to the correct fields in a database, which cannot be done automatically if they are only identified as "name" and "address" themselves. See Figure 4 for an example of integration with a downstream software application. In this example, chunks representing terms that the parties are expected to agree on are extracted and passed to a downstream application similar to Docusign to be filled out and signed.

[0115] Figure 5 is a block diagram of one embodiment of a computer system 510 that may be used in conjunction with the present invention. The steps described above may be implemented by software running on such a computer system. The computer system 510 typically includes at least one computer or processor 514 that communicates with peripheral devices via a bus subsystem 512. Typically, the computer may include a microprocessor, a graphics processing unit, or a digital signal processor, and their electronic processing equivalents such as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA), or the processor may be one of these. These peripheral devices may include a storage subsystem 524 with a memory subsystem 526 and a file storage subsystem 528, a user interface input device 522, a user interface output device 520, and a network interface subsystem 516. The input and output devices enable user interaction with the computer system 510.

[0116] A computer system can be a server computer, client computer, workstation, mainframe, personal computer (PC), tablet PC, rack-mounted “blade,” or any data processing machine capable of executing instructions (sequential or otherwise) that specify the actions to be taken by that machine.

[0117] Computer systems typically include operating systems such as Microsoft's Windows®, Sun Microsystems' Solaris®, Apple Computer's MacO®, Linux®, or Unix®. Computer systems also typically include a Basic Input / Output System (BIOS) and processor firmware. The operating system, BIOS, and firmware are used by the processor to control subsystems and interfaces connected to it. Typical processors compatible with these operating systems include Intel's Pentium® and Itanium®, Advanced Micro Devices' Opteron® and Athlon®, and ARM Holdings' ARM® processors.

[0118] The claimed inventions, embodiments, and / or examples are not limited to conventional computer applications or programmable devices that perform them. For example, the claimed inventions, embodiments, and / or examples may include optical computers, quantum computers, analog computers, etc. The computer system may be a multiprocessor or multicore system and may be used or implemented in a distributed or remote system. Here, the term “processor” is used in the broadest sense to include a single processor and a multicore or multiprocessor array, including graphics processing units, digital signal processors, digital processors, and combinations thereof. Furthermore, only a single computer system or a single machine may be exemplified, but the use of the singular form of such a term also means any set of computer systems or machines that individually or collectively perform instructions that perform any one or more of the operations discussed herein. Due to the constantly changing nature of computers and networks, the description of the computer system 510 shown in Figure 5 is intended only as an example for the purpose of illustrating a preferred embodiment. Many other configurations of the computer system 510 may have more or fewer components than the computer system shown in Figure 5.

[0119] The network interface subsystem 516 provides an interface to an external network, including an interface to the communication network 518, and is coupled to a corresponding interface device in another computer system or machine via the communication network 518. The communication network 518 may include many interconnected computer systems, machines, and communication links. These communication links may be wired links, optical links, wireless links, or any other devices for the transmission of information. The communication network 518 can be any suitable computer network, e.g., a wide area network such as the Internet, and / or a local area network such as Ethernet. The communication network may be wired and / or wireless, and may use encryption and decryption methods such as those available in a virtual private network. The communication network uses one or more communication interfaces that can receive data from and transmit data to other systems. Embodiments of the communication interface typically include Ethernet cards, modems (e.g., telephone, satellite, cable, or ISDN), (asynchronous) digital subscriber line (DSL) units, FireWire interfaces, USB interfaces, and the like. One or more communication protocols such as HTTP, TCP / IP, RTP / RTSP, IPX, and / or UDP can be used.

[0120] The user interface input device 522 may include pointing devices such as alphanumeric keyboards, keypads, mice, trackballs, touchpads, styluses, or graphic tablets; audio input devices such as scanners, touchscreens integrated into displays, speech recognition systems, or microphones; eye-tracking recognition, electroencephalogram pattern recognition, and other types of input devices. Such devices may be connected to the computer system by wire or wirelessly. Generally, the use of the term “input device” is intended to include all possible types of devices and means for inputting information into the computer system 510 or onto the communication network 518. The user interface input device typically allows the user to select several types of user interface output devices, e.g., a display subsystem, objects, icons, text, etc., to be displayed on it.

[0121] The user interface output device 520 may include non-visual displays such as a display subsystem, a printer, or an audio output device. The display subsystem may include several other devices for creating visible images, such as a flat panel device like a liquid crystal display (LCD), a projection device, or a virtual reality system. The display subsystem may also provide non-visual displays such as via an audio output or haptic output (e.g., vibration) device. In general, the use of the term “output device” is intended to include all possible types of devices and means for outputting information from the computer system 510 to the user or to another machine or computer system.

[0122] The memory subsystem 526 typically includes several memories, including a main random access memory (RAM) 530 (or other volatile memory) for storing instructions and data during program execution, and a read-only memory (ROM) 532 for storing fixed instructions. The file storage subsystem 528 provides persistent storage for program and data files and may include a hard disk drive, a floppy disk drive, a CD-ROM drive, an optical drive, flash memory, or a removable media cartridge, along with associated removable media. Databases and modules implementing the functionality of a particular embodiment may be stored in the file storage subsystem 528.

[0123] The bus subsystem 512 provides devices for enabling various components and subsystems of the computer system 510 to communicate with each other as intended. While the bus subsystem 512 is schematically shown as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. For example, RAM-based main memory can communicate directly with a file storage system using a direct memory access (DMA) system.

[0124] While the detailed description includes many details, these should be interpreted not as limiting the scope of the invention, but merely as illustrating different examples. It should be understood that the scope of this disclosure includes other embodiments not described in detail above. Various other modifications, changes, and variations, which will be apparent to those skilled in the art, can be made in the arrangement, operation, and details of the methods and apparatus disclosed herein without departing from the spirit and scope as defined in the appended claims. Therefore, the scope of the invention should be determined by the appended claims and their legal equivalents.

Claims

1. A computer implementation method performed by a computer system that executes instructions to assist a user in developing a target document belonging to a document set, Accessing a document set containing multiple documents, wherein the document set identifies chunks within the individual documents of the document set, and includes data types and semantic role labels for some of the chunks, the semantic role labels describing the semantic roles performed by the chunks within each document, (a) to derive patterns of occurrence of semantic roles performed by chunks across documents in the set of documents, and (b) to derive patterns of occurrence of counterpart chunks in different documents across the set of documents, wherein counterpart chunks perform the same semantic role in different documents. To provide a user interface for users to develop target documents belonging to the aforementioned document set, Based on the derived occurrence patterns across the document set, the system automatically generates suggestions for developing the target document and displays these suggestions within the user interface. Computer implementation methods including

2. The computer implementation method according to claim 1, wherein deriving the pattern of occurrence includes using machine learning and / or artificial intelligence to derive the pattern of occurrence.

3. The computer implementation method according to claim 1, wherein all of the multiple documents in the document set are of the same document type.

4. The process further includes deriving patterns of chunk occurrences within individual documents in the aforementioned document set, and automatically generating proposals based on such derived patterns. The computer implementation method according to claim 1.

5. The chunk in the aforementioned document set is A field chunk containing content within a document suitable for use as a field in a document template, wherein some of the field chunks are hierarchical and contain other chunks as subchunks, A structure chunk containing content from the structure within the layout of the document, wherein the semantic roll label includes semantic roll labels for some of the structure chunks, Chunks containing images or videos, The computer implementation method according to claim 1, including the method described in claim 1.

6. The computer implementation method according to claim 1, further comprising comparing chunks in the target document with the derived occurrence patterns of semantic roles and / or counterpart chunks across the set of documents, wherein several suggestions are automatically generated based on the comparison.

7. Identifying anomalies in the occurrence of semantic roles in the target document by comparing them with the derived occurrence patterns of semantic roles and / or counterpart chunks across the set of documents, wherein at least one proposal is automatically generated based on the identified anomaly. The computer implementation method according to claim 1, further comprising:

8. The identified anomalies are missing within the target document but occur commonly within the document set, and include identifying the semantic roles. The computer implementation method according to claim 7, wherein the automatically generated suggestions include adding content for the missing semantic roles.

9. The identified anomalies include identifying extra semantic roles that occur within the target document but do not typically occur within the document set. The computer implementation method according to claim 7, wherein the automatically generated suggestions include removing or modifying the corresponding chunks for the extra semantic roles.

10. The identified anomaly includes identifying a semantic role that occurs within the target document and is also common within the document set, but the content of the corresponding chunk within the target document does not match the content of the corresponding chunk within the document set. The computer implementation method according to claim 7, wherein the automatically generated suggestions include removing or modifying the mismatched content in the target document.

11. The identified chunk includes a field chunk containing content within the document suitable for use as a field in the document template. The derived pattern includes, for one of the field chunks, a pattern in which the counterpart chunk contains substantially all of the same content. The computer implementation method according to claim 1, wherein at least one automatically generated suggestion includes inputting the same content into the target document for the field chunk.

12. The computer implementation method according to claim 1, wherein displaying the proposal to the user includes displaying several proposals for which the user has the option to accept or reject the proposal.

13. (a) In response to the user accepting individual suggestions for the target document, the same suggestion is repeated for a second target document that exhibits the same pattern in the target document that led to the generation of the accepted suggestion in the target document, (b) In response to the user rejecting individual suggestions for the target document, not to repeat the same suggestion for a third target document that exhibits the same pattern in the target document that led to the generation of the rejected suggestion in the target document, The computer implementation method according to claim 1, further comprising at least one of the following.

14. The computer implementation method according to claim 1, further comprising automatically applying several suggestions, and displaying the suggestions to the user comprising displaying an option for the user to confirm the automatically applied suggestions.

15. The computer implementation method according to claim 1, wherein displaying the proposals includes displaying the proposals in an order ranked according to their reliability within the user interface.

16. A computer implementation method according to claim 1, further comprising automatically generating additional suggestions for the target document based on patterns within the target document itself and / or on patterns in documents outside the document set, wherein the suggestions based on patterns within the target document, based on patterns within the document set, and based on patterns in documents outside the document set are displayed in the user interface with different priorities.

17. The computer implementation method according to claim 1, wherein the automatically generated suggestions further depend on the user and / or the user's affiliation.

18. The computer implementation method according to claim 1, wherein at least one of the following is true: (a) the target document is an existing document being edited by the user, and the automatically generated suggestion includes a suggestion for editing the existing document; and (b) the target document is a new document being created by the user, and the automatically generated suggestion includes a suggestion for creating the new document.

19. A non-temporary computer-readable storage medium that stores executable computer program instructions for assisting a user in developing target documents belonging to a document set, wherein the computer program instructions are executable by a computer system, and the computer system Accessing a document set containing multiple documents, wherein the document set identifies chunks within the individual documents of the document set, and includes data types and semantic role labels for some of the chunks, the semantic role labels describing the semantic roles performed by the chunks within each document, (a) to derive patterns of occurrence of semantic roles performed by chunks across documents in the set of documents, and (b) to derive patterns of occurrence of counterpart chunks in different documents across the set of documents, wherein counterpart chunks perform the same semantic role in different documents. To provide a user interface for users to develop target documents belonging to the aforementioned document set, Based on the derived occurrence patterns across the document set, the system automatically generates suggestions for developing the target document and displays these suggestions within the user interface. A non-temporary computer-readable storage medium that enables the execution of methods including [specific actions].

20. A computer system for assisting a user in developing a target document belonging to a document set, wherein the computer system is A storage medium for receiving and storing a document set containing multiple documents, wherein the document set includes identifying chunks within the individual documents of the document set, and also includes data types and semantic role labels for some of the chunks, the semantic role labels describing the semantic roles performed by the chunks within each of those documents, A processor system having access to the storage medium and executing an application program for developing the target document, Equipped with, The processor system that executes the aforementioned application program is (a) to derive patterns of occurrence of semantic roles performed by chunks across documents in the set of documents, and (b) to derive patterns of occurrence of counterpart chunks in different documents across the set of documents, wherein counterpart chunks perform the same semantic role in different documents. To provide a user interface for users to develop target documents belonging to the aforementioned document set, Based on the derived occurrence patterns across the document set, the system automatically generates suggestions for developing the target document and displays these suggestions within the user interface. A computer system that performs [some action].

Images