Graphical systems and methods for human-in-the-loop machine intelligence

Abstract

Methods and systems are disclosed for creating and linking a series of interfaces configured to display information and receive confirmation of classifications made by a natural language modeling engine to improve organization of a collection of documents into an hierarchical structure. In some embodiments, the interfaces may display to an annotator a plurality of labels of potential classifications for a document as identified by a natural language modeling engine, collect annotated responses from the annotator, aggregate the annotated responses across other annotators, analyze the accuracy of the natural language modeling engine based on the aggregated annotated responses, and predict accuracies of the natural language modeling engine's classifications of the documents.

Description

CROSS REFERENCES TO RELATED APPLICATIONS[0001]This application claims the benefits of U.S. Provisional Application 62 / 089,736, filed Dec. 9, 2014, and titled, “METHODS AND SYSTEMS FOR ANNOTATING NATURAL LANGUAGE PROCESSING,” U.S. Provisional Application 62 / 089,742, filed Dec. 9, 2014, and titled, “METHODS AND SYSTEMS FOR IMPROVING MACHINE PERFORMANCE IN NATURAL LANGUAGE PROCESSING,” U.S. Provisional Application 62 / 089,745, filed Dec. 9, 2014, and titled, “METHODS AND SYSTEMS FOR IMPROVING FUNCTIONALITY IN NATURAL LANGUAGE PROCESSING,” and U.S. Provisional Application 62 / 089,747, filed Dec. 9, 2014, and titled, “METHODS AND SYSTEMS FOR SUPPORTING NATURAL LANGUAGE PROCESSING,” the disclosures of which are incorporated herein by reference in their entireties and for all purposes.[0002]This application is also related to U.S. non provisional applications (Attorney Docket No. 1402805.00006_IDB006), titled “METHODS FOR GENERATING NATURAL LANGUAGE PROCESSING MODELS,” (Attorney Docket No. 1...

AI Technical Summary

Benefits of technology

The patent is for a system that helps annotate documents automatically. It presents the document in a way that makes it easy for annotators to verify the accuracy of the system's understanding of the text. The system also suggests labels or tasks for the annotators and calculates scores to show how well the labels or tasks agree with each other. This helps the system improve the accuracy of the labeling over time. The system also displays a list of documents that have received the highest or lowest agreement to make it easier to remove any inaccurate documents from the system. Overall, the system saves time and makes annotating documents more efficient.

Problems solved by technology

It is difficult and time consuming for a human to integrate or recognize trends within these various communication formats with the volume of content involved in each, and any broad appreciation for what the creators of such communications may be expressing is therefore delayed.

Recognizing trends or underlying meanings across the vast number of human communication sources, and appropriately categorizing unique nomenclature or slang that may be embedded, cannot be efficiently obtained without the aid of computer modeling tools.

Artificial intelligence tools can attempt to analyze and classify these information sources through metadata or other identifiers, but cannot efficiently analyze the true meaning of the internal content without verification oversight that is time consuming and costly to implement.

Images