Graphical systems and methods for human-in-the-loop machine intelligence

Abstract

Methods and systems are disclosed for creating and linking a series of interfaces configured to display information and receive confirmation of classifications made by a natural language modeling engine to improve organization of a collection of documents into an hierarchical structure. In some embodiments, the interfaces may display to an annotator a plurality of labels of potential classifications for a document as identified by a natural language modeling engine, collect annotated responses from the annotator, aggregate the annotated responses across other annotators, analyze the accuracy of the natural language modeling engine based on the aggregated annotated responses, and predict accuracies of the natural language modeling engine's classifications of the documents.

Description

CROSS REFERENCES TO RELATED APPLICATIONS[0001]This application claims the benefits of U.S. Provisional Application 62 / 089,736, filed Dec. 9, 2014, and titled, “METHODS AND SYSTEMS FOR ANNOTATING NATURAL LANGUAGE PROCESSING,” U.S. Provisional Application 62 / 089,742, filed Dec. 9, 2014, and titled, “METHODS AND SYSTEMS FOR IMPROVING MACHINE PERFORMANCE IN NATURAL LANGUAGE PROCESSING,” U.S. Provisional Application 62 / 089,745, filed Dec. 9, 2014, and titled, “METHODS AND SYSTEMS FOR IMPROVING FUNCTIONALITY IN NATURAL LANGUAGE PROCESSING,” and U.S. Provisional Application 62 / 089,747, filed Dec. 9, 2014, and titled, “METHODS AND SYSTEMS FOR SUPPORTING NATURAL LANGUAGE PROCESSING,” the disclosures of which are incorporated herein by reference in their entireties and for all purposes.[0002]This application is also related to U.S. non provisional applications (Attorney Docket No. 1402805.00006_IDB006), titled “METHODS FOR GENERATING NATURAL LANGUAGE PROCESSING MODELS,” (Attorney Docket No. 1...

AI Technical Summary

Benefits of technology

[0020]In some embodiments, the annotation agreement interface includes a per document agreement list displaying information such as which documents received the highest agreement or which documents received the lowest agreement. A user o...

Problems solved by technology

It is difficult and time consuming for a human to integrate or recognize trends within these various communication formats with the volume of content involved in each, and any broad appreciation for what the creators of such communications may be expressing is therefore delayed.

Recognizing trends or underlying meanings across the vast number of human communication sources, and appropriately categoriz...

Images