Intelligent information extraction method and system

Abstract

The invention discloses an intelligent information extraction method and system. According to the method, after an NMF method is adopted to decompose a document feature matrix to obtain K clustering topics, clustering sentences in the document to obtain a set of a plurality of sentences; taking the first k topic sentence sets with the highest semantic similarity with the question text, using the question text again to respectively retrieve the k topic sentence sets, returning m most relevant sentences from each sentence set to form a corresponding document, and combining the obtained k documents into a long document; and performing answer extraction on the long document and the question text by adopting an MRC model combining a bidirectional attention flow model and a question text tensorinitialized Point Net model. According to the scheme provided by the invention, an improved MRC algorithm is adopted, and question text information is utilized in the stages of document theme aggregation and filtering, sentence retrieval recall and sorting and answer extraction; the OOV problem existing in the prior art is effectively solved, the data labeling cost is low, and the calculation efficiency and accuracy are both achieved.

Description

technical field [0001] The present invention relates to a sub-field of language processing---Information Extraction, in particular to an intelligent information extraction method and system combining traditional text retrieval and machine reading comprehension. Background technique [0002] Document-based retrieval technology has been extensively studied due to the continuous development of search services in the Internet industry. Algorithms similar to BM25 and page Rank based on empirical formulas do not rely on training models to obtain better accuracy and recall rates, but in large documents In the process of processing, there will be a problem of recalling more redundant information, so it cannot be directly used in the results of information or knowledge extraction. It often needs to be matched with a matching model such as first-order predicate logic based on rules and patterns, and subject-verb-object triplets. The design of information extraction, pattern and defini...

AI Technical Summary

Problems solved by technology

However, it is possible that the referent entity target in the text has been transferred, resulting in the recalled content not referring to the content of the question. This problem is a difficult problem for machine reading comprehension.

[0011] Requires detailed and large-scale manual labeling data: Similar to entity recognition and entity relationship classification, detailed manual labeling is required, and often faces a multi-classification task, although adding some probabilistic graphical models (such as conditional random fields) can improve multi-classification. The accuracy of the task, but multi-classification tasks will essentially face problems such as category imbalance and high sample complexity, and the cost of data scale is also an important reason for limiting the use of this method for information extraction.

Images