A Noise Reduction Method for Remotely Supervised Retrieval Data

Abstract

A method for remotely supervise noise reduction of retrieved data includes calculating probability distributions of answers to a given question on each paragraph for a given question and a plurality of paragraph retrieved based on that given question, calculating the probability distributions of the answers to the given question on each paragraph, calculating the probability distributions of the answers to the given question on each paragraph; extracting at least one specific sub-string corresponding to the answer in the paragraph using a paragraph reader based on a given question and any paragraph, and calculating a probability of the specific sub-string being an answer to the paragraph as a first probability until all paragraphs are traversed; Selecting a corresponding paragraph to answer a given question based on the probability distribution and a first probability corresponding to a particular substring in all paragraphs; Where a particular substring is a word string consisting ofone or more words. The invention can more fully utilize all paragraphs in the retrieved related text which are helpful to answer the questions, thereby more effectively improving the question answering effect of the open domain question answering, improving the stability of the model, and having good practicability.

Description

technical field [0001] The present invention relates to the technical field of natural language processing and information extraction, and more specifically, to a noise reduction method for remote supervision and retrieval data. Background technique [0002] In recent years, reading comprehension of a given document and answering specific questions about the document has become a hotspot in natural language processing research. At present, many reading comprehension systems for natural language processing have been implemented, and these reading comprehension systems have achieved good results in many open source evaluations. The multi-level structure and selective attention mechanism of these system models make it possible to reason in documents for a given question. Thus, existing reading comprehension systems have demonstrated the capabilities of neural network models for reading, processing, and understanding natural language texts. [0003] Although existing reading c...

AI Technical Summary

Problems solved by technology

Therefore, the above method of dividing answer generation into two modules cannot achieve the desired results in open domain question answering applications.

Images