An Automatic Evaluation Method of Retrieval Submodule in Question Answering System
An evaluation method and a question-and-answer system technology, applied in special data processing applications, instruments, electrical digital data processing, etc., can solve the problems that the retrieval sub-module cannot be evaluated or is difficult to evaluate, and the amount of calculation is large, so as to improve development efficiency and achieve good results , avoid the effect of linking the search results
Active Publication Date: 2018-06-15
黑龙江省工研院资产经营管理有限公司
View PDF4 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0003] In order to solve the technical problem that the existing retrieval sub-module cannot be evaluated or the evaluation is difficult and complex, and the amount of calculation is large, the present invention proposes an automatic evaluation method for the retrieval sub-module in the question answering system, and evaluates the retrieval sub-module through the prediction index MAP. The technical scheme is as follows:
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment 1
[0042] An automatic evaluation method for retrieval sub-modules in a question answering system, such as figure 1 As shown, the specific steps of the automatic evaluation method are:
[0043] Step 1: output a sorted list of documents according to the document collection in the question answering system, and obtain the confidence corresponding to each document through four calculation granularities according to the confidence model, wherein the document collection includes question documents and reference answer documents;
[0044] Step 2: Obtain the precision rate of each document through four calculation granularities according to the precision rate model, and obtain the recall rate of each document through four calculation granularities according to the recall rate model;
[0045] Step 3: For the question to be answered entered in the question answering system, automatically search for and obtain a series of related documents related to the question to be answered in the docu...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention provides an automatic evaluation method for a retrieval self-module in a question-answer system and belongs to the technical field of module self-evaluation methods. According to the method, a MAP value of a retrieval result is accurately estimated through a machine learning model (a GBDT specifically) according to the degree of correlation (expressed with various features) between documents, questions and reference answers at each position of a retrieved document list, and therefore a final evaluation process is completed. The evaluation method has the advantages of being high in evaluation certainty, simple and the like and is suitable for self-evaluation of retrieval self-modules in various question-answer systems.
Description
technical field [0001] The invention relates to a retrieval sub-module evaluation method, which belongs to the technical field of module self-evaluation methods. Background technique [0002] There are generally a certain number of questions in the question answering system, and the reference answers corresponding to these questions. The task of the retrieval sub-module is to retrieve the most likely answer candidate documents from the existing document collection, so that the subsequent modules can accurately determine the answer. Due to the retrieval evaluation index MAP, when calculating, it is necessary to know whether each retrieved document is a relevant document of the question; while in the development of the question answering system, it is generally impossible to invest manpower to mark each question in a large number of documents. Related documents. Therefore, there is a great difficulty in evaluating the retrieval module. Contents of the invention [0003] I...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Patents(China)
IPC IPC(8): G06F17/30G06Q10/06
CPCG06F16/3329G06Q10/0639
Inventor 杨沐昀郑德权朱聪慧张越徐冰曹海龙赵铁军
Owner 黑龙江省工研院资产经营管理有限公司