Corpus data processing method and device, computer readable medium and electronic equipment

Abstract

The embodiment of the invention provides a corpus data processing method and device, a computer readable medium and electronic equipment. The corpus data processing method comprises the steps of obtaining to-be-processed corpus data in a target field; Generating a first corpus template corresponding to the to-be-processed corpus data according to the entity name contained in the to-be-processed corpus data; According to the first corpus template and a second corpus template existing in the target field, calculating the similarity between the first corpus template and the second corpus template; And according to the similarity between the first corpus template and the second corpus template, filtering the to-be-processed corpus data to obtain processed corpus data. According to the technical scheme provided by the embodiment of the invention, the corpus data to be processed can be filtered by mining the corpus template corresponding to the corpus data to be processed and the existing corpus template in the target field, so that the corpus data with relatively poor correlation with the target field can be filtered out, and relatively accurate corpus data in the target field can be obtained.

Description

Technical field [0001] The present invention relates to the field of computer and communication technologies, and in particular to a method, device, computer readable medium and electronic equipment for processing corpus data. Background technique [0002] In the intelligent question answering scenario, the acquisition and expansion of domain corpus is an important issue for domain construction. Sufficient corpus of high quality and diversity can be trained to obtain more accurate deep learning models, which can more accurately classify user questions. Conversely, if there are too few relevant corpus in a field, the deep learning model will learn fewer features related to the field, and it will be difficult to distinguish it from corpus in other fields. It can be seen that the work of corpus mining has a decisive significance for the effect of deep learning models. However, the corpus mining solution proposed in the related technology has the problem of more recalled data and gr...

AI Technical Summary

Problems solved by technology

However, the corpus mining scheme proposed in the related art has the problem of recalling more data and noisy corpus data, which will not only lead to time-consuming and labor-intensive manual inspection, but also affect the accuracy of the deep learning model.

Images