A language processing method and device

Abstract

Provide a corpus processing method and device, including: obtaining a first corpus collection to be classified; determining a second corpus collection from the first corpus collection, and the evaluation objects of the second corpus in the second corpus collection are all The first evaluation object, and the evaluation contents of the second corpus about the first evaluation object are all marked as positive evaluations; a third corpus collection is determined from the first corpus collection, and the third corpus collection is The evaluation objects of the third corpus are all the first evaluation objects, and the evaluation contents of the third corpus about the first evaluation object are all marked as negative evaluations; determine the third corpus in the second corpus set Whether the second corpus is synonymous or synonymous with any piece of third corpus in the third corpus collection regarding the evaluation content of the first evaluation object; and processing the corpus collection. The corpus processing method provided by the present invention can improve the utilization rate, accuracy and coverage of classified corpus.

Description

technical field [0001] The present invention relates to a corpus processing method and device, more specifically, to a corpus processing method and device for sentiment classification. Background technique [0002] At present, when we are doing sentiment analysis of product reviews, we mainly use classification methods to build sentiment analysis models. Since most of the objects to be classified are user comments on e-commerce websites, these comments are generally their own shopping experience published by users, and they are all colloquial descriptions. There is no specific evaluation scope and evaluation rules, and may involve all aspects of the product. Even when describing the same aspect of a product, different users say it differently. This makes it difficult for us to construct classification training corpus. Because the training corpus only achieves a certain coverage, representativeness and accuracy, the trained classification model will have a better classific...

AI Technical Summary

Problems solved by technology

This will cause the following situations: First, there will be many evaluation methods that cannot be covered. Manually labeling the training corpus usually takes a lot of time and energy, and it is generally difficult to cover all possible situations

Therefore, each labeling result is very precious. If the corpus that does not reach the threshold is directly removed, not only will the labeling work be wasted, but also the coverage of the final classifier will be reduced, and the final classification effect cannot be guaranteed; , even if there are some evaluation methods in the training corpus, but the number of corpus corresponding to them is small

Due to the accidental and error-prone nature of manual annotation, the accuracy of these corpora is not reliable enough

If these corpus with low label reliability are put into the classifier, it may eventually affect the classification effect

Images