Corpus processing method and apparatus

Abstract

The invention provides a corpus processing method and apparatus. The corpus processing method comprises the steps of obtaining to-be-classified first corpus sets; determining a second corpus set from the first corpus sets, wherein evaluation objects of second corpora in the second corpus set are all first evaluation objects, and evaluation content, on the first evaluation objects, of the second corpora is labeled as positive evaluation; determining a third corpus set from the first corpus sets, wherein evaluation objects of third corpora in the third corpus set are all the first evaluation objects, and evaluation content, on the first evaluation objects, of the third corpora is labeled as negative evaluation; judging whether the second corpora in the second corpus set and the evaluation content, on the first evaluation object, of any third corpus in the third corpus set are synonyms or near synonyms; and processing the corpus sets. According to the corpus processing method provided by the invention, the utilization rate and the coverage range of the classified corpora can be increased and the accuracy of the classified corpora can be improved.

Description

technical field [0001] The present invention relates to a corpus processing method and device, more specifically, to a corpus processing method and device for sentiment classification. Background technique [0002] At present, when we are doing sentiment analysis of product reviews, we mainly use classification methods to build sentiment analysis models. Since most of the objects to be classified are user comments on e-commerce websites, these comments are generally their own shopping experience published by users, and they are all colloquial descriptions. There is no specific evaluation scope and evaluation rules, and may involve all aspects of the product. Even when describing the same aspect of a product, different users say it differently. This makes it difficult for us to construct classification training corpus. Because the training corpus only achieves a certain coverage, representativeness and accuracy, the trained classification model will have a better classific...

AI Technical Summary

Problems solved by technology

This will cause the following situations: First, there will be many evaluation methods that cannot be covered. Manually labeling the training corpus usually takes a lot of time and energy, and it is generally difficult to cover all possible situations

Therefore, each labeling result is very precious. If the corpus that does not reach the threshold is directly removed, not only will the labeling work be wasted, but also the coverage of the final classifier will be reduced, and the final classification effect cannot be guaranteed; Second, even if there are certain evaluation methods in the training corpus, the number of corpus corresponding to them is relatively small

Due to the accidental and error-prone nature of manual annotation, the accuracy of these corpora is not reliable enough

If these corpus with low label reliability are put into the classifier, it may eventually affect the classification effect

Images