Semi-supervised mass data hierarchy classification method

Abstract

Aiming at the problems of long manual labeling time and large expenses of a mass data hierarchy classifier, the invention provides a semi-supervised mass data hierarchy classification method comprising the following steps of: carrying out feature extraction on webpages in a webpage base; automatically generating the training set of a leaf node on the existing hierarchy classification body base by utilizing a rule set and an expansion rule; clustering the webpage of each existing leaf node; classifying unlabelled sets on the basis of clustering, and adding examples in the unlabelled sets, which are similar to the training set, to the training set of the corresponding leaf node to expand the scale of the training set; training the canonical linear classifier of each node by using a random gradient descent method; if the result of the classifier meets stopping conditions, stopping; otherwise, entering the step F; classifying the unlabelled sets by using the hierarchy classifier established by the steps C and D; adding the high-confidence classified webpages to the training set; and returning to the step C and repeating the steps form C to F.

Description

technical field [0001] The invention relates to the fields of data mining, machine learning and natural language processing, and is a semi-supervised hierarchical classification method for massive text data, that is, a semi-supervised hierarchical classification method for massive data. Background technique [0002] As we enter the era of information explosion, the Internet has provided people with a lot of knowledge and content, and the knowledge on the Internet has provided great help to people's basic necessities of life. Websites such as Google, Baidu, Sogou and Youdao provide Chinese search services, and people can find relevant information webpages by searching keywords. However, keyword-based retrieval often fails to provide the webpages people need, and users need to browse a large number of search results by themselves to finally find the webpages they need. Therefore, semantic-based search engines have recently aroused great interest and become a hot spot in the i...

AI Technical Summary

Problems solved by technology

[0005] 1. When the hierarchical structure is huge, a large amount of manual labeling is required to provide the training set for the classifier to achieve the required classification accuracy, and the time and cost required for this manual labeling are very large

[0006] 2. The training of high-precision text classifiers (regularized linear classifiers, including support vector machines) requires a lot of running time, and it is impossible to establish a classification model for massive text data

Images