Distributed mass short text KNN (K Nearest Neighbor) classification algorithm and distributed mass short text KNN classification system based on information entropy feature weight quantification

Abstract

The invention discloses a distributed mass short text KNN (K Nearest Neighbor) classification algorithm and a distributed mass short text KNN classification system based on information entropy feature weight quantification. The algorithm mainly includes text preprocessing, weights for measuring features with information entropies, a document vector representation method and a KNN classification algorithm adopting indexing and caching techniques, and the whole algorithm is implemented on the basis of a MapReduce computing framework. By utilizing the information entropies of features in a training set, the invention effectively measures the effects of words in texts on classification, and moreover, based on the MapReduce computing framework, the algorithm can utilize mass clusters to efficiently classify mass short texts. The mass short text classification algorithm provided by the invention has the advantages of high accuracy, high operating efficiency and strong scalability.

Description

technical field [0001] The invention belongs to the technical field of text classification, and relates to a mass short text distributed KNN classification algorithm and system based on information entropy feature weight quantization. Background technique [0002] With the rapid development of the Internet, the number of electronic documents and resources generated by users is increasing day by day. Text classification technology has become a key technology for processing and organizing massive document data. Especially with the large amount of short text data such as microblogs and various user comments appearing on the Internet, related research on short texts has gradually attracted people's attention. The classification algorithm generally includes: text preprocessing, document representation, and classifier selection and training. Specifically, text preprocessing mainly divides the text into word segmentation and removes stop words that are useless for semantic expres...

AI Technical Summary

Problems solved by technology

However, most of the traditional algorithms are used for data processing in a stand-alone environment, and have not been effectively extended to parallel and distributed computing platforms so that they can meet the needs of large data processing capabilities.

Images