Short text classification method based on CHI and classified association rule algorithm

Abstract

The invention provides a short text classification method based on CHI and a classified association rule algorithm. The frequencies of frequent word sets of different types of texts are measured, a category frequent factor (LFF) is introduced, the minimum support threshold values of text categories are reasonably allocated through the LFF, the phenomenon that frequent word set categories mined by adopting a traditional FP-Growth algorithm are deflective is avoided, meanwhile category tendency judgment is conducted on the frequent word sets, a CHI checking algorithm is adopted to measure the relevance degree of characteristic words and the categories instead of measurement based on simple word frequency statistics, the step that best parameters are determined through manual parameter setting and experiments is omitted, and the controllability of a classification system is enhanced. In addition, the invention further provides a parallel characteristic extension short text classification algorithm based on a Hadoop/MapReduce big data computing platform. MapReduce parallelization design is conducted on a category frequent factor calculating and characteristic extension method, the short text classification accuracy rate and classification efficiency are improved, and the controllability of the system is improved.

Description

technical field [0001] The invention relates to the fields of natural language processing and text mining, in particular to a short text classification method based on CHI and classification association rule algorithms. Background technique [0002] With the development of the Internet, especially social media, text content on the Internet is becoming more and more abundant. In addition to long texts such as blogs and news, due to the increasing participation of Internet users in network topics, short texts such as Weibo, emails, and comments It has also experienced explosive growth in recent years. Different from long texts, short texts are characterized by less text content, which has the disadvantages of sparse features and weak description information, which leads to the poor effect of traditional feature extraction, text representation models and text classification methods on short texts. To solve this problem, the most direct and effective way is to expand the featur...

AI Technical Summary

Problems solved by technology

[0005] 2. After excavating the feature items with co-occurrence relationship, the traditional method is to directly expand the feature of the original text without considering the category relationship of the associated feature, which will cause the introduction of noise feature words and affect the classification performance

In the existing research, the category propensity of the feature is calculated by manually setting the reliability threshold, and then the frequent word set is filtered according to the same direction relationship of the category, too much manual intervention, the threshold is difficult to determine, the flexibility of the algorithm and the controllability of the program not tall

[0006] 3. Considering the rapid expansion of network data volume in recent years, facing the high requirements of massive data on CPU, IO throughput, etc., the traditional serial text classification algorithm has the advantages of computing speed, file storage, fault tolerance, etc. in the environment of large text data volume It seems powerless, so it is necessary to study distributed algorithms that can run in multi-node big data computing mode

Images