IG TF-IDF text feature vector generation and text classification methods

Abstract

The invention particularly relates to an IG TF-IDF text feature vector generation and text classification method, and belongs to the field of text mining and machine learning. The method comprises thefollowing steps: 1) generating a text feature vector; 2) train that classifier; 3) evaluate that classification performance; 4) classify that target text set; The weight calculated by the invention can more truly reflect the importance of different terms to the text classification, so that the term with strong class discrimination ability is allocated with larger weight, the weight calculation ismore reasonable, and the accuracy of the text classification is improved. Moreover, the calculated term weights do not need to know the specific categories, thus overcoming the shortcomings of supervised methods such as TFADF in multi-category text classification.

Description

technical field [0001] The invention belongs to the field of text mining and machine learning, and in particular relates to an IG TF-IDF text feature vector generation and text classification method. Background technique [0002] With the advent of the Internet era, texts are presented in the form of electronic texts, resulting in a sharp increase in the number of electronic documents. Therefore, how to effectively organize and mine massive data texts has become more and more important. Automatic classification is one of the most widely used technical means. Classification is the division of text into predefined categories, and it is a research hotspot in the fields of information retrieval and data mining. In general, some text with category marks is used as training data, a classifier is obtained through a machine learning algorithm, and then its category is judged according to the text content. Before classifying the text, it needs to be expressed in a form that can be ...

AI Technical Summary

Problems solved by technology

However, the entry weights calculated by supervised weighting methods such as TFATF are related to the categories of specific texts, and the categories of the newsbooks to be classified are unknown. Either use the TFATF algorithm to calculate the weights for all categories one by one, or use TF- Traditional methods such as IDF calculate weights. The former needs to add additional statistical information during training and learning, while the latter increases the amount of calculation during classification or testing.

Images