New small world network model realization text feature extraction method

Abstract

The invention discloses a new small world network model realization text feature extraction method. According to a Chinese word segmentation preprocessing process, position weights and part-of-speech weights of vocabularies are determined; semantic correlation functions defined in the specification are determined by integrating "HowNet"-based two vocabulary correlation algorithms and a method for importance of the vocabularies to a text; herein, the functions are all subjected to normalization processing; value calculation conditions are more normative and stricter; two parameters including a density parameter and an edge weight parameter are set for a lexical semantic network model graph; proper thresholds are set for the parameters; and when the conditions are met, the vocabularies are text feature vocabularies. The method has higher accuracy, overcomes the deficiency that an information gain method is only suitable for extracting a type of text feature, has higher application values, and can accurately calculate contribution degrees of different vocabularies to a text thought; the data processing is more normative; the constructed lexical semantic network model graph better conforms to actual conditions; and a good theoretical basis is provided for subsequent text clustering.

Description

technical field [0001] The invention relates to the technical field of semantic network, in particular to a method for extracting text features realized by a new small-world network model. Background technique [0002] Currently commonly used text feature extraction methods include word frequency-inverse document frequency method—TF-IDF, information gain method, mutual information and other methods; the simple structure of TF-IDF cannot effectively reflect the importance of words or phrases and the characteristics of feature values. Distribution, so the accuracy of TF-IDF is not very high. The information gain method is only suitable for extracting text features of one category, but cannot be used for extracting text features of multiple categories. The mutual information method considers the ratio of the probability of category occurrence to the probability of collection, which will cause a defect, that is, the difference in the number of texts in the category collection w...

AI Technical Summary

Problems solved by technology

However, the commonly used text feature extraction methods do not consider the semantic status of the feature vocabulary and its contribution to the expression of the text, and ignore the semantic information and structural information of the document, resulting in the lack of semantic and structural information of keywords.

Images