New text feature vocabulary extraction method

Abstract

The invention provides a new text feature vocabulary extraction method. The method comprises the following steps: performing text word segmentation processing by using the word segmentation technology, performing unused word removal processing on vocabularies by using an unused word matching table, obtaining a series of vocabulary positions and vocabulary weight values according to investigation, survey and statistics, and extracting the first word c(w1), and the first m-1 words of which the RE (ci,c(w1)) values are bigger according to information quantity of the words in the text based on the abovementioned two factors, namely obtaining a feature vocabulary vector of the text. Compared with the traditional text feature vocabulary extraction method, the new text feature vocabulary extraction method provided by the invention has the advantages of high accuracy and higher application value, features of the text can be extracted when no text set is provided or no category is classified in advance and only one text condition is provided, the construction of different vocabularies to the idea of the text is calculated, and good theoretical basis is provided for subsequent text similarity and text clustering.

Description

technical field [0001] The invention relates to the technical field of semantic network, in particular to a new method for extracting text feature vocabulary. Background technique [0002] Text features refer to the collection of words that can best represent the subject of the text. Text features can not only summarize the main content and subject of the text well, but also reduce the complexity of text processing. The commonly used text feature extraction methods include word frequency-inverse document frequency method, information gain and other methods. The simple structure of the word frequency-inverse document frequency method cannot effectively reflect the importance of words or phrases and the distribution of feature values, so the accuracy of TF-IDF is not very high. The information gain method is only suitable for extracting text features of one category, but cannot be used for extracting text features of multiple categories. The above two text feature extraction...

AI Technical Summary

Problems solved by technology

[0003] For the commonly used text feature extraction methods require a large number of training sets as the premise of extraction, if only one text is given, the text feature will not be able to be extracted and the commonly used text feature extraction methods have low precision, the present invention provides a A New Method of Text Feature Vocabulary Extraction

Images