A method and device for obtaining classification weights of domain document terms

Abstract

The present invention relates to a method and device for obtaining the classification weight of domain document terms, including: based on a plurality of terms including a plurality of specific terms set in advance, the level of terms corresponding to the terms, and the plurality of terms The number of word frequencies corresponding to specific terms in the item, the total number of domain terms, the total number of common terms, the total number of domain terms, the total number of common term frequencies, the total number of terms corresponding to each level of terms, and the total number of terms in each level of terms Sum of term frequencies for specific terms; based on total domain terms, total common terms, total domain terms, total common term frequency, total number of terms for each term level and in each term level The total number of word frequencies corresponding to a specific term in , to obtain the final ranking weight at the term level. Based on the difference in the role of different terms in document topic representation, the importance of document representation can be graded for terms, so that the differences in document representation by different terms are finally reflected in the weights of different graded terms.

Description

technical field [0001] The present invention relates to the field of language processing, in particular to a method and a device for obtaining classification weights of document terms in the field. Background technique [0002] At present, most Chinese text classification systems use words as feature items, called feature words. These feature words are used as the intermediate representation of the document, and are used to realize the similarity calculation between documents and documents, documents and user targets. Usually, the score value of each feature is calculated according to a feature evaluation function, and then these features are sorted according to the score value, and several highest score values ​​are selected as feature words. [0003] The most commonly used and effective text representation method is to establish a term-document matrix. Each element value in the term-document matrix represents the weight of the term on the corresponding row to the documen...

AI Technical Summary

Problems solved by technology

However, in the general term-document matrix, the number of occurrences of terms is purely used to represent the representation of the subject of the term, and the terms that are low-frequency in a specific document term and relatively high-frequency in other documents are used as the subject terms; but The high and low frequency features of word frequency cannot well represent the theme of the document. For example, some high-frequency common word "appearance" cannot characterize the document well, but the high-frequency word "mineralization" in geological documents can better In addition, low-frequency general word segmentation has little meaning for document representation, but low-frequency field terms such as "Qibaoshan Mine" have great significance for document representation

The TextRank algorithm uses the relationship between local words (co-occurrence window) to rank subsequent keywords, and does not consider the representational differences between domain terms and high-level terms

Images