Chinese new word discovery method based on novelty

Abstract

The invention relates to a Chinese new word discovery method based on a novelty. The method sorts the novelties of all candidate new words from big to small, takes an intermediate word as a thresholdvalue, filters all words smaller than the threshold value to form a new word bank and output the new word. On the basis of an N-gram algorithm, a binary corpus model is established, word characteristics are combined on the basis of binary mutual information, an expression way from ternary to N-nary mutual information is given, and mutual information and an adjacent entropy threshold value are adopted for filtering. An old corpus is taken as a screening source by aiming at the characteristic that mutual information only can embody a word occurrence probability but can not embody whether the word is a new word or not, and the novelty is imported to optimize and filter a new word candidate set. On the basis of the shortages of complexity in calculation and big uncertainty of new word discovery evaluation indexes including accuracy, a recall rate and the like, a new t-MP evaluation index is put forward to replace original accuracy, recall rate and F-value to explain the accuracy new word discovery result. Therefore, the user can obtain the best new word from sorted candidate new words according to the amount of new words which need to be discovered and a confidence level.

Description

technical field [0001] The invention belongs to the technical field of natural language processing, and relates to a method for discovering new Chinese words based on the novelty degree and new evaluation index proposed by the invention. Background technique [0002] With the advent of an intelligent society, natural language processing has become a research hotspot for scholars. Words, as the basic unit of natural language processing, represent a semantic unit. The correctness of word division directly determines the performance of natural language processing systems. The continuous expansion of social networks has led to a large number of new words, a short cycle, and rapid spread, which has promoted the widespread use of new word discovery in Chinese word segmentation, part-of-speech tagging, and text classification. [0003] At present, the definition of "new words" is not uniform. It is generally believed that there are fixed collocations, and words used in a large rang...

Images