Method for extracting novel field words

Abstract

The invention discloses a method for extracting novel field words by the aid of combination of word2vec and Bootstrapping iteration. The method includes preprocessing field word corpus; segmenting preprocessed field texts by the aid of n-gram; counting word frequencies, left and right adjacent word numbers, left and right word entropy and mutual information six-dimensional statistics of segmented character strings; setting a group of parameters by the aid of kmeans; carrying out preliminary evaluation; carrying out filtering to obtain first round of results; respectively computing sums of cosine similarity of each candidate word and seed sets by the aid of word vector spaces and a group of field seed data; setting sum threshold values and carrying out secondary evaluation so as to extract the novel words of fields. The word vector spaces are obtained by means of word2vec training. The method has the advantages that the method is applicable to extracting the novel words from the large-scale field corpus and excellent in portability; the problem of difficulty in filtering non-field words with verb-object constructions, reduplication and the like can be fundamentally solved by the aid of the method.

Description

technical field [0001] The invention relates to the field of new word discovery in natural language processing, in particular to a method for extracting new words in the field. Background technique [0002] With the rapid development of Internet technology, a large number of new words continue to emerge. Including proper nouns, derived words, dialect words, industry words, transliterated words, words with foreign letters, words used in Hong Kong and Taiwan, and related terms in other fields, these words are called neologisms. [0003] Chinese word segmentation is the basis of most information retrieval systems. However, the existing word segmentation algorithms cannot cover all new words in the field. At this time, it is necessary to use the user dictionary to improve the accuracy of word segmentation, thereby improving the quality of the retrieval system. It can be seen that timely discovery and It is of great significance to update new words in the field. [0004] Tradit...

AI Technical Summary

Problems solved by technology

[0003] Chinese word segmentation is the basis of most information retrieval systems. However, the existing word segmentation algorithms cannot cover all new words in the field. At this time, it is necessary to use the user dictionary to improve the accuracy of word segmentation, thereby improving the quality of the retrieval system. It can be seen that timely discovery and Updating new words in the field is of great significance

Images