Unsupervised automatic extraction method of microblog new words based on repeated word strings

Abstract

The invention discloses an unsuspervised automatic extraction method of microblog new words based on repeated word strings. The method includes the steps that firstly, text segmentation is conducted on microblog documents to be processed, texts are segmented through a dynamic programming word segmentation method, the word strings to be recognized are segmented, and word segmentation fragments in the word strings to be recognized are combined into the new words to be recognized; candidate new words are extracted from the word strings to be recognized according to a statistic word selection model, and then the candidate words are filtered through a rule filtering model, and eventually the final new words are acquired. The method has the advantages that the high accuracy rate is effectively guaranteed, the method does not depend on a rule word stock too much, and the extraction speed of the new words is guaranteed.

Description

technical field

[0001] The invention belongs to the technical field of new word retrieval methods, and relates to a non-supervised automatic extraction method of microblog new words based on repeated word strings. Background technique

[0002] New word recognition is one of the main problems plaguing the field of Chinese word segmentation, and with the development of Weibo, the speed of the emergence of new words has been accelerated. Unsupervised automatic recognition of new words is crucial for other natural language processing tasks. Automatic segmentation of Chinese text is an important basic work in the field of natural language processing. The identification and processing of new words is one of the difficulties that restrict the further improvement of the accuracy of the Chinese word segmentation system. At present, the research on new word extraction mainly focuses on the extraction of entity nouns, especially the extraction of names of people, places, and institut...

Images