Method and device for automatically discovering new words from document set

Abstract

The invention discloses a method and a device for automatically discovering new words from a document set. A template acquisition unit acquires one or more templates; a word extraction unit extracts words matched with all templates of the one or more templates from the document set; a candidate template set adding unit selects at least a part of templates from the one or more templates and adds the part of templates to a candidate template set; a candidate word set adding unit selects at least a part of words from the words matched with all the templates of the one or more templates and adds the part of words to a candidate word set; a new word set adding unit sequences candidate words in the candidate word set according to the templates in the candidate template set, and adds a certain number of candidate words to a new word set according to the sequence. Compared with the prior art, by adopting the method and the device, new words can be effectively discovered.

Description

technical field [0001] The invention relates to natural language processing technology, in particular to a method and device for automatically discovering new words from a document collection. Background technique [0002] In social networks, netizens like to use their own personalized language to express their views on politics, society, culture, etc. Generally, the more people disseminate personalized language, the easier it is to become a new hot word on the Internet (referred to as "new word"). At present, new words have important applications in automatic summarization, text clustering / classification, information retrieval, etc. According to statistics, more than 1,000 new Chinese words appear on the Internet every year, and most of these new words are time-sensitive in various fields. Professional terms, since most of these new words do not exist in the dictionary, it is difficult for existing word segmentation algorithms to identify these new words from the document ...

AI Technical Summary

Problems solved by technology

At present, new words have important applications in automatic summarization, text clustering / classification, information retrieval, etc. According to statistics, more than 1,000 new Chinese words appear on the Internet every year, and most of these new words are time-sensitive in various fields. Professional terms, since most of these new words do not exist in the dictionary, it is difficult for the existing word segmentation algorithm to identify these new words from the document set

Taking the emotional new word "Give force (adjective)" and the document "The performance is very good" as an example, the existing word segmentation algorithm usually performs the following word segmentation: performance / noun very / adverb give / verb force / noun, so that the new The word "Gili" cannot be segmented as a complete word to affect the recognition of new words

Images