New word discovery method and system

Abstract

A new word discovery method and system are provided. The new word discovery method comprises the following steps: obtaining a first candidate word set by segmenting each sentence in the text; screening the first candidate word set to obtain a second candidate word set by the part-of-speech rule of the first dictionary; obtaining word segmentation from each sentence by a second dictionary, and determining a candidate word coefficient according to a relationship between a boundary word of a candidate word in a second candidate word set and a boundary word of a word segmentation; adjusting the candidate word coefficients according to the internal cohesion degree and the boundary freedom degree of the candidate words in the second candidate word set, and selecting new words from the second candidate word set according to the adjusted candidate word coefficients, wherein the first dictionary is the same as or different from the second dictionary. According to the new word discovery method and system, the influence of the specific part of speech and the dictionary on the new word discovery result can be reduced, and the candidate word coefficient can be obtained through the boundary relation, the internal cohesion degree and the boundary freedom degree, and the new word discovery result is more accurate.

Description

technical field [0001] The present invention generally relates to natural language processing, and more specifically relates to a new word discovery method and system. Background technique [0002] In the existing new word discovery method, since the results of word segmentation are used to combine candidate words and screen out new words from candidate words, but word segmentation itself relies on the establishment of a dictionary. For texts in new fields, when new words are not known, In this case, the word segmentation may be wrong, causing the new word to never be recognized. [0003] On the other hand, the results obtained by the threshold filtering method based on the characteristics of candidate words are difficult to target specific fields. For example, candidate words are usually sorted by frequency of occurrence, and most of them are common words in general fields, such as "the first time", "every time", etc. Years", etc., while special words such as "insured pers...

AI Technical Summary

Problems solved by technology

[0004] Exemplary embodiments of the present invention are to provide a new word discovery method and system to solve the problem that some words cannot be found, the results obtained based on existing methods are difficult to target specific fields, and some words with low frequency of occurrence are not found. At least one of the questions identified as a new word

Images