Self-adaptive Chinese new word recognition method and system

Abstract

The invention provides a self-adaptive new Chinese word recognition method and system. The method comprises the following steps: a text initialization step: carrying out the structural processing of an input text, and obtaining a wide character sequence which contains original Chinese characters and maintains the adjacent relation and interval relation of the Chinese characters in an original text; a non-accidental co-occurrence judgment step: using binomial distribution or Poisson distribution to approximately characterize probability distribution of front and back adjacent occurrence of a pair of Chinese characters, and determining all non-accidental co-occurrence adjacent Chinese character pairs in the text text text based on a given non-accidental co-occurrence significance level alphap; a statistical relevance judgment step: giving a relevance significance level alpha k, judging the relevance degree of adjacent Chinese characters in the text, and screening out Chinese character strings with internal strong relevance; and an existing word bank filtering step: based on an existing dictionary, screening out Chinese character strings which do not appear in the dictionary yet fromcharacter strings which meet non-accidental co-occurrence and are high in internal relevance.

Description

technical field [0001] The invention belongs to the field of Chinese language and text information processing, and relates to related technical fields such as Chinese neologism recognition, semantic analysis, automatic translation, information retrieval, and Chinese word segmentation, and in particular to a Chinese neologism recognition method and system based on accidental judgment and association judgment . Background technique [0002] With the development of the Internet and artificial intelligence technology, people's demand for semantic analysis, automatic translation, information extraction and retrieval and other applications continues to increase, all of which require Chinese words as the basic unit of processing. However, unlike languages ​​in Latin and Roman languages, Chinese does not use spaces as vocabulary separators. Therefore, when processing Chinese text, it is necessary to segment the text according to existing lexicons such as dictionaries. [0003] Howe...

AI Technical Summary

Problems solved by technology

In addition, although the statistical characteristics of its definition can characterize the characteristics of new words to a certain extent, the setting of the threshold is also manually set based on experience.

The problem that is easily caused by this is: setting the threshold too high will reduce the recognition of new words, and setting the threshold too low will introduce a large number of irrelevant strings

Due to the correlation between new word detection and the training corpus, when the type of training corpus is inconsistent with the target text type, the effect of new word detection will be affected

Images