New word discovery-based cross-domain Chinese word segmentation system and method

Abstract

The invention discloses a cross-domain Chinese word segmentation system and method based on neologism discovery. The system comprises: a neologism discovery module which achieves a neologism discovery algorithm through the use of enhanced mutual information combining statistical information and semantic information, and is used for mining a neologism list from an unlabeled corpus; an automatic tagging module, which is used for realizing initial segmentation of untagged corpora by using a new word list in combination with a reverse maximum matching algorithm to obtain incompletely segmented corpora, and completely segmenting the incompletely segmented corpora by using a word segmentation model to obtain automatically tagged corpora; a cross-domain word segmentation module, which is used for realizing a cross-domain Chinese word segmentation algorithm by using an adversarial method, and carrying out adversarial training by using the labeled source domain corpus and the automatically labeled corpus. According to the method, a new word discovery algorithm is optimized by using enhanced mutual information, and the accuracy of new word discovery and the domain of a word list are improved; in a cross-domain word segmentation algorithm, the utilization rate of unlabeled corpora is improved, and the recall rate and accuracy of word segmentation are optimized.

Description

technical field [0001] The invention relates to the technical field of natural language, in particular to a cross-domain Chinese word segmentation system and method based on new word discovery. Background technique [0002] Chinese texts use Chinese characters as the smallest writing unit, Chinese characters are combined to form words, and finally Chinese texts are composed of words. Words are the smallest structural units in Chinese text that contain semantic information and can be used independently. However, unlike English and other languages, there are no explicit separators between Chinese words. Certain technical methods are used to divide Chinese text into words to facilitate computer understanding. , this process is Chinese word segmentation. Chinese word segmentation is the most basic task in Chinese natural language processing, and it is the cornerstone of natural language processing tasks such as text classification, text generation and sentiment analysis. There...

AI Technical Summary

Problems solved by technology

The best way to solve the expression gap is to manually label the corpus of the target domain, and then mix the corpus of the two domains to retrain the model. However, large-scale manual labeling requires a lot of manpower and material resources, and it is impossible for all The domains are manually marked, so it is not feasible; the best way to solve the unregistered words is to let professionals extract words that have never appeared in the source domain corpus from the target domain corpus, and put these words as training corpus However, selecting these unregistered words requires a lot of manpower and material resources on the one hand, and on the other hand, because various new words emerge in an endless stream in today's society, it is impossible to rely on manpower to select all unregistered words

Therefore, cross-domain Chinese word segmentation has been difficult to achieve better results

Images