Chinese formal text word segmentation method based on active learning

Abstract

The invention provides a Chinese formal text word segmentation method based on active learning. The method comprises the steps that a current annotation data set L is used to train a naive bayes classifier; the current naive bayes classifier is used to annotate a to-be-annotated data set U; a sampling method is used to select a most informative fragment to be annotated for an expert; the new-sampled annotated fragment is added into the annotation data set L; and constant iteration is carried out until a preset satisfaction condition stops. The Chinese formal text word segmentation method basedon the active learning can effectively reduce artificial annotation data and obtain a tokenizer with better performance. The performance (measured by adopting an F value) of a model obtained by dataextraction and training by using an active learning method is about 5 percentage points higher than that of a model obtained by the data extraction and training by adopting a random drawing method. The performance each time of the model obtained by data extraction and training after the active learning is combined with EM iteration is improved by about 1.5 percentage points than that of the modelobtained by the data extraction and training by separately adopting the active learning method.

Description

technical field [0001] The invention relates to the technical field of word segmentation, in particular to an active learning-based Chinese formal text word segmentation method based on active learning and expectation maximization algorithms. Background technique [0002] Word segmentation is a key basic step in natural language processing, and an indispensable key link in many application systems, such as: information retrieval, named entity recognition, machine translation, syntactic analysis, etc. The effect of word segmentation directly affects the final results of these applications. Effect. However, compared with inflectional language texts such as English, there is no obvious separator like a space between words in agglutinative language texts such as Chinese. Allowing the computer to automatically recognize the boundaries between Chinese word strings and words is Chinese word segmentation. Nowadays, there have been a large number of researches on Chinese word segme...

AI Technical Summary

Problems solved by technology

Traditional greedy algorithms include forward maximum matching, reverse maximum matching, and two-way matching. This method requires a large amount of manual labeling data, and at the same time cannot solve the two major problems of Chinese word segmentation, meaning divergence and unregistered words.

In 1986, Liang Nanyuan and others applied the maximum matching method to Chinese word segmentation. The maximum matching method is a typical Chinese word segmentation method based on a dictionary. Its disadvantage is that it cannot solve the problem of boundary ambiguity and unregistered words.

Images