Iteration-based three-step unsupervised Chinese word segmentation method

Abstract

The invention discloses an iteration-based three-step unsupervised Chinese word segmentation method and belongs to the field of natural language processing technology. According to the basic thought,the method is an unsupervised word segmentation framework including local segmentation, global word selection and corpus reduction iteration execution; and in each iteration, a word formation probability model based on segmentation-context mutual independency is utilized to perform locally optimal unsupervised segmentation on text corpus, and the form is simple and effective; a document-level pulse weighting method is adopted according to the long-tail phenomenon; according to a global support degree, new words are screened, and a dictionary is incrementally generated; and last, a text is divided based on the longest matching and maximum probability principle of the dictionary, formed segmented words are filtered out, continuous non-segmented words are stitched, the words are reconstructedinto a scale-reduced training corpus, and similar iteration processing is performed on the remaining corpus till no new word is generated. The method is superior to an existing Chinese unsupervised word segmentation algorithm with best performance.

Description

Technical field [0001] The invention relates to a three-step unsupervised Chinese word segmentation method based on iteration, which belongs to the technical fields of artificial intelligence, machine learning and natural language processing. Background technique [0002] Various natural language processing tasks, including information retrieval, machine translation, text understanding and mining, etc., are all carried out with words as the basic unit. Chinese text consists of a sequence of consecutive characters, with no boundaries between words in the sentence. Therefore, the natural language processing of Chinese requires word segmentation first, that is, segmentation of consecutive character sequences into word sequences, and then processing such as grammatical analysis, semantic understanding and pragmatic analysis. [0003] The existing word segmentation algorithms can be roughly divided into supervised methods and unsupervised methods. Supervised word segmentation is to pe...

AI Technical Summary

Problems solved by technology

However, MCA is composed of five complex sub-models, and it also needs to preprocess the corpus for word alignment. The complexity is high, and there is a large room for improvement in word segmentation accuracy.

Images