Recursive and multilevel Chinese word segmentation method

Abstract

The invention discloses a recursive and multilevel Chinese word segmentation method, which comprises the following steps of: performing Chinese word segmentation on an input Chinese text by using a maximum matching algorithm, selectively shielding generated segmented words in a current dictionary tree, repeatedly performing Chinese word segmentation on the input Chinese text by using the shielded dictionary tree and the maximum matching algorithm, selectively shielding the generated segmented words till each generated segmented word does not have non-individual word prefix in the current dictionary tree, ending the word segmentation process, outputting the word segmentation result and recovering the dictionary tree before shielding. According to the recursive and multilevel Chinese word segmentation method, recursive and multilevel word segmentation is combined on the basis of a character string matched fine-grained Chinese word segmentation algorithm, so that the segmentation granularity is ensured, ambiguity is eliminated in word segmentation on each level, and the word segmentation accuracy is improved.

Description

technical field [0001] The invention relates to the technical field of natural language processing, in particular to a recursive multi-level Chinese word segmentation method. Background technique [0002] Chinese word segmentation refers to dividing a sequence of Chinese characters into individual words, and word segmentation is the process of recombining continuous sequences of Chinese characters into word sequences according to certain specifications. The existing Chinese word segmentation algorithms can be roughly divided into: word segmentation methods based on string matching and word segmentation methods based on statistics. The word segmentation method based on string matching matches the Chinese character sequence with the entry in a "sufficiently large" dictionary. If a certain string is found in the dictionary, the match is successful, that is, a word is recognized. According to different scanning directions, string matching word segmentation methods can be divide...

AI Technical Summary

Problems solved by technology

The Chinese word segmentation method based on string matching has the advantages of high efficiency, flexible update and maintenance, can include type information, and has the advantages of certain ambiguity resolution ability, but it has poor recognition ability for unregistered words

Coarse-grained segmentation can eliminate ambiguity, but the segmentation granularity is large. Fine-grained segmentation does not have the function of eliminating ambiguity, but the segmentation granularity is small

The word segmentation method based on statistics is to learn the statistical information of words into words from the corpus, so as to discover some word formation rules. It has a good ability to recognize unregistered words, but the efficiency is low, the update and maintenance are troublesome, and the cycle is long. At the same time, word segmentation The granularity is biased towards fine-grained segmentation

In many current natural language processing applications and search engine applications, considering the dual requirements of segmentation granularity and word segmentation efficiency, the word segmentation method based on string matching with fine segmentation granularity is adopted, and the generated word segmentation contains multiple levels. But there is no disambiguation function

Images