Word segmentation method based on MMseg algorithm and pointwise mutual information algorithm

Abstract

The invention relates to a word segmentation method based on an MMseg algorithm and a pointwise mutual information algorithm. A text is subjected to word segmentation processing by the MMseg algorithm based on a dictionary, and a word segmentation result is corrected by the pointwise mutual information algorithm after the word segmentation result is obtained. A specific process of correcting the word segmentation result by the pointwise mutual information algorithm comprises the following steps: calculating pointwise mutual information of a character x and a character y which are adjacent to each other in the text; judging whether the pointwise mutual information of the character x and the character y is larger than a set threshold value or not; and if so, segmenting the character x and the character y as an independent word.

Description

technical field

[0001] The present invention relates to the field of Chinese word segmentation, and more specifically, relates to a word segmentation method based on MMseg algorithm and point-by-point mutual information algorithm. Background technique

[0002] my country's research on natural language processing started relatively late, and it only established its own natural language processing model in the 1980s. Later, with the development of computers and the improvement of users' own needs, the domestic emphasis on natural language has greatly increased. The number of research institutions has increased and the research team has grown. The research team combined the characteristics of Chinese texts while drawing on foreign achievements, and proposed a new theoretical model to improve the level of research on Chinese understanding.

[0003] There are spaces between words in English word segmentation, but in Chinese text, characters between sentences are connected togethe...

Images