A method for establishing a word segmentation model, a word segmentation method and a device thereof

Abstract

The invention provides a method for establishing a word segmentation model, a word segmentation method, a device for establishing the word segmentation model and a word segmentation device. The method for establishing the word segmentation model comprises the following steps of: A1, labeling each entry of a training corpus and characteristics of each entry; B1, determining the parts of speech of each entry under corresponding characteristics; C1, counting the generation probabilities of each entry under corresponding parts of speech and the probabilities of conversion between the parts of speech by utilizing the labeled training corpus; and D1, obtaining a basic dictionary by utilizing the generation probabilities of each entry under the corresponding parts of speech, obtaining a conversion dictionary by utilizing the probabilities of the conversion between the parts of speech, and adding the basic dictionary and the conversion dictionary into the word segmentation model. When the word segmentation model is used for word segmentation, word segmentation accuracy can be improved, and characteristic labeling work can be finished at the same time of word segmentation.

Description

【Technical field】 [0001] The invention relates to the technical field of natural language processing, in particular to a method for establishing a word segmentation model, a word segmentation method and a device thereof. 【Background technique】 [0002] With the widespread use of the Internet, more and more texts and information are disseminated through the Internet. In order to retrieve and mine valuable content from these texts and information, natural language processing is an indispensable technology, and word segmentation is Fundamental work in natural language processing. [0003] In the prior art, word segmentation mainly includes rule-based word segmentation and statistics-based word segmentation. Rule-based word segmentation includes forward maximum matching, reverse maximum matching, two-way maximum matching, shortest segmentation segmentation, segmentation based on rule sets, etc. It is characterized by fast speed, but the effect on ambiguity segmentation is not g...

AI Technical Summary

Problems solved by technology

[0005] The technical problem to be solved by the present invention is to provide a method for establishing a word segmentation model, a method for word segmentation and a device thereof, so as to solve the problem of the amount of information caused by replacing the relationship between words and words with the relationship between parts of speech and parts of speech in the prior art. A large loss, thereby reducing the defect of word segmentation accuracy

Images