HMM-based part-of-speech tagging method

Abstract

The invention relates to an HMM-based part-of-speech tagging method and belongs to the field of information processing technology. According to the method, first, words in a word bank are ordered according to unicodes so that a dichotomy method can be used for quick search during word segmentation; second, an HMM is introduced, a tagged corpus serves as a training set and a test set to be used forobtaining three parameters of the HMM, and therefore a plurality of observable states in the HMM are obtained; third, secondary word segmentation is performed, the words not found in a primary word segmentation result are searched for in the observable states in the HMM, and a maximum entropy model is introduced to perform tagging on new words not found; and last, a viterbi algorithm is used to calculate an optimal hidden sequence of the HMM, and the optimal hidden sequence is combined with the tagging result of the maximum entropy model to obtain a final part-of-speech tagging result. Compared with the prior art, the phenomenon that a single part-of-speech tagging method is low in speed and low in new word recognition rate, and consequently a tagging result is low in accuracy is mainly solved, and the efficiency and accuracy of part-of-speech tagging are improved.

Description

technical field [0001] The invention relates to an HMM-based part-of-speech tagging method, which belongs to the technical field of information processing. Background technique [0002] In modern society, with the rapid development of information technology, part-of-speech tagging has become an important research direction in natural language processing. Necessary preparation for tasks such as machine translation. [0003] Generally, although the part-of-speech tagging effect based on HMM is good, it has insufficient prediction information and poor ability to identify new words, so that the accuracy of part-of-speech tagging is not high; similarly, although the part-of-speech tagging based on the maximum entropy model can effectively use the context Information has a good predictive effect, but there are problems such as slow labeling speed and labeling bias. Contents of the invention [0004] The technical problem to be solved in the present invention is to provide a ki...

AI Technical Summary

Problems solved by technology

[0004] The technical problem to be solved in the present invention is to provide a kind of part-of-speech tagging method based on HMM for the limitation and deficiency of prior art, introduce the combination of HMM and maximum entropy model to improve the part-of-speech tagging method, and solve the single part-of-speech tagging method The speed is slow and the recognition rate of new words is low, which leads to the low accuracy rate of tagging results, so as to improve the efficiency and accuracy of part-of-speech tagging

Images