Word sense disambiguation method based on hidden Markov model

Abstract

A word sense disambiguation method based on a hidden Markov model includes step 1) of training corpus, using a SemEval 2007#task5 test corpus set to parse a to-be-disambiguated sentence; performing word segmentation on the sentence; step 2) of finding ambiguous words in the sentence after the word segmentation, extracting a target ambiguous word and segmented words on the left and right of the target ambiguous word; training corpora, and calculating semantic class vocabulary transition probability and semantic class transition probability; step 3) of extracting the number of sentences containing the ambiguous word from manually annotated corpora, calculating the observation probability, and calculating the observation probability of words on the left and right of the ambiguity word; step 4) of using the value trained by the previous corpus to calculate the state transition probability, the extracted initial state probability, the observation probability and the state transition probability as parameters of the hidden Markov model, and using the well constructed disambiguation model to disambiguate the sentence in the test corpus; and step 5) of using a similarity calculation methodto verify the accuracy of disambiguation results.

Description

technical field [0001] The present invention relates to a word sense disambiguation method based on dictionaries and machine learning theories. This method has good application in natural language processing problems, such as: machine translation, information extraction, speech recognition and syntax analysis, etc., and has strong expansion Sex and flexibility. Background technique [0002] There are a large number of polysemous words in natural language. In different scenarios, the meanings expressed are not the same. How to determine which word meaning each ambiguous word corresponds to as the correct meaning of the word in a given context is the key to word sense disambiguation. solved problem. Generally, word sense disambiguation is processed. If the corresponding parts of speech are different among the multiple meanings of the ambiguous word, the correct word sense matching can be selected, which can be completed in the part-of-speech tagging stage. At present, word s...

AI Technical Summary

Problems solved by technology

[0006] The training of the word sense disambiguation model requires a large amount of word sense tagged corpus data, but building a word sense tagged corpus requires a lot of manpower and material resources

Due to the lack of corpus data, the model parameter data is relatively sparse, making this disambiguation method insufficient to support a large number of disambiguation tasks

Images