Polysemous-word word vector disambiguation method

Abstract

The invention discloses a polysemous-word word vector disambiguation method, and belongs to the cross-technical field of text mining and machine learning. The method comprises: (1) text corpus acquisition and preprocessing, wherein a Sogou news text classification corpus is adopted, and then preprocessing steps of word segmentation and stop word removing are carried out; (2) BTM topic model modeling; (3) probability calculation of corresponding of words to topics; (4) calculation of a vector model on word vectors and topic vectors; and (5) polysemous-word word vector construction, wherein thetopic vectors in a connection process are weighted in corresponding to probability P(z|w) of the topics, distinction on different meanings of the same word in different contexts is realized, and correct polysemous-word word vectors are obtained. According to the method, extension of Chinese word meaning disambiguation to the short-text field is facilitated; manpower data labeling is not needed through using combination of a topic model and the word vectors, massive short-text data mining is facilitated, and time and effort are more saved; and personalized product recommendation of e-commerce websites is facilitated.

Description

technical field [0001] The invention belongs to the cross technical field of text mining and machine learning, specifically relates to a polysemous word vector model, in particular to a word sense disambiguation method of the polysemous word vector model and its disambiguation application in short texts. Background technique [0002] There are various expressions in Chinese, and different words have different meanings in different contexts. Experts have many solutions to how to get the correct semantics of words, and there are also many problems, mainly including: [0003] 1) Knowledge-based methods, based on artificially constructed text databases and corpora, have the advantage of relatively high accuracy of results, which mainly come from various confirmed corpora, but for some fields, the corpus is relatively small. Sound, so it is inevitable to encounter the problem of sparse corpus features, which is limited by the completeness of knowledge construction. [0004] 2) T...

AI Technical Summary

Problems solved by technology

[0003] 1) Knowledge-based methods, based on artificially constructed text databases and corpora, have the advantage of relatively high accuracy of results, which mainly come from various confirmed corpora, but for some fields, the corpus is relatively small. Sound, so it is inevitable to encounter the problem of sparse corpus features, which is limited by the completeness of knowledge construction

[0004] 2) The supervised method is based on manual labeling of corpus data, but strongly depends on the already marked corpus data. However, for languages ​​without labels, this method is not suitable for word sense disambiguation. This method is not suitable for labeling Data dependency is too strong

[0005] 3) The unsupervised method is based on the incomplete need for corpus and various language annotation information, so it has better applicability, but the accuracy is relatively low

[0006] It can be seen that the existing word sense disambiguation methods face the problems of high corpus dependence, cumbersome manual annotation and low accuracy, so the word sense disambiguation method needs to be further explored

Images