Word multi-prototype vector representation and word sense disambiguation method based on CRP clustering

Abstract

The invention discloses a word multi-prototype vector representation and word sense disambiguation method based on CRP clustering, which comprises the following steps: the text in the massive text corpus is purified and pretreated to obtain plain text, CRP algorithm is used to cluster the context window representation of target polysemous word in the text corpus set. The target polysemous words inthe text corpus set are marked according to the clustering classification, and the polysemous words are trained on the marked text corpus set to obtain the multi-prototype vector representation of the polysemous words; 2, the target short text is preprocessed to obtain a short text word sequence, a target polysemous word in a word sequence is identifued, the contextual window of the target polysemous words is used to represent the similarity between the centroids of clusters corresponding to the words in the text corpus, and the word vector corresponding to the maximum similarity clusters isused as the word vector representation of the specific meaning of the polysemous words in the context to disambiguate the meanings of the polysemous words. The invention solves the problem of polysemyexpression in word expression and the problem of ambiguity identification in word meaning expression.

Description

technical field [0001] The invention relates to the field of natural language processing, in particular to a CRP clustering-based word multi-prototype vector representation and word sense disambiguation method. Background technique [0002] Among the many tasks in the field of natural language processing, the basic problem faced is how to represent language symbols as encoding patterns that can be processed by machines. Map and represent language symbols, express words, sentences, texts, etc. as a continuous low-dimensional vector, and realize semantic vector representation of words, sentences, and texts. There are a wide range of applications for tasks such as recommendation engines and automatic text summarization. [0003] Words are the most basic unit of language, and the vectorized representation of words is widely used in natural language processing tasks. A simple word vector representation is One-hot Representation. The disadvantage of this representation method is...

AI Technical Summary

Problems solved by technology

Based on the method of external knowledge base, with the help of external knowledge base (WordNet or HowNet) to explain or describe the different semantics of words, to identify the specific semantics of polysemy words, but the construction of external knowledge base or dictionary requires a lot of manpower and material resources

Images