Word sense disambiguation method and device based on word vector

Abstract

The invention relates to a word sense disambiguation method and device based on a word vector. The method comprises the following steps that: data preprocessing: carrying out processing, including punctuation removal, word segmentation and the like, on a document and a semantic item; training the word vector: using a word vector training tool to train the word vector; carrying out context vector representation: obtaining the word vector, and adopting a local weighting method to calculate the context vector; carrying out semantic item vector representation: obtaining the word vector of each word of the semantic item, and carrying out calculation to obtain a semantic item vector; carrying out similarity calculation: calculating a cosine similarity between the context vector and each semanticitem vector; carrying out semantic item distribution frequency calculation: carrying out statistics on the distribution frequency of each semantic item of an ambiguous term in a dataset; and carryingout final score statistics: calculating the cosine similarity between the context and each semantic item and the comprehensive score of each piece of semantic item frequency, wherein the semantic item with a highest score is an optimal word meaning.

Description

technical field [0001] The present invention relates to a word meaning disambiguation method and device based on word vectors, belonging to the fields of natural language processing (Natural Language Processing), machine translation (Machine Translation), artificial intelligence (Artificial Intelligence) and other fields. Background technique [0002] In recent years, with the development of science and technology, word meaning disambiguation has become increasingly important in natural language processing, machine translation, artificial intelligence and other fields. Word sense disambiguation has become an urgent problem to be solved. [0003] With the popularity of the concept of word sense disambiguation, scholars have proposed solutions to word sense disambiguation. The word sense disambiguation knowledge used in the early days was artificially woven rules, but manually writing the rules was time-consuming and laborious and there was a bottleneck problem of knowledge a...

AI Technical Summary

Problems solved by technology

[0005] Word vectors have long been used in word sense disambiguation tasks. The previous vector representation method: One-HotRepresentation, the length of a word vector represented by this method is the length of the vocabulary, and most positions of the word vector are Zero, only the dimension corresponding to the position of the word in the vocabulary is 1. Obviously, this method is difficult to express the semantic information contained in the word and the semantic relationship between words

In addition, this representation has the problem of data sparsity

Images