Word vector construction method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of big data, and discloses a word vector construction method and device, electronic equipment and a storage medium. A more accurate word vector is constructed by fusing a local word vector representing local context features and a topic distribution vector representing probability distribution of topics of texts where segmented words are located, so asto realize ambiguity elimination of polysemous words. The method comprises the steps: performing word segmentation processing on a plurality of to-be-processed texts to obtain segmented words in eachto-be-processed text; obtaining a local word vector of each segmented word based on the context of each segmented word; acquiring a topic distribution vector of each segmented word based on probability distribution of a topic to which each segmented word belongs in the plurality of to-be-processed texts; and respectively fusing the local word vector and the topic distribution vector of each segmented word to obtain a target word vector of each segmented word.

Description

technical field [0001] The present application relates to the field of big data technology, and in particular to a word vector construction method, device, electronic equipment and storage medium. Background technique [0002] Polysemy is a common phenomenon in natural language processing and a problem that needs to be solved in many scenarios. Whether it is Chinese or English, many words often have different semantics in different contexts. For example, "apple" is a type of fruit in a text such as "Apple has been produced in recent times with plenty of water," while "Apple" refers to a mobile phone brand in a text such as "Apple has released the latest version of its iPhone." When the current mainstream word vector models (such as word2vec model, etc.) construct word vectors, a word is usually represented by a unique vectorized code, resulting in the inability to distinguish polysemous words in different contexts during the word vector construction stage, reducing the subse...

AI Technical Summary

Problems solved by technology

When the current mainstream word vector models (such as word2vec model, etc.) construct word vectors, a word is usually represented by a unique vectorized code, resulting in the inability to distinguish polysemous words in different contexts during the word vector construction stage, reducing the subsequent natural Accuracy of language processing

Images