Word2vec improvement method of related factor training combining parts of speech and word orders

Abstract

The invention discloses a word2vec improvement method of related factor training combining parts of speech and word orders. The method provides a model of Structured word2vec on POS. The model includes a CWindow-POS (CWP) model and a Structured Skip gram-POS (SSGP) model. According to each of the two models, part-of-speech label information and the word orders are used as influencing factors for combined optimization, and part-of-speech correlation information is utilized for modeling inherent syntactic relationships between words in context windows; and weighted calculation is carried out on context word sequences through part-of-speech correlation weights, then vector inner-product calculation is carried out according to the word position orders, and a stochastic gradient descent (SGD) algorithm is used for learning the related weights and word embedding in a combined manner. The method directionally embeds words according to the position orders thereof, realizes combined optimization for word vectors and weighted matrices related to the parts of speech, and has high efficiency in all of word analogy tasks, word similarity tasks and qualitative analysis.

Description

technical field [0001] The invention belongs to the technical field of machine learning, and relates to a word2vec method, in particular to an improved word2vec method that combines part-of-speech and word-order correlation factor training. Sequence oriented embedding, and use part-of-speech association information to establish the inherent syntactic relationship between words in the context window; realize joint optimization of word vector and part-of-speech correlation weighting matrix. Background technique [0002] Part of speech is the basic element of natural language processing. Word order contains the conveyed semantic and grammatical information, which are the key information in natural language. How to effectively combine the two in the word embedding model is the focus of current research. Semantic vector space models of language represent each word with a real-valued vector, and word vectors can be used as features in many applications, such as document classific...

AI Technical Summary

Problems solved by technology

[0004] The shortcomings of the Word2vec model and the GloVe model are: first, it is not sensitive to word order information; second, it cannot use part-of-speech association information

This modeling preserves part-of-speech information very well, but does not preserve word order information

Images