Two-stage semantic word vector generation method

Abstract

The invention provides a two-stage semantic word vector generation method. The two-stage semantic word vector generation method comprises the following five steps of: performing text matrixizing; constructing a feature extractor; performing semantic recognition; constructing a neural language model; and generating a semantic term vector. According to the method, the corresponding word vectors aregenerated for different semantics of the polysemy by using the plurality of neural networks, the defect that the polysemy only corresponds to one word vector in a traditional word-level embedded modeis overcome, and the size of the used corpus is within an acceptable range; meanwhile, a mode of combining a convolutional neural network (CNN) and a support vector machine (SVM) is adopted, on one hand, the feature extraction capability of the convolutional neural network is utilized, and on the other hand, the generalization and robustness of the SVM are utilized, so that the word meaning recognition effect is better, and the generated semantic word vector quality is higher.

Description

technical field [0001] The invention belongs to the field of neural networks, and in particular relates to a two-stage semantic word vector generation method. Background technique [0002] Word representation is one of the key issues in natural language processing. Whether the word representation method is appropriate directly affects the modeling methods of tasks such as syntactic analysis, semantic representation and text understanding, and also affects the accuracy and robustness of application systems such as information retrieval and question answering systems. [0003] At present, the representation strategies of Chinese words can be summarized into three types: traditional 0-1 representation, distributed representation based on latent semantic information and distributed representation based on neural network language model. The traditional 0-1 representation has two problems: on the one hand, the 0-1 representation causes data sparsity, which makes the word vectors ...

AI Technical Summary

Problems solved by technology

The traditional 0-1 representation has two problems: on the one hand, the 0-1 representation causes data sparsity, which makes the word vectors generated in this way take up a lot of space; on the other hand, the 0-1 representation can only distinguish different words, but no contribution to the representation of word sense

Li Guojia used K-Means clustering to build a two-stage model in the word meaning recognition stage. The disadvantage of this method is similar to Neelakantan's method. It is necessary to set the number of center clusters for the K-Means algorithm in advance, which is equivalent to determining the number of word meanings generated in advance. number, the scalability is not good enough

[0007] Summarizing the existing methods, we can know: the disadvantage of 0-1 representation is that it causes dimensionality disaster and lack of semantic information; the disadvantage of word-level embedding is that 1) the word vectors generated by word training with multiple meanings are more biased in the corpus More semantics appear, and less semantics in the corpus are weakened; 2) In the result of calculating a high similarity with a polysemous word, the semantics are not related to each other; 3) The original triangular inequality of the word vector space is destroyed , which reduces its quality; in word sense level embedding, the fusion model can compress the generation process of word vectors, but its effect is based on the effect of the clustering algorithm it uses, and most of the current clustering algorithms are not as good as supervised models classification algorithm

The two-stage model ignores the similarity between the process of word meaning recognition and word vector generation. The two processes are completed in series, which is inefficient, but it has a high guarantee for the quality of the generated word vector.

Images