Deep dynamic context word representation model and method and computer

Abstract

The invention belongs to the technical field of computer word representation, and discloses a deep dynamic context word representation model and method. The deep dynamic context word representation model is a masking language model formed by stacking multi-layer bidirectional Transformer encoders with layers of attention mechanisms. The network is a multi-layer neural network, and each layer of the network captures context information of each word in the input statement from different angles. Then different weights are given to each layer of the network through a layer attention mechanism. Finally, different word representations are synthesized according to the weights to form context representations of the words. Words generated by using the model represent three tasks of logic reasoning(MultiNLI), named entity recognition (CoNLL2003) and reading understanding (SQuAD) on a public data set, and the tasks are improved by 2.0%, 0.47% and 2.96% respectively compared with an existing model.

Description

technical field [0001] The invention belongs to the technical field of computer word representation, and in particular relates to a model and method of deep dynamic context word representation and a computer. Background technique [0002] Currently, the closest state-of-the-art: neural network language models. The representation of words as continuous vectors has a long history. A very popular neural network language model NNLM (Neural Network Language Model) uses a linear projection layer and a nonlinear hidden layer feed-forward neural network to jointly learn word vector representation and statistical language model. Because the model has too many parameters, although the principle is simple, it is difficult to train and apply in practice. CBOW, Skip-Gram, FastText and Glove models. CBOW, Skip-Gram, FastText and GloVe and other models, among which CBOW and Skip-Gram belong to the model under the famous word2vector framework, are trained using shallow neural network lan...

AI Technical Summary

Problems solved by technology

[0006] To sum up, the problem with the existing technology is that the commonly used word embedding technology has no concept of context and dynamics, and treats words as fixed atomic units, which limits the effect on many tasks.

Previously used word embedding techniques cannot be patched with improved methods

It can only re-model conceptual word representations with context and dynamics. At the same time, it is more difficult to consider that the model generates word representations in a variety of tasks, the efficiency of generating word representations is high, and the resources required by the model are small.

Images