A method and computer for expressing deep dynamic contextual words

Abstract

The invention belongs to the technical field of computer word representation, and discloses a model and method of deep dynamic context word representation. The model of deep dynamic context word representation is a masked language stacked by multi-layer bidirectional Transformer encoders with a layer attention mechanism. model; it is a multi-layer neural network, each layer of the network captures the context information of each word in the input sentence from different angles; Represents contextual representations that combine to form words. The word representation generated by this model has performed three tasks of logical reasoning (MultiNLI), named entity recognition (CoNLL2003) and reading comprehension task (SQuAD) on public datasets, which are 2.0%, 0.47% and 2.96%.

Description

technical field [0001] The invention belongs to the technical field of computer word representation, and in particular relates to a model and method of deep dynamic context word representation and a computer. Background technique [0002] Currently, the closest state-of-the-art: neural network language models. The representation of words as continuous vectors has a long history. A very popular neural network language model NNLM (Neural Network Language Model) uses a linear projection layer and a nonlinear hidden layer feed-forward neural network to jointly learn word vector representation and statistical language model. Because the model has too many parameters, although the principle is simple, it is difficult to train and apply in practice. CBOW, Skip-Gram, FastText and Glove models. CBOW, Skip-Gram, FastText and GloVe and other models, among which CBOW and Skip-Gram belong to the model under the famous word2vector framework, are trained using shallow neural network lan...

AI Technical Summary

Problems solved by technology

[0006] To sum up, the problem with the existing technology is that the commonly used word embedding technology has no concept of context and dynamics, and treats words as fixed atomic units, which limits the effect on many tasks.

Previously used word embedding techniques cannot be patched with improved methods

It can only re-model conceptual word representations with context and dynamics. At the same time, it is more difficult to consider that the model generates word representations in a variety of tasks, the efficiency of generating word representations is high, and the resources required by the model are small.

Images