Text coding representation method based on transformer model and multiple reference systems

Abstract

The invention discloses a text coding representation method based on a transformer model and multiple reference systems, and the method comprises the steps: splicing a word vector and a segmentation character vector of a sentence where the word vector is located based on a word vector and a separator vector coding result of a context text, and obtaining a spliced word vector; mapping the spliced word vector according to at least two set semantic concepts, obtaining at least two semantic concept vectors of the word vector, and, when the absolute semantic concept number of the word vector is smaller than the set semantic concept total number, wherein the semantic concept vectors of the word vector represent convergence, finally leaving p kinds of dissimilar semantic concept vectors; selecting the most suitable semantic concept vector of the word vector in the current context from the dissimilar semantic concept vectors through maximum pooling, and taking the most suitable semantic concept vector as a semantic prediction result of the word vector in the current context; and obtaining a probability vector of the word vector, and determining a word probability under a semantic concept corresponding to the word vector according to the probability vector.

Description

technical field [0001] The invention relates to the field of machine understanding of natural language, in particular, to a text encoding representation method based on a transformer model and a multi-reference system. Background technique [0002] In order to make the machine make better use of language, words are mapped to multi-dimensional space through word vectors. Since human text expressions often need to be combined with context to form complete semantics, the semantic representation of text has become a great challenge for machine understanding. [0003] The existing pre-trained language representation method model is constructed by the word vector of words, and the human's level of understanding of text is the way of word-phrase-sentence-paragraph-chapter. The existing language model is different from the human understanding method. Differences lead to certain limitations in information granularity. [0004] In addition, in the models of existing pre-trained lang...

AI Technical Summary

Problems solved by technology

[0003] The existing pre-trained language representation method model is constructed by the word vector of words, and the human's level of understanding of text is the way of word-phrase-sentence-paragraph-chapter. The existing language model is different from the human understanding method. Differences lead to certain limitations in information granularity

[0004] In addition, in the models of existing pre-trained language representation methods, such as BERT, ELMO, ULMFiT, etc., not only the embedding layer but also the prediction layer (reference frame calculation layer) use a single word vector table as the reference frame, which makes the pre-training During the training language representation training process, different semantics of the same word converge with reference to the same target

Images