Comprehensive position coding method for vocabulary sequence data

Abstract

The invention discloses a comprehensive position coding method for vocabulary sequence data, which comprises the following steps of: before a vocabulary sequence is input into a model, adding codes ofabsolute positions of vocabularies in the sequence to each vocabulary in addition to codes of the vocabularies; when binary relation calculation of every two vocabularies is carried out in the deep learning model, adding codes of the relative positions of the two vocabularies in the sequence; and optimizing the numerical values of the two position codes, and continuously adjusting the numerical values in the training process. According to the method, on the basis that absolute position coding is carried out on original positions of vocabularies, the distance between any two vocabularies is further coded, and the two codes are combined, so that the seriality of data can be effectively reflected when language source data with serialization characteristics is input into a deep learning modelin parallel for calculation. Compared with an existing position coding method, the method has the advantages that a current mainstream machine translation model can achieve higher translation precision, and the error rate is lower.

Description

technical field [0001] The invention relates to a comprehensive position coding method for vocabulary sequence data, which belongs to the technical field of natural language processing. Background technique [0002] In natural language processing tasks, the most common source data unit used as input is a sentence, or a sequence of words, which is born with time / space / logic relationship sequence attributes. Naturally, when using neural network models to process sequences, people first think of recurrent neural networks: a deep learning model with the ability to serialize data. But from the perspective of semantic analysis, the processing of lexical sequences cannot be carried out completely in accordance with the spatial order, because the relationship between words and words is not completely consistent with their spatial order in the sequence, and the simple recurrent neural network cannot handle long-term rely. Deep learning models such as LSTM and attention-based bidire...

AI Technical Summary

Problems solved by technology

On the one hand, the data features represented by these two codes cannot replace each other. After the absolute position code is added to the vocabulary code, the model can identify the absolute position of a certain word in the vocabulary sequence. This optimized model, no matter in In the task of machine translation, or in other tasks of natural language processing, it can effectively predict which words should appear in each position in the sentence sequence in the test, and combine the words that have appeared before to effectively predict the next position. Which specific vocabulary should the vocabulary be; and after incorporating the relative position encoding into the calculation of the binary relationship, the model can effectively learn the impact of the distance between any two vocabulary on the relationship between the two, or comprehensively based on any two vocabulary The encoding and relative position encoding calculate the relationship between the two, and in the final test, find the most suitable word among the candidate words according to the distance relationship between the current position and the previous words; but on the other hand, before Efforts to fuse the two encodings have not yielded much results

Images