BPE-Learn acceleration method for sub-word segmentation

Abstract

The invention discloses a BPE-Learn acceleration method for sub-word segmentation, and the method comprises the steps: reading in training data, calculating the frequency of occurrence of each word ina corpus, and recording the frequency as a vocabulary; dividing the vocabulary into N sub-tables; using characters as basic units for byte pair statistics in the sub-process; starting to count byte pairs in respective sub-tables by the plurality of sub-processes at the same time, and informing the host process that the counting is finished through a communication queue; enabling the main processto read a temporary file, summarize the statistical result of each sub-process, select a byte pair with the highest frequency, store the byte pair into the file, and meanwhile, perform pruning; enabling the sub-process to wait for receiving a main process signal, conducting zero setting on the byte pair frequency which is not segmented, calculating and updating the byte pair frequency of the source word, and returning the byte pair frequency to the main process; and ending the statistics when byte pairs in the file stored in the host process meet the quantity requirement. According to the method, the byte pair statistical time in sub-word segmentation before training is shortened, and the GPU utilization rate is increased in neural machine translation model training.

Description

technical field [0001] The invention relates to the field of machine translation, in particular to a BPE-Learn acceleration method for subword segmentation. Background technique [0002] Neural Machine Translation (NMT) is a machine translation technology that uses words as the smallest unit. Because the NMT system involves a huge amount of computational complexity, the system limits the NMT vocabulary to a fixed-size vocabulary in order to keep resources and time consumption within an available range. Vocabularies that have not appeared in the training corpus are called unregistered word (OOV). For unregistered words, the NMT system uses the unified tag UNK to replace them. This method leads to inaccurate translations of unregistered translations, and even destroys the translation structure of the entire sentence. In order to avoid the occurrence of unregistered words, the researchers proposed the subword segmentation method BPE. After subword segmentation, words are div...

AI Technical Summary

Problems solved by technology

Under such a massive amount of data, the BPE-Learn process will consume a lot of time, as long as several hours or even more than ten hours, so that the data preprocessing process before training takes up a lot of time, and at the same time, it also consumes precious GPU resources. caused a certain waste

[0007] BPE-Learn conducts statistics on the full amount of corpus, and does not allow data to be divided into statistics. The ordinary multi-process synchronous acceleration method can only obtain local byte pair frequencies, and cannot represent the highest global frequency.

[0008] Therefore, the traditional BPE-Learn algorithm cannot complete byte pair statistics in a short period of time under massive data, nor can it achieve multi-process acceleration of byte pair statistics through data segmentation and other methods

Images