Bert model pre-training method and system and computer equipment

Abstract

The invention relates to a Bert model pre-training method and system and computer equipment, and the method comprises: an original data set obtaining step for obtaining an original data set; a data set preprocessing step for performing word segmentation processing on the original data set to obtain a word segmentation data set, performing Word2Vec model training on the word segmentation data set to obtain word embedding matrixes of all vocabularies, and sorting and encoding the vocabularies according to the occurrence frequency to obtain high-frequency vocabularies, low-frequency vocabulariesand vocabulary codes; and a Bert model pre-training step for freezing word embedding matrix parameters of the Bert model, training the Bert model based on the word embedding matrixes of all the vocabularies, reducing the learning rate and inputting the vocabulary codes to train the Bert model again. Through the method, convergence of model parameters is optimized, and model oscillation is effectively prevented.

Description

technical field [0001] The present application relates to the field of Internet technology, in particular to a Bert model pre-training method, system and computer equipment. Background technique [0002] With the rise of deep learning technology, more and more pre-trained models have been applied to natural language processing tasks, which has greatly improved the effect of the model. Early natural language pre-training used word vector methods such as word2vec to map high-dimensional sparse word vectors to dense low-dimensional vectors as input for subsequent models. With the development of deep learning, some more powerful pre-training models have been proposed, and Bert can be regarded as a representative of them. With the help of Bert's powerful pre-training, the current best results have been achieved on many tasks. [0003] In addition to being applied to natural language processing tasks, the natural language pre-training model can also be applied to other sequence d...

AI Technical Summary

Problems solved by technology

[0005] Based on the above pre-training method, before training word2vec, directly setting the words after the top n in frequency to the same word "UNK" will lose a lot of word information, so that many words representing different information in the preprocessing process use the same word. A word is represented; on the other hand, directly using the word2vec pre-trained word embedding matrix to initialize the Bert model word embedding matrix and training with the Bert model may lead to poor learning of the already trained word embedding matrix

Images