Compression method and system for multi-language BERT sequence labeling model

Abstract

The invention provides a compression method and system for a multi-language BERT sequence labeling model, and relates to the technical field of knowledge distillation of BERT models, and the method comprises the steps: 1, extracting a word list from a multi-language corpus based on a Wordpiece algorithm; 2, pre-training the multi-language/single-language BERT teacher model and the multi-language BERT student model; 3, making fine adjustment on the multi-language/single-language BERT teacher model based on downstream task data marked manually; 4, performing residual knowledge distillation on the pre-trained multi-language BERT student model by utilizing the multi-language/single-language BERT teacher model; and 5, finely adjusting the distilled multi-language BERT student model based on manually labeled downstream task data. By residual learning and many-to-one knowledge distillation, the accuracy and generalization degree of the student model are improved, and hardware resources required for deploying the BERT type sequence labeling model in a multi-language environment are reduced.

Description

technical field [0001] The present invention relates to the technical field of knowledge distillation of BERT models, in particular, to a compression method and system for multilingual BERT sequence labeling models. Background technique [0002] BERT is a large-scale pre-trained language model based on Transformers encoder. In recent years, BERT has demonstrated strong strength in many downstream tasks. Sequence tagging is a class of tasks that classify elements in a sequence. Common sequence tagging tasks include named entity recognition, part-of-speech tagging, and so on. In a multilingual environment, if multiple monolingual BERT models are used to model different language texts at the same time, it will take up a lot of computing resources; at the same time, for some language categories with scarce training corpora, both BERT and traditional models are very difficult. Difficult to achieve good results. Multilingual BERT can simultaneously model hundreds of languages ​...

AI Technical Summary

Problems solved by technology

[0004] In view of the above existing technologies, there are the following technical defects. In related technologies, the knowledge distillation technology for the multilingual BERT sequence labeling model adopts a one-to-one training method, that is, distillation from the multilingual BERT teacher model to the multilingual BERT student model. This type of method does not take into account that the multilingual BERT model is not superior to the monolingual BERT model in all language categories; in addition, due to the large difference in structural complexity between the student model and the teacher model, a single student model Inability to efficiently fit the output of the teacher model

Images