Multi-language model compression method and device based on knowledge distillation

Abstract

The invention provides a multi-language model compression method and device based on knowledge distillation, electronic equipment and a storage medium, and belongs to the technical field of machine learning. The method comprises the following steps: taking trained N language models of different languages as teacher models; training a student model by utilizing the teacher model through a knowledgedistillation algorithm, wherein the vocabulary of the student model comprises all vocabularies in the teacher model. In the embodiment of the invention, the trained N language models of different languages are taken as teacher models; through the knowledge distillation algorithm, the teacher model is used for training the student model, so that compared with a multi-language model in the prior art, the student model obtained through final training has the advantages that model parameters are few, the structure of the model is simplified, and the performance and effect of the model can be guaranteed under the condition that the model parameters are few.

Description

technical field [0001] The present application relates to the technical field of machine learning, in particular, to a multilingual model compression method and device based on knowledge distillation, electronic equipment and storage media. Background technique [0002] In recent years, the dynamic word vector training model based on the language model has performed well in NLP (Natural Language Processing, Natural Language Processing) tasks. When multiple languages ​​are required in some scenarios, multilingual models need to be used. Currently, there are two ways to train multilingual models. The first one is to prepare a large amount of corpus in multiple languages ​​to form a large vocabulary, so that the model can be trained in one training process. to learn semantic representations for multiple languages. The second is to dynamically add a vocabulary of a new language given a model that has been trained in a certain language, map the vocabulary to the weight matrix of...

AI Technical Summary

Problems solved by technology

[0003] The purpose of the embodiments of the present application is to provide a multilingual model compression method and device based on knowledge distillation, electronic equipment and storage media, so as to improve the above-mentioned problem of "complex model structure and too many model parameters".

Images