Model training method and device, text processing method and device, electronic equipment and medium

Abstract

The invention provides a model training method, a text processing method and device, electronic equipment and a medium. The model training method comprises the following steps: taking a student model corresponding to an (a-1) th round of iteration as a teacher model corresponding to an a th round of iteration; according to the first training sample corresponding to the a-th iteration and the teacher model corresponding to the a-th iteration, training the basic model corresponding to the a-th iteration to obtain a student model corresponding to the a-th iteration, wherein in the training process, a loss function is determined according to a first probability distribution vector and a second probability distribution vector, the first probability distribution vector is a probability distribution vector obtained by inputting a first training sample corresponding to the a-th iteration into a basic model corresponding to the a-th iteration, the second probability distribution vector is a probability distribution vector obtained by inputting the first training sample corresponding to the a-th round of iteration into the teacher model corresponding to the a-th round of iteration; and outputting a corresponding student model under the condition that the convergence condition is met.

Description

technical field [0001] The embodiments of the present application relate to the technical field of data processing, and in particular, to a model training method, a text processing method and device, electronic equipment, and media. Background technique [0002] Deep learning models have greatly promoted the development of various natural language processing tasks, and training these models often requires a large number of labeled samples. However, labeled samples often only exist in languages ​​with large populations such as English and Chinese, and most languages ​​in the world often have only a few or no labeled samples. Cross-language transfer is expected to solve this problem by transferring task knowledge between languages, that is, using the labeled task data of the source language to let the model learn the corresponding task of the target language. [0003] The current mainstream cross-language migration solution is to fine-tune the multilingual pre-training langua...

AI Technical Summary

Problems solved by technology

[0009] Among them, the methods based on adversarial training proposed by Keung et al. and Zhang et al. are not as simple and direct as self-training in terms of using target language unlabeled samples, and their performance is also weaker than self-training.

However, in the current self-training process, the prediction accuracy of the teacher model on unlabeled samples is low, which misleads the training of the student model. That is to say, the current cross-language model training method has poor performance.

Images