Language task model training method and device, electronic equipment and storage medium

Abstract

The invention provides a language task model training method and device, electronic equipment and a storage medium. The method comprises the steps of performing hierarchical pre-training in a languagemodel based on corpus samples of corresponding language tasks in a pre-training sample set; carrying out forward propagation on corpus samples corresponding to language tasks in a training sample setin the language task model; fixing parameters of the language model, and performing back propagation in the language task model to update the parameters of the task model; and performing forward propagation and reverse propagation on corpus samples corresponding to the language tasks in the training sample set in the language task model so as to update parameters of the language model and the task model. By means of the method and device, the catastrophic forgetting phenomenon of the language model can be prevented, and meanwhile it is guaranteed that the language model and the task model canachieve the training effect meeting the corresponding learning rate.

Description

technical field [0001] The present invention relates to artificial intelligence technology, in particular to an artificial intelligence-based language task model training method, device, electronic equipment and storage medium. Background technique [0002] Artificial Intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. [0003] Although various large-scale pre-trained language models in related technologies have strong context representation capabilities, they do not have rich interfaces for many specific tasks. For example, the application of language models to reading comprehension tasks simply puts the problem Splicing together with articles for training, the disadvantage of this training method is that the language model does not learn the a...

AI Technical Summary

Problems solved by technology

[0003] Although various large-scale pre-trained language models in related technologies have strong context representation capabilities, they do not have rich interfaces for many specific tasks. For example, the application of language models to reading comprehension tasks simply puts the problem Splicing together with articles for training, the disadvantage of this training method is that the language model does not learn the advanced features in a specific data set (such as domain-specific knowledge, interaction between questions and articles in reading comprehension)

Images