Fine tuning method and device for a language model, computing equipment and storage medium

Abstract

The invention discloses a fine tuning method and device for a language model, computing equipment and a storage medium, and the method comprises the steps: obtaining a pre-trained language model and phrases, wherein the phrases comprise discrete template cue words and discrete tag words; designing input data of a fine tuning language model, wherein the fine tuning input data comprises a text statement, a template cue word and a shielding token; and performing supervised learning of a shielding token prediction task on the language model according to the input data and the tag words to optimize model parameters of the language model. The difference between the pre-trained language model and the fine-tuned language model is bridged, so the fine-tuned language model is better in performance on a downstream shielding token prediction task; under the condition of full samples, compared with a traditional fine adjustment method, the method is better in effect, under the condition of few samples, the effect of the method is improved more remarkably, the calculation cost of a large number of parameters can be reduced, and the calculation overhead of calculation equipment is reduced.

Description

technical field [0001] The invention belongs to the technical field of natural language processing, and in particular relates to a fine-tuning method, device, computing device and storage medium of a language model. Background technique [0002] The pre-trained language model is a model trained on a large corpus data set, which is obtained by pre-training a large amount of corpus. Because the pre-trained language model has used a large amount of corpus for unsupervised learning, the knowledge in the corpus has been transferred into the embedding layer of the pre-trained language model. Fine-tuning is the main method to transfer pre-training model knowledge to downstream tasks, such as a meta-knowledge fine-tuning method and platform for multi-task language models disclosed in a patent application with a publication number of CN112100383A, and a patent application with a publication number of CN113032559A A disclosed language model fine-tuning method for text classification ...

AI Technical Summary

Problems solved by technology

However, this kind of fine-tuning method has the following defects: (1) its parameter efficiency is low: each downstream task has its own fine-tuning parameters; (2) the pre-training training target is different from the fine-tuning target, resulting in poor generalization ability of the pre-training model; (3) Compared with the pre-training stage, the network parameters are increased, and a large amount of data is required to learn the newly added parameters.

The traditional fine-tuning method adds new parameters and the training target is different from the fine-tuning target, which leads to many tasks with few samples. The traditional fine-tuning method based on the pre-training model is easy to overfit due to too few training data samples. The model may work well on the training set, but there is still a big gap between the test set and the case of using large samples. If the gap is large, it will lead to a small number of samples that cannot be used.

Images