Meta-knowledge fine adjustment method and platform for multi-task language model

Abstract

The invention discloses a meta-knowledge fine tuning method and platform for a multi-task language model, and the method comprises the steps: highly-transferable common knowledge, i.e., meta-knowledge, on different data sets of the same kind of tasks are obtained based on cross-domain typicality score learning; and learning processes of similar tasks on different domains corresponding to differentdata sets are mutually associated and mutually enhanced, so that the fine adjustment effect of similar downstream tasks on different domain data sets in language model application is improved, and the parameter initialization capability and generalization capability of a universal language model of similar tasks are improved. According to the method, fine adjustment is carried out on a downstreamtask cross-domain data set, the effect of a compression model obtained through fine adjustment is not limited by a specific data set of the task, on the basis of a pre-trained language model, fine adjustment is carried out on a downstream task through a meta-knowledge fine adjustment network, and therefore a similar downstream task language model irrelevant to the data set is obtained.

Description

technical field [0001] The invention belongs to the field of language model compression, in particular to a meta-knowledge fine-tuning method and platform for multi-task language models. Background technique [0002] Large-scale pre-trained language model automatic compression technology has achieved significant effects in the application fields of natural language understanding and generation tasks; however, when facing downstream tasks in the field of smart cities, re-fine-tuning large models based on specific data sets is still the key to improving model compression. The key step of this is that the existing fine-tuning method for downstream task-oriented language models is to fine-tune on the specific data set of the downstream task, and the effect of the compression model obtained by training is limited to the specific data set of this type of task. Contents of the invention [0003] The purpose of the present invention is to provide a meta-knowledge fine-tuning metho...

AI Technical Summary

Problems solved by technology

[0002] Large-scale pre-trained language model automatic compression technology has achieved significant effects in the application fields of natural language understanding and generation tasks; however, when facing downstream tasks in the field of smart cities, re-fine-tuning large models based on specific data sets is still the key to improving model compression. The key step of the existing downstream task-oriented language model fine-tuning method is to perform fine-tuning on the specific data set of the downstream task, and the effect of the compression model obtained by training is limited by the specific data set of this type of task

Images