Meta-knowledge fine adjustment method and platform based on domain invariant features

Abstract

The invention discloses a meta-knowledge fine adjustment method and platform based on domain invariant features, and the method comprises the steps: learning highly-transferable common knowledge, i.e., domain invariant features, on different data sets of similar tasks, and learning common domain features on different domains corresponding to different data sets of similar tasks in a fine adjustment network set; and any different domains can be quickly adapted. According to the method, the parameter initialization capability and generalization capability of the similar task universal language model are improved, and finally, the universal compression architecture of the similar downstream task language model is obtained through fine adjustment. In the meta-knowledge fine adjustment network,a domain invariant feature loss function is designed, and domain-independent general knowledge is learned, that is, a learning target of one domain invariant feature is minimized to drive a languagemodel to have domain invariant feature coding capability.

Description

technical field [0001] The invention belongs to the field of language model compression, in particular to a meta-knowledge fine-tuning method and platform based on domain-invariant features. Background technique [0002] Pretrained neural language models improve the performance of a variety of natural language processing tasks by fine-tuning on task-specific training sets. In the fine-tuning stage, the existing compression method of the pre-trained language model for downstream tasks is fine-tuned on the specific data set of the downstream task, and the effect of the trained compression model is limited to the specific data set of this type of task. Contents of the invention [0003] The purpose of the present invention is to provide a meta-knowledge fine-tuning method and platform based on domain-invariant features to address the deficiencies of the prior art. The present invention introduces meta-knowledge based on domain-invariant features, and learns common domain fea...

AI Technical Summary

Problems solved by technology

In the fine-tuning stage, the existing compression method of the pre-trained language model for downstream tasks is fine-tuned on the specific data set of the downstream task, and the effect of the compression model obtained by training is limited by the specific data set of this type of task

Images