A meta-knowledge fine-tuning method and platform for multi-task language models

A language model and multi-task technology, applied in the direction of reasoning methods, semantic analysis, unstructured text data retrieval, etc., can solve problems such as the limited effect of compression models, improve parameter initialization capabilities and generalization capabilities, and improve fine-tuning effects , Improve the effect of compression efficiency

Active Publication Date: 2021-02-19
ZHEJIANG LAB
View PDF5 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0002] Large-scale pre-trained language model automatic compression technology has achieved significant effects in the application fields of natural language understanding and generation tasks; however, when facing downstream tasks in the field of smart cities, re-fine-tuning large models based on specific data sets is still the key to improving model compression. The key step of the existing downstream task-oriented language model fine-tuning method is to perform fine-tuning on the specific data set of the downstream task, and the effect of the compression model obtained by training is limited by the specific data set of this type of task

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A meta-knowledge fine-tuning method and platform for multi-task language models
  • A meta-knowledge fine-tuning method and platform for multi-task language models
  • A meta-knowledge fine-tuning method and platform for multi-task language models

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0029] Such as figure 1 As shown, the present invention is a multi-task language model-oriented meta-knowledge fine-tuning method and platform. On the downstream task multi-domain data set of the pre-trained language model, based on cross-domain typical score learning, the meta-knowledge of typical scores is used. -Knowledge fine-tunes downstream task scenarios, making it easier for meta-learners to fine-tune to any domain. The learned knowledge is highly generalizable and transferable, rather than limited to a specific domain. The resulting compression model The effect is suitable for data scenarios in different domains of the same task.

[0030] A meta-knowledge fine-tuning method oriented to a multi-task language model of the present invention, specifically comprising the following steps:

[0031] Step 1: Calculate the class prototypes of cross-domain data sets for similar tasks: Considering that multi-domain class prototypes can summarize the key semantic features of the ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a meta-knowledge fine-tuning method and platform for a multi-task language model. Based on cross-domain typical score learning, the method obtains highly transferable shared knowledge on different data sets of similar tasks, that is, meta-knowledge. The learning process of similar tasks on different domains corresponding to different data sets is correlated and mutually strengthened to improve the fine-tuning effect of similar downstream tasks in different domain data sets in the application of language models, and improve the parameter initialization of general language models for similar tasks ability and generalization ability. The present invention performs fine-tuning on the downstream task cross-domain data set, and the effect of the compression model obtained by fine-tuning is not limited to the specific data set of this type of task. Fine-tuning is performed to obtain similar downstream task language models that are independent of the dataset.

Description

technical field [0001] The invention belongs to the field of language model compression, in particular to a meta-knowledge fine-tuning method and platform for multi-task language models. Background technique [0002] Large-scale pre-trained language model automatic compression technology has achieved significant effects in the application fields of natural language understanding and generation tasks; however, when facing downstream tasks in the field of smart cities, re-fine-tuning large models based on specific data sets is still the key to improving model compression. The key step of this is that the existing fine-tuning method for downstream task-oriented language models is to fine-tune on the specific data set of the downstream task, and the effect of the compression model obtained by training is limited to the specific data set of this type of task. Contents of the invention [0003] The purpose of the present invention is to provide a meta-knowledge fine-tuning metho...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/35G06F40/30G06N5/04
CPCG06F16/355G06F40/30G06N5/04
Inventor 王宏升王恩平单海军
Owner ZHEJIANG LAB
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap