一种基于自监督学习的多模态预训练模型迁移方法

By employing a two-stage text prompting optimization framework based on self-supervised learning and a gradient soft matching strategy, the problem of insufficient open set capability of pre-trained models in downstream tasks is solved, achieving efficient and highly generalizable image recognition applicable to various visual skeletons and models.

CN118097685BActive Publication Date: 2026-07-17NANJING UNIV
View PDF 2 Cites 0 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
NANJING UNIV
Filing Date
2024-03-15
Publication Date
2026-07-17

Smart Images

  • Figure CN118097685B_ABST
    Figure CN118097685B_ABST
Patent Text Reader

Abstract

一种基于自监督学习的多模态预训练模型迁移方法，对于预训练好的视觉语言模型，通过一个两阶段文本提示优化框架对文本提示进行优化，提升视觉语言模型的开集能力，首先利用初始化后的文本提示通过有监督任务和自监督任务进行联合优化，然后再利用自监督任务进行二阶段优化，得到鲁棒且泛化性强的文本提示，从而将视觉语言模型迁移到下游图像识别任务中。本发明提出一种新颖且简洁的框架，设计鲁棒且泛化性强的文本提示，弥补预训练任务和下游任务间差异，缓解有监督学习过程中的过拟合问题，对任意的测试类别集合自适应，且无需利用额外知识库；能够有效且高效地将预训练多模态模型迁移到下游任务中，并增强其泛化性。
Need to check novelty before this filing date? Find Prior Art