Progressive fine tuning method and system for multi-modal pre-training model

By employing a progressive fine-tuning method, utilizing cross-modal adapter bridging and dynamic resource allocation, and combining a cross-modal contrastive consistency model, the training efficiency and generalization ability issues of multimodal pre-trained models on new domain data are addressed. This achieves efficient domain transfer and task adaptation, improving model performance and resource utilization efficiency.

CN121010981AActive Publication Date: 2025-11-25SICHUAN COOLBY COMM EQUIP CO LTD
9 Cites 5 Cited by

Patent Information

Application Number
CN202511543550.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-10-28
Publication Date
2025-11-25
Estimated Expiration
2045-10-28

AI Technical Summary

Technical Problem

Existing multimodal pre-trained models require retraining or fine-tuning a large number of parameters when faced with new domain data, which is time-consuming and labor-intensive and may cause the model to lose its original knowledge. Furthermore, they lack generalization ability when transferring between different domains.

Method used

A progressive fine-tuning approach is adopted, which involves inserting a cross-modal adapter bridging module and a low-rank decomposition structure, combined with a dynamic resource allocation strategy and a progressive unfreezing scheduler. By utilizing a cross-modal contrastive consistency model and a multi-objective composite loss function, the model achieves efficient domain transfer and task adaptation.

Benefits of technology

It significantly improves the training efficiency of the model, reduces GPU memory consumption, ensures efficient adaptation and semantic consistency of the model across different domains, and improves the expressive efficiency and generalization ability of the feature space.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

The invention discloses a progressive fine tuning method and system for a multi-modal pre-training model, and belongs to the field of deep learning, and the method comprises the steps: obtaining a high-dimensional visual feature vector from a visual encoder of a pre-trained multi-modal large model, obtaining a text feature vector from a text encoder, and carrying out the construction to obtain a heterogeneous modal feature; the contribution degrees of different modes are analyzed through a resource allocation strategy, the whole fine adjustment process is dynamically guided, and limited computing resources are allocated to a multi-mode large model component which contributes to the current task most; and processing the heterogeneous modal features through a cross-modal comparison consistency model to obtain a final optimization target. According to the invention, a larger batch can be trained or a larger batch size can be used for fine tuning training under a limited hardware condition.
Need to check novelty before this filing date? Find Prior Art