Multi-modal large language model training method and system

By constructing a training set and utilizing positive-negative sample contrast learning and intra-group relative policy optimization algorithms, combined with reinforcement learning-based training methods for multimodal large language models, the challenges of knowledge updating, intermodal semantic understanding, and generation logic transparency in multimodal large language models are addressed. This approach enhances the ability to distinguish difficult samples and improves the interpretability of generated content, thereby increasing the reliability of the multimodal question answering system.

CN121660022AActive Publication Date: 2026-03-13XI AN JIAOTONG UNIV
8 Cites 2 Cited by

Patent Information

Application Number
CN202511847713.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-12-09
Publication Date
2026-03-13
Estimated Expiration
2045-12-09

AI Technical Summary

Technical Problem

Existing multimodal large language models face challenges in knowledge updating, intermodal semantic understanding, and transparency of generation logic. This leads to problems such as knowledge lag, fragmented intermodal semantic understanding, and difficulty in tracing generation logic when dealing with time-sensitive tasks. Furthermore, point-state ranking methods result in missorting of multimodal documents and weak interpretability.

Method used

A multimodal large language model training method is adopted. By constructing a training set and using positive and negative sample contrast learning and intra-group relative policy optimization algorithms, the model's ability to distinguish difficult samples is improved. Reinforcement learning is combined to enhance multimodal retrieval generation, thereby achieving fine-grained cross-modal association and interpretability enhancement.

Benefits of technology

It significantly improves the model's ability to distinguish difficult samples, enhances the interpretability of generated content and the credibility of retrieval results, provides a clear decision-making logic, and improves the technical reliability of the multimodal question-answering system.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

The invention discloses a multi-modal large language model training method and system, and relates to the technical field of multi-modal large model data processing, and the method comprises the following steps: training a first large model through querying a question set, a positive sample and a hard negative sample; inputting the plurality of test samples into the trained first large model, and generating a plurality of second answers based on a first preset prompt; sorting each test sample based on the plurality of second answers, and retrieving to obtain a previous candidate multi-modal document related to the test question; and training the second large model through the test problem and the corresponding previous multi-modal document. According to the method, the positive samples and the hard negative samples are jointly used for training, the model is forced to capture the fine-grained semantic boundary of correlation judgment through a contrast learning mechanism, the distinguishing capacity of the model for difficult samples is remarkably improved, and the mistaken arrangement phenomenon is avoided.
Need to check novelty before this filing date? Find Prior Art