Multi-modal large language model training method and system

Abstract

The invention discloses a multi-modal large language model training method and system, and relates to the technical field of multi-modal large model data processing, and the method comprises the following steps: training a first large model through querying a question set, a positive sample and a hard negative sample; inputting the plurality of test samples into the trained first large model, and generating a plurality of second answers based on a first preset prompt; sorting each test sample based on the plurality of second answers, and retrieving to obtain a previous candidate multi-modal document related to the test question; and training the second large model through the test problem and the corresponding previous multi-modal document. According to the method, the positive samples and the hard negative samples are jointly used for training, the model is forced to capture the fine-grained semantic boundary of correlation judgment through a contrast learning mechanism, the distinguishing capacity of the model for difficult samples is remarkably improved, and the mistaken arrangement phenomenon is avoided.

Description

Technical Field

[0001] This invention relates to the field of multimodal large model data processing technology, and in particular to a multimodal large language model training method and system. Background Technology

[0002] In recent years, artificial intelligence technologies, represented by Multi-modal Large Language Models (MLLMs), have made groundbreaking progress. By integrating multimodal data such as text, images, and videos, they have demonstrated outstanding capabilities in cross-modal semantic understanding and generation. However, existing generative models generally face three core challenges: First, current MLLMs generally adopt a pre-training and fine-tuning paradigm, and their knowledge systems are deeply dependent on the static datasets during the training phase. This makes it difficult to cover rapidly iterating emerging knowledge in the real world and lacks the ability to actively acquire external knowledge. As a result, the models cannot achieve dynamic knowledge updates through incremental learning. This knowledge solidification phenomenon leads to a "knowledge lag" problem when the model is dealing with time-sensitive tasks. For example, in the medical field, if the model is not updated with the latest treatment guidelines in a timely manner, it may produce incorrect diagnostic recommendations. Second, multimodal data has heterogeneous characteristics. The discrete symbolic representation of text is fundamentally different from the continuous visual features of images and videos, resulting in fragmentation in semantic understanding between modalities. While existing models attempt to address this issue using joint embedding spaces and cross-modal attention mechanisms, they still struggle to achieve accurate alignment in complex contexts. For example, when text descriptions contain abstract semantics such as metaphors and puns, the matching accuracy between visual information from the image modality and text semantics significantly decreases. Thirdly, current MLLMs, based on the Transformer architecture's self-regressive generation mechanism, essentially predict the next token through probability distributions; this black-box decision-making process makes the generation logic difficult to trace. Theoretically, while the Transformer's multi-head attention mechanism can capture complex semantic relationships, it lacks explicit modeling of the inference path, making it difficult for the model to demonstrate the contribution weights of each modality during multimodal fusion and lacking transparent interpretability of key generated content. These core challenges not only constrain the performance improvement of MLLMs but also limit their practical application in key domains.

[0003] To overcome these limitations, Retrieval-Augmented Generation (RAG) technology has emerged. This technology couples a large language model with a real-time updated external knowledge base by constructing a dynamic knowledge retrieval framework. It first retrieves relevant information from a large multimodal document collection and then uses this retrieved information to guide answer generation. Its core process includes three main modules: multimodal query parsing, cross-modal semantic retrieval, and evidence fusion reasoning. Retrieval-Augmented Generation methods often employ a dual-tower encoder architecture, where a visual encoder (such as Contrastive Language-Image Pre-training, CLIP) and a text encoder (such as Bidirectional Encoder Representations from Transformers, BERT) extract multimodal features respectively, perform cross-modal matching using cosine similarity, and finally combine the results... k The search results are input into the decoder to generate the answer. Although this method improves the accuracy of facts, it has two significant drawbacks: (1) the retrieval and generation processes are decoupled, which makes the core logic and optimization goals of the two lose their basis for coordination, resulting in insufficient alignment of the multimodal semantic space; (2) static feature representation (i.e., the feature form of data is fixed after one-time extraction of features and no longer dynamically adjusted with the task or context) cannot capture fine-grained cross-modal associations.

[0004] Recent research attempts to innovate traditional methods of multimodal document ranking by leveraging the end-to-end processing capabilities of MLLMs, specifically by applying large-scale multimodal language models to understand multimodal documents and perform point-to-point ranking. While these methods have improved retrieval accuracy, the following key issues remain to be addressed:

[0005] Limitations of Point-State Ranking: Methodologically, point-state ranking transforms the complex document ranking problem into a single-document binary classification or regression task based on a scoring function by assigning an independent relevance score to each query-document pair. This simplification strategy enables the algorithm to efficiently handle terabyte-scale document collections, supporting the infrastructure of modern search engines. Technically, it guides multimodal large language models (MLLMs) to output Boolean probability scores through preset templates such as "Is this document relevant to the question?". This direct mapping provides a standardized path for rapidly building retrieval systems. However, with the explosive growth of multimodal data and the increasing demand for real-time interaction, the inherent defects of point-state ranking have become increasingly apparent. The primary challenge stems from the limitations of the model training mechanism: In supervised learning, this method treats each document as an independent entity, focusing only on absolute relevance scores and completely ignoring the relative order information between documents. This training paradigm results in a severe deficiency in the model's ability to distinguish difficult samples (i.e., documents with similar relevance scores), especially when facing semantically ambiguous queries, easily leading to convergence of scores between high-quality and low-quality documents. Although difficult negative sample mining techniques can enhance the model's ability to identify boundary samples to some extent, they still cannot avoid misclassification because they fail to fundamentally solve the problem of missing ranking relationship modeling.

[0006] Weak interpretability mechanisms: The current system only presents the final search list and generated answers, lacking explicit modeling of "why specific documents were selected" and "how evidence-based reasoning is used." This leads to two key problems: First, it is difficult to verify the rationality of the decision-making logic. The lack of a decision-making logic verification mechanism severely weakens the credibility of the system output. In academic research, users cannot trace the logical connection between search results and research questions, making it difficult to determine whether the system follows academic norms in selecting literature. Second, it is difficult to quickly locate the failure point when errors occur. When the generated answers contain factual errors or the search list deviates from user expectations, due to the lack of explicit documentation of search strategies, evidence selection rules, and reasoning steps, developers find it difficult to locate the root cause of the problem. Whether it is a semantic understanding bias in the search model, parameter inaccuracies in the evidence matching algorithm, or logical loopholes in the reasoning process, none of these can be effectively traced through the system output.

[0007] In summary, existing point-based ranking methods suffer from multimodal document missorting, and the final retrieval lists and generated answers output by large multimodal language models have low reliability and lack effective source tracing. Summary of the Invention

[0008] Based on the shortcomings of the existing technology, the present invention provides a multimodal large language model training method and system, which solves the problems of multimodal document missorting caused by the existing point-state sorting training method, as well as the low reliability and inability to effectively trace the source of the final retrieval list and generated answers output by the multimodal large language model.

[0009] The present invention adopts the following technical solution: In a first aspect, the present invention provides a method for training a multimodal large language model, wherein the multimodal large language model includes a first large model and a second large model, and the training method includes the following steps: Construct a training set, which includes a query question set, a test question set, a candidate multimodal document set, a target multimodal document set related to the query question set, and a target answer set for the test question set; The target multimodal document set is used as positive samples, the complement of the target multimodal document set in the candidate multimodal document set is used as negative samples, and the negative samples with the highest semantic similarity to the query question are used as hard negative samples. The first model is trained using the query question set, positive samples, and hard negative samples. During training, for each sample, the first model generates multiple first answers based on a first preset prompt. Based on the multiple first answers, the model parameters of the first model are corrected using an intra-group relative strategy optimization algorithm. The first preset prompt is the probability of yes or no. The test questions in the test question set are reconstructed into multiple test samples by combining them with multiple candidate multimodal documents in the candidate multimodal document set. These test samples are then input into the trained first large model, which generates multiple second answers based on a first preset prompt. Each test sample is then ranked based on these second answers, and the top responses relevant to the test question are retrieved. One candidate multimodal document; By testing the questions and corresponding previous ones The second model is trained using a multimodal document. The second model generates multiple third answers based on a second preset prompt. Based on the multiple third answers, the model parameters of the second model are corrected through an intra-group relative strategy optimization algorithm. The second preset prompt consists of the generated document ID, thought process, and target answer.

[0010] Preferably, the step of correcting the model parameters of the first large model based on multiple first answers using an intra-group relative strategy optimization algorithm specifically includes the following steps: For each query question, perform diversified sampling to generate a set containing different first answers; The reward value for each first answer is calculated using a first-format reward function for structured evaluation and a relevance-based reward function for measuring the relevance of the answer to the query question. The relative advantage of the current first answer is obtained by taking the mean and variance of the reward values ​​of multiple first answers; the relative advantage is used to characterize the quality difference of the answer compared with other candidate results in the same group. Based on the aforementioned relative advantages, a first objective function is constructed, and the parameters of the model are adjusted based on the objective function.

[0011] Preferably, the first format reward function is as follows: ; The specific reward function for correlation determination is as follows: ; The specific reward values ​​are as follows: ; In the formula, For the first format reward function, The reward function is for correlation assessment. As a reward value, For the first i The first answer.

[0012] Preferably, the process of sorting each test sample based on multiple second answers and retrieving the top results related to the test question is described. One candidate multimodal document; specifically including the following steps: Calculate multiple similarities for multiple test samples based on multiple second answers; The point-state correlation evaluation function for each test sample is calculated based on multiple similarities. The point-state correlation evaluation functions of multiple test samples are sorted to obtain the top-ranked functions most relevant to the test problem. 1 candidate multimodal document.

[0013] Preferably, the point-state correlation evaluation function is as follows: ; In the formula, This is a point-state correlation evaluation function. For the first i One test question, In order to be with the first i The first test question corresponds to the first j Candidate multimodal documents No. s The weight of the second answer, L The number of second answers, For the first s The similarity between the second answer and the second answer.

[0014] Preferably, the third answer includes an ID list, a thought process, and a final answer, all of which are in their corresponding tag patterns. The step of correcting the model parameters of the second model based on multiple third answers using an intra-group relative strategy optimization algorithm specifically includes the following steps: The coarse match reward and the exact match reward are obtained based on the list of IDs of the third answer and the list of correct IDs; Obtain the second-format reward function based on the tag pattern of the third answer; A correctness reward function is derived based on the final answer and the target answer from the third answer. The comprehensive reward function is calculated by combining the coarse matching reward, exact matching reward, format reward function, and accuracy reward function. The relative advantage of the current third answer is obtained by taking the mean and variance of the combined reward function of multiple first answers; A second objective function is constructed based on the aforementioned relative advantages, and the parameters of the model are adjusted based on the objective function.

[0015] Preferably, the coarse matching reward and the precise matching reward are as follows: ; ; In the formula, For coarse matching rewards, This is a list of IDs for the third answer. For the correct ID list, The number of IDs in the third answer list. The number of correct IDs in the list. To ensure accurate matching of rewards; The second format reward function is shown below: ; In the formula, For the second format reward function, For the first i The third answer, There are three tag modes; The accuracy reward function is as follows: ; In the formula, For the accuracy reward function, Indicates the first The standard answer to this question.

[0016] Secondly, the present invention provides a multimodal large language model training system, wherein the multimodal large language model includes a first large model and a second large model, and the training system includes: A construction module is used to construct a training set, which includes a query question set, a test question set, a candidate multimodal document set, a target multimodal document set related to the query question set, and a target answer set for the test question set. The first training module is used to take the target multimodal document set as positive samples, the complement of the target multimodal document set in the candidate multimodal document set as negative samples, and the sample with the highest semantic similarity to the query question among the negative samples as hard negative samples. The first large model is trained by using the query question set, positive samples, and hard negative samples. During the training process, for each sample, the first large model generates multiple first answers based on a first preset prompt. Based on the multiple first answers, the model parameters of the first large model are corrected by an intra-group relative strategy optimization algorithm. The first preset prompt is the probability of yes or no. The ranking module reconstructs test questions from the test question set and candidate multimodal documents from the candidate multimodal document set into multiple test samples. These test samples are then input into the trained first large model, which generates multiple second answers based on a first preset prompt. Each test sample is ranked based on these second answers, and the top-ranked responses relevant to the test question are retrieved. One candidate multimodal document; The second training module is used to compare test questions with corresponding previous training modules. The second model is trained using a multimodal document. The second model generates multiple third answers based on a second preset prompt. Based on the multiple third answers, the model parameters of the second model are corrected through an intra-group relative strategy optimization algorithm. The second preset prompt consists of the generated document ID, thought process, and target answer.

[0017] Compared with the prior art, the above-mentioned at least one technical solution adopted by the present invention can achieve the following beneficial effects: This invention first uses the target multimodal document set as positive samples, and the complement of the target multimodal document set in the candidate multimodal document set as negative samples. The negative samples with the highest semantic similarity to the query question are then used as hard negative samples. By using both positive and hard negative samples for training, the aim is to force the model to capture the fine-grained semantic boundaries of relevance judgment through a contrastive learning mechanism, significantly improving the model's ability to distinguish difficult samples and avoiding misclassification. A group relative strategy optimization algorithm is used to guide the model to correct the first major model parameters based on the training data. Multiple test samples are input into the trained first major model, and multiple second answers are generated based on a first preset prompt. Each test sample is then sorted based on these second answers, and the top results related to the test question are retrieved. The present invention employs a coarse-grained point-based multimodal document ranking method to initially screen documents relevant to the target task.

[0018] Finally, the test questions and corresponding previous questions were compared. A second-level model is trained using multiple multimodal documents. This second-level model generates multiple third-level responses based on a second set of pre-defined prompts. Based on these third-level responses, the model parameters are adjusted using an intra-group relative strategy optimization algorithm. This invention provides a relatively clear logical structure to the output of multimodal retrieval enhancement generation tasks, enabling effective traceability and enhancing the interpretability of the generated content to a certain extent. It also makes the model's decision-making process clearer and provides a new approach to improving technical reliability. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of the reinforcement learning-based multimodal retrieval-enhancement framework designed for this invention; Figure 2 This is a specific example diagram of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Example 1 Reference Figure 1 This invention proposes a training method for a multimodal large language model. The multimodal large language model (i.e., a multimodal large model) includes a first large model and a second large model, and specifically includes the following steps: S1: Construct a training set, which includes a set of query questions, a set of test questions, a set of candidate multimodal documents, a set of target multimodal documents related to the set of query questions, and a set of target answers for the set of test questions.

[0023] In this embodiment, the candidate multimodal document set is the document set of the external knowledge base.

[0024] The original data was processed using a generative autoregressive model. Convert to the data format required for enhanced fine-tuning: (1); In the formula, For a set of questions, including a set of query questions. and test question set , N To query the number of questions, For the first i A query question, For the first i One test question, m To determine the number of test questions; For the candidate multimodal document set, i.e. , The number of candidate multimodal documents. For the first j One candidate multimodal document; for The target multimodal document set related to the query question, i.e. , In order to be with the first i The first query related to the question j A multimodal document targeting a specific object. This is the standard answer to the test question.

[0025] S2: Train the first large model by querying the question set, positive samples, and hard negative samples.

[0026] This invention proposes to guide the first large model to perform coarse-grained point-by-point multimodal document ranking through a reinforcement fine-tuning method. This process includes two core steps: reinforcement fine-tuning training and point-by-point probabilistic inference.

[0027] The purpose of enhancing fine-tuning training is to give the first model a stronger semantic understanding ability, enabling it to effectively distinguish the relevance of different modal documents to the query question, thereby optimizing the coarse-grained point-by-point multimodal document ranking effect.

[0028] Specifically, the set of query questions A collection of target multimodal documents related to the query question. The system uses a hard negative sample as the overall query input to the first large model, and generates a "yes / no" judgment based on a first preset prompt. The first preset prompt represents the probability of yes or no.

[0029] Target multimodal document collection related to the query question As a positive sample, the label is set to " The complement of the target multimodal document set that is relevant to the query question in the candidate multimodal document set is taken as a negative sample and labeled "No". The negative sample with the highest semantic similarity to the query question is defined as a hard negative sample. By using both positive and hard negative samples for training, the model is forced to capture fine-grained semantic boundaries for relevance judgment through a contrastive learning mechanism, which significantly improves the model's ability to distinguish difficult samples, especially suitable for ambiguous regions of visual-text semantic alignment in multimodal data. Subsequently, the group relative policy optimization algorithm (GRPO) is used to guide the model to correct the first major model parameter based on the training data.

[0030] In the field of information retrieval, ranking consistency modeling is one of the core challenges in improving the retrieval quality of multimodal large language models. Traditional pointwise ranking methods (such as Pointwise Ranking) focus only on the absolute relevance score of a single document, ignoring the relative order relationship between documents, which easily leads to semantic redundancy and logical contradictions in the retrieval results. The GRPO algorithm, based on the reinforcement learning paradigm, constructs a systematic ranking optimization solution with dynamic policy optimization at its core. During training, for each input problem... The algorithm first relies on the old model strategy. Perform diversified sampling to generate a sample containing G A collection of different first answers This multi-response sampling mechanism breaks the limitations of traditional single-path decision-making, providing a rich candidate space for subsequent comparative optimization. In the reward calculation stage, GRPO designed a dual-dimensional reward function system, including a formatted reward function. Relationship Judgment Reward Function Formatted reward function From a fundamental perspective, including grammatical correctness and formatting integrity, the first answer... Conduct a structured evaluation to ensure the generated content meets the format requirements of the business scenario. Relevance assessment reward function. It measures the relevance between the answer and the input sample from core dimensions such as semantic relevance and information matching. Through this dual-track parallel reward calculation mechanism, each first answer... All can obtain precisely quantified reward values. This provides crucial quantitative evidence for subsequent intra-group comparative learning. This mechanism not only achieves multi-dimensional evaluation of response quality but also effectively captures the relative merits of samples through comparison within groups, laying a solid quantitative foundation for ranking consistency modeling. The reward evaluation method is defined as follows:

[0031] (2); (3); (4); Specifically, GRPO focuses on the relative performance of the answer rather than the absolute reward value, by calculating... Different first answers The mean and variance of the rewards are used to gain a relative advantage for the first answer. Defined as: (5); Based on these relative advantages, GRPO aims to optimize the current model. The strategy updates the model parameters by maximizing the expected cumulative reward. The model parameters include parameters for the modal feature extraction layer, the cross-modal fusion layer, and the generation layer.

[0032] The first objective function of point-by-point sorting optimization The formula is as follows: (6); (7); in, (8); In the formula, For mathematical expectation, For input data, The probability distribution of the input data. For output, For the current strategy Compared to the old strategy The probability ratio at the output. For the clipping function, Let KL divergence be the KL divergence. This is a strategy for reference.

[0033] here, It is a hyperparameter related to clipping introduced in PPO (Proximal Policy Optimization). Exceed or below At this point, the objective function prunes the policy to a threshold boundary, thus limiting the magnitude of policy changes in a single update. This mechanism effectively avoids the problem of severe policy oscillations that may occur in traditional policy gradient algorithms (such as REINFORCE), and can significantly reduce variance during training, especially when dealing with high-dimensional action spaces (such as natural language generation) or complex reward functions. It is another hyperparameter used to control the strength of KL divergence regularization, when When the value is large, the algorithm imposes a strict penalty on the KL divergence between the old and new policies, forcing the new policy to... Closely follow the reference strategy Local optimization (usually of the policy updated in the previous round) is crucial when dealing with non-stationary environments (such as multi-agent interaction scenarios) or tasks with low sample efficiency. This is to prevent the policy from deviating excessively from the reference policy. This is crucial to ensuring the stability and effectiveness of the strategy update process.

[0034] S3: Reconstruct multiple test samples by combining the test questions in the test question set with multiple candidate multimodal documents in the candidate multimodal document set. Input these multiple test samples into the trained first large model and generate multiple second answers based on the first preset prompt. Sort each test sample based on the multiple second answers and retrieve the top results related to the test question. 1 candidate multimodal document.

[0035] After enhanced fine-tuning during training, the parameters of the first major language model were optimized, enabling it to better handle multimodal information. In the testing phase, this invention employs a point-state probabilistic inference mechanism to accurately evaluate the similarity between candidate multimodal documents and the test question. Specifically, based on the trained and optimized first major model, for the test question... With the corresponding candidate multimodal documents The input pairs are processed by the model, which outputs a preliminary judgment in the form of a "yes / no" binary answer. This judgment process essentially involves the model using deep semantic understanding of multimodal information such as text, images, and audio to uncover potential semantic relationships and feature matching between the input pairs. Subsequently, based on the model's output "yes / no" answer, and combined with probability calculation methods, the discrete binary judgment is transformed into a continuous similarity probability value. This probability value quantifies the degree of similarity between the target multimodal document and the query question in multiple dimensions such as semantics, structure, and sentiment, thus providing a reliable basis for subsequent document filtering and ranking, effectively improving the accuracy and efficiency of the multimodal question answering system. Specific steps:

[0036] Test questions and candidate multimodal document list The first in j Candidate multimodal documents Reconstructed into test samples , The number of candidate multimodal documents corresponding to the test query; the test samples are input into the first large model. The probability of the model generating "yes" or "no". Used to evaluate the similarity between test questions and candidate multimodal documents. Meanwhile, considering the randomness of the output of large models, the model is generated based on sampling from the current test problem. The second answer was used, and an adaptive probabilistic aggregation strategy was employed to comprehensively evaluate the similarity between the test query and the candidate multimodal documents. Therefore, this invention establishes a point-state correlation evaluation function between the test question and the multimodal document. The definition is as follows:

[0037] (9); (10); (11); In the formula, The weight of the s-th second answer.

[0038] Accordingly, point-state relevance evaluation functions are used for coarse-grained multimodal document ranking. Specifically, a test problem is used. With candidate multimodal document list Each document The relevance scores between the queries are sorted. After sorting the scores in descending order, the top queries most relevant to the query are selected. A collection of multimodal documents The definition is as follows:

[0039] (12); (13); In the formula, For point-state sorting strategy, This is a sorting function.

[0040] S4: By testing the questions and corresponding previous... The second-largest model is trained using a multimodal document.

[0041] Large models often rely on data that lacks timeliness. For example, GPT-4's knowledge base only covers information up to October 2023, and it cannot encompass new industry regulations, technological breakthroughs, or other emerging knowledge. This makes it difficult for large models to provide accurate and relevant answers when dealing with questions requiring the latest information.

[0042] In terms of accuracy, large models are prone to generating "illusory information," i.e., false content. Since their training data comes from public networks, it may inherently contain erroneous information, and large models inevitably incorporate these errors during the learning process.

[0043] To overcome the aforementioned shortcomings and achieve multimodal thinking chain (MIMCOT) reasoning during the multimodal retrieval enhancement generation process, this invention applies reinforcement learning to multimodal retrieval enhancement generation, guiding the second major model to achieve self-evolution through reinforcement fine-tuning. To overcome the limitation of single-point-state ranking in discriminating against difficult samples, this invention first performs coarse-grained document retrieval through point-state probabilistic reasoning, then performs fine-grained list-style multimodal document ranking on the retrieval results, and finally generates corresponding answers based on the retrieval information.

[0044] Fine-grained list sorting and answering aim to combine query questions with the top results obtained from point-state sorting. k The second model is fed with relevant documents and guided to generate the most relevant document IDs and answers to the query questions based on the relevant reference documents. In recent years, reinforcement learning (RL) has shown great potential in improving the reasoning capabilities of large language models (LLMs) and has achieved a series of breakthroughs. As the first LLM trained through large-scale reinforcement learning, OpenAI-O1 has broken performance limits in multiple benchmark tests, highlighting the significant improvement effect of reinforcement learning on the reasoning capabilities of language models. Similarly, the research of Deepseek-R1-Zero further demonstrates that a pure reinforcement learning framework can effectively stimulate the inherent reasoning potential of LLMs even without supervised training data. This method combines the Group Relative Policy Optimization (GRPO) algorithm with a rule-based reward mechanism to achieve iterative improvement of cognitive processes, dynamic exploration of the policy space, and error recognition based on Chair of Thought (CoT) prompts, providing a new paradigm for the autonomous evolution of unsupervised reasoning capabilities.

[0045] Building upon this foundation, researchers have explored multi-dimensional optimizations of reinforcement learning in LLM applications. For example, by extending long context modeling and improving policy optimization algorithms, the stability and generalization ability of the model in complex reasoning tasks have been further enhanced, laying a methodological foundation for the deep integration of reinforcement learning and large-scale language models. It is worth noting that although existing research has achieved significant results in pure text reasoning, the application potential of reinforcement learning in multimodal reasoning scenarios remains to be explored. Emerging research attempts to extend reinforcement fine-tuning techniques to cross-modal tasks, such as mathematical reasoning, few-shot classification, and open-vocabulary object detection. These works have initially demonstrated the unique advantages of reinforcement learning in fusing multimodal information by integrating large-scale visual language models. However, existing research has not yet fully focused on reinforcement learning-based multimodal retrieval enhancement generation mechanisms. Inspired by the above research, this invention proposes to guide a large multimodal language model to generate long thought chains for reasoning through reinforcement fine-tuning, and to jointly optimize list-based multimodal document ranking and question-based answering in question-answering sessions based on multi-source reference documents.

[0046] Fine-grained list-level sorting aims to combine queries with top results obtained through point-level sorting. The relevant documents are input into the second multimodal large language model, which is then guided to directly output the ID of the most relevant document to the query. Specifically, the test question... as well as The number of each candidate document and content The overall query is input into the model, guiding the second main model to first perform chained reasoning based on a second pre-set hint, and then answer which document numbers are most relevant. Correspondingly, each third answer will be based on the output list of IDs. With the correct ID list The matching reward function between them obtains the corresponding reward value, including the coarse matching reward. and exact match rewards Coarse matching reward It can be defined as follows:

[0047] (14); in, Represents a set and The number of elements in the intersection. This reward function evaluates the matching quality from two dimensions: : Measure the accuracy of the output list, i.e. the percentage of correct hits. Recall measures the proportion of the output list that covers the target IDs. The sum of these two metrics forms a comprehensive match metric that balances the impact of precision and recall.

[0048] The design of the rough matching reward can be further explained as follows: (15); Among them, TP This represents the number of true positives. The formula is equivalent to:

[0049] (16); When the length of the output list Length of the target list When they are equal, the reward function simplifies to: (17); This design ensures that the reward function can effectively evaluate the matching quality regardless of the list length.

[0050] Exact match reward The definition is as follows: (18); This reward function evaluates the case of a perfect match, awarding a reward only when the output list exactly matches the target list. Based on this reward mechanism, the ranking function for multimodal documents is defined as follows:

[0051] (19); in, For ranking functions, Indicates the preceding A multimodal document, This indicates a user query; MLLM stands for Multi-modal Large Language Model. This represents the sorted collection of documents generated by the model.

[0052] Regarding output format constraints, this invention draws inspiration from the structured output design of DeepSeek-R1, guiding the model to generate structured answers through a second preset prompt template. Specifically, the second major model output is explicitly divided into three logical modules within tags. The model must output a complete reasoning chain, including an understanding of the query intent, the basis for judging document relevance, and the logical steps for answer generation. A multi-dimensional reward mechanism is designed to evaluate the generation quality. This mechanism forces the model to encapsulate the thought process, relevant documents, and final answer within specific HTML tags, thereby achieving interpretability and modular evaluation.

[0053] Tags are used to output the thought process. Tags are used to output the IDs of the filtered documents, ensuring that the search results are traceable; Tags are used to generate the final answer, such as Figure 2 As shown. This structured output mechanism not only improves the interpretability of the model's answers but also provides a standardized interface for subsequent reward calculation and performance evaluation. Each third answer will receive a corresponding reward value based on the second-format reward function and the accuracy reward function. The reward evaluation method is defined as follows:

[0054] (20); This indicates three tag modes: There exists one and only one pair. <think> ...< / think> , There exists one and only one pair. <id> ...< / id> , There exists one and only one pair. <ans> ...< / ans> .

[0055] By explicitly specifying in the prompts, the output of the second major model must adhere to the following format specifications: Thought Process: In <think>Between tags, related document ID: <id>Between the tags, the final answer: In <ans>Between tags. Each generated output The evaluation will be conducted using reward functions across the following four dimensions:

[0056] (twenty one); This reward function ensures that the model output strictly follows the preset structured format.

[0057] (twenty two); in, Indicates the first The standard answer to each question. This reward directly assesses the correctness of the final answer.

[0058] Combining the two matching rewards defined in step S4, the total reward for a single output is the sum of the rewards across the four dimensions: (twenty three); The design allows for: a maximum reward of 1+1+2+1=5 (when all conditions are perfectly met), and a minimum reward of 0 (when all conditions are not met).

[0059] Correspondingly, similar to point-level sorting, fine-grained sorting, and the second objective function for answer optimization. It can be represented as: (twenty four); (25); (26); Accordingly, the answer generation function can be defined as: (27).

[0060] Example 2 Referring to Tables 1, 2, and 3, experiments were conducted on two datasets: WebQA (Web Question Answering) and MultimodalQA (Multimodal Question Answering). WebQA (Chang et al., 2021) is a multimodal, multi-hop question answering dataset designed to evaluate a model's ability to perform complex cross-modal reasoning and generation tasks. Each question requires 1-2 images or 1-2 text fragments for multi-step reasoning; the question is input in plain text, and the answer is generated as a free-form sentence. Each question in this dataset contains approximately 40 multimodal distractors (hard negative samples), requiring the model to retrieve relevant evidence from a mixed pool of correct cues and distractors.

[0061] During the experiment, this invention noticed that existing multimodal retrieval augmentation datasets suffer from labeling errors and uneven class distribution. Therefore, this invention constructed the Mini-MMRAG (Multi-Mode Retrieval-Augmented Generation) dataset. This dataset contains 5000 high-quality question-and-answer samples with multimodal references. This invention collected 35,000 and 17,000 question-and-answer data with references from WebQA and Multimodal-QA, respectively. To remove low-quality data during model training, this invention adopted two steps: First, question-and-answer pairs with obvious grammatical errors or containing difficult-to-recognize characters were excluded; second, this invention used a model fully trained on the existing MMRAG dataset to generate answers for each question and evaluated the clarity and representativeness of the samples based on the accuracy of the answers. This invention retained correctly evaluated questions to ensure that the reinforcement learning training process was not negatively affected by mislabeled samples. After filtering, 15,000 samples were retained. The filtered question-and-answer samples still suffer from class imbalance. This invention analyzed the number of question-answer pairs across various categories, including color, shape, true / false, number, and multiple-choice questions, and removed samples from severely biased categories. After category balancing, a final sample size of 5000 was retained.

[0062] The evaluation uses a dual metric: F1 score: measures the accuracy of multimodal cue retrieval. Using only the F1 score may cause the model to focus too much on retrieval and neglect answer generation (e.g., the retrieval is correct but the answer is semantically confusing). (28); in, Precision Recall (29); In the formula, TP represents a true positive, FP represents a false positive, and FN represents a false negative.

[0063] QA score: Combining fluency (QA-FL) and keyword accuracy (QA-Acc) based on BARTScore, the product of the two is used to quantify the quality of the answer. However, using QA score alone may mask retrieval defects (such as randomly generating fluent but irrelevant answers).

[0064] (30); in, For the degree of flow, This represents the generated answer text.

[0065] MultimodalQA (Talmor et al., 2021) contains multimodal question-answering pairs and aims to evaluate the multi-hop reasoning capabilities of the top two models across tables, text, and images. The dataset contains 16 manually labeled question types, 13 of which require cross-modal retrieval and reasoning. Each query typically requires one image and / or one text fragment to answer, along with approximately 20 visual or text distractors. Performance is evaluated using the following metrics: Exact Match (EM): measures the proportion of times the model's answer exactly matches the ground truth answer; and average word-level F1 score. It calculates the semantic match between predicted and actual answers at the lexical level, reflecting the completeness and accuracy of the answers.

[0066] (31); Where N is the total number of samples. For the first i one sample Scores. For WebQA and MultimodalQA, the model first retrieves a list of candidate clues, then identifies relevant evidence from multimodal sources to form a coherent answer. Both the retrieval module and the question-answering model use Qwen-2.5-VL-7B as the backbone architecture. During training, this invention utilizes Group Relative Policy Optimization (GRPO) and Low Rank Adaptive (LoRA) methods for efficient parameter fine-tuning. Experiments were conducted in parallel on eight NVIDIA H800-80GB GPUs. Key training configurations included setting the number of generation attempts to 4, limiting the input sequence length to 1024 tokens, and applying gradient pruning with a maximum threshold of 0.28 to mitigate optimization instability.

[0067] To highlight the advantages of this invention, it first introduces the advanced model architecture ETME (Enhanced Temporal Memory Encoder), which is the main model used in the experiments. ETME is a multimodal, multi-hop question-answering method based on Large Language Models (LLMIs). Its core is to transform the question-answering process into a joint task of implication tree generation and answer generation, and to improve model performance through a hybrid expert multi-task learning framework and iterative feedback mechanism. ETME transforms the question-answering task into the construction of implication trees and answer generation, and improves the interpretability and accuracy of reasoning by organizing information through logical structures. Fact base construction: Multi-hop problems are decomposed into sub-problems using LLM (such as GPT-3.5), and evidence from different modalities is processed separately.

[0068] In the WebQA benchmark tests shown in Table 2, the innovative method proposed in this invention has achieved breakthrough progress in key QA score metrics. Compared to the ETME model, which previously held the state-of-the-art (SOTA) position in this field, this method achieves... Significant improvements were achieved in both QA-FL (Question Fluency) and QA-Acc (Keyword Accuracy). and .

[0069] Table 3 presents the experimental results of MultimodalQA. For text-based problems, the method of this invention achieves a significant improvement; for image-based problems, it performs excellently overall. Specifically, the method of this invention outperforms the state-of-the-art (SOTA) model RAMQA in terms of exact match rate (EM). .

[0070] In the research and application of multimodal large models, the accuracy and efficiency of data alignment have always been key bottlenecks restricting the improvement of model performance. The experimental data above fully verify the significant effectiveness of the innovative method proposed in this invention in overcoming this bottleneck.

[0071] This invention employs a phased processing strategy. First, it utilizes a point-state probabilistic reasoning mechanism for coarse-grained document retrieval. Based on probability and statistics theory, this mechanism rapidly identifies a set of candidate documents relevant to the query by extracting and analyzing features from massive amounts of multimodal document data, thus significantly narrowing the scope of subsequent processing and greatly improving retrieval efficiency. Building upon this coarse screening, a fine-grained list-based multimodal document ranking method is further applied. This method integrates feature representations of multimodal information such as images and text. By constructing a complex cross-attention mechanism and ranking model, it deeply evaluates candidate documents from multiple dimensions, including semantics, structure, and sentiment, achieving accurate alignment between multimodal data. This two-stage processing approach, from coarse to fine, not only effectively reduces computational complexity but also significantly improves the accuracy and recall of multimodal data alignment, providing a novel technical path for enhancing the performance of large-scale multimodal retrieval models.

[0072] Furthermore, this invention innovatively visualizes the document selection process by enhancing fine-tuning techniques. During model training, specific loss functions and optimization algorithms are designed to enable the model to learn key decision-making criteria and logical relationships when selecting documents. Finally, visualization technology presents these learned selection logics in an intuitive and easy-to-understand way, providing interpretability for the model's decision-making process and offering important reference for subsequent model optimization and improvement.

[0073] The invention not only significantly improves the alignment effect of multimodal data, but also provides new research ideas and methods for the field of multimodal large model retrieval enhancement, which has important value and significance in both theoretical research and practical application.

[0074] Table 1. Baseline Dataset Statistics Table 2 WebQA Official Test Set Results Table 3. Test results of the multimodal question answering dataset. In Tables 2 and 3, Retr (Retrieval) is a metric used in large language models to measure retrieval performance. VLP includes VLP + VinVL, MuRAG, SKURG, PERQA, RAMQA, Q-only, AutoRouting, and Solar, all of which are common models in this field.

[0075] Based on the same concept, the present invention also provides a multimodal large language model training system, including a construction module, a first training module, a sorting module, and a second training module.

[0076] The building module is used to build the training set, which includes a set of query questions, a set of test questions, a set of candidate multimodal documents, a set of target multimodal documents related to the set of query questions, and a set of target answers for the set of test questions.

[0077] The first training module uses the target multimodal document set as positive samples, the complement of the target multimodal document set in the candidate multimodal document set as negative samples, and the sample with the highest semantic similarity to the query question among the negative samples as hard negative samples. The first large model is trained using the query question set, positive samples, and hard negative samples. During the training process, for each sample, the first large model generates multiple first answers based on a first preset prompt. Based on the multiple first answers, the model parameters of the first large model are corrected through an intra-group relative strategy optimization algorithm. The first preset prompt is the probability of yes or no.

[0078] The ranking module reconstructs test questions from the test question set and candidate multimodal documents from the candidate multimodal document set into multiple test samples. These test samples are then input into the trained first model, which generates multiple second answers based on a first preset prompt. Each test sample is ranked based on these second answers, and the top-ranked responses relevant to the test question are retrieved. 1 candidate multimodal document.

[0079] The second training module is used to test questions and their corresponding predecessors. The second model is trained using a multimodal document. The second model generates multiple third answers based on a second preset prompt. Based on the multiple third answers, the model parameters of the second model are corrected through an intra-group relative strategy optimization algorithm. The second preset prompt consists of the generated document ID, thought process, and target answer.

[0080] This invention applies reinforcement learning to a multimodal retrieval enhancement generation system, exploring new technical paths. This attempt enables large-scale multimodal language models to possess a certain degree of self-evolution capability, allowing them to perform multimodal thought chain reasoning even with limited supervised data. Compared to traditional techniques' reliance on large-scale labeled data, this invention actively explores ways to reduce data dependence. Simultaneously, the application of reinforcement learning provides a relatively clear logical framework for the output results of multimodal retrieval enhancement generation tasks, enhancing the interpretability of the generated content to a certain extent, making the model's decision-making process clearer, and providing new ideas for improving technical reliability.

[0081] For optimizing the training of large-scale multimodal language models, this invention proposes a two-stage reinforcement and fine-tuning approach. In the first stage, a coarse-grained point-based multimodal document sorting method is used to initially filter documents relevant to the target task. The second stage involves fine-grained list-based sorting, while simultaneously optimizing question answers. This step-by-step approach optimizes and improves both search results and question answers, attempting to enhance the relevance of search content to user questions and improve the accuracy of question answers. Compared to traditional single-stage optimization methods, this two-stage reinforcement and fine-tuning method effectively improves search efficiency and answer quality.

[0082] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0083] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.< / ans> < / id> < / think>

Claims

1. A method for training a multimodal large language model, characterized in that, The multimodal large language model includes a first large model and a second large model, and the training method includes the following steps: Construct a training set, which includes a query question set, a test question set, a candidate multimodal document set, a target multimodal document set related to the query question set, and a target answer set for the test question set; The target multimodal document set is used as positive samples, the complement of the target multimodal document set in the candidate multimodal document set is used as negative samples, and the negative samples with the highest semantic similarity to the query question are used as hard negative samples. The first model is trained using the query question set, positive samples, and hard negative samples. During training, for each sample, the first model generates multiple first answers based on a first preset prompt. Based on the multiple first answers, the model parameters of the first model are corrected using an intra-group relative strategy optimization algorithm. The first preset prompt is the probability of yes or no. The test questions in the test question set are reconstructed into multiple test samples by combining them with multiple candidate multimodal documents in the candidate multimodal document set. These test samples are then input into the trained first large model, which generates multiple second answers based on a first preset prompt. Each test sample is then ranked based on these second answers, and the top responses relevant to the test question are retrieved. One candidate multimodal document; By testing the questions and corresponding precedents The second model is trained using a multimodal document. The second model generates multiple third answers based on a second preset prompt. Based on the multiple third answers, the model parameters of the second model are corrected through an intra-group relative strategy optimization algorithm. The second preset prompt consists of the generated document ID, thought process, and target answer.

2. The multimodal large language model training method as described in claim 1, characterized in that, The process of correcting the model parameters of the first large model based on multiple first answers using an intra-group relative strategy optimization algorithm specifically includes the following steps: For each query question, perform diversified sampling to generate a set containing different first answers; The reward value for each first answer is calculated using a first-format reward function for structured evaluation and a relevance-based reward function for measuring the relevance of the answer to the query question. The relative advantage of the current first answer is obtained by taking the mean and variance of the reward values ​​of multiple first answers; the relative advantage is used to characterize the quality difference of the answer compared with other candidate results in the same group. Based on the aforementioned relative advantages, a first objective function is constructed, and the parameters of the model are adjusted based on the objective function.

3. The multimodal large language model training method as described in claim 2, characterized in that, The reward function in the first format is as follows: ； The specific reward function for correlation determination is as follows: ； The specific reward values ​​are as follows: ； In the formula, For the first format reward function, The reward function is for correlation assessment. As a reward value, For the first i The first answer.

4. The multimodal large language model training method as described in claim 1, characterized in that, The process involves sorting each test sample based on multiple second answers and retrieving the top results relevant to the test question. One candidate multimodal document; specifically including the following steps: Calculate multiple similarities for multiple test samples based on multiple second answers; The point-state correlation evaluation function for each test sample is calculated based on multiple similarities. The point-state correlation evaluation functions of multiple test samples are sorted to obtain the top-ranked functions most relevant to the test problem. 1 candidate multimodal document.

5. The multimodal large language model training method as described in claim 4, characterized in that, The point-state correlation evaluation function is as follows: ； In the formula, This is a point-state correlation evaluation function. For the first i One test question, In order to be with the first i The first test question corresponds to the first j Candidate multimodal documents No. s The weight of the second answer, L The number of second answers, For the first s The similarity between the second answer and the second answer.

6. The multimodal large language model training method as described in claim 1, characterized in that, The third answer includes an ID list, a thought process, and a final answer, all of which are in their corresponding tag patterns. Based on multiple third answers, the model parameters of the second largest model are corrected using an intra-group relative strategy optimization algorithm, specifically including the following steps: The coarse match reward and the exact match reward are obtained based on the list of IDs of the third answer and the list of correct IDs; Obtain the second-format reward function based on the tag pattern of the third answer; A correctness reward function is derived based on the final answer and the target answer from the third answer. The comprehensive reward function is calculated by combining the coarse matching reward, exact matching reward, format reward function, and accuracy reward function. The relative advantage of the current third answer is obtained by taking the mean and variance of the combined reward function of multiple first answers; A second objective function is constructed based on the aforementioned relative advantages, and the parameters of the model are adjusted based on the objective function.

7. The multimodal large language model training method as described in claim 6, characterized in that, The coarse matching reward and the precise matching reward are detailed below: ； ； In the formula, For coarse matching rewards, This is a list of IDs for the third answer. For the correct ID list, The number of IDs in the third answer list. The number of correct IDs in the list. To ensure accurate matching of rewards; The second format reward function is shown below: ； In the formula, For the second format reward function, For the first i The third answer, There are three tag modes; The accuracy reward function is as follows: ； In the formula, For the accuracy reward function, Indicates the first The standard answer to this question.

8. A multimodal large language model training system, characterized in that, The multimodal large language model includes a first large model and a second large model, and the training system includes: A construction module is used to construct a training set, which includes a query question set, a test question set, a candidate multimodal document set, a target multimodal document set related to the query question set, and a target answer set for the test question set. The first training module is used to take the target multimodal document set as positive samples, the complement of the target multimodal document set in the candidate multimodal document set as negative samples, and the sample with the highest semantic similarity to the query question among the negative samples as hard negative samples. The first large model is trained by using the query question set, positive samples, and hard negative samples. During the training process, for each sample, the first large model generates multiple first answers based on a first preset prompt. Based on the multiple first answers, the model parameters of the first large model are corrected by an intra-group relative strategy optimization algorithm. The first preset prompt is the probability of yes or no. The ranking module reconstructs test questions from the test question set and candidate multimodal documents from the candidate multimodal document set into multiple test samples. These test samples are then input into the trained first large model, which generates multiple second answers based on a first preset prompt. Each test sample is ranked based on these second answers, and the top-ranked responses relevant to the test question are retrieved. One candidate multimodal document; The second training module is used to compare test questions with corresponding previous training modules. The second model is trained using a multimodal document. The second model generates multiple third answers based on a second preset prompt. Based on the multiple third answers, the model parameters of the second model are corrected through an intra-group relative strategy optimization algorithm. The second preset prompt consists of the generated document ID, thought process, and target answer.

Images