A multimodal reasoning method based on Monte Carlo tree and dynamic retrieval

Abstract

The present disclosure provides a multimodal reasoning method based on Monte Carlo tree and dynamic retrieval. It is applied to scenarios where artificial intelligence assistants assist teachers in grading papers. Specifically, this framework includes a unified retrieval module: the retrieval module includes a mixed modal retrieval corpus and a multimodal retrieval module; the multimodal retrieval module includes two retrieval modules: a text retrieval module and a cross-modal retrieval module, which uses knowledge concept filtering to select key problem-solving insights, and uses Monte Carlo tree search and active retrieval mechanisms to automatically obtain progressive multimodal reasoning annotations for multimodal reasoning for data annotation; then the annotated data is used to gradually align the process reward model through course process reward modeling in two stages to achieve fine-grained verification of multimodal large model reasoning. The MCTS algorithm and active retrieval strategy are combined to improve multimodal reasoning. The framework can automatically obtain high-quality step-by-step reasoning annotations, thereby gradually aligning the process reward model, and ultimately achieving reliable automated multimodal reasoning verification.

Description

Technical Field

[0001] The present disclosure relates to the field of computers, and more specifically, to a multimodal reasoning enhancement method based on Monte Carlo trees and dynamic retrieval. Background Art

[0002] Reasoning ability, as a basic capability of large language models (LLMs) and multimodal large language models (MLLMs), lays the foundation for generalization of a wide range of downstream tasks, such as mathematical problem solving and visual question answering. In complex reasoning scenarios, the model usually requires multiple steps to find the final answer, and each reasoning step may generate multiple branches, leading to various candidate reasoning paths. Therefore, it is crucial to efficiently identify the correct path containing key problem-solving steps and exclude incorrect paths. To achieve this goal, reasoning verification technology enables the model to explore multiple candidate solutions and adopts a high-quality reward model for path selection, thereby providing a promising method to enhance the reliability of model reasoning. Existing technical solutions for the optimization of large language models are as follows

[0003] Multimodal reasoning optimization: The reasoning capabilities of large language models such as ChatGPT, GPT4 and multimodal large language models such as QwenVL and InternVL have attracted widespread attention and become a path to universal understanding. Early studies explored methods such as Chain-of-Thought (CoT), Tree-of-Thought (ToT), and Program-of-Thought (PoT). These methods enhance the logical coherence and response complexity of the model by guiding the model to gradually decompose complex problems, using structured prompt templates and targeted training objectives, and can be applied to multimodal tasks. Another way to enhance the reasoning ability of language models is to use reflection mechanisms such as Self-Correction. This mechanism prompts the model to evaluate its response, improve the reasoning process, and iteratively update the output, thereby promoting more accurate and coherent answers. In previous studies, it has been shown that generating diverse reasoning paths and selecting the most consistent answers can improve the reasoning ability of LLMs, such as Self-Consistency

[10] . Research has shown that building robust reward models to evaluate the quality of generated responses is a promising strategy for unlocking the potential of LLMs. Recently, with the release of OpenAI o1, the concept of a "slow thinking" mechanism has attracted the attention of researchers. Combining a Monte Carlo Tree Search (MCTS) strategy with a verification model can effectively simulate this "slow thinking" process, thereby enhancing reasoning capabilities. However, these advances have primarily focused on unimodal LLMs, while the reasoning potential of multimodal LLMs remains to be explored.

[0004] Multimodal Retrieval-Augmented Generation: In recent years, retrieval-augmented generation (RAG) technology has demonstrated excellent performance in large language models (LLMs). By integrating relevant information from multiple different sources, this technology has achieved remarkable results in various natural language processing tasks. This approach not only improves the quality of text generation, but also demonstrates strong advantages in tasks such as text understanding and information extraction. In addition, the RAG method has also shown potential in the multimodal field, especially in reasoning and question answering. Through cross-modal integration, the model is able to process text, images, and other forms of data simultaneously, thereby providing richer and more accurate answers. However, the current reasoning process still has many ambiguities and lacks an effective verification mechanism, which makes the reliability of the model output questionable.

[0005] While Monte Carlo Tree Search (MCTS)-based methods have been widely used for text-based large language models (LLMs), their adaptability to multimodal large models (MLLMs) remains largely unexplored. Indeed, the unique characteristics of multimodal scenarios require specialized adaptations of MCTS to effectively cope with their complexity. We illustrate these challenges through a theoretical analysis of the limitations of existing MCTS methods.

[0006] In each expansion step, given the input x and the best inference path y selected after simulation, this process can be modeled as:

[0007]

[0008] Among them, r and k represent the number of inference paths and sampling paths respectively, and p φ and p θ denote the probabilities of the generator and verifier, respectively. From this model, we can see that both the expansion and simulation stages are crucial to the entire process, and together they determine the success of inference. Most existing methods focus on optimizing the simulation process, while the expansion process relies on beam search based on the model's internal knowledge.

[0009] This simple strategy is effective for reasoning tasks involving only text, because large language models are well pre-trained on text data and their internal knowledge can be accurately measured. However, in multimodal reasoning tasks, the internal knowledge of MLLMs is insufficient for expanding reasoning paths, as interactions between inputs from different modalities often lead to misalignment. Such errors can be amplified over time as each step in the reasoning process depends on the previous one. Therefore, developing effective and reliable path expansion strategies in multimodal reasoning tasks poses a significant challenge. Summary of the Invention

[0010] The purpose of the embodiments of the present disclosure is to provide a multimodal reasoning enhancement method based on Monte Carlo trees and dynamic retrieval. In order to solve the problem that the interaction between inputs from different modalities often leads to misalignment and amplification of reasoning errors, the present invention proposes a method that can optimize the modeling process according to the event space structure.

[0011] In general, a multimodal reasoning method based on Monte Carlo trees and dynamic retrieval is provided. This method has a wide range of applications, including serving as an AI assistant in the education field to assist teachers in grading papers, or helping students interpret difficult questions. Specifically, it includes a unified retrieval module and a multimodal retrieval module.

[0012] The retrieval module includes a mixed-modal retrieval corpus and a multimodal retrieval module;

[0013] The multimodal retrieval module includes two retrieval modules: a text retrieval module and a cross-modal retrieval module. It adopts knowledge concept filtering to select key problem-solving insights, and automatically obtains progressive multimodal reasoning annotations for multimodal reasoning using Monte Carlo tree search and active retrieval mechanisms for data annotation. The annotated data is then used to gradually align the process reward model through course process reward modeling in two stages to achieve fine-grained verification of multimodal large model reasoning.

[0014] The method for constructing the mixed modality retrieval corpus includes constructing mathematical specific reasoning knowledge and constructing general reasoning knowledge;

[0015] The method for constructing mathematical reasoning-specific knowledge is as follows: using the multimodal mathematical datasets MATHVISTA, MathVerse, MathVision, and WE-MATH, using regular expressions to filter out the parts that overlap with the test benchmark, and combining the problem, solution, and answer of each sample into a single text format, while also attaching the corresponding image storage path;

[0016] The general reasoning knowledge construction method is: using Wikipedia and COIG large-scale question and answer database as the retrieval source of general reasoning, and performing data cleaning and block operations.

[0017] The implementation method of the text retrieval module is as follows: given a text query q of a multimodal sample, Contriever is used to obtain the hidden vectors of the query and document. The relevance score is obtained by calculating the dot product similarity between the query and document representations, as follows:

[0018] The cross-modal retrieval module is implemented as follows: using CLIP to encode the image-text pair (x, t) and obtain the image vector E I (x) and the text vector ET (t), derive the encoding vector of the entire mixed-modal corpus as follows:

[0019]

[0020] in, Represents the empty set, and for the i-th multimodal query, it is encoded as a mixed vector

[0021]

[0022] A cross-modal search is performed between the encoding of each multimodal query and the entire search database, indexed using FAISS, in order to retrieve K samples for each query:

[0023] The method of knowledge concept filtering is: given a multimodal query Q m and its knowledge concept label L kc , for the mixed modality retrieval database D H Encode the first K mixed modal samples retrieved from the dataset and calculate the similarity with the knowledge concept representation: D ins ={r∈D H ∣Sim(r,Q m )≥T r &Sim(r,L kc )≥T kc}, the grid performs the original search similarity threshold T r and knowledge concept consistency threshold T kc , only those samples that meet both criteria are allowed as query Q m Key insights from D ins , where Sim(x,y) represents the cosine similarity between vectors.

[0024] The implementation of the progressive multimodal reasoning annotation is to obtain Q-values ​​at each node to capture potential reasoning errors in the intermediate steps. It includes four core operations:

[0025] Selection operation: In the jth simulation, the process starts from s0, which represents the initial state containing the multimodal input query and the corresponding retrieved problem-solving insights r0. By iteratively selecting and exploring the Monte Carlo tree, the state s j The multimodal query is expressed as:

[0026]

[0027] where y i is the reasoning step of step i.

[0028] Expansion and active retrieval strategy: Given the state si represented by the selected leaf node, the Monte Carlo tree search method is used to trace back to the previous state to form an inference process. The multimodal input is represented as (x i ,t i ,r i ), in the expansion phase, the multimodal input is first connected with the previous reasoning steps, and then the candidate problem-solving insights of each step are dynamically retrieved as needed, and the insights of the previous step are replaced with the latest retrieved insights;

[0029] Simulation operation: Use the probability of inferring the correct answer based on the partial solution as the criterion for quality evaluation, apply one-step rolling to the nodes obtained in each expansion process, and construct the value function:

[0030]

[0031] Where k represents the number of sampled reasoning paths, I is the indicator function, and if the final answer y j Equal to the reference true value Set the value of the current node to 1; otherwise, set it to 0;

[0032] Backpropagation operation: For the terminal node and the current leaf node reached during the rolling process, the Monte Carlo tree search performs a backpropagation update on each (s, a) on the path from the current node to the root node:

[0033] N(s,a)←N(s,a)+1

[0034]

[0035] Where s represents the state, i.e., a node in the Monte Carlo tree. a represents the action, i.e., the model sampling behavior. N(s, a) represents the visit counts of both, Q(s, a) represents the action values ​​of both, and V(s) represents the value function for state s.

[0036] The implementation method of the process reward modeling is as follows: a two-stage method is constructed for reward modeling. In the first stage, the model learns to distinguish the correctness of the reasoning steps and performs step-level DPO pre-alignment: a batch of positive and negative sample pairs are naturally generated in each round of expansion and evaluation, and the preference is aligned using step-by-step direct preference optimization as the training objective; in state i, given a multimodal query and the set of sampled inference paths and the corresponding value set Set the value v j Solution paths with values ​​> 0.8 are filtered as positive samples And the value is v j = 0 is considered a negative sample. For each question Obtain K pairs of stepwise preference pairs and align their inference discrimination abilities according to the stepwise DPO as follows:

[0037]

[0038] Among them, the reference model π ref The initial setting is π θ , and remains unchanged throughout the training process, β is a hyperparameter, σ is the sigmoid function, The goal is to maximize the preference sample y + Relative to the unpreferred sample y - probability;

[0039] In the second phase, the model learns to assign scores to each step, performing point-wise fine-tuning: a step-wise cross-entropy objective is applied to the pre-aligned reward model, using the following parameters:

[0040]

[0041] in, is the true label (0 or 1), r i is the sigmoid score assigned by the reward model;

[0042] In the evaluation stage, we use the fine-tuned reward model and the reward model score as the value of each step in the evaluation stage. We adopt point-by-point soft labels to extract the nodes with the highest scores from the K extended reasoning paths in each round and discard other low-quality paths. In addition, we set an early stopping criterion of 4 and directly obtain the final result in the fourth round.

[0043] The technical effects to be achieved by the embodiments of the present invention are:

[0044] (1) For the first time, a retrieval mechanism is introduced in each step of multimodal reasoning to replace the traditional model self-sampling strategy, thereby enhancing the sampling diversity and accuracy of multi-step reasoning.

[0045] (2) We first propose a unified retrieval framework for improving multimodal reasoning, which constructs a high-quality mixed-modal retrieval corpus and combines it with a multimodal retrieval module to select key supporting evidence to solve the problem.

[0046] (3) To dynamically integrate retrieval into each step of the MCTS algorithm, we propose the AR-MCTS framework, which combines the MCTS algorithm with an active retrieval strategy to improve multimodal reasoning. This framework can automatically obtain high-quality step-by-step reasoning annotations, thereby gradually aligning the process reward model and ultimately achieving reliable automated multimodal reasoning verification. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The above and other objects and features of the present disclosure will become more apparent from the following description in conjunction with the accompanying drawings.

[0048] Figure 1 1 is a schematic diagram illustrating an architecture diagram of a multimodal reasoning enhancement method based on Monte Carlo tree and dynamic retrieval according to an embodiment of the present disclosure;

[0049] Figure 2 is a statistical diagram illustrating a data structure of a hybrid modal reasoning retrieval library according to an embodiment of the present disclosure;

[0050] Figure 3 2 is a schematic diagram showing a hybrid modality reasoning retrieval module according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0051] The following detailed description is provided to help the reader gain a comprehensive understanding of the methods, devices and / or systems described herein. However, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be clear after understanding the disclosure of the present application. For example, the order of operations described herein is merely an example and is not limited to those orders set forth herein, but can be changed as will be clear after understanding the disclosure of the present application, except for operations that must occur in a specific order. In addition, for greater clarity and conciseness, descriptions of features known in the art may be omitted.

[0052] The features described herein can be implemented in different forms and should not be construed as limited to the examples described herein. Rather, the examples described herein are provided to illustrate only some of the many possible ways to implement the methods, devices, and / or systems described herein, which will become clear after understanding the disclosure of this application.

[0053] As used herein, the term "and / or" includes any one of the associated listed items and any combination of any two or more.

[0054] Although terms such as "first," "second," and "third" may be used herein to describe various members, components, regions, layers, or portions, these members, components, regions, layers, or portions should not be limited by these terms. Instead, these terms are used solely to distinguish one member, component, region, layer, or portion from another member, component, region, layer, or portion. Thus, what is referred to as a first member, first component, first region, first layer, or first portion in the examples described herein may also be referred to as a second member, second component, second region, second layer, or second portion without departing from the teachings of the examples.

[0055] In the specification, when an element (such as a layer, region, or substrate) is described as being “on,” “connected to,” or “coupled to” another element, the element may be directly “on,” “connected to,” or “coupled to” the other element, or one or more other elements may be present therebetween. Conversely, when an element is described as being “directly on,” “directly connected to,” or “directly coupled to” another element, there may be no other elements present therebetween.

[0056] The terms used herein are intended only to describe various examples and are not intended to limit the disclosure. Unless the context clearly indicates otherwise, the singular is intended to include the plural. The terms "comprise," "include," and "have" indicate the presence of the recited features, quantities, operations, components, elements, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, quantities, operations, components, elements, and / or combinations thereof.

[0057] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure pertains after understanding the present disclosure. Unless expressly defined otherwise herein, terms (such as those defined in general dictionaries) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and should not be interpreted in an idealized or overly formal manner.

[0058] Furthermore, in describing the examples, when it is deemed that a detailed description of well-known related structures or functions would cause ambiguous interpretation of the present disclosure, such detailed description will be omitted.

[0059] Figure 1 1 is a schematic diagram showing the architecture of a multimodal reasoning enhancement method based on Monte Carlo tree and dynamic retrieval according to an embodiment of the present disclosure.

[0060] The essence of the multimodal reasoning verification method based on the combination of Monte Carlo tree search and dynamic retrieval is to establish a process-level verification framework to enhance the ability of multimodal reasoning without manual labeling, while improving the diversity and accuracy of candidate solution sampling. Therefore, the present invention proposes a multimodal reasoning enhancement (AR-MCTS) framework based on Monte Carlo tree and dynamic retrieval, which realizes fine-grained reasoning verification through active retrieval and Monte Carlo tree search. Figure 2 and 3As shown in the figure, AR-MCTS consists of two main components: 1) It introduces a unified retrieval module, including a high-quality mixed-modal retrieval corpus and a multimodal retrieval module. This module uses knowledge concept filtering to select key problem-solving insights. 2) It automatically obtains step-by-step annotations for multimodal reasoning using MCTS and active retrieval mechanisms. This annotated data is then used to gradually align the process reward model (PRM) in two stages to achieve fine-grained verification of MLLM reasoning.

[0061] Mixed-modal retrieval corpus construction

[0062] Ideally, improving reasoning through retrieval would be like taking an open-book exam for large multimodal models (MLLMs). However, the field of multimodal reasoning continues to face a shortage of high-quality reasoning retrieval corpora. To systematically build a high-quality reasoning retrieval corpus, we conducted a comprehensive survey of open-source datasets, focusing on both general and mathematics-specific reasoning knowledge in multimodal reasoning.

[0063] (1) Mathematical reasoning knowledge: Mathematical reasoning is an important capability of the basic model, and it is accompanied by the emergence of a series of high-quality datasets. In the text-only field, we selected the most widely used mathematical reasoning datasets, including GSM8K and MATH. In the multimodal field, we adopted four carefully cleaned high-quality multimodal mathematical datasets: MATHVISTA, MathVerse, MathVision, and WE-MATH. To further prevent data leakage, we use regular expressions to filter out the parts that overlap with our test benchmark, merge the question, solution process, and answer of each sample into a single text format, and attach the corresponding image storage path. In the end, we obtained 22,000 text-only question-answer pairs and 12,500 multimodal sample pairs from six data sources, covering more than 20 mathematical subfields, and each sample contains detailed solution steps.

[0064] (2) General Reasoning Knowledge: In the real world, general reasoning goes beyond natural sciences. To meet this broad demand, we follow the traditional Retrieval-Augmented Generation (RAG) approach and utilize Wikipedia and the COIG large-scale question-answer corpus as retrieval sources for general reasoning. We perform thorough data cleaning and chunking operations, and ultimately construct this extensive dataset as our general reasoning knowledge base. Statistics of the mixed-modal reasoning corpus are shown in the figure.

[0065] Mixed modal retrieval module

[0066] Given a text-image pair from a multimodal test set, the goal is to retrieve the top K multimodal relevant knowledge for each sample. Since the retrieval corpus contains mixed-modal retrieval sources, two retrieval processes are considered to obtain the top K pairs:

[0067] Text Retrieval Module: Given a text query q of a multimodal sample, we aim to retrieve k relevant documents from a corpus consisting solely of text using a dense retriever. In this work, Contriever is used to obtain latent vectors for the query and document. The relevance score is calculated by computing the dot product similarity between the query and document representations, which helps retrieve the top K documents as follows:

[0068]

[0069] Cross-modal retrieval: We utilize the widely used contrastive visual language model CLIP, which adopts a two-stream architecture consisting of an image encoder and a text encoder. In addition, we use CLIP to encode the image-text pair (x, t) to obtain the image vector E I (x) and the text vector E T (t). Since the mixed modal retrieval corpus contains multimodal samples and text-only samples, following previous research, the encoding vector of the entire mixed modal corpus is derived as follows:

[0070]

[0071] in, Represents an empty set. For the i-th multimodal query, it is encoded as a mixed vector

[0072]

[0073] A cross-modal search is performed between the encoding of each multimodal query and the entire search database, indexed using FAISS, in order to retrieve K samples for each query:

[0074]

[0075] Here, E x (Q m ) T and E x (x j ,t j ) represents the embedding of multimodal queries and samples in the mixed modal corpus, indexed from 1 to N to ensure the accuracy of retrieval.

[0076] Knowledge concept filtering:

[0077] During deployment, multimodal reasoning with retrieved knowledge is very sensitive to the consistency of fine-grained knowledge concepts. For example, algebraic knowledge cannot help solve triangle problems. It is worth noting that most high-quality visual mathematics benchmarks provide detailed category labels such as "angle and length", which motivates us to consider knowledge concepts for fine-grained filtering.

[0078] Given a multimodal query Q m and its knowledge concept label L kc , we retrieve the data from the mixed modal database D according to the formula H We encode the first K mixed modal samples retrieved from the dataset and calculate their similarity with the knowledge concept representation, following the process of “cross-modal retrieval”. We strictly enforce the original retrieval similarity threshold T r and knowledge concept consistency threshold T kc , only those samples that meet both criteria are allowed as query Q m Key insights from D ins .

[0079] D ins ={r∈D H ∣Sim(r,Q m )≥T r &Sim(r,L kc )≥T kc}

[0080] Where Sim(x,y) represents the cosine similarity between vectors.

[0081] Progressive Multimodal Reasoning Annotations:

[0082] Monte Carlo Tree Search (MCTS) is used to assist multi-language large models (MLLMs) in automatically generating step-by-step reasoning annotations through active retrieval, such as Figure 3 As shown. Through the self-exploration process, the Q value is obtained at each step (node) to capture potential reasoning errors in the intermediate steps. The following is our detailed algorithm design, which includes four core operations:

[0083] (1) Selection: In the jth simulation of AR-MCTS, the process starts from s0, which represents the initial state containing the multimodal input query and the corresponding retrieved problem-solving insights r0. The algorithm then explores the Monte Carlo tree through iterative selection, and the multimodal query of state sj can be represented as

[0084]

[0085] (2) Expansion and active retrieval strategy: Given the state si represented by the selected leaf node, the MCTS-based method backtracks to the previous state to form our reasoning process. The multimodal input is represented as (xi ,t i ,r i ). In the traditional expansion process, the temperature is empirically increased to greater than 0.6 to sample multiple potential candidate actions for the next step. In contrast, we emphasize that the supporting knowledge required for each step should vary depending on the reasoning trajectory, and therefore propose an active retrieval strategy. During the expansion phase, we first connect the multimodal input with the previous reasoning step. Then, we dynamically retrieve candidate solution insights for each step as needed, replacing the previous step's insights with the most recently retrieved insights.

[0086] (3) Simulation: We use the probability of inferring the correct answer based on the partial solution as the criterion for quality evaluation. We apply one-step rolling to the nodes obtained in each expansion process to ensure efficiency and construct the value function:

[0087]

[0088] Where k represents the number of sampled reasoning paths and I is the indicator function. If the final answer y i Equal to the reference true value We set the value of the current node to 1; otherwise, it is set to 0.

[0089] (4) Backpropagation: For the terminal node and the current leaf node reached during the rolling process, Monte Carlo Tree Search (MCTS) performs a backpropagation update on each \((s, a)\) on the path from the current node to the root node, as follows:

[0090] N(s,a)←N(s,a)+1

[0091]

[0092] Course Process Reward Modeling:

[0093] After obtaining step-by-step reasoning annotations, we designed a two-stage approach for reward modeling, drawing on concepts from curriculum learning. In the first stage, the model learns to distinguish the correctness of reasoning steps; in the second stage, the model learns to assign scores to each step, thereby achieving generalization from simple to difficult steps.

[0094] (1) Step-level DPO pre-alignment: In the first stage, each round of expansion and evaluation in AR-MCTS naturally generates a batch of positive and negative sample pairs, which motivates us to use step-by-step direct preference optimization (DPO) as the training objective to align preferences. In state i (i-th step in reasoning), given a multimodal query and the set of sampled inference paths and the corresponding value set We will value v jSolution paths with values ​​> 0.8 are filtered as positive samples And the value is v j = 0 is considered a negative sample. Therefore, for each question We can obtain K pairs of stepwise preference pairs and align their inference discrimination abilities according to the stepwise DPO as follows:

[0095]

[0096] Among them, the reference model π ref The initial setting is π θ , and remains constant throughout the training process. β is a hyperparameter and σ is the sigmoid function. The goal is to maximize the preference sample y + Relative to the unpreferred sample y - probability.

[0097] (2) Point-by-point fine-tuning: After pre-alignment, our reward model (PRM) has acquired the ability to initially distinguish the correctness of step-by-step reasoning. To further improve its reasoning scoring ability, we apply the step-by-step cross-entropy objective to the pre-aligned PRM with the following parameters:

[0098]

[0099] in, is the true label (0 or 1), r i is the sigmoid score assigned by PRM. After completing the above two stages, we gradually implement an aligned process reward model.

[0100] In the reasoning phase of AR-MCTS, we use the fine-tuned PRM to Figure 3 We follow the steps shown in the figure, using the PRM score as the value for each step in the evaluation phase. Following related research, we employ point-by-point soft labeling, and discuss PRM hard labeling in the supplementary material. Unlike the annotation process for the training data, we extract the highest-scoring nodes from the K extended reasoning paths in each round, discarding other low-quality paths. Furthermore, we set an early stopping criterion of 4, which allows us to directly obtain the final result in the fourth round, reducing computational complexity.

[0101] While some embodiments of the present disclosure have been shown and described, it will be appreciated by those skilled in the art that changes may be made to these embodiments without departing from the principles and spirit of the disclosure, the scope of which is defined by the claims and their equivalents.

Claims

1. A multimodal reasoning method based on Monte Carlo tree and dynamic retrieval, characterized in that: Applied to assist teachers in grading papers. The teacher inputs the test paper content to be reviewed, applies the retrieval model including the retrieval module, and obtains the test paper answer content that matches the test paper content. The retrieval module includes a mixed-modal retrieval corpus and a multimodal retrieval module; The multimodal retrieval module includes two retrieval modules: a text retrieval module and a cross-modal retrieval module. It uses knowledge concept filtering to select key problem-solving insights and automatically obtains progressive multimodal reasoning annotations for multimodal reasoning using Monte Carlo tree search and active retrieval mechanisms for data annotation. The annotated data is then used to gradually align the process reward model through course process reward modeling in two stages to achieve fine-grained verification of the multimodal large model reasoning. The implementation method of the text retrieval module is as follows: given a text query q of a multimodal sample, Contriever is used to obtain the hidden vectors of the query and document. The relevance score is obtained by calculating the dot product similarity between the query and document representations, as follows: ; The cross-modal retrieval module is implemented as follows: using CLIP to encode the image-text pair (x, t) and obtain the image vector and text vectors , derive the encoding vector of the entire mixed modal corpus as follows: in, Represents the empty set, and for the i-th multimodal query, it is encoded as a mixed vector ; A cross-modal search is performed between the encoding of each multimodal query and the entire search database, indexed using FAISS, to retrieve K samples for each query: ; The method of knowledge concept filtering is: given a multimodal query and its knowledge concept labels , for the mixed modality retrieval library Encode the first K mixed modality samples retrieved from , and calculate the similarity with the knowledge concept representation: , perform the original search similarity threshold and knowledge concept consistency threshold , only those samples that meet both criteria are allowed as queries Key insights ,in Represents the cosine similarity between vectors; The implementation of the progressive multimodal reasoning annotation is to obtain Q values ​​at each node to capture potential reasoning errors in the intermediate steps. It includes four core operations: Selection operation: In the jth simulation, the process starts from s0, which represents the initial state containing the multimodal input query and the corresponding retrieved problem-solving insights r0. By iteratively selecting and exploring the Monte Carlo tree, the state s j The multimodal query is expressed as: in is the reasoning step of step i; Expansion and active retrieval strategy: Given the state s represented by the selected leaf node i , the Monte Carlo tree search method backtracks to the previous state to form an inference process, and the multimodal input is represented as ,In the expansion phase, the multimodal input is first connected with the previous reasoning steps, and then the candidate problem-solving insights of each step are dynamically retrieved as needed, and the insights of the previous step are replaced with the latest retrieved insights; Simulation operation: Use the probability of inferring the correct answer based on the partial solution as the criterion for quality evaluation, apply one-step rolling to the nodes obtained in each expansion process, and construct the value function: in represents the number of sampled inference paths, is an indicator function, if the final answer Equal to the reference true value , set the value of the current node to 1; otherwise, set it to 0; Backpropagation operation: For the terminal node and the current leaf node reached during the rolling process, the Monte Carlo tree search performs a backpropagation update on each (s, a) on the path from the current node to the root node: ； in represents the state, i.e. the node of the Monte Carlo tree, Represents the action, i.e. the model sampling behavior, It represents the visit count of both. Represents the action value of the two, Represents the status The value function of .

2. The multimodal reasoning method based on Monte Carlo tree and dynamic retrieval according to claim 1, characterized in that: The method for constructing the mixed modality retrieval corpus includes constructing mathematical specific reasoning knowledge and constructing general reasoning knowledge; The method for constructing mathematical reasoning-specific knowledge is as follows: using the multimodal mathematical datasets MATHVISTA, MathVerse, MathVision, and WE-MATH, using regular expressions to filter out the parts that overlap with the test benchmark, and combining the problem, solution, and answer of each sample into a single text format, while also attaching the corresponding image storage path; The general reasoning knowledge construction method is: using Wikipedia and COIG large-scale question and answer database as the retrieval source of general reasoning, and performing data cleaning and block operations.

3. The multimodal reasoning method based on Monte Carlo tree and dynamic retrieval according to claim 1, characterized in that: The implementation method of the process reward modeling is as follows: a two-stage method is constructed for reward modeling. In the first stage, the model learns to distinguish the correctness of the reasoning steps and performs step-level DPO pre-alignment: a batch of positive and negative sample pairs are naturally generated in each round of expansion and evaluation, and the preferences are aligned using step-by-step direct preference optimization as the training objective; In state i, given a multimodal query and the set of sampled inference paths , and the corresponding value set , the value The solution path is filtered as a positive sample , and the value is The path is considered as a negative sample , for each question , obtain K pairs of stepwise preference pairs and align the reasoning discrimination abilities according to the stepwise DPO, as follows: Among them, the reference model Initial setting is , and remains constant throughout the training process, is a hyperparameter, is the sigmoid function, The goal is to maximize the preference sample Relative to the unpreferred sample probability; In the second phase, the model learns to assign scores to each step, performing point-wise fine-tuning: a step-wise cross-entropy objective is applied to the pre-aligned reward model, using the following parameters: in, is the true label (0 or 1), is the sigmoid score assigned by the reward model; In the evaluation stage, we use the fine-tuned reward model and the reward model score as the value of each step in the evaluation stage. We adopt point-by-point soft labels to extract the nodes with the highest scores from the K extended reasoning paths in each round and discard other low-quality paths. In addition, we set an early stopping criterion to 4 and directly obtain the final result in the 4th round.

Images