Multimodal reasoning method based on Monte Carlo tree and dynamic retrieval

Abstract

The invention provides a multimodal reasoning method based on a Monte Carlo tree and dynamic retrieval. The method is applied to an artificial intelligence assistant-assisted teacher paper judgment scene. Specifically, the framework comprises a unified retrieval module, wherein the retrieval module comprises a mixed modal retrieval corpus and a multi-modal retrieval module; the multi-modal retrieval module comprises two retrieval modules, namely a text retrieval module and a cross-modal retrieval module, key problem solving insights are selected by adopting knowledge concept filtering, and progressive multi-modal reasoning annotations of multi-modal reasoning are automatically obtained by utilizing Monte Carlo tree search and an active retrieval mechanism for data annotation; and gradually aligning a process reward model through course process reward modeling in two stages by utilizing the annotation data so as to realize fine-grained verification of multi-modal large model reasoning. An MCTS algorithm and an active retrieval strategy are combined to improve multi-modal reasoning. The framework can automatically obtain high-quality step-by-step reasoning annotations, so that a process reward model is aligned step by step, and finally reliable automatic multi-modal reasoning verification is realized.

Description

Technical Field

[0001] The present disclosure relates to the field of computers, and more specifically, to a multimodal reasoning enhancement method based on Monte Carlo trees and dynamic retrieval. Background Art

[0002] Reasoning ability, as a basic capability of large language models (LLMs) and multimodal large language models (MLLMs), lays the foundation for generalization of a wide range of downstream tasks, such as mathematical problem solving and visual question answering. In complex reasoning scenarios, the model usually requires multiple steps to find the final answer, and each reasoning step may generate multiple branches, leading to various candidate reasoning paths. Therefore, it is crucial to efficiently identify the correct path containing key problem-solving steps and exclude incorrect paths. To achieve this goal, reasoning verification technology enables the model to explore multiple candidate solutions and adopts high-quality reward models for path selection, thereby providing a promising method to enhance the reliability of model reasoning. Existing technical solutions for the optimization of large language models are as follows

[0003] Multimodal reasoning optimization: The reasoning ability of large language models such as ChatGPT, GPT4 and multimodal large language models such as QwenVL and InternVL has attracted widespread attention and become a way to achieve universal understanding. Early studies explored methods such as Chain-of-Thought (CoT), Tree-of-Thought (ToT), and Program-of-Thought (PoT). These methods enhance the logical coherence and response complexity of the model by guiding the model to gradually decompose complex problems, using structured prompt templates and targeted training goals, and can be applied to multimodal tasks. Another way to enhance the reasoning ability of language models is to use reflection mechanisms such as Self-Correction. This mechanism prompts the model to evaluate its response, improve the reasoning process, and iteratively update the output, thereby promoting more accurate and coherent answers. In previous studies, it has been shown that generating diverse reasoning paths and selecting the most consistent answers can improve the reasoning ability of LLMs, such as Self-Consistency

[10] . Research has shown that building a robust reward model to evaluate the quality of generated responses is a promising strategy to unlock the potential of LLM. Recently, with the release of OpenAI o1, the concept of "slow thinking" mechanism has attracted the attention of researchers. Combining the Monte Carlo Tree Search (MCTS) strategy with a verification model can effectively simulate the "slow thinking" process, thereby enhancing reasoning capabilities. However, these advances have mainly focused on unimodal LLMs, while the reasoning potential of multimodal LLMs remains to be explored.

[0004] Multimodal Retrieval-Augmented Generation: In recent years, the Retrieval-Augmented Generation (RAG) technique has demonstrated excellent performance in large language models (LLMs). By integrating relevant information from multiple different sources, this technique has achieved remarkable results in various natural language processing tasks. This approach not only improves the quality of text generation, but also shows strong advantages in tasks such as text understanding and information extraction. In addition, the RAG method has also shown potential in the multimodal field, especially in reasoning and question answering. Through cross-modal integration, the model is able to process text, images, and other forms of data simultaneously, thereby providing richer and more accurate answers. However, the current reasoning process still has many ambiguities and lacks an effective verification mechanism, which makes the reliability of the model output questionable.

[0005] Although Monte Carlo Tree Search (MCTS) based methods have been widely used in text-based large language models (LLMs), their adaptability to multimodal large models (MLLMs) remains largely unexplored. In fact, the unique characteristics of multimodal scenarios require specialized adjustments to MCTS to effectively cope with its complexity. We can illustrate these challenges by theoretically analyzing the limitations of existing MCTS methods.

[0006] In each expansion step, given the input x and the best inference path y selected after simulation, this process can be modeled as:

[0007]

[0008] Among them, r and k represent the number of inference paths and sampling paths respectively, and p φ and p θ denote the probabilities of the generator and the verifier, respectively. From this model, we can see that both the expansion and simulation stages are crucial to the whole process, and together they determine the success of reasoning. Most existing methods focus on optimizing the simulation process, while the expansion process relies on beam search based on the internal knowledge of the model.

[0009] This simple strategy is effective for reasoning tasks involving only text, since large language models are well pre-trained on text data and their internal knowledge can be accurately measured. However, in multimodal reasoning tasks, the internal knowledge of MLLMs is insufficient for the extension of reasoning paths, since the interactions between inputs from different modalities are often misaligned. Such errors can grow over time as each step in the reasoning process depends on the previous step. Therefore, developing effective reliable path extension strategies in multimodal reasoning tasks poses a major challenge. Summary of the invention

[0010] The purpose of the embodiments of the present disclosure is to provide a multimodal reasoning enhancement method based on Monte Carlo trees and dynamic retrieval. In order to solve the problem that the interactions between inputs from different modalities often result in misalignment, which leads to amplified reasoning errors, the present invention proposes a method that can optimize the modeling process according to the event space structure.

[0011] In a general aspect, a multimodal reasoning method based on Monte Carlo tree and dynamic retrieval is provided, which has a wide range of application scenarios, including serving as an artificial intelligence assistant to assist teachers in marking papers in the field of education, or helping students to interpret difficult questions. Specifically, it includes a unified retrieval module and a multimodal retrieval module;

[0012] The retrieval module includes a mixed modality retrieval corpus and a multimodal retrieval module;

[0013] The multimodal retrieval module includes two retrieval modules: a text retrieval module and a cross-modal retrieval module. It adopts knowledge concept filtering to select key problem-solving insights, and uses Monte Carlo tree search and active retrieval mechanisms to automatically obtain progressive multimodal reasoning annotations for multimodal reasoning for data annotation. The annotated data is then used to gradually align the process reward model through course process reward modeling in two stages to achieve fine-grained verification of multimodal large model reasoning.

[0014] The method for constructing the mixed modality retrieval corpus includes constructing mathematical specific reasoning knowledge and constructing general reasoning knowledge;

[0015] The method for constructing mathematical specific reasoning knowledge is as follows: using multimodal mathematical datasets MATHVISTA, MathVerse, MathVision, and WE-MATH, using regular expressions to filter out the parts that overlap with the test benchmark, merging the question, solution process, and answer of each sample into a single text format, and attaching the corresponding image storage path;

[0016] The general reasoning knowledge construction method is: using Wikipedia and COIG large-scale question and answer database as the retrieval source of general reasoning, and performing data cleaning and block operations.

[0017] The implementation method of the text retrieval module is as follows: given a text query q of a multimodal sample, Contriever is used to obtain the hidden vectors of the query and the document, and the relevance score is obtained by calculating the dot product similarity between the query and document representations, as follows:

[0018] The implementation method of the cross-modal retrieval module is: use CLIP to encode the image-text pair (x, t) to obtain the image vector E I (x) and the text vector ET (t), derive the encoding vector of the entire mixed modal corpus as follows:

[0019]

[0020] in, Represents an empty set. For the i-th multimodal query, it is encoded as a mixed vector

[0021]

[0022] A cross-modal search is performed between the encoding of each multimodal query and the entire search database, indexed using FAISS, in order to retrieve K samples for each query:

[0023] The method of knowledge concept filtering is: given a multimodal query Q m Its knowledge concept label L kc , for the mixed modality retrieval database D H Encode the first K mixed modal samples retrieved from the dataset and calculate the similarity with the knowledge concept representation: ins ={r∈D H ∣Sim(r,Q m )≥T r &Sim(r,L kc )≥T kc}, the grid performs the original retrieval similarity threshold T r and knowledge concept consistency threshold T kc , only those samples that satisfy both criteria are allowed as queries Q m Key insights from D ins , where Sim(x,y) represents the cosine similarity between vectors.

[0024] The progressive multimodal reasoning annotation is implemented by obtaining Q values ​​at each node to capture potential reasoning errors in the intermediate steps, including four core operations:

[0025] Selection operation: In the jth simulation, the process starts from s0, which represents the initial state containing the multimodal input query and the corresponding retrieved problem-solving insights r0, and through iterative selection exploration of the Monte Carlo tree, the state s j The multimodal query is expressed as:

[0026]

[0027] where y i is the reasoning step of the i-th step.

[0028] Expansion and active retrieval strategy: Given the state si represented by the selected leaf node, the Monte Carlo tree search method is used to trace back to the previous state to form a reasoning process. The multimodal input is represented as (x i ,t i ,r i ), in the expansion phase, the multimodal input is first connected with the previous reasoning steps, and then the candidate solution insights of each step are dynamically retrieved as needed, and the insights of the previous step are replaced with the latest retrieved insights;

[0029] Simulation operation: Use the probability of inferring the correct answer based on the partial solution as the criterion for quality evaluation, apply one-step rolling to each node obtained in the expansion process, and construct the value function:

[0030]

[0031] Where k represents the number of sampled reasoning paths, I is the indicator function, and if the final answer y j Equal to the reference true value Set the value of the current node to 1; otherwise, set it to 0;

[0032] Backpropagation operation: For the terminal node and the current leaf node reached during the rolling process, the Monte Carlo tree search performs a backpropagation update on each (s, a) on the path from the current node to the root node:

[0033] N(s,a)←N(s,a)+1

[0034]

[0035] Where s represents the state, i.e., the node of the Monte Carlo tree. a represents the action, i.e., the model sampling behavior. N(s,a) represents the access count of the two, Q(s,a) represents the action value of the two. V(s) represents the value function for state s.

[0036] The implementation method of the process reward modeling is as follows: a two-stage method is constructed for reward modeling. In the first stage, the model learns to distinguish the correctness of the reasoning steps and performs step-level DPO pre-alignment: a batch of positive and negative sample pairs are naturally generated in each round of expansion and evaluation, and the step-by-step direct preference optimization is used as the training objective to align the preferences; in state i, given a multimodal query and the set of sampled inference paths And the corresponding value set Set the value v j The solution paths with values ​​> 0.8 are filtered as positive samples. The value is v j = 0 is considered a negative sample. For each question K pairs of stepwise preference pairs are obtained and their inference discrimination abilities are aligned according to the stepwise DPO as follows:

[0037]

[0038] Among them, the reference model π ref The initial setting is π θ , and remains unchanged throughout the training process, β is a hyperparameter, σ is the sigmoid function, The goal is to maximize the preference sample y + Relative to the unpreferred sample y - probability;

[0039] In the second phase, the model learns to assign scores to each step, performing point-wise fine-tuning: a step-wise cross entropy objective is applied to the pre-aligned reward model with the following parameters:

[0040]

[0041] in, is the true label (0 or 1), r i is the sigmoid score assigned by the reward model;

[0042] In the stage, the fine-tuned reward model is used, and the reward model score is used as the value of each step in the evaluation stage. Point-by-point soft labels are adopted to extract the nodes with the highest scores from the K extended reasoning paths in each round, and other low-quality paths are discarded. In addition, an early stopping criterion is set to 4, and the final result is directly obtained in the 4th round.

[0043] The technical effects to be achieved by the embodiments of the present invention are:

[0044] (1) For the first time, a retrieval mechanism is introduced in each step of multimodal reasoning to replace the traditional model self-sampling strategy, thereby enhancing the sampling diversity and accuracy of multi-step reasoning.

[0045] (2) We first propose a unified retrieval framework to improve multimodal reasoning, which constructs a high-quality mixed-modal retrieval corpus and combines it with a multimodal retrieval module to select key supporting evidence to solve the problem.

[0046] (3) In order to dynamically integrate retrieval into each step of the MCTS algorithm, we propose the AR-MCTS framework, which combines the MCTS algorithm with an active retrieval strategy to improve multimodal reasoning. This framework can automatically obtain high-quality step-by-step reasoning annotations, thereby gradually aligning the process reward model and ultimately achieving reliable automated multimodal reasoning verification. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The above and other objects and features of the present disclosure will become more apparent from the following description in conjunction with the accompanying drawings.

[0048] Figure 1 is a schematic diagram showing an architecture diagram of a multimodal reasoning enhancement method based on Monte Carlo tree and dynamic retrieval according to an embodiment of the present disclosure;

[0049] Figure 2 is a data statistical schematic diagram showing a hybrid modal reasoning retrieval library according to an embodiment of the present disclosure;

[0050] Figure 3 is a schematic diagram showing a hybrid modality reasoning retrieval module according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0051] The following specific embodiments are provided to help the reader obtain a comprehensive understanding of the methods, devices and / or systems described herein. However, after understanding the disclosure of the present application, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be clear. For example, the order of operations described herein is only an example and is not limited to those orders set forth herein, but can be changed as will be clear after understanding the disclosure of the present application, except for operations that must occur in a specific order. In addition, for greater clarity and simplicity, the description of features known in the art may be omitted.

[0052] The features described herein can be implemented in different forms and should not be construed as being limited to the examples described herein. Rather, the examples described herein have been provided to illustrate only some of the many possible ways to implement the methods, devices, and / or systems described herein, which will be clear after understanding the disclosure of the present application.

[0053] As used herein, the term "and / or" includes any one of the associated listed items and any combination of any two or more.

[0054] Although terms such as "first", "second", and "third" may be used herein to describe various members, components, regions, layers, or portions, these members, components, regions, layers, or portions should not be limited by these terms. Instead, these terms are only used to distinguish one member, component, region, layer, or portion from another member, component, region, layer, or portion. Therefore, without departing from the teachings of the examples described herein, the first member, first component, first region, first layer, or first portion referred to in the examples may also be referred to as the second member, second component, second region, second layer, or second portion.

[0055] In the specification, when an element (such as a layer, a region or a substrate) is described as being “on”, “connected to” or “coupled to” another element, the element may be directly “on”, “connected to” or “coupled to” another element, or one or more other elements may be present therebetween. Conversely, when an element is described as being “directly on”, “directly connected to” or “directly coupled to” another element, there may be no other elements present therebetween.

[0056] The terms used herein are only used to describe various examples and are not intended to limit the disclosure. Unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. The terms "comprise", "include" and "have" indicate the presence of the described features, quantities, operations, components, elements and / or combinations thereof, but do not exclude the presence or addition of one or more other features, quantities, operations, components, elements and / or combinations thereof.

[0057] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by a person of ordinary skill in the art to which the present disclosure belongs after understanding the present disclosure. Unless explicitly defined as such herein, terms (such as those defined in a general dictionary) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and should not be interpreted in an idealized or overly formal manner.

[0058] Furthermore, in the description of examples, when it is considered that a detailed description of a well-known related structure or function would cause vague interpretation of the present disclosure, such a detailed description will be omitted.

[0059] Figure 1 It is a schematic diagram showing the architecture of a multimodal reasoning enhancement method based on Monte Carlo tree and dynamic retrieval according to an embodiment of the present disclosure.

[0060] The essence of the multimodal reasoning verification method based on the combination of Monte Carlo tree search and dynamic retrieval is to establish a process-level verification framework to enhance the ability of multimodal reasoning without manual labeling, while improving the diversity and accuracy of candidate solution sampling. Therefore, the present invention proposes a multimodal reasoning enhancement (AR-MCTS) framework based on Monte Carlo tree and dynamic retrieval, which realizes fine-grained reasoning verification through active retrieval and Monte Carlo tree search. Figure 2 and 3As shown in the figure, AR-MCTS consists of two main components: 1) It introduces a unified retrieval module, including a high-quality mixed-modal retrieval corpus and a multimodal retrieval module. This module adopts knowledge concept filtering to select key problem-solving insights. 2) It automatically obtains step-by-step annotations for multimodal reasoning using MCTS and active retrieval mechanisms. These annotated data are then used to gradually align the process reward model (PRM) in two stages to achieve fine-grained verification of MLLM reasoning.

[0061] Mixed modal retrieval corpus construction

[0062] Ideally, improving reasoning ability through retrieval is like taking an open-book exam for multimodal large models (MLLMs). However, the field of multimodal reasoning has always been faced with the problem of insufficient high-quality reasoning retrieval corpora. In order to systematically build a high-quality reasoning retrieval corpus, we conducted a comprehensive survey of open source datasets, focusing on general reasoning knowledge and mathematics-specific reasoning knowledge in multimodal reasoning.

[0063] (1) Mathematics-specific reasoning knowledge: Mathematical reasoning is an important capability of the base model, accompanied by the emergence of a series of high-quality datasets. In the text-only field, we selected the most widely used mathematical reasoning datasets, including GSM8K and MATH. In the multimodal field, we adopted four carefully cleaned high-quality multimodal mathematics datasets: MATHVISTA, MathVerse, MathVision, and WE-MATH. To further prevent data leakage, we use regular expressions to filter out the parts that overlap with our test benchmark, merge the question, solution process, and answer of each sample into a single text format, and attach the corresponding image storage path. In the end, we obtained 22,000 text-only question-answer pairs and 12,500 multimodal sample pairs from six data sources, covering more than 20 mathematical subfields, and each sample contains detailed problem-solving steps.

[0064] (2) General reasoning knowledge: In the real world, general reasoning goes beyond natural subjects. To meet this broad demand, we follow the traditional retrieval-augmented generation (RAG) approach and use Wikipedia and COIG large-scale question-answer corpora as retrieval sources for general reasoning. We perform thorough data cleaning and chunking operations and ultimately construct this extensive dataset as our general reasoning knowledge base. The statistics of the mixed modal reasoning corpus are shown in the figure.

[0065] Mixed modality retrieval module

[0066] Given a text-image pair from a multimodal test set, the goal is to retrieve the top K multimodal relevant knowledge for each sample. Since the retrieval corpus contains mixed modal retrieval sources, two retrieval processes are considered to obtain the top K pairs:

[0067] Text Retrieval Module: Given a text query q of a multimodal sample, we aim to retrieve k relevant documents from a corpus containing only text using a dense retriever. In this work, Contriever is used to obtain the latent vectors of the query and document. The relevance score is derived by calculating the dot product similarity between the query and document representations, which helps to retrieve the top K documents as follows:

[0068]

[0069] Cross-modal retrieval: We use the widely used contrastive visual language model CLIP, which adopts a two-stream architecture including an image encoder and a text encoder. In addition, we use CLIP to encode the image-text pair (x, t) to obtain the image vector E I (x) and the text vector E T (t). Since the mixed modal retrieval corpus contains multimodal samples and text-only samples, following previous studies, the encoding vector of the entire mixed modal corpus is derived as follows:

[0070]

[0071] in, Represents an empty set. For the i-th multimodal query, it is encoded as a mixed vector

[0072]

[0073] A cross-modal search is performed between the encoding of each multimodal query and the entire search database, indexed using FAISS, in order to retrieve K samples for each query:

[0074]

[0075] Here, E x (Q m ) T and E x (x j ,t j ) represents the embedding of multimodal queries and samples in the mixed modal corpus, indexed from 1 to N to ensure the accuracy of retrieval.

[0076] Knowledge concept filtering:

[0077] In the deployment process, multimodal reasoning with retrieved knowledge is very sensitive to the consistency of fine-grained knowledge concepts, for example, algebraic knowledge cannot help solve triangle problems. It is worth noting that most high-quality visual mathematics benchmarks provide detailed category labels such as “angle and length”, which motivates us to consider knowledge concepts for fine-grained filtering.

[0078] Given a multimodal query Q m Its knowledge concept label L kc , we use the formula to retrieve the mixed modal database D H We encode the first K mixed modal samples retrieved from the search and calculate the similarity with the knowledge concept representation, following the process of "cross-modal retrieval". We strictly enforce the original retrieval similarity threshold T r and knowledge concept consistency threshold T kc , only those samples that satisfy both criteria are allowed as queries Q m Key insights from D ins .

[0079] D ins ={r∈D H ∣Sim(r,Q m )≥T r &Sim(r,L kc )≥T kc}

[0080] Where Sim(x,y) represents the cosine similarity between vectors.

[0081] Progressive Multimodal Reasoning Annotations:

[0082] Through active retrieval, Monte Carlo Tree Search (MCTS) is used to assist multi-lingual large models (MLLMs) to automatically generate step-by-step reasoning annotations, such as Figure 3 As shown. Through the self-exploration process, the Q value is obtained at each step (node) to capture potential reasoning errors in the intermediate steps. The following is our detailed algorithm design, including four core operations:

[0083] (1) Selection: In the jth simulation of AR-MCTS, the process starts from s0, which represents the initial state containing the multimodal input query and the corresponding retrieved problem-solving insights r0. The algorithm then explores the Monte Carlo tree by iterative selection, and the multimodal query of state sj can be represented as

[0084]

[0085] (2) Expansion and active retrieval strategy: Given the state si represented by the selected leaf node, the MCTS-based method backtracks to the previous state to form our reasoning process. The multimodal input is represented as (xi ,t i ,r i ). In the traditional expansion process, the temperature is empirically increased to greater than 0.6 to sample multiple potential candidate actions for the next step. Different from this, we emphasize that the supporting knowledge required for each step should vary according to different reasoning trajectories, and thus propose an active retrieval strategy. In the expansion phase, we first concatenate the multimodal input with the previous reasoning steps. Then, we dynamically retrieve candidate solution insights for each step as needed and replace the previous step's insights with the latest retrieved insights.

[0086] (3) Simulation: We use the probability of inferring the correct answer based on partial solutions as the criterion for quality evaluation. We apply one-step rolling to each node obtained during the expansion process to ensure efficiency and construct the value function:

[0087]

[0088] Where k represents the number of sampled reasoning paths and I is the indicator function. If the final answer y i Equal to the reference true value We set the value of the current node to 1; otherwise, it is set to 0.

[0089] (4) Backpropagation: For the terminal node and the current leaf node reached during the rolling process, the Monte Carlo Tree Search (MCTS) performs a backpropagation update on each \((s,a)\) on the path from the current node to the root node, and the formula is as follows:

[0090] N(s,a)←N(s,a)+1

[0091]

[0092] Course Process Reward Modeling:

[0093] After obtaining step-by-step reasoning annotations, we design a two-stage approach for reward modeling, drawing on the concept of curriculum learning. In the first stage, the model learns to distinguish the correctness of reasoning steps; in the second stage, the model learns to assign scores to each step, thereby achieving generalization from simple to difficult.

[0094] (1) Step-level DPO pre-alignment: In the first stage, each round of expansion and evaluation in AR-MCTS naturally generates a batch of positive and negative sample pairs, which motivates us to use step-wise direct preference optimization (DPO) as the training objective to align preferences. In state i (the i-th step in reasoning), given a multimodal query and the set of sampled inference paths And the corresponding value set We will value v jThe solution paths with values ​​> 0.8 are filtered as positive samples. The value is v j = 0 is considered a negative sample. Therefore, for each question We can obtain K pairs of stepwise preference pairs and align the reasoning discrimination abilities according to the stepwise DPO as follows:

[0095]

[0096] Among them, the reference model π ref The initial setting is π θ , and remains constant throughout the training process. β is a hyperparameter and σ is the sigmoid function. The goal is to maximize the preference sample y + Relative to the unpreferred sample y - probability.

[0097] (2) Point-wise fine-tuning: After pre-alignment, our reward model (PRM) has acquired the ability to initially distinguish the correctness of step-by-step reasoning. To further improve its reasoning scoring ability, we apply a step-by-step cross-entropy objective to the pre-aligned PRM with the following parameters:

[0098]

[0099] in, is the true label (0 or 1), r i is the sigmoid score assigned by PRM. After completing the above two stages, we gradually implement an aligned process reward model.

[0100] In the reasoning phase of AR-MCTS, we use the fine-tuned PRM according to Figure 3 We perform the steps in the example, using the PRM score as the value of each step in the evaluation phase. Referring to related studies, we adopt point-by-point soft labels, and discuss the hard labels of PRM in the supplementary material. Different from the annotation process of training data, we extract the nodes with the highest scores from the K extended reasoning paths in each round and discard other low-quality paths. In addition, we set an early stopping criterion of 4, which enables us to directly obtain the final result in the 4th round, thereby reducing the computational complexity.

[0101] Although some embodiments of the present disclosure have been shown and described, it will be appreciated by those skilled in the art that modifications may be made to the embodiments without departing from the principles and spirit of the present disclosure, the scope of which is defined by the claims and their equivalents.

Claims

1. A multimodal reasoning method based on Monte Carlo tree and dynamic retrieval, characterized in that: Applied to assist teachers in marking papers, input the test paper content to be reviewed, apply the retrieval model including the retrieval module, and obtain the test paper answer content that matches the test paper content; The retrieval module includes a mixed modality retrieval corpus and a multimodal retrieval module; The multimodal retrieval module includes two retrieval modules: a text retrieval module and a cross-modal retrieval module. It adopts knowledge concept filtering to select key problem-solving insights, and uses Monte Carlo tree search and active retrieval mechanisms to automatically obtain progressive multimodal reasoning annotations for multimodal reasoning for data annotation. The annotated data is then used to gradually align the process reward model through course process reward modeling in two stages to achieve fine-grained verification of multimodal large model reasoning.

2. The multimodal reasoning method based on Monte Carlo tree and dynamic retrieval as claimed in claim 1, characterized in that: The method for constructing the mixed modality retrieval corpus includes constructing mathematical specific reasoning knowledge and constructing general reasoning knowledge; The method for constructing mathematical specific reasoning knowledge is as follows: using multimodal mathematical datasets MATHVISTA, MathVerse, MathVision, and WE-MATH, using regular expressions to filter out the parts that overlap with the test benchmark, merging the question, solution process, and answer of each sample into a single text format, and attaching the corresponding image storage path; The general reasoning knowledge construction method is: using Wikipedia and COIG large-scale question and answer database as the retrieval source of general reasoning, and performing data cleaning and block operations.

3. The multimodal reasoning method based on Monte Carlo tree and dynamic retrieval as claimed in claim 1, characterized in that: The implementation method of the text retrieval module is as follows: given a text query q of a multimodal sample, Contriever is used to obtain the hidden vectors of the query and the document, and the relevance score is obtained by calculating the dot product similarity between the query and document representations, as follows:

4. The multimodal reasoning method based on Monte Carlo tree and dynamic retrieval as claimed in claim 3, characterized in that: The implementation method of the cross-modal retrieval module is: use CLIP to encode the image-text pair (x, t) to obtain the image vector E I (x) and the text vector E T (t), derive the encoding vector of the entire mixed modal corpus as follows: in, Represents an empty set. For the i-th multimodal query, it is encoded as a mixed vector A cross-modal search is performed between the encoding of each multimodal query and the entire search database, indexed using FAISS, in order to retrieve K samples for each query:

5. The multimodal reasoning method based on Monte Carlo tree and dynamic retrieval as claimed in claim 4, characterized in that: The method of knowledge concept filtering is: given a multimodal query Q m Its knowledge concept label L kc , for the mixed modality retrieval database D H Encode the first K mixed modal samples retrieved from the dataset and calculate the similarity with the knowledge concept representation: ins ={r∈D H ∣Sim(r,Q m )≥T r &Sim(r,L kc )≥T kc }, the grid performs the original retrieval similarity threshold T r and knowledge concept consistency threshold T kc , only those samples that satisfy both criteria are allowed as queries Q m Key insights from D ins , where Sim(x,y) represents the cosine similarity between vectors.

6. The multimodal reasoning enhancement method based on Monte Carlo tree and dynamic retrieval as claimed in claim 5, characterized in that: The progressive multimodal reasoning annotation is implemented by obtaining Q values ​​at each node to capture potential reasoning errors in the intermediate steps, including four core operations: Selection operation: In the jth simulation, the process starts from s0, which represents the initial state containing the multimodal input query and the corresponding retrieved problem-solving insights r0, and through iterative selection exploration of the Monte Carlo tree, the state s j The multimodal query is expressed as: where y i is the reasoning step of step i; Expansion and active retrieval strategy: Given the state s represented by the selected leaf node i , the Monte Carlo tree search method traces back to the previous state to form the reasoning process, and the multimodal input is represented as (x i ,t i ,r i ), in the expansion phase, the multimodal input is first connected with the previous reasoning steps, and then the candidate solution insights of each step are dynamically retrieved as needed, and the insights of the previous step are replaced with the latest retrieved insights; Simulation operation: Use the probability of inferring the correct answer based on the partial solution as the criterion for quality evaluation, apply one-step rolling to each node obtained in the expansion process, and construct the value function: Where k represents the number of sampled reasoning paths, I is the indicator function, and if the final answer y j Equal to the reference true value Set the value of the current node to 1; otherwise, set it to 0; Backpropagation operation: For the terminal node and the current leaf node reached during the rolling process, the Monte Carlo tree search performs a backpropagation update on each (s, a) on the path from the current node to the root node: N(s,a)←N(s,a)+1 Where s represents the state, i.e. the node of the Monte Carlo tree, a represents the action, i.e. the model sampling behavior, N(s,a) represents the access count of the two, Q(s,a) represents the action value of the two, and V(s) represents the value function for state s.

7. The multimodal reasoning method based on Monte Carlo tree and dynamic retrieval as claimed in claim 6, characterized in that: The implementation method of the process reward modeling is as follows: a two-stage method is constructed for reward modeling. In the first stage, the model learns to distinguish the correctness of the reasoning steps and performs step-level DPO pre-alignment: a batch of positive and negative sample pairs are naturally generated in each round of expansion and evaluation, and the preference is aligned using step-by-step direct preference optimization as the training objective; In state i, given a multimodal query and the set of sampled inference paths And the corresponding value set Set the value v j The solution paths with values ​​> 0.8 are filtered as positive samples. The value is v j = 0 is considered a negative sample. For each question K pairs of stepwise preference pairs are obtained and their inference discrimination abilities are aligned according to the stepwise DPO as follows: Among them, the reference model π ref The initial setting is π θ , and remains unchanged throughout the training process, β is a hyperparameter, σ is the sigmoid function, The goal is to maximize the preference sample y + Relative to the unpreferred sample y - The probability of In the second phase, the model learns to assign scores to each step, performing point-wise fine-tuning: a step-wise cross entropy objective is applied to the pre-aligned reward model with the following parameters: in, is the true label (0 or 1), r i is the sigmoid score assigned by the reward model; In the stage, the fine-tuned reward model is used, and the reward model score is used as the value of each step in the evaluation stage. Point-by-point soft labels are adopted to extract the nodes with the highest scores from the K extended reasoning paths in each round, and other low-quality paths are discarded. In addition, an early stopping criterion is set to 4, and the final result is directly obtained in the 4th round.

Images