Large vision-language model reasoning method based on multi-modal tree search

Abstract

The invention provides a large vision-language model reasoning method based on multi-modal tree search, which is characterized in that multi-step multi-modal auxiliary information is called and generated through an auxiliary tool, a candidate reasoning path of a target task is simulated and evaluated in combination with a prediction expansion tree search mechanism, and an optimal path is selected through self-voting, so that the reasoning efficiency is improved. Therefore, through a vision-text interleaving reasoning framework and an extension strategy during testing, vision and text information are fully utilized, the reasoning ability of a large vision-language model in a complex multi-step reasoning task is remarkably improved, a more accurate reasoning result can be obtained, model fine adjustment is not needed, and the reasoning efficiency is improved. Therefore, the reasoning result can be obtained more quickly, and the scheme of the invention has good practical application value.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a large-scale vision-language model reasoning method based on multimodal tree search. Background Art

[0002] With the development of artificial intelligence (AI), large language models and multimodal models have been widely applied to various tasks. These large models have achieved considerable success in language understanding, information retrieval, and content generation. However, for complex multi-step reasoning tasks, such as solving complex geometric problems and spatial navigation, the performance of existing large models remains unsatisfactory and needs improvement.

[0003] Some existing large models use traditional reasoning methods, such as Chain-of-Thought. This traditional reasoning method relies only on unimodal text reasoning and cannot effectively utilize visual information. Therefore, it performs poorly in such complex multi-step reasoning tasks and is prone to reasoning paths that are similar in language but incorrect in facts.

[0004] Other existing large models combine visual assistance methods, such as Visual Sketchpad, which enables multimodal large models to draw using elements such as lines and marks that are closer to human drawing methods, thereby better facilitating reasoning. However, existing visual assistance methods of this type only generate single-step static visual prompt information, resulting in the lack of dynamic reasoning capabilities of large models.

[0005] At present, some large models have been combined with test-time expansion strategies, such as using Monte Carlo Tree Search (MCTS) to simulate and generate decision trees to find the best decision solution. However, such methods do not fully integrate multimodal information, resulting in low reasoning efficiency.

[0006] Therefore, in order to process complex multi-step reasoning tasks more efficiently and obtain more accurate reasoning results, a new reasoning method is needed. Summary of the Invention

[0007] To address the above issues, a large-scale vision-language model reasoning method based on multimodal tree search is provided. This method dynamically integrates visual and textual information to achieve complex reasoning tasks. The present invention adopts the following technical solutions:

[0008] The present invention provides a large-scale visual-language model reasoning method based on multimodal tree search, which has the following technical features: step S1, calling an auxiliary tool to generate multimodal auxiliary information based on input data of a target task, including text features and / or image features related to the target task; step S2, inputting the multimodal auxiliary information into a large-scale visual-language model, and the model performs multi-step simulation on candidate reasoning paths through an expansion tree search mechanism to obtain multiple candidate reasoning paths and corresponding simulation results; step S3, inputting the multiple candidate reasoning paths and their corresponding simulation results into a language model, and the model generates an optimal reasoning path through deterministic sampling, and obtains a prediction result of the target task based on the optimal reasoning path; step S4, verifying the prediction result through a verification tool; step S5, when the verification fails, regenerating the candidate reasoning path through a dynamic backtracking mechanism, and returning to step S4 for verification again.

[0009] The large-scale visual-language model reasoning method based on multimodal tree search provided by the present invention may also have such technical features, wherein step S2 includes the following sub-steps: step S2-1, the large-scale visual-language model generates several candidate child nodes of the initial reasoning node based on the multimodal auxiliary information, wherein the initial reasoning node is a description of the target task, and each of the candidate child nodes contains the current visual features and text reasoning trajectory; step S2-2, the reasoning chain is expanded through the large language model, and a multi-step reasoning simulation is performed for each candidate child node through Monte Carlo simulation until the termination state to obtain the simulation result, wherein the auxiliary tool is called in each step of the reasoning simulation to generate new multimodal auxiliary information; step S2-3, the priority score of each candidate child node is calculated based on the corresponding simulation result.

[0010] The large-scale visual-language model reasoning method based on multimodal tree search provided by the present invention may also have such a technical feature, wherein, in step S2-1, a breadth-first strategy is adopted to generate 3 to 5 candidate child nodes.

[0011] The large-scale visual-language model inference method based on multimodal tree search provided by the present invention may also have such a technical feature, wherein, in step S3, the multiple candidate inference paths, the current state of the child node and the simulation results are input into the language model, and the optimal inference path is generated by deterministic sampling with a temperature parameter τ=0.8 and a kernel sampling parameter p=0.8, and voting is performed using a large multimodal model, with the sampling temperature set to 0 and the number of votes set to 1.

[0012] The large-scale visual-language model inference method based on multimodal tree search provided by the present invention may also have such technical features, wherein the target task is a spatial navigation task, and in step S1, a real-time spatial state diagram of the spatial navigation task is obtained through an environmental interaction tool, and is encoded into the text features and / or the image features that can be processed by the large-scale visual-language model; in step S2, the generated simulation result is a simulated path planning; in step S4, the prediction result is a predicted path planning, and the spatial operation feasibility of the predicted path planning is verified by an environmental simulator.

[0013] The large-scale visual-language model inference method based on multimodal tree search provided by the present invention may also have such technical features, wherein, when used for spatial navigation tasks, in step S1, the environmental interaction tool is an Executor constructed in Python, and in step S4, the predicted path planning is executed in the Executor environment constructed in Python to verify the feasibility of the spatial operation, including the feasibility of environmental actions and collision obstacles.

[0014] The large-scale visual-language model reasoning method based on multimodal tree search provided by the present invention may also have the following technical features: wherein the target task is a geometric problem solving task. In step S1, geometric auxiliary lines in the geometric problem are generated using a drawing tool, and a drawing containing the geometric problem and the geometric auxiliary lines is output and encoded as the text features and / or image features that can be processed by the large-scale visual-language model; in step S2, the generated simulation result is a geometric relationship equation; and in step S4, the mathematical relationship of the geometric relationship equation is verified using a geometric equation solver.

[0015] The large-scale visual-language model reasoning method based on multimodal tree search provided by the present invention may also have such a technical feature, wherein, when used for geometric problem solving tasks, in step S1, the drawing tool is the Python Matplotlib library; in step S4, the geometric equation solver is the SymPy solver.

[0016] Functions and effects of the invention

[0017] According to the large-scale visual-language model reasoning method based on multimodal tree search provided by the present invention, multi-step multimodal auxiliary information is generated by calling an auxiliary tool, and the candidate reasoning paths of the target task are simulated and evaluated in combination with the predictive expansion tree search mechanism, and the optimal path is selected by self-voting. Therefore, through the visual-text interwoven reasoning framework and test-time expansion strategy, visual and text information are fully utilized, which significantly improves the reasoning ability of large-scale visual-language models in complex multi-step reasoning tasks, can obtain more accurate reasoning results, and no model fine-tuning is required, so the reasoning results can be obtained more quickly, making the solution of the present invention have good practical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a flow chart of a large-scale visual-language model reasoning method based on multimodal tree search in an embodiment of the present invention;

[0019] Figure 2 4 is a flow chart of step S2 in an embodiment of the present invention. DETAILED DESCRIPTION

[0020] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the following is a detailed description of the large-scale visual-language model reasoning method based on multimodal tree search of the present invention in combination with embodiments and drawings.

[0021] <Example 1>

[0022] Figure 1 4 is a flowchart of a large-scale visual-language model reasoning method based on multimodal tree search in this embodiment.

[0023] like Figure 1 As shown, the method of this embodiment includes the following steps:

[0024] Step S1: calling a tool to generate multimodal auxiliary information based on input data of a target task, where the multimodal auxiliary information includes text features and image features related to the target task.

[0025] In step S2, the generated multimodal auxiliary information is input into a large-scale vision-language model, which performs multi-step simulation on the candidate reasoning paths through an expansion tree search mechanism to obtain multiple candidate reasoning paths and corresponding simulation results.

[0026] In step S3, multiple candidate reasoning paths and their corresponding simulation results are input into the language model. The model generates an optimal path through deterministic sampling and obtains a prediction result of the target task based on the optimal path.

[0027] Step S4: Verify the prediction result of the target task through a verification tool.

[0028] Step S5: When verification fails, the candidate reasoning path is regenerated through the dynamic backtracking mechanism, and the process returns to step S4 for verification again.

[0029] The above steps will be described in detail below.

[0030] Step S1: Calling an auxiliary tool to generate multimodal auxiliary information based on input data of a target task, where the multimodal auxiliary information includes text features and image features related to the target task.

[0031] In this embodiment, the target task is a spatial navigation task, and the input data of the target task includes the initially set target position data and the state data during the navigation process. The target position data can be an image or text description of the target position, etc. In step S1, the auxiliary tool is an environment interaction tool (environment interaction API), which obtains the real-time state space diagram during the spatial navigation task in real time through the environment interaction API, and encodes the state space diagram into text features and / or image features that can be processed by a large visual-language model as multimodal auxiliary information.

[0032] For example, a spatial navigation task involves navigating a large language model (LLM)-driven agent (LLM Agent) to a specified target location in a simulated space. The Environment Interaction API uses an Executor built in Python to acquire real-time state data from the environment. This state data includes, for example, environmental information acquired during movement, such as the current location and nearby obstacles. Based on this state data, Python code can be used to generate a state space diagram in PNG format. The text generated by encoding the state space diagram contains the results of the previous movement, including whether the move was successful and the execution log. The generated image features are the current image and the agent's own location.

[0033] In step S2, the generated multimodal auxiliary information is input into a large-scale vision-language model, which performs multi-step simulation on the candidate reasoning paths through an expansion tree search mechanism to obtain multiple candidate reasoning paths and corresponding simulation results.

[0034] The reasoning task is represented as a tree structure, where each node in the tree represents an inference step, and each edge represents the inference process from the previous inference step to the next inference step. The root node of the tree (initial inference node) is the description of the target task, and the leaf node is the final inference result, which is the path planning prediction of the spatial navigation task in this embodiment.

[0035] Figure 2 This is a flow chart of step S2 in this embodiment.

[0036] like Figure 2As shown, step S2 specifically includes the following sub-steps:

[0037] In step S2-1, the large-scale vision-language model uses a breadth-first strategy to generate several candidate child nodes of the initial inference node based on multimodal auxiliary information. Each candidate child node contains the current visual features and text reasoning trajectory.

[0038] In this embodiment, at the initial inference node, a breadth-first strategy is adopted to generate 3 to 5 candidate child nodes, preferably 3.

[0039] In step S2-2, the inference chain is expanded using the large language model. For each candidate child node, a multi-step Monte Carlo simulation is performed until the state is terminated, resulting in the simulation result. During each step of the inference simulation, auxiliary tools are used to generate new multimodal auxiliary information and update the state space diagram.

[0040] In this embodiment, each subnode corresponds to an inference step based on the current state space diagram in the spatial navigation task. The current visual features contained in each subnode are the spatial position of the agent in the current environment. The simulation results obtained by performing Monte Carlo simulation on each candidate subnode include multiple inference steps and the final simulation path planning.

[0041] Step S2-3: Calculate the priority score of each candidate child node based on the simulation results.

[0042] In this embodiment, the scoring is performed by a self-voting algorithm, and the VLM obtains the candidate node with the most selections through multiple selections.

[0043] In step S3, multiple candidate reasoning paths and their corresponding simulation results are input into the language model. The model generates an optimal reasoning path through deterministic sampling and obtains a prediction result of the target task based on the optimal reasoning path.

[0044] In this example, for spatial navigation tasks, each candidate reasoning path (historical reasoning path), current subnode state, and simulation results are input into the language model. The language model aggregates the historical path weights of each candidate reasoning path with the current simulation results and generates the optimal path through deterministic sampling with a temperature parameter τ = 0.8 and a kernel sampling parameter p = 0.8. Voting is then performed using the multimodal large model, with the model sampling temperature set to 0 and the number of votes set to 1.

[0045] Step S4: Verify the prediction result of the target task through a verification tool.

[0046] In this embodiment, the prediction result, that is, the predicted path planning of the spatial navigation task, is verified through an environmental simulator to verify the spatial operation feasibility of the predicted path planning. For example, the predicted path planning is executed in an Executor environment built in Python to verify the feasibility of environmental actions and collision obstacles. If there is an impassable situation or a collision obstacle, the verification fails.

[0047] Step S5: When verification fails, the candidate reasoning path is regenerated through the dynamic backtracking mechanism, and the process returns to step S4 for verification again.

[0048] When verification fails, the backtracking algorithm is used to go back to the previous node of the failed node and regenerate the candidate reasoning path.

[0049] Functions and effects of embodiment 1

[0050] According to the visual-language model reasoning method based on multimodal tree search provided in this embodiment, multi-step multimodal auxiliary information (text / image features related to the state-space graph) is generated through the call of the environment interaction tool. The candidate reasoning paths of the spatial navigation task are simulated and evaluated in combination with the predictive expansion tree search mechanism, and the optimal path is selected through self-voting. Therefore, through the visual-text interwoven reasoning framework and test-time expansion strategy, the reasoning ability of large-scale visual-language models in spatial navigation tasks is significantly improved, and more accurate and feasible navigation path planning can be obtained without the need for model fine-tuning, so the navigation path planning can be obtained more quickly.

[0051] <Example 2>

[0052] This embodiment provides a visual-language model reasoning method based on multimodal tree search. In this embodiment, the same symbols are given to the same components as in the first embodiment and the corresponding descriptions are omitted.

[0053] Compared with the first embodiment, the difference is that the method of this embodiment is used for solving geometric problems, so the specific implementation of each step is different from that of the first embodiment. Each step will be described in detail below.

[0054] Step S1: Generate multimodal auxiliary information based on the input data of the target task.

[0055] In this example, the target task is to solve a geometry problem, and auxiliary lines for the geometry problem are generated using drawing tools. For example, the auxiliary lines for the geometry problem are generated using the Python Matplotlib library (version ≥ 3.7), and the output is a PNG format image containing the geometry problem drawing and the auxiliary lines, with a resolution of ≥ 300 dpi. Similarly, the output PNG image is then encoded into text features and / or image features that the model can process.

[0056] In step S2, the generated multimodal auxiliary information is input into a large-scale vision-language model, which performs multi-step simulation on the candidate reasoning paths through an expansion tree search mechanism to obtain multiple candidate reasoning paths and corresponding simulation results.

[0057] In this embodiment, each sub-node corresponds to an inference step in the geometric problem solving task based on the current geometric problem drawing. The simulation result obtained by performing Monte Carlo simulation on it includes multiple inference steps and the final simulation solution result. The solution result is, for example, a geometric relationship equation.

[0058] In step S3, multiple candidate reasoning paths and their corresponding simulation results are input into the language model. The model generates an optimal path selection through deterministic sampling, and obtains a prediction result of the target task based on the optimal path selection.

[0059] In this example, historical path weights and current simulation results are aggregated, and the optimal path is generated through deterministic sampling with a temperature parameter τ = 0.8 and kernel sampling p = 0.8. Voting is then performed using the multimodal large model, with the model sampling temperature set to 0 and the number of votes set to 1.

[0060] Step S4: Verify the prediction result of the target task through a verification tool.

[0061] In this embodiment, verification is performed using a geometric problem solver, for example, a SymPy solver is used to verify the solved geometric relationship equation, which supports solving simple equations.

[0062] Step S5: When verification fails, the candidate reasoning path is regenerated through the dynamic backtracking mechanism, and the process returns to step S4 for verification again.

[0063] In this embodiment, similarly, when verification fails, the backtracking algorithm is triggered to go back to the previous node and regenerate the candidate reasoning path.

[0064] Functions and effects of Example 2

[0065] According to the visual-language model reasoning method based on multimodal tree search provided in this embodiment, multi-step multimodal auxiliary information (text / image features related to geometric auxiliary lines) is generated through drawing tool calls, and the candidate reasoning paths of geometric problems are simulated and evaluated in combination with the predictive expansion tree search mechanism, and the optimal path is selected through self-voting. Therefore, through the visual-text interwoven reasoning framework and test-time expansion strategy, the reasoning ability of large-scale visual-language models in geometric problem-solving tasks is significantly improved, and more accurate solution results can be obtained. Without the need for model fine-tuning, the solution results can be obtained more quickly.

[0066] As shown in the above embodiments, the method of the present invention is applicable to scenarios such as large-scale vision-language model reasoning path planning and mathematical problem solving, and has the advantages of high precision, strong generalization and low implementation cost.

[0067] The above embodiments are merely illustrative of specific implementations of the present invention, and the present invention is not limited to the scope of the description of the above embodiments. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and improvements may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and improvements fall within the scope of the present invention as claimed. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. A large-scale vision-language model reasoning method based on multimodal tree search, characterized by: The following steps are involved: Step S1, calling an auxiliary tool to generate multimodal auxiliary information based on input data of a target task, including text features and / or image features related to the target task; Step S2: inputting the multimodal auxiliary information into a large-scale vision-language model, which performs a multi-step simulation on the candidate reasoning paths through an expansion tree search mechanism to obtain multiple candidate reasoning paths and corresponding simulation results; Step S3: inputting the plurality of candidate reasoning paths and their corresponding simulation results into a language model, the model generating an optimal reasoning path through deterministic sampling, and obtaining a prediction result of the target task based on the optimal reasoning path; Step S4, verifying the prediction result using a verification tool; Step S5: When verification fails, the candidate reasoning path is regenerated through the dynamic backtracking mechanism, and the process returns to step S4 for verification again.

2. The large-scale visual-language model reasoning method based on multimodal tree search according to claim 1, Its characteristics are: in, Step S2 includes the following sub-steps: Step S2-1: The large-scale vision-language model generates several candidate child nodes of an initial inference node based on the multimodal auxiliary information, wherein the initial inference node is a description of the target task, and each candidate child node includes current visual features and textual inference trajectories; Step S2-2, expanding the reasoning chain through the large language model, performing a multi-step reasoning simulation on each candidate child node through Monte Carlo simulation until a termination state is reached, and obtaining the simulation result, wherein the auxiliary tool is called in each step of the reasoning simulation to generate new multimodal auxiliary information; Step S2-3: Calculate the priority score of each candidate child node based on the corresponding simulation results.

3. The large-scale visual-language model inference method based on multimodal tree search according to claim 2, characterized in that: in, In step S2-1, a breadth-first strategy is adopted to generate 3 to 5 candidate child nodes.

4. The large-scale visual-language model inference method based on multimodal tree search according to claim 2, characterized in that: in, In step S3, the multiple candidate reasoning paths, the current status of the child nodes and the simulation results are input into the language model, and the optimal reasoning path is generated by deterministic sampling with a temperature parameter τ = 0.8 and a kernel sampling parameter p = 0.8, and voting is performed using a multimodal large model, with the sampling temperature set to 0 and the number of votes set to 1.

5. The large-scale visual-language model inference method based on multimodal tree search according to any one of claims 1 to 3, characterized in that: in, The target task is a space navigation task, In step S1, a real-time spatial state diagram of the spatial navigation task is obtained through an environment interaction tool, and is encoded into the text features and / or the image features that can be processed by the large-scale vision-language model. In step S2, the generated simulation result is the simulated path planning. In step S4, the prediction result is a predicted path plan, and the spatial operation feasibility of the predicted path plan is verified through an environment simulator.

6. The large-scale visual-language model reasoning method based on multimodal tree search according to claim 5, characterized in that: in, In step S1, the environment interaction tool is an Executor built in Python. In step S4, the predicted path planning is executed in the Executor environment constructed by Python to verify the feasibility of the spatial operation, including the feasibility of environmental actions and collision obstacles.

7. The large-scale vision-language model inference method based on multimodal tree search according to any one of claims 1 to 3, characterized in that: in, The target task is a geometric problem solving task. In step S1, a geometric auxiliary line in the geometric problem is generated by a drawing tool, a drawing containing the geometric problem and the geometric auxiliary line is output, and the drawing is encoded into the text features and / or the image features that can be processed by the large-scale vision-language model. In step S2, the generated simulation results are geometric relationship equations, In step S4, the mathematical relationship of the geometric relationship equation is verified by a geometric equation solver.

8. The large-scale visual-language model reasoning method based on multimodal tree search according to claim 7, characterized in that: in, In step S1, the drawing tool is the Python Matplotlib library. In step S4, the geometric equation solver is a SymPy solver.

Images