Large model adaptive reasoning method based on Monte Carlo tree search
By using tree structure and UCT algorithm to select nodes in large language model inference, and combining simulation and backpropagation for reward updates, the problem of failure to efficiently utilize Monte Carlo tree structure in the existing technology is solved, and a more efficient and accurate inference process is achieved.
Patent Information
- Application Number
- CN202510040263.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-10
- Publication Date
- 2025-04-15
AI Technical Summary
When using Monte Carlo Tree Search (MCTS) for large language model inference, the existing technology failed to efficiently utilize the generated tree structure, rely only on the previous inference node, and lacked the utilization of other exploration paths.
The inference process of a large language model is organized using a tree structure, the node to be expanded is selected using the UCT algorithm, and when expanding a new node, all the information on the current inference path is used to generate the next inference step. Reward updates are performed through simulation and backpropagation, new nodes are generated and fine-tuned, and finally reinforcement learning is used using the PPO algorithm.
By building a global reasoning tree, the adaptive selection ability of the big model in the inference process is improved, the quality and efficiency of reasoning are improved, and the time-consuming problem of direct application of MCTS is avoided.
Smart Images
Figure CN119831050A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of adaptive reasoning technology, and in particular to a large model adaptive reasoning method based on Monte Carlo tree search. Background Art
[0002] With the development of technology, there are many methods that apply Monte Carlo tree search to large language models for data generation, model training, and as an inference framework. However, the existing methods do not make efficient use of the generated Monte Carlo tree structure. During the inference or training process, only the previous inference node is used, and other existing exploration paths are missing.
[0003] Therefore, it is necessary to provide a large model adaptive reasoning method based on Monte Carlo tree search, which can select parent nodes according to the current reasoning state and add new reasoning nodes to the nodes, and finally generate a more accurate and reasonable answer. Summary of the invention
[0004] The purpose of the present invention is to provide a large model adaptive reasoning method based on Monte Carlo tree search, which can select parent nodes according to the current reasoning state and add new reasoning nodes to the nodes, and finally generate a more accurate and reasonable answer.
[0005] In order to solve the problems existing in the prior art, the present invention provides a large model adaptive reasoning method based on Monte Carlo tree search, which is characterized by comprising the following steps:
[0006] A tree structure is used to organize the reasoning process of a large language model, where each step of reasoning is represented by a node in the tree.
[0007] Use the UCT algorithm to select the nodes to be expanded;
[0008] When expanding a new node, the next reasoning step is generated for the currently selected node using all the information on the current reasoning path;
[0009] Execute the simulation, continue reasoning based on the existing reasoning steps until the final answer is obtained, and determine whether the final answer is correct;
[0010] Back propagation is performed according to the correct final answer. After the reward of the node on the current reasoning path is updated, the tree structure before the new node is generated is used as a prompt word. The tree structure before the new node is generated includes the reasoning process and the connection relationship between the nodes. The node generated according to the prompt word is used as the generated data;
[0011] Fine-tune the large model according to the generated data, so that the large model can reasonably select a subsequent path after fine-tuning;
[0012] Use the PPO algorithm to perform reinforcement learning on large models.
[0013] Optionally, in the large model adaptive reasoning method based on Monte Carlo tree search, the tree structure includes a current node number, a parent node number and a current node reasoning process.
[0014] Optionally, in the large model adaptive reasoning method based on Monte Carlo tree search, the PPO algorithm is used to perform reinforcement learning on the large model in the following manner:
[0015] Initialize with the initial question as the root node;
[0016] Let the large model start reasoning in the form of MCTS. The reasoning tree selects the next reasoning node and the reasoning path it depends on according to the current reasoning situation.
[0017] After the big model gives the final answer or reaches the termination condition, the reward obtained through the MCTS method is used to update the reward for each step of the reasoning process of the big model to guide the optimization of the big model.
[0018] Compared with the prior art, the present invention has the following advantages:
[0019] Through the fine-tuning process, the large model can learn the logical relationship between nodes in the existing reasoning tree structure, and adaptively select the next reasoning path based on the reasoning context. After the large model has the initial ability to select reasoning paths, the reinforcement learning process further improves the large model's adaptive selection and arrangement of future reasoning paths during the reasoning process. Through the reward mechanism and dynamic reward feedback, the large model builds a global reasoning tree during the reasoning process to find a better solution. After the two steps of fine-tuning and reinforcement learning, the large model can complete the reasoning of complex problems by building a tree, which improves the quality and efficiency of reasoning, and also avoids the problem of long time consumption when MCTS is directly applied to reasoning. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 A flowchart of a large model adaptive reasoning method provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0021] The specific implementation of the present invention will be described in more detail below in conjunction with the schematic diagram. The advantages and features of the present invention will become clearer based on the following description. It should be noted that the drawings are all in a very simplified form and are not in exact proportions, and are only used to facilitate and clearly assist in explaining the purpose of the embodiments of the present invention.
[0022] Hereinafter, if the method described herein includes a series of steps, the order in which the steps are presented herein is not necessarily the only order in which the steps may be performed, and some of the steps described may be omitted and / or some other steps not described herein may be added to the method.
[0023] With the development of technology, there are many methods that apply Monte Carlo tree search to large language models for data generation, model training, and as an inference framework. However, the existing methods do not make efficient use of the generated Monte Carlo tree structure. During the inference or training process, only the previous inference node is used, and other existing exploration paths are missing.
[0024] In order to solve the problems existing in the prior art, the present invention provides a large model adaptive reasoning method based on Monte Carlo tree search. The present invention mainly describes the large model adaptive reasoning path learning method from the data generation step, the large model fine-tuning step and the reinforcement learning step.
[0025] like Figure 1 As shown, the large model adaptive reasoning method includes the following steps:
[0026] A tree structure is used to organize the reasoning process of a large language model, where each step of reasoning is represented by each node in the tree, wherein the tree structure includes the current node number, the parent node number, and the current node reasoning process.
[0027] The UCT algorithm is used to select nodes to be expanded. The UCT algorithm (Upper Confidence Bound Apply to Tree), also known as the upper confidence interval algorithm, is a game tree search algorithm. This algorithm combines the Monte-Carlo Tree Search (MCTS) method with the UCB (Upper Confidence Bound) formula. In the search process of ultra-large-scale game trees, it has advantages in time and space compared to traditional search algorithms.
[0028] When expanding a new node, the next reasoning step is generated for the currently selected node using all the information on the current reasoning path;
[0029] Execute the simulation, continue reasoning based on the existing reasoning steps until the final answer is obtained, and determine whether the final answer is correct;
[0030] Back propagation is performed according to the correct final answer. After the reward of the node on the current reasoning path is updated, the tree structure before the new node is generated is used as a prompt word. The tree structure before the new node is generated includes the reasoning process and the connection relationship between the nodes. The node generated according to the prompt word is used as the generated data;
[0031] The large model is supervised fine-tuned based on the generated data. After fine-tuning, the large model can reasonably select subsequent paths and optimize the expansion quality of the overall inference tree. Supervised fine-tuning is SFT (Supervised Fine-Tun-ing), which is a common deep learning strategy and is usually used on pre-trained large language models.
[0032] Use the PPO algorithm to perform reinforcement learning on large models. The PPO algorithm (Proximal Policy Optimization) is a reinforcement learning algorithm that aims to solve the problems of instability and low sample efficiency in policy optimization. During reinforcement learning training, the process of generating a complete MCTS tree is used as a mini-batch. MCTS stands for Monte Carlo Tree Search, which is an algorithm used for decision-making processes, especially in the fields of games and artificial intelligence. It selects the best action by simulating and evaluating the pros and cons of decisions.
[0033] Optionally, in the large model adaptive reasoning method based on Monte Carlo tree search, the PPO algorithm is used to perform reinforcement learning on the large model in the following manner:
[0034] Initialize with the initial question as the root node;
[0035] Let the large model start reasoning in the form of MCTS. The reasoning tree selects the next reasoning node and the reasoning path it depends on according to the current reasoning situation.
[0036] After the big model gives the final answer or reaches the termination condition, the reward obtained through the MCTS method is used to update the reward for each step of the reasoning process of the big model to guide the optimization of the big model.
[0037] By performing multiple rounds of iterative training in this way, the ability of the large model to build high-quality inference trees and adaptively select inference paths is continuously optimized.
[0038] In summary, compared with the prior art, the present invention has the following advantages:
[0039] Through the fine-tuning process, the large model can learn the logical relationship between nodes in the existing reasoning tree structure, and adaptively select the next reasoning path based on the reasoning context. After the large model has the initial ability to select reasoning paths, the reinforcement learning process further improves the large model's adaptive selection and arrangement of future reasoning paths during the reasoning process. Through the reward mechanism and dynamic reward feedback, the large model builds a global reasoning tree during the reasoning process to find a better solution. After the two steps of fine-tuning and reinforcement learning, the large model can complete the reasoning of complex problems by building a tree, which improves the quality and efficiency of reasoning, and also avoids the problem of long time consumption when MCTS is directly applied to reasoning.
[0040] The above is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any technician in the relevant technical field, without departing from the scope of the technical solution of the present invention, makes any form of equivalent replacement or modification to the technical solution and technical content disclosed in the present invention, which does not depart from the content of the technical solution of the present invention and still falls within the protection scope of the present invention.
Claims
1. A large model adaptive reasoning method based on Monte Carlo tree search, characterized in that: The following steps are involved: A tree structure is used to organize the reasoning process of a large language model, where each step of reasoning is represented by a node in the tree. Use the UCT algorithm to select the nodes to be expanded; When expanding a new node, the next reasoning step is generated for the currently selected node using all the information on the current reasoning path; Execute the simulation, continue reasoning based on the existing reasoning steps until the final answer is obtained, and determine whether the final answer is correct; Back propagation is performed according to the correct final answer. After rewarding the nodes on the current reasoning path, the tree structure before the new node is generated is used as a prompt word. The tree structure before the new node is generated includes the reasoning process and the connection relationship between the nodes. The node generated according to the prompt word is used as the generated data; Fine-tune the large model according to the generated data, so that the large model can reasonably select a subsequent path after fine-tuning; Use the PPO algorithm to perform reinforcement learning on large models.
2. The large model adaptive reasoning method based on Monte Carlo tree search according to claim 1, characterized in that: The tree structure includes the current node number, the parent node number, and the current node reasoning process.
3. The large model adaptive reasoning method based on Monte Carlo tree search according to claim 1, characterized in that: Use the PPO algorithm to perform reinforcement learning on large models as follows: Initialize with the initial question as the root node; Let the large model start reasoning in the form of MCTS. The reasoning tree selects the next reasoning node and the reasoning path it depends on according to the current reasoning situation. After the big model gives the final answer or reaches the termination condition, the reward obtained through the MCTS method is used to update the reward for each step of the reasoning process of the big model to guide the optimization of the big model.
Citation Information
Cited By
Civil case cause prediction method based on large model
CN120123517A
Multi-agent cooperation capability improvement method based on agent self-supervised learning
CN121072662A
A multi-agent cooperation capability improvement method based on agent self-supervised learning
CN121072662B