Tree search inference method and apparatus based on dynamic model routing, and storage medium

Abstract

The application discloses a tree search reasoning method and device based on dynamic model routing and a storage medium. The method comprises the following steps: generating an online reasoning state set according to a long-chain reasoning task input by a user; traversing each online reasoning state in the online reasoning state set, and calling a preset prediction router to predict an estimated generation value of each candidate reasoning model under a current traversed online reasoning state; calculating a value offset based on an average value of the estimated generation values and a preset initial evaluation value, and determining a reasoning branch expansion number based on the value offset; calling a preset Softmax function, calculating a calling probability of each candidate reasoning model based on the estimated generation values, and determining a target reasoning model from the candidate reasoning models based on the calling probabilities; calling the target reasoning model to generate a plurality of intermediate reasoning paths that match the reasoning branch expansion number and that answer the current traversed online reasoning state; and processing all the online reasoning states in a loop until a target reasoning answer is generated.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a tree search reasoning method, apparatus and storage medium based on dynamic model routing. Background Technology

[0002] With the rapid development of complex reasoning tasks and large-scale intelligent decision-making, tree search reasoning solutions for long-chain complex reasoning tasks have become a key technology in the field of artificial intelligence reasoning.

[0003] Currently, tree search reasoning solutions in related technologies mainly employ fixed reasoning models to perform path deduction. However, fixed reasoning models cannot adapt to the reasoning difficulty corresponding to different reasoning states, easily leading to inconsistent reasoning path quality. Secondly, long-chain reasoning processes are characterized by dynamically changing state difficulty, making it impossible to adaptively match the optimal reasoning model based on real-time reasoning states.

[0004] Therefore, existing tree search reasoning schemes suffer from poor stability of reasoning results. Summary of the Invention

[0005] The main objective of this application is to propose a tree search reasoning method, apparatus, and storage medium based on dynamic model routing, aiming to improve the stability of reasoning results.

[0006] To achieve the above objectives, this application proposes a tree search reasoning method based on dynamic model routing, including: Generate an online reasoning state set based on the long-chain reasoning task input by the user; Traverse each online inference state in the online inference state set and call the preset prediction router to predict the estimated generation value of each candidate inference model in the current traversed online inference state; The value offset is calculated based on the average of the estimated generated values ​​and the preset initial evaluation value, and the number of inference branch expansions is determined based on the value offset. The preset Softmax function is invoked to calculate the invocation probability of each candidate inference model based on the estimated value, and the target inference model is determined from the candidate inference models based on the invocation probability. The target inference model is invoked to generate multiple intermediate inference paths that match the number of inference branch expansions and resolve the current online inference state. The process iterates through all the online reasoning states until the reasoning generates the target reasoning answer.

[0007] In some embodiments, the process of generating an online inference state set based on user-input long-chain inference task further includes: Extract complex reasoning task samples from a pre-set offline high-quality corpus; The multi-step reasoning process of the complex reasoning task sample is modeled as a Markov decision process, and an offline reasoning state set covering different reasoning depths and difficulty gradients is constructed. For any offline reasoning state in the offline reasoning state set, all candidate reasoning models are invoked in parallel to perform Monte Carlo tree search reasoning to the termination state, and the inference path results of each candidate reasoning model are obtained. Calculate the true capability value of each candidate reasoning model under each offline reasoning state based on the results of each of the inference paths; The preset dual loss function is invoked to train the initial prediction router based on the true capability value of each, thus obtaining the preset prediction router.

[0008] In some embodiments, the parallel invocation of all candidate inference models to perform Monte Carlo tree search inference to a termination state, and the resulting inference path of each candidate inference model, includes: The current offline inference state is used as the offline root node, and an independent and isolated inference environment is configured for each candidate inference model. For any candidate inference model, in the corresponding inference environment, the candidate inference model is controlled to perform multiple path simulation expansion operations on the offline root node until the inference reaches the termination state and the inference path result is obtained.

[0009] In some embodiments, calculating the true capability value of each candidate inference model under each offline inference state based on the results of each inference path includes: The cumulative reward score of each candidate inference model is calculated for all path simulation operations from the offline root node to the inference path result, and is used as the real capability value of each candidate inference model under the offline root node. By summarizing the calculation results of all offline reasoning states, the true capability value of each candidate reasoning model under each offline reasoning state is obtained.

[0010] In some embodiments, the step of invoking a preset dual loss function to train an initial prediction router based on each of the true capability values ​​to obtain the preset prediction router includes: For any of the offline inference states, extract the inference state features of the offline inference state, and extract the computing power capacity features of each of the candidate inference models; The inference state features and the computing power capacity features of each candidate inference model are input into the initial prediction router, and the estimated capability value of each candidate inference model is output. The absolute deviation between each of the estimated capability values ​​and the actual capability values ​​is measured by the mean squared error loss, and the relative order of each of the estimated capability values ​​is constrained by the dynamic boundary sorting loss. The preset dual loss function is constructed based on the mean squared error loss and the dynamic boundary sorting loss. The parameters of the initial prediction router are optimized by backpropagation based on the preset dual loss function. The training is continuously iterated until convergence to obtain the preset prediction router.

[0011] In some embodiments, generating an online inference state set based on user-input long-chain inference tasks includes: Receive the long-chain inference task input by the user; The initial text containing the original question stem but not the reasoning process in the long chain reasoning task is used as the initial state of the Markov decision process to construct an online root node; The complete context trajectory generated after each single-step reasoning is completed is used as an intermediate state of the Markov decision process to construct the corresponding online reasoning state. All the online reasoning states are then aggregated to obtain the online reasoning state set.

[0012] In some embodiments, calculating the value offset based on the average of the estimated generated values ​​and the preset initial evaluation value, and determining the number of inference branch expansions based on the value offset, includes: Obtain the initial evaluation value of the online root node, wherein the initial evaluation value is the average value of the predicted generation value of each candidate inference model under the online root node; The average value of each of the estimated generated values ​​corresponding to the current online inference state is calculated as the estimated average value; The value offset of the current traversal online inference state is calculated based on the initial evaluation value and the estimated average value. The number of reasoning branch expansions for the current traversal online reasoning state is determined based on the value offset.

[0013] In some embodiments, the step of calling the target inference model to generate multiple intermediate inference paths that match the number of inference branch expansions and resolve the current online inference state includes: Obtain the reasoning context corresponding to the current online reasoning state; The target inference model is invoked, and multiple intermediate inference paths, with the same number of expansions as the inference branches, are generated in parallel based on the inference context. The preset process reward model is invoked to evaluate the quality of each intermediate inference path and obtain the score results. Based on the scoring results, the Monte Carlo tree search backpropagation strategy is invoked to update the current traversal online inference state and the cumulative value and access count of the parent node.

[0014] This application further proposes a tree search inference device based on dynamic model routing, including: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that are executed by the at least one processor to enable the at least one processor to perform the tree search reasoning method based on dynamic model routing described above.

[0015] This application further proposes a storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, enable the processor to execute the tree search reasoning method based on dynamic model routing described above.

[0016] This application's technical solution generates an online inference state set based on a long-chain inference task input by the user; it traverses each online inference state in the set, calling a preset prediction router to predict the estimated generation value of each candidate inference model in the current traversed online inference state; calculates the value offset based on the average of each estimated generation value and a preset initial evaluation value, and determines the number of inference branch expansions based on the value offset; calls a preset Softmax function to calculate the calling probability of each candidate inference model based on each estimated generation value, and determines the target inference model from among the candidate inference models based on each calling probability; calls the target inference model to match the number of inference branch expansions and solve multiple intermediate inference paths in the current traversed online inference state; iterates through all online inference states until the target inference answer is generated; and uses a preset prediction router to predict the inference capability of each candidate inference model in real time, dynamically adjusts the number of branch expansions based on the value offset, and adaptively schedules the inference model, thereby achieving dynamic control of tree search inference and improving the adaptability of inference models and the stability of inference results for long-chain inference tasks. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating an embodiment of the tree search reasoning method based on dynamic model routing in this application; Figure 2 This is a flowchart illustrating another embodiment of the tree search reasoning method based on dynamic model routing in this application; Figure 3 This is a flowchart illustrating another embodiment of the tree search reasoning method based on dynamic model routing in this application; Figure 4This is a flowchart illustrating another embodiment of the tree search reasoning method based on dynamic model routing in this application; Figure 5 This is a flowchart illustrating another embodiment of the tree search reasoning method based on dynamic model routing in this application; Figure 6 This is a flowchart illustrating another embodiment of the tree search reasoning method based on dynamic model routing in this application; Figure 7 This is a flowchart illustrating another embodiment of the tree search reasoning method based on dynamic model routing in this application; Figure 8 This is a flowchart illustrating another embodiment of the tree search reasoning method based on dynamic model routing in this application; Figure 9 This is a schematic diagram of an embodiment of the tree search reasoning device based on dynamic model routing in this application. Detailed Implementation

[0018] The solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments in this application, and not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0019] It should be noted that all directional indicators (such as up, down, left, right, front, back, etc.) in the embodiments of this application are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicator will also change accordingly.

[0020] It should also be noted that when a component is described as "fixed to" or "set on" another component, it can be directly on the other component or there may be an intervening component present. When a component is described as "connected to" another component, it can be directly connected to the other component or there may be an intervening component present.

[0021] Furthermore, the use of terms such as "first" and "second" in this application is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. Additionally, the technical solutions of the various embodiments can be combined with each other, but only on the basis of being achievable by those skilled in the art. When the combination of technical solutions is contradictory or impossible to implement, such a combination of technical solutions should be considered non-existent and not within the scope of protection claimed in this application.

[0022] This application proposes a tree search reasoning method based on dynamic model routing, referring to... Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the tree search reasoning method based on dynamic model routing according to this application. In some embodiments, the tree search reasoning method based on dynamic model routing includes: Step S110: Generate an online reasoning state set based on the long-chain reasoning task input by the user; Step S120: Traverse each online inference state in the online inference state set and call the preset prediction router to predict the estimated generation value of each candidate inference model in the current traversed online inference state; Step S130: Calculate the value offset based on the average value of each estimated generated value and the preset initial evaluation value, and determine the number of inference branch expansions based on the value offset; Step S140: Call the preset Softmax function to calculate the calling probability of each candidate inference model based on each estimated generated value, and determine the target inference model from each candidate inference model based on each calling probability. Step S150: Call the target inference model to generate multiple intermediate inference paths that match the number of inference branches and solve the current online inference state. Step S160: Process all online reasoning states in a loop until the reasoning generates the target reasoning answer.

[0023] In this embodiment, as Figure 1 As shown, the tree search inference method based on dynamic model routing proposed in this application can be configured as a running program, functional software, or packaged as firmware, control driver, or independent functional module; then deployed to a tree search inference device based on dynamic model routing, enabling the device to run the tree search inference method based on dynamic model routing. Alternatively, it can be configured into other computer devices to enable them to run the tree search inference method based on dynamic model routing. The tree search inference device based on dynamic model routing can be simply referred to as the inference device.

[0024] When running a tree search inference method based on dynamic model routing, the inference device can first generate an online inference state set based on the long-chain inference task input by the user. For example, when a user needs to infer a complex problem, the user first configures the corresponding long-chain inference task based on the complex problem and inputs the long-chain inference task into the inference device. At this time, the inference device can receive the long-chain inference task input by the user, and then generate an online inference state set based on the long-chain inference task input by the user. For example, the inference device can divide the long-chain inference task into multiple consecutive inference steps, and then construct the complete context trajectory of each inference step as the corresponding online inference state, and take the set of all online inference states as the online inference state set.

[0025] After generating the online inference state set, the system can traverse each online inference state in the set and call a preset prediction router to predict the estimated generation value of each candidate inference model under the current traversed online inference state. For example, the inference device traverses each online inference state in the set, and for the currently traversed online inference state (the current traversed online inference state), calls the preset prediction router to predict the estimated generation value of each candidate inference model under the current traversed online inference state, thereby achieving a precise quantitative evaluation of the inference capability of each candidate inference model.

[0026] The value offset is calculated based on the average of the estimated generated values ​​and the preset initial evaluation value, and the number of inference branches to be expanded is determined based on the value offset. For example, the inference device first obtains the preset initial evaluation value, then calculates the average of the estimated generated values ​​of all candidate inference models in the current online inference state, and then calculates the value offset by the difference between the preset initial evaluation value and the average value; then, based on the sign and magnitude of the value offset, it adaptively matches and determines the number of inference branches to be expanded corresponding to the current online inference state.

[0027] The pre-defined Softmax function is invoked to calculate the invocation probability of each candidate inference model based on the estimated generated value, and the target inference model is determined from the candidate inference models based on the invocation probabilities. For example, the pre-defined Softmax function is used to convert a set of arbitrary real numbers into a probability distribution with values ​​in the range (0,1) and a sum equal to 1. The inference device invokes the pre-defined Softmax function, using the estimated generated value corresponding to each candidate inference model as input, normalizes and calculates the invocation probability of each candidate inference model, and selects the optimal model that best fits the current online inference state based on the magnitude of the invocation probability as the target inference model.

[0028] The system calls upon the target inference model to generate multiple intermediate inference paths that match the number of inference branch expansions and resolve the current online inference state. For example, the inference device extracts the inference context information corresponding to the current online inference state, inputs it into the target inference model, and controls the target inference model to generate multiple intermediate inference paths in parallel, each corresponding to a one-to-one number of inference branch expansions. Simultaneously, it calls the process reward model to score the quality of each intermediate inference path and updates the cumulative value and visit count of the current node and its parent node using the Monte Carlo tree search backpropagation mechanism, completing the single node inference expansion and parameter iteration optimization.

[0029] The system iterates through all online inference states until the target inference answer is generated. For example, the inference device continuously iterates through all online inference states, constantly executing model routing, branch expansion, path generation, and value update operations, progressively completing the deduction and expansion of each online inference state until the inference iteration reaches the termination state, generating a complete and compliant target inference answer and outputting it, thus completing the solution of the long-chain inference task.

[0030] This application's technical solution generates an online inference state set based on a long-chain inference task input by the user; it traverses each online inference state in the set, calling a preset prediction router to predict the estimated generation value of each candidate inference model in the current traversed online inference state; calculates the value offset based on the average of each estimated generation value and a preset initial evaluation value, and determines the number of inference branch expansions based on the value offset; calls a preset Softmax function to calculate the calling probability of each candidate inference model based on each estimated generation value, and determines the target inference model from among the candidate inference models based on each calling probability; calls the target inference model to match the number of inference branch expansions and solve multiple intermediate inference paths in the current traversed online inference state; iterates through all online inference states until the target inference answer is generated; and uses a preset prediction router to predict the inference capability of each candidate inference model in real time, dynamically adjusts the number of branch expansions based on the value offset, and adaptively schedules the inference model, thereby achieving dynamic control of tree search inference and improving the adaptability of inference models and the stability of inference results for long-chain inference tasks.

[0031] Reference Figure 2 , Figure 2 This is a flowchart illustrating another embodiment of the tree search inference method based on dynamic model routing in this application. In some embodiments, the process of generating an online inference state set based on the long-chain inference task input by the user, as described above, further includes: Step S170: Extract complex reasoning task samples from a pre-set offline high-quality corpus; Step S171: Model the multi-step reasoning process of complex reasoning task samples as a Markov decision process and construct an offline reasoning state set covering different reasoning depths and difficulty gradients. Step S172: For any offline reasoning state in the offline reasoning state set, all candidate reasoning models are called in parallel to perform Monte Carlo tree search reasoning to the termination state, and the inference path results of each candidate reasoning model are obtained. Step S173: Calculate the true capability value of each candidate reasoning model under each offline reasoning state based on the results of each reasoning path. Step S174: Call the preset dual loss function to train the initial prediction router based on the true capability value, and obtain the preset prediction router.

[0032] In this embodiment, as Figure 2 As shown, a preset prediction router can be trained before executing step S110. First, complex reasoning task samples are extracted from a preset offline high-quality corpus. For example, when a user needs to train a preset prediction router, the user can first configure the preset offline high-quality corpus and then configure it into the inference device. The preset offline high-quality corpus can include a large amount of long-chain complex reasoning data that has been manually verified, has accurate answers, and complete reasoning steps, covering reasoning question types of multiple scenarios and difficulties. The inference device can extract complex reasoning task samples from the preset offline high-quality corpus through random sampling and stratified sampling; ensuring that the complex reasoning task samples cover different reasoning logics and different reasoning depths.

[0033] The multi-step reasoning process of complex reasoning task samples is modeled as a Markov decision process, constructing an offline reasoning state set covering different reasoning depths and difficulty gradients. For example, the reasoning device models complex reasoning task samples, abstracting the multi-step progressive reasoning steps of the complex reasoning task samples as a Markov decision process. In this process, a single reasoning step corresponds to a decision action, changes in the reasoning context correspond to state transitions, and the completion of the problem-solving process corresponds to a termination state. By decomposing and classifying the reasoning states of complex reasoning task samples, an offline reasoning state set covering different reasoning depths and difficulty gradients is constructed.

[0034] For any offline reasoning state in the offline reasoning state set, all candidate reasoning models are invoked in parallel to perform Monte Carlo tree search reasoning until the termination state, obtaining the deduction path results of each candidate reasoning model. For example, the reasoning device traverses each offline reasoning state in the offline reasoning state set, takes the currently traversed offline reasoning state as the current offline reasoning state, and invokes all candidate reasoning models in parallel, starting from the current offline reasoning state, to continuously perform Monte Carlo tree search path simulation expansion operations, continuously completing reasoning expansion and path iteration until the reasoning outputs a complete standard answer and reaches the preset termination state; all deduction paths corresponding to each candidate reasoning model are recorded, obtaining the deduction path results corresponding to each candidate reasoning model.

[0035] The true capability value of each candidate inference model in each offline inference state is calculated based on the results of each inference path. For example, for all inference path results of each candidate inference model, the inference device calls the process reward model to score and evaluate the inference logic, accuracy, and completeness of each simulated unfolding path, and obtains the reward score for each path; the reward scores of all simulated unfolding paths in the same offline inference state are accumulated to calculate the cumulative reward score, which is used as the true capability value of the corresponding candidate inference model in the current offline inference state; the calculation data of all offline inference states are traversed and summarized to obtain the true capability value of each candidate inference model in different inference states.

[0036] The system calls a pre-defined dual loss function to train an initial prediction router based on the true capability value, thus obtaining the pre-defined prediction router. For example, the inference device calls the pre-defined dual loss function to train an initial prediction router based on the true capability value, thus obtaining the pre-defined prediction router.

[0037] Reference Figure 3 , Figure 3 This is a flowchart illustrating another embodiment of the tree search inference method based on dynamic model routing in this application. In some embodiments, the aforementioned parallel invocation of all candidate inference models to perform Monte Carlo tree search inference to the termination state yields the following inference path results for each candidate inference model: Step S180: Take the current offline inference state as the offline root node and configure an independent and isolated inference environment for each candidate inference model. Step S181: For any candidate inference model, in the corresponding inference environment, control the candidate inference model to perform multiple path simulation expansion operations on the offline root node until the inference reaches the termination state and the inference path result is obtained.

[0038] In this embodiment, as Figure 3As shown, when executing step S172, the current offline inference state can be used as the offline root node. The current offline inference state is used as the offline root node, and an independent and isolated inference environment is configured for each candidate inference model. For example, the inference device traverses each offline inference state in the offline inference state set, takes the currently traversed offline inference state as the current offline inference state, and then uses the current offline inference state as the offline root node. This offline root node carries the complete inference context information and serves as the initial state for subsequent Monte Carlo tree search. Simultaneously, for all pre-configured candidate inference models, the inference device allocates independent and data-isolated inference runtime environments for each candidate inference model. Parameters are not shared and path data does not interfere between inference environments, supporting parallel and independent inference of multiple candidate inference models. This ensures that the inference process of each candidate inference model does not affect each other, the inference results are true and reliable, and significantly improves the overall efficiency of offline sample inference.

[0039] For any candidate reasoning model, in the corresponding inference environment, the system controls the candidate reasoning model to perform multiple path simulation expansion operations on the offline root node until the reasoning reaches the termination state and the reasoning path result is obtained. For example, for any candidate reasoning model, in the corresponding inference environment, the inference device controls the candidate reasoning model to start from the offline root node and continuously perform multiple path simulation expansion operations on the current offline reasoning state, completing single-step reasoning generation, state update, and path expansion step by step. In each path expansion process, the reasoning context is continuously enriched, the reasoning steps are iteratively advanced, and tree search simulation inference is continuously carried out in a loop until the reasoning generates a complete problem-solving logic and standard answer, reaching the termination state corresponding to the Markov decision process. When the reasoning reaches the termination state, the iterative expansion of the current path is stopped, all simulation expansion paths of the candidate reasoning model from the offline root node to the termination state are completely saved, and the results are summarized to form the reasoning path result corresponding to the candidate reasoning model.

[0040] Reference Figure 4 , Figure 4 This is a flowchart illustrating another embodiment of the tree search inference method based on dynamic model routing in this application. In some embodiments, the aforementioned calculation of the true capability value of each candidate inference model in each offline inference state based on the results of each inference path includes: Step S190: Calculate the cumulative reward score of each candidate inference model for all path simulation operations from the offline root node to the inference path result, and use it as the real capability value of each candidate inference model under the offline root node. Step S191: Summarize the calculation results of all offline reasoning states to obtain the true capability value of each candidate reasoning model in each offline reasoning state.

[0041] In this embodiment, as Figure 4As shown, when executing step S173, the cumulative reward score for each candidate inference model's simulated expansion operations from the offline root node to the deduction path result can be calculated first. This cumulative reward score is used as the true capability value of each candidate inference model at the offline root node. For example, for each offline root node, the inference device retrieves all simulated deduction paths generated by each candidate inference model at that offline root node. By calling the process reward model, it quantifies and scores the single-step reasoning logic, step completeness, answer accuracy, and logical compliance of each path, obtaining the independent single-path reward score corresponding to each simulated expansion path. For the same candidate inference model, the single-path reward scores of all simulated expansion paths at that offline root node are summed to obtain the overall cumulative reward score, which is defined as the true capability value of the candidate inference model at that offline root node.

[0042] The calculation results of all offline inference states are aggregated to obtain the true capability value of each candidate inference model under each offline inference state. For example, the inference device traverses all offline inference states in the offline inference state set (i.e., traverses all offline root nodes), calculates the true capability value of all candidate inference models under each offline inference state, and uniformly summarizes and organizes the true capability values ​​corresponding to all offline inference states to form a comprehensive, multi-gradient model true capability value dataset. Finally, the true capability value of each candidate inference model under each offline inference state with different inference depths and difficulties is obtained.

[0043] In a preferred embodiment, the inference device first extracts complex reasoning task samples (e.g., math competition problems, logic puzzles, etc.) from a pre-set offline high-quality corpus. The reasoning process of the complex reasoning task samples is formalized as a Markov Decision Process (MDP), where each generated reasoning text fragment is considered an action, and the generated complete context trajectory is defined as an offline reasoning state. The inference device uses a large base model to forward generate complex inference task samples, constructing an offline inference state set covering different difficulty gradients and different inference depths.

[0044] For each offline inference state in the offline inference state set The inference device loads all candidate inference models (candidate inference models can be heterogeneous large language models, such as 7B backbone model, 3B auxiliary model and 1.5B auxiliary model) and performs Monte Carlo tree search independently in a completely isolated parallel environment.

[0045] During the Monte Carlo tree search, each candidate inference model operates in its current offline inference state. As the offline root node, it independently performs Rollout (simulated expansion) operations on multiple paths until the final answer is generated. The inference device uses external rules or a process reward model to verify the correctness of the final result of each Rollout path.

[0046] Each candidate inference model independently completes the current offline inference state. After the Monte Carlo tree search, the inference device calculates the inference state of each candidate inference model based on the search results of each candidate inference model. The actual value of its capabilities. Specifically, for the k-th candidate inference model, the inference device statistically analyzes its offline inference state. The average pass rate or cumulative reward score of all Rollout paths originating from the candidate inference model is used to quantify this value in the offline inference state. The true value of one's abilities .

[0047] Through the aforementioned parallel detection and quantization, the inference device targets each offline inference state. Each obtained a set of real capability values ​​composed of each candidate inference model. The true value of this set of abilities not only accurately reflects the state of offline reasoning. The absolute difficulty was measured, and different candidate inference models were objectively recorded in this offline inference state. The differences in abilities (relative ability order).

[0048] Reference Figure 5 , Figure 5 This is a flowchart illustrating another embodiment of the tree search inference method based on dynamic model routing in this application. In some embodiments, the aforementioned invocation of a preset dual loss function to train an initial prediction router based on the true capability value, and obtaining the preset prediction router, includes: Step S200: For any offline inference state, extract the inference state features of the offline inference state and extract the computing power capacity features of each candidate inference model. Step S201: Input the inference state features and the computing power capacity features of each candidate inference model into the initial prediction router, and output the estimated capability value of each candidate inference model. Step S202: Measure the absolute deviation between each estimated capability value and the actual capability value based on the mean square error loss, and constrain the relative order of each estimated capability value based on the dynamic boundary sorting loss. Step S203: Construct a preset dual loss function based on mean squared error loss and dynamic boundary sorting loss, optimize the parameters of the initial prediction router by backpropagation based on the preset dual loss function, and continue iterative training until convergence to obtain the preset prediction router.

[0049] In this embodiment, as Figure 5 As shown, during step S174, the inference state features of the offline inference state can be extracted first, and the computing power capacity features of each candidate inference model can be extracted. For any offline inference state, the inference state features of that offline inference state are extracted, and the computing power capacity features of each candidate inference model are extracted. For example, the inference device traverses each offline inference state, extracts the inference state features of the offline inference state, and extracts the computing power capacity features of each candidate inference model. Among them, the inference state features are used to characterize the inference depth, task difficulty, semantic complexity, and contextual logic features of the current offline inference state, which can accurately characterize the inherent attributes of different inference scenarios; the computing power capacity features are used to characterize the inherent model attributes of each candidate inference model, such as the upper limit of inference computing power, parameter magnitude, inference rate, and load capacity.

[0050] The inference state features and the computing power capacity features of each candidate inference model are input into the initial prediction router, which outputs the estimated capability value of each candidate inference model. For example, the inference device inputs the inference state features of the offline inference state and the computing power capacity features of each candidate inference model into the initial prediction router. The initial prediction router performs cross-feature fusion operations, automatically learns the correlation mapping relationship between the inference state, model computing power, and model inference capability, and finally outputs the estimated capability value corresponding to each candidate inference model in the offline inference state.

[0051] It is understandable that the estimated capability value and the estimated generation value have the same meaning. The only difference is that the output of the predictive router during the training phase is defined as the estimated capability value, while the output of the predictive router during the actual use phase is defined as the estimated generation value.

[0052] The mean squared error loss measures the absolute deviation between each estimated capability value and the actual capability value, and the dynamic boundary ranking loss constrains the relative order of each estimated capability value. For example, the inference device uses the mean squared error loss to calculate the absolute numerical deviation between the estimated capability value and the pre-labeled actual capability value, constraining the predicted value to infinitely approach the actual label, thus ensuring prediction accuracy. At the same time, the dynamic boundary ranking loss constrains the relative order of the estimated capability values ​​of different candidate inference models under the same offline inference state, ensuring that the candidate inference model with stronger inference ability can output a higher estimated capability value, correcting the problem of disordered ranking of candidate inference model predictions, and closely matching the inference performance of the actual candidate inference model.

[0053] A pre-defined dual loss function is constructed based on the mean squared error loss and the dynamic boundary ranking loss. The parameters of the initial prediction router are optimized through backpropagation based on this pre-defined dual loss function, and training is iteratively continued until convergence, resulting in the pre-defined prediction router. For example, the inference device weights and fuses the mean squared error loss and the dynamic boundary ranking loss to construct the pre-defined dual loss function. Then, the overall loss value is calculated based on the pre-defined dual loss function, and the loss gradient is propagated layer by layer through the backpropagation algorithm to iteratively update the network weights and bias parameters of the initial prediction router. The training iteration process of feature input, value prediction, loss calculation, and parameter optimization is repeated continuously until the loss value tends to stabilize and meets the pre-defined convergence threshold. Training is then stopped, and the model parameters are saved, ultimately resulting in a pre-defined prediction router that has been trained and can be directly used for online inference scheduling.

[0054] In a preferred embodiment, for any offline inference state The inference device first inputs the data into the 7B backbone model for forward propagation. The inference device then extracts the hidden state vectors from the deep hidden layers of the 7B backbone model, which serve as the offline inference state. The inference state characteristics are denoted as... Simultaneously, the inference device constructs a one-hot encoding for each candidate inference model and maps it to a low-dimensional continuous vector through a learnable linear embedding layer. This vector serves as the model category embedding representation, explicitly characterizing the intrinsic computational capacity features of the candidate inference model. The computational capacity feature of the k-th candidate inference model is denoted as... .

[0055] To deeply capture the nonlinear interaction and adaptive matching relationship between inference state features and candidate inference models, a specially designed bilinear value head structure is used internally in the initial prediction router; this structure simultaneously receives inference state features. With computing power capacity characteristics Deep cross-fusion is achieved through matrix projection and element-wise multiplication (Hadamard Product). The specific calculation formula is as follows: in, This represents the k-th candidate inference model. Represents the intermediate feature mapping function. This represents the weight matrix of the first-level linear transformation. This represents the bias vector for the first-level linear transformation. It represents the Hadamah accumulation. This represents the weight matrix of the second-level linear transformation. This represents the bias vector for the second-level linear transformation. This represents the hyperbolic tangent activation function. This indicates the offline inference state of the k-th candidate inference model. The estimated value of the capability.

[0056] To prevent online routing disorder caused by the near-similarity of the estimated capabilities of multiple candidate inference models, the mean squared error (MSE) loss between the estimated and actual capability values ​​is calculated for each candidate inference model. This forces the initial prediction router to learn the absolute difficulty of the current cognitive state. Furthermore, it is necessary to limit the prediction discrepancies between candidate inference models by introducing a dynamic boundary ranking loss to prevent overlap of confidence levels between strong and weak candidate inference models, ensuring the relative order preservation of the evaluation system. A pre-defined dual loss function is constructed based on the mean squared error loss and the dynamic boundary ranking loss, specifically: in, This indicates a pre-defined double loss function. This represents the mean squared error loss. Represents the dynamic boundary sorting loss. and To balance the mean squared error loss and the dynamic boundary ranking loss, a weighted hyperparameter is used, where N represents the total number of candidate inference models, and P represents the number of models in the same offline inference state. The following constructs an effective candidate inference model pairing The set, This represents the total number of model pairs within set P. Represents a symbolic function.

[0057] The aforementioned pre-defined dual loss function transforms the loss into two core constraints. On one hand, optimization... The loss is anchored to an absolute benchmark to prevent systematic drift in the initial prediction router's assessment of throughput; on the other hand, optimization is performed on the set P for the effective model. The loss causes a relative difference in the predictive ability value among candidate inference models. It can adaptively match the differences between the actual capabilities and values. And strictly maintain the correct relative capability order boundary. The reasoning device minimizes the total loss. Calculate the gradient, and use the backpropagation algorithm to synchronously update the bilinear value head parameters, state feature mapping matrix, and model category embedding layer parameters of the initial prediction router until the entire routing network fully converges, thus obtaining the preset prediction router.

[0058] Reference Figure 6 , Figure 6 This is a flowchart illustrating another embodiment of the tree search inference method based on dynamic model routing in this application. In some embodiments, the aforementioned generation of an online inference state set based on a long-chain inference task input by the user includes: Step S210: Receive the long chain reasoning task input by the user; Step S211: Take the initial text of the long chain reasoning task that contains the original question stem but does not contain the reasoning process as the initial state of the Markov decision process and construct the online root node. Step S212: Take the complete context trajectory generated after each single-step reasoning as the intermediate state of the Markov decision process, construct the corresponding online reasoning state, and summarize all online reasoning states to obtain the online reasoning state set.

[0059] In this embodiment, as Figure 6 As shown, during step S110, a long-chain inference task input by the user can be received first. For example, the inference device can first receive a long-chain inference task input by the user.

[0060] In a long-chain reasoning task, the initial text containing only the original question stem but without any reasoning process is used as the initial state of the Markov decision process, and an online root node is constructed. For example, the reasoning device extracts the initial text from the long-chain reasoning task that only contains the original question stem and has not yet been accompanied by any reasoning process or derivation conclusion. This initial text is defined as the initial state of the Markov decision process, and an online root node of the reasoning tree is constructed based on this initial state.

[0061] The complete context trajectory generated after each single-step inference is used as an intermediate state of the Markov decision process to construct the corresponding online inference state. All online inference states are then aggregated to obtain the online inference state set. For example, the inference device uses the complete context trajectory generated after each single-step inference as an intermediate state of the Markov decision process to construct the corresponding online inference state. Then, all online inference states are aggregated to obtain the online inference state set.

[0062] Reference Figure 7 , Figure 7 This is a flowchart illustrating another embodiment of the tree search inference method based on dynamic model routing in this application. In some embodiments, the aforementioned calculation of the value offset based on the average of each estimated generated value and the preset initial evaluation value, and the determination of the number of inference branch expansions based on the value offset, includes: Step S220: Obtain the initial evaluation value of the online root node. The initial evaluation value is the average value of the estimated generation value of each candidate inference model under the online root node. Step S221: Calculate the average value of each estimated generated value corresponding to the current traversal online inference state as the estimated average value; Step S222: Calculate the value offset of the current traversal online inference state based on the initial evaluation value and the estimated average value; Step S223: Determine the number of reasoning branch expansions for the current traversal of the online reasoning state based on the value offset.

[0063] In this embodiment, as Figure 7 As shown, when executing step S130, the initial evaluation value of the online root node can be obtained first. The initial evaluation value is the average of the estimated generation values ​​of each candidate inference model under the online root node. For example, after constructing the online root node, the inference device calls a preset prediction router to batch predict the estimated generation values ​​of all candidate inference models under the online root node, and calculates the average of the estimated generation values ​​of all candidate inference models as the initial evaluation value.

[0064] The average of the estimated generated values ​​corresponding to the current online inference state is calculated as the estimated average. For example, when the inference device traverses each online inference state, it obtains the estimated generated values ​​corresponding to all candidate inference models in the current online inference state in real time, and then calculates the average of all estimated generated values ​​to obtain the estimated average for the current online inference state.

[0065] The value offset of the current traversal online inference state is calculated based on the initial evaluation value and the estimated average value. For example, the inference device calculates the value offset of the current traversal online inference state by subtracting the initial evaluation value from the estimated average value. The value offset reflects the difficulty fluctuation of the current traversal online inference state compared to the online root node: a negative value offset indicates that the current traversal online inference state is more difficult than the initial state, with higher inference uncertainty; a positive value offset indicates that the current traversal online inference state is less difficult and more stable.

[0066] The number of inference branch expansions for the current traversal of the online inference state is determined based on the value offset. For example, the inference device is pre-configured with a mapping rule between value offsets and the number of branch expansions, and adaptively matches and determines the number of inference branch expansions for the current traversal of the online inference state based on the calculated value offset.

[0067] Reference Figure 8 , Figure 8This is a flowchart illustrating another embodiment of the tree search inference method based on dynamic model routing in this application. In some embodiments, the aforementioned multiple intermediate inference paths for matching the number of inference branches generated by the target inference model and solving the current online inference state include: Step S230: Obtain the reasoning context corresponding to the current online reasoning state. Step S231: Invoke the target inference model and generate multiple intermediate inference paths in parallel based on the inference context, with the same number of inference branch expansions. Step S232: Call the preset process reward model to evaluate the quality of each intermediate inference path and obtain the score results; Step S233: Based on the scoring results, call the Monte Carlo tree search backpropagation strategy to update the current traversal online inference state and the cumulative value and access count of the parent node.

[0068] In this embodiment, as Figure 8 As shown, when executing step S150, the reasoning context corresponding to the current online reasoning state can be obtained first. For example, the reasoning device obtains the reasoning context corresponding to the current online reasoning state. The reasoning context includes complete temporal information such as the user's original input question stem, the completed reasoning steps corresponding to the current reasoning node, historical reasoning logic, and intermediate conclusions.

[0069] The target inference model is invoked, and multiple intermediate inference paths, the same number as the number of inference branch expansions, are generated in parallel based on the inference context. For example, the inference device invokes the target inference model, takes the obtained inference context as input, and, combined with the number of inference branch expansions, generates multiple intermediate inference paths in parallel, each corresponding to one-to-one with the number of inference branch expansions.

[0070] The pre-defined process reward model is invoked to evaluate the quality of each intermediate inference path, resulting in individual scores. For example, the inference device invokes the process reward model to perform a refined quality score evaluation on each intermediate inference path generated by the target inference model. The process reward model primarily uses multi-dimensional indicators such as the logicality of the inference steps, the accuracy of the inference content, the completeness of the inference, semantic compliance, and the degree of matching with task requirements to quantitatively score each intermediate inference path, outputting the corresponding score for each inference path.

[0071] Based on the scoring results, the Monte Carlo tree search backpropagation strategy is invoked to update the cumulative value and access count of the current traversed online inference state and its parent nodes. For example, the inference device uses the Monte Carlo tree search backpropagation mechanism, based on the scoring results of each intermediate inference path, to update the cumulative value and access count of the current traversed online inference state node and all its parent nodes from bottom to top. For instance, the scoring result of a single path is used as a reward for a single inference iteration, accumulating and updating the cumulative value of the current traversed online inference state, and correspondingly updating the access count of the current traversed online inference state, thus achieving real-time correction of the value of the current traversed online inference state. Through multi-path inference and iterative backpropagation updates, the value assessment accuracy of each online inference state is continuously optimized, making subsequent model routing and branch expansion decisions more closely resemble real inference quality, effectively improving the inference accuracy and result stability of the overall long-chain inference task.

[0072] In a preferred embodiment, when the inference device infers a long-chain inference task, it reaches an online inference state. At that time, the inference device first extracts the deep inference state features of the backbone model. And combined with the computing power capacity characteristics of each candidate inference model The system performs parallel forward calls to a lightweight pre-defined prediction router. The pre-defined prediction router's internal specially designed bilinear value head outputs the online inference state in real time. The following are candidate inference models The estimated value generated .

[0073] To avoid redundant computational resources caused by blindly expanding on simple steps, and to ensure sufficient exploration breadth at logical bottlenecks, the inference device performs vertical branching pruning using the estimated generation value of each candidate inference model. The inference device calculates the average of the estimated generation value of each candidate inference model. Combined with the initial evaluation value of the online root node Calculate the online inference state Value offset And based on this, determine the status of the online inference. Number of reasoning branches expanded The specific formula is as follows: Where D represents the maximum expected value offset. This represents the scaling factor for state uncertainty. When the state has extremely high certainty (i.e., the value deviates drastically to a positive or negative direction), When the value is large, the number of inference branches is compressed to a minimum. To save computing power; when the state is in a state of extreme uncertainty, a cognitive bottleneck ( When this happens, the number of inference branches is increased to the maximum value. To fully explore ways to break the deadlock, Indicates the maximum adjustable branch interval span. This represents the floor operator. This indicates a truncation function that restricts the output value to the interval [x, y].

[0074] The inference device dynamically balances the model's capability gap with the computational cost by introducing a soft routing mechanism that incorporates a cost bias coefficient. The inference device executes a capability-aware, pre-defined Softmax function to calculate the online inference state. Assignment to candidate inference models Call probability : in, Representing candidate inference models The computational cost of a single inference operation. This represents the cost bias coefficient. This represents the Softmax temperature coefficient.

[0075] This soft routing decision implements the following three-level heterogeneous computing power on-demand scheduling logic: Dealing with highly complex logical challenges (extremely large capability gaps): When predicting extremely large capability differences between models... The difference-driven formula prioritizes the use of the 7B backbone model with high confidence, driving its deep parameters to explore in a wide range of depths in order to break through the difficulties. To handle moderately difficult transitional derivations: When the predicted difference between models is moderate, the scheduling authority of the inference device is smoothly transferred to the 3B auxiliary model, providing cost-effective computing power support while maintaining the correct inference direction. Handling low-difficulty mechanized output: When the difference between predictions between models is small With the weight of the model highlighted, the inference device will have a very high probability of directly delegating the computational task to the 1.5B auxiliary model, with extremely fast throughput and zero-latency offloading.

[0076] The inference device is based on the probability of being invoked. The final target inference model is selected. Subsequently, this target inference model is rigorously driven in the current online inference state. The following generation The inference device generates intermediate inference paths, thereby completing the two-dimensional adaptive node expansion of the search tree at the current depth. After the intermediate inference paths are generated by the target inference model, the inference device calls the process reward model to score the quality of each intermediate inference path, and refreshes the cumulative value and access count of each level of parent node from bottom to top according to the traditional backpropagation mechanism of Monte Carlo trees. Through this closed loop, the inference device jumps to the next incompletely explored online inference state, repeats the model routing, branch expansion, path generation and value update operations, and completes the inference expansion of each online inference state step by step until the inference iteration reaches the termination state, generates a complete and compliant target inference answer and outputs it, and completes the solution of the long chain inference task.

[0077] This application's technical solution generates an online inference state set based on a long-chain inference task input by the user; it traverses each online inference state in the set, calling a preset prediction router to predict the estimated generation value of each candidate inference model in the current traversed online inference state; calculates the value offset based on the average of each estimated generation value and a preset initial evaluation value, and determines the number of inference branch expansions based on the value offset; calls a preset Softmax function to calculate the calling probability of each candidate inference model based on each estimated generation value, and determines the target inference model from among the candidate inference models based on each calling probability; calls the target inference model to match the number of inference branch expansions and solve multiple intermediate inference paths in the current traversed online inference state; iterates through all online inference states until the target inference answer is generated; and uses a preset prediction router to predict the inference capability of each candidate inference model in real time, dynamically adjusts the number of branch expansions based on the value offset, and adaptively schedules the inference model, thereby achieving dynamic control of tree search inference and improving the adaptability of inference models and the stability of inference results for long-chain inference tasks.

[0078] This application further proposes a tree search reasoning device based on dynamic model routing, referring to... Figure 9 , Figure 9 This is a schematic diagram of the structure of an embodiment of the tree search inference device based on dynamic model routing according to this application. In some embodiments, the tree search inference device based on dynamic model routing includes: At least one processor; and, A memory that is communicatively connected to at least one processor; wherein, The memory stores instructions that are executed by the at least one processor to enable the at least one processor to perform the tree search reasoning method based on dynamic model routing as described above.

[0079] In this embodiment, as Figure 9As shown, the tree search inference apparatus based on dynamic model routing in this application embodiment can be a processor capable of running a tree search inference method based on dynamic model routing; there is at least one processor. Figure 9 As shown, the tree search inference device based on dynamic model routing may include: a processor 1001 (e.g., CPU), a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. The communication bus 1002 is used to establish communication between these components. The user interface 1003 may include a display screen and an input unit, such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0080] Those skilled in the art will understand that Figure 9 The tree search inference apparatus structure based on dynamic model routing shown in the figure does not constitute a limitation on the tree search inference apparatus based on dynamic model routing. It may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0081] like Figure 9 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and computer-executable instructions.

[0082] exist Figure 9 In the tree search inference device based on dynamic model routing shown, the network interface 1004 is mainly used to connect to the backend server and communicate with the backend server; the user interface 1003 is mainly used to connect to the client (user end) and communicate with the client; and the processor 1001 can be used to call computer-executable instructions stored in the memory 1005. When the processor 1001 calls and executes the instructions, it implements the steps of the tree search inference method based on dynamic model routing described above.

[0083] This application further proposes a storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, enable the processor to execute the tree search reasoning method based on dynamic model routing described above.

[0084] The above description is only a part or preferred embodiment of this application. Neither the text nor the drawings should limit the scope of protection of this application. All equivalent structural transformations made using the content of this application's specification and drawings under the overall concept of this application, or direct / indirect applications in other related technical fields, are included within the scope of protection of this application.

Claims

1. A tree search reasoning method based on dynamic model routing, characterized in that, include: Generate an online reasoning state set based on the long-chain reasoning task input by the user; Traverse each online inference state in the online inference state set and call the preset prediction router to predict the estimated generation value of each candidate inference model in the current traversed online inference state; The value offset is calculated based on the average of the estimated generated values ​​and the preset initial evaluation value, and the number of inference branch expansions is determined based on the value offset. The preset Softmax function is invoked to calculate the invocation probability of each candidate inference model based on the estimated value, and the target inference model is determined from the candidate inference models based on the invocation probability. The target inference model is invoked to generate multiple intermediate inference paths that match the number of inference branch expansions and resolve the current online inference state. The process iteratively handles all the aforementioned online reasoning states until the reasoning generates the target reasoning answer; Before generating the online inference state set based on the long-chain inference task input by the user, the following steps are also included: Extract complex reasoning task samples from a pre-set offline high-quality corpus; The multi-step reasoning process of the complex reasoning task sample is modeled as a Markov decision process, and an offline reasoning state set covering different reasoning depths and difficulty gradients is constructed. For any offline reasoning state in the offline reasoning state set, all candidate reasoning models are invoked in parallel to perform Monte Carlo tree search reasoning to the termination state, and the inference path results of each candidate reasoning model are obtained. Calculate the true capability value of each candidate reasoning model under each offline reasoning state based on the results of each of the inference paths; The preset dual loss function is invoked to train an initial prediction router based on the true capability value of each, thus obtaining the preset prediction router; The step of calling a preset dual loss function to train an initial prediction router based on the true capability value of each of the above includes: For any of the offline inference states, extract the inference state features of the offline inference state, and extract the computing power capacity features of each of the candidate inference models; The inference state features and the computing power capacity features of each candidate inference model are input into the initial prediction router, and the estimated capability value of each candidate inference model is output. The absolute deviation between each of the estimated capability values ​​and the actual capability values ​​is measured by the mean squared error loss, and the relative order of each of the estimated capability values ​​is constrained by the dynamic boundary sorting loss. The preset dual loss function is constructed based on the mean squared error loss and the dynamic boundary sorting loss. The parameters of the initial prediction router are optimized by backpropagation based on the preset dual loss function. The training is iterated until convergence to obtain the preset prediction router. The process of generating an online inference state set based on user-input long-chain inference task includes: Receive the long-chain inference task input by the user; The initial text containing the original question stem but not the reasoning process in the long chain reasoning task is used as the initial state of the Markov decision process to construct an online root node; The complete context trajectory generated after each single-step reasoning is completed is used as the intermediate state of the Markov decision process to construct the corresponding online reasoning state. All the online reasoning states are then aggregated to obtain the online reasoning state set. The step of calculating the value offset based on the average of the estimated generated values ​​and the preset initial evaluation value, and determining the number of inference branch expansions based on the value offset, includes: Obtain the initial evaluation value of the online root node, wherein the initial evaluation value is the average value of the predicted generation value of each candidate inference model under the online root node; The average value of each of the estimated generated values ​​corresponding to the current online inference state is calculated as the estimated average value; The value offset of the current traversal online inference state is calculated based on the initial evaluation value and the estimated average value. The number of reasoning branch expansions for the current traversal online reasoning state is determined based on the value offset.

2. The tree search reasoning method based on dynamic model routing according to claim 1, characterized in that, The parallel invocation of all candidate inference models to perform Monte Carlo tree search inference to the termination state yields the following inference path results for each candidate inference model: The current offline inference state is used as the offline root node, and an independent and isolated inference environment is configured for each candidate inference model. For any candidate inference model, in the corresponding inference environment, the candidate inference model is controlled to perform multiple path simulation expansion operations on the offline root node until the inference reaches the termination state and the inference path result is obtained.

3. The tree search reasoning method based on dynamic model routing according to claim 2, characterized in that, The calculation of the true capability value of each candidate reasoning model under each offline reasoning state based on the results of each of the inference paths includes: The cumulative reward score of each candidate inference model is calculated for all path simulation operations from the offline root node to the inference path result, and is used as the real capability value of each candidate inference model under the offline root node. By summarizing the calculation results of all offline reasoning states, the true capability value of each candidate reasoning model under each offline reasoning state is obtained.

4. The tree search reasoning method based on dynamic model routing according to claim 1, characterized in that, The process of calling the target inference model to generate multiple intermediate inference paths that match the number of inference branch expansions and resolve the current online inference state includes: Obtain the reasoning context corresponding to the current online reasoning state; The target inference model is invoked, and multiple intermediate inference paths, with the same number of expansions as the inference branches, are generated in parallel based on the inference context. The preset process reward model is invoked to evaluate the quality of each intermediate inference path and obtain the score results. Based on the scoring results, the Monte Carlo tree search backpropagation strategy is invoked to update the current traversal online inference state and the cumulative value and access count of the parent node.

5. A tree search inference device based on dynamic model routing, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that are executed by the at least one processor to enable the at least one processor to perform the tree search reasoning method based on dynamic model routing as described in any one of claims 1 to 4.

6. A storage medium, characterized in that, The storage medium stores a computer program, which includes program instructions that, when executed by a processor, enable the processor to perform the tree search reasoning method based on dynamic model routing as described in any one of claims 1 to 4.

Images