Inference method, system, device and medium based on heterogeneous model reinforcement learning

Abstract

The application discloses a reasoning method and system based on heterogeneous model reinforcement learning, a device and a medium, and belongs to the technical field of artificial intelligence. The method generates structured tree reasoning knowledge in a low-cost manner based on a Monte Carlo tree search mechanism, introduces a calculation value score to balance accuracy and calculation overhead, and applies a high-order reasoning mode obtained based on the tree reasoning knowledge to a reinforcement learning training process, so that high-order knowledge of a heterogeneous model group is effectively internalized to a first strategy model. Thus, the dual technical problems of high construction cost of high-quality distillation data and lack of multi-source external knowledge guidance in traditional reinforcement learning algorithms can be solved, and the first strategy model can reason the answer to a natural language task through a tree exploration type reasoning process in a reasoning stage. In this way, the generalization ability, exploration ability and decision confidence of the first strategy model can be significantly improved, so that the reasoning ability of the first strategy model is effectively improved.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, specifically relating to a reasoning method, system, device, and medium based on heterogeneous model reinforcement learning. Background Technology

[0002] With the rapid development of Large Language Models (LLMs), they have demonstrated outstanding performance in natural language processing fields such as text generation, machine translation, and question answering systems. Reasoning ability is a key indicator for measuring the cognitive abilities and higher-order skills of artificial intelligence models such as Large Language Models, and its core lies in a thorough understanding of complex environmental inputs and a precise grasp of logical relationships.

[0003] However, in the process of realizing this application, the inventors found that the current large language models have the following defects: high cost of acquiring training data, insufficient generalization ability, exploration ability and decision confidence, resulting in low reasoning ability. Summary of the Invention

[0004] The purpose of this application is to provide a reasoning method, system, device, and medium based on heterogeneous model reinforcement learning, which can solve the problems of high training data acquisition costs and insufficient generalization ability, exploration ability, and decision confidence of current large language models, resulting in low reasoning ability.

[0005] To solve the above-mentioned technical problems, this application is implemented as follows: In a first aspect, embodiments of this application provide an inference method based on heterogeneous model reinforcement learning, the method comprising: During the training phase, for each first natural language task sample, a pre-trained execution model is used to take the first natural language task sample as the root node of the search tree. Starting from the root node, the search tree is iteratively expanded with child nodes based on the Monte Carlo tree search algorithm to obtain multiple inference paths associated with the first natural language task sample. For each first natural language task sample, a pre-trained validation model is used to evaluate the quality of multiple inference paths associated with the first natural language task sample, and the n inference paths with the highest quality scores are determined as candidate inference paths, where n is a pre-determined positive integer; For each candidate inference path associated with each first natural language task sample, the computational value score of the candidate inference path is determined based on the computational complexity and robust accuracy of the candidate inference path. For each first natural language task sample, the candidate reasoning path with the highest calculated value score among the n candidate reasoning paths associated with the first natural language task sample is abstracted to obtain the reasoning thinking pattern representation associated with the first natural language task sample. Based on the reasoning thought pattern representation associated with each of the first natural language task samples, a pre-established first policy model is trained using a reinforcement learning algorithm, wherein the first policy model is established based on a large language model with a different architecture than the execution model and the verification model. In the inference phase, in response to receiving a natural language task, a first inference path is generated using a trained first policy model, and the answer to the natural language task is determined based on the first inference path. The first inference path corresponds to a tree-based exploratory inference process.

[0006] Secondly, embodiments of this application provide a reasoning system based on heterogeneous model reinforcement learning. This system includes a high-order knowledge extraction module, a heterogeneous model knowledge transfer and reasoning ability optimization module, and a reasoning module, wherein: The advanced knowledge extraction module is configured to, during the training phase, for each first natural language task sample, use a pre-trained execution model to take the first natural language task sample as the root node of a search tree, and iteratively expand the search tree from the root node using a Monte Carlo tree search algorithm to obtain multiple inference paths associated with the first natural language task sample; and for each first natural language task sample, use a pre-trained verification model to perform quality evaluation on the multiple inference paths associated with the first natural language task sample, and determine the n inference paths with the highest quality scores as candidate inference paths, where n is a pre-determined positive integer; and for each candidate inference path associated with each first natural language task sample, determine the computational value score of the candidate inference path based on the computational complexity and robustness accuracy of the candidate inference path; and for each first natural language task sample, abstract the candidate inference path with the highest computational value score among the n candidate inference paths associated with the first natural language task sample to obtain a representation of the inference thinking pattern associated with the first natural language task sample. The heterogeneous model knowledge transfer and reasoning ability optimization module is configured to train a pre-established first policy model based on a reinforcement learning algorithm according to the reasoning thinking pattern representation associated with each of the first natural language task samples. The first policy model is established based on a large language model with a different architecture than the execution model and the verification model. The reasoning module is configured to, in response to receiving a natural language task during the reasoning phase, generate a first reasoning path using a trained first policy model, and determine the answer to the natural language task based on the first reasoning path, wherein the first reasoning path corresponds to a tree-based exploratory reasoning process.

[0007] Thirdly, embodiments of this application provide an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the steps of the inference method based on heterogeneous model reinforcement learning as described in the first aspect.

[0008] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the inference method based on heterogeneous model reinforcement learning as described in the first aspect.

[0009] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the inference method based on heterogeneous model reinforcement learning as described in the first aspect.

[0010] In this embodiment, for each first natural language task sample, this application generates structured tree-shaped reasoning knowledge (i.e., the search tree) in a low-cost manner based on the Monte Carlo tree search mechanism, which can overcome the dependence on expensive labeled data and thus reduce the cost of training data acquisition. Then, the high-order reasoning pattern (i.e., the reasoning thinking pattern representation) obtained based on the tree-shaped reasoning knowledge is applied to the reinforcement learning training process, thereby effectively internalizing the high-order knowledge of the heterogeneous model group (i.e., the execution model and the verification model) into the first policy model. This can solve the dual technical problems of high cost of constructing high-quality distilled data and lack of multi-source (or multi-perspective) external knowledge guidance in traditional reinforcement learning algorithms. Moreover, it enables the first policy model to reason about the answer to the natural language task through a tree-shaped exploratory reasoning process during the reasoning stage, which can significantly improve the generalization ability, exploration ability and decision confidence of the first policy model, thereby effectively improving the reasoning ability of the first policy model. Attached Figure Description

[0011] Figure 1 A flowchart illustrating the implementation of an inference method based on heterogeneous model reinforcement learning, provided in this application embodiment; Figure 2 A schematic diagram of a high-level knowledge extraction process provided for an embodiment of this application; Figure 3A schematic diagram illustrating a reinforcement learning training process that supports knowledge transfer from heterogeneous models, provided as an embodiment of this application; Figure 4 A schematic diagram of an inference system based on heterogeneous model reinforcement learning provided in an embodiment of this application; Figure 5 A schematic diagram illustrating an inference process supporting mode switching, provided as an embodiment of this application; Figure 6 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0014] First, to facilitate understanding of the technical solutions provided in this application, the main technical concepts involved in the embodiments of this application will be briefly explained below.

[0015] Large language models, also known as large models, refer to a class of language models that are composed of artificial neural networks with a large number of parameters (usually in the billions to trillions) and are obtained through unsupervised or self-supervised training based on text.

[0016] KL divergence (KLD) is an indicator that measures the degree of matching between two probability distributions.

[0017] Pipeline process: refers to organizing a series of processing steps sequentially into a pipeline to automate and standardize the completion of complex tasks. Group Relative Policy Optimization (GRPO) is one of the mainstream reinforcement learning algorithms. As an evolution of the standard Proximal Policy Optimization (PPO) algorithm, it effectively avoids the training burden of the complex value network in PPO. Through precise control of the relative advantages within the group, it achieves efficient optimization and upgrading of the policy network (i.e., the policy model), thus exhibiting better adaptability in training complex inference tasks. The objective function of the GRPO algorithm is... Defined as:

[0018] in, , and Indicates hyperparameters, The parameters of the policy model are represented by E, which represents the expectation. Indicates the distribution of reasoning problems A specific problem example obtained from mid-sample , Indicates instances of the same specific problem The number of candidate solution groups obtained from sampling. The policy model represents the policy based on the previous round. Specific problem examples generated candidate solution set The probability distribution, Indicates the first candidate solution groups The number of candidate solutions, Indicates group-based relative reward for The advantage value calculated for the t-th candidate solution is... Indicates the current strategy and reference strategy Calculated KL divergence, The policy model represents the current policy. Specific problem examples and candidate solutions generated candidate solutions The probability distribution, The policy model represents the policy based on the previous round. Specific problem examples and candidate solutions generated candidate solutions The probability distribution, express The t-th candidate solution, express The candidate solutions located before the t-th candidate solution. This represents the operation of taking the minimum value. This indicates a cropping operation.

[0019] In the process of developing this application, the inventors discovered that the reasoning ability of current large language models is relatively low, mainly in the following aspects: (1) High cost of training data acquisition and poor generalization performance: Current high-quality distillation methods rely excessively on expensive manual annotation operations or call high-cost closed-source models to obtain high-quality fine-tuning data. This not only greatly increases the research and development and deployment costs, but also, because the model performance is limited by the distribution of a specific dataset, it shows insufficient generalization ability (such as obvious insufficient knowledge coverage) when dealing with uncovered domain knowledge or complex reasoning tasks that require broad common sense (i.e. complex natural language tasks), making it difficult to adapt to a wider range of application scenarios.

[0020] (2) Limitations of exploration mechanism and insufficient generalization ability: Large-scale reinforcement learning algorithms (such as GRPO algorithm) usually strengthen the internal prior of the model and lack the effective introduction of external knowledge. This internally guided reinforcement mechanism limits the model's exploration ability. When facing novel and complex open-domain reasoning problems, the model lacks an effective external knowledge introduction and exploration mechanism, thus limiting its exploration breadth and depth, resulting in a significant lack of generalization ability and difficulty in adapting to the diverse reasoning needs in complex open domains.

[0021] (3) Insufficient confidence in decision-making: The reasoning process of a single model currently lacks a multi-source (or multi-perspective) cross-validation mechanism. The reasoning chain (i.e. reasoning path) it generates is prone to accumulate due to local errors, resulting in low confidence in the final decision.

[0022] To address the issues of high training data acquisition costs and insufficient generalization, exploration, and decision confidence in large language models, resulting in low inference capabilities, this application provides an inference scheme based on heterogeneous model reinforcement learning. By introducing the Monte Carlo Tree Search (MCTS) mechanism, structured tree-shaped inference knowledge is generated in a low-cost manner. A heterogeneous model collaborative framework is constructed and combined with reinforcement learning optimization mechanisms. This effectively reduces the cost of training data acquisition and significantly improves the generalization, exploration, and decision confidence of large language models (such as first-policy models), thereby significantly enhancing the model's inference capabilities in complex scenarios.

[0023] The following description, in conjunction with the accompanying drawings, details a reasoning method, system, device, and medium based on heterogeneous model reinforcement learning provided in this application, through specific embodiments and application scenarios.

[0024] See Figure 1 The diagram shown is an implementation flowchart of an inference method based on heterogeneous model reinforcement learning provided in this application embodiment. The method may include the following steps: Step S101: During the training phase, for each first natural language task sample, the pre-trained execution model is used to take the first natural language task sample as the root node of the search tree. Starting from the root node, the search tree is iteratively expanded with child nodes based on the Monte Carlo tree search algorithm to obtain multiple inference paths associated with the first natural language task sample.

[0025] In this embodiment of the application, the natural language task sample (such as the first natural language task sample or the second natural language task sample) includes: a reasoning question with a pre-determined baseline answer. The reasoning question can be used to describe instructions related to the natural language task (such as text generation task, machine translation task, and question answering task) that the user wants the system to complete, such as "generate financial advice based on recent personal financial situation".

[0026] During the training phase, for each first natural language task sample, the execution model (which is responsible for generating specific reasoning paths and constructing candidate solutions or thought chains) models the first natural language task sample (i.e. the reasoning problem to be solved) as the root node of the search tree, and iteratively expands the search tree from the root node based on the Monte Carlo tree search algorithm to obtain multiple reasoning paths associated with the first natural language task sample.

[0027] This application achieves low-cost generation of structured tree-shaped reasoning knowledge associated with the first natural language task samples by iteratively expanding the child nodes of the search tree. This replaces expensive labeled data sources with a low-cost automated generation method, and the amount of information in the generated structured tree-shaped reasoning knowledge exceeds that of linear text obtained through manual annotation or application programming interface (API) calls at the same cost. This can alleviate the problem of expensive and limited sources of high-quality training data.

[0028] In some embodiments, the iterative expansion of child nodes of the search tree can be terminated when the depth of the search tree (or the depth of the leaf nodes) reaches a preset maximum depth (e.g., 10).

[0029] Step S102: For each first natural language task sample, use a pre-trained validation model to perform quality evaluation on multiple inference paths associated with the first natural language task sample, and determine the n inference paths with the highest quality scores as candidate inference paths, where n is a pre-determined positive integer.

[0030] In this embodiment, for each first natural language task sample, after iteratively expanding the child nodes of its associated search tree based on the Monte Carlo tree search algorithm, multiple different inference paths associated with the first natural language task sample can be obtained. To select higher-quality inference paths (e.g., those with more accurate answers or better matching user preferences), a pre-trained validation model (responsible for checking the logical consistency and correctness of the generated inference paths to filter out errors or low-quality solutions) can be used to evaluate and rank all inference paths associated with the first natural language task sample, and select the n inference paths (e.g., three inference paths) with the highest quality scores as candidate inference paths, thereby obtaining a set of high-quality candidate paths associated with the first natural language task sample.

[0031] Step S103: For each candidate inference path associated with each first natural language task sample, determine the computational value score of the candidate inference path based on its computational complexity and robustness accuracy.

[0032] In this embodiment, for each candidate inference path in the set of high-quality candidate paths associated with each first natural language task sample, the computational value score of the candidate inference path can be obtained by weighting the computational complexity and robust accuracy of the candidate inference path. It is understood that the computational complexity of the candidate inference path should be negatively correlated with its computational value score, and the robust accuracy of the candidate inference path should be positively correlated with its computational value score, so as to achieve a balance between efficiency and computational cost based on the computational value score.

[0033] In some embodiments, the computational complexity of a candidate reasoning path can be determined based on the number of cognitive steps in the candidate reasoning path (i.e., the number of nodes corresponding to the cognitive steps traversed in the path from the relevant leaf node to the root node in the search tree).

[0034] In some embodiments, the robustness of a candidate reasoning path can be determined based on the node value of the root node in the search tree associated with the candidate reasoning path (which reflects the expected gain when visiting the root node and can be used to measure the overall quality of the answer obtained based on the structured tree reasoning knowledge represented by the relevant search tree).

[0035] In some embodiments, to balance the robustness, accuracy, and actual computational cost of an inference path, the computational value score of the candidate inference path can be determined using the following formula:

[0036] in, Representing natural language task samples One of the associated candidate reasoning paths, Representing candidate reasoning paths The computational value score, where m represents the balance adjustment parameter between benefits and computational costs. Representing candidate reasoning paths robust accuracy Representing candidate reasoning paths The computational complexity is...

[0037] Understandably, the robust accuracy rate describes the overall probability of obtaining the correct answer using the inference path generated by the first policy model. In practical applications, it can be calculated using a natural language task sample q and its associated candidate inference paths. The input prompts are used as a reference to allow the first strategy model to generate multiple inference paths and calculate the correct answer obtained by traversing these multiple inference paths. The ratio of the number of paths P( | , ), and then P( | , The robust accuracy is obtained by taking the average value. For example, if the first strategy model generates five inference paths, then the correct answer is obtained by calculating these five inference paths. The ratio of the number of paths P( | , After that, you can get =P( | , ) / 5. The computational complexity describes the actual computational cost of generating inference paths. In practical applications, the first policy model can be used to generate candidate inference paths. The total number of required input tokens is used to determine the candidate inference paths. computational complexity .

[0038] Step S104: For each first natural language task sample, the candidate reasoning path with the highest value score among the n candidate reasoning paths associated with the first natural language task sample is abstracted to obtain the reasoning thinking pattern representation associated with the first natural language task sample.

[0039] In this embodiment, a path selection mechanism based on the trade-off between accuracy (i.e. benefit) and computational cost is introduced. The candidate inference path with the highest computational value score (i.e., a relatively balanced benefit and computational cost) is selected from the n candidate inference paths associated with the first natural language task sample as the optimal inference path (i.e. the optimal inference strategy). This allows the subsequent first strategy model to learn the optimal inference path and generate inference paths that take into account both inference efficiency and inference quality.

[0040] Reference Figure 2 The diagram illustrates the advanced knowledge extraction process. Through the path selection process that combines quality scoring and computational value scoring, a validated optimal reasoning path can be obtained for each first natural language task sample. Then, these optimal reasoning paths are abstracted (e.g., prompt word generation) to extract higher-level reasoning thought patterns (i.e., abstracted advanced reasoning patterns), so as to lay a solid foundation for the knowledge transfer and reasoning ability optimization of subsequent models.

[0041] For example, for the candidate reasoning path: single-step logical analysis → hierarchical decomposition of the problem → verification of the logical consistency and correctness of the preceding reasoning steps through a reflection mechanism, the reasoning thinking mode obtained by abstracting it can be represented as: "For this type of reasoning problem, the solution strategy should follow the following cognitive process: first, perform single-step logical analysis; second, perform hierarchical decomposition of the problem; and finally, verify the logical consistency and correctness of the preceding reasoning steps through a reflection mechanism."

[0042] Step S105: Based on the reasoning thought pattern representation associated with each of the first natural language task samples, train the pre-established first policy model using a reinforcement learning algorithm, wherein the first policy model is established based on a large language model with a different architecture than the execution model and the verification model.

[0043] Among them, the execution model can adopt a general model such as qwen2.5-7b to cover a wide range of knowledge; the strategy model can adopt any large language model (which can be a general model or a domain-specific model); and the validation model can adopt a process reward model (prm).

[0044] In this embodiment, a first policy model is established based on a large language model with a different architecture from the execution model and the verification model. This heterogeneous model, architecturally completely different from the models used for automatic inference path generation and higher-order cognitive pattern extraction, serves as the policy model (or policy function) for the reinforcement learning algorithm (e.g., the GRPO algorithm). During reinforcement learning, the inference thinking pattern representation obtained from the models used for automatic inference path generation and higher-order cognitive pattern extraction guides the first policy model in the inference path sampling process under similar natural language task samples, enabling the model to explore reinforcement learning in a higher-dimensional structured knowledge space.

[0045] It is understandable that the heterogeneity in architectural design between the model combination providing guidance information (i.e., the execution model and the verification model) and the policy model receiving guidance (i.e., the first policy model) naturally constitutes a cross-model knowledge transfer mechanism. This design can seamlessly transfer interactive cognitive information (such as higher-order problem-solving strategies or problem-solving patterns) from the execution model and the verification model to the reinforcement learning training process of the policy model through a lightweight knowledge transfer method, thereby achieving effective injection of new information from multiple external sources and complementary enhancement of knowledge between models. Compared with traditional reinforcement learning methods based on a single architecture model, this application, by introducing multi-source knowledge guidance and cross-architecture information transfer, not only solves the problem of excessively high cost of obtaining high-quality training data, but also overcomes the shortcomings of traditional reinforcement learning methods in terms of limited exploration ability, generalization ability, and decision confidence.

[0046] In some embodiments, considering that the above-described Pipeline process designed in this application can naturally construct high-quality distillation data (i.e., data such as the verified correct reasoning thought pattern representation obtained through MCTS), it can also be further used for processes such as supervised fine-tuning. For example, the relevant model can calculate the difference between the predicted output probability distribution and the standard answer based on the question input and distillation data, and fine-tune the relevant model accordingly.

[0047] Step S106: In the reasoning phase, in response to receiving a natural language task, a first reasoning path is generated using the trained first policy model, and the answer to the natural language task is determined based on the first reasoning path. The first reasoning path corresponds to a tree-based exploratory reasoning process.

[0048] In this embodiment of the application, the natural language task includes: a reasoning question to be answered, which can be used to describe instructions related to the natural language task that the user wants the system to complete, such as "generate financial advice based on recent personal financial situation", etc.

[0049] In this embodiment, since the first strategy model has effectively internalized the tree-structured high-order knowledge based on Monte Carlo tree search during the training phase, it can generate a first reasoning path for the input natural language task during the reasoning phase. Then, based on this first reasoning path, it autonomously executes a complex, multi-path tree-structured exploratory reasoning process. It is understood that when the first strategy model generates the first reasoning path and executes the tree-structured exploratory reasoning process (e.g., when performing reasoning operations or cognitive steps such as deep analysis, hypothesis generation, multi-branch exploration, and pruning evaluation on a natural language task (i.e., a reasoning problem), it can fully utilize the collaborative verification capabilities of heterogeneous models and the learned high-order cognitive paradigms to generate reasoning paths and answers with higher accuracy, robustness, and logical consistency.

[0050] As can be seen from the above technical solution, for each first natural language task sample, this application generates structured tree-shaped reasoning knowledge (i.e., the search tree) in a low-cost manner based on the Monte Carlo tree search mechanism. This can overcome the dependence on expensive labeled data, thereby reducing the cost of training data acquisition. Furthermore, it introduces a computational value score to balance accuracy and computational overhead. Then, the high-order reasoning pattern (i.e., the reasoning thinking pattern representation) obtained based on the tree-shaped reasoning knowledge is applied to the reinforcement learning training process, thereby effectively internalizing the high-order knowledge of the heterogeneous model group (i.e., the execution model and the verification model) into the first policy model. This can solve the dual technical problems of high cost of constructing high-quality distilled data and lack of multi-source (or multi-perspective) external knowledge guidance in traditional reinforcement learning algorithms. Moreover, it enables the first policy model to reason about the answer to the natural language task through a tree-shaped exploratory reasoning process during the reasoning stage. This can significantly improve the generalization ability, exploration ability, and decision confidence of the first policy model, thereby effectively improving the reasoning ability of the first policy model.

[0051] In some embodiments, the iterative expansion of the search tree from the root node using the Monte Carlo tree search algorithm includes: In the current iteration, based on the node value and access statistics of each node in the search tree, the search tree is searched from the root node until the leaf node to be expanded is found in the search tree. In the current iteration, generate each child node of the leaf node to be expanded, and assign an initial node value to each newly generated child node; In the current iteration, based on the path from each newly generated child node to the root node, determine the inference path associated with each newly generated child node; In the current iteration, based on the reasoning path associated with each newly generated child node, the simulated reasoning answer of the first natural language task sample associated with the search tree is determined; In the current iteration round, based on the simulated reasoning answer associated with each newly generated child node, starting from each newly generated child node, the node value and access statistics of all nodes traversed by the reasoning path associated with each newly generated child node are updated one by one, and a new iteration round is executed after the node value and access statistics are updated. The node value is updated based on the node value of each node itself and its child nodes.

[0052] In this application embodiment, a specific implementation method is designed to transform complex reasoning tasks into structured tree-like search optimization problems. This provides a technical foundation for the automated extraction and abstraction of high-quality reasoning patterns. Furthermore, by optimizing the node value update method, the gradual propagation and adaptive update of value information in the search tree are realized, making the value assessment of the entire search tree more accurate and convergent. This can guide subsequent reasoning and searching towards a better direction.

[0053] Specifically, this application employs an improved Monte Carlo tree search algorithm as the core inference engine, and iteratively executes a search process comprising the following four key stages for each input first natural language task sample (i.e., the seed problem): (1) Node selection stage: Based on confidence upper bound strategy, the search tree can be selected from the root node until the leaf node to be expanded in the search tree is found.

[0054] In one specific implementation, multiple samplings are performed at each node to obtain the majority of the voting reasoning paths as the optimal path associated with that node. That is, in the node selection phase, based on the confidence upper bound strategy, the child node with the highest UCT score is selected as the new decision node at each decision node until the leaf node to be expanded is found (at this time, the selected nodes can constitute the optimal path), thereby achieving the optimal balance between reasoning exploration and the utilization of known knowledge.

[0055] Optionally, the UCT score can be determined using the following formula:

[0056] in, This represents the UCT score of node s. This represents the node value of node s. This represents the number of times node s has been visited. This represents the number of times node p, the direct parent node of node s, has been visited. This represents hyperparameters.

[0057] (2) Tree structure expansion stage: For the leaf node (i.e. the leaf node to be expanded) found in the node selection stage, new child nodes are generated for the leaf node to realize the expansion of the child node, and an initial node value is assigned to each newly generated child node.

[0058] In one specific implementation, the child nodes of the leaf node to be expanded can be generated based on a predefined basic cognitive behavior pattern library. This basic cognitive behavior pattern library stores general operations of human reasoning, such as reasoning operations and cognitive steps like problem analysis, problem decomposition, and reflection.

[0059] In one specific implementation, the node value of each newly generated child node can be initialized to zero.

[0060] (3) Reasoning path simulation stage: Starting from the newly generated child node, generate a reasoning path (such as generating a complete reasoning link from the reasoning operation or cognitive step corresponding to the child node to the answer to the question) based on the path from the child node to the root node, and execute the simulated reasoning process accordingly.

[0061] (4) Value feedback and update stage: Based on the simulated reasoning answer obtained from the simulated reasoning process, update the node value and access statistics (such as the number of times the node is accessed) of each node from bottom to top along the path from the child node to the root node. Among them, the node value is used to represent the expected benefits obtained when accessing the corresponding node (such as the quality of the obtained answer (e.g., accuracy, self-consistency, etc.)), and it is updated based on the node value of the corresponding node itself and its child nodes.

[0062] In one specific implementation, any node The node value is updated using the following formula:

[0063] Among them, nodes For nodes The direct parent node, Represents the value function. , Represents a node ,node The value of nodes.

[0064] It is understandable that this application will use child nodes (such as nodes) The latest estimated value of the node (i.e., the latest node value) is dynamically fed back to the parent node (e.g., node 0). This approach integrates the node values ​​of the search tree, enabling the gradual propagation and adaptive updating of value information. This design maintains the stability of historical estimates while promptly incorporating new search results, resulting in more accurate and convergent value assessments of the entire search tree. This, in turn, guides subsequent reasoning and searching towards a better outcome.

[0065] In some embodiments, the node value of a relevant leaf node can be determined based on the difference between the simulated reasoning answer of the simulated reasoning process and the baseline answer of the seed question.

[0066] In some embodiments, a process reward model can be used to allocate node value, thereby ensuring the accuracy of the value allocation for each leaf node. Specifically, the input of the process reward model is the reasoning path associated with a certain leaf node, that is, the complete reasoning link formed by all nodes traversed from the root node (corresponding to the first natural language task sample or seed problem) to the leaf node; the reasoning link includes a step-by-step reasoning process from the problem, through various cognitive behavioral steps (such as problem analysis, problem decomposition, reflection, etc.) to the current reasoning state. The output of the process reward model is the process-level quality score given for each reasoning step in the reasoning path, that is, evaluating each intermediate step in the reasoning path, outputting the reward score of each step, and then summing them up to obtain the overall node value of the leaf node (also known as the Q value, used to characterize the expected benefits (such as accuracy, self-consistency, etc.) of the reasoning steps at that node).

[0067] For example, the node value allocation process based on the process reward model is as follows: Step 1: Construct the inference path input.

[0068] In this step, after generating new child nodes during the tree structure expansion phase, the complete inference path associated with each newly generated child node is extracted based on the path from the child node to the root node. The seed problem is then concatenated with this inference path to construct the input sequence of the process reward model.

[0069] Step 2: The process reward model scores each step.

[0070] In this step, the above input sequence is fed into a pre-trained process reward model (PRM), which outputs process reward scores r1, r2, …, r for each inference step in the inference path. n Each score reflects the reasoning quality of the corresponding step (such as whether the step is reasonable or whether it helps to solve the problem correctly).

[0071] Step 3: Summarize and calculate the node value.

[0072] In this step, based on the reward scores of each step output by prm, the node value of the leaf node is calculated using an appropriate aggregation method (such as taking the minimum value, weighted summation, or average value).

[0073] Step 4: Write the node value to the corresponding leaf node.

[0074] In this step, the node value calculated in step three is assigned to the current leaf node, completing the value allocation for that node. Then, the value backpropagation and update phase begins, based on the update formula described above: Update the value of each parent node from bottom to top.

[0075] Step 5 (optional): Take the average of multiple samples.

[0076] In this step, to further improve the stability of node value allocation, the reasoning path of the same leaf node can be sampled multiple times, and the average value after PRM scoring can be taken as the final node value, thereby reducing the random error of a single scoring.

[0077] In some embodiments, a self-allocation method based on self-consistency and majority voting can be used to allocate node value.

[0078] For example, the node value allocation process based on the above self-allocation method is as follows: Step 1: Perform multiple inference path samplings for the leaf nodes.

[0079] In this step, after generating new child nodes during the tree structure expansion phase, the inference path prefix associated with each newly generated child node (i.e., the existing inference link from the root node to the leaf node) is determined based on the path from each newly generated child node to the root node. Using this inference path prefix as the context, the subsequent inference process is independently sampled multiple times through the language model, generating k complete inference paths and their corresponding final answers for each leaf node, denoted as: {(path1, ans1), (path2, ans2), …, (path...} k , ans k )}, where k is the preset number of samplings, and each sampling is independent of the others to ensure the diversity of answers.

[0080] Step 2: Calculate the frequency of each answer based on Majority Voting.

[0081] In this step, the final answer set {ans1, ans2, …, ans} obtained from k samplings is... kPerform normalization processing (such as merging equivalent answers), and count the occurrences of each distinct answer, i.e.: Count(ans) i () = Answer ans from K samplings i The number of times an answer appears; the answer with the highest frequency is selected as the majority answer, denoted as ans*, i.e.: ans* = argmax Count(ans) i ).

[0082] Step 3: Calculate node value based on self-consistency In this step, based on the self-consistency principle, the support ratio of the majority vote answer ans* is used as a measure of the reliability of the inference path of the leaf node, and the node value Q is calculated, i.e.: Q(leaf node) = Count(ans*) / k Understandably, the node value Q reflects the consistency of the reasoning path associated with the current leaf node: if most paths converge to the same answer in k samplings, it indicates that the reasoning path has high reliability and the node value is high; conversely, if the answers are scattered, the node value is low.

[0083] Step 4: Write the node value to the corresponding leaf node. In this step, the node value Q calculated in step three is assigned to the current leaf node, completing the self-allocation of the node's value. Then, the value backpropagation and update phase begins, based on the update formula described above: It updates the node value and access statistics of each parent node from bottom to top, guiding subsequent searches toward higher self-consistency and more reliable reasoning.

[0084] Step 5 (optional): Multi-node parallel self-allocation.

[0085] In this step, if multiple new child nodes are generated during the tree structure expansion phase in the same iteration round, steps one to four above can be executed in parallel for each child node to independently complete the self-allocation of their respective node values, thereby improving the overall search efficiency.

[0086] It should be noted that this application does not impose specific restrictions on the method of node value allocation. For example, the node value allocation method can be selected based on the current resource configuration; if the current resource configuration indicates a need for low-cost or fast node value allocation (e.g., limited computing resources), a self-allocation method can be used; if the current resource configuration indicates a need for high-cost or high-precision node value allocation (e.g., sufficient computing resources), and if resources are sufficient, node value allocation can be performed based on the external supervision signal output by the prm.

[0087] In some embodiments, during the initialization phase, the node values ​​of all nodes to be explored in the search tree (corresponding to various sub-steps or sub-strategies required to solve the reasoning problem) can be initialized to zero, and then the value estimation can be dynamically updated through the above-mentioned node value update formula.

[0088] In some embodiments, during the initialization phase, the node access count of all nodes to be explored in the search tree can be initialized to zero. Then, each time a node is accessed (e.g., each time a node is selected during the node selection phase), the node access count of that node is incremented by one, thereby achieving dynamic updating of the node access count.

[0089] In some embodiments, each node in the search tree, except for the root node, corresponds to a reasoning operation or cognitive step, and the reasoning path is determined based on the path from the leaf node to the root node of the search tree; generating each child node of the leaf node to be expanded based on a predefined basic cognitive behavior pattern library includes: Based on the execution order of different reasoning operations and different cognitive steps in the reasoning process, and the reasoning operation or cognitive step corresponding to the leaf node to be expanded, select each reasoning operation and / or each cognitive step that conforms to the execution order from the basic cognitive behavior pattern library. For the leaf nodes to be expanded, generate child nodes corresponding to each selected reasoning operation and / or cognitive step.

[0090] In this embodiment of the application, each node in the search tree, except for the root node, corresponds to a specific reasoning operation or cognitive step. These reasoning operations or cognitive steps can cover a variety of high-level reasoning behavior patterns (such as single-step logical deduction, multi-step task decomposition, reflective verification, etc.). Among them, the cognitive step can be regarded as an abstract reasoning pattern, which can include single-step logical deduction, multi-step task decomposition, reflective verification, etc. The reasoning operation can be regarded as the actual content after the cognitive step is visualized, which is strongly related to the specific reasoning problem.

[0091] To ensure that the expansion of child nodes conforms to the logical order of human reasoning, the execution order of different reasoning operations and cognitive steps in the reasoning process can be obtained in advance from the basic cognitive behavior pattern library as prior information (for example, the node corresponding to reflection should exist after the node corresponding to the actual solution operation such as problem decomposition, and cannot be the first node or the parent node of the node corresponding to the actual solution operation, etc.). Subsequently, based on this prior information, various reasoning operations and / or cognitive steps that conform to the execution order can be selected from the basic cognitive behavior pattern library and child nodes can be generated accordingly. This ensures that the reasoning path obtained based on the Monte Carlo tree search algorithm conforms to the logical order of human reasoning, thereby further improving the decision confidence of the model.

[0092] In some embodiments, training a pre-established first policy model based on a reinforcement learning algorithm according to the reasoning thought pattern representation associated with each of the first natural language task samples includes: For each second natural language task sample, based on the semantic matching algorithm, each reasoning thought pattern representation that matches the second natural language task sample is selected from the reasoning thought pattern representation associated with each first natural language task sample as each knowledge guidance strategy. The first policy model is trained based on the group relative policy optimization algorithm, according to the knowledge guidance strategies associated with each second natural language task sample.

[0093] In this embodiment, a reinforcement learning training algorithm supporting knowledge transfer from heterogeneous models is designed based on the GRPO framework. This algorithm effectively injects high-order reasoning patterns (i.e., high-level cognitive knowledge extracted through the Monte Carlo tree search mechanism) from a small amount of seed data into the reinforcement learning training process, thereby simultaneously solving the dual technical problems of high cost of constructing high-quality distillation data and lack of external knowledge guidance in traditional reinforcement learning algorithms.

[0094] Specifically, during the sampling process of GRPO, for each reasoning problem sample (i.e., the second natural language task sample, which may be the same as or different from the first natural language task sample), a semantic matching algorithm is first used to retrieve several high-order knowledge patterns (i.e., reasoning thought pattern representations) that are most relevant to the reasoning problem sample as knowledge guidance strategies, so that the reasoning path sampling process of the first strategy model can be guided by these knowledge guidance strategies. Considering the inherent multi-path sampling mechanism of the GRPO algorithm, that is, it is usually necessary to generate multiple different sampling reasoning paths, the guidance information provided by the heterogeneous model should also present diversified characteristics accordingly.

[0095] In some embodiments, the semantic representation of the reasoning problem to be solved can be parsed first, and a semantic matching algorithm can be used to retrieve the most relevant high-order knowledge patterns (which may include thinking templates such as system analysis, step-by-step reasoning, decomposition and verification) from the knowledge base as a knowledge guidance strategy. The knowledge guidance strategy is used to provide prior structure and logical guidance for the reasoning path sampling of the first strategy model.

[0096] In some embodiments, to ensure the applicability of GRPO in the heterogeneous model reinforcement learning scenario described in this application, the objective function adopted by the group relative policy optimization algorithm is... It can be represented as follows:

[0097] in, Let E represent the parameters of the first strategy model, and E represent the expectation. Represents the distribution of samples from natural language tasks A specific natural language task sample obtained by mid-sampling , This represents samples from the same natural language task. The number of candidate solution groups obtained from sampling. This indicates that the first policy model is based on the previous policy. (That is, the old inference strategy, which can be used to construct the importance sampling ratio to ensure training stability) for natural language task samples generated candidate solution set The probability distribution, This represents the total number of knowledge-guided strategies. This represents the weight coefficient of the i-th knowledge guidance strategy. Indicates the first Candidate solution groups associated with each knowledge-guided strategy The number of candidate solutions, Indicates group-based relative reward for The advantage value calculated for the t-th candidate solution is... Indicates the current strategy (That is, the inference strategy currently used by the policy model, which describes the model's parameters) (Probability distribution of generated inference sequences) and reference strategy (The KL divergence and probability ratio can be calculated based on a pre-trained model or benchmark inference policy, used to constrain the policy distribution from deviating from a reasonable solution space.) It is used to measure the probability ratio of the current strategy and the old strategy under the same action, the pruning ratio. , and Indicates hyperparameters, The first policy model is based on the current policy. Samples for natural language tasks and candidate solutions generated candidate solutions The probability distribution, This indicates that the first policy model is based on the previous policy. Samples for natural language tasks and candidate solutions generated candidate solutions The probability distribution, express The t-th candidate solution, express The candidate solutions located before the t-th candidate solution. This represents the operation of taking the minimum value. This indicates a cropping operation.

[0098] For example, refer to Figure 3 The diagram shown illustrates the reinforcement learning training process that supports knowledge transfer from heterogeneous models, for the input natural language task samples (i.e., reasoning problem samples). Higher-order knowledge guidance functions can be defined. For a sample of reasoning problems Mapping converted to enhanced hint format: In other words, when the strategy model (i.e., the first strategy model) performs reasoning under the guidance of higher-order knowledge from the heterogeneous model, it generates partial solutions. These partial solutions are then compared with the original reasoning problem samples. Combined with the generation of enhanced questions containing prompts Therefore, specific knowledge guidance strategies are needed. It can sample the micro-group output set from the first policy model trained in the previous iteration. These outputs represent the knowledge-guided strategies The resulting diverse reasoning paths.

[0099] For any knowledge-guided strategy, the set of micro-group outputs (i.e., the set of various enhanced thought chains) A reward function can be used to quantitatively evaluate the output of the reasoning process associated with each reinforced mental chain. For example, it can score the solution generated by each reasoning process (i.e., the answer to the reasoning problem) or the reasoning operations or cognitive steps contained therein, thus obtaining a set of reward values ​​for each reinforced mental chain. This guides the optimization of the first strategy model towards "high-quality reasoning." Understandably, in solving reasoning problems, reward values ​​can be used to characterize the quality of the reasoning path or the final solution, such as measuring its correctness, logical consistency, completeness, and whether it meets the task objective. For example, a solution with a high reward value indicates that the reasoning chain generated by the model is more consistent with real reasoning logic and the obtained answer is closer to the correct solution. Furthermore, a reference function can be used to provide a baseline strategy or value estimate to measure the deviation between the current model output and the baseline behavior, thereby stabilizing the optimization process and preventing the model output from deviating too much from the reasonable solution space.

[0100] In obtaining the set of reward values ​​for each reinforcing mental chain. Then, group computation can be performed to compute the set of advantage values ​​for each candidate solution corresponding to each enhanced thought chain. Specifically, the GRPO group calculation process mainly includes: a. Generation group: Identify a set of candidate solutions (i.e., candidate solution group) associated with the current knowledge guidance strategy for the same input.

[0101] b. Calculate the reward value: Calculate the reward value for each candidate solution in the group.

[0102] c. Standardized Comparison: Calculate the relative advantage value of each candidate solution based on the group's average reward or reference strategy. Understandably, in solving reasoning problems, the advantage value can be used to measure the superiority or inferiority of a specific reasoning path relative to the average level or reference strategy, thus reflecting how much better or worse the reasoning path is than the "baseline reasoning."

[0103] d. Update strategy: Use these relative advantage values ​​to update the policy gradient, guiding the model towards higher-quality inference. For example, the first policy model can be updated with weights based on the advantage value, making the model more inclined to generate inference paths with higher advantage values ​​(i.e., higher quality), thereby gradually improving the overall inference ability.

[0104] Understandably, this is to guide various knowledge-sharing strategies. To better apply this to the GRPO-based model training process, this application correspondingly modifies the objective function of the aforementioned GRPO algorithm. Adjustments were made to obtain the above objective function. , where the objective function This can be understood as a weighted summation of specific objective functions guided by various heterogeneous knowledge sources. Thus, by... Applying this to the GRPO-based model training process not only enables seamless knowledge transfer from heterogeneous models but also allows for simultaneous optimization of model performance under various higher-level thinking guidance. Thus, this application constructs a heterogeneous model collaborative learning framework that effectively integrates the cognitive advantages of different model architectures and achieves a significant improvement in inference performance through improved reinforcement learning algorithms.

[0105] In some embodiments, the method further includes: During the training of the first policy model based on the group relative policy optimization algorithm, the probability of an erroneous solution appearing in each candidate solution group associated with the previous policy of the first policy model is determined. For candidate solution groups where the probability of an erroneous solution is greater than a preset value, the number of candidate solutions in the candidate solution group is increased, and / or the weight of the parameter terms associated with the candidate solution group in the objective function is decreased.

[0106] In this embodiment, considering that generating a binary accuracy value (0 or 1) can lead to gradient vanishing, thereby hindering learning efficiency and reducing sample utilization, the probability of an incorrect solution is introduced to represent the output of the micro-group based on the old inference strategy. ( The probability of a zero quasi-deterministic solution (i.e., an incorrect solution) appearing in the corresponding candidate solution set during sampling, and the probability of obtaining at least one non-zero quasi-deterministic solution (i.e., a correct solution) in the corresponding candidate solution set at this time, can be expressed as: The probability of obtaining at least one correct solution can also be expressed as: ,in, This represents the i-th reasoning strategy. express The probability of an incorrect solution appearing in the associated i-th candidate solution group. express The probability of an incorrect solution appearing in the associated j-th candidate solution group. This indicates the number of candidate solutions sampled within the group.

[0107] By using the probability of this incorrect solution occurring, the quality of samples during the training process can be constrained, making the objective function more effective. Under diversified guidance strategies, the sampling strategy can be automatically adjusted based on the probability of erroneous solutions, thereby achieving dynamic adjustment of sample utilization and sampling efficiency. For example: when When the error rate is high (larger), the number of candidate solutions sampled within the group can be increased or the weight of the candidate solution group can be decreased; when When the value is small, it indicates that most candidate solutions within the group are valid, which can improve the efficiency of policy updates. This avoids gradient update noise (here, noise actually refers to the absence of gradients) caused by a large number of invalid samples (i.e., incorrect solutions), thereby improving sampling efficiency and model convergence stability, and ultimately ensuring the stability of the model's learning performance during grouped sample training under diverse guidance.

[0108] In some embodiments, the method further includes: During the inference phase, in response to receiving a first instruction associated with the performance-first mode, a first inference path is generated using the trained first policy model, and the answer to the natural language task targeted by the first instruction is determined based on the first inference path. During the inference phase, in response to receiving a second instruction associated with the speed-first mode, a second inference path is generated using a pre-trained second policy model, and the answer to the natural language task targeted by the second instruction is determined based on the second inference path, which corresponds to a step-by-step inference process.

[0109] In this embodiment of the application, by introducing a first instruction and a second instruction, users can flexibly select inference strategies under different modes according to their own needs, thereby achieving a flexible trade-off between inference efficiency and inference quality.

[0110] Specifically, upon receiving the first instruction associated with the performance-first mode, a deeply optimized inference strategy is executed accordingly. That is, the first strategy model is invoked to generate a first inference path for the input natural language task, and based on the first inference path, a complex, multi-path tree-based exploratory inference process is executed to determine the answer to the natural language task.

[0111] Upon receiving the second instruction associated with the speed-first mode, a lightweight inference strategy is executed accordingly. This involves calling a pre-trained second strategy model (which can be trained using traditional inference model training methods) to generate a second inference path (i.e., a linear, sequential inference path, such as a typical thought chain). Based on this second inference path, the answer to the natural language task is determined. This approach guides the reader from the question to the answer through step-by-step reasoning, similar to a direct reasoning process triggered by prompts such as "let's think step by step." This avoids complex multi-path exploration and evaluation, significantly reducing computational resource consumption and latency, making it well-suited for applications requiring high real-time response speeds.

[0112] This application provides an inference system based on heterogeneous model reinforcement learning, such as... Figure 4As shown, the system includes a high-level knowledge extraction module, a heterogeneous model knowledge transfer and reasoning ability optimization module, and a reasoning module, among which: The advanced knowledge extraction module is configured to, during the training phase, for each first natural language task sample, use a pre-trained execution model to take the first natural language task sample as the root node of a search tree, and iteratively expand the search tree from the root node using a Monte Carlo tree search algorithm to obtain multiple inference paths associated with the first natural language task sample; and for each first natural language task sample, use a pre-trained verification model to perform quality evaluation on the multiple inference paths associated with the first natural language task sample, and determine the n inference paths with the highest quality scores as candidate inference paths, where n is a pre-determined positive integer; and for each candidate inference path associated with each first natural language task sample, determine the computational value score of the candidate inference path based on the computational complexity and robustness accuracy of the candidate inference path; and for each first natural language task sample, abstract the candidate inference path with the highest computational value score among the n candidate inference paths associated with the first natural language task sample to obtain a representation of the inference thinking pattern associated with the first natural language task sample. The heterogeneous model knowledge transfer and reasoning ability optimization module is configured to train a pre-established first policy model based on a reinforcement learning algorithm according to the reasoning thinking pattern representation associated with each of the first natural language task samples. The first policy model is established based on a large language model with a different architecture than the execution model and the verification model. The reasoning module is configured to, in response to receiving a natural language task during the reasoning phase, generate a first reasoning path using a trained first policy model, and determine the answer to the natural language task based on the first reasoning path, wherein the first reasoning path corresponds to a tree-based exploratory reasoning process.

[0113] As can be seen from the above technical solution, for each first natural language task sample, this application generates structured tree-shaped reasoning knowledge (i.e., the search tree) in a low-cost manner based on the Monte Carlo tree search mechanism. This can overcome the dependence on expensive labeled data, thereby reducing the cost of training data acquisition. Furthermore, it introduces a computational value score to balance accuracy and computational overhead. Then, the high-order reasoning pattern (i.e., the reasoning thinking pattern representation) obtained based on the tree-shaped reasoning knowledge is applied to the reinforcement learning training process, thereby effectively internalizing the high-order knowledge of the heterogeneous model group (i.e., the execution model and the verification model) into the first policy model. This can solve the dual technical problems of high cost of constructing high-quality distilled data and lack of multi-source (or multi-perspective) external knowledge guidance in traditional reinforcement learning algorithms. Moreover, it enables the first policy model to reason about the answer to the natural language task through a tree-shaped exploratory reasoning process during the reasoning stage. This can significantly improve the generalization ability, exploration ability, and decision confidence of the first policy model, thereby effectively improving the reasoning ability of the first policy model.

[0114] Optionally, the advanced knowledge extraction module is further configured to perform the following steps: In the current iteration, based on the node value and access statistics of each node in the search tree, a node search is performed on the search tree starting from the root node until the leaf node to be expanded in the search tree is found. In the current iteration, generate each child node of the leaf node to be expanded, and assign an initial node value to each newly generated child node; In the current iteration, based on the path from each newly generated child node to the root node, determine the inference path associated with each newly generated child node; In the current iteration, based on the reasoning path associated with each newly generated child node, the simulated reasoning answer of the first natural language task sample associated with the search tree is determined; In the current iteration round, based on the simulated reasoning answer associated with each newly generated child node and the benchmark answer of the first natural language task sample associated with the search tree, starting from each newly generated child node, the node value and access statistics of all nodes traversed by the reasoning path associated with each newly generated child node are updated one by one, and a new iteration round is executed after the node value and access statistics are updated. The node value is updated based on the node value of each node itself and its child nodes.

[0115] In this embodiment of the application, structured tree-shaped reasoning knowledge is generated in a low-cost manner through a high-order knowledge extraction module, which can overcome the dependence on expensive labeled data.

[0116] Optionally, each node in the search tree, except for the root node, corresponds to a reasoning operation or cognitive step, and the reasoning path is determined based on the path from the leaf node to the root node of the search tree; the higher-order knowledge extraction module is also configured to perform the following steps: Based on the execution order of different reasoning operations and different cognitive steps in the reasoning process, and the reasoning operation or cognitive step corresponding to the leaf node to be expanded, select each reasoning operation and / or each cognitive step that conforms to the execution order from the basic cognitive behavior pattern library. For the leaf nodes to be expanded, generate child nodes corresponding to each selected reasoning operation and / or cognitive step.

[0117] Optionally, the heterogeneous model knowledge transfer and reasoning ability optimization module is further configured to perform the following steps: For each second natural language task sample, based on the semantic matching algorithm, each reasoning thought pattern representation that matches the second natural language task sample is selected from the reasoning thought pattern representation associated with each first natural language task sample as each knowledge guidance strategy. The first policy model is trained based on the group relative policy optimization algorithm, according to the knowledge guidance strategies associated with each second natural language task sample.

[0118] In this embodiment, the heterogeneous model knowledge transfer and reasoning ability optimization module is based on an improved GRPO algorithm framework. It optimizes reasoning ability through a knowledge transfer mechanism between heterogeneous models, and constructs cross-model knowledge transfer channels using combinations of heterogeneous models with completely different architectures (e.g., execution model, validation model, and first policy model). It also introduces diverse guidance mechanisms to extend the GRPO objective function to support the simultaneous optimization of multiple higher-order knowledge patterns, thereby improving the model's exploration ability and generalization performance. Thus, based on the higher-order knowledge patterns extracted by the higher-order knowledge extraction module, the knowledge transfer mechanism between heterogeneous models can effectively reduce the error caused by prior bias within a single policy model based on guidance information provided by different architectural models, thereby further optimizing the model's reasoning ability and overcoming the inherent limitations of traditional high-quality data distillation and large-scale reinforcement learning algorithms.

[0119] Optionally, the heterogeneous model knowledge transfer and reasoning ability optimization module is further configured to perform the following steps: During the training of the first policy model based on the group relative policy optimization algorithm, the probability of an erroneous solution appearing in each candidate solution group associated with the previous policy of the first policy model is determined. For candidate solution groups where the probability of an erroneous solution is greater than a preset value, the number of candidate solutions in the candidate solution group is increased, and / or the weight of the parameter terms associated with the candidate solution group in the objective function is decreased; The objective function used in the group relative strategy optimization algorithm is... It is expressed as follows:

[0120] in, Let E represent the parameters of the first strategy model, and E represent the expectation. Represents the distribution of samples from natural language tasks A specific natural language task sample obtained by mid-sampling , This represents samples from the same natural language task. The number of candidate solution groups obtained from sampling. This indicates that the first strategy model is based on the previous strategy. Samples for natural language tasks generated candidate solution set The probability distribution, This represents the total number of knowledge-guided strategies. This represents the weight coefficient of the i-th knowledge guidance strategy. Indicates the first Candidate solution groups associated with each knowledge-guided strategy The number of candidate solutions, Indicates group-based relative reward for The advantage value calculated for the t-th candidate solution is... Indicates the current strategy and reference strategy Calculated KL divergence, probability ratio Cutting ratio , and Indicates hyperparameters, The first policy model is based on the current policy. Samples for natural language tasks and candidate solutions generated candidate solutions The probability distribution, This indicates that the first policy model is based on the previous policy. Samples for natural language tasks and candidate solutions generated candidate solutions The probability distribution, express The t-th candidate solution, express The candidate solutions located before the t-th candidate solution. This represents the operation of taking the minimum value. This indicates a cropping operation.

[0121] Optionally, the computational value score of the candidate inference path is determined using the following formula:

[0122] in, Representing natural language task samples One of the associated candidate reasoning paths, Representing candidate reasoning paths The computational value score, where m represents the balance adjustment parameter between benefits and computational costs. Representing candidate reasoning paths robust accuracy Representing candidate reasoning paths The computational complexity is...

[0123] In this embodiment of the application, the system can dynamically adjust the model's learning strategy and reasoning path based on the computational value score, according to the actual difficulty or complexity of the reasoning problem, thereby improving the model's flexibility and robustness in dealing with complex and diverse problems.

[0124] Optional, refer to Figure 5 The diagram illustrates an inference process that supports mode switching. The inference module includes an overhead-reward routing module, configured to perform the following steps: During the inference phase, in response to receiving a first instruction associated with the performance-first mode, a first inference path is generated using the trained first policy model, and the answer to the natural language task targeted by the first instruction is determined based on the first inference path. During the inference phase, in response to receiving a second instruction associated with the speed-first mode, a second inference path is generated using a pre-trained second policy model, and the answer to the natural language task targeted by the second instruction is determined based on the second inference path, which corresponds to a step-by-step inference process.

[0125] This application implements an efficient inference system based on heterogeneous model reinforcement learning. This system aims to provide an efficient, robust, and highly generalizable inference solution by constructing a heterogeneous model collaborative framework and a reinforcement learning optimization mechanism. The overall system architecture consists of two core modules during the training phase: a high-order knowledge extraction module and a heterogeneous model knowledge transfer and inference capability optimization module. The high-order knowledge extraction module is responsible for generating structured tree-like inference knowledge at low cost, overcoming the dependence on expensive labeled data. The heterogeneous model knowledge transfer and inference capability optimization module is responsible for internalizing the high-order knowledge of the heterogeneous model group into the first policy model through reinforcement learning, significantly improving its exploratory inference and generalization capabilities. During the inference (or testing) phase, the system mainly provides adaptive switching between speed and performance through a cost-benefit routing module, achieving efficient resource utilization. Through the collaborative design of these three core modules, a closed-loop technical framework of "knowledge generation - model interaction - capability optimization - flexible application" has been realized. This framework can solve key technical problems in related technologies, such as expensive information acquisition, insufficient generalization ability, limited exploration ability, and low model collaboration efficiency. As a result, it significantly improves the system's understanding and reasoning ability in complex scenarios, thus realizing a high-efficiency reasoning system that combines low cost, high generalization, strong exploration ability, and application flexibility.

[0126] Optionally, this application embodiment also provides an electronic device, including a processor 110, a memory 109, and a program or instructions stored in the memory 109 and executable on the processor 110. When the program or instructions are executed by the processor 110, they implement the various processes of the above-described inference method embodiment based on heterogeneous model reinforcement learning and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0127] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0128] Figure 6 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0129] The electronic device 100 includes, but is not limited to, components such as: radio frequency unit 101, network module 102, audio output unit 103, input unit 104, sensor 105, display unit 106, user input unit 107, interface unit 108, memory 109, and processor 110.

[0130] Those skilled in the art will understand that the electronic device 100 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 110 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 6The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0131] The processor 110 is used to perform the following steps: During the training phase, for each first natural language task sample, a pre-trained execution model is used to take the first natural language task sample as the root node of the search tree. Starting from the root node, the search tree is iteratively expanded with child nodes based on the Monte Carlo tree search algorithm to obtain multiple inference paths associated with the first natural language task sample. For each first natural language task sample, a pre-trained validation model is used to evaluate the quality of multiple inference paths associated with the first natural language task sample, and the n inference paths with the highest quality scores are determined as candidate inference paths, where n is a pre-determined positive integer; For each candidate inference path associated with each first natural language task sample, the computational value score of the candidate inference path is determined based on the computational complexity and robust accuracy of the candidate inference path. For each first natural language task sample, the candidate reasoning path with the highest calculated value score among the n candidate reasoning paths associated with the first natural language task sample is abstracted to obtain the reasoning thinking pattern representation associated with the first natural language task sample. Based on the reasoning thought pattern representation associated with each of the first natural language task samples, a pre-established first policy model is trained using a reinforcement learning algorithm, wherein the first policy model is established based on a large language model with a different architecture than the execution model and the verification model. In the inference phase, in response to receiving a natural language task, a first inference path is generated using a trained first policy model, and the answer to the natural language task is determined based on the first inference path. The first inference path corresponds to a tree-based exploratory inference process.

[0132] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described inference method embodiments based on heterogeneous model reinforcement learning and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0133] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0134] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described inference method embodiment based on heterogeneous model reinforcement learning, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0135] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0136] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0137] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0138] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A reasoning method based on heterogeneous model reinforcement learning, characterized in that, The heterogeneity in the architectural design of the model combination that provides guidance information and the first strategy model that receives guidance constitutes a cross-model knowledge transfer mechanism. This mechanism transfers the interactive cognitive information from the execution model and the validation model to the reinforcement learning training process of the first strategy model, thereby achieving effective injection of new information from multiple external sources and complementary enhancement of knowledge between models. The model combination providing guidance information includes: a pre-trained execution model and a pre-trained validation model; the method includes: During the training phase, for each first natural language task sample, a pre-trained execution model is used to take the first natural language task sample as the root node of the search tree. Starting from the root node, the search tree is iteratively expanded with child nodes based on the Monte Carlo tree search algorithm to obtain multiple inference paths associated with the first natural language task sample. The pre-trained execution model is responsible for generating the inference paths. For each first natural language task sample, a pre-trained validation model is used to evaluate the quality of multiple inference paths associated with the first natural language task sample, and the n inference paths with the highest quality scores are determined as candidate inference paths, where n is a pre-determined positive integer; the pre-trained validation model is responsible for performing logical consistency checks and correctness checks on the inference paths generated by the pre-trained execution model in order to filter out erroneous or low-quality solutions. Using the input prompt words consisting of each first natural language task sample and each candidate inference path associated with it as a reference, the pre-established first policy model generates multiple inference paths, and calculates the ratio of the number of paths that lead to the correct answer through these multiple inference paths. Then, the robust accuracy of the candidate inference path is obtained by averaging the ratio of the number of paths. The total number of input tokens required for the pre-established first strategy model to generate multiple inference paths is determined as the computational complexity of the candidate inference path. For each candidate inference path associated with each first natural language task sample, a computational value score is determined based on the computational complexity and robust accuracy of the candidate inference path; the robust accuracy describes the overall probability of obtaining the correct answer using multiple inference paths generated by the pre-established first policy model; the computational complexity describes the actual computational cost of generating multiple inference paths using the pre-established first policy model. For each first natural language task sample, the candidate reasoning path with the highest calculated value score among the n candidate reasoning paths associated with the first natural language task sample is abstracted to obtain the reasoning thinking pattern representation associated with the first natural language task sample. Based on the reasoning thought pattern representation associated with each of the first natural language task samples, a pre-established first policy model is trained using a reinforcement learning algorithm, wherein the first policy model is established based on a large language model with a different architecture than the execution model and the verification model. In the inference phase, in response to receiving a natural language task, a first inference path is generated using a trained first policy model, and the answer to the natural language task is determined based on the first inference path. The first inference path corresponds to a tree-based exploratory inference process.

2. The method according to claim 1, characterized in that, The iterative expansion of the search tree from the root node using the Monte Carlo tree search algorithm includes: In the current iteration, based on the node value and access statistics of each node in the search tree, the search tree is searched from the root node until the leaf node to be expanded is found in the search tree. In the current iteration, generate each child node of the leaf node to be expanded, and assign an initial node value to each newly generated child node; In the current iteration, based on the path from each newly generated child node to the root node, determine the inference path associated with each newly generated child node; In the current iteration, based on the reasoning path associated with each newly generated child node, the simulated reasoning answer of the first natural language task sample associated with the search tree is determined; In the current iteration round, based on the simulated reasoning answer associated with each newly generated child node, starting from each newly generated child node, the node value and access statistics of all nodes traversed by the reasoning path associated with each newly generated child node are updated one by one, and a new iteration round is executed after the node value and access statistics are updated. The node value is updated based on the node value of each node itself and its child nodes.

3. The method according to claim 2, characterized in that, Each node in the search tree, except the root node, corresponds to a reasoning operation or cognitive step. The reasoning path is determined based on the path from the leaf node to the root node in the search tree. Generating the child nodes of the leaf node to be expanded includes: Based on the execution order of different reasoning operations and different cognitive steps in the reasoning process, and the reasoning operation or cognitive step corresponding to the leaf node to be expanded, select each reasoning operation and / or each cognitive step that conforms to the execution order from the basic cognitive behavior pattern library. For the leaf nodes to be expanded, generate child nodes corresponding to each selected reasoning operation and / or cognitive step.

4. The method according to claim 1, characterized in that, The step of training a pre-established first policy model based on a reinforcement learning algorithm, according to the reasoning thought pattern representation associated with each of the first natural language task samples, includes: For each second natural language task sample, based on the semantic matching algorithm, each reasoning thought pattern representation that matches the second natural language task sample is selected from the reasoning thought pattern representation associated with each first natural language task sample as each knowledge guidance strategy. The first policy model is trained based on the group relative policy optimization algorithm, according to the knowledge guidance strategies associated with each second natural language task sample.

5. The method according to claim 4, characterized in that, The method further includes: During the training of the first policy model based on the group relative policy optimization algorithm, the probability of an erroneous solution appearing in each candidate solution group associated with the previous policy of the first policy model is determined. For candidate solution groups where the probability of an erroneous solution is greater than a preset value, the number of candidate solutions in the candidate solution group is increased, and / or the weight of the parameter terms associated with the candidate solution group in the objective function is decreased; The objective function used in the group relative strategy optimization algorithm is... It is expressed as follows: in, Let E represent the parameters of the first strategy model, and E represent the expectation. Represents the distribution of samples from natural language tasks A specific natural language task sample obtained by mid-sampling , This represents samples from the same natural language task. The number of candidate solution groups obtained from sampling. This indicates that the first policy model is based on the previous policy. Samples for natural language tasks generated candidate solution set The probability distribution, This represents the total number of knowledge-guided strategies. This represents the weight coefficient of the i-th knowledge guidance strategy. Indicates the first Candidate solution groups associated with each knowledge-guided strategy The number of candidate solutions, Indicates group-based relative reward for The advantage value calculated for the t-th candidate solution is... Indicates the current strategy and reference strategy Calculated KL divergence, probability ratio Cutting ratio , and Indicates hyperparameters, The first policy model is based on the current policy. Samples for natural language tasks and candidate solutions generated candidate solutions The probability distribution, This indicates that the first policy model is based on the previous policy. Samples for natural language tasks and candidate solutions generated candidate solutions The probability distribution, express The t-th candidate solution, express The candidate solutions located before the t-th candidate solution. This represents the operation of taking the minimum value. This indicates a cropping operation.

6. The method according to claim 1, characterized in that, The computational value score of the candidate reasoning path is determined using the following formula: in, Representing natural language task samples One of the associated candidate reasoning paths, Representing candidate reasoning paths The computational value score, where m represents the balance adjustment parameter between benefits and computational costs. Representing candidate reasoning paths robust accuracy Representing candidate reasoning paths The computational complexity is...

7. The method according to any one of claims 1-6, characterized in that, The method further includes: During the inference phase, in response to receiving a first instruction associated with the performance-first mode, a first inference path is generated using the trained first policy model, and the answer to the natural language task targeted by the first instruction is determined based on the first inference path. During the inference phase, in response to receiving a second instruction associated with the speed-first mode, a second inference path is generated using a pre-trained second policy model, and the answer to the natural language task targeted by the second instruction is determined based on the second inference path, which corresponds to a step-by-step inference process.

8. An inference system based on heterogeneous model reinforcement learning, characterized in that, The heterogeneity in the architectural design of the model combination that provides guidance information and the first strategy model that receives guidance constitutes a cross-model knowledge transfer mechanism. This mechanism transfers the interactive cognitive information from the execution model and the validation model to the reinforcement learning training process of the first strategy model, thereby achieving effective injection of new information from multiple external sources and complementary enhancement of knowledge between models. The model combination providing guidance information includes: a pre-trained execution model and a pre-trained validation model; the system includes a high-order knowledge extraction module, a heterogeneous model knowledge transfer and reasoning ability optimization module, and a reasoning module, wherein: The advanced knowledge extraction module is configured to, during the training phase, for each first natural language task sample, use a pre-trained execution model to take the first natural language task sample as the root node of a search tree, and iteratively expand the search tree from the root node using a Monte Carlo tree search algorithm to obtain multiple inference paths associated with the first natural language task sample; the pre-trained execution model is responsible for generating the inference paths; and for each first natural language task sample, use a pre-trained verification model to perform quality evaluation on the multiple inference paths associated with the first natural language task sample, and determine the n inference paths with the highest quality scores as candidate inference paths, where n is a pre-determined positive integer; the pre-trained verification model is responsible for performing logical consistency checks and correctness checks on the inference paths generated by the pre-trained execution model to filter out erroneous or low-quality solutions; and using the input prompt words composed of each first natural language task sample and each of its associated candidate inference paths as a reference, the pre-established first strategy model generates... Multiple reasoning paths are generated, and the ratio of the number of paths leading to the correct answer is calculated. The average of these ratios is then used to obtain the robust accuracy of the candidate reasoning paths. The total number of input tokens required for a pre-established first strategy model to generate multiple reasoning paths is determined as the computational complexity of the candidate reasoning paths. For each candidate reasoning path associated with a first natural language task sample, a computational value score is determined based on the computational complexity and robust accuracy of the candidate reasoning path. The robust accuracy describes the overall probability of obtaining the correct answer using multiple reasoning paths generated by the pre-established first strategy model. The computational complexity describes the actual computational cost of generating multiple reasoning paths using the pre-established first strategy model. For each first natural language task sample, the candidate reasoning path with the highest computational value score among the n candidate reasoning paths associated with the first natural language task sample is abstracted to obtain a representation of the reasoning thought pattern associated with the first natural language task sample. The heterogeneous model knowledge transfer and reasoning ability optimization module is configured to train a pre-established first policy model based on a reinforcement learning algorithm according to the reasoning thinking pattern representation associated with each of the first natural language task samples. The first policy model is established based on a large language model with a different architecture than the execution model and the verification model. The reasoning module is configured to, in response to receiving a natural language task during the reasoning phase, generate a first reasoning path using a trained first policy model, and determine the answer to the natural language task based on the first reasoning path, wherein the first reasoning path corresponds to a tree-based exploratory reasoning process.

9. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the inference method based on heterogeneous model reinforcement learning as described in any one of claims 1-7.

10. A readable storage medium, characterized in that, The program or instructions are stored on the readable storage medium, and when executed by a processor, the program or instructions implement the steps of the inference method based on heterogeneous model reinforcement learning as described in any one of claims 1-7.

Images