Tool integration reasoning method based on self-evolution preference learning
By optimizing the tool invocation behavior of the model through information entropy analysis and multi-stage self-evolutionary training paradigm, the problems of poor efficiency and accuracy in tool integrated reasoning are solved, and a more efficient, stable reasoning process and stronger generalization ability are achieved.
Patent Information
- Application Number
- CN202511368488.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-24
- Publication Date
- 2026-01-09
AI Technical Summary
Existing tool integration inference techniques struggle to balance the number of tool calls, resulting in poor efficiency and accuracy. In complex tasks, models are prone to redundant or insufficient calls, and the lack of in-depth analysis of internal states leads to unstable inference.
We employ a training data construction based on information entropy and a multi-stage self-evolutionary training paradigm. By analyzing information entropy, we identify high-entropy nodes and generate multiple branch inference paths. We then combine supervised fine-tuning and self-evolutionary direct preference optimization to optimize the model's tool invocation behavior.
It significantly improves the efficiency and accuracy of the model in complex tasks, reduces redundant calls, enhances the stability and generalization ability of the inference process, and reduces the cost of training data construction.
Smart Images
Figure CN121303330A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of computer science, and more specifically, to a tool integration reasoning method that learns through self-evolutionary preference learning. Background Technology
[0002] In recent years, large language models (LLMs) have demonstrated powerful capabilities in logical reasoning, mathematical computation, and code generation. However, when faced with complex tasks requiring deep information retrieval or precise computation, relying solely on the model's internal knowledge and reasoning abilities is often insufficient. To overcome these limitations, tool-integrated reasoning (TIR) technology has emerged. This technology allows large language models to autonomously invoke external tools (such as code interpreters and web search engines) during the reasoning process to compensate for deficiencies in their internal knowledge or capabilities, thereby improving their performance on complex tasks.
[0003] Currently, technical solutions for improving tool integration reasoning capabilities mainly revolve around fine-tuning the model, particularly using reinforcement learning (RL) for optimization. For example, some research uses reinforcement learning to train models by carefully designing reward functions to reduce unnecessary tool calls. Others use metacognitive theory or leverage the model's internal signals to control its behavior, guiding the model to use tools more rationally. These methods have improved the efficiency of models performing TIR tasks to some extent.
[0004] 2.2 Problems with existing technologies
[0005] Despite some progress in existing tool integration inference techniques, many challenges remain in practical applications, resulting in poor efficiency and accuracy of the model during the inference process.
[0006] First, existing technologies struggle to balance the "degree" of tool calls. Models often exhibit suboptimal tool-calling behavior, such as excessive tool calls (redundant tool queries for simple problems, increasing inference costs and time) or insufficient tool calls (failure to call necessary tools to obtain information in critical steps, leading to inference failure). Existing methods primarily focus on reducing excessive tool use, neglecting the problem of insufficient tool calls.
[0007] Secondly, after receiving results from the tool, the model is prone to "overthinking" or even "analysis paralysis." This is especially true when the tool's results are of low quality or complex, making it difficult for the model to effectively integrate information and continue subsequent reasoning steps, thus affecting the stability of the entire reasoning process.
[0008] Furthermore, many existing optimization methods are mainly trained for the use cases of a single tool. When the task requires the model to flexibly call multiple different types of tools, its performance is often difficult to guarantee.
[0009] Finally, most existing research optimizes models from external behaviors (such as the number of calls and the accuracy of the final answer), lacking in-depth analysis of the internal state changes during model inference. This makes the optimization process lack deeper theoretical guidance and makes it difficult to fundamentally solve the problem of instability in inference. In summary, existing technologies face the following challenges:
[0010] (1) How can we enable the model to learn to use tools prudently, ensuring task accuracy while avoiding overuse and underuse of tools, so as to improve inference efficiency?
[0011] (2) How can we utilize our understanding of the internal dynamics of the model's reasoning process (such as changes in information entropy) to more effectively guide the training of the model and enable it to master more stable and efficient tool integration reasoning capabilities? Summary of the Invention
[0012] The purpose of this disclosure is to provide a tool integration reasoning method through self-evolutionary preference learning, which aims to solve the problems of existing technologies lacking analysis and utilization of the internal state of the model reasoning process, excessive or insufficient model tool calls, and efficiently constructing high-quality training data.
[0013] In general, this paper provides a tool integration reasoning method based on self-evolutionary preference learning, comprising two parts: first, information entropy-based training data construction; second, a multi-stage self-evolutionary training paradigm.
[0014] The training data construction process based on information entropy is used to construct training data, including data source selection and entropy-guided sampling process;
[0015] The multi-stage self-evolutionary training paradigm is used to improve the model’s tool integration reasoning ability, and includes a two-stage paradigm of supervised fine-tuning and self-evolutionary direct preference optimization.
[0016] The reasoning method, given an input instruction I and a large language model θ, generates the final answer y through a calculation process, which is expressed as follows:
[0017]
[0018] Where i represents the reasoning step, N is the total number of reasoning steps, and R i For the reasoning content of step i, T C<i R represents the results of all tool calls up to step i. <i This represents the reasoning content up to step i.
[0019] The source data filtering process involves constructing a dataset D of difficult problems. source The specific steps are as follows: Use an existing supervised fine-tuning dataset D containing standard answers. sft Train a basic instruction model to obtain model M. sft Using model M sft Without providing any external tools, directly targeting D sft The problem is used to reason, and all data whose model reasoning results are incorrect are filtered out to form the source dataset D. source .
[0020] The specific steps of the entropy-guided sampling strategy are as follows:
[0021] Step 1: Generating the main inference chain: For D source For each problem in the model, model M is first used. sft Generate a complete TIR inference path, which serves as the main chain C. main ;
[0022] Step 2, High-entropy node location: Calculate the main chain C main The information entropy of the initial part of each reasoning step is calculated, and for each step, the average information entropy H of its first i lexical units is calculated. avg (i), and take the maximum value among them as the entropy value of this step, as shown in the formula: Where H(j) is the information entropy of the j-th word position, based on which the k steps with the highest entropy value in the whole main chain and their corresponding positions are selected;
[0023] Step 3, Branch Path Sampling: The inference sequence preceding each high-entropy node is used as new input to guide the model to generate multiple different branch inference paths starting from that node. All inference paths generated by combining ordinary sampling and entropy-guided sampling together constitute the candidate dataset D for subsequent training. dpo .
[0024] The supervised fine-tuning phase uses the standard SFT dataset D for the basic instruction model. sft After fine-tuning, the model Msft is obtained.
[0025] The self-evolutionary direct preference optimization stage continuously optimizes the model's inference preferences through an iterative "sampling-training" loop, and includes two steps:
[0026] Step 1: Pre-aligned DPO training: First, data filtering is performed using the candidate dataset D. dpo The system designs and constructs positive and negative sample pairs based on screening criteria. Problems are categorized into "difficult" and "easy" sets according to the proportion of correct reasoning paths. Positive samples select reasoning paths with correct reasoning, the fewest tool calls, and the lowest overall entropy. Negative samples select paths with incorrect reasoning and more tool calls than their corresponding positive samples. Next, the model is trained using the constructed positive and negative sample pairs. sft DPO training yields a higher-performing model M. dpo1 ;
[0027] Step 2, Self-evolutionary DPO alignment:
[0028] First, perform an iterative loop. In each round: first, resample: use the model M obtained from the previous round of training. dpo1 For source data D source Perform inference sampling again to generate a new path set;
[0029] Second, the criteria are updated, and new screening criteria are used to construct positive and negative sample pairs. The new criteria specifically encourage the model to select the longer correct path as a positive sample on "hard set" problems, while penalizing inefficient behavior on "easy set" problems.
[0030] Third, continue training by using new sample pairs to perform DPO training on the model, resulting in the updated model M. dpo2 ;
[0031] Step 3: Repeat the loop several times until the model performance converges. The final model is the target model to be trained in this invention.
[0032] The technical effects to be achieved by the embodiments of the present invention are as follows:
[0033] (1) To address the lack of analysis and utilization of the internal state of the model's reasoning process in existing technologies, this invention, for the first time, models the tool-integrated reasoning process from the perspective of information entropy. Analysis reveals that after receiving the tool's call results, the information entropy of the subsequently generated content exhibits a regular changing trend; furthermore, for the same problem with multiple correct reasoning paths, the path with fewer tool calls typically has a lower overall information entropy. This finding provides a new internal indicator and theoretical basis for optimizing the model's reasoning behavior.
[0034] (2) To address the issues of excessive or insufficient tool calls in the model and to efficiently construct high-quality training data, this invention designs an innovative entropy-guided sampling strategy based on the aforementioned information entropy analysis. This method first generates a main inference chain, then identifies and locates several key nodes with the highest information entropy in the inference chain (i.e., the most "uncertain" decision points in the model), and generates multiple branch inference paths from these nodes. In this way, more diverse inference possibilities can be explored at low cost, and high-quality positive (preferred) and negative (discarded) sample pairs are selected by combining dimensions such as the correctness of the inference results, the number of tool calls, and the length of the inference chain for subsequent preference learning in the model.
[0035] (3) Based on the above ideas, this invention proposes a multi-stage self-evolving preference learning framework (Tool-Light). This framework is the first to combine information entropy theory with a self-evolving training mechanism to optimize tool ensemble reasoning. The framework first enables the model to acquire basic tool-using capabilities through supervised fine-tuning (SFT); subsequently, it enters an innovative self-evolving direct preference optimization (DPO) stage. This stage is carried out in two steps: first, pre-aligned DPO training, using entropy-guided sampling data to teach the model to reduce redundant tool calls and unnecessary overthinking; then, self-evolving DPO alignment, through multiple rounds of iterative "sampling-training," allows the model to learn to make necessary tool calls while maintaining efficient reasoning capabilities. By integrating internal state analysis, efficient data sampling, and progressive preference learning, this framework systematically improves the model's reasoning accuracy, efficiency, and stability in multi-tool scenarios.
[0036] The present invention, through the above technical solution, can bring the following significant beneficial effects:
[0037] 1. Improved efficiency and accuracy: Through refined preference learning, the model learns to balance when to call, which tool to call, and how many times to call the tool, significantly reducing redundant and erroneous tool calls while ensuring the accuracy of problem solving.
[0038] 2. More stable and concise reasoning process: By guiding the model to learn low-entropy reasoning paths, the phenomenon of "overthinking" is effectively alleviated, making the model's reasoning chain shorter and more logical, and improving the stability and interpretability of the output results.
[0039] 3. Lower training data construction cost: The innovative entropy-guided sampling strategy can more efficiently explore the reasoning paths that the model may err on. Compared with the traditional brute-force sampling method, it greatly reduces the time and computational cost required to build a high-quality preference learning dataset.
[0040] 4. Stronger model generalization ability: Because the training data covers a variety of complex reasoning scenarios explored by entropy, the trained model shows stronger generalization ability when faced with unseen complex tasks that require the comprehensive application of knowledge from different domains. Attached Figure Description
[0041] The above and other objects and features of this disclosure will become clearer from the following description taken in conjunction with the accompanying drawings.
[0042] Figure 1 This is a schematic diagram illustrating an example of the basic process of tool integration reasoning in existing technologies;
[0043] Figure 2 This is a schematic diagram illustrating the architecture of a data construction method based on information entropy according to an embodiment of the present disclosure;
[0044] Figure 3 This is a schematic diagram illustrating a multi-stage self-evolutionary training paradigm architecture according to an embodiment of the present disclosure. Detailed Implementation
[0045] The following detailed embodiments are provided to aid the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but may be changed as will become clear upon understanding this disclosure, except for operations that must occur in a specific order. Furthermore, for clarity and conciseness, descriptions of features known in the art may be omitted.
[0046] The features described herein may be implemented in different forms and should not be construed as limited to the examples described herein. Rather, the examples described herein are provided only to illustrate some of the many feasible ways of implementing the methods, apparatus, and / or systems described herein, which will become clear upon understanding the disclosure of this application.
[0047] As used herein, the term “and / or” includes any one of the associated listed items and any combination of any two or more.
[0048] Although terms such as “first,” “second,” and “third” may be used herein to describe various components, assemblies, regions, layers, or parts, these components, assemblies, regions, layers, or parts should not be limited by these terms. Rather, these terms are used only to distinguish one component, assembly, region, layer, or part from another. Thus, without departing from the teaching of the examples described herein, the first component, first assembly, first region, first layer, or first part referred to as the first component, first assembly, first region, first layer, or first part may also be referred to as the second component, second assembly, second region, second layer, or second part.
[0049] In the specification, when an element (such as a layer, region, or substrate) is described as being "on" another element, "connected to," or "bonded to" another element, the element may be directly "on" another element, directly "connected to," or "bonded to" the other element, or one or more other elements may be present in between. Conversely, when an element is described as being "directly on" another element, "directly connected to," or "directly bonded to" another element, no other elements may be present in between.
[0050] The terminology used herein is for the purpose of describing various examples only and is not intended to limit disclosure. Unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. The terms “comprising,” “including,” and “having” indicate the presence of the described features, quantities, operations, components, elements, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, quantities, operations, components, elements, and / or combinations thereof.
[0051] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains upon understanding this disclosure. Unless expressly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and in this disclosure, and shall not be interpreted in an idealized or overly formalistic manner.
[0052] Furthermore, in the description of the examples, detailed descriptions of well-known related structures or functions will be omitted when it is believed that such detailed descriptions would lead to a vague interpretation of this disclosure.
[0053] Figure 2 This is a schematic diagram illustrating a data construction method based on information entropy according to an embodiment of the present disclosure.
[0054] To achieve the aforementioned objectives, the present invention employs the following technical framework: Figure 3 As shown.
[0055] This invention provides an optimization method for Tool-Integrated Reasoning (TIR) based on self-evolutionary preference learning, aiming to systematically improve the efficiency, accuracy, and stability of Large Language Models (LLMs) when invoking external tools to solve complex problems. The overall framework of this invention, namely the Too-Light method, is as follows: Figure 2 and Figure 3 As shown, it mainly includes two core parts: (1) training data construction based on information entropy; and (2) a multi-stage self-evolutionary training paradigm. The following will elaborate on each part.
[0056] Tool Integration Reasoning Task Definition:
[0057] Given an input instruction I and a large language model θ, the goal of Multi-Tool Integrated Reasoning (Multi-TIR) is to generate the final answer y. For example, given a question-and-answer instruction "What is the capital of China? Please answer using the Bing search tool. The output format should be: first, the thought process; then, the tool execution process; and finally, the answer." This instruction is input into the Qwen2.5-7B-Instruct model. Ideally, the correct output of the model should be " <think> Okay, first I need to search for the capital of China.< / think> <search> China's capital< / search> <result> The capital of China is Beijing.< / result> <think> Okay, I know the answer now.< / think> <answer> The correct answer is \boxed{Beijing}< / answer> ", that is, to give the correct answer." This process can be formally represented as:
[0058]
[0059] Where i represents the reasoning step, N is the total number of reasoning steps, and R i For the reasoning content of step i, T C<i
[0060] and R <i These represent the results of all tool calls and the inference content up to step i, respectively. The technical solution of this invention aims to optimize model θ, enabling it to generate higher-quality inference paths, thereby obtaining a more accurate answer y.
[0061] Construction of training data based on information entropy
[0062] In order to efficiently obtain high-quality training data that can teach the model how to use the tools correctly, this invention designs a construction process that includes data source filtering and entropy-guided sampling.
[0063] Source data filtering
[0064] To make model training more targeted, this invention first constructs a "difficult problem" dataset D. sourceThe specific steps are as follows: Use an existing supervised fine-tuning (SFT) dataset D that contains standard answers. sft Train a basic instruction model to obtain model M. sft Using model M sft Without providing any external tools, directly targeting D sft The problem is reasoned about. All data with incorrect model inference results are filtered out, forming the source dataset D. source The data selected through this method is usually difficult for the model to process using only its internal knowledge, thus allowing subsequent tool training to focus more on improving the model's key capabilities.
[0065] Entropy-guided sampling strategy
[0066] In obtaining the source dataset D source Subsequently, this invention employs an innovative hybrid strategy combining ordinary sampling and entropy-guided sampling to generate a set of inference paths D for preference learning. dpo The specific process of entropy-guided sampling is as follows: Figure 2 As shown, its core idea is that during the model's reasoning process, the higher the information entropy, the more "uncertain" the model is about its next decision. Exploring from these nodes with high uncertainty can generate more diverse and valuable reasoning paths at a lower cost. The steps are as follows:
[0067] Step 1, Main inference chain generation: For D source For each problem in the model, model M is first used. sft Generate a complete TIR inference path, which serves as the main chain C. main .
[0068] Step 2, High-entropy node location: Calculate the main chain C main The information entropy of the initial part of each reasoning step. Specifically, for each step, the average information entropy H of its first i tokens is calculated. avg (i), and take the maximum value among them as the entropy value of this step. The formula is as follows:
[0069]
[0070] Where H(j) is the information entropy of the j-th lexical position. Based on this, the k steps with the highest entropy values in the entire main chain and their corresponding positions are selected.
[0071] Step 3, Branch Path Sampling: The inference sequence preceding each high-entropy node is used as new input to guide the model to generate multiple different branch inference paths starting from that node. All inference paths generated by combining ordinary sampling and entropy-guided sampling together constitute the candidate dataset D for subsequent training. dpo.
[0072] Multi-stage self-evolutionary training paradigm
[0073] The overall training process of this invention is as follows: Figure 3 As shown, a two-stage paradigm involving supervised fine-tuning and self-evolutionary direct preference optimization is adopted to gradually improve the model's TIR capability.
[0074] Phase 1: Supervised Fine-Tuning (SFT)
[0075] like Figure 3 As shown in the first component, the present invention first addresses a basic instruction model (M) instruct t) Using the standard SFT dataset D sft Fine-tuning was performed to give it the initial ability to understand and perform TIR tasks, resulting in the Msft model. This stage lays the foundation for subsequent preference learning.
[0076] Phase Two: Self-Evolved Direct Preference Optimization (DPO)
[0077] This stage is the core of the invention, continuously optimizing the model's inference preferences through an iterative "sampling-training" cycle. It comprises two key steps:
[0078] Step 1: Pre-Aligned DPO Training
[0079] like Figure 3 As shown in the second component, the goal of this step is to teach the model to reduce unnecessary tool calls and "overthinking".
[0080] a) Data filtering: First, use the dataset D constructed in Section 4.2. dpo A set of screening criteria (Criteria 1) was designed to construct positive and negative sample pairs. Specifically, the problems were divided into a "difficult set" and an "easy set" based on the proportion of correct reasoning paths.
[0081] I. Positive Example: Select the inference path that is correct, has the fewest tool calls, and the lowest overall entropy value.
[0082] II. Negative Example: Select the path where the inference is incorrect and the tool is called more times than the corresponding positive example.
[0083] b) Model training: Using the constructed positive and negative sample pairs, train the model M obtained in the SFT stage. sft DPO training yields a higher-performing model M. dpo1 .
[0084] Step 2: Self-Evolved DPO Alignment
[0085] like Figure 3 As shown in the third component, the goal of this step is to teach the model to make "necessary" tool calls while maintaining its efficient reasoning capabilities, thus addressing the problem of "insufficient tool calls".
[0086] a) Iterative loop: This step is an iterative process. In each round:
[0087] I. Resampling: Using the model obtained from the previous training round (e.g., M) dpo1 ) For source data D source Perform inference sampling again to generate a new path set.
[0088] II. Updated Criteria: A new set of selection criteria (Criteria2) is adopted to construct positive and negative sample pairs. This criterion specifically encourages models to explore more thoroughly on "hard set" problems (i.e., select the longer correct path as a positive sample), while penalizing inefficient behavior on "easy set" problems.
[0089] III. Continue Training: Use the new sample pairs to continue DPO training on the model to obtain the updated model (e.g., M). dpo2 ).
[0090] (b) This loop will be repeated several times until the model performance converges. The final model obtained is the target model to be trained in this invention.
[0091] Regarding "constructing training data based on information entropy", the following alternative methods can be used:
[0092] Alternative methods for "source data filtering":
[0093] In this invention, a "difficult problem" dataset D is constructed by filtering samples where the model's reasoning is flawed. source As an alternative, the selection criteria do not have to be limited to the final correctness of the answer. For example, samples with low model output confidence or exhibiting significant uncertainty during inference (such as a flat internal probability distribution) can be selected, even if their final answer is correct, and can be considered "difficult questions" and added to the D algorithm. source Dataset. This method can also filter out key samples on the boundary of the model's capabilities, achieving a similar training focus effect.
[0094] Alternatives to the "entropy-guided sampling strategy":
[0095] a) Alternatives to uncertainty measurement methods: This invention uses information entropy as the core indicator for measuring model uncertainty. Alternatively, other indicators that reflect model uncertainty can be used, such as model perplexity, variance of the lexical probability distribution, or the Gini coefficient. Selecting the node with the highest perplexity or the most dispersed probability distribution during inference for branch sampling can also efficiently explore the diverse behaviors of the model, achieving results similar to using information entropy.
[0096] b) Alternative Sampling Process: This invention employs a "main chain + branch" sampling method. As an alternative, beam search can be used to simultaneously generate and retain multiple (e.g., k) candidate inference paths at each inference step. After completing the entire inference process, positive and negative sample pairs are directly selected from the final k complete paths based on comprehensive criteria such as correctness, tool call count, and path length. Although this method has a higher computational cost, it can systematically generate a high-quality set of candidate paths.
[0097] Regarding "self-evolutionary direct preference optimization", the following alternative approach can be adopted:
[0098] This invention employs the Direct Preference Optimization (DPO) algorithm to train the model. As an alternative, other mainstream preference learning or reinforcement learning algorithms can be used, such as Proximal Policy Optimization (PPO), Identity Preference Optimization (IPO), or Kahneman-Tversky Optimization (KTO). Although these algorithms differ in their objective functions and optimization details, their core objective is to align the model's output with the preset preferences (i.e., the better inference path defined in this invention), thus achieving performance optimization results similar to DPO.
[0099] Alternative methods for selecting positive and negative samples:
[0100] This invention designs specific positive and negative sample selection criteria (Criteria 1 and Criteria 2) during the pre-alignment and self-evolution stages. Alternatively, a unified comprehensive scoring function can be designed, which weights and sums multiple dimensions such as the correctness of the inference path, the number of tool calls, the length of the inference chain, and information entropy to obtain a final path quality score. Among all sampled paths, the one with the highest score is designated as a positive sample, and the one with the lowest score is designated as a negative sample. By adjusting the weights, this method can flexibly adapt to different optimization objectives (e.g., focusing more on efficiency or more on accuracy), while still achieving the beneficial effect of finely optimizing model behavior.
[0101] Regarding alternatives to the base model used:
[0102] The technical solution described in this invention is based on a general large language model. This solution does not depend on any specific model architecture, therefore, it can be applied to other open-source large language models with different parameter scales and architectures in the industry, such as the LLaMA series, Gemini series, and Mistral series. All of them can improve their tool integration reasoning capabilities through the method described in this invention, demonstrating good versatility.
[0103] While some embodiments of this disclosure have been shown and described, those skilled in the art will understand that modifications may be made to these embodiments without departing from the principles and spirit of this disclosure, which are defined by the claims and their equivalents.
Claims
1. A tool integration reasoning method based on self-evolutionary preference learning, characterized in that, It consists of two parts: first, the construction of training data based on information entropy; second, a multi-stage self-evolving training paradigm. The training data construction process based on information entropy is used to construct training data, including data source selection and entropy-guided sampling process; The multi-stage self-evolutionary training paradigm is used to improve the model’s tool integration reasoning ability, and includes a two-stage paradigm of supervised fine-tuning and self-evolutionary direct preference optimization. The reasoning method, given the input question-answering instruction I and the large language model θ, generates the final answer y through a calculation process, which is expressed as follows: Where i represents the reasoning step, N is the total number of reasoning steps, and R i For the reasoning content of step i, T C<i R represents the results of all tool calls up to step i. <i This represents the reasoning content up to step i.
2. The tool integration reasoning method based on self-evolutionary preference learning as described in claim 1, characterized in that, The source data filtering process involves constructing a dataset D of difficult problems. source The specific steps are as follows: Use an existing supervised fine-tuning dataset D containing standard answers. sft Train a basic instruction model to obtain model M. sft Using model M sft Without providing any external tools, directly targeting D sft The problem is used to reason, and all data whose model reasoning results are incorrect are filtered out to form the source dataset D. source .
3. The tool integration reasoning method based on self-evolutionary preference learning as described in claim 2, characterized in that, The specific steps of the entropy-guided sampling strategy are as follows: Step 1: Generating the main inference chain: For D source For each problem in the model, model M is first used. sft Generate a complete TIR inference path, which serves as the main chain C. main ; Step 2, High-entropy node location: Calculate the main chain C main The information entropy of the initial part of each reasoning step is calculated, and for each step, the average information entropy H of its first i lexical units is calculated. avg (i), and take the maximum value among them as the entropy value of this step, as shown in the formula: Where H(j) is the information entropy of the j-th word position, based on which the k steps with the highest entropy value in the whole main chain and their corresponding positions are selected; Step 3, Branch Path Sampling: The inference sequence preceding each high-entropy node is used as new input to guide the model to generate multiple different branch inference paths starting from that node. All inference paths generated by combining ordinary sampling and entropy-guided sampling together constitute the candidate dataset D for subsequent training. dpo .
4. The tool integration reasoning method based on self-evolutionary preference learning as described in claim 3, characterized in that, The supervised fine-tuning phase uses the standard SFT dataset D for the basic instruction model. sft After fine-tuning, the model Msft is obtained.
5. The tool integration reasoning method based on self-evolutionary preference learning as described in claim 4, characterized in that, The self-evolutionary direct preference optimization stage continuously optimizes the model's inference preferences through an iterative "sampling-training" loop, and includes two steps: Step 1: Pre-aligned DPO training: First, data filtering is performed using the candidate dataset D. dpo The system designs and constructs positive and negative sample pairs based on screening criteria. Problems are categorized into "difficult" and "easy" sets according to the proportion of correct reasoning paths. Positive samples select reasoning paths with correct reasoning, the fewest tool calls, and the lowest overall entropy. Negative samples select paths with incorrect reasoning and more tool calls than their corresponding positive samples. Next, the model is trained using the constructed positive and negative sample pairs. sft DPO training yields a higher-performing model M. dpo1 ; Step 2, Self-evolutionary DPO alignment: First, perform an iterative loop. In each round: first, resample: use the model M obtained from the previous round of training. dpo1 For source data D source Perform inference sampling again to generate a new path set; Second, the criteria are updated, and new screening criteria are used to construct positive and negative sample pairs. The new criteria specifically encourage the model to select the longer correct path as a positive sample on "hard set" problems, while penalizing inefficient behavior on "easy set" problems. Third, continue training by using new sample pairs to perform DPO training on the model, resulting in the updated model M. dpo2 ; Step 3: Repeat the loop several times until the model performance converges. The final model is the target model to be trained in this invention.