A large model multi-type tool collaborative reasoning method, system and device
By employing an adaptive exploration and reinforcement learning strategy based on token entropy, this study addresses the issues of single reasoning paths and rigid decision-making in large language models during complex reasoning tasks. It achieves intelligent enhancement of the reasoning tool selection strategy for large models under uncertain conditions, thereby improving the accuracy and robustness of the final answer.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- INSPUR GENERSOFT CO LTD
- Filing Date
- 2025-09-05
- Publication Date
- 2026-05-05
AI Technical Summary
When large language models call external inference tools to perform complex inference tasks, they suffer from single inference paths and rigid decision-making, especially at decision nodes where they tend to choose suboptimal paths. Existing methods ignore fine-grained decision-making, resulting in high token entropy and uncertainty.
An adaptive exploration and reinforcement learning strategy based on token entropy is adopted. By calculating the entropy change of inference nodes, the uncertainty in the inference process is monitored in real time. An exploration mechanism is triggered at nodes with high token entropy to generate diverse inference paths. The GRPO algorithm is used to train and optimize large models.
It improves the reasoning tool selection strategy for large models under uncertain conditions, enhances the accuracy and robustness of the final answer, and achieves diversified reasoning path optimization.
Smart Images
Figure CN120745849B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence and natural language processing, and in particular to a method, system and device for collaborative reasoning using large-scale models and multiple types of tools. Background Technology
[0002] Currently, in order to solve open and complex reasoning tasks, large language models (LLMs) are deeply integrated with external reasoning tools. However, when large language models call external reasoning tools to perform complex reasoning tasks, there are problems such as single reasoning paths and rigid decision-making, especially the tendency to choose suboptimal paths at decision nodes.
[0003] Reinforcement learning for intelligent agents has become a key training paradigm for realizing dynamic interaction between large language models and the environment. Mainstream methods often employ inference path-based optimization algorithms such as GRPO (Group Relative Policy Optimization), which sample the inference paths invoked by LLM inference tools and reward based on the final result.
[0004] However, this model has a significant limitation: treating inference as a whole ignores the fine-grained decisions made by the large model at each inference tool invocation step. Existing research shows that each tool feedback introduces high uncertainty, manifested as high token entropy. Current methods, by overemphasizing comparisons of the entire path, suffer from insufficient exploration at these critical nodes, making it difficult to develop diverse inference tool usage strategies. This limits the optimization potential of the model's inference path and its robustness in ultimately solving the problem. Summary of the Invention
[0005] To address the aforementioned issues, this invention proposes a collaborative reasoning method, system, and device for large-scale models using multiple tools. It designs an adaptive exploration and reinforcement learning strategy based on token entropy, which intelligently enhances the selection strategy for reasoning tools in large-scale models under uncertain conditions, thereby improving the accuracy of the final answer reasoning.
[0006] To achieve the above objectives, the present invention adopts the following technical solution:
[0007] In a first aspect, the present invention provides a collaborative reasoning method for large-scale models with multiple types of tools, including:
[0008] For the original inference path generated by the large model for the problem, calculate the preceding inference nodes generated by the large model after each inference tool call. The average entropy of each token fragment is used as the entropy value of the current inference node.
[0009] Based on the entropy values of the current inference node, the previous inference node, and the initial inference node, determine the entropy change of the current inference node. Based on the comparison of the entropy change with the set entropy change threshold, determine the high token entropy node.
[0010] When the next inference tool is invoked at a high token entropy node, a different inference tool than the original inference path is randomly invoked and the inference process continues to be executed, generating a new inference path. The entropy value of each inference node is calculated for the new inference path until there are no high token entropy nodes.
[0011] The original inference path and all the generated new inference paths are used as the training set to train the large model based on the reinforcement learning algorithm.
[0012] The final answer to the problem is generated using a large, trained model.
[0013] As an alternative implementation method, for the problem The process of collaborative reasoning based on multiple types of tools using a large model is modeled as follows:
[0014] ;
[0015] Among them, factors This represents the reasoning process for calling multiple types of tools. Represents the chain of thought The number of tokens in the data. For position token, Indicates position All previous tokens; Indicates the introduction of a set of reasoning tools Model instructions; Indicates position Feedback from all previous historical inference tools; factors This indicates the process of generating the answer. Indicate the answer The number of tokens, For position The model generation results, For position Previous model historical generation results.
[0016] As an alternative implementation method, reasoning toolset This includes local corpus search tools, online web search tools, and code tools.
[0017] As an alternative implementation method, during the reasoning process, the thought process content is filled in. <think>Internal identifier; input field for calling multiple tool types <tool>The identifier contains the name of the inference tool and related parameters; the return results of calling multiple types of tools are also included.<tool_result> The identifier serves as contextual information for subsequent inference steps; it continues until the maximum number of times the inference tool is invoked or the large model automatically terminates the inference process, at which point the inference results are filled in. <answer>The inside of the label serves as the final answer.
[0018] As an alternative implementation, the entropy value of each token fragment for:
[0019] ;
[0020] in, For dictionary size, For position Probability of obtaining a token; For position The probability of the j-th token in the corresponding dictionary; Indicated by For large models with learnable parameters.
[0021] As an alternative implementation method, the entropy change of the current inference node for:
[0022] ;
[0023] in, This is the entropy value of the current inference node; This is the entropy value of the previous inference node; The entropy value of the initial inference node; This is the size of the dictionary.
[0024] As an alternative implementation, the process of comparing the entropy change with a set entropy change threshold includes:
[0025] If the entropy change of the current inference node is less than the entropy change threshold, the multi-type tool invocation strategy of the original inference path is maintained.
[0026] If the entropy change of the current inference node is greater than or equal to the entropy change threshold, the current inference node is determined to be a high token entropy node.
[0027] As an alternative implementation method, the process of training a large model based on a reinforcement learning algorithm is modeled as follows:
[0028] ;
[0029] in, Let be the mathematical expectation function. Number of groups To generate results for the query, To limit the range of variation, The parameters are within the clip range; This represents a relative advantage within the group. Represents KL regularization terms; Importance sampling ratio; These are the learnable parameters of the model.
[0030] Secondly, the present invention provides a large-scale model multi-type tool collaborative reasoning system, comprising:
[0031] The entropy calculation module is configured to calculate the preceding inference nodes generated by the large model for each inference tool call, based on the original inference path generated by the large model for the problem. The average entropy of each token fragment is used as the entropy value of the current inference node.
[0032] The entropy change judgment module is configured to determine the entropy change of the current inference node based on the entropy values of the current inference node, the previous inference node, and the initial inference node, and to determine the high token entropy node by comparing the entropy change with the set entropy change threshold.
[0033] The adaptive exploration module is configured to randomly call a different inference tool than the original inference path when the next inference tool is called at a high token entropy node, and continue to execute the inference process to generate a new inference path. The entropy value of each inference node is calculated for the new inference path until there are no high token entropy nodes.
[0034] The training module is configured to use the original inference path and all generated new inference paths as the training set to train the large model based on a reinforcement learning algorithm.
[0035] The reasoning module is configured to use a trained large model to generate the final answer to the problem being addressed.
[0036] Thirdly, the present invention provides an electronic device including a memory and a processor, and computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the method described in the first aspect.
[0037] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0038] To address the issues of single inference paths and rigid decision-making, particularly the tendency to choose suboptimal paths at decision nodes, when large language models invoke tools to perform complex reasoning tasks, this invention designs an adaptive exploration and reinforcement learning strategy based on token entropy. This strategy is applied to collaborative reasoning tasks using multiple tools within a large model. By monitoring token entropy changes in real time during the reasoning process, the uncertainty of decision nodes is quantified. An exploration mechanism is triggered at nodes with high token entropy, generating diverse inference paths. Simultaneously, based on these diverse inference paths, the GRPO algorithm is used to train and optimize the collaborative reasoning capabilities of the large model using multiple tools. This intelligently enhances the inference tool selection strategy of the large model under uncertain conditions, improving the accuracy of the final answer reasoning.
[0039] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description
[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0041] Figure 1 This is a flowchart of the collaborative reasoning method for large-scale models and multiple types of tools provided in Embodiment 1 of the present invention;
[0042] Figure 2 This is an example diagram of collaborative reasoning using a large model and multiple types of tools provided in Embodiment 1 of the present invention. Detailed Implementation
[0043] The present invention will be further described below with reference to the accompanying drawings and embodiments.
[0044] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.
[0045] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments of the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. Furthermore, it should be understood that the terms "comprising" and "including," and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0046] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.
[0047] Example 1
[0048] This embodiment provides a collaborative reasoning method for large-scale models using multiple tool types, such as... Figure 1 As shown, it includes:
[0049] For the original inference path generated by the large model for the problem, calculate the preceding inference nodes generated by the large model after each inference tool call. The average entropy of each token fragment is used as the entropy value of the current inference node.
[0050] Based on the entropy values of the current inference node, the previous inference node, and the initial inference node, determine the entropy change of the current inference node. Based on the comparison of the entropy change with the set entropy change threshold, determine the high token entropy node.
[0051] When the next inference tool is invoked at a high token entropy node, a different inference tool than the original inference path is randomly invoked and the inference process continues to be executed, generating a new inference path. The entropy value of each inference node is calculated for the new inference path until there are no high token entropy nodes.
[0052] The original inference path and all the generated new inference paths are used as the training set to train the large model based on the reinforcement learning algorithm.
[0053] The final answer to the problem is generated using a large, trained model.
[0054] This embodiment proposes a large-scale collaborative reasoning method based on token entropy, aiming to enhance the ability of large language models to solve complex problems. This method integrates and schedules three external reasoning tools: local corpus retrieval, online web retrieval, and code execution. It guides the large model through iterative reasoning of "think-call-feedback" using structured thought chain prompts. Its core innovation lies in introducing an adaptive exploration and reinforcement learning strategy based on token entropy. By monitoring token entropy changes in real time during the reasoning process, the uncertainty of decision nodes is quantified. When the uncertainty exceeds a set threshold, an exploration mechanism is triggered to generate new reasoning paths. The GRPO algorithm is then used to train and optimize multiple paths, significantly enhancing the large model's selection of reasoning tools under uncertain conditions, thereby greatly improving the accuracy and robustness of the final answer.
[0055] The method of this embodiment will be described in detail below.
[0056] 1. Setting the type of reasoning tool.
[0057] In this embodiment, the reasoning tools used include local corpus retrieval tools, web online retrieval tools, and code tools;
[0058] Among them, the local corpus retrieval tool: targeting the local corpus (containing private knowledge and related business data), it performs Retrieval-augmented Generation (RAG) technology to retrieve relevant information.
[0059] Web online search tools: For open and real-time questions, they call the web search API (Application Programming Interface) to extract relevant content and summarize key information to parse web search results in response to queries.
[0060] Code tools: For complex logical operation problems, LLM generates Python code snippets and executes the code in a sandbox environment, returning execution results or error messages based on the correctness of the code.
[0061] 2. Large-scale model reasoning process.
[0062] The reasoning format of LLM is controlled by designing cue words. During the LLM reasoning process, the thought process is required to be filled in. <think> …< / think> The identifier is inside; operations that invoke multiple tool types are required to be filled in. <tool> …< / tool> The identifier contains the name of the inference tool (e.g., local corpus retrieval tool, web online retrieval tool, and code tool) and related parameters (e.g., local corpus retrieval query, web online retrieval query, and code function description); the return results of calling multiple types of tools are required to be filled in.<tool_result> …< / tool_result> The identifier serves as contextual information for subsequent steps, allowing the LLM inference process to continue. This process iterates until the maximum number of inference tool calls is reached, or the LLM automatically terminates the inference, at which point the inference result is entered. <answer> …< / answer> Inside the identifier, as the final answer.
[0063] 3. Definition of collaborative reasoning process using multiple tools.
[0064] Given a natural language query problem and external reasoning toolset During the LLM inference path generation process, the model autonomously invokes inference tools and links real-time feedback to the thought chain to improve the accuracy of the model inference process.
[0065] The process is modeled as follows:
[0066] ;
[0067] Among them, factors This represents the reasoning process for calling multiple types of tools. Represents the chain of thought The number of tokens in the data. For position token, Indicates position All previous tokens; Indicates the introduction of a set of reasoning tools Model instructions; Indicates position Feedback from all previous historical inference tools; factors This indicates the process of generating the answer. Indicate the answer The number of tokens, For position The model generation results, For position Previous model historical generation results.
[0068] 4. Reinforcement learning method based on token entropy.
[0069] This study utilizes token entropy to investigate the uncertainty in LLM inference. Low-entropy tokens reflect deterministic token positions, while high-entropy tokens reflect high uncertainty (such as ambiguity or additional exploration space in multi-step inference), typically corresponding to decision nodes in the model inference. Previous research has shown that maintaining high uncertainty and exploration at key token nodes in LLM inference is beneficial for improving inference quality.
[0070] First, calculate the entropy of a single token fragment. :
[0071] ;
[0072] in, Indicates dictionary size. Indicates position Probability of obtaining a token; For position The probability of the j-th token in the corresponding dictionary; Indicated by It is an LLM model with learnable parameters.
[0073] Secondly, continuously monitor the token entropy changes of the inference nodes generated by the large model after each inference tool call during the LLM inference process, adaptively select nodes with high token entropy, and infer the uncertainty of the model's prediction at the current stage.
[0074] Specifically:
[0075] For each inference path generated by the large model for the question, for the first... During the next inference tool invocation phase, the preceding inference nodes generated by the large model are calculated. The average entropy of each token fragment This is used as the entropy value of the current inference node.
[0076] Also consider the entropy value of the current inference node. The entropy value relative to the previous inference node and the entropy value of the initial inference node Regarding the gain situation, the entropy change is defined. :
[0077] ;
[0078] in, This indicates the initial response of the large model. The average entropy of each token fragment.
[0079] Then, to encourage LLM to adaptively explore the path selection of inference tools during the high token entropy stage, this embodiment designs an entropy change threshold. We monitor the token entropy changes after each inference tool call and control the distribution probability of the GRPO algorithm during inference path sampling.
[0080] Specifically:
[0081] when Less than the entropy change threshold At the same time, maintain the original inference path for multi-type tool invocation strategy.
[0082] when Greater than or equal to the entropy change threshold When this happens, the current inference node is determined to be a high token entropy node;
[0083] When the next inference tool is invoked at a high token entropy node, an inference tool different from the original inference path is randomly invoked to perform exploration. The inference tool invocation prompt is inserted into the original inference path, and the inference process continues to be executed, thereby generating a new inference path. The entropy value of each inference node is calculated on the new inference path until there are no high token entropy nodes.
[0084] For example, in the original inference path, if the next call at the current high token entropy node is a local corpus retrieval tool, then a web-based online retrieval tool or a code tool can be randomly called to enhance the LLM's exploration capabilities under uncertain conditions.
[0085] Therefore, it is generated autonomously by LLM. For each original reasoning path, new reasoning paths are generated based on the entropy change. The final result is a combination of the original reasoning paths and all the generated new reasoning paths. The inference paths are used as the training set to achieve GRPO reinforcement learning training.
[0086] The GRPO reinforcement learning algorithm is designed as follows:
[0087] ;
[0088] in, Let be the mathematical expectation function. Indicates the number of groups. This indicates the query results. Indicates a limit on the range of variation. The parameters are within the clip range; Indicates relative advantage within the group. The reward function includes two parts: reward for answer accuracy and reward for formatting. The average reward within the group. Within-group reward standard deviation; This represents the KL regularization term, used to limit the model differences from being too large in order to prevent training breakdown. Indicates the importance sampling ratio, Indicates the reference strategy, Indicate the target strategy; For the model's learnable parameters, Indicates position The query generates results; Indicates position The previous query generated the results.
[0089] like Figure 2 The above scheme is shown as an example. Figure 2 The above scheme will be explained in detail using an example.
[0090] Specifically:
[0091] Problem q is generated by LLM into two original inference paths.
[0092] (1) For the first original reasoning path, fill in the content of the initial thinking process in the first... <think>The first Think inference node is generated after the identifier is identified. Then, after the first call to the local corpus retrieval tool, the second Think inference node is generated. This process continues. After the second call to the local corpus retrieval tool, the third Think inference node is generated. After the third call to the online web retrieval tool, the fourth Think inference node is generated.
[0093] It should be noted that during the generation of the original reasoning path, the LLM autonomously invokes the reasoning tool and links the real-time feedback to the thought chain.
[0094] For each inference node, calculate the previous... The entropy value of each token fragment in the token fragment Then calculate the previous The average entropy of each token fragment is used as the entropy value of the current inference node. The mean entropy of the first Think inference node is... .
[0095] Based on the entropy value of the current inference node The entropy value of the previous inference node and the entropy value of the initial inference node Determine the entropy change of the current inference node. According to the change in entropy Comparison with the set entropy change threshold Identify nodes with high token entropy;
[0096] Depend on Figure 2 It can be seen that the second Think inference node generated after the first call to the local corpus retrieval tool is a high token entropy node. Therefore, when the inference tool is called again at this point, an inference tool different from the original inference path will be randomly selected, i.e., randomly chosen from the web online retrieval tool and the code tool. Figure 2 The program displays the selection of code tools, then continues the inference process, generating a new Think inference node;
[0097] After this new Think inference node, the LLM autonomously invokes the inference tool to execute the inference process, ultimately generating the first new inference path; and continues to calculate the entropy value of each inference node along this new inference path until there are no nodes with high token entropy. This is understandable. Figure 2 This process was not shown in the document.
[0098] (2) Similarly, for the second original reasoning path, the initial thought process content is filled into the first... <think>The first Think inference node is generated after the identifier is identified. Then, after the first call to the local corpus retrieval tool, the second Think inference node is generated, and so on. After the second call to the online web retrieval tool, the third Think inference node is generated. After the third call to the code tool, the fourth Think inference node is generated.
[0099] For each inference node, calculate the previous... The entropy value of each token fragment in the token fragment Then calculate the previous The average entropy of each token fragment is used as the entropy value of the current inference node. The mean entropy of the first Think inference node is... .
[0100] Based on the entropy value of the current inference node The entropy value of the previous inference node and the entropy value of the initial inference node Determine the entropy change of the current inference node. According to the change in entropy Comparison with the set entropy change threshold Identify nodes with high token entropy;
[0101] Depend on Figure 2 It can be seen that the third Think inference node generated after the second call to the online web search tool is a high token entropy node. Therefore, when the inference tool is called again at this point, an inference tool different from the original inference path will be randomly selected, that is, randomly chosen from the local corpus search tool and the code tool. Figure 2 The text then displays the selection of a local corpus retrieval tool, followed by the continuation of the inference process, generating a new Think inference node.
[0102] Following this new Think inference node, the LLM autonomously invokes the inference tool to execute the inference process, ultimately generating a second new inference path. The entropy value of each inference node along this new path is then calculated until no node with high token entropy remains. This is understandable. Figure 2 This process was not shown in the document.
[0103] Therefore, the training set consists of two original reasoning paths and two new reasoning paths. The large model is trained using a reinforcement learning algorithm, and the final answer is generated using the trained large model.
[0104] It is understood that the implementation of the solution in this embodiment is not limited. Figure 2 The process shown is as follows: Figure 2 This is just an example for illustration.
[0105] In this embodiment, based on the above implementation process, the implementation of the large-model multi-type tool collaborative reasoning method specifically includes the following steps:
[0106] Step 1: Initialize an external inference toolset consisting of three types: local corpus retrieval, web online retrieval, and code execution, to prepare for LLM to generate highly accurate inference paths.
[0107] Step 2: Generate structured thinking. Follow the preset prompt format, in <think>The identifier generates a structured thinking process. This step clarifies the query intent and plans the initial reasoning path and selection of reasoning tools for problem-solving.
[0108] Step 3: Invoking reasoning tools. Based on the thinking results, the large model is... <tool>Select and invoke the most suitable inference tool within the identifier. The LLM encapsulates the inference tool name and required parameters, such as the problem or code function description to be retrieved.
[0109] Step 4: Integrate the feedback from the inference tool. The large model receives the results returned by the inference tool after execution and populates them into...<tool_result> Within the identifier. This feedback serves as crucial contextual information, guiding and refining subsequent reasoning steps.
[0110] Step 5: LLM Iterative Reasoning and Answer Generation. The large model iteratively executes a "think-call-feedback" process until the termination condition is met. Finally, in <answer>The identifier generates and outputs the final answer that integrates all the information.
[0111] Step 6: Quantify the uncertainty of inference. Monitor the uncertainty of model predictions in real time by calculating the entropy value of key token fragments. High entropy values typically correspond to critical nodes requiring decision-making or exploration.
[0112] Step 7: Trigger Adaptive Exploration. For the original inference path generated by LLM, calculate the entropy change after each inference tool call and compare it with a preset threshold. This mechanism is used to determine whether exploratory inference needs to be initiated at a stage with high uncertainty.
[0113] Step 8: Perform exploratory reasoning. When the entropy gain exceeds a threshold, the model is forced to randomly select a new tool to perform exploration. This behavior aims to generate a new inference path to enhance the model's exploratory capabilities under uncertainty.
[0114] Step 9: Reinforcement Learning Strategy Optimization. The original inference paths generated by LLM and the newly generated inference paths are used together as training samples. The GRPO algorithm is applied for reinforcement learning to optimize the inference tool selection strategy of the large model at different decision points, thereby improving the overall inference quality.
[0115] To address the issues of single inference paths and rigid decision-making, particularly the tendency to choose suboptimal paths at decision nodes, when large language models invoke tools to perform complex reasoning tasks, this embodiment designs an adaptive exploration and reinforcement learning strategy based on token entropy. This strategy is applied to collaborative reasoning tasks using multiple tools based on large models. By monitoring token entropy changes in real time during the reasoning process, the uncertainty of decision nodes is quantified, and an exploration mechanism is triggered at nodes with high token entropy to generate diverse inference paths. Simultaneously, based on these diverse inference paths, the GRPO algorithm is used to train and optimize the collaborative reasoning capabilities of the large model using multiple tools. This achieves an intelligent improvement in the large model's inference tool selection strategy under uncertain conditions, thereby enhancing the accuracy of the final answer reasoning.
[0116] It should be noted that all data acquisition is conducted in accordance with laws and regulations and with user consent, and the data is used legally.
[0117] Example 2
[0118] This embodiment provides a large-scale model multi-type tool collaborative reasoning system, including:
[0119] The entropy calculation module is configured to calculate the preceding inference nodes generated by the large model for each inference tool call, based on the original inference path generated by the large model for the problem. The average entropy of each token fragment is used as the entropy value of the current inference node.
[0120] The entropy change judgment module is configured to determine the entropy change of the current inference node based on the entropy values of the current inference node, the previous inference node, and the initial inference node, and to determine the high token entropy node by comparing the entropy change with the set entropy change threshold.
[0121] The adaptive exploration module is configured to randomly call a different inference tool than the original inference path when the next inference tool is called at a high token entropy node, and continue to execute the inference process to generate a new inference path. The entropy value of each inference node is calculated for the new inference path until there are no high token entropy nodes.
[0122] The training module is configured to use the original inference path and all generated new inference paths as the training set to train the large model based on a reinforcement learning algorithm.
[0123] The reasoning module is configured to use a trained large model to generate the final answer to the problem being addressed.
[0124] It should be noted that the above modules correspond to the steps described in Embodiment 1, and the examples and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer-executable instructions.
[0125] In further embodiments, the following is also provided:
[0126] An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the method described in Embodiment 1. For brevity, further details are omitted here.
[0127] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.
[0128] Memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of memory may also include non-volatile random access memory. For example, memory may also store information about the device type.
[0129] A computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the method described in Embodiment 1.
[0130] The method in Example 1 can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.
[0131] A computer program product includes a computer program that, when executed by a processor, implements the method described in Embodiment 1.
[0132] The present invention also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions included in program modules, which execute in a device on a target real or virtual processor to perform the processes / methods described above. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc., that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules can be combined or divided among program modules as needed. The machine-executable instructions for the program modules can execute within a local or distributed device. In a distributed device, the program modules can reside in both local and remote storage media.
[0133] The computer program code used to implement the methods of the present invention may be written in one or more programming languages. This computer program code may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the computer or other programmable data processing device, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a computer, partially on a computer, as a stand-alone software package, partially on a computer and partially on a remote computer, or entirely on a remote computer or server.
[0134] In the context of this invention, computer program code or related data may be carried by any suitable carrier to enable a device, apparatus, or processor to perform the various processes and operations described above. Examples of carriers include signals, computer-readable media, and the like. Examples of signals may include electrical, optical, radio, sound, or other forms of propagation signals, such as carrier waves, infrared signals, etc.
[0135] Those skilled in the art will recognize that the units and algorithm steps described in connection with the various examples of this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.
[0136] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.< / answer> < / tool> < / think> < / think> < / think> < / answer> < / tool> < / think>
Claims
1. A collaborative reasoning method for large-scale models using multiple tool types, characterized in that, include: For the original inference path generated by the large model for natural language queries, calculate the preceding inference nodes generated by the large model after each inference tool call. The average entropy of each token fragment is used as the entropy value of the current inference node. Based on the entropy values of the current inference node, the previous inference node, and the initial inference node, determine the entropy change of the current inference node. Based on the comparison of the entropy change with the set entropy change threshold, determine the high token entropy node. When the next inference tool is invoked at a high token entropy node, a different inference tool than the original inference path is randomly invoked and the inference process continues to be executed, generating a new inference path. The entropy value of each inference node is calculated for the new inference path until there are no high token entropy nodes. The original inference path and all the generated new inference paths are used as the training set to train the large model based on the reinforcement learning algorithm. The final answer to the problem is generated using a large, trained model. For each inference path generated by the large model for the question, for the first... During the next inference tool invocation phase, the preceding inference nodes generated by the large model are calculated. The average entropy of each token fragment This is used as the entropy value of the current inference node; Also consider the entropy value of the current inference node. The entropy value relative to the previous inference node and the entropy value of the initial inference node The gain situation is used to define the entropy change of the current inference node. for: ； in, This is the entropy value of the current inference node; This is the entropy value of the previous inference node; The entropy value of the initial inference node; This is the size of the dictionary. The process of comparing the entropy change with a set entropy change threshold includes: If the entropy change of the current inference node is less than the entropy change threshold, the multi-type tool invocation strategy of the original inference path is maintained. If the entropy change of the current inference node is greater than or equal to the entropy change threshold, the current inference node is determined to be a high token entropy node. The process of training a large model using reinforcement learning algorithms is modeled as follows: ； in, Let be the mathematical expectation function. Number of groups To generate results for the query, To limit the range of variation, The parameters are within the clip range; This represents a relative advantage within the group. Represents KL regularization terms; Importance sampling ratio; These are the learnable parameters of the model; On the issue The process of collaborative reasoning based on multiple types of tools using a large model is modeled as follows: ； Among them, factors This represents the reasoning process for calling multiple types of tools. Represents the chain of thought The number of tokens in the data. For position token, Indicates position All previous tokens; Indicates the introduction of a set of reasoning tools Model instructions; Indicates position Feedback from all previous historical inference tools; factors This indicates the process of generating the answer. Indicate the answer The number of tokens, For position The model generation results, For position Previous model historical generation results.
2. The collaborative reasoning method for large-scale models with multiple tool types as described in claim 1, characterized in that, Reasoning Toolset This includes local corpus search tools, online web search tools, and code tools.
3. The collaborative reasoning method for large-scale models with multiple tool types as described in claim 1, characterized in that, During the reasoning process, the content of the thought process is filled in. <think>Internal identifier; input field for calling multiple tool types <tool>The identifier contains the inference tool name and parameters; the return results from calling multiple tool types are also included.<tool_result> The identifier serves as contextual information for subsequent inference steps; it continues until the maximum number of times the inference tool is invoked or the large model automatically terminates the inference process, at which point the inference results are filled in. <answer> The inside of the label serves as the final answer.< / answer> < / tool> < / think> 4. The collaborative reasoning method for large-scale models with multiple tool types as described in claim 1, characterized in that, The entropy value of each token fragment for: ； in, For dictionary size, For position Probability of obtaining a token; For position The probability of the j-th token in the corresponding dictionary; Indicated by For large models with learnable parameters.
5. A large-scale model multi-type tool collaborative reasoning system, characterized in that, include: The entropy calculation module is configured to calculate the original inference path generated by the large model for natural language queries, after each inference tool call, the previous inference node generated by the large model. The average entropy of each token fragment is used as the entropy value of the current inference node. The entropy change judgment module is configured to determine the entropy change of the current inference node based on the entropy values of the current inference node, the previous inference node, and the initial inference node, and to determine the high token entropy node by comparing the entropy change with the set entropy change threshold. The adaptive exploration module is configured to randomly call a different inference tool than the original inference path when the next inference tool is called at a high token entropy node, and continue to execute the inference process to generate a new inference path. The entropy value of each inference node is calculated for the new inference path until there are no high token entropy nodes. The training module is configured to use the original inference path and all generated new inference paths as the training set to train the large model based on a reinforcement learning algorithm. The reasoning module is configured to use a trained large model to generate the final answer to the problem being addressed. For each inference path generated by the large model for the question, for the first... During the next inference tool invocation phase, the preceding inference nodes generated by the large model are calculated. The average entropy of each token fragment This is used as the entropy value of the current inference node; Also consider the entropy value of the current inference node. The entropy value relative to the previous inference node and the entropy value of the initial inference node The gain situation is used to define the entropy change of the current inference node. for: ； in, This is the entropy value of the current inference node; This is the entropy value of the previous inference node; The entropy value of the initial inference node; This is the size of the dictionary. The process of comparing the entropy change with a set entropy change threshold includes: If the entropy change of the current inference node is less than the entropy change threshold, the multi-type tool invocation strategy of the original inference path is maintained. If the entropy change of the current inference node is greater than or equal to the entropy change threshold, the current inference node is determined to be a high token entropy node. The process of training a large model using reinforcement learning algorithms is modeled as follows: ； in, Let be the mathematical expectation function. Number of groups To generate results for the query, To limit the range of variation, The parameters are within the clip range; This represents a relative advantage within the group. Represents KL regularization terms; Importance sampling ratio; These are the learnable parameters of the model; On the issue The process of collaborative reasoning based on multiple types of tools using a large model is modeled as follows: ； Among them, factors This represents the reasoning process for calling multiple types of tools. Represents the chain of thought The number of tokens in the data. For position token, Indicates position All previous tokens; Indicates the introduction of a set of reasoning tools Model instructions; Indicates position Feedback from all previous historical inference tools; factors This indicates the process of generating the answer. Indicate the answer The number of tokens, For position The model generation results, For position Previous model historical generation results.
6. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the method according to any one of claims 1-4.