Tool integrated reasoning reinforcement learning system and method based on multi-dimensional rewards

Abstract

The invention discloses a tool integrated reasoning reinforcement learning system and method based on multi-dimensional rewards, and belongs to the field of natural language processing and reinforcement learning. The method comprises the steps that a tool integrated reasoning cue word design module provides predefined cue words and tag sequences to train a large language model to autonomously generate a reasoning track, and a tool is selected to be called or a final reply is output; the multi-dimensional reward calculation module constructs a multi-dimensional reward signal according to tool calling or a final reply result, wherein the multi-dimensional reward signal comprises a format reward and a multi-dimensional correctness reward; the reward fusion module integrates the reward signals of different dimensions to generate a comprehensive reward tensor; and the model training module is used for training a large language model according to the comprehensive reward tensor and an inter-group relative strategy optimization algorithm to realize autonomous reasoning. According to the method, the accuracy and training efficiency of a large language model in a tool integrated reasoning task can be remarkably improved, and in a complex tool calling scene, the model is guided to accurately generate a tool name and parameters through a multi-dimensional reward mechanism.

Description

Technical Field

[0001] This invention belongs to the fields of natural language processing and reinforcement learning, and particularly relates to a tool integration reasoning reinforcement learning system and method based on multi-dimensional rewards, which is especially suitable for the reinforcement learning training process of large language models in tool integration reasoning tasks. Background Technology

[0002] With the widespread application of large language models in tasks such as intelligent question answering, code generation, and tool invocation, how to effectively train models to accurately invoke external tools has become a research hotspot. Traditional tool invocation training methods mainly adopt a training paradigm that combines supervised fine-tuning (SFT) and reinforcement learning (RL), in which the reinforcement learning stage guides the model to learn the correct tool invocation behavior through a reward function.

[0003] However, existing reward function designs have the following limitations in providing effective learning signals: First, traditional reinforcement learning reward signals are typically calculated based on the final output of the entire inference sequence (Sequence-level). This coarse-grained reward allocation method distributes the calculated reward value evenly across all tokens in the sequence. In actual tool call sequence generation, a complete sequence may contain both correctly generated tokens (e.g., correct function names, parameter keys) and incorrectly generated tokens (e.g., incorrect parameter values, redundant characters). Sequence-level methods cannot distinguish the correctness of these tokens, especially failing to finely evaluate the correctness of key components of tool calls (i.e., function names and parameters). Assigning equivalent learning signals to correct function-level tokens and incorrect parameter-level tokens severely dilutes the guiding effectiveness of rewards, making it difficult for the model to efficiently learn the fine-grained generation rules of key details such as tool names, parameter formats, and parameter values, thus limiting the model's learning efficiency. Furthermore, the model also generates a final response, and due to the complexity and diversity of natural language, it is difficult to accurately allocate rewards to the response content using simple rule-based methods. Currently, the evaluation of the final response often employs a model with a large number of parameters, requiring detailed evaluation of multiple dimensions of the response to accurately determine whether the response content correctly solves the user's problem. However, this approach has significant limitations: on the one hand, using a single large model for multi-dimensional evaluation makes it difficult to simultaneously take into account the characteristics of different evaluation dimensions, resulting in low evaluation efficiency; on the other hand, the deployment and inference costs of large models are high, limiting the practical application of the evaluation method.

[0004] To address these issues, existing research primarily employs the following methods: First, improving the design of reward functions by incorporating multi-dimensional rewards and evaluation metrics to achieve finer-grained rewards; second, optimizing advantage calculation methods, such as using DAPO (Dynamic Sampling Policy Optimization) or GSPO (Group Sequence Policy Optimization) algorithms; third, training models using high-quality datasets, building and cleaning datasets based on expert knowledge to obtain higher-quality training data; and fourth, evaluating the model's response from multiple dimensions simultaneously using a model with a large number of parameters. However, these methods still cannot fundamentally solve the problem of multi-dimensional reward allocation, especially in tool invocation scenarios. Accurately assigning rewards to each token for the tool name and parameters, while simultaneously efficiently and accurately evaluating the model's final response, remains a challenge. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the present invention aims to provide a tool-integrated reasoning reinforcement learning system and method based on multi-dimensional rewards.

[0006] The objective of this invention is achieved through the following technical solution: a tool-integrated inference reinforcement learning system based on multi-dimensional rewards, comprising:

[0007] The tool integrates a reasoning prompt word design module, which guides the large language model to autonomously generate a complete trajectory including the reasoning process, tool call actions, and final response through predefined system prompt words and tag sequences. The large language model dynamically selects and calls external tools to obtain information based on the user query, or directly generates the final response.

[0008] A multi-dimensional reward calculation module is used to construct multi-dimensional reward signals based on the tool call results or final responses of a large language model. The multi-dimensional reward signals include format rewards and multi-dimensional correctness rewards. The format rewards are used to determine whether the format of the final response conforms to a predefined format. The multi-dimensional correctness rewards are used to evaluate the tool call results and to perform multi-dimensional semantic correctness evaluation on the final response.

[0009] The reward fusion module is used to linearly combine format rewards and multi-dimensional correctness rewards to generate a final comprehensive reward tensor for reinforcement learning training.

[0010] The model training module is used to calculate the token-level advantage value and update the gradient based on the comprehensive reward tensor through the inter-group relative policy optimization algorithm, so as to drive the optimized training of the large language model to achieve tool integration reasoning capabilities.

[0011] Furthermore, the tool integration reasoning prompt word design module includes: a predefined tag sequence comprising three types of tags: a tag for guiding the model to autonomously generate reasoning trajectories, a tag for calling callable tools, and a tag for natural language responses to user output; the results of the large language model calling callable tools and the generated final responses are appended to the dialogue history to constitute the interaction trajectory of the large language model.

[0012] Furthermore, the format reward includes: evaluating the boundary integrity and logical order normalization of the output results of the large language model, and giving different boundary integrity reward scores according to the boundary integrity; at the same time, constraining the logical timing of the complete trajectory generated by the large language model, and giving different logical order normalization reward scores according to the logical order normalization, and the boundary integrity reward score and the logical order normalization reward score are added together to form the complete format reward.

[0013] Furthermore, the multi-dimensional correctness reward includes function-level, parameter-level, and response-level rewards. If the large language model chooses to directly generate the final response, the final response generated by the large language model is evaluated at the response-level (Sequence-level) using an evaluation model with a smaller number of parameters (32B). This evaluation includes scores for three dimensions: timeliness, accuracy, and relevance, with each score ranging from 0 to 2. The scores are then normalized and averaged, outputting a Sequence-level reward score of 0 or 1. This Sequence-level reward score is then evenly distributed among each token in the response sequence. If the large language model chooses to invoke a tool, the tool invocation results are rewarded at both the function-level and parameter-level dimensions. Based on the matching of the function name, parameter key, and parameter value, a score of 0 or 1 is output, respectively.

[0014] Furthermore, the reward fusion module includes: adopting differentiated fusion strategies based on different combinations of model output content; when the model output contains a tool call key token but not a final response key token, its comprehensive reward is a linear combination of format reward and tool call key token reward, wherein the tool call key token reward includes multi-dimensional rewards at the function level and parameter level; when the model output contains a final response key token but not a tool call key token, its comprehensive reward is a linear combination of format reward and model response reward, wherein the model response reward is a response-level sequence-level reward.

[0015] Furthermore, the model training module includes: based on the comprehensive reward tensor, using an inter-group relative policy optimization algorithm to calculate token-level advantage and update gradients. For each user query, multiple responses are generated via a rollout process, forming a group. The average and variance of the rewards for all trajectories within the group are calculated. By comparing the reward values ​​of different trajectories within the same group, the inter-group relative advantage is calculated to optimize the policy, thereby driving the optimized training of the large-scale language model to achieve tool integration reasoning capabilities.

[0016] This invention also provides a tool ensemble inference reinforcement learning method based on multi-dimensional rewards, comprising the following steps:

[0017] The large language model is trained by predefined system prompt words and tag sequences to autonomously generate a complete trajectory including reasoning process, tool calling action and final response. The large language model dynamically selects and calls external tools to obtain information based on user query, or directly generates the final response and appends the tool calling results to the dialogue history.

[0018] A multi-dimensional reward signal is constructed based on the tool call results or final response of the large language model; wherein, the multi-dimensional reward signal includes a format reward and a multi-dimensional correctness reward, the format reward is used to determine whether the format of the final response conforms to a predefined format, and the multi-dimensional correctness reward is used to evaluate the tool call results and to evaluate the multi-dimensional semantic correctness of the final response;

[0019] The format reward and the multi-dimensional correctness reward are linearly combined to generate the final comprehensive reward tensor for reinforcement learning training;

[0020] Based on the comprehensive reward tensor, the token-level advantage value calculation and gradient update are realized through the inter-group relative policy optimization algorithm, so as to drive the large language model to achieve optimized training of tool integration reasoning ability.

[0021] Furthermore, the format reward includes: evaluating the boundary integrity of the output results of the large language model, and assigning different boundary integrity reward scores based on the boundary integrity. If all marker pairs are correctly matched, a boundary integrity reward of 0.5 points is assigned; if there are start or end markers but they are not completely matched, a partial integrity reward of 0.1 points is assigned; if the markers are completely incorrect or missing, the reward is 0 points. At the same time, the logical sequence of the complete trajectory generated by the large language model is constrained, and different logical sequence standardization reward scores are assigned based on the logical sequence standardization. It is required that the thinking content must be in advance, and the tool call and the final response are mutually exclusive and cannot appear at the same time. If the logical sequence standardization is met, a logical correctness reward of 0.5 points is assigned; if it is not met, the reward is 0 points. The boundary integrity reward score and the logical sequence standardization reward score are added together to form the complete format reward.

[0022] Furthermore, the multi-dimensional correctness reward includes function-level rewards, parameter-level rewards, and response-level rewards. If the large language model chooses to directly generate the final response, the final response generated by the large language model is evaluated at the response-level sequence level, including scores in three dimensions: timeliness, accuracy, and relevance. The scores are then normalized and averaged, with the output being a score of 0 or 1. If the large language model chooses to call a tool, the tool call results are rewarded at both the function-level and parameter-level dimensions, with the output being a score of 0 or 1 based on the matching of the function name, parameter key, and parameter value.

[0023] Furthermore, the linear combination of format rewards and multi-dimensional correctness rewards includes: adopting differentiated fusion strategies based on different combinations of model output content; when the model output contains a tool call key token but not a final response key token, its comprehensive reward is a linear combination of format rewards and tool call key token rewards, wherein the tool call key token rewards include multi-dimensional rewards at the function level and parameter level; when the model output contains a final response key token but not a tool call key token, its comprehensive reward is a linear combination of format rewards and model response rewards, wherein the model response rewards are response-level sequence-level rewards; the dimensions of the final generated comprehensive reward are consistent with the length of the token sequence in the model output, with each position corresponding to a reward value for a token, thus achieving an organic combination of multi-dimensional token-level rewards (function-level and parameter-level) and sequence-level rewards (format rewards and response-level).

[0024] The beneficial effects of this invention are as follows:

[0025] (1) This invention proposes a multi-dimensional reward allocation mechanism, which effectively solves the bias problem caused by the average allocation of each token in the sequence in traditional methods. This mechanism introduces multi-dimensional evaluation at the function level, parameter level, and response level, and independently calculates rewards for key tokens such as tool name, parameter key, and parameter value. At the same time, it evaluates the semantic correctness of the final response content in multiple dimensions, which significantly improves the signal strength of key tokens and the comprehensiveness of evaluation. Experiments show that this multi-dimensional reward design improves the convergence speed of the model on tool invocation tasks by about 20%, and significantly improves the generalization ability of complex parameter structures, providing more accurate gradient signals for reinforcement learning training.

[0026] (2) This invention constructs a multi-dimensional reward fusion framework, combining multi-dimensional token-level rewards at the function and parameter levels with sequence-level rewards at the response level, while also integrating structured rewards such as format rewards. At the function and parameter levels, the system provides multi-dimensional local optimization signals to the model by accurately matching key tokens such as tool names and parameter key values; at the response level, it conducts a comprehensive semantic correctness assessment of the response content through multi-dimensional evaluations such as timeliness, accuracy, and relevance; at the format level, it applies overall constraints through global indicators such as format standardization and logical coherence. This multi-dimensional reward mechanism not only ensures the accurate optimization of key tokens related to tool calls, but also ensures the comprehensive evaluation of response content and the integrity and usability of output results.

[0027] (3) This invention deeply integrates a multi-dimensional token-level reward mechanism (function-level, parameter-level, and response-level) with the GRPO algorithm, achieving a more suitable advantage function calculation for tool integration and invocation scenarios. By refining the calculation of relative advantages between groups from the sequence-level to the function-level and parameter-level, the system can provide accurate individualized reward signals for each generated token. At the same time, through multi-dimensional evaluation at the response level, the system can comprehensively evaluate the semantic correctness of the final response. This multi-dimensional optimization enables the model to not only accurately determine when to invoke the tool in the tool invocation task, but also to precisely control the generation of key elements such as tool name and parameter key values, while ensuring the accuracy, timeliness, and relevance of the final response. Experiments show that this integration scheme improves the accuracy by more than 7% on the BFCL and API-Bank benchmarks compared to TooLRL, ToRL, and SFT methods, and also improves training efficiency to a certain extent. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is an overall flowchart of the present invention;

[0030] Figure 2 This invention provides a graph showing the key token reward distribution for the tool.

[0031] Figure 3 This is a diagram showing the final response reward allocation for the model of this invention;

[0032] Figure 4 This is a flowchart of the GRPO algorithm training process of the present invention. Detailed Implementation

[0033] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of this application.

[0034] like Figure 1 As shown, this embodiment of the invention provides a tool-integrated inference reinforcement learning system based on multi-dimensional rewards, including:

[0035] The tool integrates a reasoning prompt word design module, which guides the large language model to autonomously generate a complete trajectory including the reasoning process, tool call actions, and final response through predefined system prompt words and tag sequences. The large language model dynamically selects and calls external tools to obtain information based on the user query, or directly generates the final response.

[0036] A multi-dimensional reward calculation module is used to construct multi-dimensional reward signals based on the tool call results or final responses of a large language model. The multi-dimensional reward signals include format rewards and multi-dimensional correctness rewards. The format rewards are used to determine whether the format of the final response conforms to a predefined format. The multi-dimensional correctness rewards are used to evaluate the tool call results and to perform multi-dimensional semantic correctness evaluation on the final response.

[0037] The reward fusion module is used to linearly combine format rewards and multi-dimensional correctness rewards to generate a final comprehensive reward tensor for reinforcement learning training.

[0038] The model training module is used to calculate the token-level advantage value and update the gradient based on the comprehensive reward tensor through the inter-group relative policy optimization algorithm, so as to drive the optimized training of the large language model to achieve tool integration reasoning capabilities.

[0039] As a preferred embodiment, the tool integration reasoning prompt design module specifically involves: to enable the large language model to autonomously generate reasoning trajectories and tool calls, this invention uses system prompts for guidance during the interaction phase. The "tool list" placeholder in the prompts represents the set of available tools, which includes all callable tools. This invention explicitly requires the large language model to use a special token in the instructions: <think>,<tool_call> and <response>This is used to identify the reasoning process, tool calls, and natural language responses, respectively. <observe>This indicates that the model issues a tool call command and receives a response from the environment. When the model output contains...<tool_call> During labeling, the system automatically parses tool calls into independent call instructions based on the parameters predicted by the model, and the execution results will be inserted into... <observe>The fields are then appended to the dialogue history to form the model's interaction trajectory. Similarly, if the output contains... <response>If a flag is set, the corresponding reply will be parsed and appended to the conversation history.

[0040] in,<tool_call> and <response>As they are mutually exclusive, the model either chooses to invoke the tool or respond based on its existing knowledge and outputs a reply.

[0041] For example, the prompt word is specifically:

[0042] {

[0043] You are an intelligent assistant capable of resolving user issues and providing structured dialogue responses by invoking tools. Before each response, you must carefully consider whether to invoke tools to retrieve information or to directly provide the final reply.

[0044] Available tools

[0045] You can use the following tools in your replies:

[0046] {{Tools List}}

[0047] Response process and decision-making requirements

[0048] Each response must follow these three core steps: think, decide, and answer.

[0049] Reflection: Review the dialogue history and current context to deeply analyze the user's intent and current goal.

[0050] Decision-making: Based on the results of the thinking, clearly and explicitly state whether to use a tool, which tool to use, and what parameters need to be provided.

[0051] Answer: Provide a concise, fluent, and high-quality response, ensuring the accuracy of the information and consistency with the context.

[0052] Output format

[0053] <think> {Thinking Process}< / think>

[0054] <tool_call>

[0055] {"name":"Tool Name","parameters":{"Parameter Key":"Parameter Value","... ...":"......"}}

[0056] {"name":"... ...","parameters":{"... ...":"... ...","... ...":"..."}} ...

[0058] < / tool_call>

[0059] or <response> Final reply content< / response>

[0060] Notice:

[0061] 1. Must always include< / response> < / response> < / observe> < / observe> < / response> < / think> The thought and reasoning process. It needs to include...<tool_call> or <response>One of them. The choice will be made based on the circumstances.<tool_call> (Can be called multiple times) or <response>,<tool_call> and <response>They cannot be contained simultaneously.

[0062] 2. Can be<tool_call> This field invokes multiple tools simultaneously. Each tool invocation should contain a JSON object with a "name" field and a "parameters" field, where the parameters are in dictionary format. If no parameters are needed, set "parameters" to an empty dictionary.

[0063] 3. Please refer to the conversation history to answer. The history includes the questions entered by the user and may include previous tool calls.<tool_call> And the execution result feedback of the tool call. <observe>.

[0064] }

[0065] In a preferred embodiment, the multi-dimensional reward calculation module, as a core component of the reinforcement learning training phase, is specifically used to construct a multi-dimensional reward signal, which consists of two parts, specifically including:

[0066] (1) Format rewards ( This is used to determine whether the output conforms to a predefined format. The specific reward content includes the following points:

[0067] (1.1) Boundary integrity reward:

[0068] This invention provides a precise assessment of the boundary integrity of the model's output content. The output content must be completely enclosed by corresponding start and end marker pairs, depending on its type. Specifically, this includes:

[0069] a. The reasoning must be encapsulated in... <think> {Thinking Content}< / think> Within the structure.

[0070] b. ToolCall content must be encapsulated in...<tool_call> {Tool call content}< / tool_call> Within the structure.

[0071] c. The final response content must be encapsulated in... <response> {Reply content}< / response> Within the structure.

[0072] The allocation strategy is as follows:

[0073] Complete boundary matching reward: If a specific content type in the model output is completely surrounded by the correct, paired start and end markers (i.e., structurally...) <tag> {content}< / tag> If the result is complete, a base reward value of 0.5 points will be allocated.

[0074] One-sided boundary recognition reward: If only one of the correct start or end markers appears in the model output sequence, resulting in an incomplete boundary structure (e.g., [missing information]), the boundary structure is not complete. <response> However, there is a lack of corresponding [resources / equipment].< / response> If the symbol is incorrect (or closed with an incorrect symbol), then a partial reward value of 0.1 will be allocated.

[0075] This invention employs a one-sided boundary recognition reward mechanism to alleviate the reward sparsity problem in the early stages of training. By providing positive feedback on the local correctness of boundary tokens, the model's early learning and convergence speed to the predefined output format can be effectively accelerated, ensuring that the model can quickly master the basic format generation specifications.

[0076] (1.2) Logical order normative reward:

[0077] This invention imposes strict constraints on the logical timing of structured content in the output sequence. The core content generated by the model (thinking, tool calls, and final responses) must follow a predetermined logical order.

[0078] The pre-defined structured content order requirements are as follows:

[0079] a. Content of thought ( <think> {Thinking Content}< / think> This must be a prerequisite step in the reasoning process.

[0080] b. After considering the content, you can choose to use the generation tool (<tool_call> {Tool Call}< / tool_call> ) or final reply ( <response> {reply}< / response> ).

[0081] c. The output sequence must contain at least one of the following: a tool call or a final response.

[0082] d. When the model output meets the above logical order specifications, a format reward of 0.5 points will be assigned; if it does not meet any preset logical order, the reward will be 0 points.

[0083] (1.3) Final Format Reward Determination:

[0084] Final format reward The sum of the above-mentioned boundary integrity rewards and logical order normative rewards.

[0085] It needs to be emphasized that there are format-based rewards. The reward is designed as a sequence-level reward. This reward imposes a global constraint on the structural consistency and procedural regularity of the entire sequence. If the overall format of the sequence is correct, all tokens in the sequence should have a relatively equal weight to avoid the model focusing too much on format tokens while neglecting the quality and completeness of the internally generated content (such as the actual inference text or response text). Therefore, using sequence-level reward assignment ensures a balanced optimization of the structural requirements of the entire output sequence.

[0086] (2) Multi-dimensional correctness rewards It includes a multi-dimensional reward mechanism at the function-level, parameter-level, and response-level.

[0087] The core innovation of this invention is the multi-dimensional correctness reward module, which adopts a multi-dimensional correctness reward mechanism at the function level, parameter level, and response level. The calculation employs a multi-dimensional strategy to ensure the most appropriate evaluation of different types of model outputs, as detailed below:

[0088] (2.1) Function-level and parameter-level evaluation (toolcall part): such as Figure 2 As shown, if the model output contains<tool_call> The structure employs multi-dimensional calculation logic at both the function and parameter levels to precisely allocate multi-dimensional rewards to key structural elements in tool calls, such as function names, parameter keys, and parameter values. This reward is calculated in detail across three dimensions for the tool call structure, ensuring accurate feedback on key structural elements:

[0089] a. Function-level matching: The extracted function name token is precisely matched with the token of the real label. If the match is correct, the reward for that token is 1; if the match is incorrect or there is no match, the reward is 0.

[0090] b. Parameter-level matching: The token of the extracted call parameter key is precisely matched with the token of the real tag. If the match is correct, the reward for that token is recorded as 1; if the match is incorrect or there is no match, the reward is 0.

[0091] c. Parameter-level matching: The token extracted from the call parameter value is precisely matched with the token of the real tag. If the match is correct, the reward for that token is recorded as 1; if the match is incorrect or there is no match, the reward is 0.

[0092] The reward allocation process follows these steps: First, the tool call information in the ground truth is parsed to extract the tool name, parameter key, and parameter value, which are then converted into corresponding token sequences via decoding. Next, the prediction tool call information generated by the model is parsed, similarly extracted and decoded into token sequences. Token-level pattern matching is performed on the response token sequences, and positive rewards are assigned to correctly matching function-level, parameter-level, and parameter-level tokens; zero rewards are assigned to mismatched or incorrect tokens. Finally, a multi-dimensional reward tensor is generated. The tool-related token positions contain multi-dimensional correctness rewards at both the function and parameter levels, while other token positions remain at zero, providing high-precision gradient signals for subsequent reinforcement learning training.

[0093] (2.2) Response-level assessment: such as Figure 3 As shown, if the model output contains <response>For the content, a response-level sequence-level computation logic is applied. This strategy aims to evaluate the semantic correctness of the final response. Given the inherent diversity and complexity of natural language expressions, sequence-level evaluation is more robust in this scenario. Furthermore, unlike traditional evaluation methods using a single large expert model, this invention employs a model with fewer parameters to independently evaluate the response results from three dimensions: timeliness, accuracy, and relevance. This improves evaluation efficiency and practicality while ensuring comprehensiveness. The specific evaluation process includes:

[0094] a. Extracting response content: Extracting from the model output <response>The token contains the response content, and information such as user questions, dialogue context, and tool call return results are also obtained, providing complete contextual information for subsequent multi-dimensional evaluation.

[0095] b. Independent Dimensional Evaluation: Using evaluation models with fewer parameters, responses are independently evaluated across three dimensions: timeliness, accuracy, and relevance. Each evaluation model focuses on a single dimension, enabling a more precise assessment of the response's performance in that dimension. Each evaluation model assigns a score from 0 to 2 based on its corresponding evaluation principles.

[0096] Timeliness assessment model: A timeliness score is given based on whether the response uses the latest knowledge, with a score range of 0 to 2.

[0097] Accuracy assessment model: Based on the consistency between the response and the standard answer, an accuracy score is given, ranging from 0 to 2 points.

[0098] Relevance assessment model: Based on the relevance of the response to the question and the tool, a relevance score is given, ranging from 0 to 2 points.

[0099] c. Score normalization: Normalize the scores for each dimension, mapping the score range from 0 to 2 to the range from 0 to 1.

[0100] d. Calculate the overall reward: Average the normalized scores of the three dimensions to obtain the overall reward of the model at the response level, with a value ranging from [0, 1]. This sequence-level reward is evenly distributed among each token in the response sequence.

[0101] In a preferred embodiment, the reward fusion module is designed to efficiently integrate reward signals of different granularities to generate a final comprehensive reward tensor for reinforcement learning training. The construction strategy is as follows:

[0102] Multi-dimensional integration: This module will reward correctness across multiple dimensions ( (including function-level, parameter-level, and response-level) and sequence-level format rewards ( Linear fusion is performed.

[0103] The reward distribution mechanism is as follows:

[0104] Tool call key token: For key tokens related to tool calls in the sequence (i.e. (Non-zero position), its reward value divided by The outer layer includes precise feedback signals at both the function and parameter levels. Non-instrument call tokens: For the remaining tokens in the sequence, their reward value is primarily determined by... Decide.

[0105] Response sequence tokens: For each token in the response sequence, its reward value is determined by the format reward and the response-level sequence-level reward, which are equally distributed among each token in the response sequence.

[0106] Through this hybrid reward mechanism, this invention achieves its key objectives: ensuring high-precision and discriminative feedback (function-level and parameter-level) to tool invocation-related tokens, ensuring comprehensive evaluation of response content through multi-dimensional evaluation at the response level, and maintaining the constraint of sequence-level rewards on the overall output structure quality. The final output is a comprehensive reward tensor. It will be deeply integrated with the Inter-group Relative Policy Optimization (GRPO) algorithm to achieve finer-grained advantage calculation and gradient update, thereby providing precise guidance signals for the generation behavior of each token during training.

[0107] As a preferred embodiment, the model training module aims to efficiently integrate the aforementioned multi-dimensional reward system (function-level, parameter-level, and response-level) into the Group Relative Policy Optimization (GRPO) algorithm framework to drive optimized training of the tool invocation capability of a large language model (LLM). As an advanced reinforcement learning policy optimization algorithm, the GRPO algorithm's core advantage lies in its ability to effectively overcome the inherent limitations of traditional policy gradient methods in handling the sparsity and high variance of reward signals by introducing relative advantage estimation of inter-group trajectories. Given that tool invocation tasks typically involve long, multi-step, complex decision chains, the stability and efficiency of the GRPO algorithm make it an ideal choice for guiding LLMs to learn accurate tool invocation behavior.

[0108] (1) Calculation of the advantage function based on multi-dimensional rewards:

[0109] In the reinforcement learning framework of this invention, such as Figure 4 As shown, the GRPO algorithm's advantage function calculation mechanism achieves significant enhancement by fusing multi-dimensional rewards from function-level, parameter-level, and response-level metrics to obtain a comprehensive reward signal. This allows the advantage function to move beyond a single sequence-level evaluation, providing refined value estimates for each token in the sequence (especially key function-level and parameter-level tokens related to tool calls), while simultaneously conducting a comprehensive semantic correctness assessment of the final response through multi-dimensional response-level evaluation. The specific implementation is as follows:

[0110] (1.1) Trajectory grouping and reward calculation:

[0111] For each user-input question Q in dataset D, multiple responses are generated via the Rollout process. , forming a group This group contains multiple responses and their corresponding reward values:

[0112] ;

[0113] in, This represents the response for each rollout; n is the number of responses in the group. This represents the reward for the i-th reply ( ).

[0114] (2.2) Reward distribution and integration mechanism:

[0115] (2.2.1) Tool call discrimination: After the model generates the sequence, first quickly check whether the sequence contains<tool_call> and< / tool_call> Special tool token.

[0116] If a tool call marker exists, the function-level and parameter-level reward calculation modules are activated. For each token in the sequence, a token-level correctness reward tensor is generated based on multi-dimensional calculation logic. .

[0117] If no tool call flag exists, the response-level reward calculation module is activated. For the entire sequence, based on the response-level scoring rules, a small-parameter evaluation model is used to independently evaluate the response content from three dimensions: timeliness, accuracy, and relevance, resulting in the response-level sequence-level reward tensor. .

[0118] (2.2.2) Hybrid Reward Fusion: The reward fusion module will... (Right now or ) and format rewards The components are then merged to generate the final comprehensive reward tensor. This integration ensures that key tokens invoked by the tool receive accurate feedback (at both the function and parameter levels), while also ensuring a comprehensive evaluation of the response content through the response level, and maintaining the structural normative constraints of the overall output.

[0119] (2.3) Calculate relative advantage:

[0120] (2.3.1) First, calculate the average reward of all trajectories in the GRPO rollout. and variance :

[0121] Calculate the mean;

[0122] Calculate the variance.

[0123] (2.3.2) Secondly, calculate the relative advantages between groups:

[0124] ;

[0125] in, It is a constant used to avoid division by zero; Q is the input to the model. This is the response for each rollout.

[0126] (2.3.3) Calculate the optimization objective:

[0127] ;

[0128] in, : Represents the objective function (policy objective) of GRPO, used to measure policy parameters The expected return (or advantage-weighted expectation) is used to optimize the objective and update the policy parameters so that the policy can obtain higher cumulative rewards in the environment.

[0129] : This represents the expectation of user questions Q in dataset D, where Q is the user's question and D is the dataset.

[0130] : indicates the model generated reply Take the expected value. This is the model's response.

[0131] ; Representation model Generate given Q The probability of. Current model In the given When, generate a reply The probability, These are the parameters of the model, which the model optimizes. To adjust behavior.

[0132] : Indicates the response generated by the old model (the model before the update) given Q. The probability, Used to measure the behavioral differences between the new model and the old model.

[0133] : Represents the advantage function, based on comprehensive reward Calculated and used to measure the response Given Q, take the current response. Compared to the "average response," it provides the model with a more accurate learning signal. Positive values ​​indicate that the action is better than average, while negative values ​​indicate that it is worse than average.

[0134] : Represents a clipping function that restricts the input x to the interval Inside. like Then output ;like Then output Otherwise, output x. These are pruning parameters (hyperparameters) used to limit the step size of model updates, preventing excessive model changes from causing instability, and controlling the aggressiveness of policy updates. The smaller the value, the more conservative the model updates; The larger the size, the more aggressive the updates (but the less stable they may be).

[0135] : This indicates taking the smaller value of the two parameters, used to select a more conservative update between "weighted advantage of the original model ratio" and "weighted advantage of the trimmed model ratio" (to prevent the policy update from being too large).

[0136] The model is trained using the above optimization objectives.

[0137] As a preferred embodiment, the present invention also provides a tool integration inference reinforcement learning method based on multi-dimensional rewards, comprising the following steps:

[0138] S1. Guided by predefined system prompts and tag sequences, the large language model autonomously generates a complete trajectory including the reasoning process, tool invocation actions, and final response. The large language model dynamically selects and invokes appropriate external tools to obtain information based on the user query, or directly generates the final response, and appends the tool invocation results to the dialogue history in the form of structured tags. The tool invocation results include: the model selects the invocation tool, the environment executes the invocation based on the tool selected by the model and the corresponding parameters, obtains the execution result, and returns the execution result to the model.

[0139] S2. Construct a multi-dimensional reward signal based on the tool call results or final response results of the large language model; wherein, the multi-dimensional reward signal includes format reward and multi-dimensional correctness reward; the format reward is used to determine whether the format of the output result conforms to the predefined format specification, including boundary integrity assessment and logical order standardization assessment; the multi-dimensional correctness reward includes three dimensions: function-level reward, parameter-level reward and response-level reward, which are used to accurately evaluate key structures such as function names, parameter keys and parameter values ​​in tool calls, and to conduct multi-dimensional semantic correctness assessment of the final response content.

[0140] S3. Efficiently integrate reward signals of different granularities to generate the final comprehensive reward tensor for reinforcement learning training; adopt differentiated fusion strategies according to different combinations of model output content: when the model output contains tool call key tokens but not response key tokens, its comprehensive reward is a linear combination of format reward and tool call key token reward (including multi-dimensional rewards at the function level and parameter level); when the model output contains response key tokens but not tool call key tokens, its comprehensive reward is a linear combination of format reward and model response reward (response-level sequence evaluation results);

[0141] S4. Integrate the comprehensive reward tensor into the inter-group relative policy optimization algorithm. For each user query, multiple responses are generated through the Rollout process to form a group. Calculate the average and variance of the rewards of all trajectories within the group. Calculate the inter-group relative advantage by comparing the reward values ​​of different trajectories within the same group, thereby optimizing the policy and driving the large language model to optimize the training of tool integration reasoning capabilities.

[0142] As a preferred embodiment, the present invention can be applied to the field of intelligent question-answering systems. In this application scenario, the system needs to dynamically invoke various external tools (such as search engines, knowledge base query tools, real-time data acquisition tools, etc.) to obtain information based on user queries and generate accurate and timely responses.

[0143] The specific application process is as follows: A user queries, "Please check today's weather in Beijing and tell me what clothes are suitable to wear." The system guides the large language model to generate a reasoning trajectory through the tool integration reasoning prompt word design module. The model first performs reasoning, identifying the need to call the weather query tool and the clothing recommendation tool. Then, it generates tool call instructions, calling the weather query tool to obtain today's weather information for Beijing and the clothing recommendation tool to obtain clothing suggestions. After the tool execution environment returns the execution results, the model generates a final response based on the obtained information, integrating the weather conditions and clothing suggestions into a complete answer and returning it to the user. In this application scenario, the multi-dimensional reward calculation module performs function-level and parameter-level evaluations on the tool calls generated by the model to ensure the correctness of tool names and parameters; it also performs response-level evaluations on the final response, assessing the response quality from three dimensions: timeliness, accuracy, and relevance, ensuring that the response is based on the latest weather data, accurately reflects the query results, and is highly relevant to the user's question. Through training with the multi-dimensional reward mechanism and the GRPO algorithm, the system can accurately select and call appropriate tools to generate high-quality answers, significantly improving the accuracy and user satisfaction of the intelligent question answering system.

[0144] This invention successfully implements a highly robust, scalable, and stable optimization paradigm. In systematic experimental evaluation, this method demonstrates outstanding performance improvement in key tool invocation accuracy metrics: compared to the existing baseline reinforcement learning method TooLRL, the accuracy improvement reaches 5%; compared to the instruction fine-tuning (SFT) model trained solely on static data, the performance gain is as high as 19%. The specific experimental process is as follows:

[0145] Experimental Setup: The experiment was conducted on eight NVIDIA A100 GPUs (each with 80GB of VRAM), using the PyTorch deep learning framework and the veRL reinforcement learning training framework. The base models used were Qwen-2.5-7B-Instruct and Llama3.1-8B-Instruct. The reinforcement learning training configuration was as follows: batch size set to 128, rollout count set to 4, training epochs set to 5, and the GRPO algorithm was used for training with parameter pruning. Set to 0.2, learning rate set to .

[0146] Dataset: The training dataset includes 2000 samples sampled from the Hammer dataset and 2000 samples sampled from the xLAM dataset, for a total of 4000 samples, covering various tool usage scenarios. Evaluation was conducted on three benchmark datasets: BFCL, API-Bank, and Bamboogle, comprehensively evaluating the model's tool selection, parameter imputation, and result processing capabilities.

[0147] Evaluation metrics: For tool selection, accuracy, recall, and F1 score are used for evaluation. For the final response, exact match (EM) and F1 score are used for evaluation.

[0148] Baseline Comparison: Experiments were compared with baseline methods such as the base model, single-stage SFT, SFT+PPO (Sequence-level Reward), and SFT+GRPO (Coarse-grained Reward). Experimental results show that our method (multi-dimensional reward GRPO) achieves state-of-the-art performance on all evaluation datasets. On the BFCL dataset, the accuracy reaches 0.82, a 40 percentage point improvement over the base model, a 7 percentage point improvement over SFT, and a 7 percentage point improvement over the GRPO method using coarse-grained rewards. On the API-Bank dataset, the accuracy reaches 67.32%, a 10 percentage point improvement over the base model, a 5 percentage point improvement over SFT, and a 3 percentage point improvement over the GRPO method using coarse-grained rewards. On the Bamboogle dataset, the accuracy reaches 73.05%, a 15 percentage point improvement over the base model, a 9 percentage point improvement over SFT, and a 2 percentage point improvement over the GRPO method using coarse-grained rewards, fully demonstrating the effectiveness of the multi-dimensional reward mechanism (function-level, parameter-level, and response-level).

[0149] Training Process: The training process includes reinforcement learning training based on multi-dimensional rewards. The reinforcement learning training takes approximately 48 hours. During training, the multi-dimensional reward calculation module performs precise token-level evaluation at the function and parameter levels for tool calls, performs multi-dimensional semantic evaluation at the sequence level for the final response, generates a comprehensive reward tensor through the reward fusion module, and finally calculates the dominance value and updates the gradient at the token granularity using the GRPO algorithm.

[0150] These experimental results strongly demonstrate the following core value of multi-dimensional reward design based on function level, parameter level, and response level for model training:

[0151] Enhanced tool integration reasoning capabilities: By allocating multi-dimensional token-level rewards at the function and parameter levels to key structural elements such as tool names and parameter key values, the accuracy and reliability of the model autonomously calling external tools in multi-step complex tasks are significantly improved.

[0152] Optimize key decision generation: The multi-dimensional feedback mechanism (function level, parameter level, and response level) provides highly discriminative gradient signals, enabling the model to learn the correct syntax and parameter filling rules for tool calls more efficiently, reducing redundant and erroneous token generation. At the same time, the multi-dimensional evaluation at the response level ensures the accuracy, timeliness, and relevance of the final response.

[0153] Improved task generalization performance: Under the stable and efficient GRPO framework, the introduction of multi-dimensional reward calculation (function level, parameter level and response level) significantly improves the model's generalization ability in unseen complex tool invocation scenarios.

[0154] In summary, this invention provides scientific and crucial technical support for enhancing the tool integration reasoning capabilities and task generalization performance of large-scale language models.

[0155] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process may be rearranged without departing from the scope of this disclosure. The appended method claims provide elements of various steps in an exemplary order and are not intended to limit the scope to the specific order or hierarchy described.< / response> < / response> < / observe> < / response> < / response> < / response>

Claims

1. A tool-integrated reasoning reinforcement learning system based on multi-dimensional rewards, characterized in that, include: The tool integrates a reasoning prompt word design module, which guides the large language model to autonomously generate a complete trajectory including the reasoning process, tool call actions, and final response through predefined system prompt words and tag sequences. The large language model dynamically selects and calls external tools to obtain information based on the user query, or directly generates the final response. A multi-dimensional reward calculation module is used to construct multi-dimensional reward signals based on the tool call results or final responses of a large language model. The multi-dimensional reward signals include format rewards and multi-dimensional correctness rewards. The format rewards are used to determine whether the format of the final response conforms to a predefined format. The multi-dimensional correctness rewards are used to evaluate the tool call results and to perform multi-dimensional semantic correctness evaluation on the final response. The reward fusion module is used to linearly combine format rewards and multi-dimensional correctness rewards to generate a final comprehensive reward tensor for reinforcement learning training. The model training module is used to calculate the advantage value and update the gradient based on the comprehensive reward tensor through the inter-group relative policy optimization algorithm, thereby driving the optimized training of the large language model to achieve tool integration reasoning capabilities.

2. The system according to claim 1, characterized in that, The tool integration reasoning prompt word design module includes: a predefined tag sequence comprising three types of tags: those used to guide the model to autonomously generate reasoning trajectories, those used to invoke callable tools, and those used to respond to natural language output by the user; the results of the large language model's invocation of callable tools and the final response generated are appended to the dialogue history to form the interaction trajectory of the large language model.

3. The system according to claim 1, characterized in that, The format reward includes: evaluating the boundary integrity and logical order regularity of the output results of the large language model, and giving different boundary integrity reward scores according to the boundary integrity; at the same time, constraining the logical timing of the complete trajectory generated by the large language model, and giving different logical order regularity reward scores according to the logical order regularity. The boundary integrity reward score and the logical order regularity reward score are added together to form the complete format reward.

4. The system according to claim 1, characterized in that, The multi-dimensional correctness reward includes function-level rewards, parameter-level rewards, and response-level rewards. If the large language model chooses to directly generate the final response, the generated response undergoes a response-level sequence-level evaluation, including scores for timeliness, accuracy, and relevance. These scores are then normalized and averaged, outputting a sequence-level reward score of 0 or 1. If the large language model chooses to invoke a tool, the tool invocation results are rewarded at both the function-level and parameter-level dimensions, with scores of 0 or 1 output based on the matching of the function name, parameter keys, and parameter values.

5. The system according to claim 1, characterized in that, The reward fusion module includes: adopting differentiated fusion strategies based on different combinations of model output content; when the model output contains a tool call key token but not a final response key token, its comprehensive reward is a linear combination of format reward and tool call key token reward, wherein the tool call key token reward includes multi-dimensional rewards at the function level and parameter level; when the model output contains a final response key token but not a tool call key token, its comprehensive reward is a linear combination of format reward and model response reward, wherein the model response reward is a response-level sequence-level reward.

6. The system according to claim 1, characterized in that, The model training module includes: based on the comprehensive reward tensor, using an inter-group relative policy optimization algorithm to calculate the advantage value and update the gradient at the token level. For each user question, multiple responses are generated through a Rollout process to form a group. The average and variance of the rewards of all trajectories within the group are calculated. By comparing the reward values ​​of different trajectories within the same group, the relative advantage between groups is calculated to optimize the policy, thereby driving the optimized training of the large language model to achieve tool integration reasoning capabilities.

7. A tool ensemble reasoning reinforcement learning method based on multi-dimensional rewards, characterized in that, Includes the following steps: The large language model is trained by predefined system prompt words and tag sequences to autonomously generate a complete trajectory including reasoning process, tool calling action and final response. The large language model dynamically selects and calls external tools to obtain information based on the user's question, or directly generates the final response and appends the tool calling results to the dialogue history. A multi-dimensional reward signal is constructed based on the tool call results or final response of the large language model; wherein, the multi-dimensional reward signal includes a format reward and a multi-dimensional correctness reward, the format reward is used to determine whether the format of the final response conforms to a predefined format, and the multi-dimensional correctness reward is used to evaluate the tool call results and to evaluate the multi-dimensional semantic correctness of the final response; The format reward and multi-dimensional correctness reward are linearly combined to generate the final comprehensive reward tensor for reinforcement learning training; Based on the comprehensive reward tensor, the inter-group relative policy optimization algorithm is used to calculate the advantage value and update the gradient at the token level, thereby driving the large language model to achieve optimized training of tool integration reasoning capabilities.

8. The method according to claim 7, characterized in that, The format reward includes: evaluating the boundary integrity of the output results of the large language model and assigning different boundary integrity reward scores based on the boundary integrity; and simultaneously constraining the logical timing of the complete trajectory generated by the large language model and assigning different logical order normalization reward scores based on the logical order normalization. The boundary integrity reward score and the logical order normalization reward score are added together to form the complete format reward.

9. The method according to claim 7, characterized in that, The multi-dimensional correctness reward includes function-level rewards, parameter-level rewards, and response-level rewards. If the large language model chooses to directly generate the final response, the final response generated by the large language model is evaluated at the response-level sequence level, including scores in three dimensions: timeliness, accuracy, and relevance. The scores are then normalized and averaged, with the output being a score of 0 or 1. If the large language model chooses to call a tool, the tool call results are rewarded at both the function-level and parameter-level dimensions, with the output being a score of 0 or 1 based on the matching of the function name, parameter key, and parameter value.

10. The method according to claim 7, characterized in that, The linear combination of format rewards and multi-dimensional correctness rewards includes: adopting differentiated fusion strategies based on different combinations of model output content; when the model output contains a tool call key token but not a final response key token, its comprehensive reward is a linear combination of format rewards and tool call key token rewards, wherein the tool call key token rewards include function-level and parameter-level multi-dimensional rewards; when the model output contains a final response key token but not a tool call key token, its comprehensive reward is a linear combination of format rewards and model response rewards, wherein the model response rewards are response-level sequence-level rewards.

Images