An agent evolution method based on GRPO and multi-stage verification

Abstract

The application discloses an agent evolution method based on GRPO and multi-stage verification, and belongs to the technical field of artificial intelligence. The method comprises the following steps: initializing policy parameters of an agent; delivering an input task to the agent to sample a tool calling sample; performing multi-stage verification on the tool calling sample to obtain a total reward of the tool calling sample; and updating the policy parameters of the agent according to the total reward to complete the evolution of the tool using capability of the agent. The application effectively constructs a large model agent structure and a reward function of a sampling sample, improves the adaptability and learning efficiency of the agent when facing a new tool, does not depend on heavy manual labeling, and realizes the autonomous evolution of the large model agent in the tool using capability.

Description

Technical Field

[0001] This patent belongs to the field of artificial intelligence technology, specifically referring to an agent evolution method based on GRPO and multi-stage verification. Background Technology

[0002] With the continuous development of artificial intelligence technology, Large Language Models (LLMs) have demonstrated outstanding performance in various natural language processing tasks. In recent years, researchers have begun to explore how to enable LLMs to have function call capabilities, that is, to allow the models to call corresponding application programming interfaces (APIs) to perform specific tasks based on natural language instructions. The realization of this function call capability can not only improve the practicality of the model in real-world application scenarios, but also expand its performance in complex tasks.

[0003] In the field of reinforcement learning (RL), researchers have proposed various algorithms to optimize the behavioral policies of models. Among them, the proximal policy optimization (PPO) algorithm and the relative policy optimization (GRPO) algorithm have achieved significant results in improving the decision-making ability of models. In particular, the GRPO algorithm, through relative policy optimization, can effectively improve the performance of models in complex tasks.

[0004] Currently, training large language model agents to use tools primarily relies on large-scale labeled datasets and supervised learning methods. While these methods can improve model performance to some extent, models often struggle to adapt to and utilize new, unseen tools.

[0005] In existing technologies, large language model agents have the following main drawbacks in terms of tool usage capabilities:

[0006] 1) Intelligent agents have difficulty quickly learning and adapting to the use of new and unseen tools.

[0007] 2) Existing training methods mainly rely on large-scale labeled datasets, and do not make full use of unlabeled data.

[0008] 3) The application of reinforcement learning algorithms in training tool usage ability is not deep enough, especially in how to effectively construct and utilize reward functions. Summary of the Invention

[0009] This invention proposes an agent evolution method based on GRPO and multi-stage verification. By effectively constructing the structure of large model agents and the reward function of sampled samples, the method improves the agent's adaptability and learning efficiency when facing new tools. It achieves autonomous evolution of large model agents in terms of tool usage capabilities without relying on cumbersome manual annotation.

[0010] To achieve the above objectives, the technical solution of the present invention includes the following:

[0011] An agent evolution method based on GRPO and multi-stage verification, the method comprising:

[0012] Initialize the agent's policy parameters;

[0013] The input task is passed to the agent to sample tool invocation samples; wherein, the tool invocation samples include: user task description, agent's tool invocation attempt and tool invocation execution result, and the agent's tool invocation attempt includes: tool name, tool description and parameters passed to the tool;

[0014] The tool call sample was validated in multiple stages to obtain the total reward of the tool call sample.

[0015] The agent's policy parameters are updated based on the total reward to complete the evolution of the agent's tool-using capabilities.

[0016] Furthermore, the network structure of the agent consists of a base policy LLM, two parallel multi-layer MLP networks, and a fusion network. The two parallel multi-layer MLP networks are connected to the last layer of the base policy LLM and are respectively network π. old and network π new Network π old L is used to output the token distribution for known tool calls. ExploitHead , π new Token distribution L used for outputting unknown tool calls ExploreHead The output g of the fusion network is given by: g = σ(W·avg_pooling(input_embedding)), where input_embedding represents the embedding vector of the input task, σ represents the activation function, W represents the learnable parameter vector, and avg_pooling represents average pooling. The output p of the agent is given by: output =SoftMax(g·L) Explorehead +(1-g)·L ExploitHead ).

[0017] Furthermore, the process of training the agent includes:

[0018] The fusion network is trained based on cross-entropy loss; wherein, the label of the output g of the fusion network comes from the tool call sample whose total reward is greater than a first set threshold w1. If the tool call sample calls an unknown tool, the label is 1; otherwise, the label is 0.

[0019] The base policy LLM and two parallel multilayer MLP networks are trained based on tool call samples; where, for unknown tool call samples, the network π old and network π new Simultaneously update; for known tool call samples, freeze network π. new No updates will be made and the network π will be updated. old The unknown tool call samples are tool call samples whose number of interactions with the agent is less than the second set threshold w2, and the known tool call samples are tool call samples whose number of interactions with the agent is greater than the second set threshold w2.

[0020] Furthermore, the tool call sample is subjected to multi-stage validation to obtain the total reward of the tool call sample, including:

[0021] Based on the agent's attempts to invoke tools, the format of the tool invocation samples is validated, and a format reward r is obtained. format ;

[0022] The validity of the tool call samples is verified based on the tool call execution results, and a validity reward r is obtained. vaild ;

[0023] Based on user task descriptions, agent attempts to invoke tools, and tool invocation execution results, the relevance of tool invocation samples is verified, and a relevance reward r is obtained. rel ;

[0024] Reward r according to the format format Tool effectiveness reward r vaild and correlation reward r rel The total reward for the tool's call samples is obtained.

[0025] Furthermore, based on the agent's attempts to invoke tools, the format of the tool invocation samples is validated to obtain a format reward r. format ,include:

[0026] Determine whether the parameters passed to the tool contain all the required parameters of the tool to obtain the parameter integrity verification result g1; wherein the parameter integrity verification result g1 takes the value of 0 or 1;

[0027] Determine whether the parameter type in the parameters passed to the tool is consistent with the parameter type corresponding to the parameters required by the tool, and obtain the parameter type consistency verification result g2; wherein, the parameter type consistency verification result g2 takes the value of 0 or 1;

[0028] Determine whether the parameter value range in the input tool is consistent with the parameter value range required by the tool, and obtain the parameter value range consistency verification result g3; where the parameter value range consistency verification result g3 is 0 or 1;

[0029] The tool name is used to determine whether the tool is in the list of available tools, so as to obtain the tool call legality verification result g4; where the value of the tool call legality verification result g4 is 0 or 1;

[0030] Based on the parameter integrity verification result g1, parameter type consistency verification result g2, parameter value range consistency verification result g3, and tool call legality verification result g4, the format reward r is obtained. format .

[0031] Furthermore, based on the user task description, the agent's attempts to invoke the tool, and the tool invocation execution results, the relevance of the tool invocation samples is verified, and the relevance reward r is obtained. rel ,include:

[0032] Determine whether the execution result of the tool call is abnormal to obtain a validity verification result v1; wherein the value of the validity verification result v1 is 0 or 1;

[0033] The validity verification result v2 is obtained by determining whether the tool timed out by the time the tool returned the execution result. The validity verification result v2 can be either 0 or 1.

[0034] The validity verification result v3 is obtained by verifying whether the execution result of the verification tool is empty; wherein the value of the validity verification result v3 is 0 or 1;

[0035] Based on the validity verification results v1, v2, and v3, the validity reward r is obtained. vaild .

[0036] Furthermore, based on the user task description, the agent's attempts to invoke the tool, and the tool's execution results, a correlation verification is performed to obtain the correlation reward r. rel ,include:

[0037] The correlation verification result c1 is obtained by calculating the consistency between the user task description and the tool description; wherein the value of the correlation verification result c1 is 0 or 1.

[0038] The correlation verification result c2 is obtained by calculating the consistency between the user task description and the parameters of the input tool; wherein the value of the correlation verification result c2 is 0 or 1.

[0039] The correlation verification result c3 is obtained by calculating the consistency between the user task description and the tool call execution result; wherein the value of the correlation verification result c3 is 0 or 1.

[0040] Based on the correlation verification results c1, c2, and c3, the correlation reward r is obtained. rel .

[0041] Furthermore, by calculating the consistency between the user task description and the tool description, the relevance verification result c1 is obtained, including:

[0042] Calculate the consistency between user task descriptions and tool descriptions. Where Q represents the user task description, |Q| represents the total number of words in the user task description, and T desc Indicates a tool description, |T desc | represents the total number of words in the tool description, f(q) i ,T desc ) indicates word q i In the tool description T desc The number of times it appears in the word, b represents the hyperparameter, and E(·) represents the word embedding vector representation;

[0043] The consistency E is compared with the third set threshold w3 to obtain the correlation verification result c1.

[0044] Furthermore, the correlation verification results Where P represents the parameters passed to the tool, δ(*) indicates that the internal comparison true value is 1, Q represents the user task description, E(·) represents the word embedding vector representation, and w4 represents the fourth set threshold;

[0045] The correlation verification results Where Q represents the user task description, |Q| represents the total number of words in the user task description, and R tool The result of the tool call execution is represented by δ(*), which indicates that the true value of the internal comparison is 1, E(·) represents the word embedding vector representation, and w5 represents the fifth set threshold.

[0046] An agent evolution system based on GRPO and multi-stage verification, the system comprising:

[0047] The initialization module is used to initialize the policy parameters of the agent.

[0048] A sampling module is used to pass the input task to the agent to sample tool call samples; wherein, the tool call samples include: user task description, agent's tool call attempt and tool call execution result, and the agent's tool call attempt includes: tool name, tool description and parameters passed to the tool;

[0049] The verification module is used to perform multi-stage verification of the tool call sample to obtain the total reward of the tool call sample;

[0050] An update module is used to update the agent's policy parameters based on the total reward, thereby completing the evolution of the agent's tool-using capabilities.

[0051] Compared with the prior art, the present invention has at least the following beneficial effects.

[0052] 1) Enhanced ability to adapt to new tools: The intelligent agent tool usage evolution method proposed in this invention enables the intelligent agent to gradually adapt to the usage of new, unseen tools through continuous learning and optimization. This greatly improves the flexibility and practicality of the intelligent agent in real-world application scenarios.

[0053] 2) Reduced reliance on labeled data: By utilizing unlabeled data and constructing an efficient reward function, this invention reduces reliance on large-scale labeled datasets. This not only lowers the cost of data collection and labeling but also enables agents to continue learning and evolving even in data-scarce environments.

[0054] 3) Optimization algorithm efficiency: Using the GRPO algorithm to optimize the agent can effectively improve the agent's performance in tool use tasks. The relative policy optimization mechanism of the GRPO algorithm enables the agent to explore and update the policy space efficiently, thereby converging to the optimal policy more quickly.

[0055] 4) Reasonableness of the reward function: A multi-stage verification method is proposed. The reward function constructed in this invention can accurately reflect the quality of the sampled data. This reward estimation method based on multi-stage verification not only improves the accuracy of the reward value but also provides more meaningful guidance for the optimization of the agent.

[0056] 5) Wide Applicability: The method proposed in this invention is applicable to various large-scale intelligent agents. Whether the model is based on supervised learning or reinforcement learning, its tool-using ability can be improved through this method. This provides a universal solution for the tool-using capabilities of intelligent agents in different application scenarios (including the ability of embodied intelligent agent robots to perform actions and adapt to environmental changes).

[0057] 6) In-depth exploration of unknown tools: In addition to being applicable to various large-scale intelligent agents, this invention also discloses a more effective structure for large-scale intelligent agents. This structure can not only fully explore unknown tools, changing "passive sampling" to "active discovery," but also utilize high-reward samples for high-confidence sample replay. Attached Figure Description

[0058] Figure 1 A flowchart of an agent evolution method based on GRPO and multi-stage verification.

[0059] Figure 2 A schematic diagram of the structure of a Dual-Head Router network. Detailed Implementation

[0060] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, and to make the objectives, features, and advantages of the present invention more apparent and understandable, the core technologies of the present invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0061] This invention presents an agent evolution method based on GRPO and multi-stage verification. It employs the GRPO algorithm to evolve the tool-using ability of a large language model agent and constructs an effective reward function through multi-stage verification of sampled data. For example... Figure 1 As shown, it mainly includes the following steps 1 to 4.

[0062] Step 1: Initialize the agent's policy parameters.

[0063] This invention first initializes the policy parameters θ of the large model agent based on experience, so that the agent has the function of calling tools.

[0064] In a preferred embodiment, this invention proposes a Dual-Head Router network that, without compromising the Critic-free nature of GRPO, directly enhances the characteristics and requirements of "unknown function exploration and high-reward reuse," such as... Figure 2 As shown, this network allows the model to quickly switch between "exploration vs. exploitation" and explicitly leaves "exploration gates" for unknown functions.

[0065] 1) Network structure.

[0066] A. The base strategy LLM remains unchanged, and two parallel multilayer MLP networks (denoted as π) are added to the last layer. old and π new The outputs are denoted as L. ExploitHead and L ExploreHeadπ old L is used to output the token distribution for known tool calls. ExploitHead π new Token distribution L used for outputting unknown tool calls ExploreHead .

[0067] B. Add a new route converged network router:

[0068] g=σ(W·avg_pooling(input_embedding))

[0069] Where W is a learnable parameter vector, the embedding vector prompt_embed of the input task query is averaged and then g is obtained through a routing fusion network.

[0070] C. Final output of the policy network

[0071] p output =SoftMax(g·L) Explorehead +(1-g)·L ExploitHead )

[0072] 2) Model training.

[0073] The fusion network is trained based on cross-entropy loss; wherein, the label of the output g of the fusion network comes from the tool call sample whose total reward is greater than a first set threshold w1. If the tool call sample calls an unknown tool, the label is 1; otherwise, the label is 0.

[0074] The base policy LLM and two parallel multilayer MLP networks are trained based on tool call samples; where, for unknown tool call samples, the network π old and network π new Simultaneously update; for known tool call samples, freeze network π. new No updates will be made and the network π will be updated. old The unknown tool call samples are tool call samples whose number of interactions with the agent is less than the second set threshold w2, and the known tool call samples are tool call samples whose number of interactions with the agent is greater than the second set threshold w2.

[0075] Step 2: Pass the input task to the agent to sample tool call samples.

[0076] This invention passes the input task to an agent, and generates multiple tool call samples by sampling from unlabeled tool usage data based on the agent's output. These samples include user task descriptions, agent attempts to call tools (including new tools never seen before and old tools that have been trained) (including the tool name, tool description, and parameter values ​​passed to the tool), and the corresponding tool call execution results.

[0077] Step 3: Perform multi-stage validation on the tool call sample to obtain the total reward for the tool call sample.

[0078] This invention performs multi-stage verification on the sampled data, including format verification, tool validity verification, and relevance verification, in order to determine the quality of tool calls in the sample and estimate the corresponding reward value.

[0079] Specifically, the reward function construction module needs to perform the following three stages of verification on the sampled samples:

[0080] 1. Format validation.

[0081] Check whether the utility function calls in the sampled data conform to predefined format requirements. For example, check whether the function calls contain the correct parameter types and numbers, whether they follow the calling conventions for utility functions, and whether the generated calls contain parameters or tools that are not in the given available toolset.

[0082] Specifically, as shown in Table 1, the format validation reward r is obtained by validating the parameters passed to the calling tool. format = g1·g2·g3·g4. Wherein, if any one of g1·g2·g3·g4 is 0, the format validation fails and the format validation reward value is 0.

[0083]

[0084] Table 1

[0085] 2. Tool validity verification.

[0086] This invention verifies whether a tool call conforms to the tool's expected behavior to check the execution validity of the calling tool. For example, unsuccessful executions will be filtered out. Reasons for failure may include incorrect parameter type, invalid parameters, runtime errors, timeouts, syntax errors, missing parameters, etc.

[0087] Specifically, as shown in Table 2, this invention obtains the tool validity verification result r based on the tool call execution result. valid = v1·v2·v3. Wherein, if any one of the sub-items v1·v2·v3 is 0, the format validation fails and the tool validity reward value is 0.

[0088] Sub-item symbol Value illustrate Execution successful <![CDATA[v1]]> {0,1} The tool executed without any errors. No timeout <![CDATA[v2]]> {0,1} The tool executed without a timeout. The execution result is not empty. <![CDATA[v3]]> {0,1} The tool execution result is not empty.

[0089] Table 2

[0090] 3. Correlation verification.

[0091] This invention formats and passes the results of successfully executed tool calls, the list of available tools, the generated tool calls, and the user's task description to another large model, or evaluates the consistency between function calls, execution results, and query targets by calculating text similarity.

[0092] ● "Task-tool call" pairs that produce meaningless results due to whether the task is infeasible or the calling parameters are incorrect.

[0093] ● Does the tool call align with the task objective and have appropriate parameters?

[0094] ● Whether the tool invocation and parameter selection are appropriately selected from the available toolset;

[0095] ● Whether the parameter selection for tool calls matches the user's task description.

[0096] ● Does the execution result match the purpose described in the user's task description?

[0097] The above verification evaluates the relevance between the results of the tool call and the original question. This step aims to ensure that the tool call is not only syntactically and logically correct, but also truly solves the user's problem. For example, if a user asks for recommendations for attractions in a city, check whether the function call's result provides information on attractions related to that city and whether it matches the user's interests and needs. If a function to retrieve weather information is called, check whether the returned result contains accurate weather data and whether it matches the location and time queried by the user.

[0098] Specifically, the evaluation assesses the correlation and consistency among the three elements: "task (query) - tool invocation - tool execution result," comprising three sub-items, each with a weight w. i Adjustable, but all validations must pass (otherwise, the overall validation is set to 0). Assume the task description is Q, and the tool description of the invoked tool is T. desc The description of parameter i of the called tool is denoted as The result of the tool execution is denoted as R. tool The BERT embedding of word w is denoted as E(w), and the specific calculation methods for each sub-item are as follows.

[0099] 1) Description T of the tool_call selected for invocation desc Consistent rewards with Task Q.

[0100] First, calculate the task Q and the tool description T.desc Correlation:

[0101]

[0102] Where f(q,D) represents the number of times word q appears in the document, |D| represents the number of words in the document, and b is a hyperparameter set to 0.2, which can be adjusted in conjunction with a threshold. Then, the description T of the selected invoked tool is calculated. desc The consistency reward with task Q is as follows:

[0103]

[0104] w3 is the third set threshold, with a value between 0 and 1.

[0105] 2) The parameter value P is passed into the tool_call function to be invoked. i Consistency reward with task Q:

[0106]

[0107] Where P represents all parameters in tool_call, δ(*) represents a value of 1 when the internal comparison is true, and w4 is the fourth set threshold, with a value range between 0 and 1.

[0108] 3) The result of the selected tool execution R tool Consistency reward with task Q

[0109]

[0110] Among them, w5 is the fifth set threshold, and its value ranges from 0 to 1.

[0111] Finally, the correlation verification reward:

[0112] r rel =c1·c2·c3.

[0113] In summary, the final reward value is obtained by chaining the rewards across multiple stages, as follows:

[0114] Total_Reward = r format ·r vaild ·r rel

[0115] Through the above three stages of verification, the present invention can accurately evaluate the quality of the sampled data and construct a reasonable reward function accordingly.

[0116] This invention assigns a corresponding weight to each of the above verification items. The final reward is obtained by weighted summation of all verification items. When all verifications pass, the reward value is 1.0. If a verification rule fails, the reward value is forcibly set to 0.0.

[0117] Step 4: Update the agent's policy parameters based on the total reward to complete the evolution of the agent's tool-using ability.

[0118] This invention calculates the reward value for each sample based on the reward function, optimizes the agent model parameters using the GRPO algorithm, and updates the parameters of its policy network model, thereby improving the agent's performance in tool use tasks.

[0119] Specifically, after constructing the reward function, this invention performs the aforementioned multi-stage verification process for each sample, calculates its corresponding reward value r, and updates the agent's policy parameters θ using the GRPO algorithm based on the reward value r. The GRPO algorithm is a reinforcement learning algorithm based on relative policy optimization, capable of effectively exploring and updating the policy space.

[0120] In summary, this invention uses the GRPO algorithm to evolve the tool-using capabilities of large-scale intelligent agents, which can effectively improve the agent's adaptability and learning efficiency when facing new tools.

[0121] This invention proposes to perform format verification, tool validity verification, and correlation verification on the sampled samples, thereby constructing an effective reward function that can accurately evaluate the quality of the samples and provide reliable information for the optimization of the agent.

[0122] The present invention makes full use of unlabeled data in the training and optimization of the intelligent agent tool's ability, reducing the dependence on large-scale labeled datasets.

[0123] The embodiments described above are merely illustrative of implementation methods of the present invention, and while the descriptions are detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. An agent evolution method based on GRPO and multi-stage verification, characterized in that, The method includes: Initialize the agent's policy parameters; The input task is passed to the agent to sample tool invocation samples; wherein, the tool invocation samples include: user task description, agent's tool invocation attempt and tool invocation execution result, and the agent's tool invocation attempt includes: tool name, tool description and parameters passed to the tool; The tool call sample was validated in multiple stages to obtain the total reward of the tool call sample. Based on the total reward, and by updating the agent's policy parameters according to the GRPO algorithm, the evolution of the agent's tool-using ability is completed; The network structure of the intelligent agent consists of a base policy LLM, two parallel multi-layer MLP networks, and a fusion network. The two parallel multi-layer MLP networks are connected to the last layer of the base policy LLM and are respectively network... and the Internet ,network Used to output the token distribution for known tool calls. , Token distribution used to output unknown tool calls The output of the fusion network , This represents the embedding vector of the input task. This represents the activation function. Represents a learnable parameter vector. Indicates average pooling, the agent's output ; The process involves multi-stage validation of the tool call sample to obtain the total reward for the tool call sample, including: Based on the agent's attempts to invoke tools, the format of the tool invocation samples is validated, and a format reward is obtained. ; The validity of tool call samples is verified based on the tool call execution results, and a validity reward is obtained. ; The relevance of tool call samples is verified based on user task descriptions, agent attempts to call tools, and tool call execution results, and a relevance reward is obtained. ; Rewards according to format Tool effectiveness reward and related rewards The total reward for the tool's call samples is obtained.

2. The method according to claim 1, characterized in that, The process of training the agent includes: The fusion network is trained based on cross-entropy loss; wherein the output of the fusion network is... The tags are from total rewards greater than a first set threshold. The tool call sample is labeled as follows: if the tool call sample calls an unknown tool, the label is 1; otherwise, the label is 0. The base policy LLM and two parallel multilayer MLP networks are trained based on tool call samples; where, for unknown tool call samples, the network... and the Internet Simultaneously update; freeze the network for known tool call samples. No updates will be made and the network will be updated. The unknown tool call sample is one where the number of times the agent makes contact is less than a second preset threshold. The known tool call samples are those in which the number of times the agent makes contact exceeds a second preset threshold. Tool call sample.

3. The method according to claim 1, characterized in that, Based on the agent's attempts to invoke tools, the format of the tool invocation samples is validated, and a format reward is obtained. ,include: Determine whether the parameters passed to the tool include all the required parameters for that tool to obtain the parameter integrity verification result. The parameter integrity verification result is mentioned above. The value can be 0 or 1; Determine whether the parameter types in the parameters passed to the tool are consistent with the parameter types required by the tool, and obtain the parameter type consistency verification result. The parameter type consistency verification result is mentioned above. The value can be 0 or 1; Determine whether the range of parameter values ​​in the parameters passed to the tool is consistent with the range of parameter values ​​required by the tool, and obtain the parameter value range consistency verification result. Among them, the consistency verification results of parameter value ranges. The value can be 0 or 1; Determine whether the tool is in the list of available tools based on its name to obtain the result of the tool call validity verification. Among them, the result of tool call legality verification. The value can be 0 or 1; Based on the parameter integrity verification results Parameter type consistency verification results Parameter value range consistency verification results and the results of tool call legality verification Receive format reward .

4. The method according to claim 1, characterized in that, The validity of tool call samples is verified based on the tool call execution results, and a validity reward is obtained. ,include: Determine if the tool's execution result is abnormal in order to obtain a validity verification result. The validity verification result The value can be 0 or 1; The time it takes for the tool to return the execution result determines whether the tool has timed out, thus obtaining a validity verification result. The validity verification result The value can be 0 or 1; The validity verification result is obtained by verifying whether the execution result of the verification tool is empty. The validity verification result The value can be 0 or 1; Based on the validity verification results Validity verification results and validity verification results Receive effectiveness reward .

5. The method according to claim 1, characterized in that, A correlation reward is obtained by verifying the relevance of the user task description, the agent's attempts to invoke the tool, and the tool's execution results. ,include: The relevance verification results are obtained by calculating the consistency between the user task description and the tool description. The correlation verification result The value can be 0 or 1; The correlation verification results are obtained by calculating the consistency between the user task description and the parameters passed to the tool. The correlation verification result The value can be 0 or 1; The correlation verification results are obtained by calculating the consistency between the user task description and the tool call execution results. The correlation verification result The value can be 0 or 1; Based on the correlation verification results Correlation verification results Correlation verification results Receive relevant rewards .

6. The method according to claim 5, characterized in that, The relevance verification results are obtained by calculating the consistency between the user task description and the tool description. ,include: Calculate the consistency between user task descriptions and tool descriptions. ;in, This indicates the user's task description. This indicates the total number of words in the user task description. Indicates tool description, This indicates the total number of words in the tool description. Indicator In the tool description The number of times it appears in Indicates hyperparameters, Word embedding vector representation; This consistency With the third set threshold The results of the correlation verification were obtained through comparison. .

7. The method according to claim 5, characterized in that, The correlation verification results ;in, This indicates the parameters passed to the tool. This indicates that the true value of the internal comparison is 1. This indicates the user's task description. This represents the word embedding vector representation. This indicates the fourth set threshold; The correlation verification results ;in, This indicates the user's task description. This indicates the total number of words in the user task description. This indicates the result of the tool call execution. This indicates that the true value of the internal comparison is 1. This represents the word embedding vector representation. This indicates the fifth threshold setting.

8. An agent evolution system based on GRPO and multi-stage verification, characterized in that, The system includes: The initialization module is used to initialize the policy parameters of the agent. A sampling module is used to pass the input task to the agent to sample tool call samples; wherein, the tool call samples include: user task description, agent's tool call attempt and tool call execution result, and the agent's tool call attempt includes: tool name, tool description and parameters passed to the tool; The verification module is used to perform multi-stage verification of the tool call sample to obtain the total reward of the tool call sample; An update module is used to update the agent's policy parameters based on the total reward and the GRPO algorithm to complete the evolution of the agent's tool-using ability; The network structure of the intelligent agent consists of a base policy LLM, two parallel multi-layer MLP networks, and a fusion network. The two parallel multi-layer MLP networks are connected to the last layer of the base policy LLM and are respectively network... and the Internet ,network Used to output the token distribution for known tool calls. , Token distribution used to output unknown tool calls The output of the fusion network , This represents the embedding vector of the input task. This represents the activation function. Represents a learnable parameter vector. Indicates average pooling, the agent's output ; The process involves multi-stage validation of the tool call sample to obtain the total reward for the tool call sample, including: Based on the agent's attempts to invoke tools, the format of the tool invocation samples is validated, and a format reward is obtained. ; The validity of tool call samples is verified based on the tool call execution results, and a validity reward is obtained. ; The relevance of tool call samples is verified based on user task descriptions, agent attempts to call tools, and tool call execution results, and a relevance reward is obtained. ; Rewards according to format Tool effectiveness reward and related rewards The total reward for the tool's call samples is obtained.

Images