Intelligent agent evolution method based on GRPO and multi-stage verification

Abstract

The invention discloses an agent evolution method based on GRPO and multi-stage verification, and belongs to the technical field of artificial intelligence. The method comprises the following steps: initializing strategy parameters of an intelligent agent; transmitting the input task to an intelligent agent, and sampling to obtain a tool calling sample; performing multi-stage verification on the tool calling sample to obtain a total reward of the tool calling sample; and updating strategy parameters of the intelligent agent according to the total reward so as to complete the evolution of the tool use capability of the intelligent agent. According to the method, the large model agent structure and the reward function of the sampled sample are effectively constructed, the adaptive capacity and learning efficiency of the agent facing a new tool are improved, the method does not depend on heavy manual annotation, and autonomous evolution of the large model agent on the tool use capacity is achieved.

Description

TECHNICAL FIELD

[0001] The patent belongs to the technical field of artificial intelligence, in particular to an agent evolution method based on GRPO and multi-stage verification. BACKGROUND

[0002] With the continuous development of artificial intelligence technology, large language models (LLMs) have shown excellent performance in various natural language processing tasks. In recent years, researchers have begun to explore how to enable large language models to have the ability to use tools (function calls), that is, to enable the model to call the corresponding application programming interface (API) according to natural language instructions to perform specific tasks. The implementation of this tool usage capability not only improves the practicality of the model in actual application scenarios, but also expands its performance in complex tasks.

[0003] In the field of reinforcement learning (RL), researchers have proposed various algorithms to optimize the behavior strategy of the model, among which the proximal policy optimization (PPO) algorithm and the GRPO algorithm have made remarkable achievements in improving the decision-making ability of the model. In particular, the GRPO algorithm can effectively improve the performance of the model in complex tasks through relative policy optimization.

[0004] Currently, the training of the tool usage capability of large language model agents mainly relies on large-scale labeled datasets and supervised learning methods. These methods can improve the performance of the model to some extent, but when faced with new, never-before-seen tools, the model often has difficulty adapting and using these new tools.

[0005] In the existing technology, the large language model agent has the following main shortcomings in terms of tool usage capability:

[0006] 1) For new, never-before-seen tools, the agent has difficulty quickly learning and adapting to their usage methods.

[0007] 2) Existing training methods mainly rely on large-scale labeled datasets, and the use of unlabeled data is not sufficient.

[0008] 3) Reinforcement learning algorithms are not sufficiently applied in tool usage capability training, especially in how to effectively construct and utilize reward functions. SUMMARY

[0009] The present application proposes an agent evolution method based on GRPO and multi-stage verification, which effectively constructs the reward function of the large model agent structure and the sampled samples, improves the adaptability and learning efficiency of the agent when facing new tools, does not rely on heavy manual labeling, and realizes the autonomous evolution of the large model agent in tool usage capability.

[0010] To achieve the above object, the technical scheme of the present application includes the following contents.

[0011] An agent evolution method based on GRPO and multi-stage verification, the method comprising:

[0012] Initializing the strategy parameters of the agent;

[0013] Passing the input task to the agent to sample tool invocation samples; wherein the tool invocation samples include: user task description, agent's invocation attempt on the tool, and tool invocation execution result, the agent's invocation attempt on the tool includes: tool name, tool description, and parameters passed to the tool;

[0014] Performing multi-stage verification on the tool invocation samples to obtain the total reward of the tool invocation samples;

[0015] Updating the strategy parameters of the agent according to the total reward to complete the evolution of the agent's tool usage capability.

[0016] Further, the network structure of the agent is composed of a base strategy LLM, two parallel multi-layer MLP networks and a fusion network, wherein the two parallel multi-layer MLP networks are connected to the last layer of the base strategy LLM, and are respectively network π old and network π new , network π old is used to output the token distribution L ExploitHead for known tool invocation, network π new is used to output the token distribution L ExploreHead for unknown tool invocation, the output g of the fusion network is g=σ(W·avg_pooling(input_embedding)), input_embedding represents the embedding vector of the input task, σ represents the activation function, W represents the learnable parameter vector, avg_pooling represents the average pooling, and the output p output of the agent is p Explorehead =SoftMax(g·L ExploitHead +(1-g)·L ).

[0017] Further, the process of training the agent includes:

[0018] Training the fusion network based on cross-entropy loss; wherein the label of the output g of the fusion network comes from the tool invocation sample whose total reward is greater than a first set threshold w1, if the tool invocation sample invokes an unknown tool, the label is 1, otherwise the label is 0;

[0019] The base strategy LLM and two parallel multi-layer MLP networks are trained based on tool call samples; wherein, for unknown tool call samples, the network π old and the network π new are updated simultaneously; for known tool call samples, the network π new is not updated and the network π old is updated; the unknown tool call samples are tool call samples with the number of agent contacts less than a second set threshold w2, and the known tool call samples are tool call samples with the number of agent contacts greater than the second set threshold w2.

[0020] Further, the tool call samples are verified in multiple stages to obtain the total reward of the tool call samples, including:

[0021] The format of the tool call samples is verified based on the tool calling attempt of the agent, and a format reward r format is obtained.

[0022] The effectiveness of the tool call samples is verified based on the tool calling execution result, and an effectiveness reward r vaild is obtained.

[0023] The relevance of the tool call samples is verified based on the user task description, the tool calling attempt of the agent and the tool calling execution result, and a relevance reward r rel is obtained.

[0024] The total reward of the tool call samples is obtained according to the format reward r format , the tool effectiveness reward r vaild and the relevance reward r rel .

[0025] Further, the format of the tool call samples is verified based on the tool calling attempt of the agent, and a format reward r format is obtained, including:

[0026] It is judged whether all the required parameters of the tool are contained in the parameters of the tool, to obtain a parameter integrity verification result g1; wherein, the parameter integrity verification result g1 has a value of 0 or 1.

[0027] It is judged whether the parameter type in the parameters of the tool is consistent with the parameter type corresponding to the required parameters of the tool, to obtain a parameter type consistency verification result g2; wherein, the parameter type consistency verification result g2 has a value of 0 or 1.

[0028] determine whether the parameter value range in the parameter of the incoming tool is consistent with the parameter value range corresponding to the required parameter of the tool, to obtain a parameter value range consistency verification result g3; the value of the parameter value range consistency verification result g3 is 0 or 1;

[0029] determine whether the tool is in the optional tool list based on the tool name, to obtain a tool call legality verification result g4; the value of the tool call legality verification result g4 is 0 or 1;

[0030] obtain a format reward r based on the parameter integrity verification result g1, the parameter type consistency verification result g2, the parameter value range consistency verification result g3, and the tool call legality verification result g4 format .

[0031] Further, based on the user task description, the agent's calling attempt on the tool, and the tool calling execution result, the relevance of the tool calling sample is verified, to obtain a relevance reward r rel , including:

[0032] determine whether the tool calling execution result is abnormal, to obtain an effectiveness verification result v1; the value of the effectiveness verification result v1 is 0 or 1;

[0033] determine whether the tool is executed overtime by the time of returning the tool calling execution result by the tool, to obtain an effectiveness verification result v2; the value of the effectiveness verification result v2 is 0 or 1;

[0034] obtain an effectiveness verification result v3 by verifying whether the tool calling execution result is empty; the value of the effectiveness verification result v3 is 0 or 1;

[0035] obtain an effectiveness reward r based on the effectiveness verification result v1, the effectiveness verification result v2, and the effectiveness verification result v3 vaild .

[0036] Further, based on the user task description, the agent's calling attempt on the tool, and the tool calling execution result, the relevance is verified, to obtain a relevance reward r rel , including:

[0037] obtain a relevance verification result c1 by calculating the consistency between the user task description and the tool description; the value of the relevance verification result c1 is 0 or 1;

[0038] obtain a relevance verification result c2 by calculating the consistency between the user task description and the parameter of the incoming tool; the value of the relevance verification result c2 is 0 or 1;

[0039] The correlation verification result c3 is obtained by calculating the consistency between the user task description and the tool call execution result; wherein the value of the correlation verification result c3 is 0 or 1.

[0040] Based on the correlation verification results c1, c2, and c3, the correlation reward r is obtained. rel .

[0041] Furthermore, by calculating the consistency between the user task description and the tool description, the relevance verification result c1 is obtained, including:

[0042] Calculate the consistency between user task descriptions and tool descriptions. Where Q represents the user task description, |Q| represents the total number of words in the user task description, and T desc Indicates a tool description, |T desc | represents the total number of words in the tool description, f(q) i ,T desc ) indicates word q i In the tool description T desc The number of times it appears in the word, b represents the hyperparameter, and E(·) represents the word embedding vector representation;

[0043] The consistency E is compared with the third set threshold w3 to obtain the correlation verification result c1.

[0044] Furthermore, the correlation verification results Where P represents the parameters passed to the tool, δ(*) indicates that the internal comparison true value is 1, Q represents the user task description, E(·) represents the word embedding vector representation, and w4 represents the fourth set threshold;

[0045] The correlation verification results Where Q represents the user task description, |Q| represents the total number of words in the user task description, and R tool The result of the tool call execution is represented by δ(*), which indicates that the true value of the internal comparison is 1, E(·) represents the word embedding vector representation, and w5 represents the fifth set threshold.

[0046] An agent evolution system based on GRPO and multi-stage verification, the system comprising:

[0047] The initialization module is used to initialize the policy parameters of the agent.

[0048] A sampling module is configured to pass an input task to an agent to sample a tool invocation sample, wherein the tool invocation sample includes a user task description, an agent's tool invocation attempt, and a tool invocation execution result, and the agent's tool invocation attempt includes a tool name, a tool description, and parameters passed to the tool.

[0049] A verification module is configured to perform multi-stage verification on the tool invocation sample to obtain a total reward of the tool invocation sample.

[0050] An updating module is configured to update policy parameters of the agent according to the total reward to complete evolution of tool usage capability of the agent.

[0051] Compared with the prior art, the present application has at least the following beneficial effects.

[0052] 1) Enhanced ability to adapt to new tools: The agent tool usage capability evolution method proposed by the present application enables the agent to gradually adapt to the use method of new tools by continuous learning and optimization when encountering new tools that have never been seen before. This greatly improves the flexibility and practicality of the agent in actual application scenarios.

[0053] 2) Reduced dependence on labeled data: By utilizing unlabeled data and constructing an effective reward function, the present application reduces the dependence on large-scale labeled data sets. This not only reduces the cost of data collection and labeling, but also enables the agent to continue learning and evolving in the case of data scarcity.

[0054] 3) High efficiency of the optimization algorithm: The GRPO algorithm is used to optimize the agent, which can effectively improve the performance of the agent in tool usage tasks. The relative policy optimization mechanism of the GRPO algorithm enables the agent to efficiently explore and update in the policy space, thereby converging to the optimal policy more quickly.

[0055] 4) Rationality of the reward function: The multi-stage verification method is proposed, and the reward function constructed by the present application can accurately reflect the quality of the sampled sample. This reward estimation method based on multi-stage verification not only improves the accuracy of the reward value, but also provides more meaningful guidance information for the optimization of the agent.

[0056] 5) Wide applicability: The method proposed by the present application is applicable to various large model agents, whether based on supervised learning or reinforcement learning models, and can improve their tool usage capability through the method of the present application. This provides a general solution for tool usage of agents in different application scenarios, including the ability of embodied agent robots to perform actions and adapt to environmental changes.

[0057] 6) Deep exploration of unknown tools: based on the application of various large model agents, the application also discloses a more effective large model agent structure. The structure not only can fully explore unknown tools, change "passive sampling" to "active discovery", but also can use high reward samples to perform high confidence sample playback. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 Flowchart of the agent evolution method based on GRPO and multi-stage verification.

[0059] Figure 2 Structure diagram of Dual-Head Router network. DETAILED DESCRIPTION

[0060] In order to make the technical personnel in the art better understand the technical solutions in the embodiments of the present application, and make the purposes, features and advantages of the present application more apparent, more apparent and easier to understand, the technical cores in the present application will be further described in detail below. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0061] The agent evolution method based on GRPO and multi-stage verification of the application realizes the evolution of the tool use ability of the large language model agent by using the GRPO algorithm, and constructs an effective reward function by performing multi-stage verification on the sampling samples. As shown in Figure 1 , it mainly includes the following steps 1 to 4.

[0062] Step 1: initialize the policy parameters of the agent.

[0063] The application first initializes the policy parameters θ of the large model agent according to experience, so that the agent has the function of calling tools.

[0064] In one preferred embodiment, the application proposes a Dual-Head Router network on the basis of not destroying the Critic-free characteristics of GRPO and directly improving the characteristics and needs of "unknown function exploration, high reward reuse", as shown in Figure 2 , the network can quickly switch between "exploration vs. utilization" and explicitly leave "exploration gate" for unknown functions.

[0065] 1) Network structure.

[0066] A. The base strategy LLM remains unchanged, and 2 parallel multi-layer MLP networks (denoted as π old and π new ) are added at the last layer, and the outputs are denoted as L ExploitHead and L ExploreHead.π old token distribution L for outputting for known tool calls ExploitHead .π new token distribution L for outputting for unknown tool calls ExploreHead .

[0067] B. Add a routing fusion network router:

[0068] g = σ(W avg_pooling(input_embedding))

[0069] Where W is a learnable parameter vector, the embedding vector prompt_embed of the input task query is average-pooled, and the gating input g is obtained through the routing fusion network.

[0070] C. The final output of the policy network

[0071] p output = SoftMax(g L Explorehead +(1-g) L ExploitHead )

[0072] 2) Model training.

[0073] The fusion network is trained based on cross-entropy loss; wherein the label of the output g of the fusion network comes from tool call samples with total reward greater than a first set threshold w1, and if the tool call sample calls an unknown tool, the label is 1, otherwise the label is 0;

[0074] The base policy LLM and two parallel multi-layer MLP networks are trained based on tool call samples; wherein for unknown tool call samples, the network π old and the network π new are updated at the same time; for known tool call samples, the network π new is frozen and not updated, and the network π old is updated; the unknown tool call sample is a tool call sample with less than a second set threshold w2 of agent contacts, and the known tool call sample is a tool call sample with more than a second set threshold w2 of agent contacts.

[0075] Step 2: Pass the input task to the agent to sample tool call samples.

[0076] The present application delivers input tasks to an agent, samples from unlabeled tool usage data based on the agent's output, and generates a plurality of tool invocation samples. These samples include user task descriptions, agent's invocation attempts on tools (including new tools never seen before and old tools trained before), including invoked tool names, tool descriptions, and parameter values passed into the invoked tools, and corresponding tool invocation execution results.

[0077] Step 3: Multi-stage verification is performed on the tool invocation samples to obtain the total reward of the tool invocation samples.

[0078] The present application performs multi-stage verification on the sampled samples, including format verification, tool validity verification, and relevance verification, etc., to judge the pros and cons of the tool invocation in the samples, and estimate the corresponding reward value.

[0079] Specifically, the reward function construction module needs to perform the following three stages of verification on the sampled samples:

[0080] 1. Format verification.

[0081] Check whether the tool function invocation in the sampled sample meets the pre-defined format requirements. For example, check whether the function invocation contains the correct parameter types and quantities, whether it follows the invocation convention of the tool function, and whether the generated invocation parameters or tools are not in the given available tool set, etc.

[0082] Specifically, as shown in Table 1, the format verification reward r format is obtained by verifying the parameters passed into the invoked tool. If any of g1·g2·g3·g4 is 0, the format verification fails, and the format verification reward value is 0.

[0083]

[0084] Table 1

[0085] 2. Tool validity verification.

[0086] The present application checks the execution validity of the invoked tool by verifying whether the tool invocation meets the expected behavior of the tool. For example, unsuccessful execution will be filtered out, and the reasons for unsuccessful execution can include parameter type error, invalid parameter, runtime error, timeout, syntax error, missing parameter, etc.

[0087] Specifically, as shown in Table 2, the present application obtains the tool validity verification result r valid based on the tool invocation execution result. If any of v1·v2·v3 is 0, the format verification fails, and the tool validity reward value is 0.

[0088] Subitem Symbol Value Explanation Execution success <![CDATA[v2]]> {0,1} {0,1} Tool execution result is normal. No timeout in execution <![CDATA[v2]]> {0,1} Tool execution result is normal. Non-empty execution result <![CDATA[v3]]> {0,1} Tool execution result is non-empty

[0089] Table 2

[0090] 3. Correlation verification.

[0091] This invention formats and passes the results of successfully executed tool calls, the list of available tools, the generated tool calls, and the user's task description to another large model, or evaluates the consistency between function calls, execution results, and query targets by calculating text similarity.

[0092] ● "Task-tool call" pairs that produce meaningless results due to whether the task is infeasible or the calling parameters are incorrect.

[0093] ● Does the tool call align with the task objective and have appropriate parameters?

[0094] ● Whether the tool invocation and parameter selection are appropriately selected from the available toolset;

[0095] ● Whether the parameter selection for tool calls matches the user's task description.

[0096] ● Does the execution result match the purpose described in the user's task description?

[0097] The above verification evaluates the relevance between the results of the tool call and the original question. This step aims to ensure that the tool call is not only syntactically and logically correct, but also truly solves the user's problem. For example, if a user asks for recommendations for attractions in a city, check whether the function call's result provides information on attractions related to that city and whether it matches the user's interests and needs. If a function to retrieve weather information is called, check whether the returned result contains accurate weather data and whether it matches the location and time queried by the user.

[0098] Specifically, the evaluation assesses the correlation and consistency among the three elements: "task (query) - tool invocation - tool execution result," comprising three sub-items, each with a weight w. i Adjustable, but all validations must pass (otherwise, the overall validation is set to 0). Assume the task description is Q, and the tool description of the invoked tool is T. desc The description of parameter i of the called tool is denoted as The result of the tool execution is denoted as R. tool The BERT embedding of word w is denoted as E(w), and the specific calculation methods for each sub-item are as follows.

[0099] 1) Description T of the tool_call selected for invocation desc Consistent rewards with Task Q.

[0100] First, calculate the task Q and the tool description T.desc Relevance:

[0101]

[0102] where f(q, D) represents the number of times the word q appears in the document, |D| represents the number of words in the document, and b is a hyperparameter set to 0.2, which can be adjusted in conjunction with the threshold. Then calculate the selected tool call description T desc Consistency reward with task Q is as follows:

[0103]

[0104] where w3 is a third set threshold, and the value range is between 0 and 1.

[0105] 2) The parameter value P passed in the selected tool call tool_call i Consistency reward with task Q:

[0106]

[0107] where P represents all parameters in tool_call, δ(*) represents a value of 1 when the internal comparison is true, and w4 is a fourth set threshold, and the value range is between 0 and 1.

[0108] 3) The selected tool call execution result R tool Consistency reward with task Q

[0109]

[0110] where w5 is a fifth set threshold, and the value range is between 0 and 1.

[0111] Finally, the relevance verification reward:

[0112] r rel =c1·c2·c3.

[0113] In summary, the final reward value is obtained by using chain product on the multi-stage reward, as follows:

[0114] Total_Reward=r format ·r vaild ·r rel

[0115] Through the verification of the above three stages, the quality of the sampling sample can be accurately evaluated, and a reasonable reward function can be constructed accordingly.

[0116] The final reward is obtained by weighted summation of all the check items, and the reward value is 1.0 when all the check items pass, and the reward value is forced to be 0.0 once the check rule fails.

[0117] Step 4: updating the policy parameters of the agent according to the total reward to complete the evolution of the tool use ability of the agent.

[0118] The present application calculates the reward value of each sample based on the reward function, optimizes the parameters of the agent model using the GRPO algorithm, updates the parameters of the policy network model, and thus improves the performance of the agent in the tool use task.

[0119] Specifically, after constructing the reward function, the present application performs the above multi-stage verification process for each sample, calculates its corresponding reward value r, and updates the policy parameters of the agent according to the reward value r using the GRPO algorithm. The GRPO algorithm is a reinforcement learning algorithm based on relative policy optimization, which can effectively explore and update in the policy space.

[0120] In summary, the present application evolves the tool use ability of the large model agent based on the GRPO algorithm, which can effectively improve the adaptability and learning efficiency of the agent when facing new tools.

[0121] The present application proposes to perform format checking, tool effectiveness checking and relevance checking on the sampled samples, thereby constructing an effective reward function, which can accurately evaluate the quality of the samples and provide reliable information for the optimization of the agent.

[0122] The training and optimization of the tool use ability of the agent of the present application fully utilizes unlabeled data, reducing the dependence on large-scale labeled data sets.

[0123] The above-described embodiments only express the implementation of the present application, and the description is more specific, but it should not be understood as limiting the scope of the present patent. It should be noted that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the protection scope of the present patent should be subject to the appended claims.

Claims

1. An agent evolution method based on GRPO and multi-stage verification, characterized in that, The method comprises: initializing policy parameters of an agent; passing an input task to the agent to sample a tool call sample; wherein the tool call sample comprises: a user task description, an agent's tool call attempt, and a tool call execution result, the agent's tool call attempt comprising: a tool name, a tool description, and parameters passed to the tool; performing multi-stage verification on the tool call sample to obtain a total reward of the tool call sample; updating the policy parameters of the agent according to the total reward to complete the evolution of the tool usage capability of the agent.

2. The method of claim 1, wherein, The network structure of the agent consists of a base policy LLM, two parallel multi-layer MLP networks, and a fusion network. The two parallel multi-layer MLP networks are connected to the last layer of the base policy LLM and are respectively network π. old and network π new Network π old L is used to output the token distribution for known tool calls. ExploitHead , π new Token distribution L used for outputting unknown tool calls ExploreHead The output g of the fusion network is given by: g = σ(W·avg_pooling(input_embedding)), where input_embedding represents the embedding vector of the input task, σ represents the activation function, W represents the learnable parameter vector, and avg_pooling represents average pooling. The output p of the agent is given by: output =SoftMax(g·L) Explorehead +(1-g)·L ExploitHead ).

3. The method of claim 2, wherein, The process of training the agent comprises: training the fusion network based on cross-entropy loss; wherein the label of the output g of the fusion network comes from a tool call sample whose total reward is greater than a first set threshold w1, and if the tool call sample calls an unknown tool, the label is 1, otherwise the label is 0; The base policy LLM and two parallel multilayer MLP networks are trained based on tool call samples; where, for unknown tool call samples, the network π old and network π new Simultaneously update; for known tool call samples, freeze network π. new No updates will be made and the network π will be updated. old The unknown tool call samples are tool call samples whose number of interactions with the agent is less than the second set threshold w2, and the known tool call samples are tool call samples whose number of interactions with the agent is greater than the second set threshold w2.

4. The method of claim 1, wherein, performing multi-stage verification on the tool call sample to obtain a total reward of the tool call sample, comprising: The agent attempts to call the tool based on the call of the tool, and format checking of the tool call sample is performed to obtain a format reward r format ; The effectiveness of the tool call sample is verified based on the execution result of the tool call, to obtain an effectiveness reward r vaild ; Based on the user task description, the agent's tool calling attempt, and the tool calling execution result, the relevance of the tool calling sample is verified to obtain a relevance reward r rel ; According to the format reward r format , tool effectiveness reward r vaild , and relevance reward r rel , to obtain the total reward of the tool invocation sample.

5. The method of claim 4, wherein, Based on the agent's call to the tool, the format of the tool call sample is checked, and a format reward r is obtained format , comprising: judging whether the parameters passed to the tool contain all the required parameters of the tool to obtain a parameter integrity verification result g1; wherein the parameter integrity verification result g1 has a value of 0 or 1; judging whether the parameter types in the parameters passed to the tool are consistent with the parameter types corresponding to the required parameters of the tool to obtain a parameter type consistency verification result g2; wherein the parameter type consistency verification result g2 has a value of 0 or 1; judging whether the parameter value ranges in the parameters passed to the tool are consistent with the parameter value ranges corresponding to the required parameters of the tool to obtain a parameter value range consistency verification result g3; wherein the parameter value range consistency verification result g3 has a value of 0 or 1; judging whether the tool is in the optional tool list based on the tool name to obtain a tool call legality verification result g4; wherein the tool call legality verification result g4 has a value of 0 or 1; According to the parameter integrity verification result g1, the parameter type consistency verification result g2, the parameter value range consistency verification result g3, and the tool call legality verification result g4, a format reward r is obtained format .

6. The method of claim 3, wherein, Based on the user task description, the agent's tool calling attempt, and the tool calling execution result, the relevance of the tool calling sample is verified to obtain a relevance reward r rel , comprising: judging whether the tool call execution result is abnormal to obtain an effectiveness verification result v1; wherein the effectiveness verification result v1 has a value of 0 or 1; judging whether the tool executes timeout by the time of returning the tool call execution result to obtain an effectiveness verification result v2; wherein the effectiveness verification result v2 has a value of 0 or 1; obtaining an effectiveness verification result v3 by verifying whether the tool call execution result is empty; wherein the effectiveness verification result v3 has a value of 0 or 1; According to the validity verification result v1, the validity verification result v2 and the validity verification result v3, an effectiveness reward r is obtained vaild .

7. The method of claim 3, wherein, Based on the user task description, the agent's tool calling attempt, and the tool calling execution result, correlation verification is performed to obtain a correlation reward r rel , comprising: obtaining a correlation verification result c1 by calculating the consistency between the user task description and the tool description; wherein the correlation verification result c1 has a value of 0 or 1; obtaining a correlation verification result c2 by calculating the consistency between the user task description and the parameters passed to the tool; wherein the correlation verification result c2 has a value of 0 or 1; obtaining a correlation verification result c3 by calculating the consistency between the user task description and the tool call execution result; wherein the correlation verification result c3 has a value of 0 or 1; According to the correlation verification result c1, the correlation verification result c2, and the correlation verification result c3, a correlation reward r is obtained rel .

8. The method of claim 7, wherein, obtaining a correlation verification result c1 by calculating the consistency between the user task description and the tool description, comprising: Computing consistency between user task descriptions and tool descriptions where Q denotes a user task description, |Q| denotes the total number of words in the user task description, T desc denotes a tool description, |T desc | denotes the total number of words in the tool description, f(q i ,T desc ) denotes the number of times the word q i appears in the tool description T desc , b denotes a hyperparameter, and E(·) denotes a word embedding vector representation; The consistency R is compared with a third set threshold w3 to obtain a correlation verification result c1.

9. The method of claim 7, wherein, The correlation verification result where P represents parameters of incoming tools, δ(*) represents that an internal comparison true value is 1, Q represents a user task description, E(·) represents a word embedding vector representation, and w4 represents a fourth set threshold value. The correlation verification result wherein Q denotes a user task description, |Q| denotes a total number of words in the user task description, R tool denotes a tool invocation execution result, δ(*) denotes an internal comparison true value is 1, E(·) denotes a word embedding vector representation, and w5 denotes a fifth set threshold value.

10. An agent evolution system based on GRPO and multi-stage verification, characterized in that, The system comprises: An initialization module configured to initialize a policy parameter of an agent; A sampling module configured to pass an input task to the agent to sample a tool invocation sample, wherein the tool invocation sample comprises a user task description, an invocation attempt of the agent on a tool, and a tool invocation execution result, and the invocation attempt of the agent on the tool comprises a tool name, a tool description, and parameters passed to the tool; A verification module configured to perform multi-stage verification on the tool invocation sample to obtain a total reward of the tool invocation sample; An updating module configured to update the policy parameter of the agent according to the total reward to complete evolution of a tool usage capability of the agent.

Images