Tool agent system

Abstract

The invention discloses a tool agent system, which comprises an analysis module used for analyzing a task demand input by a user to generate a structured task summary and an evaluation target; the planning module is used for analyzing and generating a first to-be-executed subtask based on task summarization; the task generation module is used for judging the completion degree of the total task according to the execution score of the current sub-task and generating a subsequent sub-task when the total task is not completed; the execution module is used for analyzing the capability preference of the current subtask, matching the optimal performance tool from the tool library based on the preference weight to execute the current subtask, and obtaining an execution result; and the evaluation module is used for performing multi-dimensional quantitative evaluation on the execution result based on the evaluation target, and generating an execution score of the current subtask and text result evaluation. Effective tool selection is achieved, and tool selection and task execution errors caused by misjudgment of the tool capacity are fundamentally reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent agents, and more particularly to a tool intelligent agent system. BACKGROUND

[0002] With the rise of large models and intelligent agent technology, more and more scholars have begun to apply intelligent agent technology to various fields. Intelligent agents (Agents) can autonomously reason, decide and execute, and are naturally suitable for coping with diversified tasks and dynamic needs. Unlike traditional fixed rule generation models, intelligent agents can understand complex requirements and adaptively adjust generation strategies through continuous interaction with users. The introduction of intelligent agents makes the generation process no longer limited to a single data set and rules, but can be flexibly adjusted according to different situations and needs, thereby achieving more user-expected results output.

[0003] Current intelligent agent technology usually uses large language models (LLM / MLLM) as the reasoning engine. It receives user input, analyzes control intent, integrates task information through a pre-set reasoning process, and passes instructions to downstream tools to gradually advance task execution.

[0004] However, in current research work, within the intelligent agent framework, the multi-tool collaboration of intelligent agents cannot be successfully completed through the ability description of each tool. In an intelligent agent system, when the number of tools participating in the task is large, simple text description methods often cannot accurately reflect the actual ability of the tool, especially in the case where different tool functions are similar. For example, although text-to-image tools based on different generation models (such as SD3 and FLUX) may have highly similar text descriptions, such as "text-to-image tool", in actual task execution, even if the functions of the models are the same, the effects produced by different generation preferences may have significant differences. Lack of accurate ability description may lead to incorrect selection of tools in task execution, thereby affecting the generation effect. Therefore, how to accurately model the ability differences between tools and effectively distinguish the performance of different tools in task execution has become the key to improving task execution efficiency and task completion degree.

[0005] In research based on large language models, GPT, Claude and other models are good at understanding and planning, but still lack in specialized division of labor and precise execution. Therefore, researchers have developed multi-agent collaboration mechanisms, mainly including the following two types of paradigms: One category is the engineering paradigm based on fixed processes, which ensures the reliability of task execution and the quality of output by predefining roles and standardized operations. MetaGPT assigns fixed roles such as product manager and engineer to agents and forces the output of PRD, design documents and other standardized intermediate artifacts to form a structured pipeline. For example, the invention patent application with publication number CN120256065A and the title of cloud task scheduling method, device, equipment and medium based on intelligent agent, through the fixed calling sequence of "description generation → topology generation → load prediction → scheme generation", realizes the full-process automation of cloud resource scheduling.

[0006] Another category is the interactive paradigm based on dynamic dialogue, which does not set fixed processes, but dynamically generates solutions through dialogue between agents, suitable for exploratory tasks. AutoGen is represented by constructing a dialogueable agent pair (such as AssistantAgent and UserProxyAgent) to realize the closed loop of code automatic execution and debugging.

[0007] In addition, in the face of the rapid growth of tools, the invention patent application with publication number CN119623478A and the title of a tool calling semantic analysis method and system for low-resource data environment generates training data for tool logic expressions through context-free grammar, and uses a self-training mechanism to iteratively optimize the semantic analysis model, so that the model can accurately understand user semantics in a low-resource environment. The invention patent application with publication number CN120256065A and the title of cloud task scheduling method, device, equipment and medium based on intelligent agent proposes to sort the candidate tools dynamically according to the priority based on semantic analysis and relevance indicators, historical data such as calling frequency and success rate, to improve the accuracy of calling and resource efficiency. The invention patent application with publication number CN118296337A and the title of a large model tool calling precision evaluation method designs specific evaluation indicators for different application scenarios, and calculates the similarity between the predicted results and the real tool labels to generate tool calling evaluation information.

[0008] Although the above methods enhance task execution capability through division of labor, process or dynamic sequencing, current research mainly focuses on task planning algorithm and tool scheduling strategy of intelligent agent, focusing on improving the rationality of planning logic. Most of these works are based on the ideal assumption that "tool calling must be successful", lack of systematic evaluation of the actual execution effect of tools, and ignore the uncertainty of tool selection and its impact on the accuracy of final decision. Moreover, existing systems usually use general text description to define tool capability, which is difficult to accurately depict its real performance boundary. This problem is particularly evident in the field of visual content generation. Although existing systems (such as MetaGPT, GenArtist, etc.) have improved task execution effect through task decomposition and multi-model scheduling, their tool descriptions are still quite general and cannot clearly distinguish the specialties and applicable scenarios of different tools. For example, a common text-to-image tool description is "can generate images consistent with text semantics", which cannot reflect the performance differences between different models, nor can it support accurate tool matching by intelligent agents in complex tasks, thus bringing risks to overall planning and execution.

[0009] Therefore, it is urgent to build an intelligent agent tool calling mechanism that can accurately depict the actual capability boundary of tools, integrate execution feedback and support dynamic evaluation, in order to improve the decision accuracy and execution robustness in complex tasks. SUMMARY

[0010] In view of the above problems, the present application provides a tool intelligent agent system to overcome the above problems or at least partially solve the above problems.

[0011] In order to achieve the above purpose, the present application adopts the following technical solutions: The present application provides a tool intelligent agent system, comprising: an analysis module for analyzing the task requirements input by the user, generating a structured task summary and evaluation target; a planning module for analyzing and generating the first subtask to be executed based on the task summary, and for judging the completion degree of the total task according to the execution score of the current subtask, and generating subsequent subtasks when the total task is not completed; an execution module for analyzing the capability preference of the current subtask, matching the best performance tool from the tool library based on the preference weight to execute the current subtask, and obtaining the execution result; an evaluation module for multi-dimensional quantitative evaluation of the execution result based on the evaluation target, generating the execution score of the current subtask and the textual result evaluation.

[0012] Further, the analysis module comprises: The semantic parsing submodule is used to perform semantic parsing on the text commands input by the user; and when the user input includes an image, it extracts the semantic information of the image and performs joint semantic parsing on the image semantic information and the text commands. The task summary generation submodule is used to generate a structured task summary based on the semantic parsing results; The evaluation target generation submodule is used to generate corresponding evaluation targets based on the task summary using preset evaluation dimensions.

[0013] Furthermore, the planning module includes: The initial subtask generation submodule is used to receive the task summary and the tool library basic capability description document, and selectively combine the image semantic information to decompose the task summary to generate the first subtask to be executed.

[0014] Furthermore, the planning module also includes: The judgment submodule is used to compare the execution score of the current subtask with a preset score threshold. If the execution score of the current subtask is less than the preset score threshold and the current iteration round is less than the preset maximum number of iterations, then the total task is determined to be incomplete. The subsequent subtask generation submodule is used to predict and generate the next subtask when the total task is not completed, based on the current historical trajectory information, the task summary and the tool library basic capability description document, and selectively combined with the image semantic information, and input into the trained large language model. The current historical trajectory information includes the sub-tasks that have been executed in the current round and their corresponding execution scores.

[0015] Furthermore, the loss function of the large language model during training is expressed as:

[0016] Among them, L( θ () represents the loss function during training; θ Indicates model parameters; E t~u Expressing expectations; This indicates a summary of the task. Indicates the first t The optimal sample for the wheel; Indicates the first t The worst sample of the wheel; B represents the tool library's basic capability description document; σ Represents the sigmoid function; This represents the regularization hyperparameter; p ref Indicates predictions from a fixed initial model. p θ The reference probability; This indicates the planning module; h t-1 Indicates the first t Historical trajectory information for round -1; This indicates the total number of iterations of the task trajectory; s i It represents the semantic information of the image.

[0017] Furthermore, the execution module includes: The capability preference analysis submodule is used to assign performance-related preference weights to the current subtask based on preset capability boundary dimensions and the tool library's basic capability description document. The matching submodule is used to perform a dot product operation between the preference weights and the capability boundary matrix composed of the multi-dimensional capability boundary values ​​of each tool in the tool library, so as to obtain the performance matching degree of each tool when executing the current subtask. The tool selection submodule is used to sort the tool index according to the performance matching degree, obtain the theoretical execution effect ranking, and select the tool with the highest performance matching degree as the best performance tool to execute the current subtask. The task execution submodule is used to call the best performance tool to execute the current subtask and obtain the execution result.

[0018] Furthermore, the execution module also includes: The capability boundary matrix update submodule is used to select the top performers with the highest performance match from the tool library. m One tool, and randomly select from the remaining tools. n An additional tool; use the selected m + n Each tool performs the same subtask, and the execution results are compared to obtain a ranking of actual execution performance. The capability boundary matrix is ​​updated based on the difference between the actual execution performance ranking and the theoretical execution performance ranking.

[0019] Furthermore, the task execution submodule includes: The input condition confirmation unit is used to obtain the usage document information of the selected tool from the basic capability description document of the tool library; and determine the input conditions required for the selected tool to perform the current subtask based on the usage document information. The execution unit is used to call the selected tool to execute the current subtask according to the input conditions and obtain a structured execution result.

[0020] Furthermore, in the evaluation module, the execution score corresponding to the current subtask is represented as follows:

[0021] in,e t represents the first t wheel task by the execution score weighted in each dimension; i represents the evaluation weight of the first L L represents the total number of o t represents the execution result of the first t g i represents the evaluation target of the first i represents the evaluation module.

[0022] Further, the tool agent system is applied to a downstream task; the downstream task at least includes one of an image editing task, an image generation task and a visual understanding task.

[0023] Via the technical solution, compared with the prior art, the tool agent system has the following beneficial effects: The tool selection process based on the task capability preference and the tool capability boundary is realized, the tool selection and task execution errors caused by the tool capability misjudgment are fundamentally reduced, and the decision accuracy and execution robustness under complex tasks are improved.

[0024] The tool agent system has dynamic updating capability, the system can be self-optimized with the accumulation of tool use experience, has stronger adaptability, and solves the problem that the predefined tool capability boundary and the actual task capability demand and the actual tool capability do not match; the planning and execution are synergistically enhanced, the execution effect is fed back to the planning stage, and a continuous improvement decision closed-loop system is beneficially constructed. BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description only are the embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of the provided drawings.

[0026] Figure 1 The tool agent system reasoning framework provided in the embodiments of the present application is shown in the figure.

[0027] Figure 2 The reasoning framework in the image generation task provided in the embodiments of the present application is shown in the figure.

[0028] Figure 3 ​​​​A schematic diagram of task decomposition and execution in an image generation task provided in an embodiment of the present application.

[0029] Figure 4 A schematic diagram of an adaptive preference updating mechanism in an image generation task provided in an embodiment of the present application.

[0030] Figure 5 A schematic diagram of a planning optimization strategy for capability alignment in an image generation task provided in an embodiment of the present application.

[0031] Figure 6 A schematic diagram of a comparison of the visual effects of the present application and other existing methods in image generation and image editing tasks provided in an embodiment of the present application.

[0032] Figure 7 A schematic diagram of the visual effects of multiple examples of an image editing task provided in an embodiment of the present application.

[0033] Figure 8 A schematic diagram of the visual effects of multiple examples of an image generation task provided in an embodiment of the present application.

[0034] Figure 9 A schematic diagram of the visual effects of multiple examples of a custom image generation provided in an embodiment of the present application.

[0035] Figure 10 A schematic diagram of a visual understanding task reasoning process provided in an embodiment of the present application.

[0036] Figure 11 A schematic diagram of a time consumption comparison provided in an embodiment of the present application.

[0037] Figure 12 A schematic diagram of a token consumption comparison provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] The technical solutions in the embodiments of the present application will be described in detail below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0039] The embodiments of the present application disclose a tool intelligent agent system, as shown in Figure 1 The planning module and the execution module are inference engines of a large language model, and the analysis module and the evaluation module are inference engines of a multi-modal large model; wherein: The analysis module is configured to analyze the task requirements input by the user, generate a structured task summary, and generate evaluation targets. The planning module is used to analyze and generate the first subtask to be executed based on the task summary; and to determine the overall task completion rate based on the execution score of the current subtask, and generate subsequent subtasks if the overall task is not completed. The execution module is used to analyze the capability preferences of the current subtask, match the best-performing tool from the tool library based on the preference weights to execute the current subtask, and obtain the execution results; The evaluation module is used to perform multi-dimensional quantitative evaluation of the execution results based on the evaluation objectives, and generate the execution score of the current subtask and textual result evaluation.

[0040] The planning module, execution module, and evaluation module are connected in sequence to form a closed loop of iterative execution based on execution scores.

[0041] Next, each of the above modules will be explained in detail.

[0042] 1. Analysis Module : This analysis module includes a semantic parsing submodule, a task summary generation submodule, and an evaluation target generation submodule; wherein: (1) Semantic parsing submodule, which is used to directly perform semantic parsing on plain text instructions when the user input is plain text instructions; and to analyze the image and extract the semantic information of the image using a visual language model when the user inputs multimodal information combining text and image. s i ; to extract semantic information from images s i Joint semantic parsing with text instructions; (3) Task summary generation submodule, which is used to extract the core user intent through logical reasoning based on the semantic parsing results and generate a structured task summary. ; (4) Evaluation target generation submodule, used to summarize the task based on preset evaluation dimensions. Generate corresponding evaluation targets; taking image generation or image editing tasks as an example, the preset evaluation dimensions may specifically include the number of objects, object types, object attributes, spatial relationships, and global image semantics, etc.

[0043] 2. Planning Module : The planning module includes an initial subtask generation submodule, a judgment submodule, and a subsequent subtask generation submodule; wherein: (1) Initial subtask generation submodule, used to receive task summary and tool library basic capability description document B And selectively combine image semantic information. si , task summary , to generate the first subtask to be executed u 1; specifically, a basic text prompt template set by the initial subtask generation submodule , which is used for the planning module to infer the subtask generation by the large language model. For the received task summary , tool library basic capability description document B and image semantic information s i , if any, fill in the corresponding variable position in the basic text prompt template , and then obtain the text prompt for generating the first subtask of the task . Input into the large language model to infer and output the first subtask 1. u

[0044] The tool library basic capability description document B mentioned above is a text description of the basic capabilities of the tool library. For example, "image generation function can realize generation based on text input and generation based on reference image and text control together" or "image editing function can realize object addition, deletion, replacement, …".

[0045] Among them, the image semantic information s i is combined selectively, that is, when there is an image in the user's input, the initial subtask generation submodule needs to combine the image semantic information s i ; at this time, the first subtask to be executed u 1 is represented as:

[0046] (2) judgment submodule, used to compare the execution score of the current subtask e t with the preset score threshold. If the execution score of the current subtask is less than the preset score threshold, and the current iteration round is less than the preset maximum iteration number, it is determined that the total task is not completed. (3) subsequent subtask generation submodule, used to generate a subsequent subtask based on the current historical trajectory information h t , task summary and tool library basic capability description document B , and selectively combine the image semantic information s i ​, input into the trained large language model, predict the next subtask u t+1 ; denoted as:

[0047] wherein the image semantic information is selectively combined s i , that is, when there is an image in the user's input, the subsequent subtask generation submodule needs to combine image semantic information s i ; The above historical trajectory information h t includes the subtasks executed based on the current round and their corresponding execution scores, denoted as .

[0048] The next subtask generated by the planning module is continuously processed by the execution module and the evaluation module u t+1 to correct or further advance the historical subtasks until the execution result score reaches the preset threshold or the historical execution step reaches the maximum step, and output the task execution result with the best evaluation score o best ; the best task execution result o best is the final visual content generated according to the user demand of the user input.

[0049] The training steps of the above large language model include: 1) In each iteration, generate k candidate subtasks by the planner, denoted as ; 2) Process the k candidate subtasks by the execution module and the evaluation module respectively, obtain the execution result and execution score of each candidate subtask; wherein, represents the t th candidate subtask in the k th round; represents the execution result of the t th candidate subtask in the k th round; the execution score of the t th candidate subtask in the k th round; 3) The candidate subtask with the highest execution score is taken as the optimal sample , and the candidate subtask with the lowest execution score is taken as the worst sample The large language model is trained; the loss function in the training process is represented as:

[0050] Wherein, L( θ ) represents the loss function in the training process; θ represents the model parameters to be updated; E t~u represents the expectation; represents the task summary; represents the optimal sample of the t th round; represents the worst sample of the t th round; B represents the tool library basic ability description document; σ represents the sigmoid function; represents the regularization hyperparameter; p ref represents the reference probability from the fixed initial model prediction p θ ; represents the planning module; h t-1 represents the historical trajectory information of the t -1th round; represents the total number of iterations of the task trajectory; s i represents the image semantic information.

[0051] 3, execution module : The execution module includes a capability preference analysis submodule, a matching submodule, a tool selection submodule, a task execution submodule, and a capability boundary matrix update submodule; wherein: (1) The capability preference analysis submodule is used to analyze the performance dimension focused on when executing the current subtask u t ; Specifically, according to the preset capability boundary dimension D and the tool library basic ability description document B , the performance-related preference weight W task for the current subtask is allocated; Specifically, for the basic text prompt template set by the capability preference analysis submodule, it is used for the large language model in the execution module to generate the preference weight W task for reasoning. For the received capability boundary dimension D , the tool library basic ability description document B , and the current subtask u tFill it into the basic text prompt template The position of the corresponding variable is then used for the subtask. u t Generate preference weights W task Text prompts .Will Input into a large language model, and adjust the preference weights. W task Perform reasoning and output.

[0052] (2) Matching submodule, used to match preference weights W task The capability boundary matrix is ​​formed by the capability boundaries of each tool in the preset tool library. M p Perform a dot product operation to obtain the corresponding subtasks for each tool to execute. u t Performance matching degree S tool , represented as:

[0053] in, This indicates a normalization operation.

[0054] The above preference weights W task and capability boundary matrix M p The dimensions corresponding to both are consistent. Among them, the preference weight... W task It is presented in vector form, where each dimension is a preset capability boundary dimension. D Each value represents a subtask. u t correspond D The allocation of attention, for example, the higher the weight assigned to a certain dimension, the higher the performance requirement for that dimension; capability boundary matrix. M p Each row of values ​​represents the capability boundary information of each tool in the tool library, and each dimension of each row is a preset capability boundary dimension. D Its numerical value represents the tool's performance. D The performance of task execution under different dimensions is evaluated, and the capability boundary values ​​of each tool are obtained through the capability assessment system.

[0055] (3) Tool selection submodule, used to select based on performance matching degree S tool Sort the tool index to obtain the theoretical performance ranking, represented as:

[0056] Wherein R represents the tool index arranged in descending order of performance matching degree; represents an operation of arranging and outputting the corresponding index in descending order. Thus, the tool in the first position of R, i.e., the tool with the highest performance matching degree, is selected as the best performance tool for executing the current subtask u t Tool top1 ; (4) A task execution submodule for calling the selected best performance tool Tool top1 to execute the current subtask and obtain the execution result. The task execution submodule includes: u t 1) An input condition confirmation unit for obtaining the usage document information of the selected best performance tool from the tool library basic capability description document; and determining the input condition required by the selected tool for executing the current subtask according to the usage document information; specifically, for the selected best performance tool, the large language model in the execution module performs inference analysis based on the usage document information of the tool and the subtask to obtain the condition input of the best performance tool meeting the format requirement. The usage document information specifically includes tool name, tool description, tool input condition description and condition example, supplementary information, etc. Tool top1 Tool top1 u t c ; specifically, for the selected best performance tool, the large language model in the execution module performs inference analysis based on the usage document information of the tool and the subtask to obtain the condition input of the best performance tool meeting the format requirement. The usage document information specifically includes tool name, tool description, tool input condition description and condition example, supplementary information, etc. Tool top1 u t Tool top1 c . Wherein the usage document information specifically includes tool name, tool description, tool input condition description and condition example, supplementary information, etc.

[0057] 2) An execution unit for calling the selected tool to execute the current subtask according to the input condition, and obtaining the structured execution result; specifically, the system enters the tool execution stage, the system executes the execution code related to the tool, and takes the input condition as the condition input to obtain the corresponding formatted output, and the execution result output in each round is the local visual content of the current task. c Tool top1 c o t

[0058] (5) A capability boundary matrix updating submodule for: 1) selecting a tool with a performance matching degree​​​​​​​​​​​​S tool top m tools, while randomly selecting n additional tools from the remaining tools to increase the likelihood of selecting potentially high-performing tools; this m + n theoretical execution effect ranking R theory is represented as:

[0059] wherein, represents the operation of selecting the top m tools; represents the operation of randomly selecting n tools; l is the total number of tools in the tool library.

[0060] 2) using the selected m + n tools to perform the same subtask u t , and comparing the execution results to obtain the actual execution effect ranking; 3) updating the capability boundary matrix according to the difference between the actual execution effect ranking and the theoretical execution effect ranking; represented as:

[0061] wherein, represents the updated capability boundary matrix; represents the normalization operation; M p represents the capability boundary matrix before updating; W task represents the preference weight; η represents the update step; Δ represents the direction coefficient, which represents the difference between the theoretical execution effect ranking R theory and the actual execution effect ranking R actual ; wherein, the actual execution effect ranking R actual is obtained by comparing and evaluating the candidate outputs corresponding to multiple tools by the multi-modal large model; for example, in the case of an image generation or image editing task, multiple tool output images are input into the multi-modal large model at the same time, and the multi-modal large model is asked to judge which image is closest to the subtask execution effect, and give the actual execution effect ranking from best to worst; in the capability boundary matrix M p updating process, when the actual use ranking of a tool exceeds its theoretical ranking, its performance boundary score will be improved according to the weighted preference and the importance distribution of the specific task in each dimension; otherwise, the score will be reduced accordingly.

[0062] It should be noted here that, in the existing technology, the tool capability boundary matrix in the tool library... M p These boundaries may originate from benchmark tests on large datasets (i.e., benchmark evaluation scores in existing authoritative papers) or expert-like evaluations based on previous tool usage. These boundaries may be inaccurate due to differences in task-related dimensions or subjective bias. To improve the accuracy of tool performance boundary scores, this invention employs an adaptive preference update mechanism through the aforementioned capability boundary matrix update submodule, iteratively adjusting the tool capability boundary matrix based on actual tool usage. M p Numerical value.

[0063] For those lacking sufficient user experience or benchmark results (i.e., lacking high-quality data) M p Newly added tools (scores) can be initialized with their scores using the average performance boundary scores of similar tools in the current tool library on the corresponding dimension, to ensure that their potential is not overlooked in subsequent tool use and iterative updates.

[0064] 4. Evaluation Module : This evaluation module is used to perform multi-dimensional quantitative evaluation of the execution results based on the evaluation objectives, and generate an execution score for the current subtask; the execution score for the current subtask is represented as follows:

[0065] in, e t Indicates the first t The execution score of the wheel task is obtained by weighting various dimensions; Indicates the first i The evaluation weights of each dimension, and ,in, L Indicates shared ownership L One dimension, Indicates the first L The evaluation weights for each dimension can be set as follows during implementation: ; o t Indicates the first t The result of the round's execution; g i Indicates the first i Evaluation objectives in several dimensions; This indicates the evaluation module.

[0066] The current subtask u The performance scores for each dimension of the text are input into the LLM for summarization, and the output is specific to the current subtask.u t Evaluation summary.

[0067] The aforementioned tool-based intelligent agent system is applied to downstream tasks; these downstream tasks include at least one of image editing, image generation, and visual understanding tasks.

[0068] Next, taking the image generation task as an example, the tool intelligent agent system provided by the present invention will be described in detail.

[0069] I. Taking the text-to-image generation task as an example, the reasoning process is as follows: Figure 2 As shown: 1. Task Requirements Analysis: After the user submits the task request, the analysis module... It receives user input, which can be plain text commands or multimodal information combining text and images, such as... Figure 2 In the example, the user entered the text requirement "A puppy wearing sunglasses and a Santa hat runs along a country road in winter, surrounded by a Christmas atmosphere" and a picture of a corgi. Therefore, in this task, the user expects the puppy in the generated image to be the corgi shown in the picture.

[0070] Subsequently, the analysis module analyzes the user input. Extract semantic information from the image and process the semantic information. s i Semantic parsing of text commands is performed, and core user intent is extracted through logical reasoning to generate a structured task summary. .like Figure 2 In the example, the analysis module integrates user input into a task summary and image semantic information through a pre-defined thought process. The task summary is "Generate a corgi wearing sunglasses and a Santa hat running on a country road decorated with Christmas decorations," the image semantic information is "A corgi stands on a rock with cherry blossom trees in the background," and the structured evaluation objective is "{number:{{sunglasses:1}, {corgi:1}}, background: country road decorated with Christmas decorations, posture: running}."

[0071] Analysis Module By pre-setting evaluation dimensions, evaluation targets corresponding to the final task output results are generated. Figure 2 In the example, since the final image semantics of this image generation task only involve "a corgi," "sunglasses," "Santa hat," and "a country road with a Christmas atmosphere," the analysis module... The evaluation objective for the task is integrated into "{Number:{{Sunglasses:1}, {Corgi:1}}, Background: Country road decorated with Christmas, Posture: Running}". It should be noted that the dimensions involved in this evaluation objective differ for different image generation tasks.

[0072] 2. Task Planning: Planning module Receive from the analysis module Task Summary Image semantics s i (Image input is available if required by the user) and a description of the basic capabilities of the tool library. B By using their own experience and common sense, the task was broken down to obtain the sub-tasks that the first-round execution module could execute. u 1; is represented as:

[0073] like Figure 2 As shown in the example, the analysis module generates the first round of sub-tasks based on existing task summaries, input image semantics, and tool library basic capability description documents. u 1, which means "generating a dog on a snowy country road".

[0074] 3. Performance-driven tool selection: For subtasks from the analysis module u t Execution module According to subtasks u t This step involves analyzing the ability preferences described in the description, in order to analyze the performance of subtasks. u t The key performance dimensions to focus on at the time. Based on the preset capability boundary dimensions. D and tool library basic capability description document B For subtasks u t Assign performance-related preference weights: In terms of specific implementation, such as Figure 2 As shown in the example, when the execution module receives a subtask u t After obtaining relevant information, it proceeds according to the established reasoning chain and the sub-tasks. u t The required capabilities are assigned preference weights to the boundaries of each dimension. For example, in the subtask "Generate a dog on a snowy country road" shown in the figure, the execution module... After reasoning, it was determined that the task has the highest weight in terms of the number of objects, while the weights of other dimensions are relatively small. Therefore, the normalized weight vectors for each dimension are directly output.

[0075] Subtasks u t Corresponding preference weights W task The capability boundary matrix formed by the capability boundaries of each tool in the pre-defined tool library M p Perform a dot product operation to calculate the subtasks executed by each tool in the tool library. Performance matching degree S tool , represented as:

[0076] in, This indicates a normalization operation.

[0077] like Figure 2 In the example, the tool performance boundary matrix M p This represents the performance capability of all tools in the corresponding tool library across different dimensions; the higher the capability, the larger the value. It can be initialized using authoritative evaluation benchmarks obtained from large datasets (which can be obtained from existing task evaluation benchmark papers or evaluated by experts).

[0078] Based on performance matching S tool The numerical values ​​are sorted from largest to smallest according to their corresponding tool indices:

[0079] Where R represents the tool index sorted by performance matching degree; This indicates operations that sort by index from largest to smallest and output the corresponding indices. Therefore, the tool ranked first in R, i.e., the tool with the highest performance match, is selected to execute the current subtask. u t The best tool Tool top1 ; like Figure 2 In the example, based on the obtained performance matching degree Rank the values ​​from largest to smallest to find the optimal tool, FLUX.

[0080] 4. Tool Invocation and Execution: Execution module Based on the best tool chosen Tool top1 User documentation and subtasks u t Analyze and predict its best tools Tool top1 implement ut Required input conditions c .like Figure 2 In this example, FLUX is chosen as the optimal tool for subtask execution. Based on the FLUX user documentation, the execution module analyzes the subtasks. u 1. And give the input conditions for FLUX. c "A dog on a snowy country road."

[0081] The system then enters the tool execution phase, where it executes the relevant executable code for the tool FLUX, based on the input conditions. c As conditional input, to obtain the corresponding formatted output. o 1.

[0082] 5. Evaluation of sub-task execution effectiveness: This evaluation module Based on the analysis module The resulting evaluation objectives, for the first t Execution result of the round o t A multi-dimensional evaluation is conducted, and an execution score is given for each dimension. The execution scores for each dimension are weighted and summed using the following formula to obtain a comprehensive score.

[0083]

[0084] in, e t Indicates the first t The wheel task obtains an execution score (i.e., a comprehensive score) by weighting various dimensions. Indicates the first i The evaluation weights of each dimension, and ,in, L Indicates shared ownership L One dimension, Indicates the first L The evaluation weights for each dimension can be set as follows during implementation: ; o t Indicates the first t The result of the round's execution; g i Indicates the first i Evaluation objectives in several dimensions; This indicates the evaluation module.

[0085] The current subtask u The performance scores for each dimension of the text are input into the LLM for summarization, and the output is specific to the current subtask. u t Evaluation summary.

[0086] As Figure 2 an example, after analyzing the execution result o 1, its weighted score is 20 points, and a comprehensive evaluation summary is given: "lack of running posture, sunglasses, and Christmas atmosphere".

[0087] 6. Decision optimization and promotion: The system scores the execution result of each stage e t and compares it with the preset score threshold to determine whether the user task is completed in a quantitative numerical manner. If the score e t does not meet the threshold requirement, the historical sub-tasks and the corresponding execution result evaluation information will be used as historical trajectory information h t , task summary , input image semantic information s i (if there is user demand for image input), and tool library basic ability document B as input to the planning module to predict the next stage of task planning, correct the historical sub-tasks, or further promote them: the next sub-task u t+1 is represented as:

[0088] Steps 3-5 are repeated to execute all sub-tasks in a cyclic iterative manner until the execution result score reaches the preset threshold or the number of historical execution steps reaches the maximum number of steps, and the task execution result with the best evaluation score is output o best .

[0089] As Figure 2 an example, the intelligent agent performs step-by-step decomposition and execution of the task for the user's task, and after multiple execution steps, it iteratively processes the base image generated in the first round to gradually approach the output result of the user's intention.

[0090] II. Taking the text-to-image generation task as an example, the task decomposition and execution process is shown in Figure 3 : Similar to the Figure 2 embodiment, when the user inputs the creation requirement "create a hand-drawn illustration". A picture depicting a warm autumn scene: a field with 8 green cabbages growing on soil covered with fallen leaves. Not far away, a red bird is foraging. This picture should have a seasonal and life-like feel. At this time, the analysis module after task analysis and summary, immediately enters the task planning process.​

[0091] In the planning module During the planning process, the planning module Based on personal experience and common sense, the first task, "Generate 8 cabbages to grow in the soil," was summarized and output. In the performance-driven tool selection process, since the number of cabbages is more important for this task, a preference weight was assigned to the output. W task The proportion of numerical dimensions is higher, thus affecting the performance boundary matrix. M p During the weighted calculation, the Creati-Layout tool received the highest score.

[0092] The tool then proceeds to execution, using the optimal tool Creati-Layout to perform the subtask "generating 8 cabbages growing on soil," thus successfully generating the basic image, which contains eight cabbages arranged in two rows against the background of the land.

[0093] Since the output did not meet the end user's needs, the process of task planning, tool selection, tool execution, and result evaluation needed to be repeated cyclically. Through multiple rounds of execution, different tools were selected to execute different sub-tasks to ensure optimal execution results for each sub-task. Finally, after multiple rounds of execution, the expected result was output.

[0094] III. Taking the text-to-image generation task as an example, the execution process of its adaptive preference update mechanism is as follows: Figure 4 As shown: For the tool capability boundary matrix in the preset tool library M p The tool performance boundary scores may originate from benchmark tests on large datasets (i.e., benchmark evaluation scores in existing authoritative papers) or expert-like evaluations based on previous tool usage. These boundaries may be inaccurate due to differences in task-related dimensions or subjective bias. To improve the accuracy of tool performance boundary scores, this invention proposes an adaptive preference update mechanism that iteratively adjusts the tool capability boundary matrix based on actual tool usage. M p Numerical value.

[0095] by Figure 2 As an extension of the embodiments described above, in the first round of sub-tasks, the planning module... The subtask is "Generate a dog on a snowy country road".

[0096] During the tool selection process, an "exploration-exploitation" strategy is used to select tools from the tool library that offer the best performance match. S toolThe highest front m tools, while randomly selecting n additional tools from the remaining tools to increase the likelihood of selecting potentially high-performance tools:

[0097] wherein, represents the operation of selecting the best front m tools; represents the operation of randomly selecting n tools; l is the total number of tools in the tool library.

[0098] In the Figure 2 embodiments, according to the performance matching degree S tool An optimal tool, a suboptimal tool, and a random tool are selected, and their theoretical rankings should be “[1, 2, 3]”. The selected three tools are executed on the same subtask and the execution results are compared to obtain the actual execution effect ranking; the difference between the theoretical ranking and the actual ranking is calculated to update the capability boundary matrix M p Numerical value: represents:

[0099] wherein, represents the updated capability boundary matrix; represents the normalization operation; M p represents the capability boundary matrix before updating; W task represents the preference weight; η represents the update step; Δ represents the direction coefficient, which represents the difference between the theoretical execution effect ranking R theory and the actual execution effect ranking R actual ; wherein, the actual execution effect ranking R actual is obtained by comparing and evaluating the candidate outputs of the multiple tools by the multi-modal large model, for example, in the image generation or image editing task, the output images of the multiple tools are input into the multi-modal large model at the same time, and the multi-modal large model is used to judge which image is closest to the subtask execution effect, and gives the actual execution effect ranking from the best to the worst; in the capability boundary matrix M p updating process, when the actual use ranking of a tool exceeds its theoretical ranking, its performance boundary score will be improved according to the weighted preference and the importance distribution of the specific task in each dimension; otherwise, the score will be reduced accordingly.

[0100] In the Figure 2In the embodiment, it can be seen that the suboptimal tool is most suitable for the subtask in visual semantics, and the random tool is more consistent with the reference image effect in the appearance of the puppy than the optimal tool. Therefore, the actual multi-modal large model will judge that the actual execution effect ranking is "[3, 1, 2]". The difference operation between the obtained theoretical ranking "[1, 2, 3]" and the actual ranking "[3, 1, 2]" obtains the ability boundary matrix M p The direction coefficient of the ability boundary matrix will be adjusted and updated according to the weight distribution of the task preference W task M p .

[0101] Four, taking the text-to-image generation task as an example, the execution process of the planning optimization strategy of the ability alignment is as shown in Figure 5 Taking the embodiment in Figure 2 as an extension, in the first round of subtasks, the planning module gives the subtask "generate a dog on a country road in the snow". After the execution process, the corresponding execution result o t-1 .

[0102] In order to further enhance the step-by-step decision-making ability of the planning module , the present application extends the step-aware preference optimization (SPO) method of Liang et al., and proposes an ability alignment planning optimization strategy to realize the alignment of the decision-making process and the tool execution performance boundary.

[0103] For each execution stage t , the planning module generates k candidate subtasks by random sampling, denoted as . In order to ensure the diversity of the candidate subtasks, a proportion of β of the subtasks are sampled without using experience information, and the rest are sampled using experience information. The experience information comes from the historical task execution correct task graph of the agent.

[0104] Each subtask is executed to obtain the corresponding execution output , and the evaluation module evaluates the result to obtain the corresponding evaluation score .

[0105] The subtask with the highest evaluation score is selected as the optimal sample , and the subtask with the lowest score is the worst sample . This process is used to train the planning module​​ Provide preference samples to train the planning module. Output the optimal sample with a higher probability.

[0106] After collecting sufficient trajectory data, the large language model is trained using the following formula: The loss function during training is expressed as:

[0107] Among them, L( θ () represents the loss function during training; θ Indicates the model parameters that need to be updated; E t~u Expressing expectations; This indicates a summary of the task. Indicates the first t The optimal sample for the wheel; Indicates the first t The worst sample of the wheel; B represents the tool library's basic capability description document; σ Represents the sigmoid function; This represents the regularization hyperparameter; p ref Indicates predictions from a fixed initial model. p θ The reference probability; This indicates the planning module; h t-1 Indicates the first t Historical trajectory information for round -1; This represents the total number of iterations of the task trajectory; s i It represents the semantic information of the image.

[0108] Final Planning Module The weights of the trained large language model are loaded and used in the system's task planning process.

[0109] Figure 6 This is a diagram illustrating the visual comparison of the effects of the present invention with other existing methods in image generation and image editing tasks, wherein: For the input of "an astronaut cat wearing a spacesuit with a white tank, fishing on a small asteroid covered with gravel. A star-shaped bait hangs on the fishing line, and a blue space fish swims in the cosmic background with spiral galaxies and twinkling stars. Next to the cat is a small bucket filled with freshly caught glowing fish. A small UFO is watching from the top right corner", the image generation task. For the input of "a peaceful tennis court scene: a smooth wooden rocking chair with green eye glasses sits next to a green mesh fence. The court has a bright blue surface and clear white lines, with a single tennis ball on the left side", the image generation task. Compared with other existing methods, the structure generated by the present invention is more reasonable, with higher accuracy in entity category, number, and attribute. Other methods may have issues such as entity loss, position error, and attribute error.

[0110] For the input of "place a blue spoon to the right of the cookie, decorate a piece of mint leaf next to the stemless strawberry, and place the entire dessert on the marble countertop", the image editing task. For the input of "please change the picture to the following scene: a child holding a dragon-shaped kite walking through the grass. The child is wearing a deep red sweater. The mountains rise in the background, and a rainbow crosses the sky", the image editing task. Compared with other existing methods, the present invention is more accurate in editing and more consistent with the input instructions. Other methods may have issues such as editing non-compliance, step loss, and over-editing.

[0111] Figure 7 The visualization effect schematic diagram of multiple examples provided by the present invention for image editing tasks. The image editing tasks involved include: 1) modify the original image, replace the wooden plate with a polished marble plate, remove the middle fork, and redraw the chocolate cake to a lighter caramel color. 2) edit the original image, change the clothing to a denim jacket style, change the background to a concert stage scene, and update the guitar body to a bright green color. 3) modify the original image, make the fire hydrant rusty and old, change the woman's shirt color to soft blue, and set the scene in the background of urban scenery. 4) edit the original image, change the chocolate cake to blue, remove the background flag, and add a bouquet with falling colored paper. 5) edit the original image, redesign the fire hydrant as a star pattern, change the ground to a lush green grass, and let a white rabbit sit on it, and set the scene at sunset.

[0112] Figure 8The visualization effect schematic diagram of multiple samples in the image generation task provided by the present application. The image generation tasks involved include: 1) draw an image: a woman wearing a yellow hat and a dress sits on a bench in the garden, holding a basket of red roses in her hand. The whole scene should create a classical and romantic atmosphere. 2) draw a detailed close-up image, showing the texture and cracks of a simple and weathered wooden wall. The image should reflect a sense of reality and long-term use, and the slight water stains on the top indicate its age and the influence of the natural environment. 3) generate a quiet landscape painting. There should be a winding stone path in the picture, passing through the flowers in the foreground, leading to the cozy red-roofed house hidden among the trees, and in the distance is the hazy mountain range. An old man should be working on the nearby ridge. 4) generate a quiet and atmospheric modern bedroom scene. The composition should focus on the bedside table of the large window, with a dark blue backpack, a soft pink toothbrush, and a glowing smart phone on the table. The night scene is shown outside the window.

[0113] Figure 9 The visualization effect schematic diagram of multiple samples in the custom image generation provided by the present application. The tasks involved include: 1) the toy is searching for treasure under the sea; 2) generate a picture: a cat wearing a colorful hat is on the riverbank, there are flowers on the bank, and the water surface is rippling. 3) turn the cat into a red-haired tiger wearing a beret and roaring.

[0114] Figure 10 For a tool agent system applied to a visual understanding task embodiment, the inference process is similar to Figure 3 the image generation embodiment. The system is general for tasks and tools in any field. For visual understanding tasks, conventional large models are often limited by their strong knowledge priors, leading to misleading phenomena for simple tasks. For example Figure 10 as shown in the sample, a general large model often gives an answer of 10 fingers for a question about the number of fingers in a character, but in actual characters, especially in anime images, the character's fingers may not be 10. Therefore, a tool agent system proposed by the present application is used to count the number of fingers in a character, as shown in Figure 10 For the user input task "Please tell me how many fingers the character in the image has?", after understanding the input image and task information, the planning module reasons and splits the task. For task execution, the planning module gives the first sub-task as "detect the position of the character's hand in the image". Then the execution module performs a performance-driven tool selection process, and Figure 3Similarly, based on the subtask, the preference weight of the hand position detection task is given, and then multiplied by the tool performance boundary matrix to obtain the matching degree ranking of different detection tools. The best tool "Groundingdino" is selected to detect the hand position. The subsequent steps are to perform finger-based segmentation on the left or right hand to obtain the actual finger position and number. After multiple rounds of execution, the detected finger number is summarized, and the answer conforming to the understanding task is output. The above image generation, image editing and image understanding tasks can effectively prove that the tool agent system proposed in the application is applicable to various tasks in various fields, and has strong robustness and generalization ability.

[0115] In order to comprehensively verify the effectiveness of the tool agent system provided in the application, the embodiment of the application adopts the following comparative experiments.

[0116] I. Quantitative results and analysis: In the embodiment of the application, three different benchmarks, namely T2I-CompBench (Huang et al., 2023), OneIG-Bench (Chang et al., 2025) and Complex-Edit (Yang et al., 2025b), are adopted, which aims to objectively evaluate the visual reasoning performance in image generation and editing tasks from multiple angles.

[0117] 1. Basic image generation comparison: In the embodiment of the application, the application is compared with various image generation methods based on basic tasks, as shown in Table 1.

[0118] Table 1: Comparison of basic image generation of the application and various image generation methods

[0119] Among them, T2I-CompBench evaluates images from attribute binding and object relationship. From Table 1, it can be known that: (1) Traditional models such as FLUX and SD3 are still competitive in texture, non-spatial and complexity indicators, and their performance is similar to or superior to CoT (Chain-of-Thought) based methods (T2I-R1, GoT). (2) CoT-based methods rely on the fine-tuning of LLM (Large Language Model), which limits their task range; simple prompts may lead to overly complex explanations and inaccurate images. (3) Agent-based methods (GenArtist, T2I-Copilot) use self-correction to regenerate low-quality outputs, thereby improving reliability. (4) The application can match the most suitable model to adjust its ability according to different tasks, thereby achieving the best performance in all dimensions.

[0120] 2. Advanced image generation comparison: To further evaluate the effectiveness of the present application in visual reasoning, various methods were evaluated on OneIG-Bench under different scenes and complex text prompts. As shown in Table 2: Table 2: Advanced image generation comparison of the present application and various image generation methods

[0121] From Table 2, it can be seen that: (1) For more complex generation tasks, FLUX and SD3 perform significantly worse than methods that integrate LLMs in terms of inference metrics, indicating that integrating LLMs improves the ability to handle complex information. (2) Regarding alignment accuracy, GoT and GenArtist perform worse than other methods, indicating that a single large model has limited capacity when handling complex tasks. (3) T2I-Copilot and the present application both utilize agent collaboration mechanisms and can more accurately plan each step of visual reasoning when handling cross-domain information. (4) Due to tool set limitations, the present application does not show a significant advantage over other methods in alignment and text metrics. However, its performance-driven tool selection enables more intelligent planning, resulting in a significant advantage in reasoning.

[0122] II. Efficiency comparison: 1. Time consumption comparison: To verify the reasoning efficiency of the present application, QWen3-VL-32B (QWen3-VL, 2025) was used as the LLM of the present application, GenArtist, and T2I-Copilot in the embodiments of the present application. Reasoning was performed on the same data set as Table 1, and the time consumption of each round of task planning, tool selection, and image evaluation was recorded in the embodiments of the present application. As shown in Table 3: Figure 11 Compared with the baseline methods, the method in the embodiments of the present application shows significantly reduced time consumption in these three processes.

[0123] In particular, although the fixed tool set of T2I-Copilot minimizes its tool selection time, the detailed tool capability text description of GenArtist requires more reasoning time when the number of tools is high. In contrast, the present application achieves tool selection time much lower than GenArtist by analyzing subtasks and outputting capability matching preference weights.

[0124] 2. Word consumption comparison: To demonstrate the efficiency of the present application in tool selection, embodiments of the present application extend this problem by conducting large-scale tool library management in future agent communities. Specifically, embodiments of the present application use GPT-4o (Fanget al., 2025b) to simulate a large tool library with the number of tools ranging from 10 to 200 and generate tool information with textual descriptions and multi-dimensional scores. Embodiments of the present application use the“complex_vel”subset in T2I-CompBench for tasks and compare the performance-driven tool selection of the present application with traditional text-based methods with a maximum output token number of 8192. Embodiments of the present application compare the total token consumption (input and output) of the two methods. As shown in Figure 12 Figure 6, the traditional text-based method consumes more tokens because it is difficult to define tool capabilities, resulting in exponential growth in token consumption as the number of tools increases, without addressing the correctness of the selection. The present application focuses on task-specific dimensions and is therefore not affected by the number of tools. As the dimension increases (from d = 4 to d = 16), the token consumption of the present application increases slowly mainly in the input prompt. This indicates that the present application has superior efficiency in tool management and selection for future agents.

[0125] The various embodiments described in this specification are presented by way of example, and each embodiment describes a specific implementation of the present application. The embodiments are not intended to limit the scope of the present application, but rather to provide a description of the various embodiments of the present application. The skilled person will be able to implement or use the present application based on the above description of the disclosed embodiments. Various modifications to the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

[0126] The above description of the disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A tool agent system, characterized by, Comprise: An analysis module for parsing user input task requirements to generate a structured task summary and evaluation targets; A planning module for analyzing and generating a first subtask to be executed based on the task summary; And for judging the completion of the total task according to the execution score of the current subtask, and generating a subsequent subtask when the total task is not completed; An execution module for analyzing the ability preference of the current subtask, matching the best performance tool from the tool library based on the preference weight to execute the current subtask, and obtaining the execution result; An evaluation module for multi-dimensional quantitative evaluation of the execution result based on the evaluation target, generating the execution score of the current subtask and the textual result evaluation.

2. The tool agent system of claim 1, wherein, The analysis module comprises: A semantic analysis submodule for semantic analysis of user input text instructions; and when the user input contains images, extracting image semantic information and jointly analyzing the image semantic information and the text instructions; A task summary generation submodule for generating a structured task summary based on the semantic analysis result; An evaluation target generation submodule for generating corresponding evaluation targets based on the task summary through a preset evaluation dimension.

3. The tool agent system of claim 2, wherein, The planning module comprises: An initial subtask generation submodule for receiving the task summary and tool library basic capability description documents, and selectively combining the image semantic information to disassemble the task summary to generate the first subtask to be executed.

4. The tool agent system of claim 1, wherein, The planning module further comprises: A judgment submodule for comparing the execution score of the current subtask with a preset score threshold, if the execution score of the current subtask is less than the preset score threshold, and the current iteration round is less than the preset maximum iteration number, it is determined that the total task is not completed; A subsequent subtask generation submodule for inputting the current historical trajectory information, the task summary and the tool library basic capability description documents, and selectively combining the image semantic information into the trained large language model to predict the next subtask when the total task is not completed; Wherein, the current historical trajectory information comprises the subtasks executed based on the current round and the corresponding execution scores.

5. The tool agent system of claim 4, wherein, The loss function of the large language model in the training process is represented as: Among them, L( The execution module comprises: () represents the loss function during training; An ability preference analysis submodule for assigning performance-related preference weights for the current subtask according to the preset ability boundary dimension and the tool library basic capability description document; Indicates model parameters; E t~u Expressing expectations; This indicates a summary of the task. Indicates the first t The optimal sample for the wheel; Indicates the first t The worst sample of the wheel; B This document describes the basic capabilities of the tool library. A matching submodule for performing dot product operation on the preference weight and the ability boundary matrix composed of the multi-dimensional ability boundary values of each tool in the tool library to obtain the performance matching degree of each tool when executing the current subtask; Represents the sigmoid function; This represents the regularization hyperparameter; p ref Indicates predictions from a fixed initial model. p θ The reference probability; This indicates the planning module; h t-1 Indicates the first t Historical trajectory information for round -1; This indicates the total number of iterations of the task trajectory; s i It represents the semantic information of the image.

6. The tool agent system of claim 1, wherein, A tool selection submodule for sorting the tool index according to the performance matching degree to obtain the theoretical execution effect ranking, and selecting the tool with the highest performance matching degree as the best performance tool to execute the current subtask; A task execution submodule for calling the best performance tool to execute the current subtask and obtaining the execution result. The execution module further comprises: The task execution submodule comprises: ​ 7. The tool agent system of claim 6, wherein, ​ The capability boundary matrix updating submodule is configured to select the top m tools with the highest performance matching degrees from the tool library, and randomly select n additional tools from the remaining tools; use the selected m + n tools to perform the same subtask, compare the execution results, and obtain an actual execution effect ranking; and update the capability boundary matrix according to the difference between the actual execution effect ranking and the theoretical execution effect ranking.

8. The tool agent system of claim 6, wherein, ​ The input condition confirmation unit is configured to acquire usage document information of the selected tool from a tool library basic capability description document, and determine input conditions required by the selected tool to execute the current subtask according to the usage document information. The execution unit is configured to call the selected tool to execute the current subtask according to the input conditions, and obtain a structured execution result.

9. The tool agent system of claim 1, wherein, In the evaluation module, the execution score corresponding to the current subtask is expressed as: wherein, e t represents the first t wheel task by the execution score weighted in each dimension; represents the evaluation weight of the first i dimension; L represents a total of L dimensions; o t represents the execution result of the first t wheel; g i represents the evaluation target of the first i dimension; represents the evaluation module.

10. The tool agent system of claim 1, wherein, The tool intelligent agent system is applied to a downstream task, and the downstream task at least includes one of an image editing task, an image generation task and a visual understanding task.

Images