Rag tool dynamic enhancement method and system for multi-modal scene intelligent agent construction

Abstract

The application provides a multi-modal scene RAG tool dynamic enhancement intelligent agent building method and system, relates to the technical field of scene building, and comprises the following steps: tool retrieval is performed by generating a tool composite vector and constructing a situational awareness retrieval vector; the best tool is selected in combination with a tool dependency graph and a four-dimensional scoring mechanism; and finally, the tool vector is updated based on the execution result feedback. The application can adaptively select the optimal tool combination, improve the execution efficiency and accuracy of the intelligent agent in the dynamic task environment, and reduce resource consumption.

Description

TECHNICAL FIELD

[0001] The present application relates to scene building technology, and in particular to a RAG tool dynamic enhancement agent building method and system for a multi-modal scene. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, retrieval-augmented generation (RAG) has become an important core technology in agent systems. In a multi-modal scene, an agent needs to process various types of information such as text, images, and audio, and complete complex tasks through appropriate tools. Existing RAG tool systems mainly rely on a predefined tool set, and an agent selects appropriate tools for task processing according to user requests.

[0003] Traditional tool selection methods usually use keyword matching or simple similarity calculation to select tools, and the tool vector representation method is single, making it difficult to fully capture the multi-aspect characteristics of tools. In addition, the dependency relationship and synergistic effect between tools are often ignored in traditional systems, resulting in the inability to form an effective tool combination strategy.

[0004] The existing RAG tool selection mechanism lacks situational awareness and cannot dynamically adjust the tool selection strategy according to the specific scene and environmental constraints of the task, resulting in low tool selection efficiency in complex and variable scenarios and an inability to accurately meet the actual needs of users.

[0005] The existing technology has a single tool evaluation dimension, mainly relying on functional relevance for tool selection, ignoring multi-dimensional evaluation indicators such as tool performance consumption, execution efficiency, and value return, and unable to maximize the value of tool use in a limited resource situation.

[0006] The existing RAG system lacks a tool usage feedback mechanism and cannot dynamically update tool representation based on historical execution data, making it difficult for the system to learn and optimize from historical experience and unable to continuously improve the accuracy and efficiency of tool selection during use. SUMMARY

[0007] The embodiments of the present application provide a RAG tool dynamic enhancement agent building method and system for a multi-modal scene, which can solve the problems in the prior art.

[0008] In a first aspect of the embodiments of the present application, a RAG tool dynamic enhancement agent building method for a multi-modal scene is provided, comprising:

[0009] Generating a tool composite vector and storing it in a tool vector space for subsequent tool retrieval, receiving a task request of an Agent, and constructing a situational awareness retrieval vector based on the task request;

[0010] perform approximate nearest neighbor search in the tool vector space using the context-aware retrieval vector to obtain initial tools of a candidate tool set, perform graph traversal based on the initial tools in a tool dependency graph to obtain associated tools of the initial tools, perform rule filtering on the initial tools and the associated tools based on a current environment constraint, and update the candidate tool set;

[0011] determine a four-dimensional score weight according to a scene type of a current task, collect real-time running indicators of each tool in the candidate tool set and calculate a dimension score, perform weighted summation on the dimension score and the four-dimensional score weight to obtain a multi-dimensional utility score of each tool, and select a tool with the highest multi-dimensional utility score as a target tool;

[0012] convert the task request into a standardized calling request according to an input interface mode of the target tool, call the target tool using the standardized calling request and obtain an execution result, collect performance value consumption indicators of the target tool in an execution process, update a tool usage statistical vector of the target tool based on the execution result and the performance value consumption indicators, and use the updated tool usage statistical vector to generate a new tool composite vector.

[0013] performing approximate nearest neighbor search in the tool vector space using the context-aware retrieval vector to obtain initial tools of a candidate tool set includes:

[0014] constructing an initial retrieval feature according to a context-aware retrieval vector, mapping the context-aware retrieval vector to a tool vector space to establish a tool knowledge graph, performing knowledge enhancement on the context-aware retrieval vector based on the tool knowledge graph to obtain an enhanced retrieval vector;

[0015] constructing an approximate nearest neighbor retrieval index in the tool vector space, performing approximate nearest neighbor search on the enhanced retrieval vector based on the approximate nearest neighbor retrieval index, and calculating a similarity score between the enhanced retrieval vector and a tool vector;

[0016] sorting tool vectors according to the similarity score, and selecting tools corresponding to a preset number of tool vectors with the highest similarity scores as initial tools of a candidate tool set.

[0017] mapping the context-aware retrieval vector to a tool vector space to establish a tool knowledge graph, performing knowledge enhancement on the context-aware retrieval vector based on the tool knowledge graph to obtain an enhanced retrieval vector includes:

[0018] constructing a dynamic mapping matrix, the dynamic mapping matrix mapping a context-aware retrieval vector to a tool vector space to obtain a tool vector, obtaining a mapping difference between a reference tool vector and the tool vector to generate mapping update information, and superimposing the mapping update information on the dynamic mapping matrix to update the dynamic mapping matrix;

[0019] constructing a tool knowledge graph in the tool vector space, mapping and converting adjacent tool nodes in the tool knowledge graph using the dynamic mapping matrix, and multiplying the conversion result with associated information of the adjacent tool nodes to construct an inter-node association strength value in the tool knowledge graph;

[0020] performing knowledge enhancement on the context-aware retrieval vector, using the tool vector as a starting node, searching for an associated node vector in the tool knowledge graph using the dynamic mapping matrix, and using the association strength value as a weight coefficient, causing the weight coefficient to change with the association strength value, and fusing the weighted associated node vector and the context-aware retrieval vector to obtain an enhanced retrieval vector.

[0021] performing graph traversal in a tool dependency graph based on the initial tool to obtain associated tools of the initial tool, and performing rule filtering on the initial tool and the associated tools based on a current environment constraint, including:

[0022] performing graph traversal in a tool dependency graph using the initial tool as a starting node, calculating a node transition probability of each step of traversal, performing traversal along edges of the tool dependency graph based on the node transition probability, and grouping tool nodes encountered during traversal into an associated tool set;

[0023] establishing a rule template library, the rule template library storing a plurality of rule templates, each rule template containing rule structure information and combination mode information, inputting a combination of the initial tool and tools in the associated tool set in a historical execution record into each rule template, calculating an influence degree and a complexity score of each rule template on tool combination, and using a weighted sum of the influence degree and the complexity score as a rule importance weight;

[0024] sorting and screening rule templates in the rule template library according to the rule importance weight, selecting a rule template with the highest rule importance weight to form a rule set, inputting each combination of the initial tool and the tools in the associated tool set into the rule set for rule verification, and extracting a current environment constraint condition to perform rule filtering on the verification result.

[0025] converting the task request into a standardized calling request according to an input interface mode of the target tool, and using the standardized calling request to call the target tool and obtain an execution result, including:

[0026] A multimodal parsing engine is constructed, and the multimodal parsing engine is used to extract features from the task request to generate an initial feature vector. The initial feature vector is then input into the multimodal parsing engine again for deep feature extraction to obtain a request feature vector with hierarchical semantic representation.

[0027] Based on the input interface mode of the target tool, the request feature vector is subjected to semantic feature parsing. According to the multimodal parsing engine, the hierarchical semantic representation in the request feature vector is mapped to the standard semantic space to generate standardized semantic features containing normalized semantic information.

[0028] The standardized semantic information in the standardized semantic features is subjected to semantic consistency verification. The semantic similarity between the request feature vector and the standardized semantic features is calculated. When the semantic similarity is greater than a preset similarity threshold, the verified standardized semantic features are converted into a standardized call request according to the input interface mode of the target tool.

[0029] The standardized invocation request is passed to the target tool for invocation, and the execution result of the target tool is collected.

[0030] Based on the execution results and the performance value consumption metrics, update the tool usage statistics vector of the target tool, and use the updated tool usage statistics vector to generate a new tool composite vector, including:

[0031] The tool usage statistical vector of the target tool is progressively updated based on the execution results and performance value consumption indicators. The progressive update is achieved by weighting the original statistical vector with the current execution statistics. The weight of the weighted combination is determined by the historical information retention coefficient, which dynamically decays with the number of executions.

[0032] The tool generates an update vector based on the progressively updated statistical vector. The update vector is obtained by progressively fusing the statistical vector with environmental context information. The progressive fusion uses the same weighting coefficients as the progressive update. A new composite tool vector is generated based on the update vector.

[0033] A second aspect of this invention provides a system for building dynamic augmented agents using RAG tools in multimodal scenarios, comprising:

[0034] The first unit is used to generate a tool composite vector and store it in the tool vector space for subsequent tool retrieval. It receives task requests from the Agent and constructs a context-aware retrieval vector based on the task requests.

[0035] The second unit is used to perform an approximate nearest neighbor search in the tool vector space using the context-aware retrieval vector to obtain an initial tool for the candidate tool set, perform graph traversal in the tool dependency graph based on the initial tool to obtain the associated tools of the initial tool, filter the initial tool and the associated tools according to the current environmental constraints, and update the candidate tool set.

[0036] The third unit is used to determine the four-dimensional scoring weights based on the scenario type of the current task, collect the real-time operation indicators of each tool in the candidate tool set and calculate the scores of each dimension, and weight and sum the scores of each dimension with the four-dimensional scoring weights to obtain the multi-dimensional utility score of each tool, and select the tool with the highest multi-dimensional utility score as the target tool.

[0037] The fourth unit is used to convert the task request into a standardized call request according to the input interface mode of the target tool, use the standardized call request to call the target tool and obtain the execution result; collect the performance value consumption index of the target tool during the execution process, update the tool usage statistics vector of the target tool based on the execution result and the performance value consumption index, and use the updated tool usage statistics vector to generate a new tool composite vector.

[0038] A third aspect of the present invention provides an electronic device, comprising:

[0039] processor;

[0040] Memory used to store processor-executable instructions;

[0041] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0042] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0043] The beneficial effects of this application are as follows:

[0044] This invention utilizes a method for dynamically enhancing intelligent agents in multimodal scenarios using RAG tools. This method enables intelligent tool recommendation and usage, allowing for the dynamic selection of the most suitable tool based on the task scenario, thereby effectively improving the task execution efficiency of intelligent agents in complex multimodal environments.

[0045] This invention constructs a tool composite vector and a tool dependency graph, which integrates multi-dimensional information such as the semantic description, interface definition, usage statistics and technical characteristics of tools. Combined with context-aware retrieval and graph traversal technology, it can not only accurately match the tools required for the current task, but also discover potential related tools, significantly enhancing the accuracy and comprehensiveness of tool retrieval.

[0046] This invention designs a tool utility evaluation mechanism based on four-dimensional scoring, dynamically adjusts the scoring weights to adapt to different scenario requirements, and continuously optimizes the tool selection strategy by collecting tool operation indicators in real time. At the same time, it establishes a standardized tool calling interface, enabling the agent to seamlessly connect with various tools, which greatly improves the agent's adaptability and task processing performance in multimodal environments. Attached Figure Description

[0047] Figure 1 This is a flowchart illustrating the method for building a dynamic augmented agent using RAG tools in a multimodal scenario according to an embodiment of the present invention.

[0048] Figure 2 This is a flowchart of the tool rule verification and filtering process in an embodiment of the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0051] Figure 1 This is a flowchart illustrating the method for building a dynamically augmented agent using RAG tools in multimodal scenarios according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0052] Generate a composite vector of tools and store it in the tool vector space for subsequent tool retrieval; receive task requests from the Agent and construct a context-aware retrieval vector based on the task requests;

[0053] The context-aware retrieval vector is used to perform an approximate nearest neighbor search in the tool vector space to obtain an initial tool for the candidate tool set. Based on the initial tool, a graph traversal is performed in the tool dependency graph to obtain the associated tools of the initial tool. The initial tool and the associated tools are filtered according to rules based on the current environmental constraints to update the candidate tool set.

[0054] The four-dimensional scoring weights are determined based on the scenario type of the current task. Real-time operating indicators of each tool in the candidate tool set are collected and scores for each dimension are calculated. The scores for each dimension are weighted and summed with the four-dimensional scoring weights to obtain the multidimensional utility score of each tool. The tool with the highest multidimensional utility score is selected as the target tool.

[0055] The task request is converted into a standardized call request according to the input interface mode of the target tool. The standardized call request is used to call the target tool and obtain the execution result. The performance value consumption index of the target tool during the execution process is collected. The tool usage statistics vector of the target tool is updated based on the execution result and the performance value consumption index. The updated tool usage statistics vector is used to generate a new tool composite vector.

[0056] In one alternative implementation, the initial tools for obtaining the candidate tool set by performing an approximate nearest neighbor search in the tool vector space using the context-aware retrieval vector include:

[0057] Initial retrieval features are constructed based on context-aware retrieval vectors. The context-aware retrieval vectors are then mapped to the tool vector space to establish a tool knowledge graph. Based on the tool knowledge graph, knowledge enhancement is performed on the context-aware retrieval vectors to obtain enhanced retrieval vectors.

[0058] An approximate nearest neighbor retrieval index is constructed in the tool vector space. An approximate nearest neighbor search is performed on the enhanced retrieval vector based on the approximate nearest neighbor retrieval index, and the similarity score between the enhanced retrieval vector and the tool vector is calculated.

[0059] The tool vectors are sorted according to the similarity scores, and the tools corresponding to the preset number of tool vectors with the highest similarity scores are selected as the initial tools of the candidate tool set.

[0060] When constructing initial retrieval features based on context-aware retrieval vectors, the system receives a user query request: "I want to analyze recent sales data and generate a chart report." This text input is encoded using a pre-trained language model, such as an encoder with a 12-layer transformer structure, an input sequence length of 128, and a hidden layer dimension of 768. During encoding, the semantic representations of the key terms "analysis," "sales data," "chart," and "report" in the query text are considered, resulting in an initial vector representation. This vector representation is then mapped through a fully connected layer to a 512-dimensional initial retrieval feature vector, used for subsequent retrieval operations in the tool vector space.

[0061] In the process of mapping context-aware retrieval vectors to the tool vector space to build a tool knowledge graph, the system employs dimensional alignment technology. Assuming the tool vector space is a 384-dimensional space, the system transforms the 512-dimensional context-aware retrieval vectors into a 384-dimensional representation using a projection matrix. This projection matrix is ​​obtained through supervised learning of 20,000 query-tool matching samples. The mapped vectors serve as nodes in the graph, establishing connections with tool nodes in the tool vector space. The system adds attributes such as functional descriptions, parameter information, and usage scenarios to each tool node; for example, a data analysis tool node includes attributes such as "supports pivot tables" and "visualization chart types." Edges between nodes are established based on functional similarity and call dependencies, with edge weights reflecting the strength of the relationship and ranging from 0 to 1.

[0062] When enhancing context-aware retrieval vectors based on a tool knowledge graph, a graph attention network mechanism is used. In practical applications, the system starts from the initial retrieval vector and explores relevant nodes within two hops along the knowledge graph, collecting the representation vectors of these nodes. For the aforementioned query example, the system identifies three core tool nodes: "data analysis tools," "visualization components," and "report generators," and obtains their representation vectors. These relevant vectors are then weighted and aggregated, with weights calculated based on their semantic relevance to the initial retrieval vector. For example, "data analysis tools" has a weight of 0.6, "visualization components" has a weight of 0.3, and "report generators" has a weight of 0.1. By combining the initial retrieval vector with the weighted relevant node vectors, a richer enhanced retrieval vector is generated, while still maintaining a 384-dimensional representation.

[0063] When constructing the approximate nearest neighbor retrieval index in the tool vector space, a hierarchical navigation graph index structure is adopted. This index is built for 10,000 tool vectors and contains a three-level hierarchical structure. The top level contains approximately 50 anchor nodes, the middle level contains approximately 500 nodes, and the bottom level contains all tool nodes. The index supports efficient retrieval of large-scale vector sets by pre-calculating the distances between nodes and constructing adjacency relationships. After the index is built, it supports retrieval operations with sublinear time complexity, with an average query time of approximately 5 milliseconds in the 384-dimensional vector space.

[0064] When performing approximate nearest neighbor search on the enhanced retrieval vector based on the approximate nearest neighbor retrieval index, the system adopts a hierarchical navigation strategy. Starting from the top layer, the distance between the query vector and the anchor nodes is calculated, and the 5 closest anchor nodes are selected to enter the next layer of search. In the intermediate layer, for each entering node, the distance to the query vector is calculated again, and the 15 closest nodes are selected to enter the bottom layer of search. At the bottom layer, the system calculates the cosine similarity between the query vector and the candidate tool nodes, with a value ranging from [-1, 1], where a larger value indicates a higher similarity.

[0065] For the aforementioned query example, the system found several potential matching tools, such as "PivotAnalyzer" (similarity 0.92), "Business Intelligence Dashboard" (similarity 0.89), "Data Visualization Generator" (similarity 0.85), and "Report Automation Tool" (similarity 0.83). The system calculates the similarity score between all tool vectors and the enhanced search vectors and records it in the results list.

[0066] When sorting tool vectors based on similarity scores, the scores are arranged in descending order, and the top N tools with the highest similarity scores are used as the initial tools for the candidate tool set. In this example, N is set to 5, and the system selects "PivotData Analyzer," "Business Intelligence Dashboard," "Data Visualization Generator," "Report Automation Tool," and "Spreadsheet Analysis Plugin" (similarity 0.81) from the above calculation results as the initial tools for the candidate tool set. These tools serve as the basis for subsequent tool selection and combination, providing services for users' data analysis and report generation needs.

[0067] By implementing the above technical means, the system can intelligently recommend the most matching set of tools based on the user's natural language query, improving the accuracy and relevance of tool recommendations, and enabling the AI ​​assistant to respond to user needs more accurately.

[0068] In one optional implementation, the context-aware retrieval vector is mapped to a tool vector space to establish a tool knowledge graph. Based on the tool knowledge graph, the context-aware retrieval vector is augmented to obtain an enhanced retrieval vector, including:

[0069] A dynamic mapping matrix is ​​constructed, which maps the context-aware retrieval vector to the tool vector space to obtain the tool vector. The mapping difference between the reference tool vector and the tool vector is obtained to generate mapping update information, and the mapping update information is superimposed on the dynamic mapping matrix to update the dynamic mapping matrix.

[0070] A tool knowledge graph is constructed in the tool vector space. The dynamic mapping matrix is ​​used to map and transform adjacent tool nodes in the tool knowledge graph. The transformation result is multiplied with the association information of the adjacent tool nodes to construct the association strength value between nodes in the tool knowledge graph.

[0071] The context-aware retrieval vector is augmented with knowledge by using the tool vector as the starting node, searching for related node vectors in the tool knowledge graph using the dynamic mapping matrix, and using the association strength value as a weight coefficient, making the weight coefficient change with the association strength value. The weighted related node vector is then fused with the context-aware retrieval vector to obtain the augmented retrieval vector.

[0072] A dynamic mapping matrix is ​​constructed to map context-aware retrieval vectors to the tool vector space. Taking a practical application scenario as an example, suppose we have a context-aware retrieval vector representing a user's search for "how to handle attachments in emails". This context-aware retrieval vector can be represented as a multi-dimensional vector, such as a 256-dimensional vector. The tool vector space refers to the vector representation space of various tool functions, such as the vector representations of "download attachments", "preview attachments", and "forward attachments". The initial values ​​of the dynamic mapping matrix can be obtained through pre-training methods, such as training with a large amount of user query and tool usage data.

[0073] The system obtains the user's current context-aware retrieval vector and maps it to the tool vector space using a dynamic mapping matrix, resulting in a tool vector. Simultaneously, the system acquires a reference tool vector, which can be the optimal tool vector determined based on the user's historical behavior data. The system calculates the mapping difference between the tool vector and the reference tool vector, generating mapping update information. For example, if the cosine similarity between the mapped tool vector and the reference tool vector is lower than a preset threshold (e.g., 0.8), a mapping difference is considered to exist, and the dynamic mapping matrix needs to be updated. The mapping update information can be the difference vector between the two vectors, which is then superimposed onto the dynamic mapping matrix with a certain weight (e.g., 0.1) to update the dynamic mapping matrix. This update method allows the mapping matrix to dynamically adjust according to user behavior, improving mapping accuracy.

[0074] A tool knowledge graph is constructed in the tool vector space. This graph contains multiple tool nodes, each representing a tool function, and these nodes are associated with each other. For example, the tool nodes "Download Attachment" and "Preview Attachment" are associated because they are both related to attachment processing. The system uses a dynamic mapping matrix to transform adjacent tool nodes in the knowledge graph. The transformation result is multiplied by the association information of the adjacent tool nodes to obtain the association strength value between the nodes. The association information can be a predefined association type and strength. For example, the association type between "Download Attachment" and "Preview Attachment" is "Attachment Processing," and the initial association strength is 0.7. After the mapping transformation using the dynamic mapping matrix, if the transformation result has a high similarity to the vector representation of the target node, the association strength is increased; if the similarity is low, the association strength is decreased.

[0075] For a concrete example, suppose the tool vector for the "Download Attachment" node is V1, and the tool vector for the "Preview Attachment" node is V2, with an initial association strength of 0.7. V1 is mapped using a dynamic mapping matrix M, resulting in M(V1). The cosine similarity between M(V1) and V2 is calculated; assuming the result is 0.85, the new association strength value can be updated by taking a weighted average of 0.7 and 0.85, for example, resulting in an updated association strength value of 0.75. In this way, through the dynamic mapping matrix, the node association strength in the tool knowledge graph is dynamically adjusted based on actual usage.

[0076] The context-aware retrieval vectors are augmented with knowledge by using tool vectors as starting nodes to search for related node vectors within the tool knowledge graph. Continuing with the previous example, after a user query "how to handle attachments in emails" is mapped to a tool vector, assuming the best-matching tool node is "download attachments," the system uses the "download attachments" node as the starting node and searches for other related nodes, such as "preview attachments" and "forward attachments." The system uses a dynamic mapping matrix to search for related node vectors in the tool knowledge graph and uses the association strength value as a weighting coefficient. For example, if the association strength between "download attachments" and "preview attachments" is 0.75, then 0.75 will be used as the weighting coefficient when merging the "preview attachments" node vector.

[0077] The system weights and sums all associated node vectors according to their association strength to obtain a composite vector. Assuming the associated nodes for "Download Attachment" are "Preview Attachment" (association strength 0.75) and "Forward Attachment" (association strength 0.6), the composite vector can be represented as: 0.75 × V2 + 0.6 × V3 (where V2 is the vector representation of "Preview Attachment" and V3 is the vector representation of "Forward Attachment"). Then, the system fuses this composite vector with the original context-aware retrieval vector to obtain an enhanced retrieval vector. The fusion method can be a simple vector addition, weighted averaging, or a more complex method such as an attention mechanism.

[0078] Through the above steps, the context-aware retrieval vector is mapped to the tool vector space and enhanced based on the tool knowledge graph, resulting in an enhanced retrieval vector. This enhanced retrieval vector contains the original query information and relevant tool knowledge, which can be used in subsequent retrieval tasks to improve retrieval accuracy and relevance. In practical applications, this method can help the system better understand user queries, recommend more suitable tools and operations, and improve the user experience.

[0079] In one optional implementation, based on the initial tool, a graph traversal is performed in the tool dependency graph to obtain the associated tools of the initial tool, and rule filtering is performed on the initial tool and the associated tools based on the current environmental constraints, including:

[0080] The initial tool is used as the starting node to perform graph traversal in the tool dependency graph. The node transition probability of each traversal step is calculated. Based on the node transition probability, the traversal is performed along the edges of the tool dependency graph. The tool nodes encountered during the traversal are combined into an associated tool set.

[0081] Establish a rule template library, which stores multiple rule templates. Each rule template contains rule structure information and combination pattern information. Input the combination of the initial tool and the tools in the associated tool set in the historical execution record into each rule template. Calculate the degree of influence and complexity score of each rule template on the tool combination. Use the weighted sum of the degree of influence and the complexity score as the rule importance weight.

[0082] The rule templates in the rule template library are sorted and filtered according to the rule importance weight. The rule template with the highest rule importance weight is selected to form a rule set. Each tool combination in the initial tool and the associated tool set is input into the rule set for rule verification. The current environmental constraints are extracted and the verification results are filtered by rules.

[0083] like Figure 2 As shown, the method includes:

[0084] When traversing the tool dependency graph with the initial tool as the starting node, a random walk algorithm is used to calculate the node transition probabilities. For tool node A, the transition probability of its neighbor B is calculated as follows: the weight of the edge from A to B divided by the sum of the weights of all edges originating from A. For example, if tool A is connected to tools B, C, and D, with edge weights of 0.5, 0.3, and 0.2 respectively, then the transition probability from A to B is 0.5 / (0.5+0.3+0.2)=0.5, the transition probability from A to C is 0.3 / 1.0=0.3, and the transition probability from A to D is 0.2 / 1.0=0.2.

[0085] During the actual traversal, starting from the initial tool node, the next tool node to be visited is selected based on the node transition probability. To avoid getting stuck in a local loop, the random restart probability is set to 0.15, meaning there is a 0.15 probability of returning to the initial tool node to restart the traversal. The traversal depth is set to 5, meaning a maximum of 5 jumps. Each tool node encountered during the traversal is added to the associated tool set. To increase the diversity of the results, 10 random walks are performed, all encountered tool nodes are merged, and the top 15 tools with the highest access frequency are selected to form the final associated tool set.

[0086] Taking language translation tools as the initial tool, the associated tools obtained through graph traversal include: speech recognition, text editing, grammar checking, vocabulary expansion, content generation, semantic analysis, image recognition, multimedia processing, document format conversion, and other tools.

[0087] During the creation of the rule template library, the stored rule templates contain rule structure information and combination mode information. The rule structure information describes the logical structure of the rule, such as "If tool A is used, then tool B must / is prohibited from being used," or "When tool A and tool B are used together, parameter X must be set to Y." The combination mode information describes the usage mode of the tool combination, such as "serial execution," "parallel execution," or "conditional triggering."

[0088] Each rule template must contain at least the following fields: rule ID, rule description, rule condition, rule result, applicable tool type, combination mode, and priority. For example, a rule template is: {rule ID: "R001", rule description: "When text processing tools and language model tools are used in combination, the text length must be less than the system limit", rule condition: "Tool A type = text processing AND Tool B type = language model", rule result: "Check that the input text length <= system limit", applicable tool type: ["text processing", "language model"], combination mode: "serial execution", priority: "high"}.

[0089] When calculating the impact of rule templates on tool combinations, the combination of the initial tool and associated tools in historical execution records is analyzed. Specifically, indicators such as success rate, user satisfaction, and execution efficiency of tool combinations are extracted from historical records. The similarity between the current tool combination and historical records is calculated, and the impact is calculated based on a weighted average of the similarity. For example, if the initial tool is "language translation" and the associated tool is "grammar checker," and historical records show that the success rate of this tool combination is 95%, user satisfaction is 4.8 / 5, and execution efficiency is improved by 30%, then the impact is 0.95×0.4+4.8 / 5×0.3+0.3×0.3=0.766.

[0090] Complexity scores are calculated considering factors such as the complexity of the rule structure, the number of tools involved, and the number of parameters. Higher complexity means greater difficulty in practical application, so rules with lower complexity are preferred. For example, if a rule involves 2 tools and 3 parameters, and its structure is a simple conditional judgment, the complexity score is 0.3; if a rule involves 5 tools and 10 parameters, and its structure includes multiple levels of nested conditions, the complexity score is 0.8.

[0091] The importance weight of a rule is calculated by weighting its impact score and complexity score: Rule importance weight = Impact score × 0.7 - Complexity score × 0.3. For example, if the impact score is 0.766 and the complexity score is 0.3, then the rule importance weight is 0.766 × 0.7 - 0.3 × 0.3 = 0.446.

[0092] Rule templates are sorted and filtered according to their importance weights, and the top 5 rule templates with the highest weights are selected to form a rule set. Each tool combination in the initial tool set and the associated tool sets is input into the rule set for rule validation to check whether the tool combinations meet the rule conditions and the rule result requirements.

[0093] When extracting current environmental constraints to filter validation results using rules, environmental factors such as system resource limitations, user permissions, network conditions, and security requirements are considered. For example, in a low-bandwidth network environment, tool combinations requiring large amounts of data transmission are filtered out; on resource-constrained devices, computationally intensive tool combinations are filtered out; and for users with restricted permissions, tool combinations requiring advanced privileges are filtered out.

[0094] Taking the combination of language translation and speech recognition tools as an example, if the current environment is a mobile device and the network conditions are average, the rule verification results show that the performance of this combination drops by 50% under high network latency. The environmental constraints include "network latency < 100ms" and "device memory > 2GB". After checking the current network latency (120ms) and device memory (3GB), the tool combination will be filtered out because it does not meet the network latency requirements.

[0095] Through the above rule filtering process, the most suitable tool combination that meets the current environmental constraints can be selected from the associated tool set of the initial tool, thereby improving the applicability and execution efficiency of the tool combination.

[0096] In one optional implementation, converting the task request into a standardized invocation request based on the input interface mode of the target tool, and using the standardized invocation request to invoke the target tool and obtain the execution result includes:

[0097] A multimodal parsing engine is constructed, and the multimodal parsing engine is used to extract features from the task request to generate an initial feature vector. The initial feature vector is then input into the multimodal parsing engine again for deep feature extraction to obtain a request feature vector with hierarchical semantic representation.

[0098] Based on the input interface mode of the target tool, the request feature vector is subjected to semantic feature parsing. According to the multimodal parsing engine, the hierarchical semantic representation in the request feature vector is mapped to the standard semantic space to generate standardized semantic features containing normalized semantic information.

[0099] The standardized semantic information in the standardized semantic features is subjected to semantic consistency verification. The semantic similarity between the request feature vector and the standardized semantic features is calculated. When the semantic similarity is greater than a preset similarity threshold, the verified standardized semantic features are converted into a standardized call request according to the input interface mode of the target tool.

[0100] The standardized invocation request is passed to the target tool for invocation, and the execution result of the target tool is collected.

[0101] A multimodal parsing engine is constructed, employing a deep neural network architecture that includes a feature extraction layer and a semantic understanding layer. The feature extraction layer consists of multiple convolutional neural networks to process input information from different modalities, such as text, images, and audio. The semantic understanding layer uses a transformer structure to achieve cross-modal semantic fusion and understanding. When the system receives a task request from a user, the multimodal parsing engine extracts features from the request content, generating an initial feature vector. For example, when a user submits a request to "query sales figures for the past three months and generate a trend chart," the system first performs word segmentation on the text content, obtaining keyword elements such as "query," "past three months," "sales figures," "generate," and "trend chart," and then converts these elements into a 512-dimensional initial feature vector.

[0102] To obtain a deeper semantic representation, the initial feature vector is input again into the multimodal parsing engine for deep feature extraction. During this process, the parsing engine captures the correlations between different feature elements through an attention mechanism and combines this with a pre-trained language model for semantic enhancement, ultimately generating a 1024-dimensional request feature vector. This feature vector possesses hierarchical semantic representation capabilities, containing information such as the intent layer of the task request (e.g., a "query" operation), the object layer (e.g., "sales figures"), and the constraint layer (e.g., a "last three months" time range). In the example above, the system identifies the user's intent as data query and visualization, requiring the invocation of database query tools and chart generation tools.

[0103] The system performs semantic feature parsing on request feature vectors based on the input interface patterns of the target tools. A tool library is maintained, containing descriptions of the input interfaces, parameter requirements, and functional specifications for various tools. For the sales data query tool, the input interface requirements include data type parameters, time range parameters, and query condition parameters. The system utilizes a pre-built interface mapping model to map the hierarchical semantic representation in the request feature vector to a standard semantic space. The mapping process is implemented using a semantic matching algorithm, which calculates the cosine similarity between the request feature vector and the tool interface description features, selecting the interface pattern with the highest similarity as the mapping target. After mapping, the system generates standardized semantic features, containing normalized semantic information such as operation type "query", data type "sales amount", time range "last three months", and output format "trend chart".

[0104] The system performs semantic consistency verification on the normalized semantic information in the standardized semantic features, calculating the semantic similarity between the request feature vector and the standardized semantic features. The similarity calculation uses a vector space-based metric. In practical applications, when the semantic similarity is greater than 0.85 (a preset similarity threshold), the system considers the standardization process to have preserved the core semantics of the user's original request. For example, the original request "Query sales figures for the past three months and generate a trend chart" becomes "Get sales data for the past 90 days and generate a line chart" after standardization. The calculated semantic similarity is 0.92, exceeding the threshold, therefore the verification passes. After successful verification, the system converts the standardized semantic features into a standardized call request based on the target tool's input interface mode. For the sales data query tool, the standardized call request is formatted as JSON: {"action":"query", "data"} type ":"sales", "time range ":"90days", "output":"trend chart "}

[0105] The system transmits standardized call requests to the target tool, which then forwards the requests to the appropriate tool service via the API gateway. For the sales data query tool, a secure connection channel is established to transmit the standardized call requests, with a timeout of 3 seconds. Upon receiving the request, the target tool executes the corresponding operations, retrieving sales data and generating a trend chart. The system collects the execution results of the target tool's calls, including operation status codes, returned data, and error messages. During the collection process, the system continuously monitors the execution status using an asynchronous listening mechanism. When an execution completion signal is detected, the system obtains the execution results and performs standardized processing. For the sales data query example, the execution results obtained by the system include the average daily sales data for the past 90 days and the URL address of the generated trend chart.

[0106] Regarding error handling, when semantic similarity verification fails, the standardized mapping strategy is readjusted, and an alternative mapping model is used for a second conversion attempt. If three consecutive conversion attempts fail verification, the system will provide the user with a request ambiguity warning, requiring the user to clarify the intent of the request. Furthermore, the system has a built-in request conversion logging function, recording intermediate results and similarity calculation values ​​during each conversion process, facilitating later optimization of the conversion algorithm. Through these technologies, the system can efficiently and accurately convert users' multimodal task requests into standardized invocation requests recognizable by the target tool, achieving intelligent and automated tool invocation.

[0107] In one optional implementation, updating the tool usage statistics vector of the target tool based on the execution result and the performance value consumption index, and using the updated tool usage statistics vector to generate a new tool composite vector includes:

[0108] The tool usage statistical vector of the target tool is progressively updated based on the execution results and performance value consumption indicators. The progressive update is achieved by weighting the original statistical vector with the current execution statistics. The weight of the weighted combination is determined by the historical information retention coefficient, which dynamically decays with the number of executions.

[0109] The tool generates an update vector based on the progressively updated statistical vector. The update vector is obtained by progressively fusing the statistical vector with environmental context information. The progressive fusion uses the same weighting coefficients as the progressive update. A new composite tool vector is generated based on the update vector.

[0110] After receiving a user's task request, the system selects a suitable target tool from the tool library to execute the task. Upon completion, the system records the execution result and corresponding performance value metrics. The execution result can be a Boolean value (success or failure), a numerical score (such as an effect score between 0 and 1), or a composite metric (such as a multi-dimensional evaluation result). Performance value metrics include quantifiable cost indicators such as execution time, computational resource consumption, and number of API calls.

[0111] Based on this information, the tool usage statistics vector of the target tool is progressively updated. The tool usage statistics vector contains multiple dimensions, such as success rate, average execution time, and average resource consumption. The update process uses a weighted combination method: the original statistics vector is merged with the current execution statistics. For example, assuming the original tool statistics vector is [0.85, 3.2, 0.45] (representing a success rate of 85%, an average execution time of 3.2 seconds, and an average resource consumption of 0.45 units), the current execution result is [1.0, 2.8, 0.4], and the historical information retention coefficient is 0.9, then the updated statistics vector is [0.9×0.85+0.1×1.0, 0.9×3.2+0.1×2.8, 0.9×0.45+0.1×0.4], which is [0.865, 3.16, 0.445].

[0112] The historical data retention coefficient is a key parameter that determines the system's reliance on historical data. This coefficient dynamically decays with the number of executions, allowing the system to gradually adjust its focus on new data. Specifically, the decay can be set as follows: the initial historical data retention coefficient is 0.95, and after every 10 executions, the coefficient is multiplied by 0.99. In this way, the system retains more historical data in the early stages of tool use to build a stable statistical foundation, while gradually increasing its sensitivity to new data as usage increases.

[0113] After the incremental update is completed, an update vector is generated based on the updated tool using statistical vectors. This statistical vector is then fused with environmental context information, including user preferences, current task type, and system load status. For example, if the system identifies the current task as time-sensitive, it will increase the weight of execution time metrics during the fusion process; if it identifies it as a precision-sensitive task, it will increase the weight of accuracy metrics.

[0114] The specific fusion process also adopts a weighted combination method. Assuming the environment context vector is [0.7, 0.3, 0.9] (representing that the current environment attaches importance to success rate of 0.7, execution time of 0.3, and resource consumption of 0.9), and using the same weight coefficient of 0.9 as the incremental update, the update vector is calculated as [0.9×0.865+0.1×0.7,0.9×3.16+0.1×0.3, 0.9×0.445+0.1×0.9], that is, [0.8485, 2.874, 0.4905].

[0115] Based on the obtained update vector, a new tool composite vector is generated. This composite vector comprehensively expresses the current state and usability of a tool, directly influencing subsequent tool selection and task allocation decisions. During generation, the system normalizes the update vector to ensure comparability across dimensions. For example, execution time typically has larger values, while resource consumption typically has smaller values; normalization allows for comparison within the 0-1 range.

[0116] Normalization can be achieved using the min-max normalization method, calculating the corresponding metrics for all similar tools in the tool library. Assuming the execution time range for similar tools in the tool library is [1.5, 8.0], the execution time of 2.874 in the above update vector will be normalized to (2.874-1.5) / (8.0-1.5)=0.212. After performing similar operations on all dimensions, the final composite tool vector will be in the form [0.78, 0.212, 0.35].

[0117] The tool's composite vector is updated globally periodically (e.g., every 100 calls or every 24 hours) to reflect long-term usage trends and performance changes. In practical applications, if a tool exhibits continuous performance degradation on a specific task type (e.g., slower API response), the system will gradually reduce the selection probability of the corresponding task through a progressive update mechanism and trigger an alarm mechanism to prompt the system administrator to check the tool's status.

[0118] The above mechanisms ensure that the system can dynamically adjust its tool selection strategy based on actual tool usage, improving task execution efficiency and resource utilization. By dynamically adjusting the historical information retention coefficient, the system retains historical usage experience while responding promptly to changes in tool performance, achieving intelligent and adaptive tool management.

[0119] A second aspect of this invention provides a system for building dynamic augmented agents using RAG tools in multimodal scenarios, comprising:

[0120] The first unit is used to generate a tool composite vector and store it in the tool vector space for subsequent tool retrieval. It receives task requests from the Agent and constructs a context-aware retrieval vector based on the task requests.

[0121] The second unit is used to perform an approximate nearest neighbor search in the tool vector space using the context-aware retrieval vector to obtain an initial tool for the candidate tool set, perform graph traversal in the tool dependency graph based on the initial tool to obtain the associated tools of the initial tool, filter the initial tool and the associated tools according to the current environmental constraints, and update the candidate tool set.

[0122] The third unit is used to determine the four-dimensional scoring weights based on the scenario type of the current task, collect the real-time operation indicators of each tool in the candidate tool set and calculate the scores of each dimension, and weight and sum the scores of each dimension with the four-dimensional scoring weights to obtain the multi-dimensional utility score of each tool, and select the tool with the highest multi-dimensional utility score as the target tool.

[0123] The fourth unit is used to convert the task request into a standardized invocation request according to the input interface mode of the target tool, use the standardized invocation request to invoke the target tool and obtain the execution result; collect the performance value consumption index of the target tool during the execution process, update the tool usage statistics vector of the target tool based on the execution result and the performance value consumption index, and use the updated tool usage statistics vector to generate a new tool composite vector.

[0124] A third aspect of the present invention provides an electronic device, comprising:

[0125] processor;

[0126] Memory used to store processor-executable instructions;

[0127] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0128] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0129] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing dynamically augmented intelligent agents using RAG tools in multimodal scenarios, characterized in that, include: Generate a composite vector of tools and store it in the tool vector space for subsequent tool retrieval; receive task requests from the Agent and construct a context-aware retrieval vector based on the task requests; The context-aware retrieval vector is used to perform an approximate nearest neighbor search in the tool vector space to obtain an initial tool for the candidate tool set. Based on the initial tool, a graph traversal is performed in the tool dependency graph to obtain the associated tools of the initial tool. The initial tool and the associated tools are filtered according to rules based on the current environmental constraints to update the candidate tool set. The four-dimensional scoring weights are determined based on the scenario type of the current task. Real-time operating indicators of each tool in the candidate tool set are collected and scores for each dimension are calculated. The scores for each dimension are weighted and summed with the four-dimensional scoring weights to obtain the multidimensional utility score of each tool. The tool with the highest multidimensional utility score is selected as the target tool. The task request is converted into a standardized call request according to the input interface mode of the target tool. The standardized call request is used to call the target tool and obtain the execution result. The performance value consumption index of the target tool during the execution process is collected. The tool usage statistics vector of the target tool is updated based on the execution result and the performance value consumption index. The updated tool usage statistics vector is used to generate a new tool composite vector.

2. The method according to claim 1, characterized in that, The initial tools used to obtain the candidate tool set by performing an approximate nearest neighbor search in the tool vector space using the context-aware retrieval vector include: Initial retrieval features are constructed based on context-aware retrieval vectors. The context-aware retrieval vectors are then mapped to the tool vector space to establish a tool knowledge graph. Based on the tool knowledge graph, knowledge enhancement is performed on the context-aware retrieval vectors to obtain enhanced retrieval vectors. An approximate nearest neighbor retrieval index is constructed in the tool vector space. An approximate nearest neighbor search is performed on the enhanced retrieval vector based on the approximate nearest neighbor retrieval index, and the similarity score between the enhanced retrieval vector and the tool vector is calculated. The tool vectors are sorted according to the similarity scores, and the tools corresponding to the preset number of tool vectors with the highest similarity scores are selected as the initial tools of the candidate tool set.

3. The method according to claim 2, characterized in that, The context-aware retrieval vectors are mapped to the tool vector space to establish a tool knowledge graph. Based on the tool knowledge graph, the context-aware retrieval vectors are augmented to obtain augmented retrieval vectors, including: A dynamic mapping matrix is ​​constructed, which maps the context-aware retrieval vector to the tool vector space to obtain the tool vector. The mapping difference between the reference tool vector and the tool vector is obtained to generate mapping update information, and the mapping update information is superimposed on the dynamic mapping matrix to update the dynamic mapping matrix. A tool knowledge graph is constructed in the tool vector space. The dynamic mapping matrix is ​​used to map and transform adjacent tool nodes in the tool knowledge graph. The transformation result is multiplied with the association information of the adjacent tool nodes to construct the association strength value between nodes in the tool knowledge graph. The context-aware retrieval vector is augmented with knowledge by using the tool vector as the starting node, searching for related node vectors in the tool knowledge graph using the dynamic mapping matrix, and using the association strength value as a weight coefficient, making the weight coefficient change with the association strength value. The weighted related node vector is then fused with the context-aware retrieval vector to obtain the augmented retrieval vector.

4. The method according to claim 1, characterized in that, Based on the initial tool, a graph traversal is performed in the tool dependency graph to obtain the associated tools of the initial tool. Then, rule filtering is applied to the initial tool and the associated tools based on current environmental constraints, including: The initial tool is used as the starting node to perform graph traversal in the tool dependency graph. The node transition probability of each traversal step is calculated. Based on the node transition probability, the traversal is performed along the edges of the tool dependency graph. The tool nodes encountered during the traversal are combined into an associated tool set. Establish a rule template library, which stores multiple rule templates. Each rule template contains rule structure information and combination pattern information. Input the combination of the initial tool and the tools in the associated tool set in the historical execution record into each rule template. Calculate the degree of influence and complexity score of each rule template on the tool combination. Use the weighted sum of the degree of influence and the complexity score as the rule importance weight. The rule templates in the rule template library are sorted and filtered according to the rule importance weight. The rule template with the highest rule importance weight is selected to form a rule set. Each tool combination in the initial tool and the associated tool set is input into the rule set for rule verification. The current environmental constraints are extracted and the verification results are filtered by rules.

5. The method according to claim 1, characterized in that, The task request is converted into a standardized invocation request based on the input interface mode of the target tool. The standardized invocation request is then used to invoke the target tool and obtain the execution result, including: A multimodal parsing engine is constructed, and the multimodal parsing engine is used to extract features from the task request to generate an initial feature vector. The initial feature vector is then input into the multimodal parsing engine again for deep feature extraction to obtain a request feature vector with hierarchical semantic representation. Based on the input interface mode of the target tool, the request feature vector is subjected to semantic feature parsing. According to the multimodal parsing engine, the hierarchical semantic representation in the request feature vector is mapped to the standard semantic space to generate standardized semantic features containing normalized semantic information. The standardized semantic information in the standardized semantic features is subjected to semantic consistency verification. The semantic similarity between the request feature vector and the standardized semantic features is calculated. When the semantic similarity is greater than a preset similarity threshold, the verified standardized semantic features are converted into a standardized call request according to the input interface mode of the target tool. The standardized invocation request is passed to the target tool for invocation, and the execution result of the target tool is collected.

6. The method according to claim 1, characterized in that, Based on the execution results and the performance value consumption metrics, update the tool usage statistics vector of the target tool, and use the updated tool usage statistics vector to generate a new tool composite vector, including: The tool usage statistical vector of the target tool is progressively updated based on the execution results and performance value consumption indicators. The progressive update is achieved by weighting the original statistical vector with the current execution statistics. The weight of the weighted combination is determined by the historical information retention coefficient, which dynamically decays with the number of executions. The tool generates an update vector based on the progressively updated statistical vector. The update vector is obtained by progressively fusing the statistical vector with environmental context information. The progressive fusion uses the same weighting coefficients as the progressive update. A new composite tool vector is generated based on the update vector.

7. A dynamic augmented agent construction system for RAG tools in multimodal scenarios, used to implement the method of any one of claims 1-6, characterized in that, include: The first unit is used to generate a tool composite vector and store it in the tool vector space for subsequent tool retrieval. It receives task requests from the Agent and constructs a context-aware retrieval vector based on the task requests. The second unit is used to perform an approximate nearest neighbor search in the tool vector space using the context-aware retrieval vector to obtain an initial tool for the candidate tool set, perform graph traversal in the tool dependency graph based on the initial tool to obtain the associated tools of the initial tool, filter the initial tool and the associated tools according to the current environmental constraints, and update the candidate tool set. The third unit is used to determine the four-dimensional scoring weights based on the scenario type of the current task, collect the real-time operation indicators of each tool in the candidate tool set and calculate the scores of each dimension, and weight and sum the scores of each dimension with the four-dimensional scoring weights to obtain the multi-dimensional utility score of each tool, and select the tool with the highest multi-dimensional utility score as the target tool. The fourth unit is used to convert the task request into a standardized call request according to the input interface mode of the target tool, use the standardized call request to call the target tool and obtain the execution result; collect the performance value consumption index of the target tool during the execution process, update the tool usage statistics vector of the target tool based on the execution result and the performance value consumption index, and use the updated tool usage statistics vector to generate a new tool composite vector.

8. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 6.

Images