Intelligent agent element debugging method and device, equipment and storage medium

Abstract

The invention provides an agent element debugging method and device, equipment and a storage medium, and is applied to the technical field of artificial intelligence. The method comprises the steps that multiple sets of element combination data to be debugged are acquired; each group of element combination data comprises a group of cue words of the intelligent agent, test data and a data combination of a large language model, and different element combination data comprise at least one element data in the cue words, the test data and the large language model; debugging the intelligent agent in parallel according to the element combination data, and obtaining debugging results corresponding to the element combination data in parallel; performing structured analysis on the debugging result of each element combination data to obtain a structured analysis result; the structured analysis result is used for assisting a user in selecting a target cue word or a target large language model with the optimal performance for the agent from the element combination data. By adopting the technical scheme of the invention, the agent debugging efficiency can be improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and storage medium for debugging intelligent agent elements. Background Technology

[0002] With the rapid development of Large Language Models (LLMs), building AI (Artificial Intelligence) agents based on LLMs—capable of autonomous planning, using tools, and completing complex tasks—has become a hot topic in current artificial intelligence application development. In the development lifecycle of an AI agent, "debugging" is an extremely critical and time-consuming stage. A successful agent typically relies on the skillful coordination of three core elements: 1. High-quality prompt engineering: instructions to guide the model's actions; 2. Representative test data: input examples covering various edge cases; 3. A suitable foundation model: models from different manufacturers and with different parameter sets exhibit significant differences in inference capabilities and instruction-following abilities.

[0003] In related technologies, the debugging process of intelligent agents is usually carried out by controlling variables. For example, the model and test data are fixed, the prompt words are repeatedly modified and the prompt results are observed; or the prompt words are fixed and different models are switched for comparison; finally, a suitable model and prompt words are selected.

[0004] However, the above techniques suffer from low debugging efficiency. Summary of the Invention

[0005] This invention provides a method, apparatus, device, and storage medium for debugging intelligent agent elements, which addresses the shortcomings of low debugging efficiency in existing technologies and aims to improve the debugging efficiency of intelligent agents by performing joint debugging of multiple elements of the intelligent agent in parallel.

[0006] This invention provides a method for debugging intelligent agent elements, comprising: Acquire multiple sets of element combination data to be debugged; each set of element combination data includes a set of prompt words, test data and data combination of the large language model of the agent, and different element combination data include prompt words, test data and at least one element data of the large language model that are different; The agent is debugged in parallel based on the combination data of each element, and the debugging results corresponding to the combination data of each element are obtained in parallel. The debugging results of the combination data of each element are subjected to structured analysis to obtain structured analysis results; the above structured analysis results are used to assist users in selecting the target prompt words or target large language models with the best performance from the combination data of each element for the agent.

[0007] According to a method for debugging intelligent agent elements provided by the present invention, the acquisition of multiple sets of element combination data to be debugged includes: Acquire multiple prompt words, multiple test data sets, and multiple large language models; Obtain the prompts, test data, and large language models selected by the user from various prompts, test data, and large language models; The user-selected prompts, test data, and large language models are combined to obtain multiple sets of element combination data to be debugged.

[0008] According to a method for debugging intelligent agent elements provided by the present invention, the above-mentioned combination of user-selected prompt words, test data, and large language model is used to obtain multiple sets of element combination data to be debugged, including: Obtain the data combination method of user input; the above data combination method includes any one of the following: full permutation combination, random sampling combination, and specified combination; The user-selected prompts, test data, and large language models are combined according to the data combination method to obtain multiple sets of element combination data to be debugged.

[0009] According to the intelligent agent element debugging method provided by the present invention, each test data includes a test case and the expected output text corresponding to the test case, and the debugging result of each element combination data includes the model output text corresponding to the corresponding element combination data. The above-mentioned structured analysis of the debugging results of each element combination data to obtain structured analysis results includes: Based on the model output text and the corresponding expected output text of each element combination data, the comprehensive quantitative value corresponding to each element combination data is determined; the magnitude of the above comprehensive quantitative value is proportional to the performance of the agent when using the corresponding element combination data. The structured analysis results are determined based on the comprehensive quantitative values ​​corresponding to the combination of data for each element.

[0010] According to the intelligent agent element debugging method provided by the present invention, the debugging results of each element combination data also include the model running time and model running lexical information corresponding to the corresponding element combination data. The above-mentioned determination of the comprehensive quantization value corresponding to each element combination data based on the model output text and the corresponding expected output text of each element combination data includes: Based on the similarity between the model output text of each element combination data and the corresponding expected output text, determine the first quantitative value corresponding to each element combination data; Based on the model running time of each element combination data, determine the second quantitative value corresponding to each element combination data; Based on the lexical information consumed by the model running of each element combination data, the third quantification value corresponding to each element combination data is determined. Based on at least one of the first, second, and third quantitative values ​​of the combined data of each element, determine the comprehensive quantitative value corresponding to each combined data of the element.

[0011] According to a method for debugging intelligent agent elements provided by the present invention, the above-mentioned debugging results of the combination data of each element are subjected to structured analysis to obtain structured analysis results, including: Pivot tables are used to perform structured analysis on the debugging results of the combination data of each element to obtain structured analysis results. The structured analysis results include multidimensional analysis views corresponding to the combination data of each element. The first dimension of the multidimensional analysis view is the large language model element, and the second dimension of the multidimensional analysis view is the prompt word element. Each cell of the multidimensional analysis view represents the structured analysis results of the large language model and the combination data of the element under the prompt word in the corresponding dimension. The first dimension and the second dimension are different and are respectively the rows or columns of the multidimensional analysis view.

[0012] According to the present invention, an intelligent agent element debugging method is provided, wherein the structured analysis results include the structured analysis results of the combination data of each element, and the method further includes: Obtain the first prompt word input by the user, and based on the structured analysis results of different large language models corresponding to the first prompt word, determine the first target large language model with the best performance among the large language models included in the combination data of each element and recommend it to the user; Alternatively, obtain the second largest language model from the user input, and based on the structured analysis results of the different prompt words corresponding to the second largest language model, determine the second target prompt word with the best performance among the prompt words included in the combination data of each element and recommend it to the user.

[0013] The present invention also provides an intelligent agent element debugging device, comprising the following modules: The combined data acquisition module is used to acquire multiple sets of element combination data to be debugged; each set of element combination data includes a set of prompt words, test data and data combination of the large language model of the agent, and different element combination data include at least one element data in the prompt words, test data and large language model. The parallel debugging module is used to debug the agent in parallel based on the combination data of each element, and obtain the debugging results corresponding to the combination data of each element in parallel. The analysis module is used to perform structured analysis on the debugging results of the combination data of various elements to obtain structured analysis results. The above structured analysis results are used to assist users in selecting the target prompt words or target large language models with the best performance from the combination data of various elements for the agent.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the intelligent agent element debugging method as described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the intelligent agent element debugging method as described above.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the intelligent agent element debugging method as described above.

[0017] The present invention provides a method, apparatus, device, and storage medium for debugging intelligent agent elements. By acquiring multiple sets of element combination data to be debugged, the intelligent agent is debugged in parallel based on each element combination data. Debugging results corresponding to each element combination data are obtained in parallel, and the debugging results of each element combination data are subjected to structured analysis to obtain structured analysis results that assist users in selecting the optimal target prompt word or target large language model for the intelligent agent from the various element combination data. Each set of element combination data includes a set of prompt words, test data, and large language model data combinations for the intelligent agent. Different element combination data sets include at least one different element data set among the prompt words, test data, and large language model. In this method, because multiple element data sets of the intelligent agent can be combined together and jointly debugged through a parallel execution mechanism, the serial iteration process that originally required hours or even days can be shortened to minutes. Developers can verify dozens of "prompt word + test data + large language model" combination schemes at once, significantly improving the debugging efficiency of the intelligent agent. Simultaneously, by performing joint structured analysis on multiple debugging results from parallel debugging, developers can be assisted in making accurate decisions quickly. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is an architecture block diagram of the intelligent agent element debugging system provided by the present invention.

[0020] Figure 2 This is a flowchart illustrating the intelligent agent element debugging method provided by the present invention.

[0021] Figure 3 This is an interactive signaling diagram of the intelligent agent element debugging method provided by the present invention.

[0022] Figure 4 This is a schematic diagram of the device for debugging intelligent agent elements provided by the present invention.

[0023] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0025] Currently, the industry typically employs a serial, isolated debugging approach when developing intelligent agents, which presents the following significant problems: 1. Low debugging efficiency (serial bottleneck): Developers can usually only control variables for single-dimensional testing. For example, they fix the model and data, repeatedly modify the prompts and observe the results; or they fix the prompts and switch between different models for comparison. This serial iterative approach of "modify one -> test once -> modify the next" greatly limits development speed. 2. Difficulty in identifying multi-variable coupling problems (lack of collaborative perspective): There are complex coupling relationships between prompts, data, and models. A seemingly perfect prompt may become completely ineffective after changing the model; a specific edge test data may only trigger the model's illusion under a specific prompt structure. Existing tools often focus on single-dimensional management (such as simple prompt management tools or model evaluation platforms), lacking the ability to place all three on the same plane for joint analysis, making it difficult for developers to pinpoint the root cause of the problem: is the prompt poorly written, the data too special, or the model itself insufficient? 3. Difficulty in comparing and analyzing results: After multiple rounds and multi-dimensional modifications, developers face a large number of discrete running result logs, making it difficult to intuitively compare the advantages and disadvantages of different combinations of solutions. This is prone to errors and consumes a lot of energy.

[0026] Based on this, embodiments of the present invention provide a method, apparatus, device, and storage medium for debugging intelligent agent elements, which can solve the above-mentioned technical problems. Specifically, it can simultaneously process three key variables / elements: prompt words, test data, and models, to achieve parallel execution and collaborative analysis, thereby improving the efficiency of intelligent agent debugging and the efficiency and accuracy of developer decision-making.

[0027] It should be noted that the execution subject of the embodiments of the present invention can be an intelligent agent element debugging device, an electronic device, an intelligent agent element debugging system, or other devices, equipment, or systems, etc. The following embodiments will use an intelligent agent element debugging system as an example for illustration. First, the architecture of the intelligent agent debugging system will be described, see [link to documentation]. Figure 1 The diagram shows the architecture of an intelligent agent element debugging system. This system includes a developer, a variable configuration module, a collaborative execution engine (core component), an external large model API (application programming interface), and a result aggregation and analysis module / platform. The developer can input information such as prompt word sets, test datasets, and large language model sets into the system through the variable configuration module. The variable configuration module can then transmit the user input to the collaborative execution engine. The collaborative execution engine, as the core component / core processing unit, typically runs on the server side. It can combine intelligent agent element data and perform parallel debugging by calling the external large model API in parallel. The debugging results are then sent to the result aggregation and analysis module. After performing structured analysis, the result aggregation and analysis module transmits / displays the structured analysis results to the developer for decision-making.

[0028] Figure 2 This is a flowchart illustrating the intelligent agent element debugging method provided by the present invention, as shown below. Figure 2 As shown, the method includes the following steps: Step 202: Obtain multiple sets of element combination data to be debugged; each set of element combination data includes a set of prompt words, test data and data combination of the large language model of the agent. Different element combination data include at least one element data in the prompt words, test data and large language model that is different.

[0029] The intelligent agent consists of three elements: prompt words, test data, and a large language model. The variable configuration module can collect multiple prompt words, multiple test data, and multiple large language models that the user needs to debug the intelligent agent in advance, and combine each test data with the prompt words and large language models it needs to use to obtain multiple sets of element combination data to be debugged.

[0030] The aforementioned multiple prompts can include multiple different versions of the same type, such as V1_Detailed Instruction Version and V2_Detailed Instruction Version, or multiple different versions of different types of prompts, such as V1_Detailed Instruction Version and V2_Mind Chain Version. These multiple prompts can cover various prompts required by developers in their business applications, such as text recognition / reasoning / inductive summarization or text recognition on images.

[0031] The aforementioned test data can include actual test data or historical test data, covering various test data used by developers in their business operations. Each test data set can include a test case (Input Query) and the corresponding expected output text (Ground Truth). The test case can include text, images, etc., from text templates used during the test, or it can include text content collected during the test. The expected output text refers to the text that the large language model should theoretically output based on the test case and prompts. For example, suppose two typical data sets are entered: "Normal Refund Request" and "Malicious Claim Request".

[0032] The aforementioned large language models can include different types of large language models, such as OpenAI, DeepSeek, Qwen3, etc. For the multiple large language models collected above, the model parameters of these multiple large language models can be collected directly to construct the corresponding large language model, or the configuration / calling parameters of these multiple large language models can be collected to call the corresponding large language model.

[0033] The collaborative execution engine can obtain multiple prompt words, multiple test data, and multiple large language models that the user needs to debug the agent through the variable configuration module. Then, it can combine every three elements in these prompt words, test data, and large language models to obtain multiple sets of element combination data to be debugged. For example, the prompt word V1, test data 1, and large language model A are combined to form the first set of element combination data, and the prompt word V2, test data 1, and large language model A are combined to form the second set of element combination data. This process can be repeated to construct many sets of element combination data to be debugged.

[0034] It is understandable that each set of element combination data here includes three elements: prompt words, test data, and large language model. Moreover, at least one element in different element combination data is different. For example, two different element combination data may include different prompt words and test data, but the same large language model.

[0035] Step 204: Debug the agent in parallel based on the combination data of each element, and obtain the debugging results corresponding to the combination data of each element in parallel.

[0036] In this step, after obtaining the combined data of various elements, the collaborative execution engine can generate parallel tasks from this combined data. Each task corresponds to a specific triple instance of "prompt word version + test data (or test data sample) + large language model (or model configuration)". Each task can then be distributed in parallel to the large language model included in the task for execution. Alternatively, the corresponding external large language model API can be called concurrently according to the configuration of the large language model included in each task, so that the corresponding task instances can be executed in parallel through each external large language model. Afterwards, the model output text of the corresponding large language model and the relevant execution information of the large language model after executing the corresponding task can be obtained in parallel. The model output text and relevant execution information of the large language model for each task are used as the debugging result for that task, i.e., the debugging result corresponding to each combination of element data is obtained.

[0037] Optionally, the execution information of the large language model for each of the above-mentioned element combination data may include: the model running time corresponding to the element combination data, the lexical information consumed by the model running, the estimated cost, and whether the model running has errors.

[0038] Alternatively, the concurrency performance of the collaborative execution engine in the agent element debugging system can be used to obtain the number of tasks processed concurrently each time. Then, according to this number of tasks, a corresponding number of task instances can be obtained for concurrent / parallel execution each time, and the debugging results of each task instance can be obtained in parallel. By combining the concurrency performance of the collaborative execution engine for concurrent debugging, the successful implementation of parallel debugging can be effectively guaranteed.

[0039] Step 206: Perform structured analysis on the debugging results of the combination data of each element to obtain the structured analysis results; the above structured analysis results are used to assist the user in selecting the target prompt word or target large language model with the best performance from the combination data of each element for the agent.

[0040] In this step, the system can collect the debugging / running results of all task instances. After obtaining the debugging results corresponding to the combination of each element data, it can directly perform joint structured analysis on the debugging results of the combination of each element data. For example, it can display them side by side in a unified view to help users intuitively compare or identify the best combination of variables / elements or discover defects under a specific combination of elements. Alternatively, it can arrange the debugging results of the combination of each element data according to a certain element and then analyze them to obtain structured analysis results. Or, it can perform preliminary structured analysis on the debugging results of the combination of each element data (for example, quantitatively evaluate the running performance of the combination of each element data through the debugging results of the combination of each element data) and then perform further joint structured analysis to obtain the final structured analysis results.

[0041] Furthermore, optionally, the structured analysis results may include a multi-dimensional analysis view. After obtaining the structured analysis results, this multi-dimensional analysis view can be displayed to the user on the front-end interface, allowing the user to quickly and accurately select the optimal target prompt word or the optimal target large language model for the agent from the combined data of various elements. After selecting the optimal target prompt word or the optimal target large language model for the agent, the user can use this optimal target prompt word or the optimal target large language model to perform reasoning / summarizing / text recognition and other processing on the data in the actual application environment, obtaining the output results under the target prompt word or the output results under the target large language model. The output results at this time are also the most accurate and reasonable results, thereby effectively improving the user's work efficiency and accuracy.

[0042] In this embodiment, multiple sets of element combination data to be debugged are acquired, and the agent is debugged in parallel based on each element combination data. Debugging results corresponding to each element combination data are obtained in parallel, and structured analysis is performed on the debugging results of each element combination data to obtain structured analysis results that assist users in selecting the optimal target prompt word or target large language model for the agent from each element combination data. Each set of element combination data includes a set of prompt words, test data, and large language model data combinations for the agent. Different element combination data sets include at least one different element data set among the prompt words, test data, and large language model. This method combines multiple element data sets of the agent and performs joint debugging of these sets through a parallel execution mechanism. This reduces the original serial iteration process, which would take hours or even days, to minutes. Developers can verify dozens of "prompt words + test data + large language model" combinations at once, significantly improving the debugging efficiency of the agent. Furthermore, joint structured analysis of multiple debugging results from parallel debugging assists developers in making quick and accurate decisions.

[0043] The above embodiments briefly illustrate the process of obtaining multiple combinations of element data to be debugged. The following embodiments will explain the specific implementation process of obtaining multiple combinations of element data to be debugged.

[0044] In one embodiment, obtaining multiple sets of element combination data to be debugged in step 202 above may include: Acquire multiple prompt words, multiple test data sets, and multiple large language models; Obtain the prompts, test data, and large language models selected by the user from various prompts, test data, and large language models; The user-selected prompts, test data, and large language models are combined to obtain multiple sets of element combination data to be debugged.

[0045] The variable configuration module can be an interactive variable configuration front-end, providing a user interface (Web UI or desktop client) for developers. This user interface offers three independent management panels: a prompt manager, a test data warehouse, and a model service hub, allowing developers to define and manage variable / feature sets across three dimensions. The prompt manager allows users to create, edit, and version different prompt templates. The test data warehouse supports users manually entering or importing real data test cases and expected output text from existing systems. The model service hub allows users to configure API access points and authentication keys for different large model service providers (such as locally deployed models from OpenAI, DeepSeek, and Qwen3), and preset different parameter configuration groups.

[0046] During the definition phase, users can input or import several cue word variables to be tested, test datasets, and a list / configuration set of models to be evaluated in the intelligent agent element debugging system. The cue word variables include multiple cue words, the test dataset includes multiple test data sets, and the model list includes multiple large language models or configuration information for multiple large language models. For the configuration information of large language models, this can include configuration information for different large language model API endpoints, versions, and core parameters (e.g., Temperature coefficient, Max Tokens maximum number of tokens). The cue word manager includes the multiple cue words imported or input during the definition phase, the test data warehouse includes the multiple test data sets imported or input during the definition phase, and the model service hub includes the multiple large language models imported or input during the definition phase. Users can filter the data defined in the cue word manager, test data warehouse, and model service hub to select the multiple cue words, multiple test data sets, and multiple large language models to be debugged, i.e., to select the range of elements / variables requiring collaborative debugging. Then, the selected prompt words, test data, and large language models are sent to the collaborative execution engine. The collaborative execution engine can combine these three elements in any way to obtain multiple sets of element combination data to be debugged.

[0047] Alternatively, the above-mentioned combination of user-selected prompts, test data, and large language models can be used to obtain multiple sets of element combination data to be debugged, including: Obtain the data combination method of user input; the above data combination method includes any one of the following: full permutation combination, random sampling combination, and specified combination; The user-selected prompts, test data, and large language models are combined according to the data combination method to obtain multiple sets of element combination data to be debugged.

[0048] The user interface of the aforementioned variable configuration module can also provide a data combination method input control / combination strategy input control. Users can filter the desired data combination method from multiple data combination methods included in the data combination method control, or input the desired data combination method through the data combination method control. These multiple data combination methods can include full permutation combination, random sampling combination, and specified combination. Full permutation combination refers to arranging all elements in a given sequence (i.e., multiple selected prompt words, multiple test data, and multiple large language models) in all possible orders without repetition. Random sampling combination refers to randomly selecting a subset of elements from the given sequence. Specified combination refers to directly generating or locating a specific combination based on specific rules, indices, or conditions, rather than traversing the entire sequence.

[0049] After obtaining the user's desired data combination method, the variable configuration module can send it to the collaborative execution engine. The collaborative execution engine can include a combination strategy generator. This generator receives the user's desired data combination method and combines multiple selected prompt words, multiple test data sets, and multiple large language models according to the user's desired combination method. This results in multiple sets of element combination data to be debugged and transforms them into task instances for the agent to run. For example, assuming the user selects 2 prompt words, 2 data sets, and 2 large language models or model configurations, the combination strategy generator generates 2×2×2=8 independent element combination data sets based on the user's input data combination method (e.g., full permutation combinations), thus obtaining 8 debugging task instances. Each task instance is a unique "prompt word + data + large language model" triple.

[0050] Furthermore, the collaborative execution engine can also include a concurrent scheduler and adaptation layer. This is a high-performance asynchronous task scheduling pool responsible for extracting multiple task instances formed by combining the aforementioned elements in parallel. Through a built-in API adaptation layer, it converts standardized inputs into specific API request formats required by different model service providers and initiates the calls. It can also handle network timeouts, retry mechanisms, and concurrency rate limits to better support concurrent debugging of intelligent agents.

[0051] In this embodiment, multiple predefined prompt words, test data sets, and large language models are first acquired. Then, multiple prompt words, test data sets, and large language models selected by the user are obtained and combined to generate multiple sets of element combination data to be debugged. This allows for debugging of prompt words, test data, and models in personalized business scenarios, meeting the user's personalized intelligent agent debugging needs. Furthermore, by acquiring the user's input data combination method and combining the user-selected element data according to the user's input data combination method, multiple sets of element combination data to be debugged are obtained, further satisfying the user's personalized debugging needs.

[0052] The above embodiments mentioned that the performance of the combined data of each element can be quantitatively evaluated by the debugging results of the combined data of each element, and then further joint structured analysis can be carried out. The following embodiments will explain the process of quantitatively evaluating the performance of the combined data of each element by the debugging results of the combined data of each element.

[0053] In one embodiment, the debugging results of the combination data of each element in step 206 above are subjected to structured analysis to obtain structured analysis results, including: Based on the model output text and the corresponding expected output text of each element combination data, the comprehensive quantitative value corresponding to each element combination data is determined; the magnitude of the above comprehensive quantitative value is proportional to the performance of the agent when using the corresponding element combination data. The structured analysis results are determined based on the comprehensive quantitative values ​​corresponding to the combination of data for each element.

[0054] Each test data set includes test cases and their corresponding expected output text. The debugging results for each element combination data set include the model output text for that combination, as well as the model execution time and lexical information consumed during model execution. After obtaining the model output text for each element combination data set, the model output text is compared with the expected output text. Based on the comparison, each element combination data set is quantified to determine its comprehensive quantization value. For example, the similarity between the model output text and the expected output text of the element combination data set is considered; the greater the similarity, the better the overall performance of the agent in text processing using that element combination data set, and thus the higher the comprehensive quantization value.

[0055] For calculating the comprehensive quantitative value of each element combination data, in addition to calculating the similarity between the model output text of the element combination data and the expected output text of the corresponding element combination data, further quantitative processing can be performed by combining other information in the debugging results. Optionally, the above-mentioned determination of the comprehensive quantitative value corresponding to each element combination data based on the model output text of each element combination data and the corresponding expected output text includes: Based on the similarity between the model output text of each element combination data and the corresponding expected output text, determine the first quantitative value corresponding to each element combination data; Based on the model running time of each element combination data, determine the second quantitative value corresponding to each element combination data; Based on the lexical information consumed by the model running of each element combination data, the third quantification value corresponding to each element combination data is determined. Based on at least one of the first, second, and third quantitative values ​​of the combined data of each element, determine the comprehensive quantitative value corresponding to each combined data of the element.

[0056] The process involves first calculating the similarity between the model's output text and the corresponding expected output text for each element combination data set, obtaining the similarity score for each set. Then, this similarity score is used as the first quantization value for each element combination data set. Alternatively, the similarity scores can be quantized using a pre-trained first quantization model to obtain the first quantization value for each element combination data set. Simultaneously, the model execution time for each element combination data set can be quantized using a pre-trained second quantization model to obtain the second quantization value. Finally, the lexical information consumed by the model execution for each element combination data set can be quantized using a pre-trained third quantization model to obtain the third quantization value. Here, the first, second, and third quantization models can be, for example, neural network models. The obtained first, second, and third quantization values ​​for each element combination data set can belong to the same data range, such as all being within the range of 0 to 1.

[0057] After obtaining the first, second, and third quantitative values ​​corresponding to each combination of element data, for each combination of element data, any one of the first, second, and third quantitative values ​​can be used as its comprehensive quantitative value, or at least two of the quantitative values ​​can be weighted and summed to obtain its comprehensive quantitative value.

[0058] After obtaining the comprehensive quantitative value corresponding to each combination of element data, the comprehensive quantitative value of each combination of element data can be directly used as the final structured analysis result, or the comprehensive quantitative value of each combination of element data can be used as the intermediate result of each combination of element data, and combined with the debugging results of each combination of element data to perform structured analysis again to obtain the final structured analysis result.

[0059] Optionally, the above-mentioned debugging results of the combined data of each element are subjected to structured analysis to obtain structured analysis results, including: Pivot tables are used to perform structured analysis on the debugging results of the combination data of each element to obtain structured analysis results. The structured analysis results include multidimensional analysis views corresponding to the combination data of each element. The first dimension of the multidimensional analysis view is the large language model element, and the second dimension of the multidimensional analysis view is the prompt word element. Each cell of the multidimensional analysis view represents the structured analysis results of the large language model and the combination data of the element under the prompt word in the corresponding dimension. The first dimension and the second dimension are different and are respectively the rows or columns of the multidimensional analysis view.

[0060] The results aggregation and analysis platform includes a structured log database, which is used to receive the raw results returned by the collaborative execution engine in real time. It not only stores the text responses of the large language model, but also records metadata, including: model execution / run time (Latency), lexical information consumed by the model (such as the number of input / output lexical tokens), estimated cost, and whether the model has errors.

[0061] In addition, the results aggregation and analysis platform also includes a multidimensional analysis view, which serves as a front-end visualization component, allowing users to dynamically drag and drop the dimensions of the multidimensional analysis view. Specifically, pivot tables can be used to perform structured analysis on the debugging results of the combined data of each element and the aforementioned comprehensive quantitative values ​​to obtain structured analysis results. These results can include multidimensional analysis views corresponding to the combined data of each element. The rows and columns of this multidimensional analysis view can be either the type of the large language model or the version of the prompt word, with each row and column representing different content. The cells formed by the rows and columns can display the structured analysis results for the corresponding large language model type and prompt word version under specific test data. For example, this could include the debugging results and comprehensive quantitative values ​​for that large language model type and prompt word version. For instance, users can use "large language model" as a column, with different columns representing different large language models, and "prompt word" as a row, with different rows representing different prompt word versions. The cross cells of the table can visually display the model debugging results and comprehensive quantitative values ​​for specific test data.

[0062] In addition, this multidimensional analysis view also supports post-processing operations such as keyword highlighting and underlining in the model output text of the debugging results, making it easier for users to view.

[0063] In this embodiment, the comprehensive quantitative value of each element combination data is determined by comparing the model output text with the corresponding expected output text. This yields a structured analysis result of the debugging results for each element combination data, allowing developers to quickly determine whether the model output meets expectations. Furthermore, by calculating the similarity between the model output text and the expected output text of the corresponding element combination data, and combining this with the model running time and lexical information consumed in the debugging results, the comprehensive quantitative value of each element combination data can be comprehensively quantified. This collaborative analysis of multiple quantitative results from various combination elements allows developers to easily identify deeper issues such as "a certain prompt is only effective under a specific model" or "a certain type of data will fail under a specific prompt structure," thus enabling more systematic optimization of the agent. Further, by employing a structured multi-dimensional analysis view interface to analyze and display the debugging results and comprehensive quantitative values ​​of each element combination data, compared to tedious manual recording, developers can quickly make decisions based on the data in the multi-dimensional analysis view (e.g., selecting the most cost-effective model or determining the most robust prompt version), accelerating the agent's deployment process.

[0064] The above embodiments illustrate how structured analysis results can help users select the best-performing prompt words or large language models. The following embodiments illustrate this.

[0065] In one embodiment, the above method further includes: Obtain the first prompt word input by the user, and based on the structured analysis results of different large language models corresponding to the first prompt word, determine the first target large language model with the best performance among the large language models included in the combination data of each element and recommend it to the user; Alternatively, obtain the second largest language model from the user input, and based on the structured analysis results of the different prompt words corresponding to the second largest language model, determine the second target prompt word with the best performance among the prompt words included in the combination data of each element and recommend it to the user.

[0066] The structured analysis results shown above include the structured analysis results of each element combination data. The structured analysis results of each element combination data include the debugging results and comprehensive quantitative values ​​of the corresponding element combination data. In other words, the structured analysis results include the debugging results and comprehensive quantitative values ​​of different prompt words and different large language models.

[0067] In practical applications after the agent has been properly debugged, users can input a first prompt word. Then, based on the structured analysis results of different combinations of the prompt word and various large language models, the system identifies the top-performing target large language model for text reasoning, summarization, and recognition using that first prompt word. For example, the system could consider the comprehensive quantization value of the combination of the first prompt word and various language models, selecting the large language model with the highest comprehensive quantization value as the top-performing target large language model for that first prompt word. This top-performing target large language model can then be recommended to the user, allowing them to deploy the agent using the first prompt word and the target large language model, and input any actual business data to obtain the optimal model output text.

[0068] Similarly, in practical applications after the agent has been debugged, users can input a second major language model (or a type of second major language model). Then, through the structured analysis results of this second major language model under different combinations of prompt words and major language models, the system can find the second target prompt word that performs best in text reasoning / summarizing / recognition functions using this second major language model among all prompt word versions. For example, it could be the comprehensive quantization value of the combination scheme of the second major language model and each prompt word version, finding the prompt word version with the largest comprehensive quantization value as the best-performing second target prompt word under this second major language model. This second target prompt word can then be recommended to the user, allowing the user to deploy the agent using this second target prompt word and the second major language model, and input any actual business data into the agent to obtain the optimal model output text.

[0069] In this embodiment, based on the first prompt word input by the user and the structured analysis results of different large language models corresponding to different prompt words, the best-performing first target large language model is determined from the large language models included in the combined data of each element and recommended to the user. Alternatively, based on the second large language model input by the user and the structured analysis results of different prompt words corresponding to different large language models, the best-performing second target prompt word is determined from the prompt words included in the combined data of each element and recommended to the user. In this way, the best-performing prompt word or large language model can be recommended to the user during the application of the intelligent agent, so that the intelligent agent outputs the best text content.

[0070] To illustrate the technical solution of the present invention in more detail, a detailed embodiment is given below. (See attached image.) Figure 3 The diagram shown is an interactive signaling diagram of the intelligent agent element debugging method provided by the present invention. See also... Figure 2 The method may include the following steps: Step 1: Developers can define the prompt word set, test dataset, and model configuration set through the variable configuration module; Step 2: Developers can select a combination strategy (i.e., data combination method) and initiate debugging through the variable configuration module; Step 3: The variable configuration module can transmit the variable set and combination strategy to the collaborative execution engine; Step 4: The collaborative execution engine can automatically generate N parallel "triple" debugging tasks according to the combination strategy; at the same time, it concurrently calls the large model API to execute each debugging task in parallel. Step 5: Task A calls the large model API to execute the triplet using the prompt "Prompt V1 + Test Data 1 + Model A"; Step 6: Task B calls the large model API to execute the triplet using the prompt "Prompt V2 + Test Data 1 + Model A"; ... Step 7: The large model API returns the results of task A in parallel (including model output text, model execution time, and lexical information consumed by the model). Step 8: The large model API returns the results of Task B in parallel (including model output text, model execution time, and lexical information consumed by the model execution, etc.). ... Step 9: The collaborative execution engine sends all task execution data and results to the result aggregation and analysis module in real time; Step 10: The results aggregation and analysis module collects data, performs structured processing, generates pivot tables / multidimensional analysis views, and presents a visual comparison interface to the developers.

[0071] As described above, the technical solution of this invention breaks the traditional serial debugging mode. By establishing an integrated debugging environment, prompt words, test data, and models are regarded as three configurable and combinable core variable dimensions. Through the automation engine, the parallel execution of multi-variable combinations and the unified analysis of results can be realized, which can effectively improve the debugging efficiency of intelligent agents. At the same time, collaborative analysis can be performed, making it easier for developers to discover multi-variable coupling problems and make timely adjustments to quickly debug the intelligent agent.

[0072] The intelligent agent element debugging device provided by the present invention is described below. The intelligent agent element debugging device described below can be referred to in correspondence with the intelligent agent element debugging method described above.

[0073] Figure 4 This is a schematic diagram of the intelligent agent element debugging method device provided by the present invention. See also: Figure 4 As shown, the device may include: The combined data acquisition module 410 is used to acquire multiple sets of element combination data to be debugged; each set of element combination data includes a set of prompt words, test data and data combination of the large language model of the agent, and different element combination data include at least one element data in the prompt words, test data and large language model. The parallel debugging module 420 is used to debug the agent in parallel based on the combination data of each element, and to obtain the debugging results corresponding to the combination data of each element in parallel. Analysis module 430 is used to perform structured analysis on the debugging results of the combination data of each element to obtain structured analysis results; the above structured analysis results are used to assist users in selecting the target prompt words or target large language models with the best performance from the combination data of each element for the intelligent agent.

[0074] In one embodiment, the aforementioned combined data acquisition module 410 is specifically used to acquire multiple prompt words, multiple test data, and multiple large language models; acquire the prompt words, test data, and large language models selected by the user from each prompt word, each test data, and each large language model; and combine the prompt words, test data, and large language models selected by the user to obtain multiple sets of element combination data to be debugged.

[0075] Optionally, the aforementioned combined data acquisition module 410 is specifically used to acquire the data combination method input by the user; the aforementioned data combination method includes any one of full permutation combination, random sampling combination, and specified combination; the prompt words, test data, and large language model selected by the user are combined according to the data combination method to obtain multiple sets of element combination data to be debugged.

[0076] In one embodiment, each test data includes a test case and the expected output text corresponding to the test case, and the debugging results of each element combination data include the model output text corresponding to the corresponding element combination data. The analysis module 430 is specifically used to determine the comprehensive quantization value corresponding to each element combination data based on the model output text and the corresponding expected output text of each element combination data. The magnitude of the comprehensive quantization value is proportional to the performance of the agent when using the corresponding element combination data. The structured analysis result is determined based on the comprehensive quantization value corresponding to each element combination data.

[0077] Optionally, the debugging results of each element combination data also include the model running time and the lexical information consumed by the model running for the corresponding element combination data. Specifically, the analysis module 430 is used to determine the first quantization value corresponding to each element combination data based on the similarity between the model output text of each element combination data and the corresponding expected output text; to determine the second quantization value corresponding to each element combination data based on the model running time of each element combination data; to determine the third quantization value corresponding to each element combination data based on the lexical information consumed by the model running for each element combination data; and to determine the comprehensive quantization value corresponding to each element combination data based on at least one of the first quantization value, the second quantization value, and the third quantization value.

[0078] Optionally, the aforementioned analysis module 430 is specifically used to perform structured analysis on the debugging results of the combination data of each element using pivot tables, and obtain structured analysis results. The structured analysis results include a multi-dimensional analysis view corresponding to the combination data of each element. The first dimension of the multi-dimensional analysis view is the large language model element, and the second dimension of the multi-dimensional analysis view is the prompt word element. Each cell of the multi-dimensional analysis view represents the structured analysis results of the large language model and the combination data of the element under the prompt word in the corresponding dimension. The first dimension and the second dimension are different and are respectively rows or columns of the multi-dimensional analysis view.

[0079] In one embodiment, the structured analysis results include structured analysis results of the combination data of each element, and the apparatus further includes: The recommendation module is used to obtain the first prompt word input by the user, and based on the structured analysis results of different large language models corresponding to the first prompt word, determine the best-performing first target large language model among the large language models included in the combination data of each element and recommend it to the user; or, obtain the second large language model input by the user, and based on the structured analysis results of different prompt words corresponding to the second large language model, determine the best-performing second target prompt word among the prompt words included in the combination data of each element and recommend it to the user.

[0080] It should be noted that the apparatus provided in this embodiment of the invention can implement all the method steps implemented in the above method embodiment and can achieve the same technical effect. Therefore, the parts and beneficial effects that are the same as those in the method embodiment will not be described in detail here.

[0081] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 5As shown, the electronic device may include a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, communications interface 520, and memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute an agent element debugging method. This method includes: acquiring multiple sets of element combination data to be debugged; each set of element combination data includes a set of prompt words, test data, and a large language model data combination for the agent; different element combination data combinations include at least one element data difference among the prompt words, test data, and large language model; debugging the agent in parallel based on each element combination data, and obtaining the debugging results corresponding to each element combination data in parallel; performing structured analysis on the debugging results of each element combination data to obtain structured analysis results; the above structured analysis results are used to assist the user in selecting the target prompt word or target large language model with the best performance from each element combination data for the agent.

[0082] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0083] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the agent element debugging method provided by the above methods. The method includes: acquiring multiple sets of element combination data to be debugged; each set of element combination data includes a set of prompt words, test data, and a large language model data combination for the agent, and different element combination data include at least one element data in the prompt words, test data, and large language model that is different; debugging the agent in parallel according to each element combination data, and obtaining the debugging results corresponding to each element combination data in parallel; performing structured analysis on the debugging results of each element combination data to obtain structured analysis results; the above structured analysis results are used to assist the user in selecting the target prompt word or target large language model with the best performance from each element combination data for the agent.

[0084] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the agent element debugging method provided by the above methods. The method includes: acquiring multiple sets of element combination data to be debugged; each set of element combination data includes a set of prompt words, test data, and a large language model data combination for the agent, and different element combination data include at least one element data in the prompt words, test data, and large language model; debugging the agent in parallel according to each element combination data, and obtaining the debugging results corresponding to each element combination data in parallel; performing structured analysis on the debugging results of each element combination data to obtain structured analysis results; the above structured analysis results are used to assist the user in selecting the target prompt word or target large language model with the best performance from each element combination data for the agent.

[0085] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0086] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for debugging intelligent agent elements, characterized in that, include: Acquire multiple sets of element combination data to be debugged; each set of element combination data includes a set of prompt words, test data and data combination of large language model of agent, and different element combination data include at least one element data of prompt words, test data and large language model; The agent is debugged in parallel based on the combination data of each element, and the debugging results corresponding to the combination data of each element are obtained in parallel. The debugging results of the combination data of each element are subjected to structured analysis to obtain structured analysis results; The structured analysis results are used to assist users in selecting the optimal target prompt word or target large language model for the agent from the combined data of each element.

2. The intelligent agent element debugging method according to claim 1, characterized in that, The acquisition of multiple sets of element combination data to be debugged includes: Acquire multiple prompt words, multiple test data sets, and multiple large language models; Obtain the prompt words, test data, and large language models selected by the user from each of the aforementioned prompt words, each of the aforementioned test data, and each of the aforementioned large language models; The user-selected prompts, test data, and large language model are combined to obtain multiple sets of element combination data to be debugged.

3. The intelligent agent element debugging method according to claim 2, characterized in that, The process of combining the user-selected prompts, test data, and large language model yields multiple sets of element combination data to be debugged, including: The data combination method input by the user is obtained; the data combination method includes any one of the following: full permutation combination, random sampling combination, and specified combination; The user-selected prompts, test data, and large language model are combined according to the data combination method to obtain multiple sets of element combination data to be debugged.

4. The method for debugging intelligent agent elements according to any one of claims 1 to 3, characterized in that, Each set of test data includes test cases and the expected output text corresponding to the test cases. The debugging results of each set of element combination data include the model output text corresponding to the respective element combination data. The structured analysis of the debugging results of each set of element combination data to obtain structured analysis results includes: Based on the model output text and the corresponding expected output text of each element combination data, a comprehensive quantitative value corresponding to each element combination data is determined; the magnitude of the comprehensive quantitative value is proportional to the performance of the agent when using the corresponding element combination data. The structured analysis results are determined based on the comprehensive quantitative values ​​corresponding to the combined data of each of the aforementioned elements.

5. The intelligent agent element debugging method according to claim 4, characterized in that, The debugging results of each element combination data also include the model running time and the word information consumed by the model running for the corresponding element combination data. The step of determining the comprehensive quantization value corresponding to each element combination data based on the model output text and the corresponding expected output text includes: Based on the similarity between the model output text of each element combination data and the corresponding expected output text, a first quantification value corresponding to each element combination data is determined; Based on the model running time of each of the element combination data, determine the second quantification value corresponding to each of the element combination data; Based on the lexical information consumed by the model operation of each element combination data, the third quantization value corresponding to each element combination data is determined. Based on at least one of the first quantification value, the second quantification value, and the third quantification value of each of the element combination data, determine the comprehensive quantification value corresponding to each of the element combination data.

6. The method for debugging intelligent agent elements according to any one of claims 1 to 3, characterized in that, The process of performing structured analysis on the debugging results of each of the aforementioned element combination data to obtain structured analysis results includes: Pivot tables are used to perform structured analysis on the debugging results of each combination of elements to obtain structured analysis results. The structured analysis results include multidimensional analysis views corresponding to each combination of elements. The first dimension of the multidimensional analysis view is the large language model element, and the second dimension of the multidimensional analysis view is the prompt word element. Each cell of the multidimensional analysis view represents the structured analysis results of the large language model and the element combination data under the prompt word in the corresponding dimension. The first dimension and the second dimension are different and are respectively the rows or columns of the multidimensional analysis view.

7. The method for debugging intelligent agent elements according to any one of claims 1 to 3, characterized in that, The structured analysis results include the structured analysis results of each combination of the aforementioned elements, and the method further includes: The system obtains the first prompt word input by the user, and based on the structured analysis results of different large language models corresponding to the first prompt word, determines the first target large language model with the best performance among the large language models included in each element combination data and recommends it to the user. Alternatively, a second language model input by the user can be obtained, and based on the structured analysis results of different prompt words corresponding to the second language model, the second target prompt word with the best performance can be determined from the prompt words included in each element combination data and recommended to the user.

8. A device for debugging intelligent agent elements, characterized in that, include: The combined data acquisition module is used to acquire multiple sets of element combination data to be debugged; each set of element combination data includes a set of prompt words, test data and data combination of the large language model of the agent, and different element combination data include at least one element data in the prompt words, test data and large language model. The parallel debugging module is used to debug the agent in parallel based on the combination data of each element, and to obtain the debugging results corresponding to each combination data of each element in parallel. The analysis module is used to perform structured analysis on the debugging results of the combination data of each element to obtain structured analysis results; The structured analysis results are used to assist users in selecting the optimal target prompt word or target large language model for the agent from the combined data of each element.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the intelligent agent element debugging method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the intelligent agent element debugging method as described in any one of claims 1 to 7.

Images