Data processing method and electronic device

Abstract

The application discloses a data processing method and an electronic device. The method comprises the following steps: processing a first task by a first to-be-tested model combination to obtain a first processing result; the first to-be-tested model combination comprises a to-be-tested model and a to-be-tested agent framework; if the first processing result meets a test condition, generating a plurality of second to-be-tested model combinations based on the to-be-tested model, the to-be-tested agent framework, a benchmark model and a benchmark agent framework; processing a second task by the plurality of second to-be-tested model combinations respectively to obtain a plurality of second processing results; and determining a test result of the first to-be-tested model combination based on the plurality of second processing results.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method and an electronic device. Background Technology

[0002] With the rapid development of large models and intelligent agent frameworks, the variety of large models and intelligent agent frameworks available to users is increasing. In actual deployment, users usually need to combine a large model with an intelligent agent framework to form a complete intelligent agent system. However, different "model-framework" combinations show significant differences in task performance, making it difficult for users to predict in advance which combination is suitable for their tasks. Existing testing methods typically only test the capabilities of the large model alone, or only bind a default model to provide an illustrative demonstration of the intelligent agent framework, failing to achieve automated testing of the "model + framework" combination. Summary of the Invention

[0003] The technical solution provided in this application is as follows:

[0004] The first aspect of this application provides a data processing method, including:

[0005] The first task is processed by the first combination of models to be tested to obtain a first processing result; the first combination of models to be tested includes a model to be tested and a framework of intelligent agent to be tested.

[0006] If the first processing result meets the test conditions, multiple combinations of second models to be tested are generated based on the model to be tested, the intelligent agent framework to be tested, the benchmark model, and the benchmark intelligent agent framework.

[0007] By combining multiple second models under test to process the second task, multiple second processing results are obtained.

[0008] Based on multiple results of the second processing, the test results of the first combination of models to be tested are determined.

[0009] In one possible implementation, the step of processing the first task through a combination of the first model under test to obtain a first processing result includes:

[0010] Obtain the target task execution chain, which includes multiple processing nodes with sequential dependencies;

[0011] By combining the first model under test, the first task is processed based on the target task to obtain adaptation data corresponding to each processing node; the adaptation data characterizes the degree of adaptation between the model under test and the intelligent agent framework under test on the corresponding processing node.

[0012] Based on the adaptation data corresponding to the multiple processing nodes, a first processing result is determined.

[0013] In one possible implementation, the data processing method further includes:

[0014] If, based on the first processing result, it is determined that the adaptation data corresponding to each processing node meets the adaptation conditions, then the first processing result is determined to meet the test conditions.

[0015] In one possible implementation, the data processing method further includes:

[0016] If at least one of the processing nodes has adaptation data that does not meet the adaptation conditions, a first prompt message is generated based on the adaptation data corresponding to the processing node that does not meet the adaptation conditions; the first prompt message is used to indicate the type and location of the adaptation anomaly.

[0017] In one possible implementation, the generation of multiple combinations of second models to be tested based on the model under test, the intelligent agent framework under test, the benchmark model, and the benchmark intelligent agent framework includes:

[0018] The test model, the test agent framework, the baseline model, and the baseline agent framework are combined in pairs to obtain multiple combinations of second test models.

[0019] By combining multiple second models under test to process the second task, multiple second processing results are obtained, including:

[0020] Under the same task scenario and the same tool configuration, multiple combinations of the second test models are used to process the second task, resulting in multiple second processing results.

[0021] In one possible implementation, the second task includes a first type of task and a second type of task;

[0022] The second task is processed by combining multiple second models under test to obtain multiple second processing results, including:

[0023] The first type of task is processed by all the second test model combinations to obtain the first type of processing result corresponding to each second test model combination. The first type of processing result is used for performance comparison between different second test model combinations.

[0024] From a plurality of second test model combinations, determine the second test model combination that has the specified capability;

[0025] The second type of task is processed by the combination of the second test models with the specified capabilities to obtain a second type of processing result, which is used to determine the performance of the combination of the second test models with the specified capabilities on the specified capabilities.

[0026] In one possible implementation, each of the second processing results includes: processing result quality information corresponding to each of the second test model combinations; the processing result quality information characterizes the completion quality of the second test model combination for the second task;

[0027] The determination of the test result of the first combination of models to be tested based on multiple second processing results includes:

[0028] Based on the quality information of the processing results corresponding to each combination of the second models to be tested, the first adaptation difference information, the second adaptation difference information, and the third adaptation difference information are determined.

[0029] Based on the first adaptation difference information, the second adaptation difference information, and the third adaptation difference information, the test result of the first combination of models to be tested is determined;

[0030] Wherein, the first adaptation difference information represents the change in the quality information of the processing result caused by the model under test; the second adaptation difference information represents the change in the quality information of the processing result caused by the intelligent agent framework under test; and the third adaptation difference information represents the change in the quality information of the processing result caused by the combined interaction between the model under test and the intelligent agent framework under test.

[0031] In one possible implementation, the data processing method further includes:

[0032] If the third adaptation difference information exceeds the target difference range, based on the adaptation data corresponding to the first model combination under test and each of the processing nodes, determine the processing nodes that generate interaction differences between the model under test and the intelligent agent framework under test in processing the second task.

[0033] Based on the processing node that generates the interaction difference, a second prompt message is generated.

[0034] In one possible implementation, the data processing method further includes:

[0035] Acquire auxiliary information collected during the processing of the second task, the auxiliary information including at least one of the following: task completion status, tool call status, execution steps, time consumption, resource consumption, and error information;

[0036] Based on the first adaptation difference information, the second adaptation difference information, the third adaptation difference information, the auxiliary information, and the adaptation data corresponding to the first model combination under test and each of the processing nodes, the difference localization result is output;

[0037] The difference localization results are used to indicate that the adaptation differences between the model under test and the intelligent agent framework under test originate from at least one of the following:

[0038] The response capability of the model under test;

[0039] The scheduling capability of the intelligent agent framework under test;

[0040] Tool call chain;

[0041] The matching between the model under test and the framework of the intelligent agent under test.

[0042] Another aspect of this application provides an electronic device, comprising:

[0043] Memory is used to store computer programs;

[0044] The processor is used to execute the computer program to enable the electronic device to perform the following method steps:

[0045] The first task is processed by the first combination of models to be tested to obtain a first processing result; the first combination of models to be tested includes a model to be tested and a framework of intelligent agent to be tested.

[0046] If the first processing result meets the test conditions, multiple combinations of second models to be tested are generated based on the model to be tested, the intelligent agent framework to be tested, the benchmark model, and the benchmark intelligent agent framework.

[0047] By combining multiple second models under test to process the second task, multiple second processing results are obtained.

[0048] Based on multiple results of the second processing, the test results of the first combination of models to be tested are determined. Attached Figure Description

[0049] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0050] Figure 1 A flowchart illustrating a data processing method provided in Embodiment 1 of this application;

[0051] Figure 2 This is a flowchart illustrating a data processing method provided in Embodiment 2 of this application;

[0052] Figure 3 This is a flowchart illustrating a data processing method provided in Embodiment 3 of this application;

[0053] Figure 4 An interactive diagram illustrating a data processing architecture provided in this application;

[0054] Figure 5 A schematic diagram of an interactive interface for an automated testing system provided in this application;

[0055] Figure 6 Another interactive interface diagram of an automatic testing system provided in this application;

[0056] Figure 7 A schematic diagram of another interactive interface of an automatic testing system provided in this application;

[0057] Figure 8 A schematic diagram of another interactive interface of an automatic testing system provided in this application;

[0058] Figure 9 A schematic diagram of another interactive interface of an automatic testing system provided in this application;

[0059] Figure 10 A schematic diagram of another interactive interface of an automatic testing system provided in this application;

[0060] Figure 11 A schematic diagram of another interactive interface of an automatic testing system provided in this application;

[0061] Figure 12 This is a schematic diagram of another interactive interface of an automatic testing system provided in this application. Detailed Implementation

[0062] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0063] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0064] The terms "first," "second," etc., used in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of units is not necessarily limited to those units, but may include other units not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0065] In the embodiments of this application, reference is made to Figure 1 This is a flowchart illustrating a data processing method provided in Embodiment 1 of this application, as shown below. Figure 1 As shown, the method may include, but is not limited to, the following steps:

[0066] Step S101: Process the first task through the first test model combination to obtain the first processing result; the first test model combination includes the test model and the test agent framework.

[0067] In this embodiment, the model to be tested and the intelligent agent framework to be tested can be determined according to the testing requirements. For example, when a user needs to determine a model suitable for pairing with a certain intelligent agent framework from multiple candidate models, the intelligent agent framework can be used as the intelligent agent framework to be tested, and each candidate model can be used as the model to be tested, forming a first combination of models to be tested one by one.

[0068] Alternatively, when a user needs to determine the appropriate framework to pair with a certain model from among multiple candidate agent frameworks, the model can be used as the model to be tested, and each candidate agent framework can be used as the agent framework to be tested, forming the first combination of models to be tested one by one.

[0069] Alternatively, when users need to select the best combination from multiple models and frameworks simultaneously, they can cross-combine the candidate models and frameworks to form multiple first test model combinations for testing. The specific sources and determination methods of the test models and test agent frameworks can be set according to actual testing needs, and this embodiment does not limit them.

[0070] In this embodiment, the model under test can include any type of artificial intelligence model. For example, the model under test can be an open-source or closed-source AI model; it can be a general-purpose AI model or a domain-specific AI model fine-tuned for a particular area; it can be a text generation model or a multimodal model. For instance, the model under test can be a Qwen series model, a Llama series model, a DeepSeek series model, a ChatGLM series model, etc., or a model obtained by the user after fine-tuning or customizing the above models. The model under test can provide inference services through local deployment, such as starting the model service locally based on frameworks like Ollama, vLLM, llama.cpp, and Transformers, or it can provide model capabilities through a remote inference interface. The specific type and provision method of the model under test can be set according to actual testing needs, and this embodiment does not limit this.

[0071] The intelligent agent framework under test can include any type of intelligent agent framework. For example, the intelligent agent framework under test can be a framework for building task planning and execution, or a framework for building multi-agent collaborative systems; it can be a general-purpose intelligent agent framework, or a dedicated intelligent agent framework for specific application scenarios.

[0072] In this embodiment, the model under test can be connected to the intelligent agent framework under test, establishing a runtime connection between the two to form a first combination of models under test. The connection process can be executed according to the model connection specifications defined by the intelligent agent framework under test, including but not limited to: configuring the network address and port of the model service, setting model inference parameters, and establishing a request-response communication channel. After successful connection, the intelligent agent framework under test has the ability to send inference requests to the model under test and receive responses.

[0073] In the first model-to-be-tested combination, the agent framework and the model under test each perform different functions, interacting and collaborating to complete task processing. Specifically, the agent framework receives task input, generates inference requests for the model under test according to its built-in scheduling logic and planning strategy, sends requests to the model to drive inference, and performs subsequent operations based on the output returned by the model. The model under test receives inference requests from the agent framework, generates corresponding response results based on its own parameters and training knowledge, and returns them to the agent framework. The two interact through the interface defined by the agent framework, forming a complete collaborative working chain.

[0074] After the first task is issued to the agent framework under test, the agent framework under test receives the first task, first generates an initial inference request according to its own scheduling logic and sends it to the model under test; the model under test generates a response according to the request and returns it to the agent framework under test.

[0075] After receiving a response, the framework of the agent under test determines the subsequent operation based on the parsing result. If the response contains a tool call request, the corresponding tool is executed and the result is sent back to the model under test to drive the next round of inference; if the response is the final answer, the task is considered complete. This interaction cycle continues for multiple rounds until the task is completed or execution is interrupted. Failure in any of the above interaction stages will cause the overall task processing to be interrupted, which allows us to record whether the task execution is complete, as well as the location and type of failure.

[0076] In this embodiment, the first task can be used to verify whether the first combination of models under test can form a workable collaborative whole, that is, whether the models under test and the intelligent agent framework under test can successfully complete the basic task processing flow according to the expected interaction method. To achieve the above verification purpose, the first task may have, but is not limited to, the following characteristics:

[0077] First, the task format is relatively simple, and the requirements for the reasoning ability and knowledge breadth of the model under test are not high, so as to avoid the failure of link verification due to the insufficient ability of the model under test itself.

[0078] Secondly, the task flow can trigger a complete interaction chain between the model under test and the intelligent agent framework under test. That is, all the links from the intelligent agent framework under test initiating a request, the model under test returning a response, the intelligent agent framework under test parsing the response and executing subsequent operations, to the final completion of the task can be covered.

[0079] Third, the task results are easily and automatically determined, meaning that the success of the task can be determined based on clearly defined rules. For example, the first task can be a question-and-answer task, such as "Please answer what 1+1 equals." This type of task does not involve tool calls, has a simple process, and mainly verifies whether the intelligent agent framework under test can successfully call the model under test and obtain a valid response. The first task can also be a task involving basic tool calls, such as "Please read the test.txt file in the current directory." This type of task, in addition to verifying the response of the model under test, can also simultaneously verify whether the intelligent agent framework under test can correctly parse the tool call request of the model under test and drive the tool to execute. The specific type and content of the first task can be set according to actual testing needs; this embodiment does not limit this.

[0080] In this embodiment, monitoring and recording can be performed throughout the entire process of the first task being processed by the first model under test. Information during the task execution process can be collected, including but not limited to: whether the task was completely executed to completion, whether an interruption occurred during execution, the execution stage at which the interruption occurred, and the intermediate data generated at each stage. Based on the above collected information, a first processing result is generated.

[0081] The first processing result reflects the handling status of the first model combination under test on the first task. The processing status can include whether the task was completed completely, whether there were any interruptions during execution, and the execution stage at which the interruption occurred. If the model under test and the intelligent agent framework under test can effectively connect, the task execution can be completed completely, and the first processing result reflects that the combination has formed a workable collaborative link. If the model under test and the intelligent agent framework under test cannot effectively connect, the task execution will fail midway, and the first processing result reflects the stage at which the failure occurred and the type of failure, such as: the request initiated by the framework cannot be correctly parsed by the model, the model does not return a valid response, the response format returned by the model does not meet the framework's parsing requirements, or the framework cannot perform subsequent operations after parsing the response.

[0082] Step S102: If the first processing result meets the test conditions, generate multiple combinations of second models to be tested based on the model to be tested, the intelligent agent framework to be tested, the benchmark model, and the benchmark intelligent agent framework.

[0083] The test conditions are used to determine whether the first combination of models under test is qualified to enter the subsequent test process. When the first processing result meets the test conditions, it indicates that the first combination of models under test has passed the basic verification, and a workable collaborative link can be formed between the model under test and the intelligent agent framework under test, thus meeting the conditions for further evaluation. When the first processing result does not meet the test conditions, it indicates that the combination has encountered problems that hinder subsequent evaluation during the processing of the first task, and it will no longer be included in the subsequent test process.

[0084] Test conditions can be set according to actual test requirements. For example, test conditions may include: whether the first task was fully executed to completion, whether there were no interruptions during execution, and whether each interactive step was properly connected. The specific content of the test conditions is not limited in this embodiment.

[0085] After determining that the first processing result meets the test conditions, a baseline model and a baseline agent framework are introduced. The baseline model is a pre-selected model used as a comparison reference; the baseline agent framework is a pre-selected agent framework used as a comparison reference. The selection principles for the baseline model and the baseline agent framework can be determined according to actual test requirements. For example, the baseline model can be a known model that performs stably on a specific capability dimension, and the baseline agent framework can be a known framework that performs stably on scheduling capabilities.

[0086] Each second test model combination can contain one model and one agent framework. Multiple second test model combinations are generated based on the test model, the test agent framework, the baseline model, and the baseline agent framework, and there are structural differences between the various second test model combinations in the model dimension and / or framework dimension, so that different second test model combinations can form a comparable relationship.

[0087] For example, a second test model combination comprising the model under test and the framework of the intelligent agent under test can be generated; a second test model combination comprising the model under test and the benchmark intelligent agent framework can also be generated; a second test model combination comprising the benchmark model and the framework of the intelligent agent under test can also be generated; and a second test model combination comprising the benchmark model and the benchmark intelligent agent framework can also be generated. The number and specific composition of the second test model combinations can be set according to actual testing requirements, and this embodiment does not limit them.

[0088] The second model combination under test can be accessed and configured in the same way as the first model combination under test, that is, the runtime connection between the model and the framework is established according to the access specifications defined by their respective frameworks, so as to ensure that each combination is in a runnable state.

[0089] Step S103: Process the second task by combining multiple second test models to obtain multiple second processing results.

[0090] The second task can be set according to actual testing needs. For example, the second task can be the same as the first task, that is, the same task content can be reused in the formal test after passing the basic verification; the second task can also be different from the first task, for example, the second task can be a task with higher complexity than the first task, or it can be a task designed for a specific capability dimension. This embodiment does not limit this.

[0091] In practice, each second test model combination operates independently. Each second test model combination receives the same second task content and does not interact or share data with other combinations during execution. Each combination performs independent reasoning and interaction according to its own included model and agent framework, completes the processing flow required by the second task, and produces a second processing result corresponding to its combination.

[0092] Because the different combinations of second test models differ in terms of model dimension and / or framework dimension, the second processing results obtained by each combination of second test models for the same second task also differ accordingly.

[0093] Step S104: Based on multiple second processing results, determine the test results of the first combination of models to be tested.

[0094] The processing result of the second task, obtained by combining the model under test (MAT) and the intelligent agent framework under test, includes the influence of the MAT's own capabilities, the intelligent agent framework's own scheduling capabilities, and the interaction between the two. These influences work together in the processing result and are not presented independently. The processing result is presented as a single overall output; it is not possible to directly determine from this overall output which parts originate from the MAT's own capabilities, which from the intelligent agent framework's own scheduling capabilities, and which from their interaction.

[0095] If the evaluation of the first model combination is based solely on this processing result, only the overall performance of the first model combination can be obtained. It cannot determine whether the good or bad performance stems from the model, the framework, or the combination of the two. In real-world applications where models and agent frameworks are used in combination, users often need to further understand the source of the problem to make targeted improvements. For example, when the overall performance of the combination is poor, users need to determine whether to replace the model, the agent framework, or adjust the way the two work together. A single processing result cannot provide this distinguishing information.

[0096] Multiple combinations of second test models differ in their composition across the model and / or framework dimensions, and each combination executes independently under the same second task. Therefore, the differences between the second processing results can reflect the impact introduced by changing the model or framework. Based on the comparison relationship between multiple second processing results, the overall performance of the first test model combination can be divided into three parts: the impact introduced by the test model itself, the impact introduced by the test agent framework itself, and the impact introduced by the interaction between the two.

[0097] Therefore, determining the test results of the first model combination based on multiple second processing results can extend the evaluation of the first model combination from an overall performance evaluation to an attributable and interpretable evaluation, thereby providing a clear decision-making basis for model selection, framework selection, and the optimization of their combination.

[0098] In this embodiment, a two-level testing architecture can be used to evaluate the first combination of models under test. In the first-level test, the first combination of models under test processes the first task, and the admission screening is completed based on whether the obtained first processing result meets the test conditions. This can pre-exclude combinations that cannot form a workable collaborative link before entering the formal evaluation, avoid performing subsequent tests on invalid combinations, thereby improving evaluation efficiency and saving computing resources.

[0099] After passing the initial screening, a second-level test is conducted. Based on the model under test, the intelligent agent framework under test, the baseline model, and the baseline intelligent agent framework, multiple combinations of second models under test are generated and executed independently under the same second task. The resulting multiple second processing results have a controlled control relationship, which can effectively distinguish the influence introduced by the model under test itself, the influence introduced by the intelligent agent framework under test itself, and the influence introduced by the interaction between the two. The test results of the first combination of models under test determined in this way not only reflect its overall performance, but also further indicate the composition source of the overall performance. This achieves an improvement from a single overall evaluation to an attributable and explainable evaluation, providing users with clear decision-making basis in the process of model selection, framework selection, and optimization of the combination of the two.

[0100] As another optional embodiment of this application, refer to Figure 2 This is a flowchart illustrating a data processing method provided in Embodiment 2 of this application. This embodiment is mainly an implementation of step S101 in Embodiment 1, such as... Figure 2 As shown, the specific steps may include, but are not limited to, the following:

[0101] Step S1011: Obtain the target task execution chain, which includes multiple processing nodes with sequential dependencies.

[0102] In this embodiment, the target task execution chain can be a serialized representation of the interaction steps required for the first model under test to process the task. During task processing, the intelligent agent framework under test and the model under test do not exchange requests and responses only once, but require multiple interactive collaborations to complete the task. These multiple steps are executed sequentially according to the task processing order, with the output of the previous step serving as the input of the next step. Failure of any step to complete normally will prevent subsequent steps from continuing, thus interrupting the overall task processing.

[0103] The processing nodes included in the target task execution chain can be configured according to actual testing needs. For example, the target task execution chain may include, but is not limited to, the following processing nodes:

[0104] The task input node is where the framework of the agent under test receives task input and generates inference requests.

[0105] The framework generates a request node, and the framework of the agent under test sends an inference request to the model under test.

[0106] The model outputs a response node, and the model under test generates a response and returns it to the framework of the agent under test.

[0107] The framework receives the output node, and the framework of the agent under test parses the model output and determines the subsequent operations.

[0108] The tool execution node is where the intelligent agent framework under test calls the corresponding tool based on the parsing results and obtains the tool execution results.

[0109] The results are returned to the node, and the tool's execution results are returned to the model under test.

[0110] The model generates the final answer node, and the model under test generates the final answer based on the tool's execution results.

[0111] The aforementioned processing nodes are executed sequentially in the order described above, together forming a complete task processing chain.

[0112] The processing nodes are sequentially dependent, meaning the execution result of any one node directly determines whether and how subsequent processing nodes can execute. For example, if the model under test does not return a valid response after the framework generates a request, the framework cannot proceed to the subsequent response parsing and operation decision-making stages; if the tool fails to execute, the tool's result feedback and the final answer based on the tool's result cannot be performed. By dividing the task processing process into these sequentially dependent processing nodes, the execution status of the first model under test in each stage of the task processing can be clearly defined.

[0113] Step S1012: Using the first model under test combination, perform link processing on the first task based on the target task to obtain adaptation data corresponding to each processing node; the adaptation data characterizes the degree of adaptation between the model under test and the intelligent agent framework under test on the corresponding processing node.

[0114] During task execution, the test agent framework within the first test model assembly interacts with the test model in the order of the processing nodes included in the aforementioned target task execution chain. At each processing node, execution information can be collected, and the corresponding adaptation data can be determined based on this execution information.

[0115] Adaptation data is used to characterize the degree of adaptation between the model under test and the intelligent agent framework under test at that processing node. The adaptation data corresponding to different processing nodes reflects whether the content or action output by the intelligent agent framework under test can be correctly received by the model under test in the interaction link corresponding to that node, or in other words, whether the output of the model under test can be correctly parsed by the intelligent agent framework under test and drive subsequent operations.

[0116] For example, for the framework-generated request node, the adapted data can reflect whether the inference request generated by the framework of the agent under test conforms to the input constraints of the model under test; for the model-output response node, the adapted data can reflect whether the model under test returns a valid and stable response to the framework request; for the framework-accepted output node, the adapted data can reflect whether the output structure of the model under test can be correctly parsed by the framework of the agent under test according to its action acceptance requirements; for the tool execution node, the adapted data can reflect whether the tool call parameters are valid and whether the tool can be executed successfully; for the result feedback node, the adapted data can reflect whether the context information when the tool execution result is fed back to the model under test is complete; for the model-generated final answer node, the adapted data can reflect whether the final answer generated by the model under test incorporates the tool execution result.

[0117] For example, the adaptation data corresponding to each processing node may include, but is not limited to:

[0118] The framework generates adaptation data (denoted as R) corresponding to the request node, which reflects the compatibility between the request generated by the framework and the input constraints of the model under test. For example, whether the request format generated by the framework meets the input format requirements of the model under test, and whether the request content exceeds the context length limit of the model under test.

[0119] The model outputs adaptation data (denoted as M) corresponding to the response node, which reflects the stability of the model under test when responding to requests. For example, whether the model under test can return a valid response for every request, and whether timeouts, garbled characters, or no response occur.

[0120] The framework receives the corresponding adaptation data (represented as O) for the output nodes, reflecting the matching between the data structure output by the model under test and the acceptance requirements of the framework. For example, if the framework requires the model to return tool call instructions with a specific format, does the output format of the model under test meet that requirement? Can the framework correctly parse the structure output by the model under test?

[0121] The adaptation data corresponding to the tool execution node may include: a tool parameter validity component (denoted as A), which reflects whether the tool call parameters extracted from the output of the model under test meet the tool definition requirements. For example, whether the parameter type, name, and required fields are complete and correct;

[0122] The tool execution success component (denoted as T) reflects whether the tool was executed successfully after being invoked. For example, whether the tool returned a success status or produced the expected tool execution result.

[0123] The adaptation data (denoted as C) corresponding to the result feedback node is used to reflect the completeness of the context returned by the tool execution result to the model under test. For example, whether the result returned by the tool is completely and untruncatedly added to the context of the model under test, and whether the model under test can obtain the complete tool result.

[0124] The adaptation data (denoted as F) corresponding to the final answer node generated by the model is used to reflect how well the final answer generated by the model under test absorbs and utilizes the tool's results. For example, whether the final answer of the model under test is actually generated based on the tool's results, and whether it correctly utilizes the information in the tool's results.

[0125] In addition, the adaptation data may also include failure log information (denoted as E), which records the stages in the task execution process where failures occurred and the corresponding error types. For example, the failure log information can indicate whether the failure occurred on the model side, the framework side, the tool side, or at the junction of the model and the framework.

[0126] For example, the adaptation data can be continuous numerical values, ranging from 0 to 1, where 0 represents no adaptation, 1 represents full adaptation, and intermediate values ​​represent different degrees of adaptation; it can also be a binary form of pass / fail.

[0127] In implementations using continuous numerical values, each adaptation data point can be obtained by statistically analyzing the proportion of successful responses at the corresponding processing node to the total number of responses. For example, at the model output response node, if the model under test is requested 10 times by the intelligent agent framework under test, and 8 of those responses successfully return responses that meet the format requirements, then the adaptation data for that node can be set to 0.8.

[0128] A higher ratio indicates a higher degree of fit between the model under test and the intelligent agent framework under test at that processing node; a lower ratio indicates a less stable connection between the two at that processing node and a lower degree of fit. The fit data corresponding to each processing node can be calculated using the above method, or other quantification methods that can characterize the degree of fit can be used. This embodiment does not limit this method.

[0129] It should be noted that among the processing nodes mentioned above, the task input node mainly involves the process of the AI ​​framework under test receiving the task and generating an initial inference request. At the task input node, the model under test is usually not yet invoked, and there has been no interaction between the model under test and the AI ​​framework under test. Therefore, in one implementation, the task input node may not have corresponding adaptation data set, and the collection of adaptation data can start from the framework generating the request node, that is, from when the AI ​​framework under test actually sends the inference request to the model under test. However, this embodiment does not limit this. In other implementations, corresponding adaptation data can also be set for the task input node according to actual testing needs, for example, to reflect the matching between the task input format and the receiving requirements of the AI ​​framework under test.

[0130] Step S1013: Determine the first processing result based on the adaptation data corresponding to the multiple processing nodes.

[0131] In this embodiment, the adaptation data corresponding to each processing node can be combined into a structured link adaptation fingerprint, and the link adaptation fingerprint is used as the first processing result. Specifically, the adaptation data R corresponding to the framework generating request node, the adaptation data M corresponding to the model output response node, the adaptation data O corresponding to the framework receiving output node, the adaptation data (including the tool parameter validity component A and the tool execution success component T) corresponding to the tool execution node, the adaptation data C corresponding to the result feedback node, the adaptation data F corresponding to the model generating final answer node, and the failure record information E are combined to obtain the link adaptation fingerprint Φ=(R, M, O, A, T, C, F, E). In this way, the first processing result completely includes the adaptation degree information and failure record information of each processing node in the form of a link adaptation fingerprint.

[0132] Furthermore, after obtaining the adaptation data for each processing node, the link closure degree can be calculated based on this adaptation data and used as a component of the first processing result. Since the processing nodes included in the target task execution link have sequential dependencies, the failure of any processing node to execute normally will block the operation of subsequent nodes, thereby interrupting the overall task processing. Therefore, calculations can be performed on each continuous adaptation data.

[0133] For example, the link closure degree L can be determined as follows: L = R × M × O × A × T × C × F.

[0134] If any of the adaptation data among R, M, O, A, T, C, and F is zero, it means that the first model combination under test cannot be adapted at the corresponding processing node at all. In this case, the link closure L is zero, that is, the target task execution link cannot be closed, and the first model combination under test cannot form a workable collaborative whole.

[0135] If all the aforementioned adaptation data are greater than zero, then the link closure degree L is greater than zero, meaning the target task execution link can be closed, and the first model combination under test possesses the basic conditions to form a collaborative whole. Furthermore, since each adaptation data point is a continuous value between 0 and 1, the magnitude of the link closure degree L reflects the overall level of adaptation data for each processing node. A larger L indicates a higher overall adaptation degree of the first model combination under test on the target task execution link; a smaller L indicates a lower overall adaptation degree. Therefore, the link closure degrees L of different first model combinations under test are comparable and can be used to measure the differences in link adaptation degrees between different combinations.

[0136] In this embodiment, the execution process of the first task is decomposed along the target task execution chain, which includes multiple processing nodes with sequential dependencies. Adaptation data characterizing the fit between the model under test and the intelligent agent framework under test is collected at each processing node. The first processing result is then determined based on the adaptation data of each processing node, thus transforming the overall processing of the first task into a refined monitoring of each processing node. Because the processing nodes have sequential dependencies, any adaptation anomaly at any processing node will prevent subsequent nodes from continuing. Therefore, the adaptation data at each processing node accurately reflects the fit of the first model combination under test at each specific processing node, rather than just providing a general overall processing result. This allows the first processing result to expand from a single judgment of whether the whole is runnable to a structured presentation of the fit of each processing node. Therefore, this embodiment provides a fine-grained decision-making basis for subsequently determining whether the first model combination under test should enter the formal evaluation, and also provides a data foundation for locating processing nodes with adaptation anomalies, avoiding a general attribution of the cause of task processing interruption.

[0137] As an optional implementation, the first processing result can be determined to meet the test conditions in the following ways, but is not limited to:

[0138] Step S11: If, based on the first processing result, it is determined that the adaptation data corresponding to each processing node meets the adaptation conditions, then the first processing result meets the test conditions.

[0139] The specific details of the adaptation conditions can be set according to actual testing needs, and this embodiment does not limit them.

[0140] For example, when the adaptation data is represented by continuous numerical values ​​between 0 and 1, the adaptation condition can be that the adaptation data is greater than zero. Since the adaptation data corresponding to each processing node represents the degree of adaptation between the model under test and the intelligent agent framework under test on that processing node, if the adaptation data corresponding to a certain processing node is greater than zero, it means that the interaction between the model under test and the intelligent agent framework under test on that node can be completed normally; conversely, if the adaptation data corresponding to a certain processing node is equal to zero, it means that the model under test and the intelligent agent framework under test on that node cannot be connected at all, that is, the interaction link cannot be completed normally.

[0141] Because the processing nodes in the target task execution chain have sequential dependencies, the failure of any processing node will prevent subsequent processing nodes from continuing to execute, thus interrupting the overall task processing. Therefore, when the adaptation data corresponding to each processing node is greater than zero, it indicates that the first model combination under test can complete the interaction normally on each processing node, and there are no blockages in any link of the target task execution chain. That is, the target task execution chain can be closed, and the first model combination under test can form a workable collaborative whole. Accordingly, it can be determined that the first processing result meets the test conditions, that is, the combination passes the admission verification and is qualified to enter the subsequent formal evaluation process.

[0142] In the implementation method that uses link closure degree for determination, based on the link closure degree L determined in step S1013, since L is the product of each continuous adaptation data component, when the adaptation data corresponding to each processing node is greater than zero, the link closure degree L must be greater than zero; when there is any processing node whose corresponding adaptation data is zero, the link closure degree L is zero. Therefore, a link closure degree L greater than zero can also be used as an equivalent judgment method for determining that the adaptation data corresponding to each processing node meets the adaptation conditions. That is, if L is greater than zero, it is determined that the adaptation data corresponding to each processing node meets the adaptation conditions, and thus the first processing result meets the test conditions; if L is equal to zero, it is determined that there is at least one processing node whose corresponding adaptation data does not meet the adaptation conditions, and thus the first processing result does not meet the test conditions.

[0143] Since the link closure degree L can be obtained by a multiplication expression, the first processing result can be judged to meet the test conditions by judging whether the link closure degree L is greater than zero. Admission determination can be completed with extremely low computational overhead, without having to compare the numerical relationship between each adaptation data component and the adaptation conditions one by one, thereby further improving the admission screening efficiency.

[0144] By employing the above method, the judgment of whether the first processing result meets the test conditions is concretized into a judgment of whether the adaptation data corresponding to each processing node meets the adaptation conditions, thus providing a clear quantitative basis for the admission judgment. Since the adaptation data is collected and determined point-by-point at each processing node of the target task execution chain, it can reflect the connection between the model under test and the intelligent agent framework under test at each specific interaction stage. Therefore, the judgment result based on the adaptation data of each processing node can accurately reflect whether the first model under test combination has formed effective collaboration at each stage of the entire chain, avoiding potential misjudgments that may occur when judging solely based on whether the overall task has been completed.

[0145] For example, in some cases, although the task may be completed as a whole, a particular step may pass only due to chance, resulting in a low actual degree of fit reflected in the adaptation data. However, by determining whether the adaptation data of each processing node meets the adaptation conditions, it is possible to ensure that the collaborative capability of the combination reaches a stable and usable level at the step level. Therefore, this implementation method can provide a reliable screening basis before the first model combination to be tested enters the formal evaluation, reducing the waste of resources caused by ineffective evaluation of unsuitable combinations.

[0146] As an optional implementation, the data processing method may include, but is not limited to, the following steps:

[0147] Step S201: Obtain the target task execution chain, wherein the target task execution chain includes multiple processing nodes with sequential dependencies;

[0148] Step S202: Using the first model to be tested, perform link processing on the first task based on the target task to obtain adaptation data corresponding to each processing node; the adaptation data characterizes the degree of adaptation between the model to be tested and the intelligent agent framework to be tested on the corresponding processing node.

[0149] Step S203: Determine the first processing result based on the adaptation data corresponding to the multiple processing nodes.

[0150] For detailed procedures of steps S201-S203, please refer to the relevant descriptions of steps S1011-S1013 above, which will not be repeated here.

[0151] Step S204: If, based on the first processing result, it is determined that the adaptation data corresponding to each processing node meets the adaptation conditions, multiple combinations of second models to be tested are generated based on the model to be tested, the intelligent agent framework to be tested, the benchmark model, and the benchmark intelligent agent framework.

[0152] Step S205: Process the second task by combining multiple second models under test to obtain multiple second processing results;

[0153] Step S206: Based on multiple second processing results, determine the test results of the first combination of models to be tested.

[0154] For a detailed description of steps S204-S206, please refer to the relevant description of steps S102-S104 in Example 1, which will not be repeated here.

[0155] Step S207: If there is at least one adaptation data corresponding to the processing node that does not meet the adaptation conditions, generate a first prompt message based on the adaptation data corresponding to the processing node that does not meet the adaptation conditions; the first prompt message is used to indicate the type and location of the adaptation anomaly.

[0156] In this embodiment, when there is at least one processing node whose corresponding adaptation data does not meet the adaptation conditions, that is, when at least one of R, M, O, A, T, C, F has a value of zero, a first prompt message can be generated based on the adaptation data with a value of zero.

[0157] Specifically, since each adaptation data point has a one-to-one correspondence with each processing node in the target task execution chain, the processing node where an adaptation anomaly occurs can be determined based on the adaptation data with a value of zero, and the position of that processing node in the target task execution chain is taken as the anomaly location. Furthermore, the type of adaptation anomaly at that processing node can be determined based on the specific meaning represented by the adaptation data.

[0158] Furthermore, the failure log information E can be used to supplement the adaptation exception types. The failure log information E can indicate whether the exception occurred on the model side, the framework side, the tool side, or at the junction of the model and the framework, thereby further refining the adaptation exception types.

[0159] For example, if the value of R is zero, the first prompt message can indicate that: the abnormal location is the framework-generated request node, and the adaptation abnormality type is that the request generated by the framework of the agent under test is not compatible with the input constraints of the model under test. For example, the request format does not meet the input format requirements of the model under test, or the request content exceeds the context length limit of the model under test.

[0160] If the value of M is zero, the first prompt message can indicate that the abnormal location is the model output response node, the adapted abnormal type is that the model under test cannot return a stable and valid response, that is, there is a response abnormality on the model side.

[0161] If the value of O is zero, the first prompt message can indicate that the abnormal location is the frame receiving output node, and the adaptation abnormality type is that the data structure output by the model under test does not meet the action receiving requirements of the intelligent agent frame under test, that is, there is an adaptation problem between the model output and the frame parsing.

[0162] If the value of A is zero, the first prompt message can indicate that the abnormal location is the parameter validity step in the tool execution node, and the appropriate abnormal type is that the tool call parameters extracted from the output of the model under test do not meet the tool definition requirements, which belongs to the parameter validity problem in the tool call chain.

[0163] If the value of T is zero, the first prompt message can indicate that the abnormal location is the tool execution stage in the tool execution node, and the appropriate abnormal type is that the tool failed to execute after being called, which belongs to the tool side problem.

[0164] If the value of C is zero, the first prompt message can indicate that the abnormal location is the result feedback node, and the appropriate abnormal type is that the tool execution result failed to be completely fed back to the context of the model under test, which belongs to the context feedback problem.

[0165] If F is zero, the first prompt message can indicate that the abnormal location is the node where the model generates the final answer, and the appropriate abnormal type is that the model under test failed to generate a valid answer based on the tool's execution result, which belongs to the final answer connection problem.

[0166] The specific format of the first prompt message can be set according to actual testing needs, and this embodiment does not limit it. For example, the first prompt message can be in text form, such as "Framework connection output node adaptation error: Model output structure cannot be parsed". The first prompt message can also be in structured data form, with the adaptation error type and error location output as field values ​​respectively. The first prompt message can also highlight the error handling node on the graphical interface.

[0167] In this embodiment, by generating the first prompt information, the specific reason why the first test model combination failed the access verification can be clearly indicated in the first-level testing stage. This provides a direct basis for subsequent troubleshooting and optimization of the adaptation problem of the first test model combination, and avoids only obtaining a general conclusion of failure without further locating the problem.

[0168] As another optional embodiment of this application, refer to Figure 3 This is a flowchart illustrating a data processing method provided in Embodiment 3 of this application. This embodiment mainly describes one implementation of step S102 described above. Figure 3 As shown, the specific steps may include, but are not limited to, the following:

[0169] Step S1021: If the first processing result meets the test conditions, the model to be tested, the intelligent agent framework to be tested, the benchmark model, and the benchmark intelligent agent framework are combined in pairs to obtain multiple combinations of second models to be tested.

[0170] In this embodiment, one can be selected from the model to be tested and the baseline model on the model side, and one can be selected from the agent framework to be tested and the baseline agent framework on the agent framework side. The selected model and the agent framework are combined to form a second model to be tested combination.

[0171] For example, the following four combinations of second models to be tested can be generated:

[0172] The second test model combination consists of the test model and the test agent framework. This combination has the same structure as the first test model combination and is used to reflect the overall performance of the test model and the test agent framework working together.

[0173] The second test model combination consists of the test model and the benchmark agent framework. This combination replaces the test agent framework with the benchmark agent framework while keeping the model unchanged. It is used to form a comparison in the framework dimension, i.e., the model is fixed and the framework is changed.

[0174] The second test model combination consists of a baseline model and the test agent framework. This combination replaces the test model with the baseline model while keeping the agent framework unchanged. It is used to form a comparison in the model dimension, i.e., the framework is fixed and the model is replaced.

[0175] The second test model combination consists of a baseline model and a baseline agent framework, which serves as a benchmark reference for the entire comparative evaluation.

[0176] The test model, the test agent framework, the benchmark model, and the benchmark agent framework are combined in pairs to obtain multiple second test model combinations. This is one implementation of step S102, which generates multiple second test model combinations based on the test model, the test agent framework, the benchmark model, and the benchmark agent framework.

[0177] Step S103 above may include, but is not limited to:

[0178] Step S1031: Under the same task scenario and the same tool configuration, the second task is processed by multiple combinations of the second test models to obtain multiple second processing results.

[0179] The task scenario can be used to define the task type and content of the second task. For example, the task scenario can be a code modification scenario, an information retrieval scenario, a file operation scenario, a multi-step planning scenario, etc. The task scenario can be set according to actual testing needs; this embodiment does not limit this.

[0180] The same task scenario means that each combination of second models under test processes the same second task. By keeping the task scenario the same, it can be ensured that each combination of second models under test faces the same processing object, avoiding the introduction of additional variables due to different task content, thus making the second processing results comparable.

[0181] Tool configuration can be used to limit the set of tools and tool parameters used during the execution of the second task. For example, tool configuration may include file reading / writing tools, code execution tools, browser tools, database query tools, API calling tools, etc. The specific content of the tool configuration can be set according to actual testing needs; this embodiment does not limit this.

[0182] The same tool configuration means that each combination of second test models uses the same set of tools when processing the second task, and the parameter settings of the tools are kept consistent. By keeping the tool configuration the same, the processing results of each combination of second test models can be avoided due to differences in tools, ensuring that the differences between the second processing results are only caused by differences in the composition of the combination.

[0183] In practice, each second model combination under test operates independently. Each combination receives the same second task, and under the same tool configuration, performs independent reasoning and interaction according to its own included model and agent framework to complete the processing flow required by the second task and produce a second processing result corresponding to its combination. During execution, each second model combination does not interact with or share data with other combinations.

[0184] Because the various combinations of second test models differ in their composition along the model and / or framework dimensions, the second processing results obtained by each combination on the same second task also differ accordingly. These differences stem from the different compositions of the combinations, rather than from differences in task or tool configurations.

[0185] In this embodiment, since the task content and tool conditions faced by each of the second test model combinations are completely identical, the impact of the task scenario and tool configuration on the processing results is controlled at the same level. Therefore, the differences between the second processing results can be attributed to the differences in the model and / or framework composition of each combination, rather than the interference of external variables. This allows for the acquisition of a set of controlled control data when determining the test results of the first test model combination based on multiple second processing results. This provides a comparable data basis for distinguishing the influence introduced by the test model itself, the influence introduced by the test agent framework itself, and the influence introduced by their interaction. Therefore, this embodiment can provide structured control data support for the testing of the first test model combination without relying on human experience judgment, which helps to improve the credibility and interpretability of the test results.

[0186] As an optional implementation, the second task may include, but is not limited to, a first type of task and a second type of task.

[0187] The first type of task can be a task that all second-test model combinations must perform. This type of task can be used to provide a unified performance benchmark for different second-test model combinations. Since the second-test model combinations differ in their composition in the model dimension and / or framework dimension, by having all second-test model combinations process the same type of task, the processing results of each combination under the same task conditions can be obtained, thus providing a consistent data foundation for horizontal comparison of the overall performance of each combination.

[0188] For example, the first type of task may include, but is not limited to, at least one of the following:

[0189] The general task package contains generic processing tasks that do not depend on specific tool capabilities, and is used to evaluate the basic processing capabilities of each second test model combination in a general scenario. All second test model combinations must execute the tasks in the general task package.

[0190] The stability task package contains the same or similar tasks that need to be executed multiple times to evaluate the consistency of the processing results of each second test model combination over multiple runs. All second test model combinations must execute the tasks in the stability task package, and the stability of the combination is evaluated through the results of multiple executions.

[0191] The resource task package contains tasks with varying levels of complexity, used to evaluate the resource consumption of each second model combination during task processing, such as time consumption, memory usage, and GPU memory usage. All second model combinations must execute the tasks in the resource task package.

[0192] In this embodiment, the second type of task can be a task selectively executed based on the capabilities of the second model combination under test. The second type of task can be used to determine the performance of a second model combination under test possessing a specified capability on that specified capability dimension. Unlike the first type of task, the second type of task does not require execution by all second model combinations under test, but rather is executed by only the second model combinations possessing the corresponding capabilities.

[0193] For example, the second type of task may include, but is not limited to, at least one of the following:

[0194] The tool invocation task package contains tasks that require invoking external tools, and is used to evaluate the tool invocation capabilities of the combined system. The tool invocation task package can be executed by a second test model that declares support for tool invocation.

[0195] File operation task packages contain tasks that require reading, modifying, or generating files, and are used to evaluate the combined file operation capabilities. File operation task packages can be executed by a second test model combination that declares support for file tools.

[0196] Code task packages, containing code generation, repair, or interpretation tasks, are used to evaluate the combined processing capabilities in code scenarios. Code task packages can be executed by a second test model tailored to the code scenario.

[0197] Multi-step task packages contain tasks requiring multi-step planning and execution, used to evaluate the planning capabilities and intermediate result delivery capabilities of the combination. Multi-step task packages can be executed by a second test model that supports multi-step planning.

[0198] In step S1031, the second task is processed by combining multiple second models to be tested to obtain multiple second processing results, which may include, but are not limited to:

[0199] Step S10311: Process the first type of task by processing all the second test model combinations respectively, and obtain the first type of processing result corresponding to each second test model combination. The first type of processing result is used for performance comparison between different second test model combinations.

[0200] Each second model combination can independently process the first type of task and obtain the corresponding first type of processing result. The first type of processing result can include information such as the correctness of task completion, the completeness of task completion, task execution time, and operational stability indicators.

[0201] Since all the second test model combinations process the same type of first-class task, each combination faces the same processing object. The differences in their type-first-class processing results stem solely from the differences in the combination's own composition, rather than the differences in the task content. Therefore, the type-first-class processing results of each combination are directly comparable, providing a unified basis for comparison in determining the test results of the first test model combination.

[0202] Step S10312: Determine the second test model combination with the specified capability from multiple combinations of the second test model combinations.

[0203] In this embodiment, the second model combination under test consists of a model and an agent framework, and its capabilities may depend on the capabilities of the model and / or agent framework in the combination. Different models may have different processing capabilities, and different agent frameworks may also declare support for different functional characteristics. Therefore, the capabilities of different second model combinations under test may differ.

[0204] The method for determining whether the second model combination under test possesses the specified capability can be set according to actual testing requirements. For example, it can be determined solely based on whether the agent framework included in the second model combination under test declares support for the specified capability; it can also be determined solely based on whether the models included in the second model combination under test possess the specified capability; or it can be determined jointly by combining the capability declarations of both the models and the agent framework.

[0205] Step S10313: Process the second type of task through the second test model combination with the specified capability to obtain the second type of processing result. The second type of processing result is used to determine the performance of the second test model combination with the specified capability on the specified capability.

[0206] After determining the specified capabilities possessed by the second model combination under test, tasks corresponding to those specified capabilities can be determined from the second category of tasks. For example, when the second model combination under test possesses tool invocation capabilities, tool invocation tasks can be identified as the second category of tasks corresponding to that combination; when the second model combination under test possesses file operation capabilities, file operation tasks can be identified as the second category of tasks corresponding to that combination; when the second model combination under test possesses code processing capabilities, code processing tasks can be identified as the second category of tasks corresponding to that combination; and when the second model combination under test possesses multi-step planning capabilities, multi-step planning tasks can be identified as the second category of tasks corresponding to that combination. This embodiment does not limit this.

[0207] Each combination of second test models can independently execute the corresponding second type of task and output the corresponding processing result, i.e., the second type of processing result.

[0208] Since the second type of task is designed for a specific capability dimension, its processing results can reflect the actual level of the second model combination under test in the specified capability. For example, if the second type of task is a tool invocation task, the processing results of the second type can reflect the performance of the second model combination under test in tool invocation capability, such as the success rate of tool invocation, parameter accuracy, and invocation process completeness.

[0209] If the second type of task is a file operation task, then the processing result of the second type can reflect the performance of the combination in file reading, modification, generation, etc.

[0210] If the second type of task is a code-related task, then the results of the second type of processing can reflect the performance of the combination in code generation, code repair, code interpretation, etc.

[0211] If the second type of task is a multi-step planning task, then the processing result of the second type can reflect the performance of the combination in terms of task planning, step-by-step execution, and intermediate result transmission.

[0212] In this embodiment, on the one hand, by making all the second test model combinations process the same type of first-class tasks, the interference factors introduced by the differences in task content are eliminated, so that the processing results of each combination under the same task conditions are established on a consistent evaluation benchmark. This enables a direct horizontal comparison between different model and framework combinations in terms of comprehensive performance, and provides a unified data foundation for the subsequent separation of the influence introduced by the model itself, the influence introduced by the framework itself, and the influence introduced by the interaction and adaptation between the two.

[0213] On the other hand, by assigning a second type of task matching the corresponding capability to the combination of the second model to be tested based on the actual capabilities of the combination, each combination can be evaluated under task conditions that are adapted to its capability structure, thereby obtaining the performance of the combination in the corresponding capability dimension. This allows the evaluation results to further reflect the performance distribution of the combination in each specified capability dimension on the basis of comprehensive performance, providing more granular data support for subsequent attribution analysis and improving the interpretability and pertinence of the evaluation results.

[0214] Therefore, by processing evaluation methods for multiple types of tasks, this embodiment not only ensures the comparability of the overall performance among all combinations, but also improves the targeting of evaluation for specific capability dimensions, enhances the adaptability of the automatic evaluation method under different combination configurations and capability conditions, and provides efficient and reliable data basis for the selection and optimization of model and agent framework combinations.

[0215] As an optional implementation, each of the second processing results may include: processing result quality information corresponding to each of the second test model combinations; the processing result quality information characterizes the completion quality of the second test model combination for the second task.

[0216] In this embodiment, the quality information of the processing results corresponding to each of the second test model combinations can be determined in the following way:

[0217] Step S1032: Obtain the task results output after the second test model is combined and processed for the second task.

[0218] For example, if the second task is a question-and-answer task, the task result is the answer text generated by the combination; if the second task is a code generation task, the task result is the code generated by the combination.

[0219] Step S1033: Conduct a quality assessment of the task results and determine the values ​​of multiple assessment indicators; the assessment indicators may include at least one of the following: task completion correctness indicator, task completion completeness indicator, task execution time indicator, and tool call success rate indicator.

[0220] Task completion accuracy metrics can be used to measure the degree of conformity between the result of one task and the expected goal of the next task. For example, in a question-and-answer task, is the answer correct? In a code generation task, can the generated code run correctly and achieve the expected function?

[0221] Task completion completeness metrics measure whether the task completion result fully covers all the requirements of the second task. For example, whether a multi-step task has completed all steps, or a file operation task has completed all required operations.

[0222] The task execution time metric can be used to measure the time consumed by the second model combination under test from receiving the second task to outputting the task processing result.

[0223] The tool call success rate metric can be used to measure the proportion of tool call operations that are successfully completed during the execution of the second task in the second test model combination.

[0224] Step S1034: Determine the quality information of the processing result based on the values ​​of multiple evaluation indicators. For example, weights can be assigned to each evaluation indicator, and the values ​​of each evaluation indicator can be weighted and summed to obtain the quality information of the processing result; alternatively, a preset scoring rule can be used to comprehensively score the values ​​of each evaluation indicator, and the scoring result can be used as the quality information of the processing result.

[0225] In this embodiment, step S104 may include, but is not limited to, the following steps:

[0226] Step S1041: Determine the source of the difference based on the quality information of the processing results.

[0227] In this embodiment, based on the quality information of the processing results corresponding to each combination of second models to be tested, the sources of difference can be determined by comparing multiple second processing results. The sources of difference may include model factors, framework factors, and co-adaptation factors.

[0228] For example, a second test model combination consisting of the test model and a baseline agent framework can be selected, as well as a second test model combination consisting of a baseline model and a baseline agent framework. Both combinations use the same agent framework, the baseline agent framework, and the only difference between them is the model itself.

[0229] Since the two combinations use the exact same framework, the difference in their processing results cannot stem from differences in framework scheduling or connection methods. Furthermore, since the model under test is not paired with the framework of the agent under test, the difference cannot originate from a compatibility issue between the model under test and the agent under test. This difference likely arises from the difference in capabilities between the model under test and the baseline model under the same framework conditions.

[0230] If there is a significant difference in the processing results between the two combinations, the source of the difference can be identified as a model-related factor, meaning there is a problem with the model under test itself. Specifically, this indicates that the model under test has a lower ability to complete the task under the same framework conditions compared to the baseline model, and the user should consider replacing the model under test.

[0231] For example, we can select a second test model combination consisting of a baseline model and a test agent framework, and a second test model combination consisting of a baseline model and a baseline agent framework. These two combinations use the same model, the baseline model; the only difference is the agent framework.

[0232] Since the two combinations use identical models, the difference in their processing results cannot stem from differences in model capabilities. Furthermore, since the agent framework under test was not paired with the model under test, the difference cannot originate from a compatibility issue between the two. The difference likely arises from the difference in scheduling capabilities between the agent framework under test and the baseline agent framework under the same model conditions.

[0233] If there is a significant difference in the processing results of the two combinations, it can be determined that the source of the difference is a framework factor, that is, there is a problem with the framework of the agent under test itself. In this case, the user should consider replacing the framework of the agent under test.

[0234] For example, if no significant differences are found in the two comparisons mentioned above, it indicates that the model under test performs normally when paired with the benchmark agent framework, and the agent framework under test also performs normally when paired with the benchmark model. Neither the model under test nor the agent framework under test has revealed any shortcomings in its own capabilities when paired with the benchmark components individually.

[0235] However, if the combination of the model under test and the intelligent agent framework under test produces significantly worse results than the combination of the baseline model and the baseline intelligent agent framework, it indicates that the problem is neither with the model under test itself nor with the intelligent agent framework under test, but rather arises when the two are used together.

[0236] This indicates that the discrepancy originates from a compatibility issue, specifically a mismatch between the model under test and the framework of the intelligent agent under test. Both can function independently, but the interaction between them incurs additional performance overhead. In this case, the user does not need to replace the model or framework, but rather should optimize the interaction between them, such as adjusting output format conventions, modifying prompts, and optimizing parameter passing methods.

[0237] Step S1041 can be understood as a qualitative attribution method. In this method, by comparing the processing results of each second test model combination, it is determined whether the source of the difference belongs to model factors, framing factors, or co-fitting factors, thereby confirming the type of problem as a whole.

[0238] In this embodiment, a quantitative attribution method can also be provided. In the quantitative attribution method, based on the processing result quality information corresponding to each combination of second models to be tested, first adaptation difference information, second adaptation difference information, and third adaptation difference information are determined to reflect the changes in processing results caused by the model to be tested itself, the changes in processing results caused by the intelligent agent framework to be tested itself, and the changes in processing results caused by the interaction of the two in a quantitative form.

[0239] In one implementation of quantitative attribution, step S104 above may include, but is not limited to, the following steps:

[0240] Step S1042: Based on the quality information of the processing results corresponding to each combination of the second models to be tested, determine the first adaptation difference information, the second adaptation difference information, and the third adaptation difference information.

[0241] Wherein, the first adaptation difference information represents the change in the quality information of the processing result caused by the model under test; the second adaptation difference information represents the change in the quality information of the processing result caused by the intelligent agent framework under test; and the third adaptation difference information represents the change in the quality information of the processing result caused by the combined interaction between the model under test and the intelligent agent framework under test.

[0242] In this embodiment, the model under test is denoted as M, the intelligent agent framework under test is denoted as F, the baseline model is denoted as M0, and the baseline intelligent agent framework is denoted as F0. The four combinations of the second models under test are (M,F), (M,F0), (M0,F), and (M0,F0), and their processing result quality information is denoted as S(M,F), S(M,F0), S(M0,F), and S(M0,F0) respectively.

[0243] In this embodiment, the first adaptation difference information can be determined using the following relationship:

[0244] ΔM = S(M,F0) - S(M0,F0)

[0245] Here, ΔM represents the first adaptation difference information. When ΔM is positive, it means that under the same baseline agent framework, the model under test improves the quality of the processing results compared to the baseline model; when ΔM is negative, it means that the model under test reduces the quality of the processing results compared to the baseline model.

[0246] In this embodiment, the second adaptation difference information can be determined using the following relationship:

[0247] ΔF = S(M0,F) - S(M0,F0)

[0248] Here, ΔF represents the second adaptation difference information. When ΔF is positive, it means that under the same benchmark model, the intelligent agent framework under test improves the quality of processing results compared with the benchmark intelligent agent framework. When ΔF is negative, it means that the intelligent agent framework under test reduces the quality of processing results compared with the benchmark intelligent agent framework.

[0249] In this embodiment, the third adaptation difference information can be determined using the following relationship:

[0250] I = S(M,F) - S(M,F0) - S(M0,F) + S(M0,F0)

[0251] Where I represents the third adaptation difference information. When the value of I is close to zero, it means that the model under test and the framework of the agent under test act independently on the processing result. There is no obvious additional gain or loss after the two are combined (that is, the combined performance is equal to the simple sum of the individual effects, and there is no unexpected synergistic gain (positive interaction) or loss (negative interaction)). The overall performance can be explained by the superposition of the model main effect (that is, the independent influence of the model itself on the quality information of the processing result) and the framework main effect (that is, the independent influence of the framework itself on the quality information of the processing result).

[0252] When I is significantly greater than zero, it indicates that there is a positive co-adaptation between the model under test and the framework of the agent under test, and the quality of the combined processing result is better than the level expected by the superposition of the main effects of the model and the main effects of the framework.

[0253] When I is significantly less than zero, it indicates that there is a negative co-adaptation between the model under test and the framework of the agent under test. The quality of the combined processing result is worse than the level expected by the superposition of the main effects of the model and the main effects of the framework. In other words, the two produce additional performance loss when used in combination.

[0254] Step S1043: Based on the first adaptation difference information, the second adaptation difference information and the third adaptation difference information, determine the test result of the first combination of models to be tested.

[0255] In this embodiment, the first adaptation difference information, the second adaptation difference information, and the third adaptation difference information may be used as the test results of the first combination of models to be tested, but are not limited to this.

[0256] In this embodiment, the overall processing result quality information S(M,F) of the first model combination under test, the first adaptation difference information ΔM, the second adaptation difference information ΔF, and the third adaptation difference information I can also be used as the test result of the first model combination under test. Through the above test results, not only can the overall performance of the first model combination under test on the second task be obtained, but the overall performance can also be decomposed into the influence introduced by the model under test itself, the influence introduced by the intelligent agent framework under test itself, and the influence introduced by the interaction and cooperation between the two.

[0257] In this embodiment, by providing a quantitative attribution method, the comprehensive evaluation reflecting the overall performance of the first model combination under test is transformed into three specifically calculable, independent, and superimposed adaptation difference information items. Specifically, the first adaptation difference information ΔM numerically represents the change in processing result quality information caused by the model itself; the second adaptation difference information ΔF numerically represents the change in processing result quality information caused by the intelligent agent framework itself; and the third adaptation difference information I numerically represents the change in processing result quality information caused by the interaction of the two. Since these three adaptation difference information items are independent and comparable numerical components, users do not need to rely on experience-based judgments of the overall score. They can directly determine the influence degree of each of the model factor, framework factor, and co-adaptation factor by the sign and magnitude of ΔM, ΔF, and I, and thereby distinguish between dominant and secondary factors. This further clarifies the improvement effects that can be brought about by changing the model under test, changing the intelligent agent framework under test, or optimizing the connection between the two, thus providing a precise, directly comparable, and executable quantitative basis for model and intelligent agent framework selection decisions and adaptation optimization.

[0258] For example, taking a code modification task package, let the baseline model M0 be Qwen2.5-Coder, the baseline agent framework F0 be OpenClaw, and the quality information S(M0,F0) of the processing result of the baseline combination (M0,F0) be 86.0. The following three examples illustrate the specific application of the quantitative attribution method.

[0259] Example 1: The test combination is the baseline combination itself.

[0260] When the model to be tested M equals the baseline model M0, and the agent framework to be tested F equals the baseline agent framework F0, the four combinations of the second models to be tested are all (M0, F0), and the quality information of their processing results is all 86.0. This is determined based on the aforementioned relationship:

[0261] ΔM = S(M,F0) - S(M0,F0) = 86.0 - 86.0 = 0.0;

[0262] ΔF = S(M0,F) - S(M0,F0)= 86.0 - 86.0 = 0.0;

[0263] I = S(M,F) - S(M,F0) - S(M0,F) + S(M0,F0) = 86.0 - 86.0 - 86.0 + 86.0= 0.0.

[0264] In this example, all variables remained unchanged, and the three adaptation differences were all zero, verifying the self-consistency of the above attribution formula.

[0265] Example 2: Only the framework is changed, while the model remains the same.

[0266] Assume the model to be tested, M, is still Qwen2.5-Coder (the same as the baseline model), and the agent framework to be tested, F, is Hermes. The quality information of the processing results for the four combinations is as follows:

[0267] The combination (M,F) is Qwen2.5 - Coder × Hermes, and the score S(M,F) = 92.0;

[0268] The combination (M,F0) is Qwen2.5-Coder × OpenClaw, with a score of S(M,F0)=86.0;

[0269] The combination (M0,F) is Qwen2.5 - Coder × Hermes, with a score S(M0,F) = 92.0;

[0270] The combination (M0,F0) is Qwen2.5-Coder × OpenClaw, with a score of S(M0,F0)=86.0.

[0271] Determined based on the relation:

[0272] ΔM = S(M,F0) - S(M0,F0) = 86.0 - 86.0 = 0.0;

[0273] ΔF = S(M0,F) - S(M0,F0) = 92.0 - 86.0 = +6.0;

[0274] I = S(M,F) - S(M,F0) - S(M0,F) + S(M0,F0) = 92.0 - 86.0 - 92.0 + 86.0= 0.0.

[0275] It is evident that the score improvement stemmed entirely from framework factors (ΔF = +6.0). This means that Hermes' scheduling strategy was more effective than OpenClaw's in code modification tasks, contributing a 6.0-point improvement to the overall performance; while, since the model remained unchanged, neither model factors nor co-adaptation factors introduced any additional impact.

[0276] Example 3: Both the model and framework have been replaced, and there is interaction loss.

[0277] Let the model M to be tested be DeepSeek-V3, and the agent framework F to be tested be Hermes. The quality information of the processing results of the four combinations is as follows:

[0278] The combination (M,F) is DeepSeek-V3 × Hermes, with a score of S(M,F) = 80.0.

[0279] The combination (M,F0) is DeepSeek-V3 × OpenClaw, with a score of S(M,F0) = 78.0.

[0280] The combination (M0,F) is Qwen2.5 - Coder × Hermes, with a score S(M0,F) = 92.0;

[0281] The combination (M0,F0) is Qwen2.5-Coder × OpenClaw, with a score of S(M0,F0)=86.0.

[0282] Calculate based on the relational formula:

[0283] ΔM = S(M,F0) - S(M0,F0) = 78.0 - 86.0 = -8.0;

[0284] ΔF = S(M0,F) - S(M0,F0) = 92.0 - 86.0 = +6.0;

[0285] I = S(M,F) - S(M,F0) - S(M0,F) + S(M0,F0) = 80.0 - 78.0 - 92.0 + 86.0= -4.0.

[0286] The following attribution conclusions can be drawn: a change of -8.0 due to model factors indicates that DeepSeek-V3's processing capability is weaker than the baseline model under the same benchmark framework; a change of +6.0 due to framework factors indicates that Hermes' scheduling strategy still has a positive contribution; a change of -4.0 due to co-adaptation factors indicates that DeepSeek-V3 and Hermes incur additional performance losses when used together, i.e., there is a negative co-adaptation between the two. Based on these quantitative results, users can directly conclude that the main reason for the overall performance decline is insufficient model capability, while the framework's advantages are still retained; at the same time, attention should be paid to the additional losses in the interaction between the model and the framework, and optimization can be made to the connection configuration between the two.

[0287] As can be seen from the three examples above, the quantitative attribution method can clearly decompose the overall score into three independent quantitative components: model factors, framework factors, and co-adaptation factors. Moreover, each component is comparable across different combinations, thus providing accurate and interpretable data for the selection and adaptation optimization of models and agent frameworks.

[0288] As an optional implementation, the data processing method may include, but is not limited to, the following steps:

[0289] Step S301: Process the first task through the first test model combination to obtain the first processing result; the first test model combination includes the test model and the test agent framework.

[0290] For a detailed description of step S301, please refer to the relevant description of step S101 in Example 1, which will not be repeated here.

[0291] Step S302: If the first processing result meets the test conditions, the model to be tested, the intelligent agent framework to be tested, the benchmark model, and the benchmark intelligent agent framework are combined in pairs to obtain multiple combinations of second models to be tested.

[0292] For a detailed explanation of step S302, please refer to the relevant description of step S1021 above, which will not be repeated here.

[0293] Step S303: Under the same task scenario and the same tool configuration, the second task is processed by multiple combinations of the second test models to obtain multiple second processing results.

[0294] Each of the second processing results may include: processing result quality information corresponding to each of the second test model combinations; the processing result quality information characterizes the completion quality of the second test model combination for the second task.

[0295] For a detailed explanation of step S303, please refer to the relevant descriptions of steps S1032-S1034 above, which will not be repeated here.

[0296] Step S304: Based on the quality information of the processing results corresponding to each combination of the second models to be tested, determine the first adaptation difference information, the second adaptation difference information, and the third adaptation difference information.

[0297] Step S305: Based on the first adaptation difference information, the second adaptation difference information and the third adaptation difference information, determine the test result of the first combination of models to be tested.

[0298] Wherein, the first adaptation difference information represents the change in the quality information of the processing result caused by the model under test; the second adaptation difference information represents the change in the quality information of the processing result caused by the intelligent agent framework under test; and the third adaptation difference information represents the change in the quality information of the processing result caused by the combined interaction between the model under test and the intelligent agent framework under test.

[0299] For detailed procedures of steps S304-S305, please refer to the relevant descriptions of steps S1042-S1043 above, which will not be repeated here.

[0300] Step S306: If the third adaptation difference information exceeds the target difference range, based on the adaptation data corresponding to the first model under test combination and each of the processing nodes, determine the processing nodes that generate interaction differences between the model under test and the intelligent agent framework under test in processing the second task.

[0301] When the value of the third adaptation difference information (I) is within the target difference range, it can be considered that the interaction between the model to be tested and the framework of the agent to be tested is at an acceptable level, that is, the combination of the two does not produce additional changes beyond the expected superposition of the main effects of the model and the main effects of the framework.

[0302] When the value of I exceeds the target difference range, it indicates that there is a significant synergistic adaptation effect between the model under test and the intelligent agent framework under test, that is, the two produce additional changes beyond the sum of their individual effects when used together. Therefore, the specific boundary of the target difference range can be set according to actual testing needs, such as a symmetrical interval centered at zero, or it can be dynamically adjusted according to historical test data or task type. This embodiment does not limit this.

[0303] Since the link adaptation fingerprint (Φ) contains multiple adaptation data such as R, M, O, A, T, C, and F, each adaptation data corresponds to a specific processing node in the target task execution link and characterizes the degree of adaptation between the model under test and the intelligent agent framework under test at that processing node, when the cooperative adaptation effect is significant, in one optional implementation, the link adaptation fingerprint of the first model combination under test (M,F) can be compared one by one with the link adaptation fingerprints corresponding to the combination of the model under test and the benchmark intelligent agent framework (M,F0), the combination of the benchmark model and the intelligent agent framework under test (M0,F), and the combination of the benchmark model and the benchmark intelligent agent framework (M0,F0).

[0304] For any processing node in the target task execution chain, if the value of the adaptation data corresponding to the processing node in the first test model combination (M,F) is significantly lower than its value in combination (M,F0) and combination (M0,F), and this decrease cannot be explained by the test model's own capabilities or the test agent framework's own scheduling capabilities, then the processing node can be determined as the processing node that generates the interaction difference.

[0305] Considering that the coordination and adaptation problem usually focuses on the connection between the model output and the framework acceptance, the adaptation data components closely related to the connection can be examined first. For example, the adaptation data O corresponding to the framework acceptance output node, the adaptation data C corresponding to the result feedback node, and the adaptation data F corresponding to the model generating the final response node can be the focus of examination. For instance, if the value of adaptation data O in the first model combination (M,F) is significantly lower than its values ​​in (M,F0) and (M0,F), then the framework acceptance output node can be identified as the processing node causing the interaction difference, i.e., there is a coordination and adaptation problem between the data structure output by the model under test and the action acceptance requirements of the framework of the agent under test. If the adaptation data C or the adaptation data value is significantly lower, then the result feedback node or the model generating the final response node can be identified as the processing node causing the interaction difference, respectively.

[0306] Of course, in this embodiment, the link closure degree L can also be used for judgment before processing based on the link adaptation fingerprint (Φ). When the link closure degree L of the combination (M,F) of the model under test and the intelligent agent framework under test is significantly lower than the link closure degree L of the combination (M,F0) of the model under test and the baseline intelligent agent framework, and the link closure degree L of the combination (M0,F) of the baseline model and the intelligent agent framework under test, it indicates that the closure degree of the entire link after the combination of the model under test and the intelligent agent framework under test has decreased. This decrease cannot be explained by the model under test itself or the intelligent agent framework under test itself alone. At this time, the specific processing node causing the decrease in link closure degree can be determined by combining the differences in the adaptation data components corresponding to each processing node.

[0307] Step S307: Based on the processing node that generates the interaction difference, generate a second prompt message.

[0308] In this embodiment, at least one of the location information of the processing node that generates the interaction difference and the type information of the interaction difference can be generated based on the processing node that generates the interaction difference. In this embodiment, the second prompt information may include, but is not limited to, at least one of the location information of the processing node that generates the interaction difference and the type information of the interaction difference.

[0309] Location information can be used to indicate the specific processing nodes that generate interactive differences in the target task execution chain, such as the framework receiving output nodes, result feedback nodes, or model generating final answer nodes.

[0310] The type information of interaction differences can be used to describe the specific form of adaptation anomaly that occurs at the processing node, such as the model output structure being unable to be accepted by the framework, the tool result feedback context being incomplete, and the final answer not making full use of the tool results.

[0311] Additionally, the second prompt message can also include targeted optimization suggestions to guide users in adjusting the connection configurations involved in the processing node. For example, if the processing node causing the interaction difference is a framework output node, the second prompt message can carry an exception type of "model output format does not match the framework parsing rules" and suggest adjusting the model output format conventions or the framework parsing rules.

[0312] If the node that processes the interaction difference is the result feedback node, the second prompt message can carry the exception type "tool result feedback is incomplete", and it is recommended to optimize the context organization method when the tool result feedback is performed.

[0313] If the node for handling interaction differences is the node for generating the final answer by the model, then the second prompt message can carry the exception type "the final answer did not fully absorb the tool results," and it is recommended to adjust the constraints on the generation of the final answer in the prompt words.

[0314] The specific presentation format of the second prompt information can be set according to actual needs, such as outputting it in text form, outputting it in structured data form, or highlighting the corresponding processing node in the graphical interface. This embodiment does not limit this.

[0315] In this embodiment, based on the fact that the third adaptation difference information exceeds the target difference range and confirms a significant collaborative adaptation effect between the model under test and the intelligent agent framework under test, the correspondence between each adaptation data component in the link adaptation fingerprint and each processing node in the target task execution link is further utilized to locate the existence of the collaborative adaptation problem from the overall level to the specific processing node at the link level. Based on this, a second prompt is generated, allowing the user to directly determine the specific link and direction of adjustment without having to check every single connection between the model and the framework. Thus, the test results, based on existing qualitative and quantitative attributions, further refine the attribution from factors to nodes, improving the diagnostic accuracy and optimization guidance effect of the automatic testing method for model-intelligent agent framework adaptation problems in actual deployment.

[0316] As an optional implementation, the data processing method may include, but is not limited to, the following steps:

[0317] Step S401: Process the first task through the first test model combination to obtain the first processing result; the first test model combination includes the test model and the test agent framework.

[0318] Step S402: If the first processing result meets the test conditions, the model to be tested, the intelligent agent framework to be tested, the benchmark model, and the benchmark intelligent agent framework are combined in pairs to obtain multiple combinations of second models to be tested.

[0319] Step S403: Under the same task scenario and the same tool configuration, the second task is processed by multiple combinations of the second test models to obtain multiple second processing results.

[0320] Step S404: Based on the quality information of the processing results corresponding to each combination of the second models to be tested, determine the first adaptation difference information, the second adaptation difference information, and the third adaptation difference information.

[0321] Step S405: Based on the first adaptation difference information, the second adaptation difference information and the third adaptation difference information, determine the test result of the first combination of models to be tested.

[0322] Wherein, the first adaptation difference information represents the change in the quality information of the processing result caused by the model under test; the second adaptation difference information represents the change in the quality information of the processing result caused by the intelligent agent framework under test; and the third adaptation difference information represents the change in the quality information of the processing result caused by the combined interaction between the model under test and the intelligent agent framework under test.

[0323] For detailed procedures of steps S401-S405, please refer to the relevant descriptions of steps S301-S305 above, which will not be repeated here.

[0324] Step S406: Obtain auxiliary information collected during the processing of the second task.

[0325] In this embodiment, during the processing of the second task by each combination of second models under test, the running data and task result information can be monitored and recorded. Auxiliary information may include, but is not limited to, at least one of the following: task completion status, tool invocation status, execution steps, time consumption, resource consumption, and error information.

[0326] Task completion status can be used to reflect whether the second model combination under test successfully completed the second task, and the quality level of completion. For example, task completion status can include whether the task was completed, the accuracy rate, and the completeness. Task completion status can be derived from the quality evaluation results of the task results, such as the values ​​of evaluation indicators such as task completion accuracy and task completeness, or it can be directly derived from the comparison results between the expected output and the actual output of the second task.

[0327] Tool call information can be used to reflect information about external tools called by the second model combination under test during the execution of the second task. For example, tool call information may include the total number of tool calls, the number of successful calls, the number of failed calls, the tool call success rate, whether the call parameters conform to the specifications, whether there are any missing or incorrect parameters, and whether the tool returns normal results.

[0328] Execution steps can be used to reflect the complete execution trajectory of the second task from receipt to completion (or interruption). For example, execution steps may include the number of rounds of task execution, the type of action of the framework in each round (e.g., inference, tool invocation, result feedback), the input-output summary of each action, and at which stage the task ends. Execution steps can be used to reconstruct the complete interaction process between the model under test and the framework of the agent under test.

[0329] The time consumed can be used to reflect the time resources consumed by the second model combination in processing the second task. For example, the time consumed may include the total end-to-end time consumed, the time consumed in a single model inference, the time consumed in a single tool execution, the time consumed in result feedback and final answer generation, etc., and may also include the average time consumed and the fluctuation of multiple runs (corresponding to the evaluation results of the stability task package).

[0330] Resource consumption can be used to reflect the computing resources used by the second model under test in processing the second task. For example, resource consumption may include CPU utilization, peak memory usage, peak GPU memory usage, and disk read / write activity. Resource consumption information can be obtained from system monitoring components or collected by resource statistics interfaces provided by the operating environment.

[0331] Error messages can be used to record exceptions and failures that occur during the execution of the second task. For example, error messages may include the processing node where the error occurred, the error type (e.g., incorrect request format, response timeout, parsing failure, invalid parameters, tool execution exception, context truncation, etc.), the corresponding original log segment, the number of failures, and their frequency. Error messages can correspond to the failure record information (E) in the link adaptation fingerprint, or they can serve as supplementary textual descriptions of E.

[0332] Step S407: Based on the first adaptation difference information, the second adaptation difference information, the third adaptation difference information, the auxiliary information, and the adaptation data corresponding to the first model under test and each of the processing nodes, output the difference localization result.

[0333] The difference localization results are used to indicate that the adaptation differences between the model under test and the intelligent agent framework under test originate from at least one of the following:

[0334] The response capability of the model under test;

[0335] The scheduling capability of the intelligent agent framework under test;

[0336] Tool call chain;

[0337] The matching between the model under test and the framework of the intelligent agent under test.

[0338] In this embodiment, the first adaptation difference information (ΔM) can be used to reflect the change in the quality of the processing results caused by the model under test itself. When ΔM is negative and has a large absolute value, it indicates that under the same baseline agent framework, the model under test has reduced the quality of the processing results compared to the baseline model. Accordingly, the source of the adaptation difference can be attributed to the response capability of the model under test, that is, the processing capability of the model under test itself in the second task is insufficient.

[0339] The second adaptation difference information (ΔF) can be used to reflect the changes in the quality of processing results caused by the framework of the agent under test itself. When ΔF is negative and has a large absolute value, it indicates that under the same baseline model conditions, the quality of processing results of the agent under test is reduced compared to the baseline agent framework. Based on this, the source of the adaptation difference can be pointed to the scheduling ability of the agent under test, that is, the scheduling ability of the agent under test itself is insufficient in the second task.

[0340] The third adaptation difference information (I) can be used to reflect the changes in the quality of processing results caused by the combined interaction between the model under test and the intelligent agent framework under test. When I is significantly negative, it indicates that neither the model under test nor the intelligent agent framework under test shows obvious abnormalities when paired with the benchmark component alone, but the combination of the two produces additional performance loss. Based on this, the source of the adaptation difference can be pointed to the matching between the model under test and the intelligent agent framework under test, that is, there is a problem in the interaction connection between the two.

[0341] In this embodiment, based on the source directions indicated by the three adaptation difference information mentioned above, the specific content of the difference localization result can be further refined based on auxiliary information and adaptation data. For example, regarding the response capability of the model under test, the judgment can be made by combining the task completion status (such as answer accuracy and completeness), error information (such as model response timeout, garbled return, no valid output) in the auxiliary information, and the adaptation data M corresponding to the model output response node. If the task completion status is poor and the M component is low, and the error information is concentrated on the model side (such as unstable response, incorrect output format), then it can be further confirmed that the source of the difference is the insufficient response capability of the model under test.

[0342] To assess the scheduling capability of the agent framework under test, we can combine the execution steps in the auxiliary information (such as whether the planning is reasonable, whether there are invalid loops, and whether the timing of tool invocation is appropriate), error messages (such as framework parsing failure and scheduling logic anomalies), and the adaptation data O corresponding to the framework's output nodes for judgment. If the execution steps show that the framework has anomalies in the multi-step planning or action decision-making stages, and the O component is significantly lower in the combination (M,F), then we can further confirm that the source of the difference is insufficient scheduling capability of the agent framework under test.

[0343] For tool call chains, the judgment can be made by combining the tool call information in the auxiliary information (such as tool call success rate, parameter validity, number of tool execution failures) with the tool call parameter validity component A and the tool execution success component T. If the tool call success rate is low, and the value of A or T is zero or significantly low, and the error information points to the tool side (such as incorrect parameter type, tool execution timeout), then the source of the difference can be determined to be the tool call chain, and it can be further located in the parameter construction stage or the tool execution stage.

[0344] For the matching between the model under test and the intelligent agent framework under test, when the third matching difference information I exceeds the target difference range, the matching data can be compared one by one by combining the execution steps and error information in the auxiliary information, as well as the matching data in the link matching fingerprint (such as O corresponding to the framework receiving output node, C corresponding to the result return node, and F corresponding to the model generating the final answer node). If the values ​​of O, C, and F in the combination (M,F) are significantly lower than the corresponding values ​​in the combination (M,F0) and combination (M0,F), it indicates that the corresponding processing node has an additional decrease when the model and framework are used together compared to when used alone. Based on this, the source of the difference can be refined and located to the specific connection link, such as "the tool call format output by the model does not match the parsing rules of the framework" (corresponding to O), "the context of the tool result returned to the model is incomplete" (corresponding to C), "the final answer does not fully absorb the tool result" (corresponding to F), etc.

[0345] In this embodiment, the output format of the differential localization results may include, but is not limited to, at least one of the following:

[0346] Text format. Describe the differences in natural language, for example, "The overall performance decline is mainly due to model factors (ΔM=-8.0), and it is recommended to replace the model under test; there is also a slight negative co-adaptation (I=-4.0), and it is recommended to check the matching between the model output format and the framework parsing rules."

[0347] Structured data format. Output difference localization results in key-value pairs or data tables. Fields can include difference source category (model / framework / toolchain / fitting match), corresponding fitting difference information value (ΔM / ΔF / I), impact level (dominant / minor / negligible), involved processing nodes (such as framework output nodes), anomaly type, and optimization suggestions, etc., to facilitate automatic program parsing and subsequent processing.

[0348] The test results of the first model combination under test can be presented in the form of radar charts or bar charts. The charts can include dimensions such as compatibility (corresponding to link closure L), end-to-end completion rate (corresponding to task completion status), latency (corresponding to time consumption), resource consumption (corresponding to resource usage), stability (corresponding to time consumption or score fluctuations across multiple runs), and error rate (corresponding to the proportion of failures in error messages out of the total number of operations). Each dimension is mapped to one or more adaptation data points in the link adaptation fingerprint, or to corresponding indicators in the auxiliary information. Through intuitive comparison of multi-dimensional graphs, users can quickly identify in which dimensions the first model combination under test lags significantly behind the baseline combination (M0, F0), thus transforming the difference localization results from a numerical level into intuitive graphical diagnostic conclusions.

[0349] In this embodiment, three sets of adaptation discrepancy information provide quantitative indications of adaptation discrepancies across three dimensions: model, framework, and interaction adaptation. Auxiliary information supplements this quantitative indication from the perspective of actual operational data, while adaptation data precisely links the adaptation discrepancy to specific processing nodes in the target task execution chain. These three elements corroborate each other, enabling the discrepancy localization results to not only indicate whether the adaptation discrepancy originates from model response capability, framework scheduling capability, tool call chain, or adaptation discrepancy, but also to further provide the location of the processing node associated with the indicated source, the anomaly type, and optimization suggestions. Therefore, users can directly understand the nature, location, and improvement direction of the adaptation problem from the discrepancy localization results, improving the efficiency and accuracy of the automated testing method in diagnosing model-agent framework adaptation problems in actual deployment.

[0350] In this embodiment, combined with Figure 4 The data processing architecture shown illustrates the overall process of the data processing method. Figure 4 The system shown comprises three progressively larger modules: Module 1 is the link verification and adaptation fingerprint module; Module 2 is the admission screening and comparative evaluation module; and Module 3 is the cause analysis, attribution, and problem localization module. These three modules work together to complete the entire automated testing process, from link verification, admission screening, comparative evaluation to cause analysis, attribution, and problem localization.

[0351] Module 1: Link Verification and Adaptation Fingerprint.

[0352] The system connects the model under test to the intelligent agent framework under test, establishing a runtime connection between the two to form the first model combination under test. The intelligent agent framework under test initiates a first task containing task requirements, tool constraints, and context information, recording the complete execution chain from task input to final response output. The system not only determines whether the chain is interrupted, but also, following the method described in step S1012 above, determines the corresponding adaptation data for each processing node on the target task execution chain of the first model combination under test, and combines them to obtain the chain adaptation fingerprint Φ=(R, M, O, A, T, C, F, E). Among them, R, M, O, A, T, C, and F are adaptation data for dimensions such as framework request adaptation, model response stability, output structure matching with framework action acceptance, tool call parameter legality, tool execution success rate, tool result feedback context integrity, and the degree to which the final response absorbs the tool result, respectively. E is the record of failure links and error types. The determination method of the above adaptation data can be found in the relevant description of step S1012 above, and will not be repeated here.

[0353] Based on this, following the method described in step S1013 above, the link closure degree L = R × M × O × A × T × C × F is calculated based on each continuous adaptation data. If the adaptation data of any key node is zero, the link closure degree L is zero, the link is not closed, and this combination is excluded from subsequent formal evaluation; when the link closure degree L is greater than zero, the link is closed, and the combination meets the conditions for entering the formal evaluation. Since each adaptation data is a continuous value between 0 and 1, the magnitudes of the link closure degrees L of different combinations are comparable and can be used to measure the degree of link adaptation between different combinations.

[0354] After introducing the link adaptation fingerprint Φ, the collaborative adaptation status of the combination can be described in a structured and quantifiable way, providing a more refined input basis for the combination admission and comparison evaluation in Module 2 and the cause analysis and attribution in Module 3. At the same time, the link interruption link can be accurately located by using the failure record component E and the position of zero in each adaptation data, avoiding general attribution.

[0355] Module Two: Admission Screening and Comparative Evaluation.

[0356] Admission is determined based on the link adaptation fingerprint Φ and link closure degree L output by Module 1: combinations with a link closure degree L of zero are excluded after the interruption is recorded and do not enter the formal evaluation; combinations with a link closure degree L of greater than zero carry their link adaptation fingerprint Φ and enter the formal evaluation.

[0357] The process of excluding interrupted links can be understood as follows: based on the processing nodes corresponding to the adaptation data with a value of zero in the link adaptation fingerprint Φ, determine the specific link where the link is interrupted (e.g., M=0 corresponds to a model-side problem, O=0 corresponds to an adaptation problem between model output and framework parsing, A=0 or T=0 corresponds to a tool call link problem, C=0 corresponds to a context return problem, and F=0 corresponds to a final answer connection problem). Record the interrupted link and the failure record E as the reason why the combination is unusable. The combination will no longer participate in subsequent evaluations, but the record will be retained for review.

[0358] The first type of task is executed by the second set of test models that have all passed the admission criteria, providing a unified performance comparison benchmark between different sets. The second type of task is selectively assigned based on whether the corresponding component in the link adaptation fingerprint Φ of the set is zero. Specifically, based on the values ​​of each adaptation data in the link adaptation fingerprint Φ, the adaptation status of each set in the corresponding capability dimension is determined, and the set with the corresponding capability is assigned the matching second type of task.

[0359] Following the admission screening, a baseline model and a baseline agent framework are introduced. Four combinations of second test models are evaluated under the same task scenario and tool configuration. These combinations are: (M,F) consisting of the test model and the test framework; (M,F0) consisting of the test model and the baseline framework; (M0,F0) consisting of the baseline model and the test framework; and (M0,F0) consisting of the baseline model and the baseline framework. After each combination processes the second task, its processing result quality information is determined according to steps S1032-S1034, namely S(M,F), S(M,F0), S(M0,F), and S(M0,F0), which serve as inputs for the factorial attribution in module three. The method for determining the processing result quality information can be found in the relevant descriptions of steps S1032-S1034 above, and will not be repeated here.

[0360] By introducing the baseline combination (M0, F0), a complete 2×2 factorial design is formed, which allows the main effects of the model, the main effects of the framing, and the effects of co-fitting to be mathematically separated, rather than relying on human experience. Meanwhile, the component-driven task packet skipping mechanism based on the link fit fingerprint Φ avoids invalid evaluations of known misfitting links, thus improving evaluation efficiency.

[0361] Module 3: Cause Analysis and Problem Localization.

[0362] Based on the processing result quality information S(M,F), S(M,F0), S(M0,F), and S(M0,F0) from the four combinations output by Module 2, the model main effect, frame main effect, and co-adaptation effect are calculated respectively according to the aforementioned step S1042: Model main effect ΔM = S(M,F0) - S(M0,F0), used to reflect the change in processing result quality information caused by changing the model under a fixed baseline frame condition; Frame main effect ΔF = S(M0,F) - S(M0,F0), used to reflect the change in processing result quality information caused by changing the frame under a fixed baseline model condition; Co-adaptation effect I = S(M,F) - S(M,F0) - S(M0,F) + S(M0,F0), used to reflect the additional change in processing result quality information caused by the combined interaction between the test model and the test agent frame. The calculation process and judgment rules for the above adaptation difference information can be found in the relevant introduction of steps S1042 to S1043 above, and will not be repeated here.

[0363] Attribution conclusions: When the co-adaptation effect I is close to zero, it indicates that the tested model and the tested agent framework act independently on the processing results, and the combined performance can be explained by the superposition of the model main effect and the framework main effect, with no significant synergy between the two; when I is significantly greater than zero, it indicates that there is a positive co-adaptation between the tested model and the tested agent framework, and the quality of the combined processing results is better than the level expected by the superposition of the model main effect and the framework main effect; when I is significantly less than zero, it indicates that there is a negative co-adaptation between the tested model and the tested agent framework, and the quality of the combined processing results is worse than the level expected by the superposition of the model main effect and the framework main effect.

[0364] Based on this, when the collaborative adaptation impact I significantly deviates from zero (i.e., exceeds the target difference range), further following the methods described in steps S306 to S307 above, and combining the differences in the link adaptation fingerprint Φ output by module one and the link closure degree L, the link adaptation fingerprint of the first model combination (M,F) to be tested is compared one by one with the link adaptation fingerprints of combination (M,F0), combination (M0,F), and combination (M0,F0) to locate which specific step occurs at which processing node in the connection between the model and the framework (such as the framework receiving the output node, the result feedback node, or the model generating the final answer node), and a second prompt message is generated. For the specific location process, please refer to the relevant descriptions of steps S306 to S307 above, which will not be repeated here.

[0365] In addition, following the steps S406-S407 described above, auxiliary information collected during the processing of the second task (including at least one of the following: task completion status, tool invocation status, execution steps, time consumption, resource consumption, and error information) can be obtained. Based on the first adaptation difference information ΔM, the second adaptation difference information ΔF, the third adaptation difference information I, the auxiliary information, and the adaptation data corresponding to each processing node, the difference localization result is output. The difference localization result is used to indicate whether the adaptation difference originates from the response capability of the model under test, the scheduling capability of the intelligent agent framework under test, the tool invocation chain, or the matching between the model under test and the intelligent agent framework under test. For the specific output method, please refer to the relevant descriptions of the steps S406-S407 described above, which will not be repeated here.

[0366] After completing the above cause analysis and difference localization, a test report can be generated based on the attribution conclusions and localization results. The test report can include two parts: problem attribution and problem localization. The problem attribution part summarizes the values ​​of the first adaptation difference information ΔM, the second adaptation difference information ΔF, and the third adaptation difference information I, along with the categories of difference sources they indicate (model factors, framework factors, co-adaptation factors). The problem localization part indicates the location of the processing node that generated the interaction difference, the type of adaptation anomaly, and the corresponding optimization suggestions. The specific format of the test report can include a text report, a structured data report, or a visual chart report; this embodiment does not limit this.

[0367] In summary, module one verifies the link and outputs the link adaptation fingerprint Φ and the link closure degree L. Module two performs admission screening based on Φ and L and organizes controlled comparison evaluation with benchmark combinations. Module three performs cause analysis, attribution and problem localization based on the processing result quality information of the four combinations and the link adaptation fingerprint Φ. The three modules work in sequence to jointly realize the automatic testing of the combination of the model under test and the intelligent agent framework under test, which is verifiable, comparable and attributable.

[0368] Next, the data processing method will be explained in conjunction with the automated testing system. The data processing method involved in this embodiment relies on an automated evaluation system, which can be physically deployed into two parts: a backend evaluation engine and an interactive interface. The interactive interface is the front-end software that users can directly operate. This interactive interface can be a standalone desktop client, a web-based system running in a browser, or a visualization component embedded in other development platforms. Users (such as model selection engineers, intelligent agent framework developers, or testers) issue evaluation commands to the backend evaluation engine and receive and view the evaluation process and results by operating the buttons, menus, or cards on the interactive interface. The user's operations in the interactive interface run throughout the entire process: first, configure the model and framework to be tested; then, start the link verification; next, view the comparison results in the admission evaluation; and finally, view the problem location and visualization report.

[0369] Before the test begins, the user first needs to select and configure the object to be tested in the interactive interface. Specifically, such as... Figure 5 As shown, the interactive interface provides a model card library. Users can select a model to be tested from the library by clicking the "Model Card" option; alternatively, they can create a new model card from the library to serve as the model to be tested. The model card can record information such as the model's name, type, deployment method, and capability declaration, which will be used for configuration and display in subsequent evaluation tasks. After selecting a model, the user continues to configure the framework of the agent to be tested.

[0370] like Figure 6As shown, the interactive interface also provides a framework card library. Users can select the intelligent agent framework to be tested from the library by clicking the "Frame Card" option; alternatively, they can create a new intelligent agent framework from the library by clicking "Create Framework Card". The framework card can record information such as the framework name, scheduling strategy, supported tool types, and model access specifications. After the model and intelligent agent framework to be tested are configured, the backend will automatically generate the test combination based on the user's selected model and framework. The user can then click "Start Link Verification" to initiate the subsequent processing flow.

[0371] The link verification phase corresponds to module one mentioned above. For example... Figure 7 As shown, after the link verification is completed, the interactive interface can display the overall results of the link verification. For example, the total number of combinations is 20, the number of combinations that passed the link verification is 16, and the number of combinations that were interrupted is 4. The interface can also summarize and display the interruption types, such as model-side problems, adaptation problems, toolchain problems, and context backhaul problems. Users can view them by clicking on the category tags such as "All", "Pass (16)", and "Interrupted (4)". If you click "Pass (16)", you can see the 16 combinations that passed the link verification. Taking the combination "Qwen2.5-Coder×OpenClaw" as an example, the link status corresponding to this combination is displayed as closed, and each processing node on the link is displayed as a green checkmark, indicating that each processing node is executing normally.

[0372] To further pinpoint the specific cause of the link interruption, users can view detailed link information for the interrupted combination. For example... Figure 8 As shown, if you click "Interruption (4)", you can see four combinations of link interruptions. For example, the combination "Llama-3.3×AutoGen" is displayed as a link interruption, and the interruption type is adaptation problem. In the link status display of this combination, some processing nodes are displayed as green checkmarks, some processing nodes are displayed as red crosses, and the processing nodes after the red crosses are displayed as gray, indicating that there has been a link interruption before this node, and the subsequent links have not been executed.

[0373] When the user further clicks on the combined details of the interrupted link, such as... Figure 9 As shown, the interactive interface displays the link adaptation fingerprint Φ of this combination. Among them, framework request adaptation R, model response stability M, tool parameter validity A, tool execution success T, context return C, and final answer absorption F are all displayed in green, with corresponding non-zero values; output structure matching O is displayed in red, with a corresponding value of 0. Correspondingly, the link closure degree L is 0.

[0374] Meanwhile, the interface can display the execution status of each processing node in the target task execution chain one by one: the task input node shows a green checkmark, indicating successful execution; the framework generating request node shows a green checkmark, indicating successful execution; the model output response node shows a green checkmark, indicating that the model returned a response, but the tool's call field format is non-standard; the framework receiving output node shows a red cross, indicating that the AutoGen parser threw a JSONDecodeError and could not extract the tool parameters; the tool execution node, result feedback node, and model generating final answer node are all displayed as gray symbols "-", indicating that the preceding steps were interrupted and the above nodes were skipped. Based on the above information, the interactive interface can output the reason for the interruption. For other combinations of chain interruptions, the same method can be used for display and explanation, without further explanation.

[0375] After the link verification is completed, the verified combinations enter the admission evaluation stage, which corresponds to Module Two mentioned above. For example... Figure 10 As shown, when a user clicks "Admission Evaluation," the interactive interface displays the number of admitted combinations (16) and the number of rejected combinations (4). In the admission evaluation, the backend uses a controlled variable control structure to organize the evaluation. The interface can display this control structure, including: fixed frame, different model; fixed model, different frame; fixed model and frame, different tools. Simultaneously, the interface can display the 2×2 factorial design of the baseline control: Let M0 represent the baseline model and F0 represent the baseline agent frame. The test model M and the test agent frame F are paired with the baseline model M0 and the baseline agent frame F0, respectively, resulting in four combinations: (M,F), (M,F0), (M0,F), and (M0,F0), with corresponding scores of S(M,F), S(M,F0), S(M0,F), and S(M0,F0).

[0376] The interface can simultaneously provide the calculation formulas for the main effects of the model, the main effects of the frame, and the co-adaptation effects: the main effects of the model ΔM = S(M,F0) - S(M0,F0); the main effects of the frame ΔF = S(M0,F) - S(M0,F0); and the co-adaptation effects I = S(M,F) - S(M,F0) - S(M0,F) + S(M0,F0).

[0377] To ensure the fairness and relevance of the comparative evaluation, the backend also needs to allocate tasks during the admission evaluation phase, such as... Figure 11As shown, the interactive interface displays the task allocation logic. The first type of task (i.e., the basic task package) is executed by all combinations, including tasks such as code modification and tool calls. The second type of task (i.e., specialized task packages) is skipped based on whether the corresponding component (i.e., adaptation data) in the link adaptation fingerprint Φ is zero; these include tasks such as browser operations and long document responses. The interface simultaneously displays the number of admitted combinations (16) and the number of disadvantaged combinations (4). For the 16 admitted combinations, each combination will undergo evaluation according to the above task allocation scheme, and a corresponding evaluation record will be generated.

[0378] After the evaluation is completed, users can view detailed evaluation results for each combination through the interactive interface. Specifically, for the 16 approved combinations, users can view their respective evaluation matrices on the interface. For example, if the combination "DeepSeek-V3×LangGraph@Code Modification" is clicked, the interactive interface can display something like this: Figure 12 Details shown:

[0379] The model card is DeepSeek-V3, the framework card is LangGraph, the tool configuration is a code toolset, the task package is code modification, and the task package is marked as belonging to a specific capability task.

[0380] Building upon this, the evaluation details page can further display attribution and localization results, corresponding to Module 3 mentioned above. This page can display attribution summaries, outputting the causal analysis results in text form, or display the proportions of model contribution, framework contribution, and synergistic contribution in chart form, such as model contribution 55%, framework contribution 25%, and synergistic contribution 20%.

[0381] The four dimensions of attribution are used to illustrate the four sources of differential localization results: model factors, framework factors, toolchain factors, and co-adaptation factors.

[0382] The interface can also display the attribution reasoning path, showcasing the steps involved in deriving attribution judgments. The combined synergy index can display the values ​​of ΔM, ΔF, and I.

[0383] Link adaptation fingerprints can display the value distribution of each component R, M, O, A, T, C, and F in a polygon (radar chart).

[0384] The model-framework capability comparison can be displayed using polygons (radar charts) to show capability dimensions such as answer quality, format / trajectory, legend / scheduling, instruction compliance, and coding / retry. Answer quality reflects the degree to which the final answer meets the task requirements and is related to the aforementioned processing result quality information S(M,F). Format / trajectory reflects the degree of matching between the model's output and the framework's parsable format, corresponding to output structure matching O and context return C in the link adaptation fingerprint Φ. Legend / scheduling reflects the framework's ability to accept model output and orchestrate subsequent tool calls, corresponding to framework request adaptation R and tool execution success T. Instruction compliance reflects the model's adherence to task requirements and tool constraints, corresponding to model response stability M and tool call parameter validity A. Coding / retry reflects the model's error correction and retry capabilities when tool execution fails, corresponding to tool execution success T and final answer absorption F. By displaying these capability dimensions as polygons, users can intuitively compare the strengths and weaknesses of different model and framework combinations in various capabilities.

[0385] In addition, the evaluation details page also displays evaluation metrics such as compatibility (86%), end-to-end completion rate (61%), latency (1.59s), resource consumption (80.78GB), stability (81%), and error rate (39%), intuitively reflecting the combination's performance across multiple dimensions. Compatibility can be expressed as a percentage of link closure degree L, measuring the overall adaptability of the combination at the link level; end-to-end completion rate, stability, and error rate correspond to task completion status in the auxiliary information, reflecting the overall success rate and reliability of the combination in executing the second task; latency corresponds to the time consumed in the auxiliary information, reflecting the response efficiency of the combination from task input to final output; and resource consumption corresponds to resource consumption in the auxiliary information, reflecting the computational and storage overhead required for the combination's operation. These metrics allow users not only to know the combination's adaptation results but also to further understand its performance and resource consumption in actual operation.

[0386] Through the aforementioned automated testing system, users can complete the entire evaluation process from link verification, admission screening, control evaluation to cause analysis and problem localization by following the sequence of configuring the model and framework → performing link verification and viewing pass / fail details → entering admission evaluation and viewing the controlled control structure → viewing task allocation → viewing combined evaluation results and attribution localization. Throughout the process, users only need to click and view operations on the interface to drive the backend to complete all data processing steps, and quickly locate the adaptation issues between the model under test and the intelligent agent framework under test through visualization. This provides an intuitive and interpretable decision-making basis for model selection, framework selection, and optimization of their combination.

[0387] It should be noted that, Figures 5 to 12The interface shown is merely a visualization of this embodiment using specific example data. The specific information involved in the figure, such as model name, framework name, link status symbol, task package type, attribution percentage, and evaluation metric values, are exemplary content set up to intuitively demonstrate the aforementioned evaluation process. Those skilled in the art can understand the correspondence between each processing stage on the interactive interface based on this, and adjust the specific names, values, and display formats according to actual evaluation needs. For other auxiliary displays not separately explained in the text, their meaning can be directly determined from the illustrations and does not affect the understanding of the solution in this embodiment; therefore, they will not be elaborated upon here.

[0388] The electronic device provided in this application will be described below. The electronic device described below can be referred to in correspondence with the data processing method described above.

[0389] Electronic devices may include:

[0390] Memory is used to store computer programs;

[0391] The processor is used to execute the computer program to enable the electronic device to perform the following method steps:

[0392] The first task is processed by the first combination of models to be tested to obtain a first processing result; the first combination of models to be tested includes a model to be tested and a framework of intelligent agent to be tested.

[0393] If the first processing result meets the test conditions, multiple combinations of second models to be tested are generated based on the model to be tested, the intelligent agent framework to be tested, the benchmark model, and the benchmark intelligent agent framework.

[0394] By combining multiple second models under test to process the second task, multiple second processing results are obtained.

[0395] Based on multiple results of the second processing, the test results of the first combination of models to be tested are determined.

[0396] It should be noted that the specific implementation of the above method steps can be referred to the description in the aforementioned method embodiments, and will not be repeated here.

[0397] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0398] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0399] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0400] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A data processing method, comprising: The first task is processed by combining the first model under test to obtain the first processing result; The first model combination to be tested includes a model to be tested and a framework of intelligent agents to be tested; If the first processing result meets the test conditions, multiple combinations of second models to be tested are generated based on the model to be tested, the intelligent agent framework to be tested, the benchmark model, and the benchmark intelligent agent framework. By combining multiple second models under test to process the second task, multiple second processing results are obtained. Based on multiple results of the second processing, the test results of the first combination of models to be tested are determined.

2. The data processing method according to claim 1, wherein the step of processing the first task through a first test model combination to obtain a first processing result includes: Obtain the target task execution chain, which includes multiple processing nodes with sequential dependencies; By combining the first model under test, the first task is processed based on the target task to obtain adaptation data corresponding to each processing node; the adaptation data characterizes the degree of adaptation between the model under test and the intelligent agent framework under test on the corresponding processing node. Based on the adaptation data corresponding to the multiple processing nodes, a first processing result is determined.

3. The data processing method according to claim 2, further comprising: If, based on the first processing result, it is determined that the adaptation data corresponding to each processing node meets the adaptation conditions, then the first processing result is determined to meet the test conditions.

4. The data processing method according to claim 2, further comprising: If at least one of the processing nodes has adaptation data that does not meet the adaptation conditions, a first prompt message is generated based on the adaptation data of the processing node that does not meet the adaptation conditions. The first prompt message is used to indicate the type and location of the adaptation exception.

5. The data processing method according to claim 1 or 2, wherein generating a plurality of second test model combinations based on the test model, the test agent framework, the benchmark model, and the benchmark agent framework includes: The test model, the test agent framework, the benchmark model, and the benchmark agent framework are combined in pairs to obtain multiple combinations of second test models. By combining multiple second models under test to process the second task, multiple second processing results are obtained, including: Under the same task scenario and the same tool configuration, multiple combinations of the second test models are used to process the second task, resulting in multiple second processing results.

6. The data processing method according to claim 5, wherein the second task includes a first type of task and a second type of task; The second task is processed by combining multiple second models under test to obtain multiple second processing results, including: The first type of task is processed by all the second test model combinations to obtain the first type of processing result corresponding to each second test model combination. The first type of processing result is used for performance comparison between different second test model combinations. From a plurality of second test model combinations, determine the second test model combination that has the specified capability; The second type of task is processed by the combination of the second test models with the specified capabilities to obtain a second type of processing result, which is used to determine the performance of the combination of the second test models with the specified capabilities on the specified capabilities.

7. The data processing method according to claim 5, wherein each of the second processing results includes: Quality information of the processing results corresponding to each combination of the second model under test; The quality information of the processing result characterizes the quality of the second test model combination in completing the second task; The determination of the test result of the first combination of models under test based on multiple second processing results includes: Based on the quality information of the processing results corresponding to each combination of the second models to be tested, the first adaptation difference information, the second adaptation difference information, and the third adaptation difference information are determined. Based on the first adaptation difference information, the second adaptation difference information, and the third adaptation difference information, the test result of the first combination of models to be tested is determined; Wherein, the first adaptation difference information represents the change in the quality information of the processing result caused by the model under test; the second adaptation difference information represents the change in the quality information of the processing result caused by the intelligent agent framework under test; and the third adaptation difference information represents the change in the quality information of the processing result caused by the combined interaction between the model under test and the intelligent agent framework under test.

8. The data processing method according to claim 7, further comprising: If the third adaptation difference information exceeds the target difference range, based on the adaptation data corresponding to the first model combination under test and each of the processing nodes, determine the processing nodes that generate interaction differences between the model under test and the intelligent agent framework under test in processing the second task. Based on the processing node that generates the interaction difference, a second prompt message is generated.

9. The data processing method according to claim 7, further comprising: Acquire auxiliary information collected during the processing of the second task, the auxiliary information including at least one of the following: task completion status, tool call status, execution steps, time consumption, resource consumption, and error information; Based on the first adaptation difference information, the second adaptation difference information, the third adaptation difference information, the auxiliary information, and the adaptation data corresponding to the first model combination under test and each of the processing nodes, the difference localization result is output; The difference localization results are used to indicate that the adaptation differences between the model under test and the intelligent agent framework under test originate from at least one of the following: The response capability of the model under test; The scheduling capability of the intelligent agent framework under test; Tool call chain; The matching between the model under test and the framework of the intelligent agent under test.

10. An electronic device, comprising: Memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to perform the following method steps: The first task is processed by the first combination of models to be tested to obtain a first processing result; the first combination of models to be tested includes a model to be tested and a framework of intelligent agent to be tested. If the first processing result meets the test conditions, multiple combinations of second models to be tested are generated based on the model to be tested, the intelligent agent framework to be tested, the benchmark model, and the benchmark intelligent agent framework. By combining multiple second models under test to process the second task, multiple second processing results are obtained. Based on multiple results of the second processing, the test results of the first combination of models to be tested are determined.

Images