End-to-end AI agent evaluation method, system, equipment and medium

Abstract

The invention provides an end-to-end AI agent evaluation method, system and device and a medium, and the method comprises the steps: receiving a CSV file uploaded by a user, analyzing the CSV file into structured data, and writing the structured data into an intermediate storage through a preset transmission mechanism; calling each model API in a preset model list in batches, and processing the structured data in real time in combination with a streaming data processor and a multi-modal data processor to obtain a processing result; and carrying out multi-dimensional performance index calculation on a processing result based on a preset extensible evaluation system, realizing transverse comparative analysis of models of different scales through a dynamic column generation technology in combination with automatic quality scoring, and finally integrating multi-dimensional performance data, quality scores and a comparison result to generate a structured evaluation report. According to the method, comprehensive, accurate and efficient evaluation of the AI agent in multiple scenes is realized, and a reliable decision basis is provided for optimization and deployment of the AI agent.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an end-to-end AI agent evaluation method, system, device and medium. Background Technology

[0002] Existing AI agent evaluation systems face four core problems: (1) Limited evaluation dimensions and insufficient scenario adaptability. Current evaluations rely heavily on single models and prompts, failing to cover complex multi-dimensional evaluation needs and lacking comprehensive assessment capabilities; (2) Data processing efficiency and scalability bottlenecks. Systems are inadequate in batch processing capabilities and have poor configuration flexibility, making it difficult to handle large-scale and diverse evaluation tasks; (3) Lack of real-time and multimodal support. Limited real-time data stream processing capabilities result in performance bottlenecks. Evaluations primarily focus on text, lacking effective support for multi-source data types such as images and videos, thus limiting the expansion of application scenarios; (4) Weak result analysis and insight capabilities. Insufficient in-depth mining of performance indicators and generation of visual reports, failing to provide strong decision-making basis for AI agent optimization. In summary, existing AI evaluation systems have technical deficiencies in five key areas: evaluation methods, data processing, architecture design, functional support, and result analysis, making it difficult to meet the complexity, diversity, real-time requirements, multimodal nature, and industrial-grade needs of AI agent application scenarios. Therefore, there is an urgent need for an effective end-to-end AI agent evaluation method to solve the above problems. Summary of the Invention

[0003] In view of the above problems, the present invention is proposed to provide an end-to-end AI agent evaluation method, system, device and medium that overcomes or at least partially solves the above problems.

[0004] To achieve the above and other related objectives, the present invention provides an end-to-end AI agent evaluation method, the method comprising: Receive CSV files uploaded by users, parse them into structured data, and write the structured data into intermediate storage through a preset transmission mechanism; The APIs of each model in the preset model list are called in batches, and the structured data is processed in real time by combining the streaming data processor and the multimodal data processor to obtain the processing results. Based on a pre-defined scalable evaluation system, the processing results are used to calculate multi-dimensional performance indicators. Combined with automated quality scoring, the results are used to achieve horizontal comparative analysis of models of different scales through dynamic column generation technology. Finally, multi-dimensional performance data, quality scores and comparison results are integrated to generate a structured evaluation report.

[0005] Optionally, before the step of receiving the user-uploaded CSV file and parsing it into structured data, the method further includes: Configure the AI ​​agent runtime environment, including specifying API endpoints, setting the model list, configuring user IDs, setting request timeouts, and defining the preset maximum number of retries; Build and maintain a prompt word template library; Configure the parameters and timeout thresholds output by the control model; Define a scalable evaluation system to clarify the various evaluation dimensions and standards on which the evaluation is based.

[0006] Optionally, the batch invocation of model APIs from a preset model list, combined with a streaming data processor and a multimodal data processor, performs real-time processing on the structured data to obtain processing results, including: The system iterates through a pre-defined list of models and calls the APIs of each model in high concurrency and parallel operation according to a scheduling algorithm and a concurrency control strategy to process the structured data and obtain the corresponding API responses. The API response is cleaned, processed, and recorded, and the processed data is converted into a format suitable for streaming transmission. A real-time communication channel from the server to the client is established using SSE technology to transmit the converted data. A multimodal data processor is used on the server or client side to perform type identification and targeted processing on the converted data to obtain the processing result.

[0007] Optionally, after the step of traversing the preset model list and calling each model API in parallel with high concurrency according to the scheduling algorithm and concurrency control strategy, the method further includes: During API calls, the execution status of each call is monitored in real time; the execution status includes response time, status code, error type, resource usage, request parameters and response data; Once an API call failure is detected, the predefined exception handling and retry mechanism is immediately triggered.

[0008] Optionally, the exception handling and retry mechanism that is immediately triggered once an API call failure is detected includes: When an API call fails, an error log for the API call failure is recorded. If the current number of retries is less than the preset maximum number of retries, a retry log is recorded, and the delay time is calculated according to the exponential backoff strategy. After waiting for the specified delay, call the API again and increment the retry count by 1; If the retry fails and the number of retries reaches or exceeds the preset maximum number of retries, the retry process is terminated and an error object containing error information is returned. The error object includes the specific reason for the API call failure, the unique identifier of the original request, and the reset of unnecessary fields.

[0009] Optionally, the calculation of multi-dimensional performance indicators for the processing results based on a preset scalable evaluation system includes: Using a pre-defined scalable evaluation system, the processing results are analyzed in multiple dimensions using a performance indicator calculator, and multi-dimensional performance data is output in a quantitative manner; wherein, the multi-dimensional performance data includes the first token time and the complete response time.

[0010] Optionally, the process combines automated quality scoring with dynamic column generation technology to achieve cross-sectional comparative analysis of models of different scales. Ultimately, it integrates multi-dimensional performance data, quality scores, and comparison results to generate a structured evaluation report, including: The quality assessment engine, based on multi-dimensional scoring criteria, completes the hierarchical quality assessment of each model and outputs the quality score of each model through multi-model cross-validation, stratified sampling review and vertical domain adaptation technology. The comparison analyzer uses dynamic column generation technology to adapt to different numbers of comparison models, generating comparison results of multidimensional performance and quality indicators. The report generator aggregates multi-dimensional performance data, quality scores, and comparison results, generates a structured evaluation report through a template engine, and associates specific optimization suggestions based on comprehensive analysis.

[0011] Secondly, the present invention also provides an end-to-end AI intelligent agent evaluation system, the system comprising: The receiving module is used to receive CSV files uploaded by users, parse them into structured data, and write the structured data into intermediate storage through a preset transmission mechanism. The processing module is used to call the APIs of each model in the preset model list in batches, and combine the streaming data processor and the multimodal data processor to process the structured data in real time to obtain the processing results. The analysis module is used to calculate multi-dimensional performance indicators of the processing results based on a preset scalable evaluation system, and combined with automated quality scoring, to realize horizontal comparative analysis of models of different scales through dynamic column generation technology. Finally, it integrates multi-dimensional performance data, quality scores and comparison results to generate a structured evaluation report.

[0012] Thirdly, the present invention provides an electronic device comprising: a memory and a processor; the memory for storing a computer program; and the processor for executing the computer program stored in the memory to cause the electronic device to perform the steps of the end-to-end AI agent evaluation method as described above.

[0013] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by an electronic device, implements the steps of the end-to-end AI agent evaluation method as described above.

[0014] The above-described one or more technical solutions provided by this invention can have the following advantages or at least achieve the following technical effects: This invention achieves comprehensive, accurate, and efficient evaluation of AI agents in multiple scenarios through multimodal fusion processing, dynamic resource scheduling, scalable evaluation framework, and automated comparative analysis technology, providing a reliable decision-making basis for the optimization and deployment of AI agents. Attached Figure Description

[0015] Figure 1 The diagram shows a flowchart of an end-to-end AI agent evaluation method according to an embodiment of the present invention.

[0016] Figure 2 This is a schematic diagram of multi-dimensional configuration management in one embodiment of the present invention;

[0017] Figure 3 The diagram shows an API schematic for batch scheduling of multiple models in one embodiment of the present invention.

[0018] Figure 4 The diagram shown is a schematic diagram of streaming data processing in one embodiment of the present invention;

[0019] Figure 5 The diagram shown is a schematic diagram of the exception handling and retry mechanism in one embodiment of the present invention;

[0020] Figure 6 The diagram shows a functional module schematic of an end-to-end AI intelligent agent evaluation system according to an embodiment of the present invention.

[0021] Figure 7 The diagram shown is a schematic representation of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0022] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0023] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0024] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0025] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0026] Unless otherwise stated, the term "multiple" means two or more.

[0027] In this embodiment of the disclosure, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0028] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0029] The technical solutions of the present invention will now be described in detail with reference to the accompanying drawings.

[0030] The end-to-end AI agent evaluation method of this application can be applied to an end-to-end AI agent evaluation system. The end-to-end AI agent evaluation system can have a large model pre-set for AI agent evaluation. The end-to-end AI agent evaluation system can be set in a terminal or server that communicates with the AI ​​agent, or it can be set directly in the AI ​​agent. There is no limitation here.

[0031] Please see Figure 1 An embodiment of the present invention provides an end-to-end AI agent evaluation method, the method including the following steps S10~S30: Step S10: Receive the CSV file uploaded by the user, parse it into structured data, and write the structured data into intermediate storage through a preset transmission mechanism.

[0032] Among them, CSV files refer to the standard structured carriers used for batch input data in AI agent evaluation. They can store the core data of the evaluation task, including task instructions, input parameters, expected output (standard answer), and environmental configuration information.

[0033] Structured data refers to standard format data generated after CSV data is parsed by a CSV data parser and can be directly processed by the system. The CSV data parser supports automatic parsing of UTF-8 encoded CSV files, has intelligent column recognition capabilities, can handle complex tabular data, and has a parsing accuracy of up to 94%.

[0034] The preset transmission mechanism represents the data transmission scheme that ensures data transmission efficiency and reliability. It mainly includes fragmented transmission, breakpoint resumption, and verification mechanisms. The verification mechanism uses algorithms such as CRC32 and MD5 to verify the integrity of fragments, ensuring data integrity.

[0035] This transmission mechanism supports large file chunked uploads and file integrity verification. By splitting large files into multiple smaller chunks for independent transmission, it effectively reduces resource consumption per transmission. It also provides a resume function, recording transmission progress to ensure that transmission can resume from the point of interruption after an abnormal interruption, significantly improving transmission reliability and stability.

[0036] Intermediate storage refers to a storage layer (such as Redis) used to temporarily store raw data or intermediate results during the data processing flow. Its core functions include decoupling the data production and consumption chain, implementing traffic shaping and buffering, and ensuring consistency and reliability during data processing.

[0037] In its implementation, the system can receive CSV files uploaded by users and automatically parse them into structured data using a CSV data parser. Then, based on a preset transmission mechanism (such as fragmented transmission and breakpoint resume), the structured data can be written into intermediate storage efficiently and reliably.

[0038] Step S20: Batch call each model API in the preset model list, and combine the streaming data processor and multimodal data processor to process the structured data in real time to obtain the processing results.

[0039] The pre-defined model list represents a set of AI models pre-configured and entered into the system to participate in evaluation, designed to achieve specific evaluation objectives (such as model performance comparison, functional effectiveness verification, failure scenario detection, etc.). This list includes basic information, calling parameters, and evaluation-related configurations for each model, and serves as the core input for the batch scheduling engine to achieve high-concurrency calls and parallel evaluations.

[0040] The streaming data processor is a real-time data processing module built on Server-Sent Events (SSE) technology. It supports the server to actively and continuously push data streams to the client and is suitable for incremental output scenarios of AI models (such as text generation and real-time analysis). Its core functions include real-time stream processing, protocol optimization, and front-end integration.

[0041] The multimodal data processor is a unified module that integrates the processing capabilities of heterogeneous data from multiple sources, such as text, images, audio, and video. It supports cross-modal feature fusion and scenario adaptation, and its core functions include data preprocessing, feature fusion and alignment, and scenario adaptation.

[0042] The processing result represents the standardized, structured output data generated after model API calls, real-time processing of multiple types of data, and an anomaly fallback mechanism, which can be directly used for subsequent evaluation and analysis. The core includes two types of information: the structured transformation result of the model's original response and metadata and status information related to the evaluation process.

[0043] In practical implementation, high-concurrency API calls and resource optimization can be achieved through a batch scheduling engine. High-concurrency calls can be made to the APIs of each model in the preset model list to process structured data. Then, combined with a streaming data processor (based on SSE technology to support real-time processing of streaming data) and a multimodal data processor (integrating the ability to process multimodal data such as text, images, and videos), the API responses are processed and fused in real time. At the same time, the stability and reliability of the evaluation process are ensured by relying on exception handling and retry mechanisms, and finally accurate processing results are obtained.

[0044] Step S30: Based on a preset scalable evaluation system, multi-dimensional performance indicators are calculated on the processing results. Combined with automated quality scoring, horizontal comparative analysis of models of different scales is achieved through dynamic column generation technology. Finally, multi-dimensional performance data, quality scores and comparison results are integrated to generate a structured evaluation report.

[0045] The pre-defined, scalable evaluation system represents a modular evaluation framework pre-defined to support the evaluation needs of multiple scenarios and models. Its core consists of "evaluation dimensions, indicator rules, and adaptation mechanisms," featuring both pre-defined basic capabilities and on-demand expansion and adaptation. Specifically, it includes: core evaluation dimensions (such as response efficiency and accuracy), an indicator rule library (defining the quantitative calculation methods for each dimension, such as the first token time formula), a scenario adaptation mechanism (supporting dynamic loading of industry-specific indicators), and extended interfaces (providing APIs for users to customize indicators or dimensions).

[0046] Multi-dimensional performance data represents a quantitative dataset calculated based on performance dimensions within a scalable evaluation system, using performance metric counters. This dataset covers three core dimensions: response efficiency, stability, and resource consumption. For example, response efficiency metrics include first token time and complete response time; stability metrics include API call success rate, no-retry completion rate, and response time volatility; and resource consumption includes CPU / memory usage, network bandwidth consumption, and concurrent scheduling latency during model calls.

[0047] Quality score is used to represent the quantitative score and graded conclusion of the core output content of the model generated by the quality assessment engine based on the quality dimensions of the assessment system through automated algorithms. It includes assessment dimensions (such as factual accuracy and logical consistency), generation logic (multi-model cross-validation, stratified sampling review), and output format (dimensional score 0-100 points, comprehensive rating, such as A, B, C).

[0048] The comparison results represent the structured conclusions output by the comparison analyzer after conducting a horizontal comparison of the multi-dimensional performance data and quality scores of multiple models to be evaluated, based on dynamic column generation technology. Specifically, these include: indicator comparison data (numerical comparison of each model on key indicators), ranking and stratification (ranking of the overall capabilities of the models and gradient division), difference attribution hints (analysis of key difference points), and dynamic adaptation presentation (automatically adjusting the report column width and chart type to adapt to the number of models).

[0049] A structured evaluation report is a standardized evaluation result document generated by a report generator based on a preset template, which integrates "multi-dimensional performance data, quality scores, and comparison results".

[0050] In practical implementation, based on a pre-defined scalable evaluation system, a performance indicator calculator can quantify key performance indicators (such as first token time, complete response time, etc.) of the processing results, generating multi-dimensional performance data. Then, the quality evaluation engine uses an automated scoring system (combining multi-model cross-validation, stratified sampling review, and other technologies) to conduct multi-dimensional quality evaluation of the model response and output a quality score. Subsequently, a comparative analyzer can use dynamic column generation technology to conduct horizontal comparative analysis of the results of models of different scales / types to be evaluated, outputting comparison results. Finally, a report generator integrates the above multi-dimensional performance data, quality scores, and comparison results to automatically generate a structured evaluation report containing key indicators, quality scores, and improvement suggestions.

[0051] In this embodiment, through multimodal fusion processing, dynamic resource scheduling, scalable evaluation framework and automated comparative analysis technology, a comprehensive, accurate and efficient evaluation of AI agents in multiple scenarios is achieved, providing a reliable decision-making basis for the optimization and deployment of AI agents.

[0052] Based on the foregoing embodiments, a second embodiment of the end-to-end AI agent evaluation method of the present invention is proposed. In this embodiment, before step S10, the method may further include the following steps S40~S70: Step S40: Configure the AI ​​agent runtime environment, including specifying API endpoints, setting model lists, configuring user IDs, setting request timeouts, and defining the preset maximum number of retries.

[0053] The runtime environment can include specifying API endpoints (multiple API endpoints can be dynamically configured via the agent configurator to adapt to different model call requirements), setting a model list (the model parameter configurator supports concurrent evaluation of multiple models, with a maximum of 10 models evaluated at a time), configuring user IDs (the runtime environment needs to configure authentication information, such as the Basic Auth username and password for API nodes), setting request timeout time, and defining the preset maximum number of retries (such as configuring the alarm exit time for anomaly detection tasks).

[0054] Please see Figure 2 The diagram shows a constructor that initializes a configuration object, covering basic API configuration items (such as URL, Prompt, model list, user ID, timeout (e.g., default 30000 milliseconds) and maximum number of retries (e.g., default 3 times)) as well as video recording related configurations.

[0055] In practical implementation, API endpoints can be specified through the agent configurator, supporting dynamic configuration of multiple API endpoints to adapt to the calling requirements of different models; the model parameter configurator can be used to set the model list, supporting concurrent evaluation of multiple models and personalized parameter configuration; runtime parameters such as user ID, request timeout, and maximum retries can be configured. Among them, authentication information such as user ID needs to be configured in the runtime environment to ensure the security and legitimacy of API calls; the request timeout and maximum retries can refer to the configuration ideas of "alarm exit time" in the anomaly detection task to achieve unified timeout and retry policy management. Thus, through the collaboration of the agent configurator and the model parameter configurator, flexible and unified management of the basic environment for AI agent operation is achieved.

[0056] Step S50: Build and maintain the prompt word template library.

[0057] In practical implementation, a prompt word template manager can be used to achieve flexible injection and reuse of prompt words by combining fixed role definitions with dynamic placeholders. The template library should support hierarchical permission management; for example, administrators can edit and maintain templates, while ordinary users can only view or copy template content. Intelligent linkage between templates should be supported; for example, meeting minutes templates can automatically generate tasks for the weekly manager template to call, improving the efficiency of collaboration between templates. The template library decouples dynamic data from fixed quality elements, ensuring the standardization and flexibility of prompt word generation. Therefore, by building and maintaining a prompt word template library, not only is the efficiency and consistency of prompt word management improved, but it also provides strong support for the high-quality output of AI agents.

[0058] Step S60: Configure the parameters and timeout thresholds output by the control model.

[0059] In practical implementation, model generation parameters (such as temperature and diversity) can be customized and tuned to control output quality and style; multiple models can be configured and evaluated concurrently independently, for example, in scoring mode, each model can run evaluation tasks independently; request timeouts are typically managed uniformly in API node configurations (such as status code handling strategies), and "alarm exit time" is explicitly set in anomaly detection tasks to achieve unified management of timeout and retry strategies. Thus, the model parameter configurator effectively ensures the stability and flexibility of model output through the synergy of parameter tuning and timeout thresholds.

[0060] Step S70: Define a scalable evaluation system to clarify the various evaluation dimensions and standards on which the evaluation is based.

[0061] In its implementation, customizable metrics (such as accuracy, security, and usability) and a standardized dimensional framework (basic capabilities, technical performance, etc.) can be provided to meet diverse evaluation needs. Automated evaluation supports dynamic prompt templates (such as scoring modes and comparison modes), while manual evaluation can be configured with multi-dimensional weights (such as weighted total score calculation), thus achieving flexible and scenario-based scoring standards. Through an extensible dimensional framework, the evaluation system can be customized for different testing scenarios, ensuring the relevance and effectiveness of the evaluation results. Therefore, the evaluation dimension configurator, through its extensible evaluation system, enables flexible configuration and scenario-based application of evaluation dimensions and standards.

[0062] In this embodiment, the above design achieves dynamic runtime parameter management, scalable evaluation system, and intelligent resource scheduling, ensuring the stability and maintainability of end-to-end AI agent evaluation.

[0063] Based on the foregoing embodiments, a third embodiment of the end-to-end AI agent evaluation method of the present invention is proposed. In this embodiment, step S20 may include the following sub-steps S201~S203: Sub-step S201 involves traversing the preset model list and, based on the scheduling algorithm and concurrency control strategy, calling each model's API in high concurrency and parallelism to process the structured data and obtain the corresponding API response.

[0064] API response refers to the raw feedback data set returned by the model server to the caller (i.e., the core processing layer of the end-to-end AI agent evaluation system) after the batch scheduling engine initiates a call request (including structured input data and call parameters) to the target AI model in the preset model list. It is the direct output result of the model in response to the input requirements. Its core components usually include status code (the result of the request processing), response header (containing metadata) and response body (the actual returned data, including success data, error objects, and empty responses).

[0065] Please see Figure 3 The diagram illustrates a function for batch scheduling a preset list of models. By traversing the list of models, the API is called sequentially to obtain the results of each model, and the returned data is cleaned and organized before the results are added to the result row.

[0066] In practical implementation, to optimize resources and improve evaluation efficiency, the batch scheduling engine can use scheduling algorithms (such as first-come, first-served, shortest job first, multi-level feedback queues, priority scheduling), concurrency control (such as semaphore-based, thread pool optimization) and dynamic adjustment strategies (such as load-based adjustment, rule-based strategies, real-time feedback mechanisms) to efficiently schedule a preset list of models, enabling high-concurrency parallel API calls for multiple models; then, these APIs are used to process structured data and collect the API responses of each model.

[0067] Sub-step S202 involves cleaning, processing, and recording the API response, converting the processed data into a format suitable for streaming transmission, and using SSE technology to establish a real-time communication channel from the server to the client to transmit the converted data.

[0068] Please see Figure 4 The diagram illustrates the streaming process based on SSE technology. By receiving, decoding, and parsing real-time data pushed by the server, it enables one-way real-time communication between the client and the server. It is suitable for application scenarios that require immediate feedback, such as real-time notifications, data monitoring, and dynamic updates.

[0069] In practical implementation, a streaming data processor based on SSE technology can perform real-time cleaning, parsing, and structuring of the collected API responses, converting the segmented raw responses into a standard streaming data format. This streaming data processor supports the server to actively and continuously push processed data to the client, and through server-side optimization, front-end adaptation, and protocol tuning, it ensures the stability and real-time performance of the transmission, perfectly adapting to the output scenarios of AI models.

[0070] Sub-step S203 involves using a multimodal data processor on the server or client to perform type identification and targeted processing on the converted data to obtain the processing result.

[0071] In practice, the multimodal data processor integrates the processing capabilities of multimodal data such as text, images, and videos. It can perform type identification and targeted processing (such as data preprocessing, feature fusion, and scene adaptation) on the converted data on the server or client side, and finally generate usable processing results.

[0072] Furthermore, in one embodiment, after step S201, the following sub-steps A10~A20 are further included: Sub-step A10 involves monitoring the execution status of each API call in real time during the API call process; the execution status includes response time, status code, error type, resource usage, request parameters, and response data. Sub-step A20: Once an API call failure is detected, a predefined exception handling and retry mechanism is immediately triggered.

[0073] The execution status is a set of real-time monitoring metrics for each call process, used to evaluate the health and performance of the call; the core metrics include response time, status code, error type, resource usage, request parameters, and response data.

[0074] A predefined exception handling and retry mechanism is used to indicate that when an API call fails, the mechanism identifies the exception type, performs recovery operations, and controls retry behavior through a hierarchical strategy.

[0075] Please see Figure 5 The diagram illustrates the intelligent exception handling and retry mechanism. By retrying API calls when they fail and returning a uniform error format after reaching the maximum number of retries, the robustness of the system and the user experience are improved. This mechanism can effectively cope with situations where the network is unstable and services are temporarily unavailable.

[0076] In practical implementation, the execution status of each API call can be monitored in real time, including response time, status code, error type, resource usage, request parameters, and response data. Once a call failure is detected, a predefined exception handling and retry mechanism is immediately triggered. This intelligent retry strategy and exception recovery measures ensure the stability and reliability of the evaluation process by automatically retrying failed tasks.

[0077] Furthermore, in one embodiment, sub-step A20 includes the following sub-steps A201 to A203: Sub-step A201: When the API call fails, record the API call failure error log. If the current number of retries is less than the preset maximum number of retries, record the retry log and calculate the delay time according to the exponential backoff strategy. Sub-step A202: After waiting for the delay time, call the API again and increment the retry count by 1; Sub-step A203: If the retry still fails and the number of retries reaches or exceeds the preset maximum number of retries, the retry process is terminated and an error object is returned; the error object includes the specific reason for the API call failure, the unique identifier of the original request, and the reset of unnecessary fields.

[0078] The preset maximum number of retries represents the upper limit for retry operations, preventing resource exhaustion or request avalanche due to infinite retries. When the actual number of retries reaches this threshold, the system will terminate retries and enter the final failure handling process.

[0079] Delay time is used to represent the waiting interval between two retry operations; it can avoid the avalanche effect, improve the success rate, and optimize resources.

[0080] Error objects are used to generate structured data containers after a complete retry failure, encapsulating the complete failure context of the API call and supporting problem diagnosis and business process processing; the core includes the specific reason for the API call failure, the unique identifier of the original request, and the reset of unnecessary fields.

[0081] Unnecessary fields are reset to set the thought process and recommended video information to null values ​​to avoid interference from invalid data and ensure the consistency of the response structure; and the first token time and total processing time are set to 0 because the API call did not complete successfully and there is no valid processing time data.

[0082] In the implementation, when an API call fails, exception handling is triggered, and an error log can be recorded, detailing the failure time, API interface, error type, and specific error information for easy problem tracing. If the current number of retries is less than the preset maximum number of retries, a retry log is recorded (including the number of retries, current time, and reason for retry), and an exponential backoff strategy is used to calculate the delay time (e.g., 2^n seconds, where n is the current number of retries) to avoid high-frequency retries causing service pressure. After waiting for the delay time, the API call is re-initiated, and the retry count is incremented by 1. Subsequently, if the retry still fails, and the number of retries has reached or exceeded the maximum number of retries, the retry process is terminated, and an error object containing detailed error information is returned. This error object must include the specific reason for the API call failure (e.g., timeout, service error code, HTTP status code), retain the unique identifier of the original request (e.g., Request ID, supporting full-link tracing), and set the thought process and recommended video information to null values, and the first token time and total processing time to 0. By employing exponential backoff, state management, and precise error information encapsulation, it balances retry efficiency with system stability, effectively addressing scenarios of network instability or temporary service unavailability, while providing a complete context for troubleshooting.

[0083] In this embodiment, a collaborative architecture of batch scheduling engine, streaming data processor and multimodal data processor is used to establish real-time communication between server and client using SSE technology, ensuring efficient and stable transmission of processed data and program response.

[0084] Based on the foregoing embodiments, a fourth embodiment of the end-to-end AI agent evaluation method of the present invention is proposed. In this embodiment, step S30 may include the following sub-steps S301 to S304: Sub-step S301: Using a preset scalable evaluation system, the processing results are analyzed in multiple dimensions using a performance indicator calculator, and various performance indicators are quantitatively output; wherein, the performance indicators include the first token time and the complete response time.

[0085] Among them, multi-dimensional performance data, which are parameters used to represent the efficiency of the quantitative model, may include the first token time (used to measure inference efficiency) and the full response time.

[0086] In practical implementation, a pre-set scalable evaluation system can be used to perform multi-dimensional performance analysis on the processing results through a performance indicator calculator, enabling accurate calculation of key indicators such as first token time and complete response time, and providing a quantitative basis for model performance.

[0087] Sub-step S302 involves the quality assessment engine performing a stratified quality assessment of each model based on multi-dimensional scoring criteria, through multi-model cross-validation, stratified sampling review, and vertical domain adaptation techniques, and outputting the quality score of each model.

[0088] The multi-dimensional scoring criteria, used to represent the core basis for the quality assessment engine to quantitatively evaluate the quality of model responses, is a hierarchical and scalable scoring dimension system covering the entire chain of model response capabilities. Its design needs to adapt to the evaluation requirements of general scenarios and vertical fields; specifically, it includes five core dimensions: accuracy, completeness, logic, vertical field adaptability, and adaptability. Each dimension constructs a quantitative evaluation benchmark for model response quality from different levels.

[0089] The quality score represents the quantitative evaluation of the model's response to instructions based on the aforementioned multi-dimensional scoring criteria. The reliability of the score is ensured through techniques such as multi-model cross-validation (verifying score consistency) and stratified sampling verification (ensuring sample representativeness). The score measures the model's ability to understand and execute instructions. The overall score reflects the model's comprehensive performance in terms of accuracy, completeness, and vertical adaptability, providing targeted guidance for model optimization.

[0090] In its implementation, the quality assessment engine incorporates a multi-dimensional response quality assessment algorithm. It can employ an automated assessment system, combined with multi-model cross-validation, stratified sampling verification, and vertical domain adaptation techniques, to complete stratified quality assessment of each model and output the quality score of each model.

[0091] In sub-step S303, the comparison analyzer uses dynamic column generation technology to adapt to different numbers of comparison models and generate comparison results of multidimensional performance and quality indicators.

[0092] The comparison results represent the horizontal comparison dataset generated by the comparison analyzer for different evaluation models. This dataset is produced by automatically adapting to differences in the number of models using dynamic column generation technology, and integrating multi-dimensional performance data (such as first token time and complete response time) with instruction scores (accuracy, completeness, vertical fit, etc.). The results visually present the differences in efficiency and quality among the models in a structured format, providing a basis for model selection and optimization.

[0093] In its implementation, the comparative analyzer supports horizontal comparative analysis of results from multiple models. It can use dynamic column generation technology to automatically adapt to the analysis needs of different numbers of comparative models and output comparative results of multi-dimensional performance and quality indicators.

[0094] Sub-step S304 involves the report generator aggregating multi-dimensional performance data, quality scores, and comparison results, generating a structured evaluation report through a template engine, and associating specific optimization suggestions based on comprehensive analysis.

[0095] Among them, the optimization suggestions are used to represent the report generator's in-depth analysis based on multi-dimensional performance data (such as first token time, complete response time), instruction scores (accuracy, completeness, logicality, vertical domain adaptability, and adaptability) and comparison results. These suggestions provide the model developers or users with directions covering performance tuning (such as shortening the first token time and reducing resource consumption to improve throughput by optimizing the algorithm for response efficiency) and quality improvement (such as focusing on the model output effect and formulating improvement strategies from dimensions such as enhancing instruction logic matching, improving key information coverage, improving vertical domain knowledge depth, and optimizing the ability to adapt to complex contexts).

[0096] In practice, the report generator can aggregate multi-dimensional performance data, quality scores and comparison results, and automatically generate a structured evaluation report through the template engine, and associate specific optimization suggestions based on comprehensive analysis.

[0097] In this embodiment, the comprehensive evaluation mechanism described above enables quantitative assessment of model performance, precise problem localization, and scientific decision-making on optimization strategies, effectively improving the quality and reliability of AI agent applications.

[0098] Based on the same inventive concept, the fifth embodiment of this invention also provides an end-to-end AI agent evaluation system corresponding to the end-to-end AI agent evaluation method of the foregoing embodiments. Since the principle of the system in the fifth embodiment of this invention for solving the problem is similar to the end-to-end AI agent evaluation method of the foregoing embodiments, the implementation of the system can refer to the implementation of the method; repeated details will not be elaborated further. Please refer to... Figure 6 The present invention provides an end-to-end AI intelligent agent evaluation system, the system comprising: The receiving module 10 is used to receive the CSV file uploaded by the user, parse it into structured data, and write the structured data into intermediate storage through a preset transmission mechanism. The processing module 20 is used to call the APIs of each model in the preset model list in batches, and to process the structured data in real time by combining the streaming data processor and the multimodal data processor to obtain the processing results. The analysis module 30 is used to perform multi-dimensional performance analysis on the processing results based on a preset scalable evaluation system, and combined with automated quality scoring, use dynamic column generation technology to realize horizontal comparative analysis of models of different scales, and finally integrate multi-dimensional performance data, quality scores and comparison results to generate a structured evaluation report.

[0099] In addition, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described end-to-end AI agent evaluation method.

[0100] Figure 7 This is a schematic block diagram of the electronic device provided in an embodiment of this application. Figure 7 As shown, the electronic device includes at least one processor 401, a memory 402, at least one network interface 403, and a user interface 405. The various components in the electronic device are coupled together via a bus system 404. It is understood that the bus system 404 is used to implement communication between these components. In addition to a data bus, the bus system 404 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 7 The general will label all buses as bus systems.

[0101] The user interface 405 may include a monitor, keyboard, mouse, trackball, clicker, button, touchpad, or touch screen.

[0102] It is understood that memory 402 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM) or programmable read-only memory (PROM), used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memories described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable categories of memory.

[0103] In this embodiment of the invention, the memory 402 is used to store various types of data to support the operation of the electronic device 400. Examples of this data include: any executable program for operation on the electronic device 400, such as the operating system 4021 and application programs 4022; the operating system 4021 includes various system programs, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and handling hardware-based tasks. The application program 4022 may include various applications, such as a media player, browser, etc., for implementing various application services. The end-to-end AI agent evaluation method provided in this embodiment of the invention can be included in the application program 4022.

[0104] The methods disclosed in the above embodiments of the present invention can be applied to processor 401, or implemented by processor 401. Processor 401 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 401 or by instructions in the form of software. The processor 401 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 401 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. General-purpose processor 401 may be a microprocessor or any conventional processor, etc. The steps of the end-to-end AI agent evaluation method provided in the embodiments of the present invention can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in a memory. The processor reads the information in the memory and combines it with its hardware to complete the steps of the aforementioned method.

[0105] In an exemplary embodiment, the electronic device 400 may be used by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs) to perform the aforementioned method.

[0106] In summary, this invention achieves comprehensive, accurate, and efficient evaluation of AI agents in multiple scenarios through multimodal fusion processing, dynamic resource scheduling, scalable evaluation framework, and automated comparative analysis technology, providing a reliable decision-making basis for the optimization and deployment of AI agents.

[0107] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. An end-to-end AI agent evaluation method, characterized in that, The method includes: Receive CSV files uploaded by users, parse them into structured data, and write the structured data into intermediate storage through a preset transmission mechanism; The APIs of each model in the preset model list are called in batches, and the structured data is processed in real time by combining the streaming data processor and the multimodal data processor to obtain the processing results. Based on a pre-defined scalable evaluation system, the processing results are used to calculate multi-dimensional performance indicators. Combined with automated quality scoring, the results are used to achieve horizontal comparative analysis of models of different scales through dynamic column generation technology. Finally, multi-dimensional performance data, quality scores and comparison results are integrated to generate a structured evaluation report.

2. The method according to claim 1, characterized in that, Before the step of receiving the user-uploaded CSV file and parsing it into structured data, the method further includes: Configure the AI ​​agent runtime environment, including specifying API endpoints, setting the model list, configuring user IDs, setting request timeouts, and defining the preset maximum number of retries; Build and maintain a prompt word template library; Configure the parameters and timeout thresholds output by the control model; Define a scalable evaluation system to clarify the various evaluation dimensions and standards on which the evaluation is based.

3. The method according to claim 1, characterized in that, The process involves batch invoking the APIs of each model in a preset model list, and combining this with a streaming data processor and a multimodal data processor to process the structured data in real time, obtaining the processing results, including: The system iterates through a pre-defined list of models and calls the APIs of each model in high concurrency and parallel operation according to a scheduling algorithm and a concurrency control strategy to process the structured data and obtain the corresponding API responses. The API response is cleaned, processed, and recorded, and the processed data is converted into a format suitable for streaming transmission. A real-time communication channel from the server to the client is established using SSE technology to transmit the converted data. A multimodal data processor is used on the server or client side to perform type identification and targeted processing on the converted data to obtain the processing result.

4. The method according to claim 3, characterized in that, After the steps of traversing the preset model list and calling each model API in parallel with high concurrency according to the scheduling algorithm and concurrency control strategy, the method further includes: During API calls, the execution status of each call is monitored in real time; the execution status includes response time, status code, error type, resource usage, request parameters and response data; Once an API call failure is detected, the predefined exception handling and retry mechanism is immediately triggered.

5. The method according to claim 4, characterized in that, The mechanism for immediately triggering exception handling and retrying upon detecting an API call failure includes: When an API call fails, an error log for the API call failure is recorded. If the current number of retries is less than the preset maximum number of retries, a retry log is recorded, and the delay time is calculated according to the exponential backoff strategy. After waiting for the specified delay, call the API again and increment the retry count by 1; If the retry fails and the number of retries reaches or exceeds the preset maximum number of retries, the retry process is terminated and an error object is returned. The error object includes the specific reason for the API call failure, the unique identifier of the original request, and the reset of unnecessary fields.

6. The method according to claim 1, characterized in that, The pre-defined scalable evaluation system performs multi-dimensional performance index calculations on the processing results, including: Using a pre-defined scalable evaluation system, the processing results are analyzed in multiple dimensions using a performance indicator calculator, and multi-dimensional performance data is output in a quantitative manner; wherein, the multi-dimensional performance data includes the first token time and the complete response time.

7. The method according to claim 1 or 6, characterized in that, The method combines automated quality scoring with dynamic column generation technology to achieve cross-sectional comparative analysis of models of different scales. Finally, it integrates multi-dimensional performance data, quality scores, and comparison results to generate a structured evaluation report, including: The quality assessment engine, based on multi-dimensional scoring criteria, completes the hierarchical quality assessment of each model and outputs the quality score of each model through multi-model cross-validation, stratified sampling review and vertical domain adaptation technology. The comparison analyzer uses dynamic column generation technology to adapt to different numbers of comparison models, generating comparison results of multidimensional performance and quality indicators. The report generator aggregates multi-dimensional performance data, quality scores, and comparison results, generates a structured evaluation report through a template engine, and associates specific optimization suggestions based on comprehensive analysis.

8. An end-to-end AI intelligent agent evaluation system, characterized in that, The system includes: The receiving module is used to receive CSV files uploaded by users, parse them into structured data, and write the structured data into intermediate storage through a preset transmission mechanism. The processing module is used to call the APIs of each model in the preset model list in batches, and combine the streaming data processor and the multimodal data processor to process the structured data in real time to obtain the processing results. The analysis module is used to calculate multi-dimensional performance indicators of the processing results based on a preset scalable evaluation system, and combined with automated quality scoring, to realize horizontal comparative analysis of models of different scales through dynamic column generation technology. Finally, it integrates multi-dimensional performance data, quality scores and comparison results to generate a structured evaluation report.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to cause the processor to perform the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program that, when executed, performs the steps of the method according to any one of claims 1 to 7.

Images