Evaluation method, apparatus, computing device, and storage medium for agent system

Abstract

The specification provides an evaluation method and device of an Agent system, a natural language task draft is generated based on a task source using an evaluation task generation model; the task draft is decomposed into a structured task description using the evaluation task generation model, and the structured task description is checked; input data related to the evaluation task is obtained, and the correct answer corresponding to the input data is synchronously generated based on deterministic calculation; based on the task draft and the input data, a task prompt word describing the target state and not revealing the solution is generated, and the generated task prompt word is subjected to information leakage checking; a standard solution program is generated using the evaluation task generation model, and a verification script containing strong assertions or tolerance assertions is generated, the standard solution program is executed in an isolated environment and the standard solution program and its output are verified using the verification script; the target Agent system to be evaluated is called to execute the evaluation task based on the task prompt word and the input data.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence evaluation technology, and in particular to an evaluation method, apparatus, computing device and storage medium for an agent system. Background Technology

[0002] To accurately assess the task execution capabilities of different AI agents across various professional fields, it is necessary to construct a large number of high-quality evaluation tasks and conduct objective, reproducible, and quantitative evaluations of the agent's system capabilities based on these tasks. However, existing agent evaluation technologies face the following technical challenges: First, manual design is inefficient. High-quality evaluation benchmarks, such as ScienceAgentBench (ICLR 2025), require domain experts to spend approximately 3 hours manually designing each evaluation task. This purely manual approach severely limits the scalable output of evaluation tasks.

[0003] Second, the template parameterization method lacks flexibility. Existing parameterized variant generation methods based on task templates are limited by the patterns of existing templates and cannot generate entirely new task types not covered by the templates.

[0004] Third, the limitations of existing automated evaluation data generation technologies. Existing methods (such as CN119227818A and CN119443061A) generate evaluation data in the form of text question-and-answer pairs or multiple-choice questions, which can only evaluate the text understanding and generation capabilities of task generation models. For the evaluation of AI agents—that is, evaluating their ability to perform multi-step tasks in real-world environments—an executable evaluation task package is needed, containing specific input data files, standard solution programs, and automated verification scripts, rather than simple text question-and-answer pairs.

[0005] Fourth, there is the issue of the reliability of the correct answer. Existing methods typically rely on manual calculation or extraction from real datasets for the correct answer, lacking a systematic approach that can deterministically generate the correct answer while generating input data.

[0006] Fifth, the synthesis of multimodal data is difficult. The evaluation tasks of AI agents often involve multimodal data inputs (such as geological exploration maps, medical images, etc.), and the procedural synthesis of this multimodal data is extremely difficult.

[0007] In summary, existing technologies do not yet possess a systematic method that can start from natural language descriptions, use statistical characteristic parameters to drive the joint generation of data and the deterministic nature of correct answers, combine multiple automatic verification loops and dual-path data instantiation, and automatically construct a complete evaluation task package containing executable standard solutions and automated verification scripts. Summary of the Invention

[0008] In view of this, the purpose of this specification is to solve at least one technical problem existing in the prior art, and to provide an evaluation method and system for agent systems. This system can automatically construct an executable evaluation task containing input data, task prompts, standard solution programs, and verification scripts by starting from multiple task sources and going through a series of refinement stages, including task draft generation, structured decomposition, data instantiation, task description refinement, solution and verification generation, and quality verification. Based on this evaluation task package, the system to be evaluated is automatically evaluated.

[0009] To achieve the above objectives, according to a first aspect of this specification, a method for evaluating an agent system is provided, comprising: Based on at least one task source, use an evaluation task generation model to generate a natural language task draft that includes a description of data manipulation and a description of expected output. The task draft is decomposed into a structured task description containing an ordered chain of steps using an evaluation task generation model, and the structured task description is then subjected to format and logic validation. Acquire input data related to the evaluation task, and synchronously generate the correct answer corresponding to the input data based on deterministic computation; Based on the task draft and the input data, the evaluation task generation model is used to generate task prompts that describe the target state without revealing the solution, and information leakage verification is performed on the generated task prompts. A standard solution program is generated using an evaluation task generation model, along with a verification script containing strong assertions or tolerance assertions. The standard solution program is then executed in an isolated environment, and the verification script is used to verify the standard solution program and its output. The strong assertions are used to precisely match the output with the correct answer; the tolerance assertions are used to match the output with the correct answer within a specified tolerance range. When the standard solution program passes the verification script, the target agent system to be evaluated is invoked to perform the evaluation task based on the task prompt and the input data. Based on the result of the target agent system performing the evaluation task, as well as the standard solution program and / or the verification script, the evaluation result of the target agent system is generated.

[0010] According to a second aspect of this specification, an evaluation device for an agent system is provided, comprising: A generation unit is configured to: generate a natural language task draft containing data operation descriptions and expected output descriptions based on at least one task source using an evaluation task generation model; decompose the task draft into a structured task description containing an ordered chain of steps using the evaluation task generation model; acquire input data related to the evaluation task and synchronously generate the correct answer corresponding to the input data based on deterministic computation; generate task prompts describing the target state without revealing the solution based on the task draft and the input data using the evaluation task generation model; generate a standard solution program and a verification script containing strong assertions or tolerance assertions using the evaluation task generation model; wherein, the strong assertions are used to precisely match the output with the correct answer; and the tolerance assertions are used to match the output with the correct answer within a deviation range, wherein the deviation has a tolerance range. The verification unit is used to perform format and logic verification on the structured task description, perform information leakage verification on the generated task prompts, execute the standard solution program in an isolated environment, and use the verification script to verify the standard solution program and its output. And, an evaluation unit, configured to, when the standard solution program passes the verification script, invoke the target Agent system to be evaluated to perform an evaluation task based on the task prompt and the input data, and generate an evaluation result of the target Agent system based on the result of the target Agent system performing the evaluation task, the standard solution program / and the verification script.

[0011] According to a third aspect of this specification, an electronic device is provided, including a processor and a memory, wherein a computer program is stored on the memory, and the computer program, when executed by the processor, implements the method described in any one of the first aspects above.

[0012] According to a fourth aspect of this specification, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the method described in any one of the first aspects above.

[0013] Compared with existing technologies, the Agent system evaluation method provided in this specification features full-process automation, significantly improving the efficiency of evaluation task construction and solving the problem of low efficiency in manual construction. Starting with a natural language task draft, it does not rely on any predefined task template and can cover Agent task descriptions in multiple domains. The evaluation task output by this specification includes components such as input data files, task prompt word documents, standard solution programs, and verification scripts, which can be directly used for end-to-end evaluation of Agent systems in an isolated environment, covering the complete operation chain of Agent from data loading to result output. Moreover, the evaluation method provided in this specification uses a joint generation mechanism of synthetic data driven by statistical characteristic parameters and the deterministic nature of the correct answer, making the correct answer a deterministic function of the input data and its statistical characteristic parameters, thus ensuring the correctness and traceability of the answer. Attached Figure Description

[0014] To more clearly illustrate the technical solutions of the embodiments in this specification, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a flowchart illustrating an evaluation method for an agent system in one embodiment of this specification; Figure 2 This is a flowchart illustrating a structured task generation process in one embodiment of this specification; Figure 3 This is a flowchart illustrating a method for programmatically synthesizing data in one embodiment of this specification; Figure 4 This is a flowchart illustrating the specific process of generating and verifying the standard solution program in one embodiment of this specification; Figure 5 This is a schematic structural diagram of an electronic device shown in one embodiment of this specification; Figure 6 This is a structural block diagram of an evaluation device for an Agent system shown in one embodiment of this specification. Detailed Implementation

[0016] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0017] The "Agent system" (or "AI Agent" or "intelligent agent system") described in this specification refers to a software entity or collection of software entities built based on a large language model or other artificial intelligence model, possessing environmental awareness, autonomous decision-making capabilities, and tool invocation capabilities. An Agent system can receive task instructions presented in natural language or structured form, execute multi-step operations by invoking one or more external tools (including but not limited to code interpreters, data analysis libraries, file system interfaces, application programming interfaces, network services, database query engines, etc.), and generate structured or unstructured output results that meet the task objectives. The core characteristic of an Agent system lies in its autonomy—that is, under given task objectives and constraints, it can autonomously plan execution paths, select invoking tools, process intermediate results, and dynamically adjust execution strategies based on environmental feedback, rather than relying solely on pre-prepared fixed scripts or templated question-and-answer processes. In this specification, "Agent system" includes the target Agent system as the evaluation object, the quality inspection Agent system used for quality verification during the evaluation task construction phase, and the construction Agent system as the automatic construction executor of the evaluation task in the preferred embodiment.

[0018] Figure 1 This is an overall flowchart of an evaluation method for an agent system provided in an exemplary embodiment of this specification. The executing entity of this evaluation method can be a computer system or a collection of software modules, or it can be an agent system with autonomous reasoning and tool invocation capabilities; this specification does not limit this. Figure 1 As shown, the method in this embodiment includes the following steps.

[0019] Step S102: Based on at least one task source, use an evaluation task generation model to generate a natural language task draft that includes a description of data manipulation and a description of expected output.

[0020] The "evaluation task generation model" mentioned in this specification refers to an artificial intelligence model with generative capabilities, including but not limited to evaluation task generation models and multimodal generation models. In specific implementations, a "large model" with millions or more model parameters is used as a preferred implementation of the evaluation task generation model, but this specification is not limited to this. Moreover, it is worth noting that the evaluation task generation model designed in multiple steps of this specification can be the same model or different models, as long as they can achieve the functions described in the corresponding steps, and this specification does not limit this.

[0021] A task draft is a summary description of a data analysis or processing task, presented in the form of natural language text. A task draft must contain at least the following two core elements: (1) a description of data operations—the type and general content of the data processing, calculation, analysis, or transformation operations involved in the task; and (2) a description of expected output—the form and content of the final result that the task should produce.

[0022] In this step, the sources of tasks can include, but are not limited to, the following five types.

[0023] Task Source 1: Domain Knowledge-Based Generation. The domain name is input into the evaluation task generation model, which then lists common data analysis or processing tasks frequently encountered by practitioners in that domain. For example, when the domain is set to "proteomics," the task drafts output by the evaluation task generation model might include: "Load mass spectrometry data, perform peak detection and quantification, and output a protein abundance matrix," or "Perform FDR correction on protein identification results and output a list of significantly differentially expressed proteins." The advantage of this source is its ability to quickly cover typical task types within the target domain, providing an efficient starting point for building domain-specific agent evaluation benchmarks.

[0024] Task Source Two: Extraction from the Methodology Description Paragraphs of Academic Papers. The methodology section of a paper is input into the evaluation task generation model, which identifies each independent computational or data analysis step involved and rewrites each step as an independent task draft. For example, the methodology section of a bioinformatics paper might contain descriptions such as "differential expression analysis using DESeq2" or "GO enrichment analysis using ClusterProfiler." The evaluation task generation model extracts these descriptions and rewrites them into forms such as "perform differential expression analysis on a given gene expression count matrix, outputting a list of differentially expressed genes and the corresponding log2FC and adjusted p-values." This source enables the rapid transformation of analytical methods from cutting-edge research into evaluation tasks, keeping agent evaluation in sync with the forefront of scientific research.

[0025] Task Source 3: Extracted from tutorials or course assignments. The descriptions of assignments from tutorials or course outlines are input into the evaluation task generation model, which then rewrites them into task drafts conforming to the evaluation task format. For example, an assignment from a data science course might be described as "Preprocess the Titanic dataset using Pandas, and train a random forest model using Scikit-learn to predict survival." The evaluation task generation model could rewrite this as "Preprocess a tabular dataset containing passenger information (including missing value imputation and categorical variable encoding), train a classification model to predict binary labels, and output the model's accuracy and F1 score on the test set." The advantage of this source is that it allows for the rapid expansion of the evaluation task library using a large number of existing teaching resources.

[0026] Task Source 4: Reverse Generation Based on Target Skill Combinations. This is a unique task source in this manual, employing a reverse approach of "selecting skills first, then generating tasks," complementing the previous three types of forward approaches of "having tasks first, then labeling skills." Specific methods include: firstly, constructing a predefined skill library containing various data processing skills. These skills include, but are not limited to, at least two of the following: data cleaning (missing value handling, outlier detection and handling, format normalization), statistical testing (t-test, ANOVA, chi-square test, nonparametric tests), visualization (histograms, box plots, scatter plot matrices, heatmaps), machine learning modeling (classification, regression, clustering, dimensionality reduction), file format conversion (CSV / JSON / Excel / Parquet conversion), and optimization solving (linear programming, integer programming, constrained optimization).

[0027] Select two or more skills from the skill library to form a skill combination. For example, select "data cleaning" and "statistical testing" to form a combination, or select "data cleaning," "statistical testing," and "visualization" to form a three-skill combination. Input the skill combination as a constraint into the evaluation task generation model. The additional constraint requires that the generated task must use each skill in the skill combination simultaneously. The evaluation task generation model generates a task draft that meets the constraints. For example, when the skill combination is {"data cleaning," "statistical testing," and "visualization"}, the evaluation task generation model generates: "Clean a sales data set containing missing and outlier values, perform an ANOVA test on the sales volume in different regions, and plot the results as a box plot with significance labels"; when the skill combination is {"data cleaning," "machine learning modeling," and "visualization"}, the evaluation task generation model generates: "Preprocess a housing price dataset containing noise and missing values, train a regression model to predict housing prices, and plot the comparison between the predicted results and the true values ​​as a scatter plot."

[0028] Furthermore, by traversing different skill combinations in the skill library (including pairwise, triadic, and higher-order combinations), a systematic set of task drafts covering the target skill space can be generated. During the traversal, skill combinations lacking collaborative application scenarios are eliminated (for example, "file format conversion" and "optimization solution" lack a meaningful collaborative scenario, making it difficult to construct tasks that require the simultaneous use of both), and semantic deduplication is performed on the generated task drafts to avoid homogenization. Semantic deduplication can be achieved by calculating the semantic vector similarity between task drafts; when the similarity exceeds a preset threshold (e.g., 0.85), only one of them is retained.

[0029] The core advantage of this source lies in its ability to systematically cover the skill set space and generate targeted tasks for evaluating an agent's performance on specific skill sets, providing structured support for fine-grained diagnosis of agent capabilities. For example, when an agent performs poorly on tasks such as "data cleaning + statistical testing + visualization," evaluators can further subdivide the skill set to pinpoint specific weaknesses.

[0030] Task Source 5: Generating Task Drafts Based on Existing Data. For multimodal data (such as geological exploration maps, medical images, mineral thin section photographs, etc.), procedural synthesis often struggles to achieve sufficient realism and domain validity. This source adopts a reverse "data-first" approach: pre-collecting and organizing multimodal data materials, inputting the content description and data characteristics of these materials into the evaluation task generation model, and then generating a task draft based on the data materials as input. Specifically, this involves pre-collecting and organizing multimodal data materials specific to a particular domain. These multimodal data materials include at least one of the following: domain-specific images (such as medical CT / MRI images, geological remote sensing images, and microscopic cell images), experimental record data (such as spectral data, chromatographic data, and mass spectrometry data), sensor acquisition data (such as IoT device time-series data and environmental monitoring data), and professional documents (such as PDF format experimental reports and scanned documents). The data materials undergo standardized preprocessing, including format unification (e.g., converting images of different formats to PNG or DICOM standard formats), field naming standardization (e.g., unifying column naming styles), and data quality annotation (e.g., recording data acquisition conditions, known noise levels, and effective data range). Then, the content description of the data materials (e.g., "a set of lung CT images" or "daily air quality sensor data for a certain region from 2020 to 2023") and data characteristics (e.g., representing data dimensions, data types, numerical ranges, and special attributes) are input into the evaluation task generation model. The model then generates a task draft based on the data materials as input. For example, given "a set of lung CT images and their corresponding expert-annotated nodule locations," the evaluation task generation model generates: "Analyze a set of lung CT images, detect and annotate all suspicious nodule areas, count the number of nodules and their maximum diameter, and output a detection report." The advantage of this approach is that it fully utilizes existing multimodal data assets, avoids the difficulties of procedurally synthesizing multimodal data, and ensures the authenticity and domain validity of the input data.

[0031] Furthermore, this step also includes validating the generated task draft. This validity validation includes, but is not limited to: operability validation—verifying whether the operations described in the draft correspond to executable calculation steps (e.g., "analyze data" is an inoperable description and should be refined to "calculate the mean and standard deviation" or "draw a distribution histogram," etc.); input / output integrity validation—verifying whether the draft clearly describes the source of the input data and the expected output format; and domain relevance validation—verifying whether the task described in the draft has practical significance in the target domain. Drafts that fail validation are fed back to the evaluation task generation model for correction, and retries are limited to a preset number.

[0032] The natural language task draft output in step S102 is stored in the task queue for processing in subsequent step S104.

[0033] Step S104: Use the evaluation task generation model to decompose the task draft into a structured task description containing an ordered chain of steps, and perform format verification and logical verification on the structured task description.

[0034] This step transforms the natural language task draft generated in step S102 into a machine-parseable structured description, providing a formal task specification for subsequent data instantiation, solution generation, and verification script generation. Figure 2 This illustration shows a detailed generation process of a structured task as described in an exemplary embodiment of this specification, including steps 201 to 205.

[0035] The structured task description is represented in a predefined format (such as JSON or YAML) and may contain the following fields: The `steps` field is an ordered chain of steps, a required array. Each step may contain: an `id` field: a unique identifier for the step (e.g., "step_1"); an `action` field: an operation description, using natural language to describe the specific operation performed by the step (e.g., "load CSV data file", "remove missing values", "perform one-way ANOVA"); an `input` field: the input type or source for the step (e.g., "sales_data.csv", "step_1.output"); and an `output` field: the output type or filename for the step (e.g., "cleaned_data.csv", "anova_results.json").

[0036] The `final_outputs` field: A list of final output files, specifying the final outputs of the evaluation task.

[0037] The difficulty_estimate field contains an estimate of the difficulty level, which can be either "low", "medium", or "high".

[0038] Format validation uses a predefined structured data schema to enforce format validation on the output of the model generated by the evaluation task. The structured data schema is defined in JSON Schema or a similar form, specifying the structural requirements for the output JSON: steps is a required array, each element must contain four required fields: id, action, input, and output, and the type constraint for each field is string; the id field should follow the naming pattern "step_positive integer"; final_outputs is an optional array; and so on.

[0039] When the output does not conform to the structured data pattern, the system automatically extracts the non-conforming field information and validation error details, appends the error information to the original prompt, and requests the evaluation task generation model to regenerate. For example, if the JSON output by the model is missing a "steps" field or "steps" is an empty array, the system will append the message "Validation error: Required field 'steps' is missing or 'steps' is an empty array. Please regenerate a structured task description containing a complete chain of steps" to the end of the prompt and request a new generation.

[0040] Logical verification includes the following two core checks: (1) Verify whether the ordered chain of steps constitutes a directed acyclic graph (DAG), i.e., there is no circular dependency in the chain of steps. Circular dependency is manifested as: step A depends on the output of step B, and step B directly or indirectly depends on the output of step A, making it impossible to form an executable sequence of steps. The system performs this verification by constructing a step dependency graph (with steps as nodes and input-output dependencies as directed edges) and detecting whether there are cycles in the graph.

[0041] (2) Verify that the input for each step can be provided by the output of the preceding step or the initial input data. That is, for each step in the step chain, the data source referenced by its input field must be one of the following two: the initial input data file (such as "sales_data.csv") or the output field of the preceding step. If the input of a step references the output of a non-existent step (such as the input of a step being "step_5.output" but step_5 does not exist in the step chain), or references the output of a subsequent step (i.e., "cross-step forward reference"), then the input of that step cannot be satisfied, and the logic check fails.

[0042] If format or logic validation fails, the failed item is fed back to the evaluation task generation model for regeneration, with retries limited to a preset number. After format and logic validation, the output for the next stage is ensured to have a unified, parsable, machine-readable format. A structured task description that passes format validation is then passed to the next step.

[0043] Step S106: Obtain input data related to the evaluation task, and synchronously generate the correct answer corresponding to the input data based on deterministic computation.

[0044] The "ground truth" referred to in this specification refers to the expected standard output result in the evaluation task, generated by a deterministic computation process, used to verify the correctness of the output of the Agent system under evaluation or the standard solution program. This step is the core element in this specification that ensures the correctness of the answer and has a mathematical basis. In some of the illustrated implementation methods, depending on the method of obtaining the input data, there are two implementation paths: Path A (programmatic synthesis) and Path B (based on existing data materials). The two paths can be used individually or in combination.

[0045] Path A: Programmed synthetic data (suitable for structured numerical data) This approach is based on the data pattern defined in the structured task description. It procedurally synthesizes input data through a data generation process controlled by statistical characteristic parameters, and synchronously generates the correct answer based on deterministic computation: the statistical characteristic parameters of the data are set by the generation process, and the correct answer is a deterministic function of these statistical characteristic parameters. In other words, as long as the statistical characteristic parameters are determined, the probability distribution of the input data and the value of the correct answer are simultaneously determined—the reliability of the correct answer has mathematical traceability.

[0046] Figure 3 A detailed flowchart of path A is shown. (For example...) Figure 3 As shown, path A includes the following sub-steps: S301A: Determines the data mode of the input data.

[0047] Based on the data fields and field constraints defined in each step of the structured task description, determine the data schema of the input data. The data schema includes: field names (e.g., "region", "date", "sales", "product"), field types (categorized by type / numeric / date / text), field value ranges (e.g., "region values ​​are {'North', 'South', 'East', 'West'}" and "sales are positive"), and constraints between fields (e.g., "date field is no later than the current date").

[0048] Taking a sales data ANOVA task as an example, the data schema is defined as follows: the region field is categorical, the date field is date type, the sales field is numeric type, and the product field is categorical. The constraints of each field include: sales and product are related (the average sales value is different for different products), and region and sales are related (the average sales value is different for different regions).

[0049] S302: Set the statistical characteristic parameters of the input data.

[0050] Define the statistical characteristic parameters for each data field. These parameters include: the distribution type of each field and its parameters. For example, for a numeric field such as `sales`, set its distribution type to a normal distribution; for a categorical field such as `region`, set its categorical distribution to a uniform distribution or a specified probability distribution; for a date field such as `date`, set its range to a uniform distribution within a specified interval; and for the correlation coefficient between fields, for example, the effect size between `sales` and `region` (Cohen's value or...). This ensures that there are statistically significant differences in the means between different groups. Taking an ANOVA task using sales data as an example, if the mean sales revenue for each region is set to 1200, 950, 1100, and 800, with a standard deviation of 200 and a sample size of 50 for each region, then the effect size f-value corresponding to the ratio of between-group variance to within-group variance is approximately 0.68. According to statistical power analysis, the sample size required for this effect size is much smaller than 50. Therefore, the conclusion of the ANOVA test (rejecting the null hypothesis, i.e., there are significant differences in sales revenue across regions) is determined before the data is generated. This characteristic of "the conclusion being determined before the data is generated" is the mathematical basis for the certainty of the correct answer.

[0051] S303: Input data is generated programmatically based on statistical characteristic parameters.

[0052] Based on the statistical characteristic parameters set in step S302, input data is generated programmatically using a random process. Specifically, each field is randomly sampled according to its set distribution type and parameters to generate an original dataset that conforms to the statistical characteristic parameters.

[0053] Taking the aforementioned sales data ANOVA task as an example: For each region (4 regions), 50 samples are generated, totaling 200 rows of data. The `region` field of each sample is the corresponding region name; the `sales` field is randomly sampled from a normal distribution centered at the region's mean with a standard deviation of 200; the `date` field is uniformly randomly sampled from the interval from January 1, 2024 to December 31, 2024; and the `product` field is randomly sampled according to a preset probability distribution (e.g., A accounts for 40%, B for 35%, and C for 25%). To ensure the repeatability of the generation process, a fixed random seed is used during the generation process.

[0054] S304: Data quality issue injection.

[0055] Based on the data cleaning steps included in the structured task description, determine the types of data quality issues that need to be injected. When the task description includes expressions such as "cleaning," "handling missing values," or "detecting outliers," it indicates that the task is expected to include data cleaning operations, and data quality issues need to be intentionally injected into the input data to make the cleaning step a necessary component of the task.

[0056] According to a preset injection ratio, the data quality issues are injected into the programmatically generated input data. The data quality issues include at least one of the following: missing values ​​(such as setting the values ​​of certain fields to NaN or empty strings), outliers (such as setting certain values ​​to extreme values ​​that are far beyond the normal range, for example, setting sales figures to -100 or 10000), and format errors (such as setting date fields to invalid formats such as "2024-13-01").

[0057] The injection ratio ranges from 0% to 30% of the total input data. The injection ratio is positively correlated with the task difficulty: the higher the injection ratio, the greater the difficulty of data cleaning, and the higher the overall difficulty of the task.

[0058] After injecting data quality issues, the deterministic algorithm is re-executed to calculate the updated correct answer, ensuring that the correct answer remains consistent with the injected input data. For example, if missing values ​​were injected, the number of rows after proper cleaning should be reduced accordingly; if outliers were injected, these outlier rows should be removed after proper cleaning.

[0059] S305: Calculate the correct answer using a deterministic algorithm.

[0060] Based on statistical characteristic parameters and generated input data (including injected data quality issues), a deterministic algorithm is used to calculate the correct answer. A deterministic algorithm is defined as one that, given the same input data and the same algorithm implementation, will necessarily produce the same output result. The selection of the deterministic algorithm should match the operation type in the structured task description.

[0061] Taking the aforementioned sales data ANOVA task as an example, the calculation process for the correct answer includes: performing standard cleaning logic on the generated input data (deleting rows with missing values ​​and removing outliers using the interquartile range (IQR) method), then performing one-way ANOVA by region, and outputting the following information as the correct answer: the number of cleaned rows, the F-statistic, the p-value, and the significance (a Boolean value indicating whether the p-value is less than 0.05).

[0062] It is important to emphasize that, because the statistical characteristic parameters are set to indicate significant differences in the average sales revenue across different regions, and the injected data quality issues are correctly handled in the data cleaning logic, the significance judgment in the correct answer above (true in this example) is a deterministic conclusion, independent of the specific implementation of random fluctuations. Even if a different set of specific data is generated by changing the random seed, as long as the setting of the statistical characteristic parameters remains unchanged (i.e., the differences in the averages across regions remain significant), the significance conclusion of ANOVA will remain unchanged. This is the meaning of "the correct answer is a deterministic function of the statistical characteristic parameters".

[0063] Path A is applicable to, but is not limited to: descriptive statistics (mean, median, standard deviation, quantiles), hypothesis testing (t-test, ANOVA, chi-square test, nonparametric test), deterministic data transformation (logarithmic transformation, standardization, normalization, One-Hot encoding), and deterministic optimization solutions (linear programming, integer programming). For machine learning training tasks involving random initialization (such as random forests and K-Means clustering), determinism can be ensured by fixing the random seed, and a relatively tolerant range can be adopted for output differences caused by randomness (see the tolerance range setting for tolerance assertions in step S105), focusing on verifying qualitative conclusions (such as "whether the classification accuracy exceeds 0.8" or "whether the cluster silhouette coefficient is greater than 0.5"). For scenarios where the correct answer cannot be obtained through deterministic calculation (such as open-ended generation tasks), existing labeled data should be used via Path B.

[0064] When the evaluation task involves multimodal data (such as medical images, geological images, professional document images, sensor data, etc.) or domain-specific data that is difficult to achieve sufficient realism through procedural synthesis, Path B is adopted: directly use pre-collected existing data materials as input data, and perform standard analysis procedures on the existing data to calculate the correct answer.

[0065] Specifically, path B may include the following sub-steps: First, select multimodal data materials that match the task draft from the data material library, and then perform standardized preprocessing on the multimodal data materials.

[0066] The data resource library pre-collects and organizes multimodal data resources specific to a particular domain. These multimodal data resources include at least one of the following: domain-specific images, experimental records, sensor-collected data, and professional documents. Selection criteria include: whether the content description of the data resources matches the subject area of ​​the task draft; and whether the data characteristics (dimensions, format, completeness of annotation information) of the data resources meet the task requirements. Standardization preprocessing includes: format unification, standardized field naming, and data quality annotation. After standardization preprocessing, the existing data resources possess basic annotation information or have defined standard analysis procedures. The basic annotation information includes, but is not limited to: target region annotations in images, true labels for the data, and data acquisition parameters.

[0067] Next, a predefined standard analysis procedure is used to perform deterministic calculations on the data to generate the correct answer.

[0068] The standard analysis procedure described is a deterministic computational process tailored to specific data types and analysis tasks. For example, in medical image analysis tasks (such as lung CT nodule detection), the data material includes expert-annotated nodule locations (represented in pixel coordinates) and diameter measurements. The standard analysis procedure uses this expert-annotated data to count the number of nodules, calculate the maximum and average diameters, and output the nodule locations in a preset format. The correct answer is essentially a deterministic mapping of the expert-annotated information, rather than a subjective inference by the program.

[0069] For mineral thin section analysis tasks (such as lithological identification of rock thin sections), the data materials already include mineral composition annotations (type and area percentage of each mineral grain). Standard analysis programs calculate the area percentage of each mineral type, statistically analyze the main mineral assemblages, and output the rock type according to a preset rock classification standard based on the existing annotations. The correct answer is a definitive summary of the existing annotation information.

[0070] For sensor data analysis tasks (such as anomaly detection in environmental monitoring data), the data materials already include timestamps of the anomaly events. Standard analysis programs use this information to count the number of anomalies and calculate the distribution characteristics of their durations. The correct answer is to perform deterministic statistics on the annotation information.

[0071] Therefore, the correct answer in path B is essentially a deterministic mapping of existing basic annotation information. As long as the quality of the basic annotation information is reliable (guaranteed by domain experts) and the standard analysis procedure is implemented correctly, the correct answer has the same deterministic basis as path A.

[0072] In some other embodiments shown in this specification, part of the input data for an evaluation task is synthesized via path A, and another part of the input data is selected from existing materials via path B. The correct answers from both parts are merged into a unified correct answer.

[0073] For example, an evaluation task requires an agent to comprehensively judge whether a region has the potential for a specific mineral resource based on a geological survey table of data (synthesized via path A) and a set of remote sensing images of the corresponding area (selected via path B). In this task, the table of data (such as latitude, longitude, altitude, lithology codes, etc.) is generated procedurally via path A, and the correct answer is determined by the statistical characteristic parameters (such as the relationship coefficient between different lithology codes and mineralization potential) when the table of data was generated; the remote sensing images are selected from an existing geological image database, and the correct answer is determined based on existing expert interpretation and annotation information. After the data and answers from both paths are generated separately, they are merged into a unified dataset and correct answer according to the task requirements.

[0074] Step S108: Based on the task draft and the input data, use the evaluation task generation model to generate task prompts that describe the target state without revealing the solution, and perform information leakage verification on the generated task prompts.

[0075] The task prompt is the actual task description document received by the evaluation object (the Agent system to be evaluated). Its quality directly affects the fairness and effectiveness of the evaluation. If the task prompt reveals solution information, the evaluation will degenerate into a test of whether the Agent remembers a specific API, rather than an assessment of its task understanding and execution capabilities.

[0076] Specifically, the task draft output in step S102 and the input data information generated in step S106 (such as actual file path, data format, field list, data summary statistics, data quality problem description, etc.) are merged into a unified prompt word input, and the task prompt word for the evaluation object is generated by the evaluation task generation model.

[0077] Task prompts include the following elements: (1) Overview of the input file path and file content. For example, inform the evaluation subject that "the input file is sales_data.csv, which contains four fields: region, date, sales (sales amount, unit: RMB 10,000), and product (product category), with a total of 200 rows of data. There are a small number of missing values ​​and outliers in the data."

[0078] (2) Output file path and output format requirements. For example, explicitly specify the path of each output file (e.g. " / output / anova_result.json" ". / output / boxplot.png") and the format requirements (e.g., ANOVA results are output in JSON format, containing three fields: f_statistic, p_value, and significance; box plots are output in PNG format, and the significance level needs to be marked on the plot).

[0079] (3) Evaluation criteria. Clearly state the evaluation criteria for the task completion status of the evaluated subjects (e.g., "The significance judgment of ANOVA must be consistent with the standard answer" and "The box plot must include grouped data from all four regions"). However, do not include specific threshold values ​​or algorithm details.

[0080] The purpose of information leakage verification is to ensure that the task prompt does not contain any information that may hint at the solution. The verification includes one or more of the following sub-items: (1) Semantic similarity comparison. The generated task prompts are compared with the key code snippets (or semantic vectors of key API names) of the standard solution program. If the similarity exceeds a preset threshold (e.g., 0.85), information leakage may exist.

[0081] (2) Specific solution information detection. Check whether the task prompt contains library names, function names, or algorithm names. If any of the above are detected, it is determined that solution information has been leaked.

[0082] (3) Step Instruction Detection. Detect whether the task prompt contains overly specific instructions on the operation steps (such as "Step 1, use pandas.read_csv to load the data; Step 2, use df.dropna to delete missing values..."). Task prompts should describe the target state rather than the operation steps; excessive step instructions constitute a disguised leak of solution information.

[0083] If any validation fails, the failed item is fed back to the evaluation task generation model, requesting it to regenerate the task prompt. For example, if the original task prompt contains the statement "use the f_oneway function in scipy.stats", the information leakage validator detects "scipy.stats" and "f_oneway", and appends the feedback message "Solution information leakage detected: the specific library name and function name 'scipy.stats.f_oneway' are mentioned. Please delete all content mentioning specific libraries, functions or algorithms, and only describe the task objective" to the prompt and request regeneration.

[0084] Step S110: Generate a standard solution program and a verification script containing strong assertions or tolerance assertions using the evaluation task generation model, thereby executing the standard solution program in an isolated environment and verifying the standard solution program and its output using the verification script.

[0085] The task of this step is to generate two key executable components: a standard solution program and a verification script, and to ensure their consistency and correctness through closed-loop verification. The "isolated environment" mentioned in this specification refers to a code execution environment isolated from the host system resources, including but not limited to containerized environments (such as Docker containers), virtual machines, or operating system-level sandboxes. The core requirement of an isolated environment is that when the Agent system under evaluation or the standard solution program is executed within it, it cannot access external network resources, cannot modify the file system outside the isolated environment, and cannot interfere with the normal operation of the host system.

[0086] Figure 4 A detailed flowchart of this step is shown, specifically including the contents shown in S401 to 406.

[0087] S401: Generate the standard solution program.

[0088] The refined task prompts and input data from step S108 are input into the evaluation task generation model, which then generates a complete standard solution program. This standard solution program is an executable script written in a programming language, implementing the entire operation flow from loading input data to generating the final output.

[0089] In yet another embodiment shown, after generating the standard solution program, a static check should also be performed on the generated standard solution program. This static check may include the following sub-items: (1) Check whether the standard solution program references external resources not mentioned in the task prompt. For example, if the standard solution program attempts to download a data file from an external URL, but the task prompt does not mention that external data needs to be downloaded, it is considered unqualified.

[0090] (2) Examine whether the standard solution program hardcodes the correct answer into the solution. The hard-coding detection includes: checking whether constants (such as statistical values ​​or final output values) close to the correct answer are directly written into the code; and checking whether logical branches (such as ifTrue: return expected_answer) directly return the correct answer without performing actual calculations. Hard-coding behavior renders the standard solution program meaningless as a "standard solution" and cannot be used as an evaluation benchmark. This detection can be achieved through abstract syntax tree analysis of the code combined with semantic judgment of key constants.

[0091] (3) Verify whether the standard solution program can run independently. That is, check whether the code dependencies are complete, whether the entry point is clear, and whether the input and output paths use relative paths instead of absolute paths. For example, if the standard solution program references a third-party library that is not listed in the dependency declaration, or uses a hard-coded absolute file path, it is determined that it cannot run independently.

[0092] When static analysis reveals a problem, the problem description is fed back to the evaluation task generation model for correction, and retries are limited to a preset number of times.

[0093] S402: Generate verification script.

[0094] Based on the correct answer pre-calculated in step S401, a verification script is generated using the evaluation task generation model. The verification script uses an assertion-based testing framework (such as Python's unittest or pytest) to verify the output of the standard solution program (and the subsequent Agent system to be evaluated) item by item.

[0095] The core of the validation script is the assertion mechanism. The "strong assertion" and "tolerance assertion" described in this manual are two validation rules for the program output in the validation script. A strong assertion requires the output to match the correct answer exactly (or be completely identical item by item), and is applicable to core conclusions and qualitative judgments (such as "does the file exist?", "is the significance judgment correct?", "are the classification labels accurate?"). Its result is a binary "pass" or "fail". A tolerance assertion allows for a deviation between the output and the correct answer, limited by a tolerance range. It is applicable to specific numerical results (such as statistics, coordinate positions, percentages, etc.), and its result is whether the deviation falls within the tolerance range.

[0096] Strong assertions are applicable in scenarios including, but not limited to: whether the output file exists, whether the output format is correct, whether the qualitative conclusion is correct, whether the Boolean flags match, and whether the enumerated type output is within the expected value range. Tolerance assertions are applicable in scenarios including, but not limited to: statistical values, model performance indicators, coordinate positions (such as bounding box coordinates in object detection), and count-type outputs (such as "number of rows of cleaned data" or "number of detected targets"); the tolerance range is adaptively determined according to the output type.

[0097] In one embodiment of this specification, the tolerance range is determined according to the following rules: For numerical outputs (such as statistics, model performance indicators), the tolerance range is calculated based on the effect size of the statistical characteristic parameters in step S106. When the effect size is greater than a preset threshold, a relative tolerance (e.g., a relative tolerance of 10%) is used, meaning that if the correct answer is 100, the output is allowed to be between 90 and 110; when the effect size is less than or equal to the preset threshold, an absolute tolerance (e.g., an absolute tolerance of 0.01) is used, meaning that if the correct answer is 0.05, the output is allowed to be between 0.04 and 0.06. The principle of this design is that when the statistical effect is significant, even if different reasonable solution paths lead to small deviations in the numerical values, it will not affect the direction of the conclusion; when the statistical effect is not significant (close to the critical value), more accurate numerical matching is required to ensure the reliability of the conclusion. For count-type outputs (such as the number of rows of cleaned data, the number of detected targets), the tolerance range is determined based on a preset proportion of the total amount of input data, wherein the preset proportion is in the range of 1% to 10%. For categorical and Boolean outputs, strong assertions are used for exact matching without tolerance.

[0098] Furthermore, in yet another illustrated implementation, the verification script may also include the following process compliance checks: (1) Network Access Verification. This verifies whether network resources outside the isolated environment are accessed during the execution of the standard solution procedure (or evaluation agent system) in the isolated environment. Specifically, it is implemented by intercepting external outbound requests through container network policies (e.g., blocking all non-localhost outbound connections) and capturing abnormal network call behavior. Any external network access attempts detected are recorded as a process compliance violation.

[0099] (2) File Modification Verification. This verifies whether files outside the task-specified path were modified during the execution of the standard solution program (or evaluation agent system) in the isolated environment. Specifically, a hash snapshot (e.g., using SHA-256) is generated for all directories during isolated environment initialization, and a second hash comparison is performed after execution to detect whether any files in unexpected paths were created, modified, or deleted. If illegal file modification is detected, it is recorded as a process compliance violation.

[0100] (3) Execution Time Verification. Verify whether the execution time of the standard solution program exceeds the preset upper limit. The preset upper limit can be dynamically determined according to the task complexity and the scale of the input data. For example, for a data cleaning + ANOVA + visualization task containing 200 rows of data, the preset execution time upper limit can be set to 60 seconds. If the execution time exceeds the upper limit, it is recorded as a process compliance violation.

[0101] S403: Perform closed-loop verification in an isolated environment.

[0102] Build an isolated runtime environment (e.g., start a Docker container and configure the network policy to deny all external access), copy the input data file to the specified working directory of the isolated environment, and execute the standard solution program in the isolated environment. After the standard solution program completes execution, execute a verification script to check its output. The verification script checks the output of the standard solution program, including strong assertions, tolerance assertions, and optional process compliance checks.

[0103] When all verifications pass (S404: Yes), the evaluation task is confirmed to be valid. The input data, task prompts, standard solution program and verification script are packaged into an executable evaluation task package, which serves as a complete evaluation unit for evaluating the Agent system under evaluation.

[0104] When the standard solution fails verification, the automatic repair sub-step (S405) is initiated. This sub-step feeds back error information (such as the specific error description output by the verification script, the location and reason for assertion failure) to the evaluation task generation model, which then corrects the standard solution based on the error information. The corrected code is then re-executed for verification in the isolated environment. If verification still fails after reaching the preset limit for the number of repetitions, the evaluation task is marked as "requiring manual intervention," and a detailed error log is output for manual analysis.

[0105] Before invoking the target Agent system to be evaluated to perform the evaluation task, step S112 may be executed: invoking multiple quality inspection Agent systems to perform the evaluation task based on the task prompt words and the input data, and generating a difficulty evaluation of the evaluation task based on the results of the multiple quality inspection Agent systems performing the evaluation task, as well as the standard solution program and / or the verification script.

[0106] This step uses multiple AI Agents with different capability levels (referred to as "quality control Agents" in this specification) to perform the evaluation task, and calculates the pass rate of each Quality Control Agent to obtain a quantitative estimate of the difficulty of the evaluation task. The Quality Control Agent can be an Agent system driven by a large language model, and its capability level can be determined through pre-screening (e.g., dividing it into high, medium, and low capability levels based on its performance on existing evaluation benchmarks).

[0107] Specifically, in an isolated environment, the following process is executed for each quality inspection agent: (1) Provide the task prompts and input data files to the quality inspection agent; (2) The quality inspection agent performs tasks in an isolated environment and generates output files; (3) Use the verification script generated in step S110 to verify the output of the quality inspection agent; (4) Record the verification results (pass / fail) of the quality inspection agent and its execution log.

[0108] Based on the above verification results, the pass rate of each quality inspection agent can be calculated. For example, if 6 quality inspection agents (2 high, 2 medium, and 2 low) are called, and 4 of them pass the verification, the pass rate is 67%. The system can generate a difficulty evaluation of the assessment task based on the pass rate and optionally adjust the difficulty: when the pass rate is greater than a first preset threshold, the task difficulty is determined to be too low, triggering a difficulty increase operation; when the pass rate is less than a second preset threshold, the task difficulty is determined to be too high, triggering a difficulty decrease operation or an automatic diagnosis operation; when the pass rate is between the second preset threshold and the first preset threshold, the task difficulty is determined to be moderate, and no adjustment is needed.

[0109] The difficulty-increasing operations described in this embodiment include at least one of the following: increasing the proportion of data quality issues injected into the input data; increasing the number of constraints in the structured task description; increasing the data size of the input data; increasing the number of steps in the ordered step chain; and reducing the level of detail in the prompts provided in the task prompts. Correspondingly, the difficulty-reducing operations are the opposite of the above difficulty-increasing operations, and will not be elaborated upon here.

[0110] The automatic diagnostic operation described in this embodiment may include the following sub-items: Error type analysis of the execution logs of the quality inspection agent: collecting the execution logs of all quality inspection agents that failed verification, extracting the error types, and statistically analyzing the frequency of each type of error. If the execution logs of more than a preset proportion of quality inspection agents report the same type of error, it is determined that the evaluation task may have its own defects (such as unclear task description, ambiguous output format requirements, or bugs in the verification script), and the task is submitted to the manual review queue or marked as "requiring manual intervention." If each quality inspection agent fails at different positions in the ordered chain of steps, it is determined that the task is too difficult rather than defective, and the task is marked as a high-difficulty task. If a verified standard solution program fails verification even when re-executed in the same isolated environment, it is determined to be an environmental problem (such as incorrect isolation environment configuration, incompatible dependency library versions, etc.), triggering the environment check process.

[0111] Step S112 is executed after the standard solution program has been verified and before the target Agent system is called, to ensure that the quality and difficulty of the evaluation task meet the usable standard. The task after difficulty adjustment returns to step S106 (data instantiation) or step S108 (task description refinement) to be regenerated and verified again until the difficulty is appropriate or the maximum number of iterations is reached.

[0112] Step S114: When the standard solution program passes the verification script, the target Agent system to be evaluated is invoked to perform the evaluation task based on the task prompt and the input data. Based on the result of the target Agent system performing the evaluation task, the standard solution program and / or the verification script, the evaluation result of the target Agent system is generated.

[0113] After the standard solution procedure verification is completed in step S110 and the quality verification is completed in step S112 (if executed), the executable evaluation task package is applied to the evaluation of the target Agent system. Specifically, in an isolated environment, the following process is executed: (1) Construct an isolated operating environment (which can be consistent with the environment configuration used when verifying the standard solution program in step S110) to ensure the consistency of the evaluation conditions.

[0114] (2) Provide the task prompt words to the target Agent system and copy the input data file to the specified working directory in the isolation environment.

[0115] (3) The target agent system performs the evaluation task in the isolated environment and generates output files. During this process, the network policy of the isolated environment denies external access, the file system monitors and records all file read and write operations, and the timer records the task execution time.

[0116] (4) Use the verification script generated in step S110 to verify the output of the target Agent system. The verification script performs the following operations: verify the correctness of qualitative conclusions through strong assertions (such as "whether the significance judgment is correct", "whether the output file exists", "whether the output format is compliant"); verify the reasonableness of specific numerical results through tolerance assertions (such as whether the statistic is within the tolerance range); and verify the compliance of the execution process through process compliance checks (whether there is illegal network access, illegal file modification, or timeout).

[0117] (5) Generate evaluation results for the target Agent system based on the verification results. The evaluation results include, but are not limited to: overall judgment: pass / fail (pass when all strong assertions pass and all tolerance assertions are within the tolerance range, otherwise fail); detailed results of each assertion item: specific test results of each strong assertion and tolerance assertion (pass / fail / actual value / expected value / deviation); process compliance record: whether illegal network access occurred, whether illegal file modification occurred, and whether the execution time exceeded the preset limit; execution log: standard output and standard error output of the target Agent system during the execution process.

[0118] (6) Write the evaluation results into the evaluation report and store them in persistent storage for subsequent analysis and auditing.

[0119] When it is necessary to track the capability changes of the same target agent system in multiple rounds of evaluation, task variants with similar difficulty but different content can be generated by changing the data instance (changing the random seed to generate new input data instances), which prevents the agent system from gaining false capability improvements by memorizing evaluation data.

[0120] To prevent evaluation data from being "memorized" by the AI ​​model's training data, thus generating artificially inflated performance metrics during evaluation, another embodiment provided in this specification embeds a data decontamination mechanism during the automatic construction of the evaluation task. The data decontamination strategy comprises two levels: Level 1: Data Instance Variation. During the data instantiation process described in step S116, a different random seed is used each time input data is generated, ensuring that the specific values ​​of each generated data instance are different. For example, in a sales data ANOVA task, the sales value generated with a random seed of 42 is completely different from the value generated with a random seed of 123, but the correct answer (whether there are significant differences between regions) remains the same for both sets of data. This level ensures that even if a data instance is included in the AI ​​model's training data, the model cannot map its memory to other data instances.

[0121] Level Two: Task Logic Variations. In addition to changes at the numerical data level, this specification also supports changes at the task logic level. By altering the number and names of fields in the data schema (e.g., changing "sales" to "revenue" or "region" to "area"), changing the operation types in the structured task description (e.g., changing ANOVA to t-test or nonparametric test), and changing the output format requirements in the task prompts (e.g., changing JSON output to CSV or tabular format), the logical structures of different task variations become distinct. This level ensures that even if an AI model memorizes all the details of a specific task, it cannot transfer that knowledge to task variations with different logical structures.

[0122] By combining the two levels, even if the AI ​​model has been exposed to a certain data instance or task description during the training phase, it cannot directly transfer the memorized solution to task variants with different specific values ​​and logical structures, thus effectively ensuring the authenticity and validity of the evaluation results.

[0123] Based on the Agent system evaluation method described in one or more of the above embodiments, by constructing a complete technical chain of "automatic task construction - deterministic answer guarantee - isolated closed-loop verification - automated evaluation," the paradigm shift of Agent evaluation from being driven by human experience to a fully automated execution process based on data and logic determinism has been realized. This solves the efficiency bottleneck of large-scale evaluation. Through the joint generation mechanism of data and the determinism of correct answers driven by statistical characteristic parameters, the answers are mathematically verifiable, ensuring the objectivity and reliability of the evaluation benchmark. By "describing the target state without revealing the solution," the fairness of the evaluation is ensured. The hierarchical design of strong assertions and tolerance assertions accommodates reasonable path deviations to improve the robustness of the evaluation. By performing closed-loop verification in an isolated environment, the safety and reproducibility of the evaluation process are ensured. Thus, the efficiency of Agent evaluation is improved in terms of efficiency, accuracy, fairness, robustness, and security.

[0124] Figure 5 This is a schematic structural diagram of an electronic device provided in an exemplary embodiment. Please refer to... Figure 5 At the hardware level, the device includes a processor 502, an internal bus 504, a network interface 506, memory 508, and non-volatile memory 510, and may also include other necessary hardware. The evaluation method for the Agent system described in one or more embodiments of this specification can be implemented in software, for example, the processor 502 can read the corresponding computer program from the non-volatile memory 510 into memory 508 and then run it. Of course, besides software implementation, the evaluation method for the Agent system described in one or more embodiments of this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0125] Figure 6This specification illustrates a structural block diagram of an evaluation device 60 for an agent system according to an exemplary embodiment. The device 60 includes: a generation unit 602, configured to: generate a natural language task draft containing a data operation description and a desired output description based on at least one task source using an evaluation task generation model; decompose the task draft into a structured task description containing an ordered chain of steps using the evaluation task generation model; acquire input data related to the evaluation task and synchronously generate the correct answer corresponding to the input data based on deterministic computation; generate task prompts describing the target state without revealing the solution based on the task draft and the input data using the evaluation task generation model; generate a standard solution program and a verification script containing strong assertions or tolerance assertions using the evaluation task generation model; wherein the strong assertions are used to precisely match the output with the correct answer, applicable to the verification of core conclusions and qualitative judgments; the tolerance assertions are used to match the output with the correct answer within a deviation range, applicable to the verification of specific numerical results, and setting a tolerance range for the deviation.

[0126] The verification unit 604 is used to perform format verification and logical verification on the structured task description, perform information leakage verification on the generated task prompt words, execute the standard solution program in an isolated environment, and use the verification script to verify the standard solution program and its output. The evaluation unit 606 is used to, when the standard solution program passes the verification script, invoke the target agent system to be evaluated to perform an evaluation task based on the task prompt and the input data, and generate an evaluation result of the target agent system based on the result of the target agent system performing the evaluation task, the standard solution program and the verification script.

[0127] More preferably, the source of the task includes at least one of the following: Task sources can be generated from domain knowledge, extracted from academic papers, extracted from tutorial assignments, generated in reverse from skill combinations, or generated from existing data materials.

[0128] More preferably, the generation unit 602 is further configured to: select two or more skills from a pre-built skill library containing multiple data processing skills to form a skill combination; input the skill combination as a constraint into an evaluation task generation model, and use the evaluation task generation model to generate a task draft containing a data operation description and expected output, which requires the use of each skill in the skill combination to complete.

[0129] More preferably, the generation unit 602 is further configured to: traverse different skill combinations in the skill library to generate a set of task drafts covering the target skill space; during the traversal process, skill combinations that do not have collaborative application scenarios are eliminated, and semantic deduplication is performed on the generated task drafts.

[0130] More preferably, the verification unit 604 is further configured to perform format-force verification on the output of the evaluation task generation model using a predefined structured data pattern. When the output does not conform to the structured data pattern, the non-conforming field information is extracted and appended to the original prompt words of the task generation model to request regeneration. It also verifies whether the ordered chain of steps constitutes a directed acyclic graph and whether the input of each step can be provided by the output of the preceding step or the initial input data.

[0131] More preferably, the generation unit 602 is further configured to: programmatically synthesize input data based on the data patterns defined in the structured task description, and calculate the correct answer corresponding to the input data using a deterministic algorithm based on preset statistical characteristic parameters; and, using pre-collected existing data materials with basic annotation information as input data, and performing deterministic calculations on the data materials using a predefined standard analysis program to generate the correct answer corresponding to the input data.

[0132] More preferably, the generation unit 602 is further configured to: determine the data pattern of the input data according to the data fields and field constraints defined in the structured task description; set statistical characteristic parameters of the input data, the statistical characteristic parameters including the distribution type and parameters of each data field, and the correlation coefficient between fields; and generate input data programmatically through a random process based on the statistical characteristic parameters.

[0133] The generation unit 602 is further configured to calculate the correct answer using a deterministic algorithm based on the same statistical characteristic parameters, such that the correct answer is a deterministic function of the input data and its statistical characteristic parameters.

[0134] More preferably, the generation unit 602 is further configured to: select multimodal data materials matching the task draft from a data material library, wherein the multimodal data materials include at least one of domain-specific images, experimental record data, sensor acquisition data, and professional documents; perform standardized preprocessing on the data materials, including format unification, field naming standardization, and data quality labeling, so that the existing data materials have basic labeling information or a defined standard analysis procedure; and use the standard analysis procedure to perform deterministic calculations on the data materials to generate a correct answer, wherein the correct answer is a deterministic mapping of the basic labeling information.

[0135] More preferably, the verification unit 604 is further configured to: perform semantic similarity comparison between the generated task prompt words and the key code segments of the standard solution program; detect whether the task prompt words contain library names, function names, or algorithm names; and when the verification fails, feed back the failed items to the evaluation task generation model to regenerate the task prompt words.

[0136] More preferably, the verification unit 604 is further configured to: verify whether network resources outside the isolation environment are accessed during the execution of the standard solution program in the isolation environment; verify whether files outside the path specified by the evaluation task are modified during the execution of the standard solution program in the isolation environment; and verify whether the execution time of the standard solution program in the isolation environment exceeds a preset upper limit.

[0137] More preferably, the verification unit 604 is further configured to: verify whether the standard solution program references external resources not mentioned in the task prompt; verify whether the standard solution program hardcodes the correct answer into the solution; and verify whether the standard solution program can run independently.

[0138] More preferably, before invoking the target Agent system to be evaluated to execute the evaluation task, the evaluation unit 606 is further configured to: Based on the task prompts and the input data, multiple quality inspection agent systems are invoked to execute the evaluation task. Based on the results of the evaluation task executed by the multiple quality inspection agent systems, the standard solution program, and the verification script, a difficulty evaluation of the evaluation task is generated.

[0139] More preferably, the device 60 further includes a difficulty adjustment unit 308, used to adjust the difficulty of the evaluation task based on the difficulty evaluation, including: when the pass rate is greater than a first preset threshold, determining that the task difficulty is too low and triggering a difficulty increase operation; when the pass rate is less than a second preset threshold, determining that the task difficulty is too high and triggering a difficulty decrease operation or automatic diagnosis. The automatic diagnosis includes: performing error type analysis on the execution logs of the quality inspection agents; if the execution logs of quality inspection agents exceeding a preset threshold report the same type of error, it is determined to be a defect in the evaluation task itself; if each quality inspection agent fails at different positions in the ordered chain of steps, it is determined to be too difficult and marked as a high-difficulty task; if a verified standard solution program fails to pass verification when re-executed in the same isolated environment, it is determined to be an environmental problem.

[0140] More preferably, the difficulty increase operation performed by the difficulty adjustment unit 308 includes at least one of the following: increasing the proportion of data quality problems injected into the input data; increasing the number of constraints in the structured task description; increasing the data size of the input data; increasing the number of steps in the ordered step chain; and reducing the level of detail of the prompt information provided in the task prompt words.

[0141] It should be noted that the technical solution of this device and the technical solution of the aforementioned Agent system evaluation method belong to the same concept. For details not described in detail in the technical solution of the evaluation device, please refer to the description of the technical solution of the aforementioned Agent system evaluation method.

[0142] Accordingly, this specification also provides a computer-readable storage medium having executable computer program instructions stored thereon; wherein, when executed by a processor, the instructions implement the steps in the evaluation method flow of the Agent system described above.

[0143] Accordingly, this specification also provides a computer program product having executable computer program instructions stored thereon; wherein, when the computer program instructions are executed by a processor, they implement the steps in the evaluation method flow of the Agent system described above.

[0144] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, when implementing one or more of these specifications, the functions of each module can be implemented in one or more software and / or hardware components, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between devices or units, and may be electrical, mechanical, or other forms.

[0145] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0146] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0147] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0148] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, system, or computer program product. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0149] One or more embodiments of this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a particular task or implement a particular abstract data type. One or more embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0150] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, system embodiments are basically similar to method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. In the description of this specification, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0151] The above description is merely an embodiment of one or more embodiments of this specification and is not intended to limit the scope of this specification. Various modifications and variations can be made to the one or more embodiments of this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims.

Claims

1. A method for evaluating an agent system, characterized in that, include: Based on at least one task source, use an evaluation task generation model to generate a natural language task draft that includes a description of data manipulation and a description of expected output. The task draft is decomposed into a structured task description containing an ordered chain of steps using an evaluation task generation model, and the structured task description is then subjected to format and logic validation. Acquire input data related to the evaluation task, and synchronously generate the correct answer corresponding to the input data based on deterministic computation; Based on the task draft and the input data, the evaluation task generation model is used to generate task prompts that describe the target state without revealing the solution, and information leakage verification is performed on the generated task prompts. A standard solution program is generated using an evaluation task generation model, along with a verification script containing strong assertions or tolerance assertions. The standard solution program is then executed in an isolated environment, and the verification script is used to verify the standard solution program and its output. The strong assertions are used to precisely match the output with the correct answer; the tolerance assertions are used to match the output with the correct answer within a specified deviation range, where a tolerance range is defined for each deviation. When the standard solution program passes the verification script, the target agent system to be evaluated is invoked to perform the evaluation task based on the task prompt and the input data. Based on the result of the target agent system performing the evaluation task, as well as the standard solution program and the verification script, the evaluation result of the target agent system is generated.

2. The method according to claim 1, characterized in that, The source of the task includes at least one of the following: Task sources can be generated from domain knowledge, extracted from academic papers, extracted from tutorial assignments, generated in reverse from skill combinations, or generated from existing data materials.

3. The method according to claim 2, characterized in that, Based on task sources generated inversely from skill combinations, a natural language task draft containing data manipulation descriptions and expected output descriptions is generated using an evaluation task generation model, including: Select two or more skills from a pre-built skill library containing various data processing skills to form a skill combination; The skill combination is input as a constraint into the evaluation task generation model, which then generates a task draft that includes a description of data operations and the expected output, and requires the use of each skill in the skill combination to complete.

4. The method according to claim 3, characterized in that, The process of generating a natural language task draft containing data operation descriptions and expected output descriptions using an evaluation task generation model, based on task sources generated inversely from skill combinations, also includes: Iterate through the different skill combinations in the skill library to generate a set of task drafts that cover the target skill space; During the traversal, skill combinations that do not have collaborative application scenarios are eliminated, and semantic deduplication is performed on the generated task drafts.

5. The method according to claim 1, characterized in that, The format validation includes: The output of the evaluation task generation model is subjected to format-forced validation using a predefined structured data pattern. When the output does not conform to the structured data pattern, the non-conforming field information is extracted and appended to the original prompt words of the task generation model to request regeneration. The logical verification includes: Verify whether the ordered chain of steps forms a directed acyclic graph, and whether the input of each step can be provided by the output of the preceding step or the initial input data.

6. The method according to claim 1, characterized in that, The process of acquiring input data related to the evaluation task and synchronously generating the correct answer corresponding to the input data based on deterministic computation includes at least one of the following processes: Input data is synthesized programmatically based on the data patterns defined in the structured task description, and the correct answer corresponding to the input data is calculated by a deterministic algorithm based on preset statistical characteristic parameters. Furthermore, pre-collected existing data materials with basic annotation information are used as input data, and a predefined standard analysis program is used to perform deterministic calculations on the data materials to generate the correct answer corresponding to the input data.

7. The method according to claim 6, characterized in that, The process of programmatically synthesizing input data based on the data patterns defined in the structured task description, and calculating the correct answer corresponding to the input data using a deterministic algorithm based on preset statistical characteristic parameters, includes: The data pattern of the input data is determined based on the data fields and field constraints defined in the structured task description; Set the statistical characteristic parameters of the input data, including the distribution type and parameters of each data field, and the correlation coefficient between fields; Input data is generated programmatically through a random process based on the aforementioned statistical characteristic parameters; The correct answer is calculated using a deterministic algorithm based on the same statistical characteristic parameters, such that the correct answer is a deterministic function of the input data and its statistical characteristic parameters.

8. The method according to claim 7, characterized in that, The process involves using pre-collected existing data materials with basic annotation information as input data, and performing deterministic calculations on the data materials using a predefined standard analysis program to generate the correct answer corresponding to the input data, including: Select multimodal data materials that match the task draft from the data material library, wherein the multimodal data materials include at least one of domain-specific images, experimental record data, sensor acquisition data, and professional documents; The data materials are standardized preprocessed, including format unification, field naming standardization, and data quality labeling, so that the existing data materials have basic labeling information or have a defined standard analysis procedure; The standard analysis procedure is used to perform deterministic calculations on the data material to generate a correct answer, which is a deterministic mapping of the basic annotation information.

9. The method according to claim 1, characterized in that, The step of performing information leakage verification on the generated task prompt words includes: The generated task prompts are semantically similar to key code snippets in the standard solution program. Detect whether the task prompt contains a library name, function name, or algorithm name; If the verification fails, the failed items will be fed back to the evaluation task generation model to regenerate task prompts.

10. The method according to claim 1, characterized in that, The verification script also includes process compliance checks, and the verification of the standard solution program using the verification script includes: Verify whether network resources outside the isolated environment are accessed during the execution of the standard solution program in the isolated environment; Verify whether files outside the specified path of the evaluation task are modified during the execution of the standard solution program in the isolated environment; Verify whether the execution time of the standard solution program in the isolated environment exceeds a preset upper limit.

11. The method according to claim 1 or 10, characterized in that, The step of verifying the standard solution program using the verification script includes: Check whether the standard solution procedure references any external resources not mentioned in the task prompt; Examine whether the standard solution procedure hardcodes the correct answer into the solution; Verify whether the standard solution program can run independently.

12. The method according to claim 1, further comprising, before invoking the target Agent system to be evaluated to execute the evaluation task: Based on the task prompts and the input data, multiple quality inspection agent systems are invoked to execute the evaluation task. Based on the results of the evaluation task executed by the multiple quality inspection agent systems, the standard solution program, and the verification script, a difficulty evaluation of the evaluation task is generated.

13. The method of claim 12, further comprising: Based on the aforementioned difficulty assessment, the difficulty of the evaluation task is adjusted, including: When the pass rate is greater than the first preset threshold, the task difficulty is determined to be too low, and a difficulty increase operation is triggered. When the pass rate is less than the second preset threshold, the task difficulty is determined to be too high, triggering a difficulty reduction operation or automatic diagnosis. The automatic diagnosis includes: Error type analysis is performed on the execution logs of the quality inspection agents. If the execution logs of quality inspection agents exceeding a preset threshold report the same type of error, it is determined to be a defect in the evaluation task itself. If each quality inspection agent fails at different positions in the ordered chain of steps, it is determined to be too difficult and marked as a high-difficulty task. If a standard solution program that has been verified fails to pass verification when re-executed in the same isolated environment, it is determined to be an environmental problem.

14. The method according to claim 13, wherein the difficulty-increasing operation includes at least one of the following: Increase the proportion of data quality issues injected into the input data; Increase the number of constraints in the structured task description; Increase the size of the input data; Increase the number of steps in the ordered chain of steps; Reduce the level of detail in the prompts provided in the task prompts.

15. An evaluation device for an agent system, comprising: A generation unit is used to generate a natural language task draft containing a data manipulation description and a desired output description based on at least one task source and using an evaluation task generation model. The task draft is decomposed into a structured task description containing an ordered chain of steps using an evaluation task generation model. Acquire input data related to the evaluation task, and synchronously generate the correct answer corresponding to the input data based on deterministic computation; Based on the task draft and the input data, a task prompt word describing the target state without revealing the solution is generated using an evaluation task generation model; a standard solution program and a verification script containing strong assertions or tolerance assertions are generated using the same evaluation task generation model; wherein, the strong assertion is used to precisely match the output with the correct answer; the tolerance assertion is used to match the output with the correct answer within a deviation range, wherein the deviation has a tolerance range. The verification unit is used to perform format and logic verification on the structured task description, perform information leakage verification on the generated task prompts, execute the standard solution program in an isolated environment, and use the verification script to verify the standard solution program and its output. And, an evaluation unit, configured to, when the standard solution program passes the verification script, invoke the target Agent system to be evaluated to perform an evaluation task based on the task prompt and the input data, and generate an evaluation result of the target Agent system based on the result of the target Agent system performing the evaluation task, the standard solution program / and the verification script.

16. An electronic device, characterized in that, include: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the evaluation method for the Agent system as described in any one of claims 1 to 14 by running the executable instructions.

17. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by the processor, this instruction implements the steps of the evaluation method for the Agent system as described in any one of claims 1 to 14.

Images