An AI programming agent fuzz testing method and system based on real abnormal patterns

Abstract

The application discloses an AI programming agent fuzzy test method and system based on a real abnormal mode, which extracts an interaction mode and an action type based on a real abnormal report, generates a seed template, and executes a test after warehouse-level task instantiation, wherein a test case is in line with a real fault scene, can effectively reproduce the actual abnormal behavior of an AI programming agent, and improves test effectiveness and scene coverage; a use case isolation and continuous conversation execution mechanism is adopted to guarantee independent test environment and coherent state, combined with a multi-source evidence track package to realize full-link data retention, and through automatic abnormal determination and classification and cross-model analysis, abnormal grading induction is completed, the test result has strong reproducibility, the abnormal determination precision is high, and the test analysis efficiency is significantly improved.

Description

Technical Field

[0001] This invention relates to the field of AI programming technology, specifically to a fuzz testing method and system for AI programming agents based on real-world anomaly patterns. Background Technology

[0002] With the rapid development of AI programming agent technology, its application in development scenarios such as code generation, defect repair, and file organization is becoming increasingly widespread. The reliability of the agent's behavior and the accuracy of its output directly affect development efficiency and code quality. Fuzz testing, as a core method of automated reliability testing, has been gradually applied to the behavior verification of AI programming agents.

[0003] Although existing technologies offer many benefits, the following problems still exist:

[0004] Existing fuzzing solutions mostly use randomized test case generation or fixed rule construction, without targeting test case construction based on abnormal behavior patterns reported by real users. The generated test cases have a low degree of matching with fault scenarios in the actual production environment, making it difficult to reproduce the abnormal behavior of the agent in real use. The effectiveness and scenario coverage of the test are significantly insufficient.

[0005] Secondly, existing testing solutions mostly adopt a shared workspace reuse execution mode, which can easily cause file environment interference between different test cases, resulting in poor reproducibility of test results. Moreover, anomaly detection often relies on the verification of a single output result, lacks cross-support from multi-source evidence across the entire chain, has low anomaly localization accuracy, requires a large amount of manual intervention for screening, and has low overall testing efficiency. Summary of the Invention

[0006] The purpose of this section is to outline some aspects of the embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0007] 1. Technical problems to be solved:

[0008] To address the issues mentioned above, existing fuzz testing solutions often employ randomized test case generation or fixed rule construction, failing to target test case construction based on abnormal behavior patterns reported by real users. Consequently, the generated test cases have a low degree of matching with fault scenarios in the actual production environment, making it difficult to reproduce the abnormal behavior of the agent during real-world use. This results in significant deficiencies in test effectiveness and scenario coverage.

[0009] Secondly, existing testing solutions mostly adopt a shared workspace reuse execution mode, which can easily cause file environment interference between different test cases, resulting in poor reproducibility of test results. Furthermore, anomaly detection often relies on the verification of a single output result, lacks cross-support from multi-source evidence across the entire chain, has low anomaly localization accuracy, requires a large amount of manual intervention for screening, and has low overall testing efficiency. Therefore, this invention is proposed to address these issues.

[0010] Therefore, the purpose of this invention is to provide a fuzz testing method and system for AI programming agents based on real anomaly patterns. It extracts interaction patterns and action types based on real anomaly reports and generates seed templates. It also employs test case isolation and continuous session execution mechanisms to ensure the independence of the test environment and the continuity of the state.

[0011] 2. Technical Solution:

[0012] To address the aforementioned technical problems, according to one aspect of the present invention, the present invention provides the following technical solution:

[0013] A fuzz testing method for AI programming agents based on real-world anomaly patterns includes the following steps:

[0014] S1. Collect and filter abnormal reports and evidence, collect real abnormal reports and screen out valid abnormal samples with complete execution trajectories;

[0015] S2. Extract interaction patterns and action types from valid abnormal samples;

[0016] S3. Perform compatibility screening and seed template generation. Perform compatibility verification between the interaction mode and the action type, and generate seed templates after retaining valid pairs that can be combined and executed.

[0017] S4. Instantiate the repository-level task by binding the seed template to the target code repository context and instantiating it to generate multiple independent repository-level test tasks.

[0018] S5. Using a test case isolation and continuous session execution mechanism, an independent running environment is created for each of the warehouse-level test tasks, and the AI ​​programming agent under test is scheduled to execute all the steps of the warehouse-level test task in sequence.

[0019] S6. Perform multi-source evidence collection and trajectory package construction, collect multi-source data at each step of the task execution process, and generate multi-source evidence trajectory packages with unified identifiers;

[0020] S7. Perform automatic anomaly judgment and classification, perform multi-dimensional verification on the multi-source evidence trajectory package, and output the anomaly classification result after classifying the anomaly level according to the degree of deviation.

[0021] S8. Perform anomaly reproduction and cross-model analysis. Reproduce and verify the classified anomaly samples and compare them across models to complete anomaly clustering and root cause induction.

[0022] As a preferred embodiment of the AI ​​programming agent fuzzy testing method based on real anomaly patterns of the present invention, the interaction mode is set with a task flow of multi-step operation and cross-step state continuation, the action type is set with an operation category corresponding to the coding tool that can be called; the compatibility filtering is set with execution adaptability verification rules, and only the interaction mode and action type that can be combined are paired.

[0023] As a preferred embodiment of the AI ​​programming agent fuzz testing method based on real anomaly patterns of the present invention, the seed template is configured with a predefined Prompt framework, an adaptive seed field and a target repository context, and is configured with step number boundaries, path constraint range and single seed test case generation quantity configuration items, and is configured with executable test case filtering rules.

[0024] As a preferred embodiment of the AI ​​programming agent fuzz testing method based on real anomaly patterns of the present invention, the warehouse-level test task is configured with a basic identifier field, an instruction sequence field, a rollback verification field, an expected state field, and an execution constraint field; the basic identifier field is configured with a unique test case ID and an associated seed template ID storage location, and the execution constraint field is configured with a full relative path constraint, a path traversal prohibition rule, and a step number limit.

[0025] As a preferred embodiment of the AI ​​programming agent fuzz testing method and system based on real anomaly patterns of the present invention, the test case isolation and continuous session execution mechanism in step S5 is configured with an independent repository baseline snapshot and a dedicated work area, and a continuous agent session unit is configured. All execution steps within the same test case are executed sequentially in the continuous agent session unit, and a single-step independent timeout rule is configured.

[0026] As a preferred embodiment of the AI ​​programming agent fuzzy testing method based on real anomaly patterns of the present invention, in step S6, when constructing the multi-source evidence trajectory package, a unified identifier for the trajectory package in the form of case_id+step_id is set, and a prompt layer evidence set, an action layer evidence set, a state layer evidence set, and a response layer evidence set are set. The prompt layer evidence set is set with storage items for step instructions, sequence constraints, and prohibition conditions; the action layer evidence set is set with storage items for tool call records, terminal commands, and return results; the state layer evidence set is set with storage items for file addition, deletion, and modification sets, generated artifacts, exit codes, and time consumption; and the response layer evidence set is set with storage items for completion status statements, failure reason explanations, and final response text.

[0027] As a preferred embodiment of the AI ​​programming agent fuzzy testing method based on real anomaly patterns of the present invention, the automatic anomaly determination and classification in step S7 is provided with an evidence validity verification unit, an instruction compliance check unit and an action consistency check unit, a three-level anomaly level classification rule, and a standardized anomaly classification result output item.

[0028] A system for fuzz testing of AI programming agents based on real anomaly patterns, comprising an input layer, a core processing layer, and an output storage layer;

[0029] The input layer is equipped with a real anomaly report library, a real code repository set, and the AI ​​programming agent under test.

[0030] The core processing layer includes an anomaly report collection module, a behavior abstraction module, a seed template generation module, a warehouse-level task instantiation module, an isolation execution module, an evidence collection module, an anomaly judgment module, and a review and analysis module.

[0031] The output storage layer includes a trajectory package database, an anomaly classification result set, and a cross-model analysis report library.

[0032] As a preferred embodiment of the system for a fuzzy testing method for AI programming agents based on real anomaly patterns according to the present invention, the behavior abstraction module is provided with an interaction mode and action type extraction unit and a compatibility matching unit; the seed template generation module is provided with a constraint configuration unit; and the repository-level task instantiation module is provided with a target code repository binding unit.

[0033] As a preferred embodiment of the system for a fuzzy testing method for AI programming agents based on real anomaly patterns according to the present invention, the evidence collection module is provided with a multi-source data collection unit and a hierarchical trajectory package construction unit; the anomaly determination module is provided with a validity verification unit, an instruction compliance verifier, and an action consistency verifier; and the review and analysis module is provided with an anomaly reproduction verification unit and a cross-model comparison analysis unit.

[0034] 3. Beneficial effects:

[0035] Compared with the prior art, the beneficial effects of the present invention are:

[0036] This fuzz testing method and system for AI programming agents based on real-world anomaly patterns extracts interaction patterns and action types from real-world anomaly reports and generates seed templates. After instantiation into warehouse-level tasks, the tests are executed. The test cases closely match real-world fault scenarios, effectively reproducing the actual abnormal behavior of AI programming agents and improving test effectiveness and scenario coverage.

[0037] This AI programming agent fuzzy testing method and system based on real anomaly patterns employs test case isolation and continuous session execution mechanisms to ensure the independence of the test environment and the continuity of the state. It combines multi-source evidence trajectory packages to achieve full-link data retention. Through automatic anomaly judgment and classification and cross-model analysis, it completes the hierarchical summarization of anomalies. The test results are highly reproducible, the anomaly judgment accuracy is high, and the test analysis efficiency is significantly improved. Attached Figure Description

[0038] To more clearly illustrate the technical solutions of the embodiments of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and detailed embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0039] Figure 1 This is a schematic diagram illustrating the working principle of an AI programming agent fuzzy testing method and system based on real anomaly patterns according to the present invention.

[0040] Figure 2 This is a flowchart illustrating the overall process of a fuzz testing method and system for AI programming agents based on real-world anomaly patterns, as described in this invention.

[0041] Figure 3 This is a system architecture diagram of an AI programming agent fuzz testing method and system based on real anomaly patterns according to the present invention.

[0042] Figure 4 This invention provides a flowchart of the behavior abstraction and seed template generation process for an AI programming agent fuzzy testing method and system based on real anomaly patterns.

[0043] Figure 5 This is a warehouse-level test task object structure diagram of an AI programming agent fuzzy testing method and system based on real anomaly patterns according to the present invention.

[0044] Figure 6 This is a schematic diagram of the isolated execution environment mechanism of an AI programming agent fuzz testing method and system based on real anomaly patterns according to the present invention.

[0045] Figure 7 This invention presents a hierarchical structure diagram of multi-source evidence trajectory packets in an AI programming agent fuzzy testing method and system based on real anomaly patterns.

[0046] Figure 8 This is a flowchart illustrating the automatic anomaly determination and classification process of an AI programming agent fuzzy testing method and system based on real anomaly patterns, as described in this invention. Detailed Implementation

[0047] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0048] This invention is described in detail with reference to the schematic diagrams. When describing the embodiments of this invention, for ease of explanation, the cross-sectional views illustrating the device structure may be partially enlarged, not according to the usual scale. Furthermore, the schematic diagrams are merely examples and should not be construed as limiting the scope of protection of this invention. In actual fabrication, the three-dimensional spatial dimensions of length, width, and depth should be included.

[0049] The orientation or positional relationship indicated in the terminology is based on the orientation or positional relationship shown in the accompanying drawings and is only for the convenience of describing the invention and simplifying the description, and is not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention.

[0050] The term "connection method" should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be a connection within two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0051] The embodiments of the present invention will now be described in further detail with reference to the accompanying drawings.

[0052] Please see Figures 1-8 This embodiment of the fuzz testing method for AI programming agents based on real anomaly patterns includes the following steps:

[0053] S1. Collect and filter abnormal reports and evidence, collect real abnormal reports and screen out valid abnormal samples with complete execution trajectories;

[0054] S2. Extract interaction patterns and action types from valid abnormal samples;

[0055] S3. Perform compatibility screening and seed template generation. Verify the compatibility of interaction modes and action types, retain valid pairs that can be combined and executed, and then generate seed templates.

[0056] S4. Instantiate repository-level tasks by binding the seed template to the target code repository context and instantiating multiple independent repository-level test tasks.

[0057] S5. Employs test case isolation and continuous session execution mechanisms to create an independent runtime environment for each repository-level test task and schedules the AI ​​programming agent under test to execute all steps of the repository-level test task sequentially.

[0058] S6. Perform multi-source evidence collection and trajectory package construction, collect multi-source data at each step of the task execution process, and generate multi-source evidence trajectory packages with unified identifiers;

[0059] S7. Perform automatic anomaly judgment and classification, perform multi-dimensional verification on multi-source evidence trajectory packages, and output the anomaly classification results after classifying the anomaly level according to the degree of deviation.

[0060] S8. Perform anomaly reproduction and cross-model analysis. Reproduce and verify the classified anomaly samples and compare them across models to complete anomaly clustering and root cause induction.

[0061] It is worth noting that, in order to extract reusable behavioral elements from real-world failure scenarios and ensure that the generated test cases fit actual abnormal scenarios and have the ability to recreate scenarios, the interaction patterns are specifically extracted from the complete interaction chain of valid abnormal samples, including multi-step operations and task flows with cross-step state continuation; the action types correspond one-to-one with the coding tool interfaces that the tested AI programming agent can call, including the operation categories of the corresponding callable coding tools; the compatibility screening has built-in execution adaptability verification rules, which verify the logical connection and tool support of the interaction patterns and action types through the rule base, and only retain the pairing of interaction patterns and action types that can be combined and executed.

[0062] Next, to generate standardized, executable test case templates in batches based on extracted behavioral elements, and to constrain the execution boundaries and generation scale of test cases, specifically, the seed template embeds a predefined Prompt framework, adaptive seed fields, and target repository context. The predefined Prompt framework fixes the input format of the test task, and the adaptive seed fields automatically fill in variable content according to the characteristics of the target repository. The seed template is configured with step number boundaries, path constraint range, and single seed test case generation quantity configuration items to limit the upper limit of steps, operation path range, and number of derived test cases per template for a single test case. The seed template has built-in executable test case filtering rules, which filter out template instances that cannot be parsed and executed normally through syntax parsing and feasibility verification.

[0063] Meanwhile, to map the generalized seed template to a complete test task that can run in a specified code repository, and to clarify the execution content, recovery rules, and expected state of the task, the repository-level test task is specifically divided into a basic identifier field, an instruction sequence field, a rollback verification field, an expected state field, and an execution constraint field. The basic identifier field is located in the header area of ​​the task object and internally stores a unique test case ID and an associated seed template ID, used to uniquely identify the task and trace its source template. The instruction sequence field concatenates the single-step operation instructions in the execution order. The rollback verification field records the environment recovery operation and verification logic after the task is executed. The expected state field records the file status and expected execution result after the task is completed. The execution constraint field is embedded in the task's runtime configuration and internally limits the relative path constraints, prohibits path traversal rules, and limits the number of steps, used to limit the task's execution rules and security boundaries.

[0064] Furthermore, to ensure complete isolation and non-interference between the runtime environments of each test case, while maintaining the continuity of state and context of multi-step operations within a single test case, and improving the accuracy and reproducibility of test results, specifically, step S5's test case isolation and continuous session execution mechanism is configured with independent repository baseline snapshots and dedicated workspaces. The independent repository baseline snapshot is generated by cloning a full copy of the target code repository, with each test case corresponding to an independent snapshot copy. The dedicated workspace corresponds to an independent file system directory for each test case, with files isolated from each other. The mechanism is configured with continuous agent session units, with each test case corresponding to an independent agent session. All execution steps within the same test case are executed sequentially in the continuous agent session unit, and the workspace file state and session context are preserved across steps. The mechanism is configured with independent timeout rules for each step, integrated into the execution scheduler, to separately control the execution time of each step. If a timeout occurs, the current step is terminated and an exception is recorded.

[0065] It is worth noting that, in order to fully record the entire chain of data during the test execution process, covering the entire process of instruction input, tool execution, state change, and agent output, and to provide multi-dimensional and traceable evidence support for subsequent anomaly detection, specifically, when constructing the multi-source evidence trajectory package in step S6, the trajectory package header is configured with a unified trajectory package identifier in the form of case_id+step_id, which serves as the unique associated primary key for single-step evidence data; the trajectory package is configured with hierarchical layers including a prompt layer evidence set, an action layer evidence set, a state layer evidence set, and a response layer evidence set; the prompt layer evidence set is collected from the instruction data issued by the test task and internally stores step instructions, Sequence constraints and prohibition conditions are used to store instructions and constraint information issued by the test terminal; the action layer evidence set is collected from the call logs and terminal outputs of the coding tool, and internally stores tool call records, terminal commands and return results, used to store the full operation record of the tool; the state layer evidence set is collected from the periodic file scan results of the work area, and internally stores the file addition, deletion and modification set, generated artifacts, exit codes and time consumption, used to store the actual environment state after execution; the response layer evidence set is collected from the output content of the AI ​​programming agent, and internally stores the completion status statement, failure reason explanation and final response text, used to store the descriptive output of the agent.

[0066] Finally, to automatically identify abnormal agent behavior during the testing process, classify it according to the degree of impact of the abnormality, and improve the efficiency of abnormality analysis by replacing manual screening, specifically, the automatic abnormality judgment and classification process in step S7 has a built-in evidence validity verification unit, which is used to verify the data integrity of the trajectory package. Use cases that lack core evidence are marked as invalid use cases. The process is configured with an instruction compliance check unit and an action consistency check unit. The instruction compliance check unit compares the actual execution content of the agent with the instruction requirements to check the compliance of instruction execution. The action consistency check unit compares the output description of the agent with the actual file status to check the consistency between the behavior description and the actual execution result. The process has a built-in three-level abnormality classification rule. According to the deviation degree and impact range of the inspection results, abnormalities are divided into three levels: serious abnormality, expected result abnormality, and minor abnormality. If there is no deviation, it is marked as no abnormality. The end of the process is configured with a standardized abnormality classification result output interface to output structured abnormality classification results.

[0067] A system for implementing the above-mentioned fuzz testing method according to this embodiment includes an input layer, a core processing layer and an output storage layer. The input layer is connected to the input end of the core processing layer through a data interface, and the output end of the core processing layer is connected to the output storage layer through a data read / write interface.

[0068] The input layer has a built-in real exception report library, a real code repository set, and the AI ​​programming agent under test. The real exception report library is used to store the raw exception report data reported by users, the real code repository set is used to provide the code running environment required for testing, and the AI ​​programming agent under test is the target object of testing.

[0069] The core processing layer consists of an anomaly report collection module, a behavior abstraction module, a seed template generation module, a warehouse-level task instantiation module, an isolation execution module, an evidence collection module, an anomaly judgment module, and a review and analysis module, which are sequentially connected through a data transmission link to complete the entire process of anomaly handling, task generation, execution scheduling, evidence collection, and anomaly analysis.

[0070] The output storage layer has a built-in trajectory package database, anomaly classification result set, and cross-model analysis report library. The trajectory package database is used to store multi-source evidence trajectory package data generated by full-scale testing, the anomaly classification result set is used to store structured anomaly classification results, and the cross-model analysis report library is used to store analysis reports on anomaly reproduction and cross-model comparison.

[0071] The behavior abstraction module integrates an interaction pattern and action type extraction unit and a compatibility matching unit. The extraction unit extracts interaction pattern and action type elements from the input exception report, and the compatibility matching unit verifies the execution compatibility of the two. The seed template generation module integrates a constraint configuration unit, which configures the steps, paths, and other constraint parameters of the template to generate standardized seed templates. The repository-level task instantiation module integrates a target code repository binding unit, which binds the seed template to the specified repository context to generate an executable repository-level test task.

[0072] The evidence collection module integrates a multi-source data collection unit and a hierarchical trajectory package construction unit. The multi-source data collection unit collects four types of data during the execution process: instructions, tools, status, and responses. The hierarchical trajectory package construction unit assembles and generates hierarchical multi-source evidence trajectory packages using case_id + step_id as identifiers. The anomaly detection module integrates a validity verification unit, an instruction compliance verifier, and an action consistency verifier, corresponding to evidence validity verification, instruction compliance verification, and behavior consistency verification, respectively, and classifies anomalies into three levels according to the degree of deviation. The review and analysis module integrates an anomaly reproduction verification unit and a cross-model comparison analysis unit, used to reproduce and confirm the identified anomalies, compare the anomaly performance of different AI programming agents, and complete anomaly clustering and root cause induction.

[0073] Example 1: Single-warehouse, single-model basic fuzzy testing scenario

[0074] This embodiment represents a standard test scenario for a single code repository and a single AI programming agent, focusing on verifying the reliability of agent behavior in open-source tool code repositories. The specific implementation process is as follows:

[0075] 1. Define the test object and input samples. The target repository is a Python command-line tool project with 50,000 lines of code, including a unit test suite and a log output module. The test object is a mainstream code generation agent that supports three types of coding tools: file reading and writing, command execution, and code editing. The system accesses more than 300 historical user anomaly reports from this repository. After evidence filtering, 120 valid samples with complete reproduction paths are retained.

[0076] 2: Complete the extraction of behavioral elements and compatibility screening. The behavior abstraction module extracts three types of interaction modes, namely file operation, test execution, and code repair, and nine corresponding action types from the valid samples. After verifying the logical connection and tool support by the execution adaptability verification rules, sixty-eight sets of valid pairs that can be combined and executed are formed.

[0077] 3: Generate standardized test seeds and task instances. The seed template generation module generates standardized seed templates based on valid pairings. Each template is configured to generate twenty test cases, each task has a maximum of eight steps, and the operation path is limited to the project's src and tests directories. After validation by executable filtering rules, more than 1,200 executable repository-level test tasks are generated.

[0078] 4: Perform isolation tests and collect evidence synchronously. The isolation execution module creates an independent repository baseline snapshot and a dedicated workspace for each test task, and schedules the tested agent to execute the task steps sequentially within a single session, with a single-step timeout set to sixty seconds. During the execution, the evidence collection module constructs multi-source evidence trajectory packages step by step, and all data is stored in the trajectory package database.

[0079] 5: Completed automatic anomaly detection and classification. The anomaly detection module performed multi-dimensional verification on all trajectory packages and finally identified 12 serious anomalies, 37 cases of expected result anomalies, and 61 cases of minor anomalies. The remaining cases were judged as normal.

[0080] 6: Output analysis results and verify application value. The review and analysis module reproduces and verifies serious anomalies. After summarizing the root causes of anomalies, it outputs a single warehouse test analysis report. The entire batch of tests was executed in parallel and took about 3.5 hours. Compared with the manual test case design and verification mode, the efficiency is improved by more than 7 times. All anomalies are accompanied by a complete step-by-step evidence chain, which can be directly used for the localization and repair effect verification of intelligent agent defects.

[0081] Example 2: Multi-model lateral comparison test scenario

[0082] This embodiment presents a comparative test scenario of multiple AI programming agents. Using the same test case set, it verifies the differences in abnormal behavior and reliability levels of different agents. The specific implementation process is as follows:

[0083] 1. Configure a basic environment for multi-model comparison testing. The input layer connects to three AI programming agents with different architectures. The warehouse-level task instantiation module generates three sets of warehouse-level test tasks with completely identical parameters for the same seed template. Each set of tasks corresponds to one AI agent under test.

[0084] 2: Establish a unified standard isolated execution environment. The isolated execution module allocates an independent repository baseline snapshot and a dedicated workspace with the same configuration to each group of tasks, ensuring that the initial test environment, instruction content, and constraint rules of the three agents are completely consistent. The three agents execute the same test tasks synchronously in their respective continuous agent session units, all following the unified single-step timeout rules and path constraints.

[0085] 3: Construct standardized evidence packages that can be aligned horizontally. The evidence collection module constructs an independent multi-source evidence trajectory package for each intelligent agent. All trajectory packages adopt a unified case_id+step_id identification rule to ensure that evidence data of the same step can be accurately aligned horizontally.

[0086] 4: Implement unified standard automatic anomaly detection. The anomaly detection module adopts unified verification rules and hierarchical standards to synchronously perform automatic anomaly detection and classification on all trajectory packages, ensuring that the detection results of different models are horizontally comparable.

[0087] 5. Conduct multi-dimensional horizontal comparative analysis. The review and analysis module conducts horizontal comparison of the abnormal performance of different agents under the same test case, and statistically analyzes indicators such as the abnormality occurrence rate, abnormality type distribution, and severe abnormality ratio of each model. Set four core evaluation indicators: single-step execution accuracy, multi-step task completion rate, abnormal self-recovery capability, and instruction compliance. Summarize the commonalities and differences in abnormal behaviors of different models.

[0088] 6. Output multi-model comparative analysis results, and finally generate a multi-model comparative analysis report with visual statistical charts. This can provide quantitative basis for agent selection and provide precise defect scenario input for the R&D team to optimize the reliability of model behavior.

[0089] Combination Figures 1-8 The specific usage process of the AI ​​programming agent fuzz testing method and system based on real anomaly patterns in this embodiment is as follows:

[0090] 1. Connect to the test data source, and connect the real exception report library and the real code repository to be tested to the system input layer to complete the interface configuration and permission verification of the AI ​​programming agent under test. It supports mainstream general interface protocols and private repository access authentication. After connection, it automatically performs connectivity detection to ensure that the system can call the agent and access repository resources normally.

[0091] 2: Configure test run parameters. In the seed template generation module, set the upper limit of the number of steps, the range of operation path constraints, and the number of test cases generated by a single seed template. In the exception judgment module, configure the three-level exception grading threshold. The parameter configuration can be saved as a dedicated test plan for later reuse. Enable the cross-model comparison function as needed and add models to be tested.

[0092] 3: Start the automated test task. The system will automatically complete the abnormal report filtering, behavior element extraction, seed template generation, repository-level task instantiation and isolated execution in sequence. It supports parallel scheduling of up to 100 test cases. During the execution process, you can view the progress and abnormal statistics in real time. It automatically collects multi-source data of each step and builds multi-source evidence trajectory packages. No manual intervention is required throughout the process.

[0093] 4. Read test analysis results. After the test is completed, retrieve the anomaly classification result set from the output storage layer. It supports filtering and viewing by anomaly level and type. For the target anomaly, the corresponding multi-source evidence trajectory package can be retrieved to view the full-link details. When cross-model analysis is enabled, the cross-model analysis report with accompanying visualization charts can be read directly to view the anomaly clustering and root cause induction conclusions. The results can be exported to a standardized format for subsequent secondary analysis.

[0094] Although the present invention has been described above with reference to embodiments, various modifications can be made and components can be replaced with equivalents without departing from the scope of the invention. In particular, as long as there is no structural conflict, the features in the disclosed embodiments can be combined with each other in any manner. The lack of an exhaustive description of these combinations in this specification is merely for the sake of brevity and resource conservation. Therefore, the present invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A fuzz testing method for AI programming agents based on real-world anomaly patterns, characterized in that, Includes the following steps: S1. Collect and filter abnormal reports and evidence, collect real abnormal reports and screen out valid abnormal samples with complete execution trajectories; S2. Extract interaction patterns and action types from valid abnormal samples; S3. Perform compatibility screening and seed template generation. Perform compatibility verification between the interaction mode and the action type, and generate seed templates after retaining valid pairs that can be combined and executed. S4. Instantiate the repository-level task by binding the seed template to the target code repository context and instantiating it to generate multiple independent repository-level test tasks. S5. Using a test case isolation and continuous session execution mechanism, an independent running environment is created for each of the warehouse-level test tasks, and the AI ​​programming agent under test is scheduled to execute all the steps of the warehouse-level test task in sequence. S6. Perform multi-source evidence collection and trajectory package construction, collect multi-source data at each step of the task execution process, and generate multi-source evidence trajectory packages with unified identifiers; S7. Perform automatic anomaly judgment and classification, perform multi-dimensional verification on the multi-source evidence trajectory package, and output the anomaly classification result after classifying the anomaly level according to the degree of deviation. S8. Perform anomaly reproduction and cross-model analysis. Reproduce and verify the classified anomaly samples and compare them across models to complete anomaly clustering and root cause induction.

2. The fuzz testing method for AI programming agents based on real anomaly patterns according to claim 1, characterized in that, The interaction mode is set with a task flow that includes multi-step operations and cross-step state continuation. The action type is set with corresponding operation categories that can call coding tools. The compatibility filtering is set with execution adaptability verification rules, which only retain the interaction modes and action types that can be combined and executed.

3. The fuzz testing method for AI programming agents based on real anomaly patterns according to claim 2, characterized in that, The seed template is configured with a predefined Prompt framework, adaptive seed fields and target repository context, and includes configuration items for the number of steps, path constraints and the number of single seed test cases generated, as well as executable test case filtering rules.

4. The fuzz testing method for AI programming agents based on real anomaly patterns according to claim 3, characterized in that, The repository-level test task is configured with a basic identifier field, an instruction sequence field, a rollback verification field, an expected state field, and an execution constraint field. The basic identifier field is configured with a unique test case ID and an associated seed template ID storage location. The execution constraint field is configured with a full relative path constraint, a path traversal prohibition rule, and a step number limit.

5. The fuzz testing method for AI programming agents based on real anomaly patterns according to claim 4, characterized in that, The use case isolation and continuous session execution mechanism in step S5 is configured with an independent repository baseline snapshot and a dedicated workspace, and a continuous agent session unit. All execution steps within the same use case are executed sequentially in the continuous agent session unit, and a single-step independent timeout rule is set.

6. The fuzz testing method for AI programming agents based on real anomaly patterns according to claim 5, characterized in that, In step S6, when constructing the multi-source evidence trajectory package, a unified identifier for the trajectory package in the form of case_id+step_id is set, and a prompt layer evidence set, an action layer evidence set, a state layer evidence set, and a response layer evidence set are set. The prompt layer evidence set includes storage items for step instructions, sequence constraints, and prohibition conditions. The action layer evidence set includes storage items for tool call records, terminal commands, and return results. The state layer evidence set includes storage items for file addition, deletion, and modification sets, generated artifacts, exit codes, and time consumption. The response layer evidence set includes storage items for completion status statements, failure reason explanations, and final response text.

7. The fuzz testing method for AI programming agents based on real anomaly patterns according to claim 6, characterized in that, The automatic anomaly determination and classification in step S7 includes an evidence validity verification unit, an instruction compliance check unit and an action consistency check unit, a three-level anomaly classification rule, and a standardized anomaly classification result output item.

8. A system for implementing the AI ​​programming agent fuzz testing method based on real anomaly patterns as described in any one of claims 1 to 7, characterized in that, It includes an input layer, a core processing layer, and an output storage layer; The input layer is equipped with a real anomaly report library, a real code repository set, and the AI ​​programming agent under test. The core processing layer includes an anomaly report collection module, a behavior abstraction module, a seed template generation module, a warehouse-level task instantiation module, an isolation execution module, an evidence collection module, an anomaly judgment module, and a review and analysis module. The output storage layer includes a trajectory package database, an anomaly classification result set, and a cross-model analysis report library.

9. The AI ​​programming agent fuzz testing system based on real anomaly patterns according to claim 8, characterized in that, The behavior abstraction module is equipped with an interaction mode and action type extraction unit and a compatibility matching unit; the seed template generation module is equipped with a constraint configuration unit; and the repository-level task instantiation module is equipped with a target code repository binding unit.

10. The AI ​​programming agent fuzz testing system based on real anomaly patterns according to claim 9, characterized in that, The evidence collection module is equipped with a multi-source data collection unit and a hierarchical trajectory package construction unit; the anomaly determination module is equipped with a validity verification unit, an instruction compliance verifier, and an action consistency verifier; the review and analysis module is equipped with an anomaly reproduction verification unit and a cross-model comparison analysis unit.

Images