Robustness testing method and apparatus for code agents

Abstract

The application provides a kind of robustness test and device of code agent, and it relates to artificial intelligence technical field.The method comprises: obtaining original instance from original test data set;The original instance includes code repository, problem description, test case and reference repair patch;According to the reference repair patch, determine target file from the corresponding original instance;Based on the target file, carry out repository layer disturbance to the original instance, obtain disturbance instance, to obtain robustness test data set according to the disturbance instance;Run the code agent on the robustness test data set to carry out robustness test, can comprehensively reflect the robustness of code agent when facing real code repository.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a robustness testing method and apparatus for code-based intelligent agents. Background Technology

[0002] In recent years, code agents based on Large Language Models (LLMs) have shown great potential in automating software engineering tasks such as code generation, defect localization, and automated software remediation. To evaluate the real-world capabilities of these agents, researchers have constructed repository-level code generation benchmark suites such as SWE-bench. These benchmarks typically provide instances containing complete code repositories, problem descriptions, test cases, and reference fix patches, enabling tests to determine whether agents can locate problems and generate fix code in complex repository environments.

[0003] However, existing benchmarks primarily focus on evaluating the agent's ultimate ability to generate correct code patches, with test cases mostly derived from well-structured, well-named open-source projects. In such an ideal environment, the agent can rely on surface cues such as clear directory structures and semantic filenames to navigate and understand the repository. Real-world code repositories, however, suffer from structural inconsistencies, non-standard naming conventions, and an abundance of redundant files. Therefore, current code generation benchmark results fail to fully reflect the agent's robustness when facing real-world code repositories. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention provides a robustness testing method and apparatus for code-based intelligent agents.

[0005] This invention provides a robustness testing method for code-based intelligent agents, comprising: Obtain the original instance from the original test dataset; the original instance includes the code repository, issue description, test cases, and reference fix patch; The target file is determined from the corresponding original instance based on the reference repair patch; Based on the target file, the original instance is perturbed at the repository layer to obtain a perturbed instance, and a robustness test dataset is obtained based on the perturbed instance; The code agent is run on the robustness test dataset to perform robustness testing.

[0006] According to the robustness testing method for a code agent provided by the present invention, the structural obfuscation type is determined; The original instance is subjected to the repository layer perturbation in terms of structure based on the structure obfuscation type and the target file.

[0007] According to the robustness testing method for a code agent provided by the present invention, the structural obfuscation type includes import redirection; The structural repository layer perturbation of the original instance based on the structural obfuscation type and the target file includes: The import items that the target file directly imports from an external module are determined based on the first original import statement of the target file; An intermediate proxy module is created based on the imported item; the intermediate proxy module is used to import the imported item from the external module and re-export the imported item. Modify the first original import statement of the target file to obtain a first perturbation statement for indirectly importing the import item from the intermediate proxy module, and perform the repository layer perturbation on the original instance in terms of structure.

[0008] According to the robustness testing method for a code-based intelligent agent provided by the present invention, the structural obfuscation type includes data flow redirection; The structural repository layer perturbation of the original instance based on the structural obfuscation type and the target file includes: Identify the constant configuration parameters of the target file based on its original code. A perturbation configuration file is created based on the constant configuration parameters; the perturbation configuration file contains the constant configuration parameters. Remove the definition of the constant configuration parameter from the target file and add an import statement to the target file to import the constant configuration parameter from the perturbation configuration file, thereby performing the repository layer perturbation on the original instance in terms of structure.

[0009] According to the robustness testing method for a code agent provided by the present invention, the structural obfuscation type includes source file redirection; The structural repository layer perturbation of the original instance based on the structural obfuscation type and the target file includes: Based on the second original import statement of the target file, determine the source file of the import module in the target file; In the directory where the source file is located, a subdirectory with the same name as the source file is created. The source file is moved to the internal implementation file in the subdirectory with the same name, and a package initialization file is created in the subdirectory with the same name. The package initialization file is used to identify the subdirectory with the same name as a package and export the public interface in the internal implementation file. Modify the second import statement of the target file according to the package initialization file to obtain the second perturbation statement for importing the import module from the subdirectory with the same name, and perform the repository layer perturbation on the original instance in terms of structure.

[0010] According to the robustness testing method for a code agent provided by the present invention, the structural obfuscation type includes obfuscated file perturbation; The structural repository layer perturbation of the original instance based on the structural obfuscation type and the target file includes: Based on the naming characteristics of the target file, at least two obfuscated files are created in the directory where the target file is located; the filenames of the obfuscated files are similar to the filenames of the target files.

[0011] According to the robustness testing method for a code-based intelligent agent provided by the present invention, after creating at least two obfuscated files in the directory where the target file is located, the method further includes: Each of the obfuscated files is populated with misleading content; wherein the misleading content is used to interfere with the code agent's ability to identify the target file.

[0012] According to the robustness testing method for a code-based intelligent agent provided by the present invention, the method further includes: Based on the problem description, the original instance is subjected to a prompting layer perturbation; The original instance is perturbed at the code level based on the target file; The obtained perturbation instances include: The original instance is subjected to the prompt layer perturbation, the code layer perturbation, and / or the repository layer perturbation to obtain the perturbation instance.

[0013] According to the robustness testing method for a code-based intelligent agent provided by the present invention, obtaining the perturbation instance includes: Based on the target file, the original instance is subjected to structural perturbation of the repository layer to obtain candidate perturbation instances; If the candidate perturbation instance passes the functional integrity and consistency verification, then the candidate perturbation instance is determined to be a perturbation instance; and / or, Apply the hint layer perturbation, the code layer perturbation, and / or the repository layer perturbation to the original instance to obtain candidate perturbation instances; If the candidate perturbation instance passes the functional integrity and consistency verification, then the candidate perturbation instance is determined to be a perturbation instance.

[0014] According to a robustness testing method for a code-based intelligent agent provided by the present invention, the step of running the code-based intelligent agent on the robustness testing dataset to perform robustness testing includes: The code agent is run on the robustness test dataset to obtain the first test result based on the preset metrics; The code agent is run on the original test dataset to obtain a second test result with preset metrics. Robustness tests were conducted by comparing the results of the first test with those of the second test.

[0015] According to the robustness testing method of the code intelligence agent provided by the present invention, the preset indicators include resolution rate, total number of steps, target file location steps, and number of files opened.

[0016] The present invention also provides a robustness testing apparatus for code-based intelligent agents, comprising: The original instance acquisition module is used to obtain original instances from the original test dataset; the original instances include code repositories, problem descriptions, test cases, and reference fix patches; The target file determination module is used to determine the target file from the corresponding original instance based on the reference repair patch; The perturbation data acquisition module is used to perform warehouse layer perturbation on the original instance based on the target file to obtain perturbation instances, so as to obtain a robustness test dataset based on the perturbation instances; A robustness testing module is used to run the code agent on the robustness testing dataset to perform robustness tests.

[0017] This invention provides a robustness testing method and apparatus for code agents. It obtains original instances from the original test dataset, determines target files from the corresponding original instances based on reference patch repairs, and perturbs the original instances at the repository layer based on the target files to obtain perturbed instances. A robustness test dataset is then obtained based on these perturbed instances, simulating the structural obfuscation of a real code repository. Running the code agent on this robustness test dataset comprehensively reflects the robustness of the code agent when facing a real code repository. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the robustness testing method for code-based intelligent agents provided by this invention.

[0020] Figure 2 This is a schematic diagram of the robustness testing device for code-based intelligent agents provided by the present invention.

[0021] Figure 3This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0023] Early code generation benchmarks primarily evaluated the model's performance on independent programming problems. For example, benchmarks such as HumanEval and LiveCodeBench focused on function-level code generation tasks. However, these benchmarks failed to examine the model's ability to understand and manipulate files across complex project environments.

[0024] Subsequently, repository-level benchmarks such as SWE-bench emerged, reflecting the cross-file reasoning capabilities required of code agents, including understanding module dependencies, tracing function call relationships, and identifying the scope of files that need modification. These benchmarks more closely approximate the actual deployment requirements of code agents. However, these benchmarks are primarily based on well-structured, naming-consistent open-source projects, which typically have clear directory structures, semantic filenames and symbol names, and comprehensive documentation. In this idealized environment, code agents can locate target files using simple heuristics such as keyword matching and filename inference, without needing a deep understanding of the repository's dependencies and organizational structure.

[0025] However, in real-world applications, many code repositories often present more problems, such as: 1) disorganized directory structures, with legacy code mixed with new code; 2) non-standard or even misleading file naming; 3) lack of documentation and comments, resulting in poor code readability; and 4) a large number of redundant files, obsolete modules, and temporary patches. Based on existing repository-level benchmarks, it is impossible to effectively evaluate the performance of agents when faced with disorganized repositories.

[0026] The following is combined with Figures 1 to 3 The present invention describes a robustness testing method and apparatus for code agents, in order to evaluate the robustness of large language model-driven code agents in understanding and manipulating code repositories.

[0027] Figure 1 This is a flowchart illustrating the robustness testing method for code-based intelligent agents provided by the present invention, as shown below. Figure 1 As shown, the method includes the following steps.

[0028] Step 101: Obtain the original instance from the original test dataset.

[0029] The original test dataset refers to the repository-level code generation benchmark dataset used to evaluate the performance of the code agent on repository-level tasks. For example, the original test dataset can be SWE-bench or SWE-bench Verified, etc.

[0030] SWE-bench extracts real issues and corresponding fixes from GitHub projects, requiring the agent to locate defects and generate fixes within a complete code repository environment. SWE-bench Verified is a carefully selected subset of SWE-bench, manually verified to ensure accurate problem descriptions, complete test cases, and clear fixes for each instance.

[0031] Furthermore, repository-level code generation benchmarks are a classification of code generation benchmarks based on task complexity and evaluation scope. Code generation benchmarks are used to evaluate the model's ability to generate correct code, and in addition to repository-level code generation benchmarks, code generation benchmarks also include at least functional code generation benchmarks.

[0032] Here, the original instance refers to the test sample that has not been disturbed in any way. It can be understood that the original test dataset includes multiple original instances.

[0033] In some embodiments, the original instance includes a code repository, an issue description, a test suite, and a reference fix patch.

[0034] In this context, a code repository refers to a complete collection of software project source code containing multiple files, directory structures, and dependencies between files. A problem description is a description in natural language of a problem that needs to be solved or a functional requirement to be implemented in the code repository. Test cases are executable programs used to verify the correctness of the code repository's functionality. Reference patches are standardized representations of modifications needed to the code repository to address the problems described in the code repository, indicating the correct location and content of the modifications.

[0035] It should be noted that there are many ways to obtain the original instance from the original test dataset, such as by reading the dataset's storage file or calling the application programming interface provided by the dataset. This embodiment does not limit the methods used.

[0036] Step 102: Determine the target file from the corresponding original instance based on the reference repair patch.

[0037] It should be noted that there are many ways to determine the target file from the corresponding original instance based on the reference repair patch, such as parsing the uniform difference format of the reference repair patch or performing regular expression matching on the file name of the reference repair patch. This embodiment does not limit this method.

[0038] Step 103: Based on the target file, perform warehouse layer perturbation on the original instance to obtain a perturbed instance, and obtain a robustness test dataset based on the perturbed instance.

[0039] Among them, repository layer perturbation refers to the systematic modification of the file organization, dependencies, naming conventions, etc. of the code repository, in order to reduce the clarity and information content of the code repository structure.

[0040] A perturbation instance refers to a test sample obtained by applying a warehouse layer perturbation to the original instance.

[0041] Robustness test datasets are datasets consisting of multiple perturbation instances used to evaluate the performance of code agents in the face of structurally chaotic environments.

[0042] It should be noted that different repository layer perturbations can be repeatedly applied to the same original instance, or different repository layer perturbations can be repeatedly applied to different original instances to obtain multiple perturbation instances. Based on multiple perturbation instances, a robust test dataset containing diverse structural confusion scenarios can be constructed.

[0043] Step 104: Run the code agent on the robustness test dataset to perform robustness testing.

[0044] It should be noted that perturbation instances in the robustness test dataset can be used as input to provide the code agent under test. This allows the agent to locate defects and generate fix patches in the corresponding perturbation-post-code repository based on the problem description in each perturbation instance. The agent's running process and result data under the perturbation environment can be collected, and the sensitivity and adaptability of the agent to repository structure perturbations can be evaluated based on the running process and result data.

[0045] The robustness testing method for code agents provided in this invention obtains original instances from the original test dataset, determines target files from the corresponding original instances based on reference patch repairs, and performs repository-level perturbation on the original instances based on the target files to obtain perturbed instances. Based on the perturbed instances, a robustness test dataset is obtained, which can simulate the structural obfuscation of a real code repository. In this way, running the code agent on the robustness test dataset to perform robustness testing can comprehensively reflect the robustness of the code agent when facing a real code repository.

[0046] Based on the above embodiments, the step of perturbing the original instance based on the target file includes: Determine the type of structural obfuscation; The original instance is subjected to the repository layer perturbation in terms of structure based on the structure obfuscation type and the target file.

[0047] In this context, structural obfuscation types refer to methods of modifying the organizational structure of a code repository. It's understandable that different structural obfuscation types correspond to different obfuscation scenarios, and each type aims to reduce the clarity of the repository across different dimensions.

[0048] For example, structural obfuscation types may include import redirection, data stream redirection, source file redirection, and obfuscated file perturbation. Among them, import redirection can also be called proxy import perturbation, data stream redirection can also be called dynamic dependency perturbation, source file redirection can also be called in-situ hiding perturbation, and obfuscated file perturbation can also be called dummy file perturbation.

[0049] It should be noted that, based on the structural obfuscation type and target file, perturbations can be applied to the structural level of the original instance's code repository, such as file organization, dependency structure, data storage location, and file naming, to obtain the final perturbed instance.

[0050] Understandably, by perturbing the original instance's repository layer based on the structure obfuscation type and target file, it is possible to specifically simulate the problem of different types of repository structure disorder in real code repositories, making the test dataset more diverse and representative, thereby enabling more refined testing of the robustness of the code agent under different structure obfuscation scenarios.

[0051] Based on the above embodiments, the structural obfuscation type includes import redirection; The structural repository layer perturbation of the original instance based on the structural obfuscation type and the target file includes: The import items that the target file directly imports from an external module are determined based on the first original import statement of the target file; An intermediate proxy module is created based on the imported item; the intermediate proxy module is used to import the imported item from the external module and re-export the imported item. Modify the first original import statement of the target file to obtain a first perturbation statement for indirectly importing the import item from the intermediate proxy module, and perform the repository layer perturbation on the original instance in terms of structure.

[0052] The first original import statement refers to the statement in the target file that was originally used to directly import the required code elements from an external module.

[0053] External modules refer to code module files that are independent of the target file but depend on it.

[0054] Imported items refer to specific code elements that the target file imports from external modules through import statements, including but not limited to classes, functions, variables, and constants.

[0055] The intermediate proxy module, also known as the intermediate proxy layer or proxy module file, is a new module file created in the directory where the target file is located. It is used to indirectly connect the target file and external modules. It imports the required import items from the external module and then re-exports them.

[0056] The first perturbation statement refers to the modified statement in the target file used to import the required import items from the intermediate proxy module.

[0057] It should be noted that dependencies can be indirectly mitigated by introducing an intermediate proxy layer, thereby perturbing the original instance's structure at the repository layer. For example, a proxy module file can be created in the directory containing the target file. The required modules from the target file can be imported into the proxy module file, and the first original import statement in the target file can be changed to a first perturbation statement that imports indirectly from the proxy module file, thus achieving structural repository layer perturbation of the original instance.

[0058] Specifically, you can first analyze the first original import statement in the target file, identify all import items such as classes, functions or variables directly imported from external modules, and create a new proxy module file in the directory where the target file is located.

[0059] The first original import statement can be a direct import containing "from module_a import ClassB, function_c".

[0060] The proxy module file can be proxy_imports.py.

[0061] Next, in the proxy module file, write the first perturbation statement to import the required symbols from the external module, and re-export these symbols using the __all__ list or an explicit export statement to ensure that the proxy module file provides the same interface as the original external module.

[0062] Then modify the import statement of the target file, changing it from importing directly from the external module module_a to importing from the proxy module file, such as from .proxy_imports import ClassB, function_c.

[0063] Understandably, the aforementioned import redirection perturbation can transform the originally clear direct dependencies into indirect dependencies that require cross-file parsing, thereby increasing the difficulty for the code agent to understand the dependency path. This forces it to actively identify and parse intermediate proxy modules in order to trace back to the real dependency source, thus effectively evaluating the agent's navigation ability and deep semantic understanding ability when facing complex dependency structures.

[0064] Based on any of the above embodiments, the structural obfuscation type includes data stream redirection; The structural repository layer perturbation of the original instance based on the structural obfuscation type and the target file includes: Identify the constant configuration parameters of the target file based on its original code. A perturbation configuration file is created based on the constant configuration parameters; the perturbation configuration file contains the constant configuration parameters. Remove the definition of the constant configuration parameter from the target file and add an import statement to the target file to import the constant configuration parameter from the perturbation configuration file, thereby performing the repository layer perturbation on the original instance in terms of structure.

[0065] Among them, constant configuration parameters, also known as numerical constants, string constants, or configuration parameters, refer to fixed values ​​defined in the target file for configuring functional behavior.

[0066] Constant configuration parameters typically exist as variables at the module or class level.

[0067] The perturbation configuration file is a file created in the directory where the target file is located, used to centrally store the constant configuration parameters in the target file.

[0068] The definition of constant configuration parameters in the target file refers to the statements that originally exist in the target file and assign values ​​to constant configuration parameters, such as MAX_LENGTH=100.

[0069] It should be noted that by externalizing the hard-coded constants in the target file into the configuration file, cross-file data dependencies can be formed, thereby perturbing the repository layer of the original instance in terms of structure.

[0070] Specifically, first, scan the abstract syntax tree of the target file to identify all constant configuration parameters defined at the module or class level. Then, create a perturbation configuration file, such as config.py or constants.py, in the directory where the target file is located. Next, move all the identified constant configuration parameters, keeping their variable names and values ​​unchanged, into this configuration file. Finally, modify the target file, deleting the original constant definitions and adding an import statement at the beginning of the target file to import the constant configuration parameter from the perturbation configuration file.

[0071] Adding an import statement at the beginning of the target file can be done as `from .config import MAX_LENGTH, DEFAULT_TIMEOUT`.

[0072] Understandably, by using the aforementioned data flow redirection perturbation, the originally static data within a file can be transformed into dynamic cross-file dependencies. This ensures that the location of all code using the aforementioned constant configuration parameters in the target file remains unchanged, but the values ​​of the constant configuration parameters now come from an external perturbation configuration file. This forces the agent to recognize the configuration file and understand the data flow in order to fully understand the functionality of the target file, thereby effectively evaluating the agent's tracking ability and deep semantic understanding ability when facing cross-file data dependencies.

[0073] Based on any of the above embodiments, the structural obfuscation type includes source file redirection; The structural repository layer perturbation of the original instance based on the structural obfuscation type and the target file includes: Based on the second original import statement of the target file, determine the source file of the import module in the target file; In the directory where the source file is located, a subdirectory with the same name as the source file is created. The source file is moved to the internal implementation file in the subdirectory with the same name, and a package initialization file is created in the subdirectory with the same name. The package initialization file is used to identify the subdirectory with the same name as a package and export the public interface in the internal implementation file. Modify the second import statement of the target file according to the package initialization file to obtain the second perturbation statement for importing the import module from the subdirectory with the same name, and perform the repository layer perturbation on the original instance in terms of structure.

[0074] The second original import statement refers to the statement in the object file that was originally used to directly import the module from the source file. For example, the second original import statement could be `import module` or `from module importSomeClass`, etc.

[0075] In this context, the source file of the imported module in the target file refers to the original code file containing the actual implementation of the imported module that is referenced by the target file through the import statement.

[0076] Among them, a subdirectory with the same name refers to a subdirectory created in the directory where the source file is located, which has the same name as the source file.

[0077] The internal implementation file refers to the file containing the original code of the source file in the subdirectory with the same name.

[0078] Among them, the package initialization file is the file that identifies the subdirectory with the same name as a Python package.

[0079] Internal implementation files are typically named with an underscore prefix. For example, if the source file importing the module in the object file is `module.py`, you can create a subdirectory with the same name as `module / `, and within that `module / `, create an internal implementation file named `__init__.py`, as well as a package initialization file named `__init__.py`.

[0080] The second perturbation statement refers to the statement in the modified target file used to import modules from the transformed package structure.

[0081] It's worth noting that the actual implementation can be hidden by converting a single file into a package structure, thus perturbing the original instance's repository layer. For example, assuming the source file of an imported module in the target file is `module.py`, first create a subdirectory named `module / ` with the same name as the source file in the directory containing that file. Then, move all the contents of the original `module.py` to an internal implementation file in this subdirectory, for example, named `_impl.py`. Create an `__init__.py` file in the `module / ` subdirectory, whose content is `from._implimport*` or explicitly importing and re-exporting classes and functions that need to be exposed, thereby identifying the subdirectory as a package and re-exporting the public interfaces in the internal implementation file. Finally, delete the original `module.py` file and modify the import statements in the target file to ensure they point to the new package structure.

[0082] The original import statements `importmodule` or `frommoduleimportSomeClass` are still valid because Python will recognize `module / ` as a package and execute the import logic in `__init__.py`.

[0083] Understandably, by redirecting and perturbing the source files as described above, the original single source file can be transformed into a package structure containing package initialization files and internal implementation files. This increases the directory hierarchy and the number of files, forcing the code agent to not simply open the expected file to view the implementation, but to understand the package import mechanism, identify the re-export function of the package initialization file, and further find the actual code in the internal implementation file. This effectively evaluates the agent's navigation and deep understanding capabilities when faced with module encapsulation and package structure.

[0084] Based on any of the above embodiments, the structural obfuscation type includes obfuscated file perturbation; The structural repository layer perturbation of the original instance based on the structural obfuscation type and the target file includes: Based on the naming characteristics of the target file, at least two obfuscated files are created in the directory where the target file is located; the filenames of the obfuscated files are similar to the filenames of the target files.

[0085] The naming characteristics of a target file refer to features such as core keywords, naming patterns, or file type suffixes contained in its filename. For example, these characteristics may include main words, version number hints, and feature descriptions in the filename.

[0086] Obfuscated files are code files created in the same directory as the target file, with similar filenames but misleading content.

[0087] It should be noted that, while retaining the core keywords of the target file, the filename of the obfuscated file can be made similar to the filename of the target file by adding suffixes, prefixes, random strings, or modifying some characters. For example, if the target file is named fields.py, the obfuscated file could be named fields_old.py, fields_v2.py, etc.

[0088] In some embodiments, the number of obfuscated files ranges from 2 to 4 to ensure that the obfuscated files create sufficient interference in the file list without being overly redundant.

[0089] Based on any of the above embodiments, after creating at least two obfuscated files in the directory where the target file is located, the method further includes: Each of the obfuscated files is populated with misleading content; wherein the misleading content is used to interfere with the code agent's ability to identify the target file.

[0090] Misleading content may include incomplete class definitions, outdated version comments, partial import statements and global variable definitions but lacking actual functional code, and code snippets that are related to the function of the target file but are logically incomplete.

[0091] For example, an incomplete class definition may only contain class declarations and partial method signatures, lacking method implementations; an outdated version comment may be "This file is deprecated since v1.0" or "Use new_fields.pyinstead", etc.; a code snippet that is related to the functionality of the target file but is logically incomplete may only contain data validation logic and lack the main processing logic.

[0092] Understandably, misleading content may be grammatically correct, but semantically incomplete or misleading.

[0093] It should be noted that the recognizability of target files can be reduced by creating obfuscated files with similar names, thus perturbing the repository layer of the original instance. For example, the naming characteristics of the target files can be analyzed first to extract core keywords and naming patterns. Based on the target file names, naming strategies such as adding suffixes, adding action hints, or adding random strings can be used to determine the names of the obfuscated files. After generating the obfuscated files, incomplete class definitions, outdated version comments, and other misleading content can be added to each obfuscated file to obtain a complete obfuscated file.

[0094] Understandably, by obfuscating files as described above, the code agent is forced to open and analyze the file content instead of relying solely on filenames for quick matching. This allows the agent to determine which file is the real target that needs modification, thus effectively evaluating its ability to identify filename noise and misleading content and its ability to understand file content.

[0095] Based on any of the above embodiments, the method further includes: Based on the problem description, the original instance is subjected to a prompting layer perturbation; The original instance is perturbed at the code level based on the target file; The obtained perturbation instances include: The original instance is subjected to the prompt layer perturbation, the code layer perturbation, and / or the repository layer perturbation to obtain the perturbation instance.

[0096] Among them, the prompt layer perturbation refers to modifying the problem description in the original instance, changing its expression form without changing the core requirements.

[0097] For example, the problem description in the original instance can be input into the large language model to obtain the perturbed problem description output by the large language model, thereby realizing the perturbation of the prompt layer.

[0098] The large language model can be a pre-trained model or a general-purpose large language model. When the large language model is a general-purpose large language model, the problem description can be rewritten under the guidance of prompt words to obtain a problem description that adopts colloquial expression without changing the original core requirements, avoiding direct mention of file paths and / or the introduction of meaningless filler words.

[0099] Code-level perturbation refers to modifying the code in the original instance to change its syntactic structure without altering its semantics.

[0100] For example, the code of the target file can be modified by renaming local variable names, converting if-else conditional statements into ternary expressions, injecting branches that are never executed, and using the abstract semantic tree of the original code to verify whether the semantics are equivalent, thus achieving code-level perturbation.

[0101] It should be noted that perturbations at the hint layer, code layer, and repository layer can be combined and applied to the original instance to obtain a perturbated instance.

[0102] Understandably, the multi-layered perturbations applied by the combination of cue layer perturbations, code layer perturbations, and repository layer perturbations, working together, can comprehensively eliminate surface clues that the code agent can rely on.

[0103] Experiments have shown that the test results generated by multi-level combined perturbations are significantly better than those generated by single-level perturbations.

[0104] The combined approach has been verified to significantly reduce the accuracy of code agents and more accurately reflect their robustness in repository-level tasks.

[0105] Based on any of the above embodiments, obtaining a disturbance instance includes: Based on the target file, the original instance is subjected to structural perturbation of the repository layer to obtain candidate perturbation instances; If the candidate perturbation instance passes the functional integrity and consistency verification, then the candidate perturbation instance is determined to be a perturbation instance; and / or, Apply the hint layer perturbation, the code layer perturbation, and / or the repository layer perturbation to the original instance to obtain candidate perturbation instances; If the candidate perturbation instance passes the functional integrity and consistency verification, then the candidate perturbation instance is determined to be a perturbation instance.

[0106] Functional integrity and consistency verification refers to verifying whether the functions of the perturbed instance and the original instance are equivalent, and whether all test cases pass and produce consistent results before and after the perturbation. For example, functional integrity and consistency verification can be a PASS_TO_PASS functional verification mechanism.

[0107] Understandably, by performing functional integrity and consistency verification on candidate perturbation instances, and only determining the candidate perturbation instance as a perturbation instance if the verification passes, it can be ensured that each perturbation instance is functionally completely equivalent to the original instance, so that test cases can pass and the results are consistent before and after the perturbation. This avoids the problem that repository-level perturbations can easily disrupt project building, importing and running, and reduces the risk of misjudging errors introduced by perturbations as insufficient capabilities of the code intelligence agent.

[0108] Based on any of the above embodiments, the step of running the code agent on the robustness test dataset to perform robustness testing includes: The code agent is run on the robustness test dataset to obtain the first test result based on the preset metrics; The code agent is run on the original test dataset to obtain a second test result with preset metrics. Robustness tests were conducted by comparing the results of the first test with those of the second test.

[0109] Among them, the preset metrics refer to a set of metrics used to quantitatively evaluate the performance of code agents in repository-level tasks.

[0110] In some embodiments, the preset metrics include resolution rate, total number of steps, number of steps to locate the target file, and number of files opened.

[0111] For example, the resolution rate can be the proportion of patches generated by the code agent that pass test cases; the total number of steps can be the total number of interaction steps performed by the code agent in generating the patch; the target file location steps can be the number of execution steps from the start of the code agent's attempt to access the target file to the first successful access to the target file; and the number of files opened can be the number of files opened by the code agent during the patch generation process.

[0112] The first test result refers to the numerical values ​​or statistical data of various preset indicators obtained by running the code agent on the robustness test dataset after perturbation.

[0113] The second test result refers to the numerical values ​​or statistical data of various preset indicators obtained by running the code agent on the original test dataset.

[0114] It is understandable that the preset indicators of the first test result and the second test result correspond one-to-one.

[0115] It should be noted that the difference between the first and second test results can be analyzed by calculating the decrease and trend of various preset indicators of the first and second test results. This allows for a comprehensive evaluation of the performance degradation of the code agent when facing warehouse structure disturbances, thereby quantifying its robustness.

[0116] It is understandable that running the code agent on the robustness test dataset yields a first test result with preset metrics, and running the code agent on the original test dataset yields a second test result with preset metrics. Comparing the first test result with the second test result for robustness testing can provide multi-dimensional robustness feedback and quantify the impact of structural perturbations in the repository layer on the performance of the code agent, thus providing a foundation for targeted improvements to the code agent's ability to understand and manipulate the code repository.

[0117] The robustness testing apparatus for code-based intelligent agents provided by the present invention will be described below. The robustness testing apparatus for code-based intelligent agents described below can be referred to in correspondence with the robustness testing method for code-based intelligent agents described above.

[0118] Figure 2 This is a schematic diagram of the robustness testing device for the code-based intelligent agent provided by the present invention, as shown below. Figure 2 As shown, the device includes: The original instance acquisition module 210 is used to acquire original instances from the original test dataset; the original instances include code repositories, problem descriptions, test cases, and reference fix patches; The target file determination module 220 is used to determine the target file from the corresponding original instance based on the reference repair patch; The perturbation data acquisition module 230 is used to perform warehouse layer perturbation on the original instance based on the target file to obtain a perturbation instance, so as to obtain a robustness test dataset based on the perturbation instance; Robustness testing module 240 is used to run the code agent on the robustness testing dataset to perform robustness tests.

[0119] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3As shown, the electronic device may include a processor 310, a communications interface 320, a memory 330, and a communication bus 340, wherein the processor 310, communications interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 can invoke logical instructions in the memory 330 to execute a robustness testing method for the code agent. This method includes: obtaining an original instance from the original test dataset; the original instance includes a code repository, problem description, test cases, and a reference patch; determining a target file from the corresponding original instance based on the reference patch; perturbing the original instance at the repository layer based on the target file to obtain a perturbed instance, thereby obtaining a robustness test dataset based on the perturbed instance; and running the code agent on the robustness test dataset to perform robustness testing.

[0120] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0121] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the robustness testing method for the code agent provided by the above methods. The method includes: obtaining an original instance from an original test dataset; the original instance includes a code repository, a problem description, test cases, and a reference patch; determining a target file from the corresponding original instance based on the reference patch; perturbing the original instance at the repository layer based on the target file to obtain a perturbed instance, so as to obtain a robustness test dataset based on the perturbed instance; and running the code agent on the robustness test dataset to perform robustness testing.

[0122] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a robustness testing method for the code agent provided by the methods described above. The method includes: obtaining an original instance from an original test dataset; the original instance including a code repository, a problem description, test cases, and a reference patch; determining a target file from the corresponding original instance based on the reference patch; perturbing the original instance at the repository layer based on the target file to obtain a perturbed instance, thereby obtaining a robustness test dataset based on the perturbed instance; and running the code agent on the robustness test dataset to perform robustness testing.

[0123] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0124] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A robustness testing method for a code-based intelligent agent, characterized in that, include: Obtain the original instance from the original test dataset; The original instance includes the code repository, problem description, test cases, and reference fix patch; The target file is determined from the corresponding original instance based on the reference repair patch; Based on the target file, the original instance is perturbed at the repository layer to obtain a perturbed instance, and a robustness test dataset is obtained based on the perturbed instance; The code agent is run on the robustness test dataset to perform robustness testing.

2. The robustness testing method for code-based intelligent agents according to claim 1, characterized in that, The process of perturbing the original instance based on the target file includes: Determine the type of structural obfuscation; The original instance is subjected to the repository layer perturbation in terms of structure based on the structure obfuscation type and the target file.

3. The robustness testing method for code-based intelligent agents according to claim 2, characterized in that, The structure obfuscation type includes import redirection; The structural repository layer perturbation of the original instance based on the structural obfuscation type and the target file includes: The import items that the target file directly imports from an external module are determined based on the first original import statement of the target file; An intermediate proxy module is created based on the imported item; the intermediate proxy module is used to import the imported item from the external module and re-export the imported item. Modify the first original import statement of the target file to obtain a first perturbation statement for indirectly importing the import item from the intermediate proxy module, and perform the repository layer perturbation on the original instance in terms of structure.

4. The robustness testing method for code-based intelligent agents according to claim 2, characterized in that, The structural obfuscation type includes data stream redirection; The structural repository layer perturbation of the original instance based on the structural obfuscation type and the target file includes: Identify the constant configuration parameters of the target file based on its original code. A perturbation configuration file is created based on the constant configuration parameters; the perturbation configuration file contains the constant configuration parameters. Remove the definition of the constant configuration parameter from the target file and add an import statement to the target file to import the constant configuration parameter from the perturbation configuration file, thereby performing the repository layer perturbation on the original instance in terms of structure.

5. The robustness testing method for code-based intelligent agents according to claim 2, characterized in that, The obfuscation type includes source file redirection; The structural repository layer perturbation of the original instance based on the structural obfuscation type and the target file includes: Based on the second original import statement of the target file, determine the source file of the import module in the target file; In the directory where the source file is located, a subdirectory with the same name as the source file is created. The source file is moved to the internal implementation file in the subdirectory with the same name, and a package initialization file is created in the subdirectory with the same name. The package initialization file is used to identify the subdirectory with the same name as a package and export the public interface in the internal implementation file. Modify the second import statement of the target file according to the package initialization file to obtain the second perturbation statement for importing the import module from the subdirectory with the same name, and perform the repository layer perturbation on the original instance in terms of structure.

6. The robustness testing method for code-based intelligent agents according to claim 2, characterized in that, The structure obfuscation type includes obfuscated file perturbation; The structural repository layer perturbation of the original instance based on the structural obfuscation type and the target file includes: Based on the naming characteristics of the target file, at least two obfuscated files are created in the directory where the target file is located; the filenames of the obfuscated files are similar to the filenames of the target files.

7. The robustness testing method for code-based intelligent agents according to claim 6, characterized in that, After creating at least two obfuscated files in the directory where the target file is located, the method further includes: Each of the obfuscated files is populated with misleading content; wherein the misleading content is used to interfere with the code agent's ability to identify the target file.

8. The robustness testing method for code-based intelligent agents according to claims 1-7, characterized in that, The method further includes: Based on the problem description, the original instance is subjected to a prompting layer perturbation; The original instance is perturbed at the code level based on the target file; The obtained perturbation instances include: The original instance is subjected to the prompt layer perturbation, the code layer perturbation, and / or the repository layer perturbation to obtain the perturbation instance.

9. The robustness testing method for a code-based intelligent agent according to any one of claims 8, characterized in that, The obtained perturbation instances include: Based on the target file, the original instance is subjected to structural perturbation of the repository layer to obtain candidate perturbation instances; If the candidate perturbation instance passes the functional integrity and consistency verification, then the candidate perturbation instance is determined to be a perturbation instance; and / or, Apply the hint layer perturbation, the code layer perturbation, and / or the repository layer perturbation to the original instance to obtain candidate perturbation instances; If the candidate perturbation instance passes the functional integrity and consistency verification, then the candidate perturbation instance is determined to be a perturbation instance.

10. The robustness testing method for a code-based intelligent agent according to claim 1, characterized in that, The robustness test performed by running the code agent on the robustness test dataset includes: The code agent is run on the robustness test dataset to obtain the first test result based on the preset metrics; The code agent is run on the original test dataset to obtain a second test result with preset metrics. Robustness tests were conducted by comparing the results of the first test with those of the second test.

11. The robustness testing method for code-based intelligent agents according to claim 10, characterized in that, The preset metrics include resolution rate, total number of steps, number of steps to locate the target file, and number of files opened.

12. A robustness testing device for a code-based intelligent agent, characterized in that, include: The original instance acquisition module is used to obtain original instances from the original test dataset; The original instance includes the code repository, problem description, test cases, and reference fix patch; The target file determination module is used to determine the target file from the corresponding original instance based on the reference repair patch; The perturbation data acquisition module is used to perform warehouse layer perturbation on the original instance based on the target file to obtain perturbation instances, so as to obtain a robustness test dataset based on the perturbation instances; A robustness testing module is used to run the code agent on the robustness testing dataset to perform robustness tests.

Images