Evaluation methods and computing devices for vulnerability remediation capabilities of code-based intelligent agents.
By constructing proactive vulnerability remediation tasks, the vulnerability remediation capabilities of code agents within a specified scope are evaluated. This solves the evaluation bias caused by reliance on issue reports in existing technologies, achieves accurate evaluation even in the absence of issue reports, and improves the ability of code agents to reflect real-world scenarios.
Patent Information
- Application Number
- CN202611170505.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-03
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies rely on detailed issue reports when evaluating the vulnerability remediation capabilities of code agents. This leads to an overestimation of their capabilities in the absence of high-quality issue reports, and fails to accurately reflect their proactive vulnerability remediation capabilities in the absence of issue reports.
By acquiring code files, reference code patches, and test cases from the evaluation sample, an active vulnerability remediation task is constructed, requiring the code agent to autonomously discover and generate patches within a specified scope, and evaluating its vulnerability remediation capabilities under conditions where no issues are reported.
It enables objective evaluation of the proactive vulnerability remediation capabilities of code agents in the absence of problem reports, which is closer to real code review scenarios, overcomes the dependence on high-quality problem reports, and improves the accuracy and authenticity of the evaluation.
Smart Images

Figure CN123114949A_ABST
Abstract
Description
Technical Field
[0001] This specification relates to the field of artificial intelligence technology, and in particular to an evaluation method and computing device for the vulnerability repair capability of code intelligent agents. Background Technology
[0002] In recent years, code agents driven by large language models have been increasingly used in real-world software engineering tasks, such as fixing specific vulnerabilities in large code repositories based on issue reports. Specifically, the code agent can invoke tools such as code search, file editing, and test execution, interacting with the software repository in multiple rounds to complete the vulnerability remediation task. Traditional evaluation methods typically assume the existence of detailed issue reports to provide the code agent with the crucial information needed for vulnerability remediation, including but not limited to error messages, developer descriptions, or fault locations.
[0003] However, obtaining issue reports in the real world is difficult. On one hand, issue reports are inherently reactive, meaning they are recorded only after the error has occurred in production, resulting in extremely high acquisition costs. On the other hand, the creation of issue reports relies on expert knowledge and writing standards; not all issue reports can support code agents in resolving vulnerabilities. Therefore, if the evaluation method always assumes that the agent can obtain comprehensive issue reports, it will overestimate the code agent's ability to conduct pre-deployment code review and vulnerability remediation. Summary of the Invention
[0004] This specification describes one or more embodiments of an evaluation method and computing device for the vulnerability remediation capability of a code agent, enabling an objective evaluation of the vulnerability remediation capability of a code agent under conditions where no problem reports are received.
[0005] Based on the first aspect, an evaluation method for the vulnerability remediation capability of code-based intelligent agents is provided, including:
[0006] Obtain an evaluation sample, which includes a code file sample containing vulnerabilities, a reference code patch for the vulnerabilities recorded in the code file sample, and a reference test case;
[0007] The code file sample and task instructions are used as input to the code agent to be evaluated, and a predicted code patch is output. The task instructions instruct the code agent to be evaluated to perform vulnerability checks on the code in the code file sample and generate a code patch. The input to the code agent to be evaluated does not include information related to describing the vulnerability.
[0008] The reference test cases are used to test the code file sample after the predicted code patch is fixed, and the recorded vulnerability fixation evaluation results are obtained based on the test results;
[0009] Based on the recorded vulnerability remediation assessment results, the vulnerability remediation capability assessment result of the code agent to be evaluated is determined.
[0010] According to the second aspect, an evaluation device for the vulnerability remediation capability of code-based intelligent agents is provided, comprising:
[0011] The acquisition module is used to acquire evaluation samples, which include code file samples with vulnerabilities, reference code patches for vulnerabilities recorded in the code file samples, and reference test cases.
[0012] The patch generation module is used to take the code file sample and task instructions as input to the code agent to be evaluated, and output predicted code patches. The task instructions instruct the code agent to be evaluated to perform vulnerability checks on the code in the code file sample and generate code patches. The input of the code agent to be evaluated does not include relevant information describing the vulnerability.
[0013] The recorded vulnerability remediation assessment module is used to test the code file sample after the predicted code patch is repaired using the reference test cases, and obtain the recorded vulnerability remediation assessment result based on the test results;
[0014] The overall evaluation module is used to obtain the vulnerability remediation capability evaluation result of the code agent to be evaluated based on the recorded vulnerability remediation evaluation results.
[0015] According to a third aspect, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the method described in the first aspect.
[0016] According to a fourth aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method described in the first aspect.
[0017] According to a fifth aspect, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method of the first aspect.
[0018] In the embodiments of this specification, by replacing sample-specific issue reports with proactive vulnerability remediation tasks as input to the code agent, the focus of the evaluation paradigm is shifted from "remediating known vulnerabilities based on issue reports" to "proactively discovering and remediating vulnerabilities within a specified code scope." This is closer to the general code review scenario, that is, ensuring code quality in the absence of issue reports and overcoming the dependence of existing evaluation methods on high-quality issue reports. Attached Figure Description
[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0020] Figure 1 A schematic diagram of the structure of an evaluation system 100 according to one embodiment is shown;
[0021] Figure 2 A flowchart is shown for an evaluation method for vulnerability remediation capabilities of code-based intelligent agents according to one embodiment;
[0022] Figure 3 A schematic diagram illustrating the evaluation of recorded vulnerability fixes for a code agent to be evaluated, according to one embodiment;
[0023] Figure 4 A flowchart illustrating a potential vulnerability remediation assessment of a code agent to be evaluated, according to one embodiment, is shown.
[0024] Figure 5 A structural block diagram of an evaluation apparatus for the vulnerability remediation capability of a code agent according to one embodiment is shown. Detailed Implementation
[0025] To facilitate understanding of the solutions provided in the embodiments of this specification, a brief introduction will first be given to the technical terms that may be involved in the embodiments of this specification.
[0026] Code agents: Typically driven by large language models, they can invoke tools such as code search, file editing, and test execution, and interact with software repositories in multiple rounds to complete tasks such as vulnerability patching and feature implementation.
[0027] Proactive vulnerability remediation: In the absence of detailed problem reports, the code agent autonomously inspects the code within a specified scope of the repository, discovers vulnerabilities in the code, and generates corresponding code patches.
[0028] Recorded vulnerabilities: In the code repository, vulnerabilities discovered and recorded by developers are usually accompanied by code patches and test cases provided by the developers to verify their effectiveness. That is, if the corresponding test cases fail before the code patch is applied, and pass after the code patch is applied, then the vulnerability is considered effective.
[0029] Potential vulnerabilities: In the process of fixing vulnerabilities, the code agent may find and fix some undocumented vulnerabilities. These vulnerabilities often lack code patches and test cases provided by the developers to verify their effectiveness.
[0030] Large-scale language models-driven code agents have been an important area of focus in the industry community and have already achieved clear practical applications, such as various commercial paid services like Coding Plans.
[0031] In the development of code-based intelligent agents, capability assessment methods are an important foundation for guiding the design, optimization, and application of the next generation of code-based intelligent agents.
[0032] However, most existing industry evaluation methods still rely on detailed issue reports, placing code agents in a "known problem, passive fix" scenario for testing. This type of evaluation differs from the proactive vulnerability remediation requirements in real-world development, making it difficult to verify the ability of code agents to proactively discover and fix vulnerabilities in the absence of issue reports. The ability to proactively discover and fix vulnerabilities (which can be simply referred to as proactive vulnerability remediation capability) has practical application value, for example, preventing severe economic losses caused by code failures after deployment.
[0033] In view of this, the embodiments of this specification propose an evaluation method for the vulnerability repair capability of code agents. By replacing sample-specific issue reports with proactive vulnerability repair tasks as input to the code agent, the focus of the evaluation paradigm is shifted from "repairing known vulnerabilities based on issue reports" to "proactively discovering and repairing vulnerabilities within a specified code scope". This is closer to the general code review scenario, that is, ensuring code quality in the absence of issue reports and overcoming the dependence of existing evaluation methods on high-quality issue reports.
[0034] The solution provided in this specification will now be described with reference to the accompanying drawings.
[0035] Figure 1 An evaluation system 100 according to one embodiment is shown. Figure 1 As shown, the evaluation system 100 may include a sample construction end 101, an evaluation execution end 102, and a data storage end 103. The evaluation execution end 102 includes an environment construction agent 1021 and a judgment agent 1022.
[0036] The sample building end 101 can pull historical patch request data from the code hosting platform and generate evaluation samples. The evaluation execution end 102 is used to issue task instructions to the code agent to be evaluated and receive predicted code patches. The environment building agent 1021 is used to build an executable environment and test scripts for the code file samples. The evaluation agent 1022 is used to determine the reproducibility relationship between potential vulnerabilities and test cases. The data storage end 103 is used to save evaluation samples, predicted code patches, test results, and intermediate evaluation metrics.
[0037] The code agent to be evaluated can be an agent driven by a large language model (LLM) and capable of calling code search, file editing, and test execution tools. Figure 1 The connections in the data structure can represent data flow relationships. The sample construction end 101 can write evaluation samples to the data storage 103. The evaluation execution end 102 can read evaluation samples from the data storage 103. The code agent to be evaluated can access the code file samples in an isolated environment. The judging agent 1022 can read predicted code patches and candidate test cases from the data storage 103.
[0038] Figure 2 A flowchart illustrating an evaluation method for vulnerability remediation capabilities of code-based intelligent agents according to one embodiment is shown. It can be understood that this method can... Figure 1 The evaluation execution terminal 102 shown is used for execution. This evaluation execution terminal 102 can be any device, equipment, platform, or device cluster with computing and processing capabilities. For example... Figure 2 As shown, the method includes steps S210 to S240.
[0039] In step S210, an evaluation sample is obtained, which includes a code file sample with vulnerabilities, a reference code patch for the vulnerabilities recorded in the code file sample, and a reference test case.
[0040] There are several ways to obtain evaluation samples.
[0041] One method is to collect historical pull request data from multilingual open-source repositories from code hosting platforms (such as GitHub, Gitee, GitLab, etc.), and then process this collected historical pull request data to obtain evaluation samples.
[0042] Code file samples can originate from a historical commit state of a real code repository and may include at least some source code files, dependency description files, build scripts, and test directories from a repository snapshot before a fix. In other words, code file samples can be code files from a repository snapshot.
[0043] A logged vulnerability can be defined as a vulnerability that the developer has identified and provided a fix for on the code hosting platform.
[0044] Reference code patches can represent the code changes made before and after fixing a documented vulnerability. They can be represented as diff files, patch files, diff text, or a collection of structured edited snippets.
[0045] Reference test cases can be test scripts or test functions that verify the validity of the difference file. In one example, a reference test case fails when the reference code patch is not applied, but passes after the reference code patch is applied. Utilizing this failure-to-success testing characteristic, subsequent testing can determine whether the predicted code patch at least fixes the corresponding logged vulnerability.
[0046] For example, the sample build endpoint 101 can obtain multiple historical fix request data from the code hosting platform. Each historical fix request data may include a repository snapshot before the fix, reference code patches provided by the developer, and reference test cases.
[0047] A pre-fix repository snapshot represents the state of the code repository before merging the reference code patch. Repository snapshots can be frozen by commit number or tag so that the same code state can be reproduced during subsequent testing.
[0048] Optionally, the sample builder 101 can perform integrity verification on historical fix request data. Integrity verification may include checking whether the reference code patch can be applied to the repository snapshot before the fix, whether the reference test case has an executable entry point, and whether the repository snapshot is missing dependency declarations. For historical fix request data that cannot build an executable environment, the sample builder 101 can discard it or mark it as an unusable sample.
[0049] After obtaining historical remediation request data, the sample building end 101 can use the vulnerability classification system to filter multiple historical remediation request data.
[0050] For example, reference code patches and reference test cases can be input into multiple large language models. These models can use the same prompt words or different prompt words but output a unified label space. Multi-model annotation reduces misclassification caused by biases in the semantic understanding of patches by individual models. The sample construction end 101 can vote based on the consistency of multiple vulnerability category labels to obtain the final classification label. The voting process can be represented as: y = mode(Y), where y represents the final classification label, Y represents the set of vulnerability category labels output by M large language models, and mode represents the operation of selecting the label that appears most frequently in the set.
[0051] When there is a tie in votes, a tag can be selected according to a preset priority, or the historical repair request data can be marked as pending manual review.
[0052] In a different embodiment, the consistency ratio can also be used as a screening threshold. The consistency ratio can be expressed as C=N / M, where C represents the consistency ratio, N represents the number of models with the same final classification label y, and M represents the total number of large language models participating in the annotation.
[0053] When C is below the threshold, the historical repair request data can be discarded to reduce the perturbation of the evaluation results by samples with uncertain categories.
[0054] When the final classification label belongs to a preset vulnerability category in the vulnerability classification system, the sample creation terminal 101 can retain the corresponding historical remediation request data. When the final classification label does not belong to a preset vulnerability category, the sample creation terminal 101 can discard the corresponding historical remediation request data.
[0055] Preset vulnerability categories can include null pointer references, unreleased resources, boundary condition errors, concurrency races, and missing exception handling. The vulnerability classification system can be configured according to the assessment objectives. By filtering by category, the assessment sample can focus on remediation records with real vulnerability semantics, reducing the interference of feature optimizations, format adjustments, or document modifications on the assessment of vulnerability remediation capabilities.
[0056] The historical remediation request data obtained after filtering can form a candidate sample set. Each candidate sample in the candidate sample set can include a repository snapshot before the remediation, a reference code patch, and a reference test case.
[0057] In one embodiment, the code file sample may contain a single recorded vulnerability. The sample building end 101 can select candidate sample data that includes a single recorded vulnerability from the candidate sample set as evaluation samples.
[0058] A single logged vulnerability helps attribute test results for predictive code patches to a single remediation target. When a reference test case fails, the evaluation execution end can more easily determine whether the failure stems from an uncovered vulnerability or an environment build anomaly.
[0059] In another embodiment, the code file sample may contain multiple recorded vulnerabilities. The sample building end 101 can sort multiple candidate sample data within the same code repository according to the merge time. The sample building end 101 can select at least two adjacent candidate sample data from the sorted candidate sample data based on a time window. The sample building end 101 can aggregate the at least two candidate sample data to obtain an evaluation sample that includes multiple recorded vulnerabilities.
[0060] Thus, a time window can be used to limit the maximum time interval or the maximum number of commits between adjacent candidate samples. Aggregated samples can include multiple reference code patches, multiple reference test cases, and a specified merged scope. Multiple vulnerability samples increase task complexity and more closely resemble the scenario in real-world code reviews where a single commit contains multiple defect fixes. The time window controls the degree of coupling between different vulnerabilities. If the time span is too large, different vulnerabilities may involve refactoring or interface changes, increasing evaluation noise. If the time span is too small, the number of aggregable samples may be insufficient. The time window can be configured based on repository activity and file overlap.
[0061] Other methods of obtaining data include: obtaining it from the company's internal vulnerability database, obtaining it from security vulnerability remediation records, or obtaining it through manually constructed vulnerability injection data.
[0062] In the evaluation sample obtained in step S210, a traceable correspondence can be maintained between the code file sample, the reference code patch, and the reference test cases. The reference code patch can serve as a reference for vulnerability location assessment and remediation assessment. The reference test cases can serve as the basis for verifying the effectiveness of the remediation.
[0063] Since the input to the code agent to be evaluated does not include information describing the vulnerability, the reference code patches and reference test cases in the evaluation sample can be used as offline references during the task distribution phase, rather than being provided as input to the code agent to be evaluated. This allows the evaluation process to more closely resemble a proactive code review scenario without issue reports.
[0064] Optionally, the information used to describe the vulnerability may include at least one of the following: error message, vulnerability location, and expected behavior. For example, the information used to describe the vulnerability may also be a problem report regarding a sample code file.
[0065] After obtaining the evaluation sample in step S210, the evaluation execution terminal 102 can construct an active vulnerability remediation task based on the evaluation sample. The core of the task construction process is to combine the repository snapshot with the task instructions as the input to the code agent to be evaluated, without including reference code patches or reference test cases in the input.
[0066] In step S220, the code file sample and task instructions are used as inputs to the code agent to be evaluated, and the output is a predicted code patch. The task instructions instruct the code agent to be evaluated to perform vulnerability checks on at least a specified range of code in the code file sample and generate code patches. The input of the code agent to be evaluated does not include information related to describing the vulnerability.
[0067] Task instructions can include a task template and specified scope information. The task template can use a uniform text structure, requiring the agent of the code to be evaluated to autonomously check for vulnerabilities and generate code patches within the specified file scope.
[0068] Optionally, the task template may include a description of the task objective, output format constraints, and operation permission instructions.
[0069] The task objective description can instruct the code agent to be evaluated to perform vulnerability scanning and remediation on source code within a specified scope.
[0070] Output format constraints can specify that predictive code patches should use a uniform difference file format.
[0071] Operation permission descriptions can limit the file paths that the code agent to be evaluated can access and the types of operations that can be performed in the repository snapshot.
[0072] The specified scope can be determined based on the scope of modifications extracted from the reference code patch. Optionally, the specified scope can be a set of code files, a directory, a function signature, or an expanded code region, and can omit precise line numbers, vulnerability types, and remediation recommendations.
[0073] The evaluation execution end 102 can parse the modification scope from the reference code patch and use the files covered by the modification scope and their context files as the specified scope.
[0074] In this way, the specified scope covers the code region where the recorded vulnerability is located, but does not include information such as vulnerability type, error message, or remediation direction. The agent evaluating the code can only know which files need to be checked, but cannot know what kind of vulnerabilities exist in those files.
[0075] In one embodiment, the code file sample may contain a single documented vulnerability, corresponding to a simple task. The scope specified in the simple task may cover only the set of files involved in a single reference code patch.
[0076] In one different embodiment, the code file sample may contain multiple documented vulnerabilities, corresponding to a complex task. The specified scope in the complex task can cover the union of the sets of files involved in multiple reference code patches. Complex tasks require the agent of the code to be evaluated to discover and fix multiple vulnerabilities in a single execution, which places higher demands on the agent's context management and multi-turn interaction capabilities.
[0077] For example, the code agent to be evaluated can receive repository snapshots and task instructions via an input terminal. When performing proactive vulnerability remediation tasks in an isolated environment, the code agent can invoke a set of tools from the tool invocation list. Tool invocation can include code search tools, file reading tools, code editing tools, and test run tools. Code search tools can be used to retrieve code snippets related to a specified range in the repository snapshot. File reading tools can be used to read the complete contents of source code files within a specified range. Code editing tools can be used to perform insertion, deletion, or replacement operations on source code files. Test run tools can be used to execute existing test scripts in an isolated environment to verify whether intermediate modifications introduce regression issues.
[0078] The code agent to be evaluated can interact with multiple tools in multiple rounds based on the reasoning capabilities of a large language model. In each round of interaction, the code agent can decide its next action based on the currently observed code content. For example, the code agent can first locate a suspicious function using a code search tool, then obtain the complete context of the function using a file reading tool, then execute modifications using a code editing tool, and finally verify whether the modifications break existing functionality using a test run tool. If the test run tool returns a failure result, the code agent can roll back the modifications or adjust the repair strategy.
[0079] After the agent under evaluation completes all interactions, it can output a predicted code patch. The predicted code patch represents the set of all modifications performed by the agent on the code file sample. The predicted code patch can use a uniform difference file format, recording the original line number, new line number, and modification content for each modified file. The predicted code patch will be used in subsequent repair and localization evaluations; therefore, its format needs to be consistent with the reference code patch for structured comparison.
[0080] During the execution of step S220, the input to the code agent to be evaluated does not include information describing the vulnerability, which constitutes a core constraint of the evaluation paradigm. Traditional evaluation methods use detailed problem reports as input, placing the code agent to be evaluated in a scenario of known problems and passive fixes. However, in the embodiments of this specification, the code agent to be evaluated needs to discover defects by autonomously understanding the code logic, even in the absence of key information such as vulnerability type, error stack, and fault location. In other words, the embodiments of this specification do not require problem reports as input to the code agent to be evaluated, allowing the code agent to proactively discover vulnerabilities and generate predictive patches even without problem reports, thus better demonstrating the proactive vulnerability remediation capability of the code agent.
[0081] This approach more closely resembles real-world code review scenarios, where developers proactively inspect code before deployment, rather than passively fixing issues after they are exposed. By excluding vulnerability alerts, the evaluation results more objectively reflect the true capabilities of the evaluated code agent in proactively fixing vulnerabilities.
[0082] In one example, the evaluation execution end can set the maximum number of interaction rounds and the maximum number of tool calls in the task instructions. The maximum number of interaction rounds can constrain the inference depth of the code agent being evaluated, preventing resource exhaustion due to infinite loops. The maximum number of tool calls can constrain the scope of access to repository snapshots by the code agent being evaluated, preventing excessive time costs due to excessive searching. These two constraint parameters can be configured according to the computational budget of the evaluation scenario.
[0083] In some other embodiments, the task instructions may not be limited to a specified scope, and by default the code agent to be evaluated performs vulnerability checks on all code in the code file sample and generates code patches.
[0084] After obtaining the prediction code patch in step S220, the method can proceed to step S230.
[0085] In step S230, the code file sample after the predicted code patch is fixed is tested using reference test cases, and the recorded vulnerability fixation evaluation result is obtained based on the test results.
[0086] The evaluation execution end can apply the predicted code patch to the code file sample before the fix, and obtain the code file sample after the predicted code patch is fixed.
[0087] Next, the evaluation execution end can call the environment building agent to build an executable environment and generate test scripts for the code file sample.
[0088] For example, the environment building agent can generate a dependency resolution tree based on the dependency declaration file in the code file sample. This dependency declaration file can include package management manifests, version lock files, build toolchain declarations, and test framework declarations.
[0089] The environment building agent can select a compatible version based on version constraints. If an unsolvable conflict exists, the agent can attempt to remove optional dependencies, downgrade the testing framework, or switch the language runtime version. If the build still fails, the sample can be marked as an unavailable environment. In this way, dependency conflicts can be distinguished into fixable and unbuildable conflicts, reducing the interference of dependency conflicts on patch repair evaluation results.
[0090] The environment building agent can also create isolated execution environments based on container runtimes. Container runtimes can include Docker container runtimes or equivalent container runtimes.
[0091] Optionally, the isolated execution environment can include a read-only image layer, a writable temporary layer, an independent network namespace, an independent mount namespace, and resource quotas. Code file samples can be mounted as read-only volumes. Predictive code patches can be written to the temporary layer or the patch workspace. After the test script execution is complete, the isolated execution environment can be destroyed to reduce contamination between samples. The isolated execution environment can be restricted from accessing the external network or only allowed to access the offline dependency cache. Resource quotas can limit processor time, memory usage, and disk writes. In this way, different evaluation samples can be executed in mutually isolated environments, reducing the impact of residual environment state on subsequent samples.
[0092] The environment building agent can categorize build failures into types such as missing dependencies, compilation errors, insufficient permissions, or insufficient resources. These failure types can serve as a basis for distinguishing between patch fix failures and environment unavailability. When a failure type indicates environment unavailability, the corresponding evaluation sample can be excluded from the patch fix failure count. When a failure type indicates an error in the test script itself, the environment building agent can either regenerate the test script or mark the corresponding sample as a low-availability sample. In this way, the recorded vulnerability fix evaluation results can better reflect the predicted code patch's ability to fix code defects, rather than environment build anomalies or test script anomalies.
[0093] The executable environment can include dependent libraries, a compiler, a runtime environment, and a sandbox. Test scripts can encapsulate the invocation order, environment variables, and timeout configurations of reference test cases. Multiple reference test cases can be included. The evaluation execution end can run the test scripts within the executable environment, executing multiple reference test cases on a sample code file after the predicted code patch fixes, and obtaining the test results.
[0094] Test results can include pass, failure, exception, timeout, or crash status. The evaluation execution end can determine the recorded vulnerability remediation evaluation results based on the pass / fail status of multiple reference test cases.
[0095] In one example, if all reference test cases pass, the recorded vulnerability remediation assessment result indicates that the recorded vulnerability remediation was successful. If any of the reference test cases fail, the recorded vulnerability remediation assessment result indicates that the recorded vulnerability remediation failed.
[0096] In another implementation, the recorded vulnerability remediation assessment results may include the remediation pass rate. The remediation pass rate can be expressed as: G = N1 / N2, where G represents the remediation pass rate, N1 represents the number of reference test cases that passed, and N2 represents the total number of reference test cases.
[0097] The repair pass rate can distinguish between the intermediate state between partial test success and all test failures, and reduce the impact of fluctuations in a single test on the evaluation conclusion.
[0098] Optionally, for test errors caused by environment build failures, the evaluation execution end can mark the corresponding sample as an unavailable environment and exclude it from the patch fix failure count.
[0099] For unstable tests, the evaluation execution end can repeat the test a preset number of times and use the majority result as the judgment. In this way, the recorded vulnerability remediation evaluation results can better reflect the predicted code patch's ability to fix code defects, rather than environmental noise or occasional failures.
[0100] Figure 3 A schematic diagram illustrating the evaluation of recorded vulnerability fixes for a code agent to be evaluated, according to one embodiment, is shown. Figure 3 As shown, the code file samples and task instructions in the evaluation sample are used as inputs to the code agent to be evaluated. The code agent to be evaluated performs vulnerability checks on the code in the code file samples within at least a specified range and generates predictive code patches.
[0101] Subsequently, the predicted code patch was used to fix the vulnerabilities in the code file sample, resulting in a patched code file sample. Then, the patched code file sample was tested using reference test cases, and the test results were obtained.
[0102] Finally, the test results are used to obtain the recorded vulnerability remediation assessment results. In one example, the recorded vulnerability remediation assessment results can include successful remediation (i.e., all reference test cases passed the test, indicating that the predicted code patch will fix all recorded vulnerabilities, and the recorded vulnerability remediation is successful) or failed remediation (i.e., not all reference test cases passed the test, indicating that the predicted code patch did not fix all recorded vulnerabilities, and the recorded vulnerability remediation is failed).
[0103] In another example, the recorded vulnerability remediation assessment results can include the remediation rate of recorded vulnerabilities. This remediation rate can be expressed as the pass rate of reference test cases, which represents the proportion of reference test cases that passed out of all test cases. In this way, the recorded vulnerability remediation assessment results can provide a more detailed understanding of the remediation status of recorded vulnerabilities.
[0104] In step S240, based on the recorded vulnerability remediation assessment results, the vulnerability remediation capability assessment result of the code agent to be evaluated is determined.
[0105] The assessment execution end can use the recorded vulnerability remediation assessment results as a basic indicator.
[0106] Recorded vulnerability remediation assessment results can indicate whether the predicted code patch enables the reference test case to pass. If the recorded vulnerability remediation assessment results indicate successful remediation, the vulnerability remediation capability assessment results can indicate that the code agent to be evaluated possesses the capability to remediate the recorded vulnerability. If the recorded vulnerability remediation assessment results indicate failed remediation, the vulnerability remediation capability assessment results can indicate that the code agent to be evaluated failed the corresponding assessment sample.
[0107] When the evaluation sample includes multiple samples, the evaluation execution end can count the number of successfully repaired samples to the total number of samples, and determine the vulnerability repair capability evaluation result based on the ratio.
[0108] In one example, before determining the overall capability assessment result, the assessment execution end can also analyze the predicted code patch and the reference code patch, and obtain the vulnerability location assessment result based on the analysis results.
[0109] For example, the evaluation execution end can convert the predicted code patch and the reference code patch into sets of editable fragments, respectively, to obtain a first set of editable fragments corresponding to the predicted code patch and a second set of editable fragments corresponding to the reference code patch. Each editable fragment can include a file path, a starting line number, an ending line number, and a modification type. The starting and ending line numbers determine the range of line numbers, which indicates the range of lines of modified code.
[0110] The evaluation execution end can obtain the line number range of each edit segment in the first edit segment set and the line number range of each edit segment in the second edit segment set.
[0111] For any edit segment in the second set of edit segments, if there is an edit segment in the first set of edit segments that overlaps with the line number range of that edit segment, the evaluation execution end can determine that the edit segment is an overlapping edit segment.
[0112] One range of line numbers can be represented as [a, b], and another range of line numbers can be represented as [c, d]. a and b represent the start and end line numbers of one range, respectively. c and d represent the start and end line numbers of the other range, respectively. If min(b, d) ≥ max(a, c), then the two ranges of line numbers can be considered to overlap.
[0113] In another example, the condition for two line numbers to overlap can also be that the ranges of the two line numbers are completely identical.
[0114] By judging the overlap of line number ranges, the evaluation execution end can identify whether the predicted patch covers the code area modified by the reference patch.
[0115] The evaluation execution end can analyze the overlap of segments based on the first and second edited segment sets to obtain the segment overlap rate.
[0116] Optionally, the evaluation execution end can determine the segment overlap rate based on the ratio of the number of overlapping edit segments in the second edit segment set to the total number of edit segments in the second edit segment set. The segment overlap rate can be expressed as: O=N o / N t Where O represents the fragment overlap rate, and N o N represents the number of overlapping edit segments in the second set of edit segments. t This indicates the total number of edited segments in the second edited segment set. The segment overlap rate indicates the degree of overlap between the edited segments in the first edited segment set and the edited segments in the second edited segment set.
[0117] The assessment execution end can determine the vulnerability location assessment result based on the fragment overlap rate.
[0118] Vulnerability location assessment results can include fragment overlap rate.
[0119] In one different embodiment, the vulnerability location assessment result may include a location hit label. The assessment execution end can compare the fragment overlap rate with a location threshold. If the fragment overlap rate is higher than the location threshold, the location assessment can indicate that the reference modification range has been hit. If the fragment overlap rate is lower than the location threshold, the location assessment can indicate that the reference modification range has not been hit. In cases where a predicted code patch causes a reference test case to pass but the fragment overlap rate is low, the assessment execution end can determine that the predicted patch fixes the recorded vulnerability through an alternative path, or that the predicted patch may trigger a test pass but has not located the original vulnerability. This distinction helps subsequent capability assessment results differentiate between remediation capability and location capability.
[0120] The evaluation execution end can combine the recorded vulnerability remediation evaluation results and vulnerability location evaluation results. If the remediation is successful and the fragment overlap rate is higher than the location threshold, the vulnerability remediation capability evaluation result can indicate successful location and remediation. If the remediation is successful but the fragment overlap rate is lower than the location threshold, the vulnerability remediation capability evaluation result can indicate that the test passed but the location was off-target. In the case of remediation failure, even if the fragment overlap rate is high, the vulnerability remediation capability evaluation result can still indicate that the remediation was unsuccessful.
[0121] Thus, the evaluation results of the vulnerability remediation capability of the code agent to be evaluated include the recorded vulnerability remediation evaluation results and the vulnerability location evaluation results.
[0122] Recorded vulnerability remediation assessment results can verify the effectiveness of the code agent generating patches without vulnerability hints. Fragment overlap rate can verify the consistency between the predicted patch and the reference patch in terms of modification locations, and reduce the assessment bias caused by relying solely on test pass results.
[0123] In another example, considering that predicted code patches may not only fix recorded vulnerabilities, but also potential vulnerabilities in the code file sample that have not been recorded, the process proceeds to the potential vulnerability remediation assessment stage after obtaining the recorded vulnerability remediation assessment results in step S230.
[0124] Because potential vulnerabilities lack reference code patches and reference test cases provided by developers, the evaluation execution end needs to verify the effectiveness of potential vulnerability fixes through additional mechanisms.
[0125] Figure 4 A flowchart illustrating a potential vulnerability remediation assessment of a code agent to be evaluated, according to one embodiment, is shown. Figure 4 As shown, the potential vulnerability remediation assessment of the code agent to be evaluated includes steps S231 and S232.
[0126] In step S231, several candidate test cases are generated.
[0127] The evaluation execution end can generate several candidate test cases.
[0128] Candidate test cases differ from reference test cases, and candidate test cases must meet preset conditions. The preset conditions are: executing candidate test cases using a sample code file before the predicted code patch fix results in a failure; executing candidate test cases using a sample code file after the predicted code patch fix results in a success.
[0129] Optionally, candidate test cases can be generated by the code agent to be evaluated. The evaluation execution end can take a code file sample, a predicted code patch, and a generation task instruction as input to the code agent to be evaluated, and output multiple candidate test cases. The generation task instruction can be used to instruct the code agent to be evaluated to generate multiple candidate test cases based on the code file sample and the predicted code patch.
[0130] The code agent to be evaluated can analyze and predict the code changes involved in the code patch, infer the type of defect that the change may fix, and construct test inputs that can trigger the corresponding defect.
[0131] In one example, the evaluation execution end can select target candidate test cases that meet preset conditions and are different from the reference test cases from multiple candidate test cases, thus obtaining several candidate test cases.
[0132] The verification process under preset conditions may be executed twice.
[0133] First execution: The evaluation execution unit runs candidate test cases on a code file sample without the predicted code patch applied. If a candidate test case passes, it does not meet the failure-to-success condition and can be discarded by the evaluation execution unit. If a candidate test case fails, the second execution begins.
[0134] Second execution: The evaluation execution unit runs the candidate test case on a sample code file after applying the predicted code patch. If the candidate test case passes, it meets the preset conditions, and the evaluation execution unit can retain it as a target candidate test case. If the second execution still fails, the evaluation execution unit can discard the candidate test case.
[0135] Through two rounds of screening, each test case in the candidate test case set can distinguish the behavioral differences between the code before and after the fix.
[0136] In one example, the differences between a candidate test case and a reference test case can be compared through test case identifiers, test function names, or test input hash values. If a candidate test case is identical to a reference test case, the evaluation execution end can exclude it to avoid duplicate calculations of remediation contributions for already recorded vulnerabilities.
[0137] In step S232, based on several candidate test cases, the potential vulnerability remediation assessment results are obtained.
[0138] The evaluation execution end can obtain the evaluation results of potential vulnerability remediation based on several candidate test cases.
[0139] For example, the evaluation execution end can input the predicted code patch and several candidate test cases into the judging agent, and output an association matrix between potential vulnerabilities and candidate test cases. The association matrix represents the mapping relationship between which candidate test cases reproduce each potential vulnerability. The judging agent can analyze the code change segments involved in the predicted code patch, identify the potential vulnerabilities that each change segment may fix, and determine whether each candidate test case can trigger the corresponding potential vulnerability. The association matrix can be represented as: M(i,j) = 1 or 0, where M(i,j) represents the element in the i-th row and j-th column of the association matrix. i represents the potential vulnerability index, and j represents the candidate test case index. A value of 1 for M(i,j) indicates that the j-th candidate test case can reproduce the i-th potential vulnerability, and a value of 0 for M(i,j) indicates that the j-th candidate test case cannot reproduce the i-th potential vulnerability.
[0140] The correlation matrix can reveal the correspondence between different code changes in a predicted code patch and different test cases, and help identify cases where some fixes are effective.
[0141] In one example, the judging agent can determine the association matrix based on dynamic execution paths and semantic analysis. The judging agent can then invoke a dynamic instrumenter to inject probes during the execution of candidate test cases.
[0142] The probe can record function entry points, function exit points, branch jumps, code line executions, exception throwing, and assertion failure locations. The judging agent can form a set of covered code lines Pⱼ corresponding to the j-th candidate test case based on the recorded results. The judging agent can extract a set of patch-modified code lines Mᵢ corresponding to the i-th potential vulnerability from the predicted code patch. For potential vulnerabilities where original code lines are deleted or replaced, Pⱼ can be derived from the execution coverage of the pre-fix code file sample. For potential vulnerabilities where code lines are added or replaced, Pⱼ can be derived from the execution coverage of the fixed code file sample.
[0143] The judging agent can calculate the coverage of candidate test cases to the code lines that modify potential vulnerabilities. Coverage can be expressed as: Cᵢⱼ = |Pⱼ∩Mᵢ| / |Mᵢ|, where Cᵢⱼ represents the coverage of the j-th candidate test case to the code lines that modify the i-th potential vulnerability. Pⱼ represents the set of code lines covered when the j-th candidate test case is executed. Mᵢ represents the set of code lines that modify the patch for the i-th potential vulnerability.
[0144] Next, the judging agent can determine the elements of the association matrix based on the coverage and the reproducibility flag of the candidate test cases. The elements of the association matrix can be represented as: M(i,j)=I(Cᵢⱼ≥τ∧Rⱼ=1), where M(i,j) represents the element between the i-th potential vulnerability and the j-th candidate test case in the association matrix; I represents the indicator function, which takes the value 1 when the condition within the parentheses is true, and 0 when the condition within the parentheses is false; τ represents the coverage threshold, which can be configured to a small value greater than zero, for example, τ can be 1 / |Mᵢ|, to indicate that the candidate test case covers at least one line in the set of patched code lines; Rⱼ represents the reproducibility flag of the j-th candidate test case. Rⱼ can take the value 1 when the j-th candidate test case fails before the fix and passes after the fix; otherwise, Rⱼ can take the value 0. ∧ represents the logical AND operation. When Cᵢⱼ≥τ and Rⱼ=1, M(i,j) can be 1, and it can be determined that the j-th candidate test case reproduces the i-th potential vulnerability. When Cᵢⱼ<τ or Rⱼ=0, M(i,j) can be 0. When Mᵢ is empty, Cᵢⱼ does not need to participate in the calculation, and the corresponding potential vulnerability can be handled by semantic analysis or manual review.
[0145] In some other examples, the judging agent can also correct dynamic decisions based on semantic analysis. Semantic analysis can be performed collaboratively by a large language model and a static analyzer.
[0146] Semantic analysis can include abstract syntax tree parsing, call graph construction, modified function identification, modified conditional branch identification, and exception type identification for predicted code patches. The judging agent can extract the input construct, assertion object, captured exception type, and failure stack of candidate test cases. If the set of covered lines of code does not match Mᵢ, but the failure stack falls within the function call chain corresponding to Mᵢ, and the exception type matches the defect type fixed by the predicted code patch, the judging agent can mark the candidate test case as needing review, instead of directly setting Aᵢⱼ to 1. If the set of covered lines of code matches Mᵢ, but the assertion object of the candidate test case is unrelated to Mᵢ, the judging agent can reduce the association confidence.
[0147] The association matrix can retain only elements that satisfy the dynamic coverage condition and the reproducibility condition as 1. The dynamic coverage condition can include Cᵢⱼ≥τ. Semantic analysis can serve as an auxiliary verification and provide clues for review when the dynamic coverage condition is not met. In this way, the association matrix can form the main decision-making basis based on the dynamic execution path and reduce the uncertainty caused by relying solely on semantic judgment.
[0148] The evaluation execution can determine the number of successfully patched potential vulnerabilities based on an association matrix. A successfully patched potential vulnerability must be reproduced by at least one candidate test case.
[0149] For the i-th row of the correlation matrix, if at least one element in the row has a value of 1, then the i-th potential vulnerability can be determined to have been successfully fixed. If all elements in the row have a value of 0, then the i-th potential vulnerability can be determined to have not been effectively reproduced, meaning that the fix for this potential vulnerability lacks test verification support.
[0150] The evaluation execution end can obtain the potential vulnerability remediation evaluation result based on the number K of successfully remediated potential vulnerabilities. The potential vulnerability remediation evaluation result can include the number of successfully remediated potential vulnerabilities, the total number of potential vulnerabilities, and the potential vulnerability remediation success rate.
[0151] In one different embodiment, the potential vulnerability remediation assessment results may also include a list of candidate test cases for each potential vulnerability, so as to facilitate subsequent manual review or automatic regression verification.
[0152] The assessment execution end can determine the potential vulnerability remediation assessment result based on the number of successfully remediated potential vulnerabilities. When recorded vulnerability remediation assessment results, vulnerability location assessment results, and potential vulnerability remediation assessment results are all included in the assessment, the vulnerability remediation capability assessment result can include the recorded vulnerability remediation score, vulnerability location score, and potential vulnerability remediation score. The assessment execution end can weight and summarize these scores to obtain a comprehensive capability score. The comprehensive capability score can serve as a basis for distinguishing between targeted remediation capabilities and proactive vulnerability discovery and remediation capabilities.
[0153] This evaluation process combines recorded vulnerability remediation assessment results, vulnerability location assessment results, and potential vulnerability remediation assessment results to form a multi-dimensional capability assessment conclusion. Recorded vulnerability remediation assessment results reflect the ability of the evaluated code agent to remediate known vulnerabilities. Vulnerability location assessment results reflect the accuracy with which the evaluated code agent identifies vulnerability locations. Potential vulnerability remediation assessment results reflect the additional ability of the evaluated code agent to discover and remediate unrecorded vulnerabilities while remediating known vulnerabilities. The assessment results from these three dimensions can be weighted and aggregated or displayed hierarchically to avoid a single indicator masking the differences in the evaluated code agent's capabilities across different dimensions.
[0154] The candidate test case generation and potential vulnerability assessment processes can expose over-fixing or ineffective modifications in predicted code patches. If a predicted code patch introduces a large number of code changes but fails to pass any candidate test cases, most elements in the correlation matrix output by the judging agent may be 0, and the potential vulnerability repair assessment results can indicate that the additional repairs in the predicted code patch have low credibility. If a predicted code patch modifies only a small amount of code but can pass multiple candidate test cases to verify different potential vulnerabilities, the correlation matrix can exhibit a high density of non-zero elements, and the potential vulnerability repair assessment results can indicate that the predicted code patch has high repair efficiency and generalization ability.
[0155] In summary, the technical concept of the various embodiments in this specification allows for an alternative to evaluation methods that rely on issue reports by constructing proactive vulnerability remediation tasks and using code file samples and task instructions as inputs to the code agent to be evaluated. Task instructions can instruct the code agent to perform vulnerability checks and generate code patches within a specified scope. Since the input to the code agent does not include information describing the vulnerability, proactive discovery and remediation capabilities can be evaluated even in the absence of issue reports. By filtering samples containing reference code patches and reference test cases from historical remediation request data and constructing a single recorded vulnerability task or multiple recorded vulnerability aggregation tasks, both single vulnerability attribution and complex multi-vulnerability scenarios can be addressed. Testing the code file sample after the predicted code patch has been fixed using reference test cases yields the recorded vulnerability remediation evaluation results, thus verifying whether the recorded vulnerability has been fixed. Converting the predicted code patch and reference code patch into a set of editable fragments and determining the fragment overlap rate based on line number range overlap yields the vulnerability location evaluation results, allowing for the differentiation between cases where the test passed but the location was offset. By generating candidate test cases that meet the condition of failing before patching and passing after patching, and using an evaluation agent to establish a correlation matrix between potential vulnerabilities and candidate test cases, a potential vulnerability patching assessment result can be obtained. This allows for the identification of potential vulnerability patches lacking developer patches and test cases, as well as the identification of invalid or partial vulnerability patches. The overall vulnerability patching capability assessment result is based on the recorded vulnerability patching assessment results and can be combined with vulnerability location assessment results and potential vulnerability patching assessment results. This provides a basis for the design and optimization of code intelligence agents and the verification of pre-deployment code review capabilities.
[0156] According to another embodiment, an evaluation apparatus for the vulnerability remediation capability of a code agent is provided. Figure 5 This diagram illustrates the structure of an evaluation apparatus for remediating vulnerabilities in code-based intelligent agents according to one embodiment. This apparatus can be deployed in any device, platform, or device cluster with data storage, computing, and processing capabilities. Figure 5 As shown, the device 500 includes:
[0157] The acquisition module 501 is used to acquire an evaluation sample, which includes a code file sample with vulnerabilities, a reference code patch for the vulnerabilities recorded in the code file sample, and a reference test case.
[0158] The patch generation module 502 is used to take the code file sample and task instructions as input to the code agent to be evaluated, and output a predicted code patch. The task instructions instruct the code agent to be evaluated to perform vulnerability checks on the code in the code file sample and generate a code patch. The input of the code agent to be evaluated does not include relevant information describing the vulnerability.
[0159] The recorded vulnerability remediation assessment module 503 is used to test the code file sample after the predicted code patch is repaired using the reference test cases, and obtain the recorded vulnerability remediation assessment result based on the test results.
[0160] The overall evaluation module 504 is used to obtain the vulnerability remediation capability evaluation result of the code agent to be evaluated based on the recorded vulnerability remediation evaluation results.
[0161] In one embodiment, the evaluation device 500 for the vulnerability repair capability of the code agent further includes a vulnerability location evaluation module 505, which is used to analyze the predicted code patch and the reference code patch, and obtain the vulnerability location evaluation result based on the analysis result.
[0162] The determination of the vulnerability remediation capability assessment result of the code agent to be evaluated is also related to the vulnerability location assessment result.
[0163] According to one embodiment, the vulnerability location and assessment module 505 is specifically used for:
[0164] The predicted code patch and the reference code patch are respectively converted into an editable fragment set to obtain a first editable fragment set corresponding to the predicted code patch and a second editable fragment set corresponding to the reference code patch;
[0165] An analysis is performed based on the overlap of segments in the first set of edited segments and the second set of edited segments to obtain the segment overlap rate, which indicates the overlap between edited segments in the first set of edited segments and edited segments in the second set of edited segments;
[0166] Based on the fragment overlap rate, the vulnerability location assessment result is determined.
[0167] According to one embodiment, the edit fragment includes a range of line numbers indicating the range of lines of code to be modified;
[0168] When performing segment overlap analysis based on the first and second edited segment sets to obtain the segment overlap rate, the vulnerability location and assessment module 505 is used to:
[0169] Obtain the line number range of each edit segment in the first edit segment set and the line number range of each edit segment in the second edit segment set;
[0170] For any edit segment in the second set of edit segments, if there is an edit segment in the first set of edit segments that overlaps with the line number range of the edit segment, then the edit segment is determined to be an overlapping edit segment.
[0171] The overlap rate is determined based on the ratio of the number of overlapping edited segments in the second set of edited segments to the total number of edited segments in the second set of edited segments.
[0172] According to one embodiment, the reference test cases include multiple types;
[0173] The recorded vulnerability remediation assessment module 503 is specifically used for:
[0174] The environment is invoked to build an intelligent agent that constructs an executable environment and generates test scripts for the code file sample.
[0175] Run the test script in the executable environment to execute the multiple test cases on the code file sample after the predicted code patch is fixed;
[0176] Based on the pass / fail status of the multiple test cases, the recorded vulnerability remediation assessment results are determined.
[0177] According to one implementation, the evaluation device 500 for the vulnerability repair capability of code agents further includes a potential vulnerability repair evaluation module 506. This potential vulnerability repair evaluation module 506 is used to: generate several candidate test cases, wherein the candidate test cases are different from the reference test cases, and the candidate test cases meet preset conditions. The preset conditions are: executing the candidate test cases using a code file sample before the predicted code patch repair yields a failure result; executing the candidate test cases using a code file sample after the predicted code patch repair yields a success result.
[0178] Based on the aforementioned candidate test cases, the potential vulnerability remediation assessment results are obtained;
[0179] The determination of the vulnerability remediation capability assessment result of the code agent to be evaluated is also related to the potential vulnerability remediation assessment result.
[0180] Furthermore, in one embodiment, when generating several candidate test cases, the potential vulnerability remediation assessment module 506 is used to:
[0181] The code file sample, the predicted code patch, and the generation task instruction are used as inputs to the code agent, and the multiple candidate test cases are output. The generation task instruction is used to instruct the code agent to generate multiple candidate test cases based on the code file sample and the predicted code patch.
[0182] From the plurality of candidate test cases, target candidate test cases that meet the preset conditions and are different from the reference test cases are selected to obtain the plurality of candidate test cases.
[0183] In a further embodiment, when executing the potential vulnerability remediation assessment results based on the plurality of candidate test cases, the potential vulnerability remediation assessment module 506 is used to:
[0184] The predicted code patch and the plurality of candidate test cases are input into the evaluation agent, and the correlation matrix between potential vulnerabilities and candidate test cases is output. The correlation matrix represents the mapping relationship between which candidate test cases reproduce each potential vulnerability.
[0185] Based on the correlation matrix, the number of potential vulnerabilities that have been successfully fixed is determined, and each successfully fixed potential vulnerability is reproduced by at least one candidate test case.
[0186] The potential vulnerability remediation assessment result is obtained based on the number of potential vulnerabilities that have been successfully remediated.
[0187] According to one embodiment, the task instruction instructs the code agent to be evaluated to perform vulnerability checks on at least a specified range of code in the code file sample and generate code patches, wherein the specified range is determined based on the modification range extracted from the reference code patch, and the modification range indicates the range of code modifications made by the reference code patch to the code file sample.
[0188] In one possible implementation, the acquisition module 501 is specifically used for:
[0189] The code hosting platform retrieves multiple historical fix request data, which includes a repository snapshot before the fix, reference code patches provided by the developer, and reference test cases.
[0190] The vulnerability classification system is used to filter the multiple historical remediation request data, and the historical remediation request data that hit the preset vulnerability category is used as candidate sample data to obtain a candidate sample set;
[0191] Based on the candidate sample set, the sample is determined as the evaluation sample.
[0192] In another possible implementation, when filtering the multiple historical remediation request data using a vulnerability classification system, the acquisition module 501 is used to:
[0193] For any historical patch request, the reference code patch and reference test cases of the historical patch request data are input into multiple large language models respectively to obtain the vulnerability category label output by each large language model;
[0194] The final classification label is obtained by voting based on the consistency of multiple vulnerability category labels;
[0195] When the final classification label belongs to a preset vulnerability category in the vulnerability classification system, the corresponding historical remediation request data is retained;
[0196] If the final classification label does not belong to a preset vulnerability category in the vulnerability classification system, the corresponding historical remediation request data is discarded.
[0197] In another possible implementation, the code file sample contains a single documented vulnerability;
[0198] When determining the evaluation sample based on the candidate sample set, the acquisition module 501 is used to:
[0199] Candidate sample data, including a single recorded vulnerability, are selected from the candidate sample set as the evaluation sample.
[0200] In another possible implementation, the code file sample contains multiple documented vulnerabilities;
[0201] When determining the evaluation sample based on the candidate sample set, the acquisition module 501 is used to:
[0202] Sort multiple candidate sample data within the same code repository according to the merge time;
[0203] Based on the time window, at least two adjacent candidate sample data are selected from the sorted candidate sample data;
[0204] The data from the at least two candidate samples are aggregated to obtain an evaluation sample that includes multiple recorded vulnerabilities.
[0205] For specific implementation examples of each unit in the above device, please refer to the previous examples. Figure 2 The above device enables an objective assessment of the vulnerability remediation capabilities of a code-based intelligent agent in the absence of problem reports.
[0206] According to another embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed in a computer, causes the computer to perform a combination Figure 2 The method described.
[0207] According to yet another embodiment, a computer program product is also provided, including a computer program / instructions that, when executed by a processor, implement the foregoing combinations. Figure 2 The described method and steps.
[0208] According to another embodiment, a computing device is also provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements a combination... Figure 2 The method described.
[0209] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this invention can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.
[0210] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.
Claims
1. An evaluation method for the vulnerability remediation capability of code-based intelligent agents, comprising: Obtain an evaluation sample, which includes a code file sample containing vulnerabilities, a reference code patch for the vulnerabilities recorded in the code file sample, and a reference test case; The code file sample and task instructions are used as input to the code agent to be evaluated, and a predicted code patch is output. The task instructions instruct the code agent to be evaluated to perform vulnerability checks on the code in the code file sample and generate a code patch. The input to the code agent to be evaluated does not include information related to describing the vulnerability. The reference test cases are used to test the code file sample after the predicted code patch is fixed, and the recorded vulnerability fixation evaluation results are obtained based on the test results; Based on the recorded vulnerability remediation assessment results, the vulnerability remediation capability assessment result of the code agent to be evaluated is determined.
2. The method according to claim 1, wherein, The process of determining the vulnerability remediation capability assessment result of the code agent to be evaluated based on the recorded vulnerability remediation assessment results also includes the following prior steps: The predicted code patch and the reference code patch are analyzed, and the vulnerability location assessment result is obtained based on the analysis results; The determination of the vulnerability remediation capability assessment result of the code agent to be evaluated is also related to the vulnerability location assessment result.
3. The method according to claim 2, wherein, The analysis of the predicted code patch and the reference code patch, and the resulting vulnerability location assessment, includes: The predicted code patch and the reference code patch are respectively converted into an editable fragment set to obtain a first editable fragment set corresponding to the predicted code patch and a second editable fragment set corresponding to the reference code patch; An analysis is performed based on the overlap of segments in the first set of edited segments and the second set of edited segments to obtain the segment overlap rate, which indicates the overlap between edited segments in the first set of edited segments and edited segments in the second set of edited segments; Based on the fragment overlap rate, the vulnerability location assessment result is determined.
4. The method according to claim 3, wherein, The edit segment includes a range of line numbers, which indicates the range of lines of code to be modified. The analysis of fragment overlap based on the first set of edited fragments and the second set of edited fragments to obtain the fragment overlap rate includes: Obtain the line number range of each edit segment in the first edit segment set and the line number range of each edit segment in the second edit segment set; For any edit segment in the second set of edit segments, if there is an edit segment in the first set of edit segments that overlaps with the line number range of the edit segment, then the edit segment is determined to be an overlapping edit segment. The overlap rate is determined based on the ratio of the number of overlapping edited segments in the second set of edited segments to the total number of edited segments in the second set of edited segments.
5. The method according to claim 1, wherein, The reference test cases include multiple ones; The process of testing the code file sample after the predicted code patch is applied using the reference test cases to obtain the recorded vulnerability remediation assessment results includes: The environment is invoked to build an intelligent agent that constructs an executable environment and generates test scripts for the code file sample. Run the test script in the executable environment to execute the multiple test cases on the code file sample after the predicted code patch is fixed; Based on the pass / fail status of the multiple test cases, the recorded vulnerability remediation assessment results are determined.
6. The method according to any one of claims 1-5, wherein, The process of determining the vulnerability remediation capability assessment result of the code agent to be evaluated based on the recorded vulnerability remediation assessment results also includes the following prior steps: Several candidate test cases are generated, which are different from the reference test cases and meet preset conditions. The preset conditions are: executing the candidate test cases using a sample code file before the predicted code patch fix results in a failure; executing the candidate test cases using a sample code file after the predicted code patch fix results in a success. Based on the aforementioned candidate test cases, the potential vulnerability remediation assessment results are obtained; The determination of the vulnerability remediation capability assessment result of the code agent to be evaluated is also related to the potential vulnerability remediation assessment result.
7. The method according to claim 6, wherein, The generation of several candidate test cases includes: The code file sample, the predicted code patch, and the generation task instruction are used as inputs to the code agent, and the multiple candidate test cases are output. The generation task instruction is used to instruct the code agent to generate multiple candidate test cases based on the code file sample and the predicted code patch. From the plurality of candidate test cases, target candidate test cases that meet the preset conditions and are different from the reference test cases are selected to obtain the plurality of candidate test cases.
8. The method according to claim 6, wherein, The potential vulnerability remediation assessment results obtained based on the aforementioned candidate test cases include: The predicted code patch and the plurality of candidate test cases are input into the evaluation agent, and the correlation matrix between potential vulnerabilities and candidate test cases is output. The correlation matrix represents the mapping relationship between which candidate test cases reproduce each potential vulnerability. Based on the correlation matrix, the number of potential vulnerabilities that have been successfully fixed is determined, and each successfully fixed potential vulnerability is reproduced by at least one candidate test case. The potential vulnerability remediation assessment result is obtained based on the number of potential vulnerabilities that have been successfully remediated.
9. The method according to claim 1, wherein, The task instruction instructs the code agent to be evaluated to perform vulnerability checks on at least a specified range of code in the code file sample and generate code patches. The specified range is determined based on the modification range extracted from the reference code patch, and the modification range indicates the range of code modifications made by the reference code patch to the code file sample.
10. The method according to claim 1, wherein, The acquisition of evaluation samples includes: The code hosting platform retrieves multiple historical fix request data, which includes a repository snapshot before the fix, reference code patches provided by the developer, and reference test cases. The vulnerability classification system is used to filter the multiple historical remediation request data, and the historical remediation request data that hit the preset vulnerability category is used as candidate sample data to obtain a candidate sample set; Based on the candidate sample set, the sample is determined as the evaluation sample.
11. The method according to claim 10, wherein, The process of filtering the multiple historical remediation request data using a vulnerability classification system includes: For any historical patch request, the reference code patch and reference test cases of the historical patch request data are input into multiple large language models respectively to obtain the vulnerability category label output by each large language model; The final classification label is obtained by voting based on the consistency of multiple vulnerability category labels; When the final classification label belongs to a preset vulnerability category in the vulnerability classification system, the corresponding historical remediation request data is retained; If the final classification label does not belong to a preset vulnerability category in the vulnerability classification system, the corresponding historical remediation request data is discarded.
12. The method according to claim 10, wherein, The code file sample contains a single documented vulnerability; The process of determining the evaluation sample based on the candidate sample set includes: Candidate sample data, including a single recorded vulnerability, are selected from the candidate sample set as the evaluation sample.
13. The method according to claim 10, wherein, The code file sample contains multiple documented vulnerabilities; The process of determining the evaluation sample based on the candidate sample set includes: Sort multiple candidate sample data within the same code repository according to the merge time; Based on the time window, at least two adjacent candidate sample data are selected from the sorted candidate sample data; The data from the at least two candidate samples are aggregated to obtain an evaluation sample that includes multiple recorded vulnerabilities.
14. A computing device comprising a memory and a processor, wherein, The memory stores executable code, and when the processor executes the executable code, it implements the method of any one of claims 1-13.
