Method and system for evaluating assertion error attribution capability based on large model

Abstract

The invention discloses an assertion error attribution capability assessment method and system based on a large model, and the method comprises the steps: collecting original data in a real defect report, and constructing a confusion test data set containing multiple difficulty levels; the method comprises the following steps of: pre-defining three test scenes, executing an automatic test process, analyzing a log and recording corresponding test error report information under each scene: extracting context information of a focus method code, and performing semantic enhancement by utilizing an inference type large model; integrating the test error information and the code context information subjected to semantic enhancement, and constructing and generating an input prompt word for capability assessment; after the large model receives processing, outputting a deduction conclusion of the large model on the assertion error; and calculating a core evaluation index for measuring the assertion error attribution capability of the large model by comparing the consistency degree between a result output by the large language model and a predefined standard answer. By utilizing the method, the assertion error attribution capability of the large model facing real defects can be objectively and accurately evaluated.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of software testing and large language model generation, and in particular to a large model-based assertion error attribution capability evaluation method and system. BACKGROUND

[0002] In the field of modern software development, software testing, as a core link to ensure software quality, plays a crucial role in software stability and reliability. However, traditional manual testing methods have long been plagued by low writing efficiency, incomplete coverage of complex business logic, and difficulty in maintaining test cases. Although traditional automated testing tools can generate unit test code, the generated code often has poor readability, further exacerbating the complexity of testing work.

[0003] In recent years, with the rapid development of large language model (LLM) technology, it can quickly generate unit test code with good readability and basic functions by virtue of its efficient processing capability for natural language instructions, greatly improving the efficiency of unit test writing. This technological breakthrough has brought new changes to the field of software testing.

[0004] Chinese patent document with publication number CN119576799A discloses a dynamic language test case automatic generation method based on a large language model. Chinese patent document with publication number CN119782178A discloses a unit test generation method based on a large language model.

[0005] However, it cannot be ignored that the current large language model-based unit test generation tools still have major defects. In the testing evaluation process, existing industry tools mostly focus on test coverage, ignoring the key indicator of assertion accuracy, resulting in high assertion error rates of unit tests generated by these tools. In addition, the current large language model lacks precise error attribution capability, and when assertion failure occurs, it cannot accurately determine whether it is a software defect or a problem with the generated test case itself, thus greatly reducing its usability and requiring a lot of manpower for subsequent maintenance.

[0006] Currently, there is still a lack of standardized evaluation system for large language model assertion error attribution capability analysis, which seriously restricts the further development and application of software testing technology based on large language models. SUMMARY

[0007] In order to fill the gap of the lack of standardized evaluation system for the assertion error attribution capability of large language model in the industry at present, the present application provides a large model-based assertion error attribution capability evaluation method and system, which can objectively and accurately evaluate the assertion error attribution capability of large model in the face of real defects through large model generation technology and containerized scene execution technology.

[0008] A large model-based assertion error attribution capability evaluation method, comprising:

[0009] (1) Collecting original data in real defect reports, including focus method source code, problem description and original test code; generating obfuscated test code based on the collected original data, and constructing an obfuscated test dataset containing multiple difficulty levels after preliminary verification and screening of the obfuscated test code;

[0010] (2) Predefining three test scenarios, executing an automated test process using the obfuscated test dataset, parsing logs and recording corresponding test error information under each scenario:

[0011] (3) Extracting context information of the focus method code using static analysis technology, and performing semantic enhancement using an inference-based large model to obtain the code context after semantic enhancement;

[0012] (4) Integrating the test error information and the code context information after semantic enhancement to construct input prompts for capability evaluation; after the large language model accepts the input prompts, it outputs its inference conclusion on the assertion error;

[0013] (5) Comparing the consistency between the output results of the large language model and the predefined standard answers to calculate the core evaluation index for measuring the assertion error attribution capability of the large model.

[0014] In step (1), based on the attribution capability difference of the benchmark large model in the pre-test process, a dataset hierarchical system containing three levels of attribution difficulty is dynamically constructed.

[0015] In step (1), the preliminary verification and screening of the obfuscated test code includes: basic syntax checking and compilation attempt, and screening out invalid tests that cannot pass this stage of verification.

[0016] In step (1), there are three ways to generate obfuscated test code, including:

[0017] Rule mutation: at least one mutation operation based on predetermined heuristic rules is performed on the original test code to generate obfuscated test code;

[0018] Large model generation: according to the key information extracted from the original data, the large language model is driven to generate obfuscated test code by following a pre-set prompt template.

[0019] Artificial generation: combine the generation capability of large models with the review and verification process of professionals to generate more deceptive and confusing test code.

[0020] The predetermined heuristic rules include:

[0021] Assertion value variation: directly modify the specific value set in the assertion statement, or perform logical inversion on the expected result of the Boolean type assertion;

[0022] Assertion type variation: modify the target type setting required for type detection operations in the assertion;

[0023] Set operation variation: change the specific type or form of set-related operations involved in the assertion;

[0024] Conditional logic variation: reverse the logical relationship in the conditional judgment statement, or directly remove the conditional statement from the code;

[0025] Loop operation variation: adjust the parameter setting for defining the loop boundary in the loop control structure, or add new termination conditions to the loop control;

[0026] Null value check variation: reverse the execution result of the null value check logic, or remove the null value check statement in the code.

[0027] In step (2), three test scenarios are predefined as follows:

[0028] (2-1) Use the test after the confusion variation to implement the correct focus method, simulate the case where the unit test has defects and the business logic itself is correct;

[0029] (2-2) Only use the original correct test with the wrong focus method implementation, simulate the case where the unit test is correct but the business logic itself has defects;

[0030] (2-3) Use the test after the variation with the wrong focus method implementation, simulate the case where the unit test has defects and the business logic has defects at the same time.

[0031] In step (3), the reasoning large model is used for semantic enhancement, which specifically includes:

[0032] The focus method code context obtained by static analysis is understood by the reasoning large model to understand its intended function and design intent, systematically deconstruct its internal execution logic and process, identify and label the understanding bottlenecks caused by missing or ambiguous information, and propose potential exploration or verification paths, and finally generate a semantically enhanced code context.

[0033] In step (5), the core evaluation indicators include:

[0034] Full pass rate: reflects the proportion of the complete question and answer result sequence output by the large language model for three scenarios, each position of which is completely correct;

[0035] Partial correct rate: the proportion of correct results that can match the standard answer in the output results of any one of the three question and answer tasks independently executed by the model.

[0036] A large model-based assertion error attribution capability evaluation system, comprising:

[0037] A data set construction module for constructing a confusion test data set containing multiple difficulty levels;

[0038] A unit test execution module for configuring three predefined test scenarios and collecting log results of executing tests in different scenarios, and extracting corresponding test error information;

[0039] A context enhancement module for extracting the focus code context and generating a semantically enhanced code context using an inferential large model, and generating an evaluation context in combination with the test error information output by the unit test execution module;

[0040] A capability evaluation module for guiding the large model to ask and answer using the evaluation context output by the context enhancement module, and obtaining the final assertion error attribution capability evaluation indicators.

[0041] A large model-based assertion error attribution capability evaluation system, comprising a memory and one or more processors, the memory storing executable code, and the one or more processors executing the executable code to implement the above-mentioned assertion error attribution capability evaluation method.

[0042] Compared with the prior art, the present application has the following beneficial effects:

[0043] 1. The present application innovatively proposes a large model-based assertion error attribution capability evaluation method, which helps to identify and select large language models or specific model versions that perform well in assertion error root cause positioning, and can significantly accelerate the precise positioning and repair of defects in the software development process.

[0044] 2. The present application innovatively designs a large model-based assertion error attribution capability system, and the data set construction module in the system can quickly and conveniently generate high-quality evaluation benchmark data sets covering different difficulty levels and containing diversified variation types. The generated high-quality and high-complexity data sets can not only be used to evaluate the assertion error attribution capability of the large model, but also can be used to train or fine-tune the capability of the large model in error attribution.

[0045] 3、The context enhancement module in the application accurately locks the focus code through static analysis technology, and uses a large inference model to deeply analyze the code logic of the extracted focus method, to generate a semantically enhanced code context, and to convert the original, possibly insufficient error context information into more fitting prompt words for the focus method code logic. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 A flow chart of a large model-based assertion error attribution capability evaluation method is provided for an embodiment of the application.

[0047] Figure 2 A system structure diagram of a large model-based assertion error attribution capability evaluation system is provided for an embodiment of the application. DETAILED DESCRIPTION

[0048] The application will be further described in detail below in conjunction with the drawings and embodiments, and it should be noted that the following embodiments are intended to facilitate the understanding of the application and do not limit it in any way.

[0049] As shown in Figure 1 A large model-based assertion error attribution capability evaluation method, comprising the following steps:

[0050] S10, constructing a confusion test data set containing multiple difficulty levels according to the basic data set swe-bench-java.

[0051] S20, deploying corresponding patch configurations for three pre-defined simulation scenarios in the configured containerized running environment, and executing corresponding automated test processes.

[0052] S30, extracting the context information of the focus method code, and enriching and enhancing the context semantically with the help of the large model.

[0053] S40, integrating the extracted error description information and the semantically enhanced code context information to construct and generate input prompt words for capability evaluation.

[0054] S50, receiving and processing the evaluation prompt words generated in step S40 by the large model, and outputting its inference conclusion for the problem type attribution under the three pre-defined scenarios.

[0055] S60, by comparing the consistency between the question and answer result sequence output by the large model and the pre-defined standard answer sequence, obtaining and calculating the core evaluation indicators for measuring the attribution capability of the model.

[0056] Further, the specific implementation steps of step S10 are:

[0057] S101, write the container internal automation running process with java language, the specific logic is to extract the focus method code, problem description, test code first; generate obfuscated test code; verify detection, screen out the test with compilation failure;

[0058] S102, package the java code in S101 as a jar package, mount it in the container to run, and get the obfuscated test code.

[0059] Further, the specific implementation steps of S101 are as follows:

[0060] S101.1, extract from the basic data set swe-bench-java:

[0061] fix_patch field (Java code patch to solve the problem)

[0062] test_patch field (Java test code patch to solve the problem)

[0063] resolved_issues field (associated problem list);

[0064] S101.2, extract from test_patch:

[0065] modified_file_path

[0066] modified_method

[0067] S101.3, locate and load the target source code according to modified_file_path;

[0068] S101.4, execute the rule mutation path when the system configuration parameter test_generation_mode="rule_mutation":

[0069] S101.4.1, extract the abstract syntax tree AST of the source code using JavaParser;

[0070] S101.4.2, inject the pre-defined mutation rules of table 1 into AST;

[0071] S101.4.3, generate the mutated Java test code;

[0072] Table 1 Mutation type and operation example

[0073] Mutation type Operation example Assert value mutation Modify assertEquals(5, result) to assertEquals(6, result) Assert type mutation Change assertInstanceOf(String.class, obj) to assertInstanceOf(Integer.class, obj) Collection operation mutation Replace assertContains("a", list) with assertNotContains("a", list) Conditional logic mutation Invert the if(x>0) logic to if(x<=0) or remove the conditional statement altogether Loop operation mutation Modify for(int i=0; i<5; i++) to for(int i=0; i<10; i++) Null check mutation Change assertNotNUll(obj) to assertNull(obj)

[0074] S101.5, when the system configuration parameter test_generation_mode = "llm_generation" is executed, the large model generation path is executed:

[0075] S101.5.1, fill the source code of S101.3 and resolved_issues description to the obfuscated test generation prompt template;

[0076] S101.5.2, call the large model with the obfuscated test generation prompt word generated by S101.5.1;

[0077] S101.5.3, output the initial Java unit test code generated_test_code;

[0078] S101.6, perform run verification on the test code generated by S101.4.3 or S101.5.3, execute the Java test code generated by S101.5.3 using the JUnit framework, and then screen out the test code that cannot run.

[0079] Further, the specific implementation steps of S102 are as follows:

[0080] S102.1, compile the code implementation of S101.1 to S101.6 into an executable JAR file;

[0081] S102.2, mount the JAR file obtained in S102.1 to a container, write a script in Shell language to automatically run the JAR file, and return the generated obfuscated code to the host;

[0082] S102.3, combine the obfuscated code obtained in S102.2 with the basic defect dataset swe-bench-java to build an obfuscated test dataset.

[0083] Further, the specific implementation steps of step S20 are as follows:

[0084] S201, implement the deployment of three preset simulation scenarios;

[0085] S202, execute tests under different simulation scenarios and extract test error information from the run logs;

[0086] Further, the specific implementation steps of S201 are as follows:

[0087] S201.1, apply the mutated test patch and the correct focus method patch at the same time to simulate the condition that the unit test has errors and the business code logic is correct;

[0088] S201.2, only apply the correct test patch without mutation, but not the correct focus method patch, to simulate the condition that the unit test is correct but the business code logic is wrong;

[0089] S201.3, only apply the mutated test patch, do not apply the correct focus method patch, to simulate the condition that the unit test error and the business code logic error occur at the same time.

[0090] Further, the specific implementation steps of S202 are as follows:

[0091] S202.1, for the 9 kinds of java projects appearing in the data set, corresponding shell scripts are written respectively to deal with java projects using different build tools (maven or gradle);

[0092] S202.2, use the log analysis tool to extract test error information and return it to the host.

[0093] Further, the specific implementation steps of step S30 are as follows:

[0094] S301, extract the context information of the focus method code;

[0095] S302, and use the reasoning large model to enrich and enhance the context information extracted in S301 in the semantic level.

[0096] Further, the specific implementation steps of step S301 are as follows:

[0097] S301.1, use JavaParser to parse the project code and get the abstract syntax tree (AST);

[0098] S301.2, test execution error mapping table, get the focus method name corresponding to the test execution error, and then use AST to lock the specified focus method body;

[0099] S301.3, use program slicing technology in backward analysis to further obtain the method signature of other methods called in the focus method.

[0100] Further, the specific implementation steps of step S302 are as follows:

[0101] S302.1, generate a functional description of the input code context and infer the implementation purpose of this part of code;

[0102] S302.2, for the input code context, decompose the logic steps according to the code execution order, involving operation subject, action, result and other elements;

[0103] S302.3, annotate the part of the understanding obstacle caused by the lack of sufficient information in the context, and give a suggestion for verification direction.

[0104] Specifically, the context template used in step S302 is a heuristic prompt word template for semantic enhancement.

[0105] Further, the specific implementation steps of step S40 are as follows:

[0106] S401, integrate the information obtained in the above steps;

[0107] S402, construct an evaluation prompt word using dynamic context information.

[0108] Further, the specific implementation steps of step S401 are as follows:

[0109] S401.1, fuse the test case code that triggers the error, the error information in the test execution process, and the focus method context after semantic enhancement processing;

[0110] S401.2, according to the context restriction of different large models, the fused context is cropped, and information with low importance is preferentially discarded, such as reference method signature obtained by program slicing.

[0111] Further, the specific implementation steps of step S402 are as follows:

[0112] S402.1, generate an evaluation prompt word using the evaluation context generated in S401 and the evaluation prompt word template

[0113] S402.2, reorder the evaluation prompt words generated in S402.1 for the three simulation scenarios, which are used to guide the large model to perform three independent inference question answering tasks for these three scenarios.

[0114] Further, the specific implementation of step S50 is:

[0115] S501, compare and verify the large language model answer sequence obtained in step S402 with the known standard answer sequence one by one;

[0116] S502, calculate the question and answer accuracy index of the model on the complete scene sequence, and the correctness index of the single independent question and answer task.

[0117] Specifically, the number of times the question and answer result sequence is consistent with the standard answer sequence is the total evaluation number N; the number of times the output result of any one of the three independent question and answer tasks performed by the model can match the standard answer is the number of partial correct answers; and the total evaluation number is the data entry number of the evaluation data set. Let N be the total evaluation number, P fFor complete pass number, P p For partial pass number, the evaluation index calculation formula of the part is:

[0118] Complete pass rate R f :

[0119] Partial correct rate R p :

[0120] As Figure 2 shown, a large model-based assertion error attribution capability evaluation system, the input is a basic defect dataset (swe-bench-java is used in this embodiment), and the output is the assertion error attribution capability evaluation result of mainstream large models.

[0121] The system includes the following modules:

[0122] Dataset construction module: used for constructing a confusion test dataset containing multiple difficulty levels.

[0123] Unit test execution module: used for configuring three preset test scenarios in a container and collecting log results of executing tests in different scenarios, and extracting corresponding test error information.

[0124] Context enhancement module: used for extracting the focus code context and generating a semantically enhanced code context using an inferential large model, and generating an evaluation context in combination with the test error information output by the unit test execution module.

[0125] Capability evaluation module: used for guiding the large model to ask questions and answer using the evaluation context output by the context enhancement module, and obtaining the final assertion error attribution capability evaluation index.

[0126] Specifically, the dataset construction module contains four functional units, including a container execution unit, a rule variation unit, a large model generation unit, and a manual generation unit.

[0127] The container execution unit is mainly responsible for calling the pre-compiled Java program (JAR package) to automatically execute the following processes:

[0128] Extract the focus method code, original problem description, and correct test code in the basic dataset;

[0129] Pass the above inputs to the rule variation unit / large model generation unit / manual generation unit, and receive the generated confusion test code;

[0130] Verify the compilation passability of the confusion test code and filter out the compilation failure cases.

[0131] The rule variation unit is specifically responsible for applying the six types of heuristic rules in Table 1 to generate obfuscated test code.

[0132] The large model generation unit is mainly responsible for using the obfuscated test generation prompt template combined with the extracted information to drive the large model to generate obfuscated test code.

[0133] The artificial generation unit is mainly based on the obfuscated test generated by the large model to further select some complex tests for deeper variation, making the obfuscated test more confusing and more difficult to identify.

[0134] Specifically, the unit test execution module includes a simulation scenario deployment, a test execution engine, and an error capture tool.

[0135] The simulation scenario deployment is to combine patches according to scenarios in a container, and the three preset scenarios are shown in S201.

[0136] The test execution engine is mainly responsible for using different shell scripts to automatically execute unit tests in docker containers for different projects to deal with java projects using different build tools (maven and gradle).

[0137] The error capture tool is mainly responsible for collecting test execution error information in the log and sending it back to the host.

[0138] Specifically, the context enhancement module includes primary context extraction, semantic enhancement processing, and dynamic context combination.

[0139] Primary context extraction is to use static code analysis techniques to accurately locate the focus method directly related to the error.

[0140] Semantic enhancement processing is to use a reasoning large language model to deeply analyze the code logic of the extracted focus method and generate a semantically enhanced code context.

[0141] Dynamic context combination is to use a prompt word template to fuse the error information captured during test execution and the focus method context processed by semantic enhancement to form a dynamic context information containing more semantic information.

[0142] Specifically, the capability evaluation module includes a prompt word constructor, a multi-round question and answer mechanism, and a result verification unit.

[0143] The prompt word constructor is mainly responsible for filling the fused dynamic context information into the prompt word template to construct an input prompt word for guiding the large language model to evaluate the capability.

[0144] The multi-round question and answer mechanism is to reorder the evaluation prompt words generated for the above three simulation scenarios, and then guide the large language model to perform three independent inference question and answer for the three scenarios.

[0145] The result verification unit is mainly responsible for comparing the sequence of three numbers (representing the question and answer results for the three scenarios) output by the large language model with the known standard answer sequence one by one, and calculating the question and answer accuracy rate of the model on the complete scenario sequence and the correct rate of the single independent question and answer task.

[0146] In summary, the assertion error attribution ability evaluation method and system based on a large model effectively solves the problem of lacking reliable benchmarks to systematically evaluate the software testing error root cause analysis ability of LLM in the prior art. The method can objectively compare the performance of different LLMs in complex error diagnosis tasks and provides a powerful tool, filling an important gap in the evaluation benchmark in this field, and has important significance for promoting the development of intelligent software testing technology.

[0147] The above embodiments have described the technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the present application. Any modification, supplement and equivalent replacement made within the principle range of the present application shall be included in the protection scope of the present application.

Claims

1. A method for evaluating the ability of large model-based assertion error attribution, characterized in that, The method comprises the following steps: (1) Collecting original data in real defect reports, including focus method code, problem description and original test code; Based on the collected original data, generate obfuscated test code, and after preliminary verification and screening of the obfuscated test code, construct an obfuscated test dataset containing multiple difficulty levels; (2) Predefine three test scenarios, execute automated testing processes using the obfuscated test dataset, parse logs and record corresponding test error information under each scenario: (3) Extract the context information of the focus method code using static analysis techniques, and use the reasoning-based large model for semantic enhancement to obtain the code context after semantic enhancement; (4) Integrate the test error information and the code context information after semantic enhancement to construct and generate input prompts for capability evaluation; After the large language model accepts the input prompt, it outputs its inference conclusion on the assertion error; (5) By comparing the consistency between the output results of the large language model and the pre-defined standard answers, calculate the core evaluation index for measuring the large model's assertion error attribution capability.

2. The large model-based assertion error attribution capability evaluation method according to claim 1, characterized in that, In step (1), the preliminary verification and screening of the obfuscated test code includes: performing basic syntax checking and compilation attempts to filter out invalid tests that cannot pass this stage of verification.

3. The large model-based assertion fault attribution capability evaluation method according to claim 1, characterized in that, In step (1), there are three ways to generate obfuscated test code, including: Rule mutation: At least one mutation operation based on predetermined heuristic rules is performed on the original test code to generate obfuscated test code; Large model generation: Based on the key information extracted from the original data, follow the pre-set prompt template to drive the large language model to generate obfuscated test code; Artificial generation: Combine the generation capability of the large model with the review and verification process of professional personnel to generate obfuscated test code with higher obfuscation.

4. The large model-based assertion error root cause capability evaluation method according to claim 3, characterized in that, The predetermined heuristic rules include: Assertion value mutation: Directly modify the specific value set in the assertion statement, or perform logical inversion on the expected result of the Boolean type assertion; Assertion type mutation: Modify the target type setting required for type detection operations in the assertion; Set operation mutation: Change the specific type or form of set-related operations involved in the assertion; Conditional logic mutation: Invert the logical relationship in the conditional judgment statement, or remove the conditional statement directly from the code; Loop operation mutation: Adjust the parameter setting for defining the loop boundary in the loop control structure, or add new termination conditions for loop control; Null value check mutation: Invert the execution result of the null value check logic, or remove the null value check statement in the code.

5. The large model-based assertion fault attribution capability evaluation method according to claim 1, characterized in that, In step (2), the three predefined test scenarios are as follows: (2-1) Use the obfuscated test after mutation with the correct focus method implementation to simulate the case where the unit test has defects but the business logic is correct; (2-2) Use only the original correct test with the wrong focus method implementation to simulate the case where the unit test is correct but the business logic has defects; (2-3) Use the mutated test with the wrong focus method implementation to simulate the case where the unit test has defects and the business logic has defects.

6. The large model-based assertion fault attribution capability evaluation method according to claim 1, characterized in that, In step (3), semantic enhancement is performed using a reasoning large model, specifically including: The focus method code context obtained by static analysis is used to understand its intended function and design intent using a reasoning large model, systematically deconstructing its internal execution logic and flow, identifying and labeling understanding bottlenecks caused by information gaps or ambiguity, and proposing potential exploration or verification paths, ultimately generating a semantically enhanced code context.

7. The large model-based assertion fault attribution capability evaluation method according to claim 1, characterized in that, In step (5), the core evaluation indicators include: Complete pass rate: reflects the proportion of complete and correct results in the output of the large language model for the three scenarios; Partial correct rate: the proportion of correct results that match the standard answer in any one of the three question and answer tasks performed independently by the model.

8. A large model-based assertion error attribution capability evaluation system, characterized in that, Including: Dataset construction module: used to construct a confusion test dataset containing multiple difficulty levels; Unit test execution module: used to configure three pre-defined test scenarios and collect log results for executing tests in different scenarios, and extract corresponding test error information; Context enhancement module: used to extract the focus code context and generate a semantically enhanced code context using a reasoning large model, and generate an evaluation context combining the test error information output by the unit test execution module; Capability evaluation module: used to guide the large model to ask questions and answer using the evaluation context output by the context enhancement module, and obtain the final assertion error attribution capability evaluation indicators.

9. A large model-based assertion error attribution capability evaluation system, characterized in that, A memory and one or more processors, the memory having stored therein executable code, the one or more processors executing the executable code to implement the assertion error attribution capability evaluation method of any one of claims 1-7.

Images