A logic-driven test case generation method based on a large language model
Through the logic-driven test case generation method, combined with the logic-driven thinking chain and layered repair strategy, the problems of low test case coverage and error repair in generating large language models are solved, achieving higher coverage and quality.
Patent Information
- Application Number
- CN202511013692.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-23
- Publication Date
- 2025-10-10
- Estimated Expiration
- 2045-07-23
AI Technical Summary
Existing test case generation methods based on large language models have problems such as low coverage, incomplete logic coverage, and difficulty in error repair in dealing with complex errors.
A logic-driven test case generation method is adopted. By extracting key information from the code to be tested, a logic-driven thinking chain prompt template is used to guide the large language model to generate test cases. The coverage is calculated by combining white-box and black-box testing strategies, and a layered repair strategy is used to repair test cases.
It achieves higher coverage and lower manual intervention requirements, and is particularly suitable for cross-module testing scenarios in large projects, significantly improving the quality and reliability of test cases.
Smart Images
Figure CN120523743B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of software testing, and in particular to a logic-driven test case generation method based on a large language model. Background Art
[0002] Unit testing, a core practice in software quality assurance, has made significant progress in automated generation technology in recent years, but it still faces many challenges. Existing methods are mainly divided into two categories: search-based techniques (SBST) and large language model (LLM)-based techniques, each of which has significant shortcomings.
[0003] Search-based test generation (SBST): Tools like EvoSuite generate test cases using genetic algorithms. While these tools offer high coverage, they suffer from poor readability and a lack of understanding of code semantics. These include confusing variable naming, with over 80% of identifiers being meaningless; randomized input data, often containing a high level of invalid data; and assertions lacking business logic, validating only basic properties like non-nullability.
[0004] Test generation based on large language models (LLMs): Test cases generated by tools such as ChatUniTest are highly readable, but suffer from incomplete logic coverage (such as missing boundary conditions) and high error rates (such as compilation errors and assertion failures). This is mainly because LLMs are not systematically guided to focus on the logical paths of the code. Furthermore, their repair strategies are overly simplistic and cannot effectively fix errors in the test code.
[0005] In existing technologies, the prompting process for LLM test case generation is overly simplified, which results in the generated test cases being incomplete and the coverage rate failing to reach a high level. At the same time, error repair often relies on a single strategy, usually only LLM iterative repair, which makes it difficult to deal with complex errors (such as missing dependencies and syntax errors).
[0006] Therefore, there is an urgent need for an automated test generation method that can take into account coverage, correctness and semantic understanding. Summary of the Invention
[0007] The purpose of the present invention is to provide a logic-driven test case generation method based on a large language model to solve the problems of the existing test case generation based on a large language model being insufficiently comprehensive, having low coverage, and difficulty in error repair in dealing with complex errors.
[0008] In order to achieve the above object, the technical solution of the present invention is as follows:
[0009] The present invention relates to a method for generating logic-driven test cases based on a large language model, which comprises the following steps:
[0010] S1. Extract key information from the code to be tested;
[0011] S2. Based on the key information, a logical driving thinking chain prompt template is used to guide the large language model to generate test cases;
[0012] S3. Run the test cases and calculate the coverage;
[0013] S4. Based on the test case running results and coverage, determine whether the test cases need to be repaired, if so, use a hierarchical repair strategy to repair the test cases, if not, output the test cases.
[0014] Preferably, the specific steps of S1 extracting key information from the code to be tested include:
[0015] S1.1. Scan the test project, identify all source files under the test project directory, and filter out non-test class source files through a whitelist filtering mechanism;
[0016] S1.2. Use abstract syntax tree to parse the code in the non-test class source files to generate code information for testing;
[0017] S1.3. Standardize the storage of the extracted code information.
[0018] Preferably, the specific steps of S2 guiding the large language model to generate comprehensive test cases based on the logical driving thinking chain prompt template are as follows:
[0019] S2.1. Form a test case generation prompt structure through the logical driving thinking chain prompt template;
[0020] S2.2. Send the test case generation prompt structure to the large language model and match out the test cases.
[0021] Preferably, the specific way of S2.1 forming the test case generation prompt structure through the logical driving thinking chain prompt template is to use the logical driving thinking chain to generate the following prompt structures in sequence:
[0022] S2.1.1. Prompt the large language model to analyze the core logic of the method under test and the code function;
[0023] S2.1.2. Prompt the large language model to enumerate all execution paths in the code;
[0024] S2.1.3. Prompt the large language model to perform condition combination deduction and enumerate all condition combinations;
[0025] S2.1.4. Prompt the large language model to generate corresponding test cases according to the execution paths and condition combinations obtained by deduction, and use boundary value analysis and equivalence class division method to generate boundary test scenarios.
[0026] Preferably, S2.1 forming a prompt structure for generating a test case by using a logic-driven thinking chain prompt template further includes:
[0027] S2.1.5. Prompt the large language model to compare the generated test cases with the execution paths and condition combinations.
[0028] Preferably, the S3 coverage calculation includes calculating line coverage and branch coverage, and the calculation formula is:
[0029] (1),
[0030] (2),
[0031] in, is the row coverage, is the branch coverage, is the number of executed rows, is the total number of lines of code, is the executed branch, For the total code branch.
[0032] Preferably, the specific steps of using the layered repair strategy in S4 to repair the test case are:
[0033] S4.1. Collect error information and error types for test cases that cannot be run, organize the error type, error information, test source code, and test cases into repair prompts, and send them to the large language model.
[0034] S4.2. The large language model uses regular expression matching to identify the fixed test case from the model response and replaces the original test case with the fixed test case.
[0035] S4.3. Run the repaired test case. If it works, output the test case. If it doesn't, determine whether the number of repairs has reached the maximum number of repairs. If not, return to S4.2 and increment the number of repairs by 1. If it has, proceed to S4.4.
[0036] S4.4. Determine whether the error type is a missing dependency. If so, use the automatic search and import mechanism to supplement the missing dependency. After fixing the dependency error, proceed to S4.5. If not, skip the dependency error repair step and proceed to S4.5.
[0037] S4.5. Determine whether the test case still contains errors. If so, use runtime checks to identify the incorrect code structure and the test case that fails to run properly due to the use of these incorrect codes. Pinpoint the lines where the errors occur and delete the test case containing these errors.
[0038] S4.6. Correct the assertion logic for the test case fixed in S4.5.
[0039] Compared with the prior art, the technical solution provided by the present invention has the following beneficial effects:
[0040] 1. The present invention relates to a logic-driven test case generation method based on a large language model. This method uses a logic-driven thinking chain prompt template to guide the large language model in generating test cases. Through the logic-driven thinking chain (LCoT), it integrates white-box testing (path coverage, condition combination coverage) and black-box testing (boundary value analysis, equivalence class partitioning) strategies to systematically guide the large language model in generating test cases, completely resolving the problems of missing paths and incomplete condition combinations when generating test cases using traditional LLM.
[0041] 2. The present invention relates to a logic-driven test case generation method based on a large language model. When the generated test case fails to run, a layered repair strategy is used to repair the test case, giving priority to handling syntax and compilation errors, and then specifically solving dependency missing, resistance errors (such as object construction errors, outdated API calls) and assertion logic problems. Through dynamic dependency library matching and automatic assertion correction, the need for manual intervention is significantly reduced. It is particularly suitable for cross-module testing scenarios in large projects. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 Flowchart of the logic-driven test case generation method based on large language models. DETAILED DESCRIPTION
[0043] In order to further understand the content of the present invention, the present invention is described in detail with reference to the examples. The following examples are used to illustrate the present invention but are not used to limit the scope of the present invention.
[0044] Refer to the attached Figure 1 As shown, the present invention relates to a logic-driven test case generation method based on a large language model, which includes the following steps:
[0045] S1. Preprocessing stage: extract key information from the code to be tested. The specific steps include:
[0046] S1.1. Select a Java project to be tested, scan the project, identify all Java source files in the project directory, and filter non-test source files using a whitelist filtering mechanism. Only non-test code files with a Java extension are retained, excluding all non-relevant resources such as configuration files and test case files.
[0047] S1.2. Parsing the code in the non-test class source file using an abstract syntax tree to generate code information for testing: It is a common technical means in the art to parse the code in the non-test class source file using an abstract syntax tree and extract key information. The specific method is as follows: structurally analyze the extracted non-test class source file, parse the source code into an abstract syntax tree (a kind of intermediate representation that represents the syntax composition of a program in a tree structure), traverse the abstract syntax tree nodes iteratively, identify key node types, extract class signatures and method signatures, extract method body code blocks, and finally obtain package declaration information from the compilation unit root node (a top-level structure node representing the entire source file) and collect the fully qualified names in all import declaration nodes (which record the external class information referenced by the current file).
[0048] S1.3. Standardize the extracted code information in JSON format for each class or method to provide support for subsequent stages.
[0049] S2. Test case generation phase: Based on the key information collected in S1.2, use a logic-driven thinking chain prompt template to guide the large language model to generate test cases. The invention uses a dual-channel design to construct the prompt, which consists of two parts: System Prompt and User Prompt.
[0050] The System Prompt part clearly defines the test specification (such as the Junit5 framework, Mockito3 simulation library, and reflection mechanism usage guidelines, which can also be customized).
[0051] The User Prompt is divided into two parts: basic test information and a logic-driven thinking chain (LCoT). The logic-driven thinking chain combines the ideas of path coverage and condition combination coverage in white-box testing, as well as the ideas of boundary value analysis and equivalence class division in black-box testing. Through the prompt method, the LLM analyzes the core logic of the tested method and the code functionality. The specific steps are as follows:
[0052] S2.1. Form the prompt structure for test case generation through the logic-driven thinking chain prompt template, including the following prompt structures in sequence:
[0053] S2.1.1. Prompt the large language model to analyze the core logic of the tested method and the code functionality;
[0054] S2.1.2. Prompt the large language model to enumerate all execution paths in the code;
[0055] S2.1.3. Prompt the large language model to perform condition combination derivation and enumerate all condition combinations;
[0056] S2.1.4. Prompt the large language model to generate corresponding test cases based on the derived execution paths and condition combinations, and use boundary value analysis and equivalence class partitioning to generate boundary test scenarios;
[0057] S2.1.5. Prompt the large language model to compare the generated test cases with the execution paths and condition combinations.
[0058] S2.2. Send the prompt structure generated by the test case to the large language model to match the test case.
[0059] The core innovation of the test case generation stage lies in the design of a logic-driven prompt template, which combines white-box test path coverage criteria with condition combination coverage criteria to systematically guide the generation of test cases. It also integrates black-box testing strategies (boundary value analysis, equivalence class partitioning) to cover scenarios that are difficult to reach with traditional methods, aiming to guide LLM to generate comprehensive and high-quality test cases.
[0060] S3. Run test cases and calculate coverage. Coverage calculation is used to evaluate test completeness, identify untested code, optimize test efficiency, and ensure compliance with quality requirements, thereby improving software quality and reliability. This includes calculating line coverage and branch coverage. The calculation formula is:
[0061] (1),
[0062] (2),
[0063] in, is the row coverage, is the branch coverage, is the number of executed rows, is the total number of lines of code, is the executed branch, For the total code branch.
[0064] S4. Determine whether the test case needs repair based on the test case execution results and coverage. If the test case cannot be run, it is considered repairable; if it can be run, it is considered not repairable. If repair is necessary, a layered repair strategy is used to repair the test case. If not, the test case is output.
[0065] The specific steps to repair test cases using a layered repair strategy are:
[0066] S4.1. Collect error information and error types for test cases that cannot be run, organize the error type, error information, test source code, and test cases into repair prompts, and send them to the large language model.
[0067] S4.2. Send the constructed repair hint to the large language model via an API. The large model responds, using regular expression matching to identify the repaired test case from the model response. The repaired test case replaces the original test case.
[0068] S4.3. Run the repaired test case. If it works, output the test case. If it doesn't, determine whether the number of repairs has reached the maximum number of times set (7). If not, return to S4.2 and increment the number of repairs by 1. If it has, proceed to S4.4.
[0069] S4.4. Run the test case matched in S4.3. If it works, output the test case. If it still doesn't work, meaning the error hasn't been fixed, determine whether the error is a missing dependency. If so, use the automatic search and import mechanism to add the missing dependency. After fixing the dependency error, proceed to S4.5. If not, skip fixing the dependency error and proceed to S4.5.
[0070] The principle of determining missing dependencies is as follows: after running the code, error information is extracted from the log, console, or exception stack through regular matching, focusing on the following key error types: class not found exception, class definition not found error, method does not exist error, and field does not exist error. If the above problems can be detected, it proves that there is a missing dependency problem. For missing dependency errors, all dependencies used in the source code file are added to the test code. Then an external dependency library is built based on the Maven central repository, which stores commonly used Java packages. Dependency missing detection is performed based on the error log, and the missing dependencies are supplemented through the automatic retrieval-import mechanism. Specifically, the error log is parsed, and the missing class name, such as java.lang.ClassNotFoundException, is extracted using regular matching. Then the dependency scope is determined. If the class belongs to a JDK built-in package (such as java., javax), it is determined to be an environment problem and dependency retrieval is not performed. If it belongs to a third-party library (such as org.apache, com.google), a search is performed. The same dependency is matched in the external dependency library based on the missing dependency name. If it can be retrieved, the dependency is imported into the test case execution environment to fix the dependency error.
[0071] S4.5. Determine whether there are still errors in the test case. If so, define the error that cannot be solved after multiple rounds of repair as a stubborn error. Stubborn errors mainly involve the following situations: incorrect code structure (such as class or method definition error), use of incorrect object instance when calling a method, and use of outdated API interface, etc. For stubborn errors, the present invention uses runtime verification to accurately locate the erroneous code structure itself, as well as the test cases that cannot be executed normally due to the use of the erroneous code structure or the erroneous calling method, locate the specific location in the source code file through the line number in the error log, and then determine the test case to which the line belongs through syntax tree analysis. Finally, the test case containing the located problem code is deleted, thereby ensuring the overall executableness of the test case set;
[0072] S4.6. For the test cases fixed in S4.5, perform assertion logic corrections. Specifically, when the test result does not match the expected result, the test case will be marked as failed. This situation is usually caused by an assertion error. Construct an assertion error repair template. When encountering basic assertions (such as assertTrue, assertFalse) and object or array assertions (such as assertNull, assertNotNull), first locate the line where the failed assertion is located based on the error information in the log. Then repair it by flipping the logic, that is, replacing the assertTrue assertion with assertFalse and the assertNull assertion with assertNotTrue. When encountering equal value assertions (such as assertEquals), repair them by replacing the expected value in the test case with the actual value. Similarly, first locate the line where the failed assertion belongs, extract the actual value from the error log AssertionError, and replace the expected value in the code with the actual value. By combining the above multiple repair strategies, high coverage and high availability test cases can be obtained.
[0073] Experimental example:
[0074] Experimental object selection: The dataset constructed in this study comes from two channels: (1) the evaluation corpus of the cutting-edge LLM testing tool ChatUniTest; (2) widely used open source projects in the Maven central repository.
[0075] Baseline: To comprehensively evaluate the effectiveness of TestLoter, we selected two representative baseline methods for comparative experiments:
[0076] Traditional SBST method: EvoSuite is used as a benchmark tool. This tool optimizes test case generation through genetic algorithms and has an authoritative position in the field of Java unit testing. Its core advantages are:
[0077] Population-based evolutionary mechanisms enable path space exploration;
[0078] Integrated dynamic symbolic execution enhances complex constraint solving;
[0079] Test case generation adheres to JUnit best practice guidelines.
[0080] LLM-driven approach: ChatUniTest is chosen as the comparative baseline. This framework enhances generation quality through the following innovative mechanisms:
[0081] Adaptive context focusing technique reduces semantic interference.
[0082] Multi-round generation-verification loop optimizes assertion completeness.
[0083] TestLoter configuration: DeepSeek-V3 is used as the core inference engine in this experiment. Test generation and repair are achieved through API calls. Key parameter configurations are as follows: Context window: 60k tokens, fully utilizing the model's long context processing capability to avoid prompt truncation; Temperature coefficient: set to 0.5 to balance generation diversity and certainty; Iteration mechanism: a maximum of 4 LLM queries (maxRounds=4) are performed in each method generation process. This threshold balances cost and benefit through pre-experiments.
[0084] ChatUniTest experimental setup: To rule out the influence of LLM on experimental results, the GPT-3.5-turbo-0125 model originally used by ChatUniTest is replaced with the DeepSeek-V3 model. The context length is also set to 60k, and the temperature is set to 0.5. Test cases are generated through the default maxRounds of 4 iterations.
[0085] EvoSuite experimental setup: EvoSuite version 1.1.0 is used, and the default configuration of EvoSuite is used.
[0086] Sufficiency evaluation: Whether the test cases generated by TestLoter are superior to the baseline methods in terms of coverage ability.
[0087] Evaluation indicators:
[0088] Line coverage (Line Coverage): The proportion of code lines covered by execution tests.
[0089] Branch coverage (Branch Coverage): The proportion of conditional branches covered.
[0090] Higher line coverage and branch coverage indicate that the generated test cases have higher sufficiency, whereas lower sufficiency indicates lower sufficiency. The comparison results are shown in Table 1:
[0091] Table 1: Comparison of TestLoter coverage with baseline coverage
[0092]
[0093] Table 1 shows the performance of the TestLoter method in generating test case adequacy compared to baseline methods. Compared to ChatUniTest, TestLoter achieved higher line and branch coverage across all test items, with an average increase of 8.5% in line coverage and 10.9% in branch coverage. Compared to EvoSuite, TestLoter also achieved improvements in line and branch coverage across all items, increasing line coverage by over 10% and branch coverage by over 15%. These significant coverage improvements demonstrate the effectiveness of the TestLoter method, which can achieve better results than ChatUniTest and EvoSuite.
[0094] The present invention has been described in detail above with reference to the embodiments. However, the contents described are only preferred embodiments of the present invention and should not be considered as limiting the scope of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent coverage of the present invention.
Claims
1. A logic-driven test case generation method based on a large language model, characterized in that: It includes the following steps: S1. Extract key information from the code to be tested; S2. Based on key information, a logic-driven thinking chain prompt template is used to guide the large language model to generate test cases. The specific steps are as follows: S2.
1. Use logic-driven thinking chain prompt templates to form prompt structures for test case generation; S2.
2. Send the prompt structure generated by the test case to the large language model to match the test case; The specific method of forming the prompt structure of test case generation by using the logic-driven thinking chain prompt template in S2.1 is to use the logic-driven thinking chain to sequentially generate the following prompt structures: S2.1.
1. Prompt the large language model to analyze the core logic and code functions of the method under test; S2.1.
2. Prompt the large language model to enumerate all execution paths in the code; S2.1.
3. Prompt the large language model to deduce conditional combinations and enumerate all conditional combinations; S2.1.
4. Prompt the large language model to generate corresponding test cases based on the derived execution paths and condition combinations, and use boundary value analysis and equivalence class partitioning to generate boundary test scenarios; S2.1.
5. Prompt the large language model to compare the generated test cases with the execution paths and condition combinations; S3. Run the test cases and calculate the coverage; S4. Determine whether the test case needs to be repaired based on the test case running results and coverage. If necessary, use a layered repair strategy to repair the test case. If not, output the test case.
2. The method for generating logic-driven test cases based on a large language model according to claim 1, wherein: The specific steps of S1 extracting key information from the code to be tested include: S1.
1. Scan the project to be tested, identify all source files in the project directory, and filter out non-test source files through a whitelist filtering mechanism. S1.
2. Use the abstract syntax tree to parse the code in non-test source files and generate code information for testing. S1.
3. Standardize and store the extracted code information.
3. The method for generating logic-driven test cases based on a large language model according to claim 1, wherein: The S3 coverage calculation includes calculating line coverage and branch coverage, and the calculation formula is: (1)， （2）， in, is the row coverage, is the branch coverage, is the number of executed rows, is the total number of lines of code, is the executed branch, For the total code branch.
4. The method for generating logic-driven test cases based on a large language model according to claim 1, wherein: The specific steps of S4 using the layered repair strategy to repair the test case are as follows: S4.
1. Collect error information and error types for test cases that cannot be run, organize the error type, error information, test source code, and test cases into repair prompts, and send them to the large language model. S4.
2. The large language model uses regular expression matching to identify the fixed test case from the model response and replaces the original test case with the fixed test case. S4.
3. Run the repaired test case. If it works, output the test case. If it doesn't, determine whether the number of repairs has reached the maximum number of repairs. If not, return to S4.2 and increment the number of repairs by 1. If it has, proceed to S4.
4. S4.
4. Determine whether the error type is a missing dependency. If so, use the automatic search and import mechanism to supplement the missing dependency. After fixing the dependency error, proceed to S4.
5. If not, skip the dependency error repair step and proceed to S4.
5. S4.
5. Determine whether the test case still contains errors. If so, use runtime checks to identify the incorrect code structure and the test case that fails to run properly due to the use of these incorrect codes. Pinpoint the lines where the errors occur and delete the test case containing these errors. S4.
6. Correct the assertion logic for the test case fixed in S4.5.