Directional test case generation method based on large language model and symbolic execution tool

Abstract

The invention belongs to the related technical field of software engineering, and particularly relates to a directional test case generation method based on a large language model and a symbolic execution tool, which comprises the following steps: generating a cue word p based on a to-be-tested program P and a test object code branch T, and generating a test case i through the large language model; using i to test P, and if T is covered, completing generation; otherwise, the code coverage information of the P in the current test is added to the current p in an accumulated mode, the test cases i are regenerated and tested, when the number of retimes reaches the preset retry upper limit m, T is still not covered, the m test cases i serve as input templates, the program literal quantity in each input template is modified into symbolization variables of the symbolization execution tool, and the program literal quantity in the input templates is changed into the symbolization variables of the symbolization execution tool. Constructing a test entry function f of the symbolic execution tool; and synchronously inputting f, P and T into a symbolic execution tool to generate a corresponding test case j, and testing P to complete the generation of the test case. According to the invention, the software test coverage rate can be improved.

Description

Technical Field

[0001] The present invention belongs to the technical field related to software engineering, and more specifically, relates to a method for generating directed test cases based on a large language model and a symbolic execution tool. Background Art

[0002] In software engineering, generating targeted test input (i.e., directed test input) is a crucial task in automated testing. The primary goal of this type of input is to improve test effectiveness by automatically generating specific test cases. Directed input is widely used in tasks such as reproducing software defects, ensuring test case coverage, and integrating with other testing tools to improve overall performance. The formal definition of generating directed test cases is as follows: let the input space be I, the target be a function T whose output is 0 or 1, indicating whether a particular input achieves the target (covers the target code line or branch), and the constraint set be C, which describes the conditions that the input must satisfy to ensure validity. The generation process can be represented as a function G, which generates a satisfying input from the input space, the target, and the constraints. Finally, the definition of directed test input can be summarized as G(I, T, C), where the generated input satisfies the requirements of the target and the constraints.

[0003] The paper "Targeted test input generation using symbolic-concrete backward execution" proposes a method for generating test cases that can reach the target code location using reverse symbolic execution. This method searches from the target location toward the program entry, collecting constraints along the path. After reaching the entry, a constraint solver is used to solve the set of constraints along the path, thereby obtaining input that meets the constraints and reaches the target. This method based on reverse symbolic execution has significant advantages. Its solution process is built on a solid mathematical foundation and can provide a high success rate. However, this method also faces the inherent drawbacks of symbolic execution, such as the complex construction of the program function call graph, the path explosion problem, and the difficulty of solving nonlinear constraints. These issues not only lead to solution failures but also may cause runtime timeouts, making it difficult to effectively apply in real-world software development and testing projects.

[0004] Another approach, proposed in the paper "Code-Aware Prompting: A Study of Coverage-Guided Test Generation in Regression Setting using LLM," uses lightweight static analysis to collect constraints between program entry points and targets, synthesizes corresponding prompts, and then queries a large language model. Leveraging the model's code understanding and generation capabilities, the model generates inputs that are likely to reach the target with a certain probability. This approach first extracts path constraints and then passes them to the large language model for test case generation. Thanks to the powerful understanding and reasoning capabilities of large language models, they significantly outperform traditional symbolic execution tools in areas involving external knowledge of the code and operating system interactions. This enables large-model-based generation methods to better handle complex software testing scenarios. However, large language models still have inherent limitations when handling mathematical problems and logical reasoning. This leads to poor performance on problems such as long array indexing, long string generation, and complex memory management.

[0005] Therefore, there is an urgent need in the field for a method that can ensure the comprehensiveness and accuracy of test case generation. Summary of the Invention

[0006] In response to the above defects or improvement needs of the prior art, the present invention provides a method for generating targeted test cases based on a large language model and a symbolic execution tool, the purpose of which is to improve the success rate of generating targeted test cases.

[0007] To achieve the above objectives, according to one aspect of the present invention, a method for generating targeted test cases based on a large language model and a symbolic execution tool is provided, comprising:

[0008] Based on the program to be tested P and the test target code branch T, generate the prompt word p of the large language model to generate the test case i through the large language model;

[0009] Test case i is used to test P. If T is covered, the generation of a test case is completed. Otherwise, the code coverage information of P in the current test is accumulated and added to the current prompt word p. Test case i is regenerated and tested. When T is still not covered after the number of retry attempts reaches the preset retry limit m, the m test cases i generated by the large language model are used as input templates. According to the usage requirements of the symbolic execution tool, the program literals in each input template are modified into symbolic variables of the symbolic execution tool. Each modified input template is used as the configuration information of the symbolic execution tool to construct a test entry function f of the symbolic execution tool.

[0010] Each test entry function f is used as the driving function of the symbolic execution tool, P is used as the analysis object of the symbolic execution tool, and T is used as the analysis target of the symbolic execution tool. The symbolic execution tool is input synchronously to generate the corresponding test case j; test case j is used to test P, and the test case j corresponding to T being covered in the test is output to complete the generation of the test case.

[0011] Furthermore, the value of m is set to 1-10.

[0012] Furthermore, when the large language model is a code large language model, its hyperparameter temperature is set to 0-0.2.

[0013] Furthermore, the value of m is set according to the hyperparameter temperature of the large language model. The higher the temperature, the higher the value of m.

[0014] According to another aspect of the present invention, an electronic device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0015] According to another aspect of the present invention, a computer-readable storage medium is provided, which includes a stored computer program, wherein when the computer program is executed by a processor, the device where the storage medium is located is controlled to execute the steps of the above-mentioned method.

[0016] According to another aspect of the present invention, a computer program product is provided, comprising a computer program or instructions, which implement the steps of the method described above when executed by a processor.

[0017] In general, the above technical solutions conceived by the present invention have the following beneficial effects compared with the prior art:

[0018] 1. The present invention provides a method for generating targeted test cases based on a large language model and a symbolic execution tool. By designing a collaborative framework for a large model and a symbolic execution tool, a higher success rate for generating targeted test cases is achieved than the method of using a large model / symbolic execution tool alone, or the combined method of intersecting the outputs of the large model and the symbolic execution tool. Specifically, it includes the following two aspects: (1) Semantic reasoning stage: Based on the understanding ability of the large language model and combined with the external knowledge of the large language model, the test case requirements of the target program are understood and analyzed. Through the large language model, the code context semantics can be deeply analyzed, and the logical type, data structure characteristics and associated constraints of the input parameters can be intelligently inferred, effectively overcoming the type misjudgment and structural reasoning loss problems caused by the lack of semantic understanding in traditional symbolic execution technology, and avoiding the path enumeration explosion or constraint solution timeout caused by this. (2) Precise solution stage: The literal parameters obtained by reasoning are selectively symbolized. The symbolic execution tool can use the symbolic execution engine to dynamically collect the path constraint set along the target path, and perform precise mathematical deduction through the constraint solver to generate test case input values ​​that meet the boundary conditions. This process specifically addresses the inherent shortcomings of large language models in discrete mathematical computation and logical equation derivation, ensuring the numerical accuracy and logical completeness of literal reasoning. By combining semantic-level reasoning (large model capabilities) with mathematical-level solution (symbolic execution capabilities), this collaborative mechanism maintains the semantic understanding advantages of large models while inheriting the mathematical rigor of symbolic execution. Through the complementary enhancement of these two technologies, the ultimate goal of increasing the success rate is achieved.

[0019] 2. When the method of the present invention uses a large language model to iteratively derive test cases, it also uses existing test results (if any) to update the prompt words. Specifically, counterexamples are added to the prompt words to avoid the large language model from repeatedly generating incorrect inputs, thereby improving generation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A flowchart of a method for generating directed test cases based on a large language model and a symbolic execution tool provided by an embodiment of the present invention;

[0021] Figure 2 A flowchart of a method for determining whether a generated directional input reaches a test target provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0023] Example 1

[0024] A directed test case generation method based on large language models and symbolic execution tools, such as Figure 1 Shown, including:

[0025] Based on the program to be tested P and the test target code branch T (that is, the given code line or branch to be tested), generate a prompt word p of the large language model to generate a test case i through the large language model;

[0026] Test case i is used to test P. If T is covered, the generation of a test case is completed. Otherwise, the code coverage information of P in the current test is accumulated and added to the current prompt word p. Test case i is regenerated and tested. When T is still not covered after the number of retry attempts reaches the preset retry limit m, the m test cases i generated by the large language model are used as input templates. According to the usage requirements of the symbolic execution tool, the program literals in each input template are modified into symbolic variables of the symbolic execution tool. Each modified input template is used as the configuration information of the symbolic execution tool to construct a test entry function f of the symbolic execution tool.

[0027] Each test entry function f is used as the driving function of the symbolic execution tool, P is used as the analysis object of the symbolic execution tool, and T is used as the analysis target of the symbolic execution tool. The symbolic execution tool is input synchronously to generate the corresponding test case j; test case j is used to test P, and the test case j corresponding to T being covered in the test is output to complete the generation of the test case.

[0028] The method of this embodiment automatically generates test cases that cover specified code lines or branches, and is used to improve software test coverage. The program literals in the above-mentioned input templates, such as immediate values, constant strings, etc.

[0029] As a preferred implementation, the retry upper limit m is generally set to 1 to 10, and is set to 5 by default.

[0030] In particular, in this embodiment, the prompt word of the large model includes three parts: (A) the prompt word "generate a test case for program P that can reach target T" and other equivalent prompt words with similar meanings; (B) the relevant code of program P from the entry address to target T, such as source code or partial slice code; (C) existing test results (if any), to prevent the large language model from repeatedly generating incorrect input, are added to the prompt word in the format of counterexamples.

[0031] In particular, in this embodiment, the prompt word technology includes a prompt word technology using a few-sample learning prompt words and a thought chain prompt word.

[0032] This embodiment provides a method for generating targeted test cases based on a large language model and a symbolic execution tool. By designing a collaborative framework for a large model and a symbolic execution tool, a higher success rate for generating targeted test cases is achieved than the method of using a large model / symbolic execution tool alone, or the combined method of intersecting the outputs of the large model and the symbolic execution tool. Specifically, it includes the following two aspects: (1) Semantic reasoning stage: Based on the understanding ability of the large language model and combined with the external knowledge of the large language model, the test case requirements of the target program are understood and analyzed. Through the large language model, the code context semantics can be deeply analyzed, and the logical type, data structure characteristics and associated constraints of the input parameters can be intelligently inferred, effectively overcoming the type misjudgment and structural reasoning loss problems caused by the lack of semantic understanding in traditional symbolic execution technology, and avoiding the path enumeration explosion or constraint solution timeout caused by this. (2) Precise solution stage: The literal parameters obtained by reasoning are selectively symbolized. The symbolic execution tool can use the symbolic execution engine to dynamically collect the path constraint set along the target path, and perform precise mathematical deduction through the constraint solver to generate test case input values ​​that meet the boundary conditions. This process specifically addresses the inherent shortcomings of large language models in discrete mathematical computation and logical equation derivation, ensuring the numerical accuracy and logical completeness of literal reasoning. By combining semantic-level reasoning (large model capabilities) with mathematical-level solution (symbolic execution capabilities), this collaborative mechanism maintains the semantic understanding advantages of large models while inheriting the mathematical rigor of symbolic execution. Through the complementary enhancement of these two technologies, the ultimate goal of increasing the success rate is achieved.

[0033] This method overcomes the limitations of existing targeted test case generation approaches based solely on symbolic execution or large language models. It offers a method for generating targeted test cases that combines the advantages of large language models and symbolic execution. This approach, while leveraging the complementary technology of symbolic execution in scenarios where one technology is limited, delivers improved generation results, ultimately increasing the success rate of neat method generation. Compared to using symbolic execution or large language models alone, this approach offers greater compatibility and a higher success rate across different code scenarios.

[0034] This method combines symbolic execution tools with large language model tools. Symbolic execution tools can use different tools for different target programming languages. For example, LLVM-based KLEE can be used to generate test cases for software written in C, while tools such as SymbolicPathFinder can be used for software written in Java. Dynamic symbolic execution (also known as concolic execution) tools such as Triton, EvoSuite, and CrossHair can also be used.

[0035] Regarding large language models, GPT-3.5 or higher-performance models are most effective. CodeLlama, CodeQwen, and other large code models are also suitable. When using large code language models, setting the hyperparameter temperature to 0 to 0.2 yields good results. When retrying large language models, the number of retries can be set based on the temperature, typically between 2 and 10. Higher temperatures increase the number of retries.

[0036] To determine whether the test case has successfully reached the test target. You can use the corresponding software to write the code coverage detection tool of the programming language, such as gCov for C language and JaCoCo for Java language. The specific process is as follows Figure 2 shown.

[0037] Targeted test cases generated by the large model but failing to achieve the test target will be added to the next round's prompts. This inclusion can be achieved by appending a description of the test case's inclusion and exclusion semantics after the original prompt, such as "Test case X failed to reach the target" or "Test case X output was Y instead of Z." This prevents the large model from repeatedly generating incorrect answers and increases output diversity when temperature is set to 0. Note that this is accumulated in each round. For example, if m is 5, after five rounds of failed test case generation, the prompts can contain up to five exclusionary statements for the failed test cases.

[0038] After reaching the retry limit of the large model, the failed test case is used to generate the symbolic definition of the variable for the symbolic execution tool. For example, in KLEE, the abstract syntax tree of the test case is first analyzed, and the literals therein are marked and collected, and their type information and structure information (B) are collected. Type information includes integers, floating-point types, arrays, etc., and structure information includes array length, structure definition, etc. Then, these literals are symbolized. Definitions are performed through the interface klee_make_symbolic provided by KLEE. In other programming languages ​​and other tools, generation is performed according to the tool's instructions. In tools that do not support customized symbolic variable declarations, all parameters of the parameterized entry function are used by default. You can create a new function, use the parameters that need to be symbolically defined as parameters of the new function, and call the entry function that actually requires symbolic execution in the function body.

[0039] Example 2

[0040] The present application also relates to an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0041] The electronic device may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The processor may be a central processing unit (CPU), or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The memory may be used to store computer programs and / or modules, and the processor may perform various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory.

[0042] The relevant technical solutions are the same as above and will not be repeated here.

[0043] Example 3

[0044] The present application also relates to a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above method when the computer program is executed by a processor.

[0045] Specifically, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0046] The relevant technical solutions are the same as above and will not be repeated here.

[0047] Example 4

[0048] The present invention provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of the method of the above-described embodiment of the present invention.

[0049] The relevant technical solutions are the same as above and will not be repeated here.

[0050] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for generating targeted test cases based on a large language model and a symbolic execution tool, characterized in that: include: Based on the program to be tested P and the test target code branch T, generate the prompt word p of the large language model to generate the test case i through the large language model; Use test case i to test P. If T is covered, the generation of a test case is completed; Otherwise, the code coverage information of P in the current test is accumulated and added to the current prompt word p. The test case i is regenerated and tested. When the number of retry attempts reaches the preset retry limit m and T is still not covered, the m test cases i generated by the large language model are used as input templates. According to the usage requirements of the symbolic execution tool, the program literals in each input template are modified into symbolic variables of the symbolic execution tool. Each modified input template is used as the configuration information of the symbolic execution tool to construct a test entry function f of the symbolic execution tool. Each test entry function f is used as the driving function of the symbolic execution tool, P is used as the analysis object of the symbolic execution tool, and T is used as the analysis target of the symbolic execution tool. The symbolic execution tool is input synchronously to generate the corresponding test case j. Use test case j to test P, output the test case j corresponding to T covered in the test, and complete the generation of test cases.

2. The method for generating a directed test case according to claim 1, wherein: The value of m is set to 1 to 10.

3. The method for generating a directed test case according to claim 1, wherein: When the large language model is a code large language model, its hyperparameter temperature is set to 0-0.

2.

4. The method for generating a directed test case according to claim 1, wherein: The value of m is set according to the hyperparameter temperature of the large language model. The higher the temperature, the higher the value of m.

5. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

6. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed by a processor, the device where the storage medium is located is controlled to perform the steps of the method according to any one of claims 1 to 4.

7. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images