Artificial intelligence guidance generation program testing method combined with symbolic execution

Abstract

According to the artificial intelligence guidance generation program testing method combined with symbolic execution provided by the invention, the code automatically generated by artificial intelligence guidance is effectively tested by combining the strict reasoning ability of symbolic execution and the fuzzy reasoning ability of an artificial intelligence model; the artificial intelligence technology is combined and utilized to enhance and solve program path constraints collected by symbolic execution, and more high-quality and high-coverage test cases are generated, so that the test coverage rate and the correctness of test results are improved, program errors which are difficult to capture by a traditional test framework are found, the test coverage rate and the code error detection capability are improved, and the test efficiency is improved. And no matter whether the to-be-tested program has reference implementation and predefined output or not, the test method can be successfully implemented, developers can be helped to better utilize the program generated by artificial intelligence guidance, and the reliability of the program is improved.

Description

Technical Field

[0001] The present invention involves combining the use of symbolic execution technology to implement efficient testing on programs generated by artificial intelligence, thereby significantly improving test coverage and the ability to detect code defects, discovering software errors that are difficult to capture with traditional testing frameworks, and ensuring the correctness and reliability of generated code. The present invention is a method for testing programs generated by artificial intelligence combined with symbolic execution, and belongs to the field of computer program testing technology. Background Art

[0002] The rapid development of artificial intelligence, especially the emergence of deep learning technology and large language model technology, has effectively improved the quality of automatically generated programs, thereby significantly improving the development efficiency of developers. Therefore, more and more software developers are seamlessly integrating AI-guided automatically generated code into software projects to further optimize their development workflow.

[0003] For this reason, how to ensure the correctness of programs automatically generated by artificial intelligence has attracted the attention of academia and industry. Software testing is a common paradigm for ensuring software quality and has been widely studied in recent years. However, software testing usually relies on predefined test cases or manually written test scripts, which are not flexible enough when facing artificial intelligence-guided automatic generation of programs. In addition, the automatically generated programs often lack clear specifications and may also be highly complex and dynamic, making it difficult for traditional test cases to evaluate the correctness of the test results and cover all possible code paths and behaviors.

[0004] Existing related work tests the automatically generated code through fuzz testing, random testing and other techniques, but these methods face difficulties in achieving comprehensive path coverage because they rely on probability-based use case generators and lack rigorous reasoning, which cannot achieve high test coverage and therefore cannot better evaluate the correctness and effectiveness of the test results. Therefore, how to further ensure the correctness of the programs automatically generated by artificial intelligence and improve the test coverage remains a challenging problem.

[0005] The present invention proposes an artificial intelligence guided program generation testing method combined with symbolic execution. The method effectively tests the code automatically generated by artificial intelligence guidance by combining the strict reasoning capability of symbolic execution and the fuzzy reasoning capability of artificial intelligence models, so as to improve the test coverage and the correctness of the test results, thus making up for the shortcomings of the prior art.

[0006] Symbolic execution is a rigorous white-box testing technique that generates tests for deep execution paths by solving the path constraints in the program, thereby achieving better test coverage. For the symbolic execution engine, it can generate test inputs by solving the path constraints to discover errors in the program. Its symbolic execution process consists of two main components: the symbolic constraint collection module and the satisfiability modulo theory solver module. The symbolic constraint collection module traverses the execution paths in the program according to the symbolic execution strategy, collects the loop and branch conditions encountered along each path, reverses the conditions that evaluate to False, and generates a conjunctive normal form for the conditional groups that evaluate to True. This process propagates constants and variables to construct constraints. Subsequently, the satisfiability modulo theory solver module solves these constraints using the satisfiability modulo theory to generate test inputs for the current path.

[0007] The present invention systematically derives the path constraints of programs automatically generated by artificial intelligence through symbolic execution, and assigns the task of solving these constraints to the artificial intelligence model, thereby making up for the shortcomings of existing symbolic execution technology when faced with complex or unsolvable constraints, generating a large number of complementary boundary test cases, and being able to discover difficult-to-capture program errors that are challenging to current software testing frameworks. Summary of the invention

[0008] Technical problem: The purpose of the present invention is to provide an AI-guided program generation testing method combined with symbolic execution, which can generate a reliable set of test cases regardless of whether the AI-guided generated program has a benchmark implementation and predefined output, while solving the problem of combining strict reasoning and fuzzy reasoning, improving test coverage, ensuring the correctness of the generated code, and discovering software errors that are difficult to capture with traditional testing frameworks.

[0009] Technical solution: The present invention proposes a test method for program generation guided by artificial intelligence combined with symbolic execution. The code automatically generated by artificial intelligence is effectively tested by combining the strict reasoning ability of symbolic execution and the fuzzy reasoning ability of artificial intelligence model. The program path constraints collected by symbolic execution are enhanced and solved by artificial intelligence technology to generate more high-quality and high-coverage test cases, thereby improving the test coverage and the correctness of the test results. Regardless of whether the program to be tested has a benchmark implementation and predefined output, this test method can be successfully implemented. The specific steps are as follows:

[0010] Step 1-1. Introduce symbolic execution technology into the test framework of AI-guided program generation, effectively combine the strict reasoning ability of symbolic execution with the fuzzy reasoning ability of AI models, build a complete test process, automatically generate input and output of test cases, and perform high-coverage functional testing;

[0011] Step 1-2. For the test framework of step 1-1, the program path constraints collected by symbolic execution are enhanced and solved by combining artificial intelligence technology, so as to make up for the shortcomings of symbolic execution technology when facing complex or unsolvable constraints and generate a large number of complementary boundary test cases;

[0012] Step 1-3. When implementing the test based on step 1-1, when there is a lack of benchmark implementation and predefined output, the large language model is used to predict the test standard output, thereby realizing automated functional testing and obtaining accurate results; regardless of whether the program to be tested has a benchmark implementation and predefined output, this method can obtain accurate test results.

[0013] in,

[0014] The specific steps in step 1-1 of the test framework are as follows:

[0015] Step 2-1. Generate the program code to be tested using the artificial intelligence model based on the natural language description and function template provided by the user, wherein the natural language description outlines the problem to be solved and specifies the expected functionality of the code, and the function template defines the format of the code, including the function name, parameter types and names, and any necessary libraries and packages for different programming languages;

[0016] Step 2-2. Use the artificial intelligence model to generate the seed input I in the fuzz testing method based on the natural language description and the test input template seed , where the natural language description describes the problem to be solved, and the test input template defines the format of the test input, including the type and name of the input, as well as the value range of the input;

[0017] Step 2-3. Apply fuzz testing methods, using the seed input generated by the AI ​​model to generate more test inputs by mutating the case. fuzz , merge the seed input with the test input generated by mutation as the initial test input set, denoted as I Original , then:

[0018] I Original =I seed ∪I fuzz

[0019] Step 2-4: Input the program to be tested generated by the above artificial intelligence model into the symbolic execution engine, set symbolic constraints, collect constraints, and use the symbolic execution engine and artificial intelligence model to solve them respectively to obtain the corresponding test input set I SMTSolved and I AISolved , and compare them with the initial test input set I Original Combined as the test input set for functional testing, denoted as I, we have:

[0020] I=IOriginal ∪I SMTSolved ∪I AISolved

[0021] Step 2-5. If the program under test has a benchmark implementation or predefined output, use it as the corresponding expected output. If it does not have a benchmark implementation or predefined output, use the large language model to generate the expected output and replace the test input {in 1 ,in 2 ,…} and their corresponding expected outputs are organized into test case pairs and combined into a test case set T, as follows:

[0022] T={(in 1 ,exp_out 1 ),(in 2 ,exp_out 2 ),…}

[0023] Step 2-6. Perform functional testing on the automatically generated program to be tested through the test case set T to verify whether the test coverage reaches the expected level. At the same time, check whether the program to be tested can pass all test cases correctly, so as to verify the correctness of the program automatically generated by artificial intelligence guidance.

[0024] The specific steps of steps 1-2 are as follows:

[0025] Step 3-1. Design a symbolic constraint template to guide the symbolic execution engine to perform accurate constraint analysis, including the format of the use case, symbolic declaration, constraint preconditions, and other adaptation information of the engine;

[0026] Step 3-2. Set symbolic constraints, apply the symbolic constraint template to the program under test, use the program under test as the input of the symbolic execution engine, and the symbolic constraint collection module in the symbolic execution engine traverses the execution path in the program according to the symbolic execution strategy, collects the loop and branch conditions encountered along each path, reverses the conditions evaluated to False, and generates a conjunctive normal form for the condition groups evaluated to True, propagates constants and variables to construct constraints, that is, the path of the jth path j The corresponding constraints are constraints(path j );

[0027] Step 3-3. For solvable constraints, the satisfiability modulo theory solver module SMTSolver in the symbolic execution engine uses the satisfiability modulo theory to solve these constraints and generate the test input of the current path as follows:

[0028] SMTSolve(constraints(path j ))=in j

[0029] Among them, j Corresponding to the execution path path j SMTSolve is a satisfiability modulo theory solution method. The test input set I SMTSolved consists of all the solvable inputs generated during this process;

[0030] Step 3-4. For constraints that are difficult to solve, by providing requirements to the AI ​​model, that is, requesting to solve the following path constraints in symbolic execution, and then pasting the symbolic constraints collected from the symbolic execution engine, the test input for the corresponding path can be generated, as follows:

[0031] AISolve(constraints(path j ))=supp_in j

[0032] Among them, supp_in j Corresponding to the execution path path j The test input generated by the artificial intelligence model, AISolve is an enhanced constraint solving method for artificial intelligence models, and the test input set I AISolved Consists of all test inputs generated during this process.

[0033] The specific steps of steps 1-3 are as follows:

[0034] Step 4-1. In the case where the program to be tested has a benchmark implementation or predefined output, the test input set is input into the program of the benchmark implementation, and its corresponding output is used as the corresponding expected output in the functional test;

[0035] Step 4-2. In the case where the program under test lacks a benchmark implementation and the test input has no predefined output, use the large language model to generate the expected output, and write the natural language description desc and the corresponding test input in i ∈I is input to the large language model, so that the large language model can determine the expected output. After multiple iterations, the most frequently occurring value is selected as the expected output, as follows:

[0036] LLMCompute(desc,in j )=out i1 ,out i2 ,…

[0037] exp_out i =max_freq(out i1 ,out i2 ,…)

[0038] Among them, LLMCompute is a method for calculating the expected output of a large language model, max_freq is a method for selecting the most frequently occurring value, and exp_out i For the test input j The corresponding expected output;

[0039] Step 4-3. Add the test input {in 1 ,in 2 ,…} and their corresponding expected outputs are organized into test case pairs and combined into a test case set T, as follows:

[0040] T={(in 1 ,exp_out 1 ),(in 2 ,exp_out 2 ),…}

[0041] The test case set T can be used to perform functional testing on the program to be tested and obtain accurate test results.

[0042] Beneficial effects: The artificial intelligence-guided program generation test method combined with symbolic execution described in the present invention can effectively improve the test coverage by using symbolic execution, ensure the correctness of the generated code, generate a large number of complementary boundary test cases while reducing manual intervention, and discover program errors that are difficult to capture in traditional test frameworks. Specifically, it has the following beneficial effects:

[0043] 1. Compared with the existing artificial intelligence-guided program testing technology, the method of the present invention combines strict reasoning and fuzzy reasoning processes through symbolic execution, so that it can generate tests for deep execution paths, covering program execution paths that are difficult to reach with existing methods, and obtaining higher test coverage.

[0044] 2. The present invention utilizes artificial intelligence models to effectively enhance the constraint solving capabilities of symbolic execution, thereby effectively solving complex constraints that are difficult to handle with existing methods, generating more high-quality, high-coverage test cases, and significantly improving the comprehensiveness and accuracy of the test.

[0045] 3. The method of the present invention can automatically generate expected outputs in the absence of benchmark implementations and predefined outputs, obtain accurate test results, reduce manual intervention, and improve the degree of automation of the test process.

[0046] 4. The method of the present invention can generate a large number of complementary boundary test cases, discover program errors that are difficult to capture with traditional testing frameworks, and significantly improve the reliability of programs automatically generated by artificial intelligence guidance. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1The overall flow of the program testing method for artificial intelligence guided generation combined with symbolic execution.

[0048] Figure 2 Flow of a testing framework for introducing AI-guided program generation combined with symbolic execution.

[0049] Figure 3 A process for enhancing the solution of program path constraints collected by symbolic execution by combining artificial intelligence technology.

[0050] Figure 4 The present invention is a process for obtaining accurate test results regardless of whether the program under test has a benchmark implementation and predefined outputs. DETAILED DESCRIPTION

[0051] The present invention proposes a testing method for AI-guided program generation combined with symbolic execution. By introducing symbolic execution technology into the testing framework of AI-guided program generation, the strict reasoning ability of symbolic execution and the fuzzy reasoning ability of AI models are effectively combined, and the programs generated under the guidance of AI models are tested, which significantly improves the test coverage and defect detection rate.

[0052] Step 1-1. Introduce symbolic execution technology into the test framework of AI-guided program generation, effectively combine the strict reasoning ability of symbolic execution with the fuzzy reasoning ability of AI models, build a complete test process, automatically generate input and output of test cases, and perform high-coverage functional testing;

[0053] Step 1-2. For the test framework of step 1-1, the program path constraints collected by symbolic execution are enhanced and solved by combining artificial intelligence technology, so as to make up for the shortcomings of symbolic execution technology when facing complex or unsolvable constraints and generate a large number of complementary boundary test cases;

[0054] Step 1-3. When implementing the test based on step 1-1, when there is a lack of benchmark implementation and predefined output, the large language model is used to predict the test standard output, thereby realizing automated functional testing and obtaining accurate results; regardless of whether the program to be tested has a benchmark implementation and predefined output, this method can obtain accurate test results.

[0055] The following sections further describe some specific details of the implementation process:

[0056] 1. Strategy of Introducing Symbolic Execution Technology into the Testing Framework of AI-Guided Generative Programs

[0057] Step 2-1. Generate the program code to be tested using the artificial intelligence model based on the natural language description and function template provided by the user, wherein the natural language description outlines the problem to be solved and specifies the expected functionality of the code, and the function template defines the format of the code, including the function name, parameter types and names, and any necessary libraries and packages for different programming languages;

[0058] Step 2-2. Use the artificial intelligence model to generate the seed input I in the fuzz testing method based on the natural language description and the test input template seed , where the natural language description describes the problem to be solved, and the test input template defines the format of the test input, including the type and name of the input, as well as the value range of the input;

[0059] Step 2-3. Apply fuzz testing methods, using the seed input generated by the AI ​​model to generate more test inputs by mutating the case. fuzz , merge the seed input with the test input generated by mutation as the initial test input set, denoted as I Original , then:

[0060] I Original =I seed ∪I fuzz

[0061] Step 2-4: Input the program to be tested generated by the above artificial intelligence model into the symbolic execution engine, set symbolic constraints, collect constraints, and use the symbolic execution engine and artificial intelligence model to solve them respectively to obtain the corresponding test input set I SMTSolved and I AISolved , and compare them with the initial test input set I Original Combined as the test input set for functional testing, denoted as I, we have:

[0062] I=I Original ∪I SMTSolved ∪I AISolved

[0063] Step 2-5. If the program under test has a benchmark implementation or predefined output, use it as the corresponding expected output. If it does not have a benchmark implementation or predefined output, use the large language model to generate the expected output and replace the test input {in 1 ,in 2 ,…} and their corresponding expected outputs are organized into test case pairs and combined into a test case set T, as follows:

[0064] T={(in 1 ,exp_out 1 ),(in 2 ,exp_out2 ),…}

[0065] Step 2-6. Perform functional testing on the automatically generated program to be tested through the test case set T to verify whether the test coverage reaches the expected level. At the same time, check whether the program to be tested can pass all test cases correctly, so as to verify the correctness of the program automatically generated by artificial intelligence guidance.

[0066] Introducing symbolic execution technology into the test framework of AI-guided program generation can make full use of the strict reasoning ability of symbolic execution and the fuzzy reasoning ability of AI models to achieve efficient test case generation. In the existing AI-guided program generation test framework, test cases are often generated based on the fuzzy testing method, which is an efficient testing method that generates inputs by randomly mutating seed inputs to find errors in the program. However, AI-guided programs may have extremely complex input spaces, which makes it difficult for inputs generated by random mutations in fuzz testing to cover all possible scenarios, resulting in low test coverage. Therefore, by introducing symbolic execution technology into the test framework, test cases for deep execution paths can be generated by solving path constraints in the program, thereby achieving better test coverage.

[0067] The symbolic execution engine first traverses all possible execution paths of the program. During the traversal process, the symbolic execution engine records the conditional branches and loop conditions encountered on each path. For each execution path, the symbolic execution engine collects the constraints of all conditional branches and loop conditions on the path. These constraints are usually expressed in the form of logical expressions, describing the value ranges and relationships of variables on the path. The symbolic execution engine reverses the evaluation of the conditions on the path. For example, for a condition if(x>0), the symbolic execution engine generates two constraints: x>0 and x<=0, corresponding to the true and false conditions of the condition, respectively. The symbolic execution engine combines all the constraints on the path into a conjunctive normal form (CNF), which is the logical AND of all constraints. This step ensures that all conditions on the path are taken into account. The symbolic execution engine inputs the generated conjunctive normal form into the satisfiability module theory solver (SMT Solver). The solver attempts to solve these constraints and find variable values ​​that satisfy all constraints. Once the solver finds solutions that satisfy the constraints, the symbolic execution engine uses these solutions as test inputs. Each test input corresponds to a specific execution path, ensuring that different paths of the program are tested.

[0068] 2. Strategy for enhancing the solution of program path constraints by combining artificial intelligence technology

[0069] Step 3-1. Design a symbolic constraint template to guide the symbolic execution engine to perform accurate constraint analysis, including the format of the use case, symbolic declaration, constraint preconditions, and other adaptation information of the engine;

[0070] Step 3-2. Set symbolic constraints, apply the symbolic constraint template to the program under test, use the program under test as the input of the symbolic execution engine, and the symbolic constraint collection module in the symbolic execution engine traverses the execution path in the program according to the symbolic execution strategy, collects the loop and branch conditions encountered along each path, reverses the conditions evaluated to False, and generates a conjunctive normal form for the condition groups evaluated to True, propagates constants and variables to construct constraints, that is, the path of the jth path j The corresponding constraints are constraints(path j );

[0071] Step 3-3. For solvable constraints, the satisfiability modulo theory solver module SMTSolver in the symbolic execution engine uses the satisfiability modulo theory to solve these constraints and generate the test input of the current path as follows:

[0072] SMTSolve(constraints(path j ))=in j

[0073] Among them, j Corresponding to the execution path path j SMTSolve is a satisfiability modulo theory solution method. The test input set I SMTSolved consists of all the solvable inputs generated during this process;

[0074] Step 3-4. For constraints that are difficult to solve, by providing requirements to the AI ​​model, that is, requesting to solve the following path constraints in symbolic execution, and then pasting the symbolic constraints collected from the symbolic execution engine, the test input for the corresponding path can be generated, as follows:

[0075] AISolve(constraints(path j ))=supp_in j

[0076] Among them, supp_in j Corresponding to the execution path path j The test input generated by the artificial intelligence model, AISolve is an enhanced constraint solving method for artificial intelligence models, and the test input set I AISolved Consists of all test inputs generated during this process.

[0077] During symbolic execution, the constraints of program paths can sometimes become extremely complex or even impossible to solve. This complexity may stem from the complex logical structure of the program itself or the special properties of a specific path, making it difficult for traditional symbolic execution engines to generate corresponding test inputs, resulting in insufficient test coverage. To address this challenge, we introduced artificial intelligence technology to enhance the ability to solve these complex constraints.

[0078] Specifically, when the symbolic execution engine encounters constraints that are difficult to solve, we hand them over to the AI ​​model for processing. First, the AI ​​model learns the patterns and characteristics of different types of constraints by analyzing a large amount of historical data. Then, it uses this learned knowledge to try to infer input values ​​that may satisfy the constraints.

[0079] In this process, the AI ​​model uses advanced techniques such as generative adversarial networks (GAN) or reinforcement learning to continuously optimize its output by generating and evaluating a large number of candidate inputs. These generated inputs not only help solve specific complex constraints, but also explore uncovered paths in the program. In this way, the AI ​​model is able to generate a large number of boundary test cases covering a variety of edge cases.

[0080] In addition, we adopt a hybrid strategy to combine AI-generated inputs with inputs obtained from symbolic execution to form a more comprehensive test set, which ensures that test coverage is significantly improved even when facing extremely complex or unsolvable constraints.

[0081] Through this strategy, the combination of artificial intelligence and symbolic execution can make up for their respective shortcomings, so that rich and effective test inputs can be generated even when faced with complex or unsolvable constraints, thereby comprehensively improving the quality and efficiency of testing.

[0082] Strategies for Generating Expected Output Using Large Language Models

[0083] Step 4-1. In the case where the program to be tested has a benchmark implementation or predefined output, the test input set is input into the program of the benchmark implementation, and its corresponding output is used as the corresponding expected output in the functional test;

[0084] Step 4-2. In the case where the program under test lacks a benchmark implementation and the test input has no predefined output, use the large language model to generate the expected output, and write the natural language description desc and the corresponding test input in i ∈I is input to the large language model, so that the large language model can determine the expected output. After multiple iterations, the most frequently occurring value is selected as the expected output, as follows:

[0085] LLMCompute(desc,in j)=out i1 ,out i2 ,…

[0086] exp_out i =max_freq(out i1 ,out i2 ,…)

[0087] Among them, LLMCompute is a method for calculating the expected output of a large language model, max_freq is a method for selecting the most frequently occurring value, and exp_out i For the test input j The corresponding expected output;

[0088] Step 4-3. Add the test input {in 1 ,in 2 ,…} and their corresponding expected outputs are organized into test case pairs and combined into a test case set T, as follows:

[0089] T={(in 1 ,exp_out 1 ),(in 2 ,exp_out 2 ),…}

[0090] The test case set T can be used to perform functional testing on the program to be tested and obtain accurate test results.

[0091] In functional testing, verifying the correctness of a program requires comparing the actual output with the expected output. However, for programs without a benchmark implementation or predefined output, determining the expected output is a major challenge. To address this problem, we can leverage large language models to generate expected outputs from natural language descriptions and test inputs.

[0092] Specifically, when there is a lack of benchmark implementations, we provide detailed natural language descriptions and specific test inputs to the large language model. The model understands the complex relationship between the description and the input through in-depth learning of large-scale text data, and thus infers possible output results. In this process, the model not only considers the surface information of the input, but also analyzes its deep semantics and related background knowledge, making the generated output more accurate and reasonable.

[0093] In the process of generating expected output, the large language model generates multiple candidate outputs based on the input. We assign a confidence score to each candidate output and evaluate and optimize these outputs through multiple iterations. Finally, the value with the highest frequency or the highest confidence is selected as the final expected output. This method can effectively handle complex input-output relationships and ensure that the generated expected output meets the expected function of the program.

[0094] By using a large language model to generate expected outputs, accurate test results can be obtained even without a benchmark. This approach not only improves the automation of testing, but also flexibly handles complex functions and ambiguous input-output relationships. It provides greater flexibility and reliability for the testing process, allowing developers to verify the correctness of the program in a shorter time and quickly adapt to changing needs.

Claims

1. A method for testing program generated by artificial intelligence combined with symbolic execution, characterized in that By combining the strict reasoning ability of symbolic execution and the fuzzy reasoning ability of artificial intelligence models, the code generated by artificial intelligence-guided automation is effectively tested. In combination with the use of artificial intelligence technology, the program path constraints collected by symbolic execution are enhanced to solve more high-quality and high-coverage test cases, thereby improving the test coverage and the correctness of the test results. Regardless of whether the program to be tested has a benchmark implementation and predefined output, this test method can be successfully implemented. The specific steps are: Step 1-1. Introduce symbolic execution technology into the test framework of AI-guided program generation, effectively combine the strict reasoning ability of symbolic execution with the fuzzy reasoning ability of AI models, build a complete test process, automatically generate input and output of test cases, and perform high-coverage functional testing; Step 1-2. For the test framework of step 1-1, the program path constraints collected by symbolic execution are enhanced and solved by combining artificial intelligence technology, so as to make up for the shortcomings of symbolic execution technology when facing complex or unsolvable constraints and generate a large number of complementary boundary test cases; Step 1-3. When implementing the test based on step 1-1, when there is a lack of benchmark implementation and predefined output, the large language model is used to predict the test standard output, thereby realizing automated functional testing and obtaining accurate results; regardless of whether the program to be tested has a benchmark implementation and predefined output, this method can obtain accurate test results.

2. The method for testing program generated by artificial intelligence combined with symbolic execution according to claim 1, characterized in that The specific steps in step 1-1 of the test framework are as follows: Step 2-1. Generate the program code to be tested using the artificial intelligence model based on the natural language description and function template provided by the user, wherein the natural language description outlines the problem to be solved and specifies the expected functionality of the code, and the function template defines the format of the code, including the function name, parameter types and names, and any necessary libraries and packages for different programming languages; Step 2-2. Use the artificial intelligence model to generate the seed input I in the fuzz testing method based on the natural language description and the test input template seed , where the natural language description describes the problem to be solved, and the test input template defines the format of the test input, including the type and name of the input, as well as the value range of the input; Step 2-3. Apply fuzz testing methods, using the seed input generated by the AI ​​model to generate more test inputs by mutating the case. fuzz , merge the seed input with the test input generated by mutation as the initial test input set, denoted as I Original , then: I Original =I seed ∪I fuzz Step 2-4: Input the program to be tested generated by the above artificial intelligence model into the symbolic execution engine, set symbolic constraints, collect constraints, and use the symbolic execution engine and artificial intelligence model to solve them respectively to obtain the corresponding test input set I SMTSolved and I AISolved , and compare them with the initial test input set I Original Combined as the test input set for functional testing, denoted as I, we have: I=I Original ∪I SMTSolved ∪I AISolved Step 2-5. If the program under test has a benchmark implementation or predefined output, use it as the corresponding expected output. If it does not have a benchmark implementation or predefined output, use the large language model to generate the expected output. Organize the test inputs {in1, in2, ...} in the test input set I of the functional test and their corresponding expected outputs into test case pairs, and combine them into a test case set T, as follows: T={(in1,exp_out1),(in2,exp_out2),…} Step 2-6. Perform functional testing on the automatically generated program to be tested through the test case set T to verify whether the test coverage reaches the expected level. At the same time, check whether the program to be tested can pass all test cases correctly, so as to verify the correctness of the program automatically generated by artificial intelligence guidance.

3. The method for testing program generated by artificial intelligence combined with symbolic execution according to claim 1, characterized in that The specific steps of steps 1-2 are as follows: Step 3-1. Design a symbolic constraint template to guide the symbolic execution engine to perform accurate constraint analysis, including the format of the use case, symbolic declaration, constraint preconditions, and other adaptation information of the engine; Step 3-2. Set symbolic constraints, apply the symbolic constraint template to the program under test, use the program under test as the input of the symbolic execution engine, and the symbolic constraint collection module in the symbolic execution engine traverses the execution path in the program according to the symbolic execution strategy, collects the loop and branch conditions encountered along each path, reverses the conditions evaluated to False, and generates a conjunctive normal form for the condition groups evaluated to True, propagates constants and variables to construct constraints, that is, the path of the jth path j The corresponding constraints are constraints(path j ); Step 3-3. For solvable constraints, the SMT Solver module in the symbolic execution engine uses the satisfiability modulo theory to solve these constraints and generate the test input of the current path as follows: SMTSolve(constraints(path j ))=in j Among them, j Corresponding to the execution path path j SMTSolve is a satisfiability modulo theory solution method. The test input set I SMTSolved consists of all the solvable inputs generated during this process; Step 3-4. For constraints that are difficult to solve, by providing requirements to the AI ​​model, that is, requesting to solve the following path constraints in symbolic execution, and then pasting the symbolic constraints collected from the symbolic execution engine, the test input for the corresponding path can be generated, as follows: AISolve(constraints(path j ))=supp_in j Among them, supp_in j Corresponding to the execution path path j The test input generated by the artificial intelligence model, AISolve is an enhanced constraint solving method for artificial intelligence models, and the test input set I AISolved Consists of all test inputs generated during this process.

4. The method for testing program generated by artificial intelligence combined with symbolic execution according to claim 1, characterized in that The specific steps of steps 1-3 are as follows: Step 4-1. In the case where the program to be tested has a benchmark implementation or predefined output, the test input set is input into the program of the benchmark implementation, and its corresponding output is used as the corresponding expected output in the functional test; Step 4-2. In the case where the program under test lacks a benchmark implementation and the test input has no predefined output, use the large language model to generate the expected output, and write the natural language description desc and the corresponding test input in i ∈I is input to the large language model, so that the large language model can determine the expected output. After multiple iterations, the most frequently occurring value is selected as the expected output, as follows: LLMCompute(desc,in j )=out i1 ,out i2 ,… exp_out i =max_freq(out i1 ,out i2 ,…) Among them, LLMCompute is a method for calculating the expected output of a large language model, max_freq is a method for selecting the most frequently occurring value, and exp_out i For the test input j The corresponding expected output; Step 4-3. Organize the test inputs {in1, in2, ...} and their corresponding expected outputs in the test input set I obtained in step 2-4 into test case pairs and combine them into a test case set T, as follows: T={(in1,exp_out1),(in2,exp_out2),…} The test case set T can be used to perform functional testing on the program to be tested and obtain accurate test results.

Images