An artificial intelligence guided generation program testing method combined with symbolic execution

Abstract

The application provides an artificial intelligence guidance generation program test method combined with symbolic execution, effectively tests the code automatically generated by the artificial intelligence guidance by combining the strict reasoning ability of the symbolic execution and the fuzzy reasoning ability of the artificial intelligence model, enhances the solving of the program path constraints collected by the symbolic execution by using the artificial intelligence technology, generates more high-quality and high-coverage test cases, thereby improving the test coverage and the correctness of the test results, discovering program errors that are difficult to capture by a traditional test framework, improving the test coverage and the ability of detecting code errors, and the test method can be successfully implemented regardless of whether the to-be-tested program has a benchmark implementation and a predefined output, and can help developers better utilize the program generated by the artificial intelligence guidance and improve the reliability of the program.

Description

TECHNICAL FIELD

[0001] The present application relates to a method for testing programs generated by artificial intelligence guidance using symbolic execution techniques, which significantly improves test coverage and the detection of code defects, discovers software errors that traditional testing frameworks cannot capture, and ensures the correctness and reliability of generated code. BACKGROUND

[0002] The rapid development of artificial intelligence, especially the emergence of deep learning technology and large language model technology, has effectively improved the quality of automatically generated programs, significantly improving the development efficiency of developers. Therefore, more and more software developers seamlessly integrate the code generated by artificial intelligence guidance into software projects to further optimize their development workflow.

[0003] Therefore, how to ensure the correctness of programs automatically generated by artificial intelligence guidance has attracted attention from academia and industry. Software testing is a common paradigm for ensuring software quality and has been extensively studied in academia in recent years. However, software testing usually relies on pre-defined test cases or manually written test scripts, which is not flexible enough when facing automatically generated programs by artificial intelligence guidance. Moreover, automatically generated programs often lack explicit specifications and may have high complexity and dynamics, making it difficult for traditional test cases to evaluate the correctness of test results and cover all possible code paths and behaviors.

[0004] Existing related work tests automatically generated code through techniques such as fuzz testing and random testing, but these methods face difficulties in achieving comprehensive path coverage because they rely on probability-based test case generators and lack strict reasoning, making it difficult to achieve high test coverage and evaluate the correctness and effectiveness of test results. Therefore, how to further ensure the correctness of programs automatically generated by artificial intelligence guidance and improve test coverage remains a challenging problem.

[0005] The present application proposes a method for testing programs generated by artificial intelligence guidance using symbolic execution, which effectively tests code automatically generated by artificial intelligence guidance by combining the strict reasoning ability of symbolic execution and the fuzzy reasoning ability of artificial intelligence models to improve test coverage and the correctness of test results, addressing the shortcomings of existing technologies.

[0006] Symbolic execution is a rigorous white-box testing technique that generates tests for deep execution paths by solving path constraints in the program, thus achieving better test coverage. For a symbolic execution engine, it can generate test inputs by solving path constraints to discover errors in the program. Its symbolic execution process consists of two main components: a symbolic constraint collection module and a satisfiability modulo theories solver module. The symbolic constraint collection module traverses the execution paths in the program according to the symbolic execution strategy, collects the loop and branch conditions encountered along each path, inverts the conditions that evaluate to False, and groups the conditions that evaluate to True into a conjunctive normal form. This process propagates constants and variables to construct constraints. Subsequently, the satisfiability modulo theories solver module solves these constraints using satisfiability modulo theories to generate test inputs for the current path.

[0007] The present invention systematically derives path constraints of the program generated by artificial intelligence guided automation using symbolic execution, and hands over the task of solving these constraints to an artificial intelligence model, thus making up for the shortcomings of existing symbolic execution techniques in the face of complex or unsolvable constraints, generating a large number of complementary boundary test cases, and being able to discover difficult-to-catch program errors that are challenging to current software testing frameworks. SUMMARY

[0008] Technical problem: The purpose of the present invention is to provide an artificial intelligence guided generation of program testing method combined with symbolic execution, which can generate reliable test case sets regardless of whether the artificial intelligence guided generated program has a benchmark implementation and predefined output, while solving the problem of needing to combine rigorous reasoning and fuzzy reasoning, improving test coverage, ensuring the correctness of the generated code, and discovering software errors that are difficult to catch by traditional testing frameworks.

[0009] Technical solution: The present invention proposes an artificial intelligence guided generation of program testing method combined with symbolic execution, which effectively tests the code automatically generated by artificial intelligence guidance by combining the rigorous reasoning ability of symbolic execution and the fuzzy reasoning ability of artificial intelligence models, and enhances the solving of program path constraints collected by symbolic execution using artificial intelligence technology to generate more high-quality and high-coverage test cases, thus improving test coverage and the correctness of test results, and regardless of whether the program to be tested has a benchmark implementation and predefined output, this testing method can be successfully implemented, the specific steps are as follows:

[0010] Step 1-1. Introduce symbolic execution technology into the testing framework of the program generated by artificial intelligence guidance, effectively combine the rigorous reasoning ability of symbolic execution and the fuzzy reasoning ability of artificial intelligence models, build a complete testing process, automatically generate test case inputs and outputs, and perform high-coverage functional testing;

[0011] Step 1-2. For the test framework of step 1-1, the program path constraints collected by symbolic execution are enhanced for solving by using artificial intelligence technology, which makes up for the shortcomings of symbolic execution technology when facing complex or unsolvable constraints, and generates a large number of complementary boundary test cases;

[0012] Step 1-3. When the reference implementation and predefined output are missing during the implementation of the test based on step 1-1, the test standard output is predicted by the large language model, so as to realize the automatic functional test and obtain accurate results; Whether the to-be-tested program has a reference implementation and predefined output, this method can obtain accurate test results.

[0013] Wherein,

[0014] The specific steps of step 1-1 in the test framework are as follows:

[0015] Step 2-1. Use an artificial intelligence model to generate a to-be-tested program code according to a natural language description and a function template provided by a user, wherein the natural language description outlines a problem to be solved and specifies an expected function of the code, and the function template defines a format of the code, including a function name, parameter types and names, and any necessary libraries and packages of different programming languages;

[0016] Step 2-2. Use an artificial intelligence model to generate a seed input I seed in a fuzz testing method according to a natural language description and a test input template, wherein the natural language description describes a problem to be solved, and the test input template defines a format of the test input, including types and names of the input, and a value range of the input;

[0017] Step 2-3. Apply the fuzz testing method to generate more test inputs I fuzz by using the seed input generated by the artificial intelligence model in a mutation case method, and combine the seed input and the test inputs generated by mutation to serve as an initial test input set, denoted as I Original , and have:

[0018] I Original = I seed ∪ I fuzz

[0019] Step 2-4. Input the to-be-tested program generated by the above artificial intelligence model into a symbolic execution engine, set a symbolic constraint, collect the constraint, and use the symbolic execution engine and the artificial intelligence model to solve, respectively, to obtain corresponding test input sets I SMTSolved and I AISolved , and combine the two with the initial test input set I Original to serve as a test input set for functional testing, denoted as I, and have:

[0020] I = IOriginal ∪I SMTSolved ∪I AISolved

[0021] Step 2-5. If the to-be-tested program has a reference implementation or predefined output, use it as the corresponding expected output. If it does not have a reference implementation or predefined output, use a large language model to generate the expected output. Organize the test inputs {in1, in2, …} in the test input set I of the functional test and the expected outputs corresponding thereto into test case pairs, and combine them into a test case set T, as follows:

[0022] T = {(in1, exp_out1), (in2, exp_out2), …}

[0023] Step 2-6. Perform functional testing on the automatically generated to-be-tested program through the test case set T, check whether the test coverage reaches the expectation, and verify whether the to-be-tested program can correctly pass all test cases, so as to verify the correctness of the program guided by artificial intelligence.

[0024] The step 1-2 is specifically as follows:

[0025] Step 3-1. Design a symbolic constraint template to guide the symbolic execution engine to perform accurate constraint analysis, including the format of the use case, the declaration of the symbol, the premise condition of the constraint, and other adaptation information of the engine;

[0026] Step 3-2. Set the symbolic constraint, apply the symbolic constraint template to the to-be-tested program, and use the to-be-tested program as the input of the symbolic execution engine. The symbolic constraint collection module in the symbolic execution engine traverses the execution paths in the program according to the symbolic execution strategy, collects the loop and branch conditions encountered along each path, reverses the conditions evaluated as False, and generates a conjunction normal form for the conditions evaluated as True, and propagates constants and variables to construct constraints, that is, the jth path path j The corresponding constraint is constraints(path j );

[0027] Step 3-3. For solvable constraints, the satisfiability modulo theories solver module SMTSolver in the symbolic execution engine solves these constraints using satisfiability modulo theories, to generate the test input of the current path, as follows:

[0028] SMTSolve(constraints(path j )) = in j

[0029] in j is the test input corresponding to the execution path path jSMTSolve is a satisfiability modulo theories solving method, and I is the set of test inputs SMTSolved consists of all the solvable inputs generated in this process;

[0030] Step 3-4. For the constraints that are difficult to solve, by providing the requirements to the artificial intelligence model, that is, requesting the following path constraints in symbolic execution, and then pasting the symbolic constraints collected from the symbolic execution engine, that is, the test input corresponding to the path can be generated, as follows:

[0031] AISolve(constraints(path j ))=supp_in j

[0032] Where supp_in j is the test input generated by the artificial intelligence model corresponding to the execution path path j , and AISolve is an artificial intelligence model enhanced solving constraint method, and I is the set of test inputs AISolved consists of all the test inputs generated in this process.

[0033] The specific steps of the above-mentioned step 1-3 are as follows:

[0034] Step 4-1. In the case where the program to be tested has a benchmark implementation or a predefined output, input the test input set into the program of the benchmark implementation, and take the corresponding output as the expected output in the functional test;

[0035] Step 4-2. In the case where the program to be tested lacks a benchmark implementation and the test input has no predefined output, use a large language model to generate an expected output, input the natural language description desc and the corresponding test input in i ∈I to the large language model, so that the large language model determines the expected output, and through multiple iterations, the most frequently occurring value is selected as the expected output, as follows:

[0036] LLMCompute(desc,in j )=out i1 ,out i2 ,…

[0037] exp_out i =max_freq(out i1 ,out i2 ,…)

[0038] Where LLMCompute is a large language model method for calculating the expected output, max_freq is a method for selecting the most frequently occurring value, and exp_out i is the test input inj corresponding expected output;

[0039] Step 4-3. The test inputs {in1, in2, …} in the test input set I obtained in step 2-4 and the corresponding expected output are organized into test case pairs and combined into a test case set T as follows:

[0040] T = {(in1, exp_out1), (in2, exp_out2), …}

[0041] The function test of the program to be tested can be performed through the test case set T, and accurate test results can be obtained.

[0042] Beneficial effects: The artificial intelligence guided program test method combined with symbolic execution described in the present application can effectively improve the test coverage by using symbolic execution, ensure the correctness of the generated code, generate a large number of complementary boundary test cases while reducing manual intervention, and find program errors that are difficult to capture by traditional test frameworks. Specifically, the present application has the following beneficial effects:

[0043] 1. Compared with existing artificial intelligence guided program test technology, the method of the present application combines strict reasoning and fuzzy reasoning processes through symbolic execution, so as to generate tests for deep execution paths, cover program execution paths that are difficult to achieve by existing methods, and obtain higher test coverage.

[0044] 2. The present application effectively enhances the constraint solving capability of symbolic execution by using artificial intelligence models, so as to effectively solve complex constraints that are difficult to handle by existing methods, generate more high-quality and high-coverage test cases, and significantly improve the comprehensiveness and accuracy of the test.

[0045] 3. The method of the present application can automatically generate expected output in the absence of benchmark implementation and pre-defined output, obtain accurate test results, reduce manual intervention, and improve the automation degree of the test process.

[0046] 4. The method of the present application can generate a large number of complementary boundary test cases, find program errors that are difficult to capture by traditional test frameworks, and significantly improve the reliability of the program automatically generated by artificial intelligence guidance. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 The total flow of the artificial intelligence guided program test method combined with symbolic execution.

[0048] Figure 2 The flow of the test framework for introducing the artificial intelligence guided program combined with symbolic execution.

[0049] Figure 3A process for enhanced solving of program path constraints collected by symbolic execution using artificial intelligence technology.

[0050] Figure 4 A process that can obtain accurate test results regardless of whether the program to be tested has a reference implementation and a predefined output. DETAILED DESCRIPTION

[0051] The present application proposes a program test method guided by artificial intelligence combined with symbolic execution. By introducing symbolic execution technology into the test framework of artificial intelligence guided program generation, the strict reasoning ability of symbolic execution and the fuzzy reasoning ability of artificial intelligence model are effectively combined to test the program generated by artificial intelligence model, which significantly improves the test coverage and defect detection rate.

[0052] Step 1-1. Introduce symbolic execution technology into the test framework of artificial intelligence guided program generation, effectively combine the strict reasoning ability of symbolic execution and the fuzzy reasoning ability of artificial intelligence model, build a complete test process, automatically generate the input and output of test cases, and perform high-coverage functional testing.

[0053] Step 1-2. For the test framework of step 1-1, combine the use of artificial intelligence technology to enhance the solving of program path constraints collected by symbolic execution, make up for the shortcomings of symbolic execution technology when facing complex or unsolvable constraints, and generate a large number of complementary boundary test cases.

[0054] Step 1-3. When implementing testing based on step 1-1, when there is no reference implementation and predefined output, use a large language model to predict the test standard output, thereby realizing automated functional testing and obtaining accurate results. This method can obtain accurate test results regardless of whether the program to be tested has a reference implementation and a predefined output.

[0055] The following section further describes some specific details in the implementation process:

[0056] I. Strategy for introducing symbolic execution technology into the test framework of artificial intelligence guided program generation

[0057] Step 2-1. Use an artificial intelligence model to generate a program to be tested code according to a user-provided natural language description and a function template, wherein the natural language description outlines the problem to be solved and specifies the expected functionality of the code, and the function template defines the format of the code, including function name, parameter type and name, and any necessary libraries and packages for different programming languages.

[0058] Step 2-2. Use an artificial intelligence model to generate a seed input I in a fuzzy test method according to the natural language description and a test input template seedWherein, the natural language description describes the problem to be solved, and the test input template defines the format of the test input, including the type and name of the input, and the value range of the input;

[0059] Step 2-3. Apply the fuzz testing method to generate more test inputs I using the seed input generated by the artificial intelligence model in the method of mutating the use case fuzz Step 2-4. Merge the seed input and the test input generated by mutation into the initial test input set, denoted as I Original Then we have:

[0060] I Original = I seed ∪ I fuzz

[0061] Step 2-4. Input the program to be tested generated by the above artificial intelligence model into the symbolic execution engine, set the symbolic constraints, collect the constraints, and use the symbolic execution engine and the artificial intelligence model to solve respectively, to obtain the corresponding test input set I SMTSolved and I AISolved Merge the two with the initial test input set I Original as the test input set for functional testing, denoted as I, then we have:

[0062] I = I Original ∪ I SMTSolved ∪ I AISolved

[0063] Step 2-5. If the program to be tested has a benchmark implementation or predefined output, use the large language model to generate the expected output, and if it does not have a benchmark implementation or predefined output, use the large language model to generate the expected output. Organize the test inputs {in1, in2, …} in the test input set I for functional testing and the expected outputs corresponding to them into test case pairs, and combine them into a test case set T, as follows:

[0064] T = {(in1, exp_out1), (in2, exp_out2), …}

[0065] Step 2-6. Perform functional testing on the automatically generated program to be tested through the test case set T, check if the test coverage reaches the expected value, and verify if the program to be tested can pass all test cases correctly, to verify the correctness of the program generated by artificial intelligence guidance.

[0066] Introducing symbolic execution techniques into the testing framework of artificial intelligence guidance generation programs can fully leverage the rigorous reasoning capabilities of symbolic execution and the fuzzy reasoning capabilities of artificial intelligence models, and achieve efficient test case generation. In the existing testing framework of artificial intelligence guidance generation programs, test cases are often generated based on the fuzzy testing method. This testing method generates inputs by randomly mutating seed inputs to discover errors in the program, which is an efficient testing method. However, artificial intelligence guidance generation programs may have extremely complex input spaces, making it difficult for randomly mutated inputs in fuzzy testing to cover all possible scenarios, resulting in low test coverage. Therefore, introducing symbolic execution techniques into the testing framework can generate test cases for deep execution paths by solving path constraints in the program, thereby achieving better test coverage.

[0067] The symbolic execution engine first traverses all possible execution paths of the program. During the traversal process, the symbolic execution engine records the conditional branches and loop conditions encountered on each path. For each execution path, the symbolic execution engine collects all the constraints of the conditional branches and loop conditions on the path. These constraints are usually represented in the form of logical expressions, describing the range and relationship of the variables on the path. The symbolic execution engine performs reverse evaluation on the conditions on the path. For example, for a condition if(x>0), the symbolic execution engine generates two constraints: x>0 and x<=0, corresponding to the true and false conditions respectively. The symbolic execution engine combines all the constraints on the path into a conjunctive normal form (CNF), i.e. the logical and of all constraints. This step ensures that all conditions on the path are considered. The symbolic execution engine inputs the generated conjunctive normal form into a satisfiability modulo theories solver (SMT Solver). The solver attempts to solve these constraints and find variable values that satisfy all constraints. Once the solver finds a solution that satisfies the constraints, the symbolic execution engine takes these solutions as test inputs. Each test input corresponds to a specific execution path, ensuring that different paths of the program are tested.

[0068] II. Strategies for enhanced solving of program path constraints using artificial intelligence technology

[0069] Step 3-1. Design symbolic constraint templates to guide the symbolic execution engine to perform accurate constraint analysis, including the format of the use case, symbolic declaration, preconditions of the constraints, and other adaptation information of the engine;

[0070] Step 3-2. Set up the symbolic constraints, apply the symbolic constraint template to the program under test, use the program under test as the input to the symbolic execution engine, the symbolic constraint collection module in the symbolic execution engine traverses the execution paths in the program according to the symbolic execution strategy, collects the loops and branch conditions encountered along each path, reverses the conditions evaluated as False, and groups the conditions evaluated as True to form a conjunctive normal form, propagates constants and variables to construct constraints, that is, the jth path path j The corresponding constraints are constraints(path j );

[0071] Step 3-3. For solvable constraints, the satisfiability modulo theories solver module SMTSolver in the symbolic execution engine uses satisfiability modulo theories to solve these constraints, generating test inputs for the current path, as follows:

[0072] SMTSolve(constraints(path j ))=in j

[0073] Where in j is the test input corresponding to the execution path path j , SMTSolve is the satisfiability modulo theories solving method, and the test input set I SMTSolved is composed of all solvable inputs generated in this process;

[0074] Step 3-4. For constraints that are difficult to solve, by providing the artificial intelligence model with the requirement that the following path constraints in symbolic execution are requested to be solved, and then pasting the symbolic constraints collected from the symbolic execution engine, test inputs corresponding to the path can be generated, as follows:

[0075] AISolve(constraints(path j ))=supp_in j

[0076] Where supp_in j is the test input generated by the artificial intelligence model corresponding to the execution path path j , AISolve is the artificial intelligence model enhanced constraint solving method, and the test input set I AISolved is composed of all test inputs generated in this process.

[0077] During symbolic execution, the constraints of program paths sometimes become extremely complex and even unsolvable. This complexity can arise from the complex logical structure of the program itself or the special nature of specific paths, making it difficult for traditional symbolic execution engines to generate corresponding test inputs, resulting in insufficient test coverage. To address this challenge, we introduce artificial intelligence technology to enhance the ability to solve these complex constraints.

[0078] Specifically, when the symbolic execution engine encounters constraints that are difficult to solve, we hand these constraints over to an artificial intelligence model for processing. First, the artificial intelligence model learns the patterns and characteristics of different types of constraints through analysis of a large amount of historical data. Then, it uses this learned knowledge to try to infer input values that may satisfy the constraints.

[0079] In this process, the artificial intelligence model uses advanced techniques such as generative adversarial networks (GAN) or reinforcement learning to continuously optimize its output by generating and evaluating a large number of candidate inputs. These generated inputs not only help solve specific complex constraints, but also explore paths in the program that have not been covered. Through this method, the artificial intelligence model can generate a large number of boundary test cases, covering various edge cases.

[0080] In addition, we also use a hybrid strategy to combine the inputs generated by artificial intelligence with the inputs obtained by symbolic execution to form a more comprehensive test set. This ensures that even when faced with extremely complex or unsolvable constraints, test coverage can be significantly improved.

[0081] Through this strategy, the combination of artificial intelligence and symbolic execution can compensate for their respective shortcomings, enabling the generation of rich and effective test inputs even when faced with complex or unsolvable constraints, thereby comprehensively improving the quality and efficiency of testing.

[0082] III. Strategy for generating expected output using large language models

[0083] Step 4-1. In the case where the program under test has a benchmark implementation or predefined output, input the test input set into the program of the benchmark implementation, and take the corresponding output as the expected output in the functional test.

[0084] Step 4-2. In the case where the program under test lacks a benchmark implementation and the test input has no predefined output, use a large language model to generate the expected output. Input the natural language description desc and the corresponding test input in i ∈I into the large language model to determine the expected output. Through multiple iterations, select the most frequently occurring value as the expected output, as follows:

[0085] LLMCompute(desc,in j) = out i1 , out i2 ,…

[0086] exp_out i = max_freq(out i1 , out i2 , …)

[0087] where LLMCompute is a large language model computing expected output method, max_freq is a method for selecting the most frequently occurring value, exp_out i is the corresponding expected output of the test input in j ;

[0088] Step 4-3. Organize the test inputs {in1, in2, …} in the test input set I obtained in step 2-4 and their corresponding expected outputs into test case pairs, and combine them into a test case set T as follows:

[0089] T = {(in1, exp_out1), (in2, exp_out2), …}

[0090] Through this test case set T, the function of the program under test can be tested and accurate test results can be obtained.

[0091] In functional testing, verifying the correctness of the program requires comparing the actual output with the expected output. However, for programs without a benchmark implementation or predefined output, determining the expected output is a significant challenge. To solve this problem, we can use a large language model to generate the expected output through natural language description and test input.

[0092] Specifically, when lacking a benchmark implementation, we provide detailed natural language descriptions and specific test inputs to the large language model. Through deep learning on large-scale text data, the model understands the complex relationship between the description and the input, and infers the possible output results. In this process, the model not only considers the surface information of the input, but also analyzes its deep semantics and related background knowledge, making the generated output more accurate and reasonable.

[0093] In the process of generating expected output, the large language model generates multiple candidate outputs based on the input. We assign a confidence score to each candidate output, and through multiple iterations, we evaluate and optimize these outputs. Finally, the value with the highest frequency or the highest confidence is selected as the final expected output. This method can effectively handle complex input-output relationships and ensure that the generated expected output meets the expected function of the program.

[0094] By using large language models to generate expected outputs, accurate test results can be obtained even without benchmarks. This approach not only improves the automation of testing, but also flexibly handles complex functions and ambiguous input-output relationships. It provides greater flexibility and reliability for the testing process, allowing developers to verify the correctness of the program in a shorter time and quickly adapt to changing requirements.

Claims

1. A method of generating program tests for artificial intelligence guided by symbolic execution, characterized by By combining the rigorous reasoning ability of symbolic execution and the fuzzy reasoning ability of artificial intelligence model, the code generated by artificial intelligence guided automation is effectively tested, and the program path constraints collected by symbolic execution are enhanced by artificial intelligence technology to generate test cases, the specific steps are as follows: Step 1-1. Introduce symbolic execution technology into the test framework of the program generated by artificial intelligence guidance, combine the rigorous reasoning ability of symbolic execution and the fuzzy reasoning ability of artificial intelligence model, build a complete test process, and automatically generate the input and output of test cases for functional testing; The specific steps in the test framework are as follows: Step 1-1-1. Use an artificial intelligence model to generate program code to be tested according to user-provided natural language descriptions and function templates, wherein the natural language description outlines the problem to be solved and specifies the expected function of the code, and the function template defines the format of the code, including function name, parameter type and name, and any necessary libraries and packages for different programming languages; Step 1-1-2. Generating a seed input I in the fuzz testing method according to the natural language description and the test input template using an artificial intelligence model seed wherein the natural language description describes a problem to be solved, and the test input template defines the format of the test input, including the type and name of the input, and the value range of the input; Step 1-1-3. Apply fuzzing methods to generate more test inputs I using the seed inputs generated by the AI model in a mutation of the use case approach fuzz Combine the seed inputs with the mutation-generated test inputs as the initial test input set, denoted as I Original Then, there is: I Original = I seed ∪ I fuzz Step 1-1-4. The above artificial intelligence model guided generated test program input is input into the symbolic execution engine, the symbolic constraints are set, the constraint collection is performed, and the symbolic execution engine and the artificial intelligence model are solved respectively to obtain the corresponding test input set I SMTSolved and I AISolved The two are combined with the initial test input set I Original , as the test input set of the function test, denoted as I, then: I = I Original ∪I SMTSolved ∪I AISolved Step 1-1-5. If the program to be tested has a benchmark implementation or predefined output, use it as the corresponding expected output, if it does not have a benchmark implementation or predefined output, use a large language model to generate the expected output, organize the test inputs {in1, in2, …} in the test input set I and the corresponding expected output into test case pairs, and combine them into a test case set T, as follows: T={(in1,exp_out1),(in2,exp_out2),…} Step 1-1-6. Perform functional testing on the automatically generated program to be tested through the test case set T, check if the test coverage reaches the expected value, and verify if the program to be tested can correctly pass all test cases, to verify the correctness of the program generated by artificial intelligence guidance; Step 1-2. For the test framework of step 1-1, combine the enhanced solving of program path constraints collected by symbolic execution using artificial intelligence technology to generate multiple complementary boundary test cases; Step 1-3. When there is no benchmark implementation or predefined output, use a large language model to predict the test standard output to achieve automated functional testing and obtain accurate results; The specific steps are as follows: Step 1-3-1. In the case where the program to be tested has a benchmark implementation or predefined output, input the test input set into the program of the benchmark implementation, and use the output as the expected output in the functional test; Step 1-3-2. In the case where the program under test lacks a baseline implementation and the test input has no pre-defined output, generate the expected output using a large language model, using the natural language description desc and the corresponding test input in i ∈Iinput to the large language model, which determines the expected output. Through multiple iterations, the most frequently occurring value is selected as the expected output, as follows: LLMCompute(desc,in i ) = out i1 , out i2 ,... exp_out i = max_freq(out i1 , out i2 , …) Wherein, LLMCompute is a large language model computing expected output method, max_freq is a method of selecting the most frequently occurring value, exp_out i For testing input in i The corresponding expected output; Step 1-3-3. Organize the test inputs {in1, in2, …} in the test input set I obtained in step 1-1-4 and the corresponding expected output into test case pairs, and combine them into a test case set T, as follows: T={(in1,exp_out1),(in2,exp_out2),…} Through the test case set T, the program to be tested can be functionally tested and accurate test results can be obtained.

2. The artificial intelligence guided generation procedure test method in conjunction with symbolic execution of claim 1, wherein The specific steps of step 1-2 are as follows: Step 1-2-1. Design a symbol constraint template to guide the symbol execution engine to perform accurate constraint analysis, including the format of the use case, symbol declaration, preconditions of the constraint, and other adaptation information of the engine; Step 1-2-2. Set up the symbolic constraints, apply the symbolic constraint template to the program under test, use the program under test as the input of the symbolic execution engine, and the symbolic constraint collection module in the symbolic execution engine traverses the execution paths in the program according to the symbolic execution strategy, collects the loops and branch conditions encountered along each path, reverses the conditions evaluated as False, and groups the conditions evaluated as True to form a conjunctive normal form, and propagates constants and variables to construct constraints, that is, the path of the jth path j The corresponding constraints are constraints(path j ); Step 1-2-3. For solvable constraints, the satisfiability modulo theory solver module SMTSolver in the symbol execution engine uses satisfiability modulo theory to solve these constraints, generating test inputs for the current path, as follows: SMTSolve(constraints(path j )) = in j wherein in j is the test input corresponding to the execution path path j , SMTSolve is a satisfiability modulo theories solver, and the set of test inputs I SMTSolved consists of all the solvable inputs generated in this process; Step 1-2-4. For difficult-to-solve constraints, by providing requirements to the artificial intelligence model, i.e., requesting to solve the following path constraints in symbol execution, and then pasting the symbol constraints collected from the symbol execution engine, i.e., the test inputs for the corresponding path can be generated, as follows: AISolve(constraints(path j )) = supp_in j wherein supp_in j is a test input generated by the artificial intelligence model corresponding to the execution path path j AISolve is an artificial intelligence model enhanced solving constraint method, and the test input set I AISolved is composed of all test inputs generated in this process.

Images