Automatic generation method of dynamic language test cases based on large language model

Abstract

The present invention discloses a method for automatically generating dynamic language test cases based on a large language model. First, in the preprocessing stage, the relevant information of the program to be tested is organized as prompts. The large language model analyzes the parameter types of the program to be tested according to these prompts, and generates the code information required for the test through an abstract syntax tree; in the test case generation stage, a dynamic multi-objective optimization algorithm is used to continuously cross and mutate the test cases through branch coverage feedback. In the partial evolution process, the large language model is used to generate test data that conventional mutations cannot generate, and finally the selected excellent test cases are organized into a complete test suite; in the program repair stage, the test suite is run to detect potential errors, and the large language model is used to analyze and iteratively repair the errors to generate an executable and error-free test suite. The present invention significantly improves the test coverage of dynamic languages ​​and accelerates the achievement of coverage goals.

Description

Technical Field

[0001] The present invention relates to the technical field of automatic test case generation for dynamic languages, and in particular to a method for automatically generating dynamic language test cases based on a large language model. Background Art

[0002] Automatic test case generation is a method of automatically creating test cases through algorithms, which aims to verify the functionality, performance, and security of software systems without having to manually write each test case. In this field, evolutionary algorithms occupy a very important position. Evolutionary algorithms are based on the principles of natural selection and genetic algorithms, simulating the biological evolution process, and optimizing the generated test case set through continuous iteration.

[0003] At present, there are many excellent evolutionary algorithms for automatically generating test cases, including the Multi-Objective Sorting Algorithm (MOSA), the Multi-Independent Objectives Algorithm (MIO) and the Dynamic Multi-Objective Sorting Algorithm (DynaMOSA). These methods have been integrated into automatic test case generation tools such as Evosuite and Pynguin, and have achieved good results, but they still have some problems.

[0004] On the one hand, evolutionary algorithms need to rely on parameter type information when generating test cases in order to provide appropriate parameter types for method calls or assembling payload objects. However, when generating test cases for dynamic languages, the search space becomes extremely large because the code written in dynamic languages ​​lacks parameter type information. Different parameter types need to be tried continuously to find the correct parameter type, which makes test case generation inefficient. For example, the tools Pynguin and Nessie both use evolutionary algorithms to generate test cases, but their performance is limited by the problem of missing parameter types.

[0005] On the other hand, the DynaMosa evolutionary algorithm treats test cases as individuals, and N such individuals form a population, thereby simulating the process of biological evolution to generate a new generation of populations. The evolutionary strategy adopted by the algorithm includes operations such as crossover between individuals and random mutation of individual data. However, due to the high randomness of these operations, they are insufficient in their ability to process data that requires a specific format. For example, when it is necessary to generate data in a specific date format, the probability of generating data that meets the requirements through random mutation is very low. This may result in insufficient validity of the data generated by the algorithm in some cases, which may cause the algorithm to fall into a local optimal solution.

[0006] In addition, when automatically generating test cases, in order to verify the correctness of the functions, it is usually necessary to generate regression assertions to ensure that each test case has a clear expected result. These assertions can not only effectively detect whether the program under test meets the expected behavior, but can also be used for subsequent regression testing. Therefore, the executable nature of the assertions is crucial. However, the currently automatically generated test suites usually provide inspiration for manually making high-coverage test suites, and cannot be used directly due to errors. For example, some test cases generated by Pynguin cannot be run directly due to environmental problems.

[0007] In summary, the automatic generation of test cases using evolutionary algorithms still faces the following challenges:

[0008] (1) When the object to be tested is a dynamic language, the characteristic of no parameter type leads to a large search space and low search efficiency;

[0009] (2) The method that relies solely on random mutation generates insufficient data availability, making it difficult to find the global optimal solution;

[0010] (3) The generated test cases cannot be used directly due to runtime errors. Summary of the invention

[0011] In view of the above shortcomings, the present invention proposes a dynamic language test case generation method based on a large language model. It mainly includes three stages: a preprocessing stage, a test case generation stage, and a program repair stage. In the preprocessing stage, a parameter analysis prompt is constructed and sent to the large language model to perform parameter analysis on the program to be tested, the parameter type after analysis is marked in the program to be tested, and finally the information required for the test generation step is extracted; in the test case generation stage, a new mutation strategy is introduced for the dynamic multi-objective selection algorithm, the large language model is used to mutate the original data, and the improved dynamic multi-objective algorithm is used to generate test cases for the program to be tested; in the program repair stage, the erroneous test code and the cause of the error are constructed into a program repair prompt, which is sent to the large language model, and the errors in the test case are repaired, and finally a test case that can be directly run is obtained.

[0012] The present invention provides a method for generating dynamic language test cases based on a large language model, comprising the following stages:

[0013] Preprocessing stage: First, obtain the program to be tested, and construct the program to be tested and its related information into parameter analysis prompts;

[0014] Then, the parameter analysis prompt is sent to the large language model for parameter type analysis, and the parameter type annotation is extracted from the content returned by the large language model and added to the program under test;

[0015] Finally, the program under test with parameter type annotations is parsed using the abstract syntax tree to extract the metadata required for generating the test;

[0016] Test case generation phase: The test phase uses a dynamic multi-target selection algorithm with a large language model mutation strategy.

[0017] First, a set of test cases is randomly initialized as the initial population;

[0018] Then, the initial population is randomly operated to generate new offspring. After a certain number of iterations in the evolution process, the large language model is called to mutate the offspring, and the fitness value is calculated to retain the excellent offspring. After several rounds of iterations, the mutation score of the offspring is calculated, and the offspring with repeated coverage are removed to obtain a test suite.

[0019] Program repair phase: First, run the test suite to check whether there are any syntax errors. If so, construct a syntax error repair prompt based on the error cause and error code, and send it to the large language model for iterative repair;

[0020] Then, the test suite is run to check whether there are runtime errors. If there are runtime errors, the error cause and error code are constructed into a runtime error repair prompt and sent to the large language model for iterative repair. Finally, if the error still cannot be repaired, the erroneous part of the code is deleted to obtain the final executable test suite.

[0021] Preferably, the specific process of the pretreatment stage includes the following steps:

[0022] 1-1. Obtain the program to be tested, wherein the program to be tested is a Python module to be tested;

[0023] 1-2. Construct parameter type analysis prompts based on the program to be tested, the document content of the program to be tested, and the task description, and send them to the large language model;

[0024] 1-3. Parse the response returned by the large language model, obtain the message content field, match the Python code part in the message field through regular expressions, and combine the parameter type annotations and the program to be tested described in step 1-1 to form a new program to be tested;

[0025] 1-4. Recursively import the constructed classes, called functions and other information from the modules that the program under test depends on;

[0026] 1-5. Retrieve the program to be tested and the modules on which the program to be tested depends, and save the variable values ​​used therein as the source of initial values ​​of the program to be tested;

[0027] 1-6. Use the ast module to parse the program under test into an abstract syntax tree, and use the inspect module to inspect the program under test and retrieve the available classes, functions, and methods as metadata required for testing;

[0028] 1-7. Construct the metadata required for the test into sets and mappings for storage: the set contains information about all callable or accessible elements in the module under test, and the accessible content in the Python module under test is selected from the set to generate input; the mapping stores callable or accessible elements, including elements in the module under test and elements of modules on which the module under test depends. The elements include classes, functions, and methods, and are used to record type usage information of the elements.

[0029] Preferably, the test case generation phase can be specifically divided into the following phases: an initial phase, an evolution phase, a large language model mutation phase, and a mutation test phase.

[0030] Preferably, the initial stage specifically includes the following steps:

[0031] 2-1-1. Set the coverage target to have no control dependency branch, and get the initial coverage target Bench={b1, b2, ..., b n};

[0032] 2-1-2. Select a method or function from the set described in 1-7, use the mapping described in 1-7 and the corresponding value randomly selected from the initial value source described in 1-5 to construct an initial test case, and continue to generate test statements until the maximum number of statements is reached. Continue to generate test cases as individuals in the population until the maximum number of populations is reached, and obtain the initial population population = {solution1, solution2, ..., solution n};

[0033] 2-1-3. For the initial coverage target Bench={b1, b2, ..., b n}, for the initial population population={solution1, solution2, ..., solution n}, and divide the individuals with the same fitness value in the population into the same front, and obtain the front set front = {front0, front1, front2, ..., front n}, where front0 holds the individual with the smallest fitness, front1 holds the individual with the second smallest fitness, and so on;

[0034] 2-1-4. For the initial population population = {solution1, solution2, ..., solution n} calculate the crowding distance of the individuals in the cluster and obtain the crowding distance of each individual;

[0035] 2-1-5. Select excellent individuals based on the frontier set described in 2-1-3 and the crowding distance of each individual described in 2-1-4: traverse all current frontiers. If the number of individuals in the frontier with the smallest fitness does not exceed the population size, all of them are added to the offspring population until the number of remaining individuals in the offspring population is less than the number of individuals in the current smallest frontier. Then, continue to add individuals to the offspring population from large to small according to the crowding distance until the maximum population size is reached, and obtain the offspring population offspring. i ;

[0036] 2-1-6. Recalculate the coverage target Bench = {b1, b2, ..., b n}, remove the covered targets and add targets whose control dependencies are satisfied. The targets whose control dependencies are satisfied refer to the new targets that can be reached after this iteration because their control dependencies are satisfied. Generate new coverage targets and record the reached targets and the corresponding test cases through archiving;

[0037] 2-1-7. Entering the evolution stage, offspring population described in 2-1-5 i Perform crossover and mutation to generate new offspring.

[0038] Preferably, the specific process of the evolution stage is as follows:

[0039] 2-2-1. Offspring in the current offspring population i In the process, two test cases are selected for crossover operation, a crossover point is randomly selected in the test case, each test case is divided into two parts according to the crossover point, the second half of the crossover point of the two test cases is exchanged, and the variable data in the repair statement is ensured to ensure that the test case after crossover is still valid;

[0040] 2-2-2. Mutate the crossover test case to obtain a mutated test case. The mutation method is one of the following three methods:

[0041] Insert operation: offspring in the current generation population i In the test case, a test case is randomly selected and a new statement is inserted at a random position in the test case with probability , after each insertion, the probability of reinsertion is , until no more insertions are made, the insertions including new method calls or existing method calls;

[0042] Modify operation: Offspring in the current offspring population i In the test case, test cases are randomly selected. For the existing test cases of length L, each statement is mutated with a probability of 1 / L. For int and float, new values ​​are generated and existing values ​​are modified; for str, characters are deleted, replaced or inserted; for complex types, including lists, sets, and dictionaries, elements are replaced; method and function parameters may be modified or removed;

[0043] Delete operation: Offspring in the current generation population i In the test case, test cases are randomly selected. For test cases of length L, each statement is deleted with a probability of 1 / L. If the deletion causes dependency problems, the dependent statements need to be repaired or recursively deleted.

[0044] 2-2-3. After the individuals in the current population are mutated into new individuals through crossover, insertion, modification, and deletion operations, the fitness and crowding distance are calculated as described in 2-1-3 and 2-1-4, and the excellent individuals are selected according to the steps described in 2-1-5 to obtain the next generation population offspring. i+1 , at the same time, carry out the above-mentioned 2-1-6 and update the coverage target, archive the test cases that can reach the coverage target, and choose to continue the evolution or enter the large language model mutation stage according to whether the current number of iterations reaches the threshold of the large language model mutation stage;

[0045] 2-2-4. If the maximum number of test iterations or test time is reached, enter the mutation test phase, streamline the archived test cases that can reach the coverage target, and combine the test cases into a test suite.

[0046] Preferably, the specific process of the large language model mutation stage is as follows:

[0047] 2-3-1. Evolution stage After n rounds of iterations, n can be selected by yourself and is recommended to be a multiple of 10. Use the large language model to perform mutation operations based on the large language model on the individuals in the current population, identify the methods or functions in the new program to be tested described in 1-3, and save the methods or functions in the list UTfun;

[0048] 2-3-2. Obtain the function to be processed by randomly sampling the methods or functions in UTfun;

[0049] 2-3-3. Construct a difficulty prompt template for the function to be processed, call the large language model Deepseek to analyze the function to be processed, and obtain the difficulty of testing the function;

[0050] 2-3-4. Construct a mutation prompt template for the function to be processed, call the large language model Deepseek to generate new test data for the difficult problems obtained in 2-3-3;

[0051] 2-3-5. Perform the crossover and mutation operations described in 2-2-1 and 2-2-2 on the new test data together with the individuals in the current population to generate new individuals. Perform the fitness calculations and crowding distance calculations described in 2-1-3 and 2-1-4. Select the excellent individuals according to the steps described in 2-1-5 to obtain the next generation of the population. At the same time, perform the update coverage target described in 2-1-6, and archive the test cases that achieve the coverage target. Finally, choose to enter the evolution phase or mutation test phase based on whether there are still iterations available.

[0052] Preferably, the specific process of the mutation testing stage is as follows:

[0053] 2-4-1. Use the MUTPY tool to generate a variant version of the new program to be tested described in 1-3, wherein the variant version is obtained by making a small amount of grammatical modification on the original module;

[0054] 2-4-2. Execute the test cases archived in the evolution stage 2-2-3 and the large language model mutation stage 2-3-5 on the variant version, and determine whether the variant version is alive in the following way. If a test case can produce a result different from the original version in the variant version, the variant version is considered to be killed, otherwise the variant version is considered to be alive;

[0055] 2-4-3. Test cases are retained based on the number of variant versions killed by the test cases. The test cases that kill more variant versions have a higher retention priority. From the archived test cases, assertions that did not work when testing variant versions are deleted, thereby minimizing the assertion set, retaining the most effective assertions, and obtaining a test suite.

[0056] Preferably, the specific process of the program repair stage is as follows:

[0057] 3-1. Input is the test suite described in 2-4-3, check whether the test suite has grammatical errors, if the test suite has grammatical errors, organize the grammatical error information and the grammatical error part code into a grammar repair prompt template in combination with the task description, send the prompt to the large language model Deepseek for 5 rounds of iterative error correction, if the error correction is successful, end early, and obtain the test suite with grammar repair;

[0058] 3-2. Run the test suite after grammar repair to check whether there are runtime errors in the test suite after grammar repair. If there are runtime errors, organize the runtime error information and the runtime error code into a runtime error repair prompt template in combination with the task description, and send the prompt to the large language model Deepseek for 5 rounds of iterative error correction. If the error correction is successful, it ends early and finally obtains the repaired test suite;

[0059] 3-3. If there are still errors in the repaired test suite, remove the error code lines and finally get a repaired executable test suite.

[0060] The present invention has the following characteristics and beneficial effects:

[0061] (1) The present invention introduces parameter type analysis in the preprocessing stage, which reduces the search space, improves the search efficiency, and frees SBST from the limitations brought by parameter types to a certain extent;

[0062] (2) This paper proposes a new mutation strategy that uses a large language model to mutate test data, improves data availability, and increases the possibility of finding the optimal solution;

[0063] (3) When encountering a test case that cannot be executed, the present invention constructs a prompt based on the error code and error message, and uses a large language model to repair the program, thereby improving the quality of the test suite. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is an overall flow chart of the method for automatically generating dynamic language test cases based on a large language model of the present invention;

[0065] Figure 2 It is a flow chart of the test case generation phase of the method for automatically generating dynamic language test cases based on a large language model of the present invention; DETAILED DESCRIPTION

[0066] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the invention, but are not intended to limit the present invention in any form. It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other.

[0067] like Figure 1 As shown, the test case generation method based on the large language model and dynamic language includes a preprocessing stage, a test case generation stage and a program repair stage, wherein:

[0068] In the preprocessing phase, the program to be tested and its related information are first determined, and parameter analysis prompts are constructed. Then, these prompts are sent to the large language model for parameter type analysis, and parameter type annotations are extracted from the returned content and added to the program to be tested. Subsequently, the program with parameter type annotations is parsed using an abstract syntax tree to extract the data required to generate the test;

[0069] In the test case generation phase, a dynamic multi-objective selection algorithm is used to randomly initialize a set of test cases as the initial population. New offspring are generated through operations such as crossover and mutation, and the large language model is called for mutation during the evolution process. The fitness value is calculated to retain excellent offspring, and the process is repeated until resources are exhausted. Finally, the mutation score of the offspring is calculated, and the offspring with repeated coverage are removed to form a test suite;

[0070] In the program repair phase, the test suite is first run to check for syntax errors. If a syntax error is found, the cause of the error and the code are constructed into a syntax repair prompt and sent to the large language model for iterative repair. Next, check for runtime errors. If a runtime error is found, the cause of the error and the code are constructed into a runtime repair prompt. If the error still cannot be repaired, the error code is finally deleted to obtain an executable test suite.

[0071] I. The pretreatment stage of the present invention specifically comprises the following steps:

[0072] 1-1. Select a Python module to be tested as the program to be tested.

[0073] 1-2. Organize the program to be tested, the document content of the program to be tested, and the task description into a parameter type analysis prompt prompt_pa. The task description part of the parameter type analysis prompt should clearly indicate that this is a parameter type analysis task, and its content is: "Please help me indicate the parameter type and return type foreach function

[0074] and class in this code.", by calling the large language model API, the prompt is sent to the large language model Deepseek, and Deepseek returns a response to the prompt content.

[0075] 1-3. Parse the response returned by the large language model, retain its message content, use the regular expression "'''(.*?)'''|"""(.*?)"""" to match the Python code in the message field, and combine the parameter type annotations with the program to be tested in step 1-1 to form a new program to be tested.

[0076] 1-4. Use the open source module inspect to recursively traverse the dependent modules of the program under test. Use the getsource method in the inspect toolkit to import information such as constructed classes and called functions from the modules that the program under test depends on.

[0077] 1-5. Initialize a constant pool constant_pool, read the code path where the dependent modules are located, use the open source module AST to parse the module to be tested and the dependent modules into an abstract syntax tree, traverse the different nodes of AST, identify and collect the used variable values, and save them in the constant pool as the source of the initial value of the program to be tested.

[0078] 1-6. Use the AST module to parse the program under test into an abstract syntax tree, traverse the AST nodes, collect the available classes, functions, and methods, use the inspect module to inspect the program to obtain more detailed information, check whether it can work as expected, and retrieve the available classes, functions, and methods as metadata required for testing.

[0079] 1-7. Construct the program information to be tested into maps and collections for storage: collection: contains all callable or accessible element information in the module to be tested, such as classes, functions, and methods. Select callable or accessible elements from this collection to generate test input; map: records which functions, methods, or classes can create or return a specific data type, helping to determine which elements should be called when a specific type of object is needed during the test process.

[0080] II. The test case generation phase described in the present invention specifically includes the following steps: an initial phase, an evolution phase, a large language model mutation phase, and a mutation test phase.

[0081] 2-1. The initial stage includes the following steps:

[0082] 2-1-1. Initially, set the coverage target to have no control over dependent branches:

[0083] Use the toolkit bytecode to generate a CFG graph from the bytecode of the module under test, use the Lengauer-Tarjan algorithm to calculate the post-dominator tree and generate the control dependency graph CDG. Initially, select the branch without control dependency as the initial coverage target Bench={b1, b2, ..., b n}, where Bench represents the initial coverage target set, b n Represents the nth coverage target without control dependencies. Bench changes dynamically during the evolution process.

[0084] 2-1-2. Randomly select a method or function from the collection of information about the program to be tested obtained in step 1-7 and use the mapping obtained in step 1-7 and the initial value source constant pool constant_poll obtained in step 1-5 to randomly select an initial value to construct an initial test case for the program to be tested. The initial value should correspond to its parameter type. The test case consists of statements s, and the statement types include assignment statements and assertion statements. The maximum number of statements in a test case is Smax (recommended to be 20). Continue to add statements to the test case until the maximum number of statements is reached. The constructed test case is used as the individual solution in the initial population. i , continuously generate test cases as individuals in the population until the maximum population size Qmax (recommended to be 50) is reached, and the constructed Qmax individuals are used as the initial population population = {solution1, solution2, ..., solution n}.

[0085] 2-1-3. For the initial coverage target Bench={b1, b2, ..., b n}, for the initial population population={solution1, solution2, ..., solution n} calculates the fitness of the individuals in the code, and divides the frontier into generations according to the fitness. For a code object u without branches, its fitness calculation formula is as follows:

[0086] (1)

[0087] For a branch v, its fitness calculation formula is as follows:

[0088] (2)

[0089] Among them, cond represents a specific branch, x represents a test case, represents the degree of closeness, which is the number of control dependencies between the most recently executed branch and the coverage target b. It stands for branch distance, which is the minimum recorded branch distance of branch v when executing test case x. It consists of the true branch distance, the distance to the branch with the condition true, and the false branch distance, the distance to the branch with the condition false. If the target branch can be reached, the branch distance is 0. If the target branch cannot be reached, the distance is calculated using the conventional branch distance calculation method. The calculation formula is as follows:

[0090] (3)

[0091] (4)

[0092] After obtaining the fitness value, the individuals with the same fitness value in the population are divided into the same frontier. First, the level 0 frontier front0 is calculated. The individual solution saved in the level 0 frontier front0 is the individual solution that can reach the coverage target. i (Fitness value is 0), iterate from b1 coverage target to coverage target b n , save the individuals that can reach these coverage targets into the 0-level frontier list front0. If the number of individuals is less than the maximum population size Qmax, calculate the 1-level frontier solution and save it into the 1-level frontier list front1. The 1-level frontier is defined as the solution closest to the target except the 0-level frontier. By analogy, the 2-level frontier is the solution closest to the target except the 0-level and 1-level frontiers. Until the maximum population size Qmax is reached, stop calculating the frontier, and finally get the frontier set front={front0, front1, front2, ..., front n}.

[0093] 2-1-4. For the initial population population = {solution1, solution2, ..., solution n} calculates the crowding distance of the individuals in the target space, and calculates the average distance between the adjacent individuals in the target space for each individual in the population. Individuals with greater crowding can bring more diversity and should be protected; initialize the crowding distances in all fronts to 0, and for the coverage target Bench={b1, b2, ..., b n}Traverse and calculate the current front under each target i The number of solutions with the minimum fitness is len(test_set), cd represents the crowding distance of a certain offspring, len(front) represents the length of the frontier of the offspring, and the calculation formula of the crowding distance is as follows:

[0094]

[0095] The crowding distance cd of each individual is obtained.

[0096] 2-1-5. For the front set obtained in step 2-1-3, front = {front0, front1, front2, ..., front n}Traverse and process the current front in turn i , if the current front i The number is less than the maximum population size minus the number of offspring Qmax-len (offspring i ), then add the current frontier to the offspring i If the current front i The number is greater than the maximum population size minus the number of offspring Qmax-len (offspring i ), terminate the loop, sort the crowding distances of the individuals obtained in step 2-1-4, and select Qmax-len (offspring i ) individuals joined the offspring i In the offspring i The number of offspring reaches the maximum population size, and finally the offspring population is obtained. i .

[0097] 2-1-6. Regenerate the control dependency graph CDG and recalculate the coverage target Bench={b1, b2, ..., b n}, remove the covered target and add the target whose control dependency is satisfied. The target whose control dependency is satisfied refers to the new target that can be reached after this iteration because its control dependency is satisfied. Update the coverage target Bench={b1, b2, ..., b n}, the currently covered target b i And its corresponding test case, saved in the archive.

[0098] 2-1-7. The evolution from the initial population to the first generation of offspring has been completed. The offspring population obtained in step 2-1-5 is offspring. i Perform crossover and mutation to generate new offspring.

[0099] 2-2. Evolution stage, specifically including the following steps:

[0100] 2-2-1. Offspring in the current offspring population iSelect any two test cases solution1 and solution2, and any statement in solution1 i1 Then insert an intersection point1, and any statement in solution2 i2 Then insert an intersection point2, swap the content after intersection point1 with the content after intersection point2, and form two new test cases solution1', solution2'. It is necessary to pay attention to repairing the dependency of variables. That is, after the exchange, the variables and data in the two new test cases need to be repaired to ensure that the newly generated test cases are still syntactically and logically valid test cases.

[0101] 2-2-2. Mutate the crossover test case to obtain the mutated test case. Each mutation operation adjusts the structure and content of the test case to generate new individuals and enhance the diversity of the population. Define the test case length L as the number of statements in the test case x, and each statement in the test case is represented by s. The mutation method is one of the following three:

[0102] Insert operation: offspring in the current generation population i In the test case selection, the probability , insert a new statement at a random position p∈[0,l], where p is the insertion position, 0 is the start position of the test case, and l is the end position of the test case. If this statement is inserted, the probability of inserting another statement is , and so on, until the i-th statement is not inserted; for each insertion, either a new method call is inserted or an existing method call is inserted.

[0103] Modify operation: Offspring in the current offspring population i In the example, randomly select a test case. For the existing test case x= <s0,s1,...,s l-1 >, each statement s iMutation with probability 1 / L. Depending on the different types of variables, mutation operations include the following: Numeric types (int and float): Generate a random value according to the normal distribution, add to the existing value or change its decimal places; String type (str): Delete, replace or insert characters, and the probability of each character being deleted or replaced is 1 / string length; Complex types (lists, sets and dictionaries): Randomly replace elements in them. For example, randomly replace an element in a list, and randomly replace a key or value in a dictionary; Method or function parameters: Modify its parameter value with a certain probability. If the parameter is optional, the modification operation may remove it or select a new value from the existing object set as the parameter.

[0104] Delete operation: Offspring in the current generation population i In the example, a test case is randomly selected. For a test case x of length L, <s0,s1,...,s l-1 >, each statement s i It is deleted with a probability of 1 / L. If the deleted statement affects subsequent statements (such as a statement that depends on the variable value in the deleted statement), the subsequent statements need to be repaired to maintain the validity of the test case. If these dependencies cannot be repaired, all subsequent statements that depend on the statement will be recursively deleted.

[0105] 2-2-3. After the crossover, insertion, modification, and deletion operations are completed, the individuals in the current population have mutated into new individuals. Repeat steps 2-1-3 to 2-1-6 for the new individuals, calculate the fitness and crowding distance, select the excellent individuals, and obtain the next generation of offspring. i+1 At the same time, the coverage target is updated and the test cases that can reach the coverage target are archived. According to the current number of iterations, if the number of iterations reaches the threshold n (n can be set arbitrarily, and it is recommended to be set to a multiple of 10), the large language model mutation phase is entered. If the threshold n is not reached, the evolution phase is repeated.

[0106] 2-2-4 If the maximum number of test iterations or test time is reached, the archived test cases that can achieve the coverage target are streamlined through mutation testing, and the test cases are combined into test suites.

[0107] 2-3. The large language model mutation phase includes the following steps:

[0108] 2-3-1. After n rounds of iterations (n ​​can be set arbitrarily, and it is recommended to be set to a multiple of 10), in order to further improve the effect of test generation, a large language model is introduced to perform mutation operations based on the large language model on individuals in the current population. First, all functions defined in the program to be tested in steps 1-3 are identified by parsing the abstract syntax tree (AST) structure. The ast.walk() method can be used to traverse all nodes in the AST and identify function definitions. The identified functions or methods are saved in a list UTfun, which contains all function names and method names in the module to be tested and their corresponding parameters.

[0109] 2-3-2. Use the choice() method in the random toolkit to randomly sample the functions or methods in UTfun and select the function to be processed.

[0110] 2-3-3. First, let the large language model analyze the difficulties of testing the function to be processed, and construct a difficulty prompt difficulty_prompt for the function to be processed. The difficulty prompt is used to request the large language model to test the difficulties of the function to be processed. The prompt consists of the content of the function to be processed and the description of the task of finding difficulties instruction_difficulty, where the task description is: "Based on the above code content, if you want to generate therequired test case, what difficulties will there be? Please analyze the difficulties in an organized manner". Send the prompt to the large language model Deepseek to get the difficulty of testing the function.

[0111] 2-3-4. Then, the large language model is required to generate some test data for the difficult points, and a mutation prompt mutation_prompt is constructed for the function to be processed. The mutation prompt is used to request the large language model for the test data for the difficult points. The prompt consists of code code, test difficulty difficulty, and task description instruction_data for generating test data for the difficult points. The task description is: "Please complete the above test function to generate a set of excellent test cases

[0112] Pay special attention to its difficulties for the target to be tested. ". Call the large language model Deepseek to generate new test data for the difficult questions obtained in step 2-3-3;

[0113] 2-3-5. Parse the test data returned by Deepseek into a new individual and offspring it from the existing one i Combine to form a new population population_llm. In this new population, continue to perform the previously defined crossover and mutation operations on the test cases, steps 2-2-1 to 2-2-2: Crossover operation: randomly select test cases for partial exchange to ensure diversity; Insert, modify, and delete operations: adjust each part of the test case to ensure that the new individuals can explore more test paths. Then, continue with steps 2-1-3 to 2-1-6 to calculate fitness and crowding distance, select excellent individuals, and finally reach the next generation of offspring of the population. i+1 , and update the coverage target at the same time, and archive the test cases that can reach the coverage target. If there are still remaining iterations or evolution time, return to the evolution stage for a new round of evolution. If the maximum number of iterations or test time is reached, enter the mutation test stage.

[0114] 2-4. Mutation testing phase, specifically including the following steps:

[0115] 2-4-1. Use the MUTPY tool to generate variant versions of the new program to be tested obtained in step 1-3. Variant versions are obtained by making a small amount of grammatical modifications to the original code, such as replacing operators and deleting code branches. These variant versions are used to evaluate the effectiveness of test cases.

[0116] The minor grammatical modifications adopted include the following, as shown in Table 1:

[0117] Table 1: Mutation testing strategies

[0118]

[0119] 2-4-2. Execute the test cases archived in the evolution phase 2-2-3 and the large language model mutation phase 2-3-5 on the variant version. If a test case produces different results on the original module and the variant version, the variant version is considered to have been "killed"; if the test results are consistent, the variant version "survives". Only assertions that can cause test failures are considered "excellent" assertions. Assertions that do not cause errors in all variant version tests will be considered invalid and deleted from the test case to reduce assertion redundancy.

[0120] 2-4-3. Evaluate the priority of test cases based on the number of variants killed by each test case. Test cases that kill more variants have a higher retention value. For those assertions that do not work when testing variants, delete them from the archive to minimize the assertion set. The test cases that are finally retained will generate test suites in the form of pytest framework to ensure that the generated test suites are efficient and comprehensive.

[0121] III. The program repair phase of the present invention specifically includes the following steps:

[0122] 3-1. First, input the test suite to be repaired obtained in step 2-4-3, and check whether there are any syntax errors. If a syntax error is found, extract the syntax error message syntax_message and the syntax error code error_code, and combine them with the task description syntax_instruction to form a syntax repair prompt template syntax_repair_prompt, where the task description is: "The above code is incorrect and caused by syntaxerrors. Please help me fix this error code so that it can run normally." The syntax repair prompt template is used to request the large language model to repair the syntax errors in the code. The prompt is sent to the large language model through i rounds of iteration (recommended 5). The model returns the repair result syntax_result i If the repair is successful, the loop is terminated; if not, the next round of repair is continued until the repair is completed or the maximum number of iterations is reached, and finally a test suite with grammar repair is obtained.

[0123] 3-2. Run the test suite after the syntax repair, and check whether there is a runtime error in the test suite after the syntax repair. If there is a runtime error in the test suite, combine the runtime error message runtime_message and the runtime error code error_code with the task description runtime_instruction to form a runtime error repair prompt template runtime_repair_prompt, where the task description is: "The above code is incorrect

[0124] and caused by runtime error. Please help me fix this error code sothat it can run normally." The runtime error repair prompt template is used to request the large language model to repair the runtime error in the code. Similar to the grammar repair process, the runtime error repair prompt is sent to the large language model through i rounds of iteration (recommended 5), and the model returns the repair result runtime_result i If the repair is successful, the loop ends; if not, the iteration continues until the repair is completed or the maximum number of iterations is reached, and finally a repaired test suite is obtained.

[0125] 3-3. If there are still unresolved errors in the test suite after multiple rounds of repairs on the large language model, delete the code lines of the error section. Finally, you will get a fixed executable test suite.

[0126] The above specific implementation modes are only used to further illustrate the present invention, and are not the only implementation modes of the present invention, and should not be construed as limitations on the present invention.

[0127] In order to verify the effectiveness of the present invention, three sets of experiments were conducted. In the scenario of parameterless type, the method of the present invention, pytLMtester, was compared with the advanced python unit test generation algorithm CODAMOSA, and evaluated on two data sets. The first data set is 130 modules of 19 projects identified from the evaluation of Pynguin; the second is HumanEval, which consists of 164 simple to medium-level manually written programming problems.

[0128] Experimental results:

[0129] Experiment 1: On Dataset 1, pytLMtester achieved significantly better coverage than CODAMOSA in 34 modules, accounting for about 26% of the total; in 90 modules, there was no significant difference in coverage between the two, accounting for about 69% of the total; and in the remaining 6 modules, pytLMtester was inferior to CODAMOSA. On Dataset 2, pytLMtester achieved significantly better coverage than CODAMOSA in 13 modules, accounting for about 8% of the total; in 147 modules, there was no significant difference in coverage between the two, accounting for about 90% of the total; and in the remaining 4 modules, pytLMtester was inferior to CODAMOSA. At the same time, when achieving the same coverage, pytLMtester can achieve the same coverage at a faster speed.

[0130] Experiment 2: pytLMtester represents our complete framework, DYNAMOSA_PA represents the conventional DYNAMOSA algorithm plus our parameter analysis module, DYNAMOSA_Lm represents the conventional DYNAMOSA algorithm plus our large language model evolution strategy, and DYNAMOSAONLY represents the use of only the conventional DYNAMOSA algorithm. On Dataset 1, pytLMtester has a higher coverage rate than DYNAMOSAONLY on 28.5% of the modules, DYNAMOSA_PA has a higher coverage rate than DYNAMOSAONLY on 21.5% of the modules, and DYNAMOSA_Lm has a higher coverage rate than DYNAMOSAONLY on 10.8% of the modules; on Dataset 2, the coverage rate was significantly improved in 13, 8, and 4 modules, respectively, which further proves the effectiveness of the method.

[0131] Experiment 3: The test suite repair effect is shown in Table 2.

[0132] Table 2. Executable rate of program after repair

[0133]

[0134] After asking the language model five times, 67.69% of the errors can be fixed in Python Module, while 82.32% of the errors can be fixed in HumanEval. These repair results demonstrate the effectiveness of the repair module.

[0135] Result analysis:

[0136] Parameter type analysis provides clear guidance information for test cases. This guidance information helps the evolutionary algorithm generate legal and meaningful inputs, improves the effectiveness of the initial test population, and reduces the generation of meaningless test cases. This means that we start iterations with higher-quality test cases, thereby achieving higher coverage faster and reducing meaningless iterations.

[0137] During the evolution process, a large language model is used to mutate the original test cases, rather than relying solely on random mutation. Compared to traditional random mutation methods, LLM can use its understanding of code semantics to generate more diverse and meaningful individuals. For example, we can generate reasonable values ​​for str type dates, which is almost impossible to achieve by relying solely on traditional random mutation methods. Traditional random mutation methods may cause the population to linger around the local optimum, while using a large language model for mutation can help jump out of the local optimum and explore a wider solution space, thereby improving the coverage and quality of test cases.

[0138] The repair effect of the large language model improves significantly with the increase in the number of iterations. After multiple iterations, most runtime errors in the test suite can be effectively resolved. However, for a few complex errors, manual intervention is still required. Combining manual repair with automatic repair not only improves the overall quality of the test suite, but also reduces the burden on developers and improves development efficiency.

[0139] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and descriptions are only preferred examples of the present invention and are not intended to limit the present invention. The present invention may have various changes and improvements without departing from the spirit and scope of the present invention, and these changes and improvements fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A dynamic language test case generation method based on a large language model, characterized in that: The following phases are included: Preprocessing stage: First, obtain the program to be tested, and construct the program to be tested and its related information into parameter analysis prompts; Then, the parameter analysis prompt is sent to the large language model for parameter type analysis, and the parameter type annotation is extracted from the content returned by the large language model and added to the program under test; Finally, the program under test with parameter type annotations is parsed using the abstract syntax tree to extract the metadata required for generating the test; Test case generation phase: The test phase uses a dynamic multi-target selection algorithm with a large language model mutation strategy. First, a set of test cases is randomly initialized as the initial population; Then, the initial population is randomly operated to generate new offspring. After a certain number of iterations in the evolution process, the large language model is called to mutate the offspring, and the fitness value is calculated to retain the excellent offspring. After several rounds of iterations, the mutation score of the offspring is calculated, and the offspring with repeated coverage are removed to obtain a test suite. Program repair phase: First, run the test suite to check whether there are any syntax errors. If so, construct a syntax error repair prompt based on the error cause and error code, and send it to the large language model for iterative repair; Then, the test suite is run to check whether there are runtime errors. If there are runtime errors, the error cause and error code are constructed into runtime error repair prompts and sent to the large language model for iterative repair. Finally, if the error still cannot be repaired, the error code is deleted to obtain the final executable test suite. The specific process of the pre-processing stage includes the following steps: 1-1. Obtain the program to be tested, which is a Python module to be tested; 1-2. Construct parameter type analysis prompts based on the program to be tested, the document content of the program to be tested, and the task description, and send them to the large language model; 1-3. Parse the response returned by the large language model, obtain the message content field, match the Python code part in the message field through regular expressions, and combine the parameter type annotations and the program to be tested described in step 1-1 to form a new program to be tested; 1-4. Recursively import the constructed classes and called function information from the modules that the program under test depends on; 1-5. Retrieve the program to be tested and the modules on which the program to be tested depends, and save the variable values ​​used therein as the source of initial values ​​of the program to be tested; 1-6. Use the ast module to parse the program under test into an abstract syntax tree, and use the inspect module to inspect the program under test and retrieve the available classes, functions, and methods as metadata required for testing; 1-7. Construct the metadata required for the test into sets and mappings for storage: the set contains information about all callable or accessible elements in the module under test, and the accessible content in the Python module under test is selected from the set to generate input; the mapping stores callable or accessible elements, including elements in the module under test and elements of modules on which the module under test depends, and the elements include classes, functions, and methods.

2. The method for generating dynamic language test cases based on a large language model as claimed in claim 1, characterized in that: The test case generation phase can be specifically divided into the following phases: an initial phase, an evolution phase, a large language model mutation phase, and a mutation test phase.

3. The method for generating dynamic language test cases based on a large language model as claimed in claim 2, characterized in that: The initial stage specifically includes the following steps: 2-1-1. Set the coverage target to have no control dependent branches, and obtain the initial coverage target Bench = {b1, b2, ..., b n}； 2-1-2. Select a method or function from the set described in 1-7, use the mapping described in 1-7 and the corresponding value randomly selected from the initial value source described in 1-5 to construct an initial test case, and continue to generate test statements until the maximum number of statements is reached. Continue to generate test cases as individuals in the population until the maximum number of populations is reached, and obtain the initial population population = {solution1, solution2, ..., solution n }; 2-1-3. For the initial coverage target Bench = {b1, b2, ..., b n }, for the initial population population = {solution1, solution2, ..., solution n } calculates the fitness of individuals in the population, and divides individuals with the same fitness value in the population into the same front, and obtains the front set front = {front0, front1, front2, ..., front n }, where front0 holds the individual with the smallest fitness, front1 holds the individual with the second smallest fitness, and so on; 2-1-4. For the initial population population = {solution1, solution2, ..., solution n } calculate the crowding distance of the individuals in the cluster and obtain the crowding distance of each individual; 2-1-5. Select excellent individuals based on the frontier set described in 2-1-3 and the crowding distance of each individual described in 2-1-4: traverse all current frontiers. If the number of individuals in the frontier with the smallest fitness does not exceed the population size, all of them are added to the offspring population until the number of remaining individuals in the offspring population is less than the number of individuals in the current smallest frontier. Then, continue to add individuals to the offspring population from large to small according to the crowding distance until the maximum population size is reached, and obtain the offspring population offspring. i ; 2-1-6. Recalculate coverage target Bench = {b1, b2, ..., b n }, remove the covered goals, add the goals that control the dependencies to be satisfied, generate new coverage goals, and record the goals that have been reached and the corresponding test cases through archiving; 2-1-7. Entering the evolution stage, offspring population described in 2-1-5 i Perform crossover and mutation to generate new offspring.

4. The method for generating dynamic language test cases based on a large language model as claimed in claim 3, characterized in that: The specific process of the evolutionary stage is as follows: 2-2-1. Offspring in the current offspring population i In the process, two test cases are selected for crossover operation, a crossover point is randomly selected in the test case, each test case is divided into two parts according to the crossover point, the second half of the crossover point of the two test cases is exchanged, and the variable data in the repair statement is ensured to ensure that the test case after crossover is still valid; 2-2-2. Mutate the crossover test case to obtain a mutated test case. The mutation method is one of the following three methods: Insert operation: offspring in the current generation population i In the test case, a test case is randomly selected and a new statement is inserted at a random position in the test case with probability σ s , after each insertion, the probability of reinsertion is Until no more insertions are made, the insertions may include new method calls or existing method calls; Modify operation: Offspring in the current offspring population i In the test case, test cases are randomly selected. For the existing test cases of length L, each statement is mutated with a probability of 1 / L. For int and float, new values ​​are generated and existing values ​​are modified; for str, characters are deleted, replaced or inserted; for complex types, including lists, sets, and dictionaries, elements are replaced; method and function parameters may be modified or removed; Delete operation: Offspring in the current generation population i In the test case, test cases are randomly selected. For a test case of length L, each statement is deleted with a probability of 1 / L. If the deletion causes dependency problems, the dependent statements need to be repaired or recursively deleted. 2-2-3. After the individuals in the current population are mutated into new individuals through crossover, insertion, modification, and deletion operations, the fitness and crowding distance are calculated as described in 2-1-3 and 2-1-4, and the excellent individuals are selected according to the steps described in 2-1-5 to obtain the next generation population offspring. i+1 , at the same time, update the coverage target as described in 2-1-6, archive the test cases that can reach the coverage target, and choose to continue evolution or enter the large language model mutation stage based on whether the current number of iterations reaches the threshold of the large language model mutation stage; 2-2-4. If the maximum number of test iterations or test time is reached, enter the mutation test phase, streamline the archived test cases that can reach the coverage target, and combine the test cases into a test suite.

5. The method for generating dynamic language test cases based on a large language model as claimed in claim 4, characterized in that: The specific process of the large language model mutation stage is as follows: 2-3-1. Evolution phase After every n rounds of iterations, use the large language model to perform mutation operations based on the large language model on the individuals in the current population, identify the methods or functions in the new program to be tested described in 1-3, and save the methods or functions in the list UTfun; 2-3-2. Obtain the function to be processed by randomly sampling the methods or functions in UTfun; 2-3-3. Construct a difficulty prompt template for the function to be processed, call the large language model Deepseek to analyze the function to be processed, and obtain the difficulty of testing the function; 2-3-4. Construct a mutation prompt template for the function to be processed, call the large language model Deepseek to generate new test data for the difficult problems obtained in 2-3-3; 2-3-5. Perform the crossover and mutation operations described in 2-2-1 and 2-2-2 on the new test data together with the individuals in the current population to generate new individuals. Perform the fitness calculations and crowding distance calculations described in 2-1-3 and 2-1-4. Select the excellent individuals according to the steps described in 2-1-5 to obtain the next generation of the population. At the same time, perform the update coverage target described in 2-1-6, and archive the test cases that achieve the coverage target. Finally, choose to enter the evolution phase or mutation test phase based on whether there are still iterations available.

6. The method for generating dynamic language test cases based on a large language model as claimed in claim 5, characterized in that: The specific process of the mutation test phase is as follows: 2-4-1. Use the MUTPY tool to generate a variant version of the new program to be tested described in 1-3; 2-4-2. Execute the test cases archived in the evolution stage 2-2-3 and the large language model mutation stage 2-3-5 on the variant version, and determine whether the variant version is alive in the following way. If a test case can produce a result different from the original version in the variant version, the variant version is considered to be killed, otherwise the variant version is considered to be alive; 2-4-3. Test cases are retained based on the number of variant versions killed by the test cases. The test cases that kill more variant versions have a higher retention priority. From the archived test cases, assertions that did not work when testing variant versions are deleted to obtain a test suite.

7. The method for generating dynamic language test cases based on a large language model as claimed in claim 6, characterized in that: The specific process of the program repair phase is as follows: 3-1. Input is the test suite described in 2-4-3, check whether the test suite has grammatical errors, if the test suite has grammatical errors, organize the grammatical error information and the grammatical error part code into a grammar repair prompt template in combination with the task description, send the prompt to the large language model Deepseek for 5 rounds of iterative error correction, if the error correction is successful, end early, and obtain the test suite with grammar repair; 3-2. Run the test suite after grammar repair to check whether there are runtime errors in the test suite after grammar repair. If there are runtime errors, organize the runtime error information and the runtime error code into a runtime error repair prompt template in combination with the task description, and send the prompt to the large language model Deepseek for 5 rounds of iterative error correction. If the error correction is successful, it ends early and finally obtains the repaired test suite; 3-3. If there are still errors in the repaired test suite, remove the error code lines and finally get a repaired executable test suite.

Images