A solver performance defect detection method based on multi-objective genetic algorithm

Through the method based on multi-objective genetic algorithm, test cases are automatically generated, which solves the problems of insufficient boot information and insufficient diversity evaluation in solver performance testing, and effectively discovers solver performance defects, ensuring the diversity and coverage of test cases. Reduces errors caused by performance defects.

CN113377676BActive Publication Date: 2024-12-06DALIAN UNIV OF TECH
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202110767573.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-07-07
Publication Date
2024-12-06
Estimated Expiration
2041-07-07

AI Technical Summary

Technical Problem

The existing solver performance testing methods automatically generate test cases inadequate boot information, which leads to the inability to effectively discover the performance defects of the solver, and lacks diversity evaluation during the search process, resulting in high repetition of test cases and inability to cover the test range, ensuring solution The correctness of the device.

Method used

Using a method based on multi-objective genetic algorithm, the target solver is compiled through instrumentation, the test case file is randomly generated, and new test cases are generated using binary tournament selection and cross-mutation operators, and the fitness is evaluated in combination with code coverage and runtime difference. Perform non-dominant Pareto sorting and congestion distance calculations to filter out test cases that can trigger the performance defect of the solver.

Benefits of technology

Through the multi-objective search algorithm, maximizes the running time difference and code coverage, shortens the time for discovering performance defects, prevents test cases from expanding, ensures the diversity of generated results, effectively detects potential performance defects of solvers, and reduces errors caused by performance defects.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113377676B_ABST
    Figure CN113377676B_ABST
Patent Text Reader

Abstract

The present invention belongs to the field of software testing, and relates to a technology for automatically generating test cases for detecting solver performance defects, specifically a solver performance defect detection method based on a multi-objective genetic algorithm. The present invention increases the guidance information in the search process by using a multi-objective search algorithm to maximize the running time difference between the target solver and the benchmark solver, and the code coverage of the target solver, thereby reducing the time required to discover performance defects; by minimizing the complexity of the test cases, the volume of the test cases is prevented from being excessively expanded; at the same time, a dynamic tracking file is used to calculate the similarity between the test cases to ensure the diversity of the generated result set, so that the algorithm can find more potential performance defects. This method is applicable to each version of the solver development, and can help developers discover performance defects in the solver development process, and effectively reduce unforeseen errors that occur during use due to the solver's own performance defects.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the field of software testing, and relates to a technology for automatically generating test cases for detecting solver performance defects, in particular to a solver performance defect detection method based on a multi-objective genetic algorithm. Background Art

[0002] SMT (The Satisfiability Modulo Theories) is a decision problem used to solve the satisfiability of first-order logic formulas. Solvers are software tools designed to solve a variety of theoretical constraint problems. It has been applied to various fields, including software verification and program testing. However, errors in solvers may cause application crashes or even worse. For example, in symbolic execution using solvers, if the solver has performance defects, correct analysis results cannot be obtained. The performance defect of the solver means that for a given input, the solver cannot complete the judgment within the specified time.

[0003] Existing papers and invention patents pay little attention to the performance issues of solvers. Most invention patents focus on how to use solvers to solve problems in specific fields. For example, a solver design method for rocket online trajectory planning (patent application number: CN202011262143.1), this patent proposes a solver design method for rocket trajectory planning problems, which can meet the real-time requirements of rocket online trajectory planning problems. However, if the solver itself has performance defects, the solution speed cannot be guaranteed. Patent A core numerical solver easily expandable software system, calling method and terminal (patent application number: CN202011262143.1) applies the solver to the terminal interface, which is convenient for integrating new numerical solution algorithms and deriving numerical solution interfaces that meet different needs. These invention patents focus on how to use solvers to solve practical problems, but not much attention is paid to the testing of the solver itself. 2018D Blotsky proposed a method of using genetic algorithms to automatically generate test cases and detect solver bugs. However, this method uses a random search strategy and takes the solver running time as the optimization target, which cannot guarantee the convergence of the algorithm and the effectiveness of the results. There are problems such as long algorithm running time and few performance defects found, and it cannot guarantee that the solver performance problems can be effectively found.

[0004] In summary, the current performance testing of solvers has the following two problems: First, there is insufficient guiding information when automatically generating test cases, so it is impossible to find effective timeout cases to detect the performance defects of the solver, and it is impossible to effectively detect the potential performance errors of the solver; second, there is a lack of diversity evaluation in the search process, which will lead to a high degree of repetitiveness of the test cases in the generated result set, which cannot effectively cover the test scope and ensure the correctness of the solver. Summary of the invention

[0005] To solve the above problems, the purpose of the present invention is to provide a method for automatically generating test cases that can trigger solver performance defects. By generating test cases that trigger solver performance defects, the present invention is conducive to detecting potential performance defects in the solver, helping developers to test updated versions of the solver, and ensuring the high availability of the solver.

[0006] The technical solution of the present invention:

[0007] A solver performance defect detection method based on a multi-objective genetic algorithm, the specific steps are as follows:

[0008] Step 1: Instrument and compile the target solver

[0009] Specifically: Since the code statement coverage of the test case needs to be calculated during the algorithm operation, the target solver needs to be instrumented and compiled using a code coverage detection tool; the code coverage detection tool can generate an accurate count of the execution of each statement in the program; instrumented compilation does not affect the normal operation of the target solver, but only generates a file containing the number of statement executions after the operation, which is used to count the code statement coverage of the test case in step 7.

[0010] Step 2: Randomly generate a set of seed test files to form the initial population

[0011] Specifically: randomly generate test case files that conform to the SMT grammar rules. The test case files constitute the initial population, that is, each individual in the population is a test file; the size of the population represents the total number of test files contained in the population.

[0012] Step 3: Screen the seed test files in the initial population through the binary tournament selection method

[0013] Specifically: Use the binary tournament selection method to screen out the test files with excellent performance in the initial population; the binary tournament method is to take out two test files from the population each time, and the initial screening will select the one that makes the target solver run longer as the test file with excellent performance.

[0014] Step 4: Perform crossover mutation on the excellent test files within the population to generate the offspring population

[0015] Specifically, after selecting excellent test files, new test files are generated through crossover operators and mutation operators to form the offspring population; each time, a test file to be operated is selected from the test files with excellent performance, and then the crossover rate and mutation rate are used to determine which operation should be performed on the test file to generate a new test file. The operation is performed continuously until the number of new test files reaches the specified population number.

[0016] The crossover operator selects two test files to exchange each other's assertion statements. The crossover operator is: in order to ensure sufficient crossover amount, select a part of the assertion statements in the two test files for exchange; if the number of assertions in the parent test file does not reach the number to be exchanged, then exchange one of the assertion statements, that is, to ensure that each test case has at least one assertion statement.

[0017] The mutation operator operates on a single test file. The mutation operators include: creating new variables, deleting variables, adding assertion statements, deleting assertion statements, replacing strings and numbers in assertion statements, replacing functions in assertion statements, and replacing nodes of the same type.

[0018] Step 5: Determine whether the algorithm has converged or reached the maximum number of iterations of the algorithm

[0019] Specifically: determine whether the maximum number of iterations of the algorithm has been reached or whether the algorithm has converged. If not, jump to step 6 to iterate the algorithm; if the maximum number of iterations has been reached or the algorithm has converged, jump to step 11; algorithm convergence means that the average fitness of the population has not changed within a certain number of generations.

[0020] Step 6: Merge the good test files in the parent generation and the test files in the child population to generate a new population

[0021] Specifically, the excellent test files in the parent generation and the test files in the child generation population are merged into a new test file set, that is, a new population, for the next iteration operation.

[0022] Step 7: Evaluate the fitness of the test files in the population and calculate the solver running time difference, code coverage, and test case complexity.

[0023] Specifically: a fitness function is evaluated for each test file in the population. Three optimization target indicators are used in the algorithm: the running time difference between the target solver and the benchmark solver, the code coverage of the target solver, and the complexity of the test case. The first optimization target is the running time difference between the target solver and the benchmark solver. Under the specified timeout period, the target solver and the benchmark solver are used to run the test case respectively, and the time taken by the two is counted. The running time difference between the target solver and the benchmark solver is used as the first fitness score. In this process, a code coverage information file and a dynamic tracking file are generated. The second optimization target is the code statement coverage of the target solver. The code statement execution information contained in the code coverage information file is used to calculate the running time of the test case. The code statement coverage rate of the target solver (i.e., the ratio of the number of code statements covered when the test case is executed to the total number of code statements of the target solver) is used as the second fitness score; the third optimization goal is to prevent the test case from over-expanding, and to keep the test case in a smaller size by minimizing the complexity of calculating the test case; the complexity of the test case is affected by the number of assertion statements in the test case and the nesting depth of the nodes in the statement. The specific calculation method uses weighted scores, and the nodes at the nth layer accumulate n points. All nodes in each statement are traversed to calculate the weighted score and as the third fitness; through the operation of this step, each test case will obtain a three-tuple as the fitness score, and participate in the sorting in step 8.

[0024] Step 8: Perform non-dominated Pareto sorting on the test cases in the population and calculate the crowding distance between the test cases

[0025] Specifically, perform non-dominated Pareto sorting: first select all non-dominated test case sets from the population as a Pareto front; then remove these test cases from the population, select non-dominated test cases again to form a new Pareto front, and repeat this operation until all test cases are added to the Pareto front; then use the dynamic tracking file generated by each test case in step 7 when the target solver is running to calculate the crowding distance between test cases for the selection operation in step 9.

[0026] Step 9: Select the test cases with good performance in the population

[0027] Specifically: use dominance-based binary tournament selection to screen excellent test cases; after non-dominated Pareto sorting, each test case has two attributes: non-dominated sorting and crowding distance; when test case I and test case J are compared, the following rules are followed: if the non-dominated sorting of test case I is better than that of test case J, test case I wins; if test case I has the same non-dominated sorting as test case J, but the crowding distance of test case I is greater than that of test case J, test case I wins; repeat the selection operation until the number of test cases with excellent performance reaches the specified population size to form a new population.

[0028] Step 10: Perform crossover mutation on the test cases with good performance to generate new sub-populations

[0029] Specifically: This step is the same as step 4, and a new sub-population is obtained.

[0030] Step 11: Verify the generated test files and filter out the test cases that trigger performance defects

[0031] Specifically: let the target solver re-execute the test file in the population under a sufficiently long timeout period; if the test file still cannot be solved, then add it to the final result set; the test case in the result set will trigger the performance defects of the solver and the algorithm ends.

[0032] Beneficial effects of the present invention: The method of the present invention can detect potential performance defects of the solver by automatically generating test cases. By using a multi-objective search algorithm, the running time difference between the target solver and the benchmark solver and the code coverage of the target solver are maximized to increase the guidance information in the search process and reduce the time required to find the test files that trigger performance defects; by minimizing the complexity of the test cases, the volume of the test cases is prevented from being over-expanded. At the same time, dynamic tracking files are used to calculate the similarity between test cases to ensure the diversity of the generated result set, so that the algorithm can find more potential performance defects. This method is applicable to each version of the solver development, which can help developers discover performance defects in the solver development process and effectively reduce unforeseen errors that occur during use due to the solver's own performance defects. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a schematic diagram of the workflow of a solver performance defect detection method based on a multi-objective genetic algorithm of the present invention. DETAILED DESCRIPTION

[0034] The method of the present invention is described in detail below in conjunction with the accompanying drawings, technical solutions and examples.

[0035] In this embodiment, the method of the present invention is deployed on a Linux server, and the specific configuration of the server is shown in Table 1. In this embodiment, z3str3 is used as the target solver, cvc4 is used as the benchmark solver, the population size is 40, the crossover rate is 0.75, the mutation rate is 0.25, and the maximum iteration generation is 60 generations. The method is composed of a seed test file generation module, a fitness evaluation module, a test file crossover mutation module, a non-dominated Pareto sorting module, a test case crowding calculation module and other modules.

[0036] Table 1 Linux server configuration information table

[0037] Processor model Memory operating system Intel Core i9-10900 32GB Ubuntu 18.04.01

[0038] like Figure 1 As shown, the solver performance defect detection method based on the multi-objective genetic algorithm of the present invention is carried out according to the following process. Plug and compile the target solver; randomly generate an initial set of seed test files to construct an initial population; screen the initial population and generate a sub-population through crossover mutation; merge the test cases with good performance of the parent and the test cases in the sub-population, and evaluate the fitness of the new population; perform non-dominated Pareto sorting on the test cases in the population and calculate the crowding distance; select excellent test cases to perform crossover mutation to generate a new population; verify the test files in the final population, and output the test files that can trigger the performance defects of the target solver.

[0039] The steps in this embodiment are described using z3str3 as the target solver. The specific steps can be adjusted for other solvers. The specific steps are as follows:

[0040] Step 1: Instrument and compile the target solver

[0041] Specifically: use the command LDFLAGS += "-fprofile-arcs" CFLAGS += "-fprofile-arcs-ftest-coverage" CXXFLAGS += "-fprofile-arcs-ftest-coverage" CPPFLAGS += "-fprofile-arcs-ftest-coverage" python3 scripts / mk_make.py to compile z3str3, and then when z3str3 runs, it will generate a ".gcda" file containing the number of times the code statement is run.

[0042] Step 2: Randomly generate a set of seed test files to form the initial population

[0043] Specifically: Use the syntax generator in StringFuzz to generate a random test file and encapsulate it into an Instance class, which includes four parts: setlogic, declare, assert, and checksat. The setlogic part specifies that the syntax used by the test case is string theory; the declare part declares the variables that may be used and their types; the assert part uses a list to store the abstract syntax tree, and each syntax tree is an assertion statement. The checksat part is the fixed syntax of the test case, which allows the solver to check the satisfaction of the test case. The initial population is a collection containing 80 Instance object instances.

[0044] Step 3: Screen the seed test files in the initial population through the binary tournament selection method

[0045] Specifically, Instance object instances are selected from the population in pairs, and the Instance object instances with good performance are placed in the excellent Instance object instance set, and the Instance object instances with poor performance are placed back into the population. This operation is repeated until 40 excellent Instance object instances are obtained to form a new population.

[0046] Step 4: Perform crossover mutation on the excellent test files within the population to generate the offspring population

[0047] Specifically: select two test cases from the excellent test cases, and then generate a random number between 0 and 1. If the generated random number is less than 0.25, use the crossover operator to cross the two test cases to obtain a new sub-test case; if the generated random number is between 0.25 and 1, use the mutation operator to mutate the two test cases separately to obtain two new sub-test cases; add the new sub-test cases to the sub-population, and repeat this operation until the size of the sub-population reaches 40.

[0048] Step 5: Determine whether the algorithm has converged or reached the maximum number of iterations of the algorithm

[0049] Specifically, the average fitness will be calculated in each subsequent iteration. If the average fitness of the population has not changed for 10 consecutive generations, the algorithm is considered to have converged. If the convergence condition is not met, but the algorithm has run for 60 generations, the iteration is also terminated. Then jump back to step 11. If the judgment fails, jump to step 6.

[0050] Step 6: Merge the good test files in the parent generation and the test files in the child population to generate a new population

[0051] Specifically, the set of test cases with good performance in the parent generation is merged with the newly generated offspring population to generate a new population, which will participate in the fitness evaluation in step 7.

[0052] Step 7: Evaluate the fitness of the test files in the population and calculate the solver running time difference, code coverage, and test case complexity.

[0053] Specifically, each test case is sequentially taken out from the new population, converted into a test file that can be recognized by z3str3 and cvc4, and its fitness is evaluated. First, the test file is run using z3str3 and cvc4 respectively under a timeout of 5s to obtain two running times. The running time difference is used as the first fitness score. In this process, a ".gcda" file containing code coverage information is generated, and all ".gcda" files are traversed by a python script. The code statement coverage of the test case is calculated by counting the number of times the code statement is run, and this is used as the second fitness score. Finally, the complexity of the test file is calculated. That is, the nodes in each abstract syntax tree are counted, and the nodes in the first layer are scored one point, and the nodes in the nth layer are scored n points. Then the scores of all nodes are summed up, and the final result is used as the third fitness score. Through this step, each test case will obtain a three-tuple as a fitness score, and participate in the sorting in step 8.

[0054] Step 8: Perform non-dominated Pareto sorting on the test cases in the population and calculate the crowding distance between the test cases

[0055] Specifically: After step 8, all test cases in the population contain a fitness score tuple. To sort, first select the test cases that are not dominated by other test cases, that is, no other test cases have better fitness scores than these test cases. These test cases will be selected as the first Pareto front. Then repeat the above operation for the remaining test cases in the population until all test cases are assigned to the Pareto front. Then calculate the crowding distance for each test case on the front, and use the diffoscope tool to analyze the dynamic tracking file generated in step 7 to calculate the crowding distance of each test case.

[0056] Step 9: Select the test cases with good performance in the population

[0057] Specifically, two test cases are randomly selected from the population and compared. If the non-dominated sorting of test case I is better than that of test case J, test case I wins; if the non-dominated sorting of test case I is the same as that of test case J, but the crowding distance of test case I is greater than that of test case J, test case I wins. Repeat the selection operation until the number of test cases with good performance is 40.

[0058] Step 10: Perform crossover mutation on the test cases with good performance to generate new sub-populations

[0059] Specifically: This step is the same as step 4, and a new sub-population is obtained.

[0060] Step 11: Verify the generated test files and filter out the test cases that trigger performance defects

[0061] Specifically: set the timeout to 40 minutes and let z3str3 solve the test files in the population again. If the test case still times out and the solver cannot solve it, then a test file that can trigger the solver performance problem has been found and recorded in the result set. The result set can then be reported to the z3str3 developers.

[0062] The method of the present invention uses code statement coverage information to guide the search, which can discover the performance defects of the solver in a shorter time; by minimizing the complexity of test cases, it can help developers locate the defect locations more quickly; using a unique test case crowding distance calculation method, the diversity of the final test files can be guaranteed; this method can effectively detect potential performance defects in the target solver and reduce unforeseen errors caused by the solver's own defects.

Claims

1. A solver performance defect detection method based on a multi-objective genetic algorithm, characterized in that: The specific steps are as follows: Step 1: Instrument and compile the target solver Specifically, since the code statement coverage of the test case needs to be calculated during the algorithm operation, the target solver needs to be instrumented and compiled using a code coverage detection tool; the code coverage detection tool can generate an accurate count of the execution of each statement in the program; instrumentation compilation does not affect the normal operation of the target solver, but only generates a file containing the number of statement executions after the operation, which is used to count the code statement coverage of the test case in step 7; Step 2: Randomly generate a set of seed test files to form the initial population Specifically: randomly generate test case files that conform to the SMT syntax rules. The test case files constitute the first generation population, that is, each individual in the population is a test file; the size of the population represents the total number of test files contained in the population; Step 3: Screen the seed test files in the initial population through the binary tournament selection method Specifically, the binary tournament selection method is used to select the test files with good performance in the initial population. The binary tournament method is to take two test files from the population each time, and the initial screening will select the one that makes the target solver run longer as the test file with good performance. Step 4: Perform crossover mutation on the excellent test files within the population to generate the offspring population Specifically, after selecting the excellent test files, new test files are generated through the crossover operator and mutation operator to form the offspring population; each time, the test files to be operated are selected from the excellent test files, and then the crossover rate and mutation rate are used to determine which operation should be performed on the test files to generate new test files, and the operation is performed continuously until the number of new test files reaches the specified population number; The crossover operator selects two test files to exchange each other's assertion statements. The crossover operator is: to ensure sufficient crossover, select a part of the assertion statements in the two test files for exchange; if the number of assertions in the parent test file does not reach the number of exchanges, then exchange one of the assertion statements, that is, to ensure that each test case has at least one assertion statement; The mutation operator operates on a single test file. The mutation operators include: creating new variables, deleting variables, adding assertion statements, deleting assertion statements, replacing strings and numbers in assertion statements, replacing functions in assertion statements, and replacing nodes of the same type. Step 5: Determine whether the algorithm has converged or reached the maximum number of iterations of the algorithm Specifically: determine whether the maximum number of iterations of the algorithm has been reached or whether the algorithm has converged. If not, jump to step 6 to iterate the algorithm. If the maximum number of iterations has been reached or the algorithm has converged, jump to step 11. Algorithm convergence means that the average fitness of the population has not changed within a certain number of generations. Step 6: Merge the good test files in the parent generation and the test files in the child population to generate a new population Specifically, the excellent test files in the parent generation and the test files in the child generation population are merged into a new test file set, i.e., a new population, for the next iteration operation; Step 7: Evaluate the fitness of the test files in the population and calculate the solver running time difference, code coverage, and test case complexity. Specifically, the fitness function is evaluated for each test file in the population. Three optimization target indicators are used in the algorithm: the first optimization target is the running time difference between the target solver and the benchmark solver. Under the specified timeout period, the target solver and the benchmark solver are used to run the test case respectively, and the time taken by the two is counted; the running time difference between the target solver and the benchmark solver is used as the first fitness score. In this process, a code coverage information file and a dynamic tracking file are generated; the second optimization target is the code statement coverage rate of the target solver. The code statement execution information contained in the code coverage information file is used to calculate the code statement coverage rate of the target solver when the test case is running, that is, the ratio of the number of code statements covered when the test case is executed to the total number of code statements of the target solver is counted as the second fitness score; The third optimization goal is to prevent the test cases from over-expanding. By minimizing the complexity of calculating the test cases, the test cases are kept small. The complexity of the test cases is affected by the number of assertion statements in the test cases and the nesting depth of the nodes in the statements. The specific calculation method is to use weighted scores, accumulate n points for the nodes at the nth layer, and traverse all the nodes in each statement to calculate the weighted score and the sum as the third fitness. Through the operation of this step, each test case will obtain a three-tuple as the fitness score, which will participate in the sorting in step 8. Step 8: Perform non-dominated Pareto sorting on the test cases in the population and calculate the crowding distance between the test cases Specifically, perform non-dominated Pareto sorting: first select all non-dominated test case sets from the population as a Pareto front surface; then remove these test cases from the population, select non-dominated test cases again to form a new Pareto front surface, and repeat this operation until all test cases are added to the Pareto front surface; then use the dynamic tracking file generated by each test case in step 7 when the target solver is running to calculate the crowding distance between test cases for the selection operation in step 9; Step 9: Select the test cases with good performance in the population Specifically, the dominance-based binary tournament selection is used to select excellent test cases; after non-dominated Pareto sorting, each test case has two attributes: non-dominated sorting and crowding distance; when test case I and test case J are compared, the following rules are followed: if the non-dominated sorting of test case I is better than that of test case J, test case I wins; if test case I has the same non-dominated sorting as test case J, but the crowding distance of test case I is greater than that of test case J, test case I wins; repeat the selection operation until the number of test cases with excellent performance reaches the specified population size to form a new population; Step 10: Perform crossover mutation on the test cases with good performance to generate new sub-populations Specifically: the same operation as step 4 is performed to obtain a new sub-population; Step 11: Verify the generated test files and filter out the test cases that trigger performance defects Specifically: let the target solver re-execute the test files in the population under a sufficiently long timeout period; If the test file still cannot be solved, it will be added to the final result set; the test cases in the result set will trigger the performance defects of the solver and the algorithm will end.

Citation Information

Patent Citations

  • Solver design method for rocket online trajectory planning

    CN112287560A

  • Statement coverage and defect detection based multi-objective test data reduction method

    CN104281522A

  • Method and device for testing code file

    CN110059009A

  • Test case priority ranking method based on multi-objective optimization

    CN111666209A

  • Methods for generating test suites and devices thereof

    US20150261657A1