Feedback-based SMT solver performance test case reduction method

Through feedback-based methods and binary search method and technical means of replacing nodes in sequence, the problem of time-consuming reduction of SMT solver performance failure test cases in the prior art is solved, and the rapid and effective test case reduction is achieved, which improves the improvement of Test efficiency and quality.

CN113377675BActive Publication Date: 2024-12-06DALIAN UNIV OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110767566.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-07-07
Publication Date
2024-12-06
Estimated Expiration
2041-07-07

AI Technical Summary

Technical Problem

The prior art is difficult to effectively reduce the test cases that trigger performance failures of SMT solvers in a short time, resulting in a long time-consuming and affecting developers to quickly analyze and locate performance failures.

Method used

Using feedback-based methods, by verifying whether the test cases can trigger performance failures, using the binary search method to compare code coverage similarity, and simplifying the nesting depth of the formula by replacing the assertion nodes in sequence, generating minimization test cases.

Benefits of technology

Effectively reduce test cases that trigger performance failures of SMT solvers in a short time, improve test efficiency and quality, help developers quickly find the cause of performance failures, and reduce test costs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113377675B_ABST
    Figure CN113377675B_ABST
Patent Text Reader

Abstract

The present invention belongs to the field of software testing, and relates to a technology for simplifying test cases that trigger SMT solver performance failures, specifically a method for simplifying SMT solver performance test cases based on feedback. First, it is verified whether the input test case can trigger the SMT solver performance failure, and then the number of test case assertions that trigger the SMT solver performance defect is simplified by comparing the code coverage similarity using a binary search method, and finally the formula nesting depth of each assertion in the test case is simplified by sequentially replacing the nodes in the assertion, thereby generating a minimized test case. The present invention can effectively simplify the test case that triggers the SMT solver performance failure, and use the simplest possible test case to fully meet the given test objectives, thereby improving the test efficiency and reducing the test cost, and can help developers better find the performance failures related to the SMT solver and improve the quality of the solver.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the field of software testing, and relates to a technology for simplifying test cases that trigger SMT solver performance failures, and specifically to a feedback-based SMT solver performance test case simplification method. Background Art

[0002] The Satisfiability Modulo Theories (SMT) solver is a tool for verifying whether a first-order logic formula can be satisfied. It has been widely used in formal methods, programming languages, software engineering, and computer security. However, as a software, the SMT solver will inevitably have performance failures, such as solution timeouts, which makes it difficult for other tools that rely on the SMT solver to output reliable results. Therefore, for test cases that trigger SMT solver performance failures, effective reduction methods must be developed to help developers better analyze and locate SMT solver-related performance failures and improve the performance of SMT solvers.

[0003] Although some existing papers and invention patents can be used to simplify test cases, there are still many deficiencies in simplifying SMT solver performance test cases. For example, a test case set simplification method based on error detection capability (patent application number: CN202010146247.X), this patent first constructs a relationship matrix between the original test case set and the corresponding test requirement set; then constructs a matrix based on the error detection capability of each element in the test case set and calculates the error detection capability of the test case set; finally, the essential strategy and the greedy strategy of error detection capability are used to simplify the test cases. However, this method is not targeted at SMT test cases. In 2020, Mansur et al. proposed the STORM tool, which generates SMT solver test cases through disassembly and combination methods and simplifies the test cases that trigger bugs, but STORM cannot simplify performance fault test cases well.

[0004] Since it may take a long time to trigger the SMT timeout problem, it takes a long time to reduce the test cases that trigger performance failures using existing methods. Therefore, it is necessary to propose a method to reduce the test cases that trigger SMT solver performance failures in a short time. Summary of the invention

[0005] To solve the above problems, the present invention provides a feedback-based SMT solver performance test case simplification method, which can simplify the test cases that trigger the SMT solver performance failure in a short time, help developers find the performance failure of the SMT solver faster, and improve the quality of the SMT solver. The method of the present invention first verifies whether the input test case can trigger the SMT solver performance failure, and then uses a binary search method to simplify the number of test case assertions that trigger the SMT solver performance defects by comparing the code coverage similarity, and finally simplifies the formula nesting depth of each assertion in the test case by sequentially replacing the nodes in the assertion, thereby generating a minimized test case. Code coverage is a metric in software testing, which describes the proportion and degree of source code being tested in the program, and the resulting proportion is called code coverage.

[0006] The present invention can be used for SMT solvers, such as Z3, CVC4, etc.; the relevant steps of the present invention can also be adapted according to the characteristics of a specific solver, thereby simplifying the test cases of the solver.

[0007] The technical solution of the present invention:

[0008] A feedback-based SMT solver performance test case reduction method, the specific steps are as follows:

[0009] Step 1: Verify whether the input test case can trigger the performance failure.

[0010] For the test cases to be simplified, first manually verify the test cases; input the test cases into the SMT solver to detect whether they can trigger performance faults; if the test cases can trigger performance defects, simplify the test cases; otherwise, output the information that the test cases do not meet the simplification conditions.

[0011] Step 2: Parse the test case into an Instance class instance.

[0012] Parse the test cases that trigger performance defects into operable Instance class instances. The Instance class consists of four parts: set logic, declare, assert, and check sat. The set logic part specifies that the syntax used by the test case is string theory; the declare part declares the variables that may be used and their types; the assert part uses a list to store abstract syntax trees, and each syntax tree is an assertion statement. The check sat part is the fixed syntax of the test case, allowing the solver to check the satisfaction of the test case.

[0013] Abstract Syntax Tree (AST) is an abstract representation of the syntax structure of source code. It represents the syntax structure of a programming language in a tree form, and each node in the tree represents a structure in the source code.

[0014] Step 3: Set the code coverage similarity threshold and the execution time of the test case in the SMT solver. The test time limit can be set by the tester according to the requirements.

[0015] Step 4: Use binary search to reduce the number of test case assertions. The specific steps are as follows:

[0016] Step 4.1: First, store a copy of the test case, set Dmin to 0, and set Dmax to the total number of test case assertions that need to be reduced.

[0017] Step 4.2: Compare Dmax and Dmin. If Dmax is less than or equal to the value of Dmin, the number of test case assertions has reached the minimum. End the loop and save the test case; otherwise, go to step 4.3.

[0018] Step 4.3: Randomly take out the assertion number D=1 / 2(Dmax+Dmin) of the new test case from the test case; input the new test case into the solver to run and obtain the code coverage information, compare the coverage similarity between the new test case and the original test case, if the coverage similarity can reach the similarity threshold, go to step 4.4; if the coverage similarity is less than the similarity threshold, go to step 4.5.

[0019] Step 4.4: Set Dmax to the value of D and go to Step 4.2.

[0020] Step 4.5: Set the Dmin value to the value of D and go to step 4.2.

[0021] Step 5: Use the node replacement method to simplify the test case formula nesting depth with the minimum number of assertions; a test case has multiple assertions, each of which is encoded as an abstract syntax tree AST; traverse the AST encoded by each assertion in sequence and replace the nodes; the specific steps are as follows:

[0022] Step 5.1: Sequentially traverse the assertions in the test case to determine whether the assertion is the last assertion in the test case; if the assertion is the last assertion in the test case, go to step 5.2; otherwise, go to step 5.3.

[0023] Step 5.2: The formula nesting depth of this test case has reached the minimum, that is, at this time, this test case has the least number of assertions and the assertions with the minimum nesting depth, and is the simplest test case; save the simplest test case and end the loop.

[0024] Step 5.3: Sequentially replace nodes in assertions: Use constants of the same type to replace the node to generate a new test case, and run the test case in the SMT solver to collect coverage information.

[0025] Step 5.4: Compare the new test case coverage with the original test case coverage. If the similarity reaches the threshold, or the node is the last node of the AST, go to step 5.1; if the similarity threshold is not reached, replace the node back to the original node, and continue to replace the next node in the AST, go to step 5.3.

[0026] The beneficial effects of the present invention are as follows: the present invention can effectively simplify the test cases that trigger the performance failures of the SMT solver, and use the simplest possible test cases to fully meet the given test objectives, thereby improving the test efficiency and reducing the test cost, and can help developers better find the performance failures related to the SMT solver and improve the quality of the solver. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a schematic diagram of the workflow of a feedback-based SMT solver performance test case simplification method of the present invention. DETAILED DESCRIPTION

[0028] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0029] The workflow of the present invention is as follows Figure 1 As shown, the method of the present invention is deployed on a Linux server, and the specific configuration of the server is shown in Table 1. According to the experimental requirements, the z3str3 solver is selected and the corresponding software is installed according to the test steps.

[0030] Table 1 Linux server configuration information table

[0031] Processor model Memory operating system Intel Core i9-10900 32GB Ubuntu 20.04.2LTS

[0032] Step 1: Verify whether the input test case can trigger the performance failure.

[0033] Specifically, for the test case to be simplified, first run the test case in the z3str3 solver to detect whether it can trigger a performance fault. If the test case can trigger a performance fault, simplify the test case; otherwise, output the information that the test case does not meet the simplification conditions.

[0034] Step 2: Parse the test case into an Instance class instance.

[0035] The z3str3 solver parses the test cases that trigger the performance defects into actionable Instance class instances.

[0036] Step 3: Set the code coverage similarity threshold sim to 0.9 and set the test case running time in the z3str3 solver to 5s.

[0037] Step 4: Use binary search to reduce the number of test case assertions. The specific steps are as follows:

[0038] a: Use the z3str3 solver, first store the test cases in the solver cache, and then use binary search to reduce the number of assertions. Set Dmin to 0 and Dmax to the total number of assertions for each test case.

[0039] b: Compare Dmax and Dmin. If Dmax is less than or equal to the value of Dmin, the number of assertions of the test case has reached the minimum. End the loop and save the test case; otherwise, go to step c.

[0040] c: Randomly take out a new test case from the test case, and the number of assertions is D = 1 / 2 (Dmax + Dmin), then put the new test case into z3str3 and run it for 5 seconds to get the code coverage information, and compare the coverage similarity s1 between the new test case and the original test case. If s1 ≥ sim1, go to step d, otherwise go to step e.

[0041] d: Set Dmax to D and go to step b;

[0042] e: Set Dmin to D and go to step b.

[0043] Step 5: Use the node replacement method to simplify the test case formula nesting depth with the minimum number of assertions. A test case has multiple assertions, each of which is encoded as an AST. Traverse the AST encoded by each assertion in order and replace the nodes. The specific steps are as follows:

[0044] Step a: Sequentially traverse the assertions in the test case. Determine whether the assertion is the last assertion in the test case. If the assertion is the last assertion in the test case, go to step b; otherwise, go to step c.

[0045] Step b: The formula nesting depth of the test case has reached the minimum, that is, the test case has the minimum number of assertions and the minimum nesting depth, and is the simplest test case. Save the simplest test case and end the loop.

[0046] Step c: Sequentially traverse the nodes in the AST encoded in the assertion. Replace the node with a constant of the same type of the node to generate a new test case, and run this test case in the z3str3 solver for 5s to collect coverage information.

[0047] Step d: Compare the similarity between the coverage rate of the new test case and that of the original test case. If the similarity s1 ≥ sim1, or this node is the last node of the AST, go to Step a; if s1 < sim1, then backtrack to this node, replace this node back to its original node, and continue to replace the next node in the AST, then go to Step c.

[0048] Step 6: Through the reduction in Steps 4 and 5, simpler test cases can be obtained, which can more effectively help developers locate the reasons for the failure of the SMT solver's performance, thus quickly fixing the failure.

[0049] Step 7: Write the reduced test cases into the test report and submit them to the developers.

Claims

1. A feedback-based SMT solver performance test case reduction method, characterized in that: The specific steps are as follows: Step 1: Verify whether the input test case can trigger performance failures; For the test cases to be simplified, first manually verify the test cases; input the test cases into the SMT solver to detect whether they can trigger performance faults; if the test cases can trigger performance faults, simplify the test cases; otherwise, output the information that the test cases do not meet the simplification conditions; Step 2: Parse the test case into an Instance class instance; Parse the test cases that trigger performance defects into operable Instance class instances; the Instance class consists of four parts: set logic, declare, assert, and check sat; the set logic part specifies that the syntax used by the test case is string theory; the declare part declares the variables that may be used and their types; the assert part uses a list to store abstract syntax trees, each syntax tree is an assertion statement; the check sat part is the fixed syntax of the test case, allowing the solver to check the satisfaction of the test case; Step 3: Set the code coverage similarity threshold and the execution time of the test case in the SMT solver; the test time limit can be set by the tester according to the needs; Step 4: Use binary search to reduce the number of test case assertions. The specific steps are as follows: Step 4.1: First, store a copy of the test case, set Dmin to 0, and set Dmax to the total number of test case assertions that need to be reduced; Step 4.2: Compare Dmax and Dmin. If Dmax is less than or equal to the value of Dmin, the number of test case assertions has reached the minimum. End the loop and save the test case. Otherwise go to step 4.3; Step 4.3: Randomly extract the assertion number D=1 / 2(Dmax+Dmin) of the new test case from the test case; input the new test case into the solver to run and obtain the code coverage information, compare the coverage similarity between the new test case and the original test case, if the coverage similarity can reach the similarity threshold, go to step 4.4; If the coverage similarity is less than the similarity threshold, go to step 4.5; Step 4.4: Set Dmax to the value of D and go to step 4.2; Step 4.5: Set the Dmin value to the value of D and go to step 4.2; Step 5: Use the node replacement method to simplify the test case formula nesting depth with the minimum number of assertions; a test case has multiple assertions, each of which is encoded as an abstract syntax tree AST; traverse the AST encoded by each assertion in sequence and replace the nodes; the specific steps are as follows: Step 5.1: Sequentially traverse the assertions in the test case to determine whether the assertion is the last assertion in the test case; if the assertion is the last assertion in the test case, proceed to step 5.2; Otherwise go to step 5.3; Step 5.2: The formula nesting depth of the test case has reached the minimum, that is, the test case has the least number of assertions and the assertions with the minimum nesting depth, and is the simplest test case; save the simplest test case and end the loop; Step 5.3: Replace the nodes in the assertion sequentially; Use the same type of constant to replace the node to generate a new test case, and run the test case in the SMT solver to collect coverage information; Step 5.4: Compare the new test case coverage with the original test case coverage. If the similarity reaches the threshold, or the node is the last node of the AST, go to step 5.1; if the similarity threshold is not reached, replace the node back to the original node, and continue to replace the next node in the AST, go to step 5.3.