A method for detecting defects of an SMT solver empowered by historical defect use cases

Abstract

The application provides a history defect case empowerment SMT solver defect detection method, and the main steps are as follows: 1) automatically collecting history defect cases from a defect tracking system of a solver; 2) extracting a skeleton structure (i.e. a logic structure composed of logical connection words in a formula) of a logic formula and an atomic formula in the history defect cases; 3) abstracting the extracted atomic formula and performing correlation rule learning; and 4) using the atomic formula to instantiate the extracted skeleton structure under the guidance of the correlation rule to obtain a logic formula that can be used for solver testing, and performing differential testing on multiple solvers. The application solves the limitation that effective test inputs are difficult to generate in solver defect detection, and provides a new solution for solver defect detection based on the method for constructing new test cases with strong bug revealing capability based on history defect cases.

Description

Technical Field

[0001] This invention relates to the field of software defect detection, particularly defect detection in SMT solvers, and is a defect detection method for SMT solvers powered by historical defect test cases. Background Technology

[0002] SMT (Satisfiability Modulo Theories) solvers are automated reasoning tools used to determine the satisfiability of first-order logic formulas based on a given theory. Some popular SMT solvers include Z3, CVC4, Yices, and MathSAT. As an important foundational software, SMT solvers are widely used in various applications, such as computer hardware and software verification, symbolic execution, test case generation, and automated program repair. However, as large-scale and complex software systems, SMT solvers themselves have inherent flaws. These flaws inevitably affect the performance of downstream application software. Therefore, effectively discovering these flaws is crucial. Recently, researchers have proposed a series of automated solver testing techniques. By constructing diverse test formulas and utilizing metamorphic testing or differential testing techniques, the correctness of SMT solvers can be verified. Existing methods mainly generate diverse test inputs through methods such as seed input mutation. However, diverse inputs do not necessarily equate to strong error-detection capabilities. Therefore, focusing solely on the diversity of test inputs may have limited impact on the effectiveness of testing techniques, thus restricting the ability of testing tools to detect defects in SMT solvers. To address this limitation, this invention proposes a defect detection method for SMT solvers empowered by historical defect test cases. This invention uses expressions from a large number of historical defect test cases of the SMT solver as a foundation, extracts skeleton information with strong error-detection capabilities, and guides the construction of new expressions for testing the solver by mining association rules, thereby more effectively detecting defects in SMT solvers. Summary of the Invention

[0003] This invention provides a defect detection method for SMT solvers empowered by historical defect test cases, addressing the critical issue of a lack of effective test inputs for SMT solver testing. By utilizing historical defects recorded in different SMT solver defect tracking systems, key logical structures and atomic formulas are extracted. The correlation rules between atomic formulas are then mined, and different logical structures are instantiated under the guidance of these rules to generate effective test inputs. This exposes defects in the solver, thereby improving the reliability and quality of the SMT solver.

[0004] To effectively detect defects in solvers, this invention discloses a method for detecting defects in SMT solvers empowered by historical defect test cases, which specifically includes the following steps:

[0005] Step 1: Collect historical defect test cases recorded in the solver's defect tracking system;

[0006] Step 2: Extract the core logical structure from historical defect test cases, and use all atomic formulas as itemsets to mine the association rules between atomic formulas using an association rule learning algorithm;

[0007] Step 3: Guided by association rules, instantiate the extracted logical structure using atomic formulas from historical defect test cases to synthesize valid SMT solver test inputs;

[0008] Step 4: Solve the generated test inputs using different SMT solvers and record the outputs obtained;

[0009] Step 5: Inconsistent results obtained by different SMT solvers for the same test input, as well as solver crashes or memory errors, are considered potential defects.

[0010] In step 1, when collecting defective use cases for solvers, the primary focus is on commonly used open-source SMT solvers such as Z3 and CVC5, which have been extensively tested and used by researchers and users. For example, nearly 3,000 defective use cases can be collected from Z3's defect tracking system. Mature solvers often contain richer defective use cases, which facilitates the implementation of this method.

[0011] In step 2, for first-order logic formulas, this method considers the sequence of logical connectives and quantifiers in the formula as its key structure. According to the SMT-LIB v2.6 standard input for the SMT solver, logical connectives in logic formulas include: and, or, impiles, and not; quantifiers include forall and exist. After extracting the core logical structure of the defective test cases, they are saved for subsequent test input generation. In addition, atomic formulas are extracted from the defective test cases, and the Apriori algorithm is used to learn association rules for the set of atomic formulas in each defective test case. The obtained association rules and extracted atomic formulas are also saved and used in subsequent test case generation.

[0012] In step 3, when generating test input, a core logic structure extracted in step 2 is first randomly selected. Then, atomic formulas are chosen to fill this selected structure. Here, if a filled atomic formula conforms to a certain association rule, the corresponding atomic formula is selected according to that rule; otherwise, an atomic formula is randomly selected for filling. This process is repeated until the logic structure is instantiated, and then one or more fully filled structures are integrated into a single test input.

[0013] In steps 4 and 5, the solution results of different SMT solvers are compared to check for differences, thereby identifying defects in the solvers. Here, the defects in the solvers that this method can detect mainly include the following three categories: 1) Soundness defects: The solver makes an incorrect judgment on the satisfiability of a formula; 2) Invalid model defects: The solver gives the correct satisfiability but provides an incorrect model, i.e., a solution that cannot satisfy the formula; 3) Crash: The SMT solver terminates abnormally due to an assertion violation or other error. If any possible defect is found, the input that triggers the defect (including test cases and corresponding commands) and the corresponding output are saved for subsequent review.

[0014] This invention uses defective test cases from solver defect trackers as a foundation, extracting valuable information from these cases to generate new test inputs. This information includes the core logic structure, atomic formulas, and the associated rules inherent in the defective test cases. The association rules then guide the generation of test inputs, filling in the core logic structure with the extracted atomic formulas to complete instantiation, which is then used to test the SMT solver. This solves the key problem of lacking effective, high-impact test inputs in SMT solver testing technology. This invention can automatically collect and extract defective test cases from commonly used solver defect trackers. The execution process is simple and efficient, effectively detecting solver defects and improving the solver's reliability and quality.

[0015] Beneficial effects: This method can effectively detect defects in SMT solvers and generate new valid test inputs using historical defect test cases. It effectively addresses the key challenge of lacking test inputs with strong error-revealing capabilities in SMT solver testing, and provides a brand-new solution for solver defect detection. Attached Figure Description

[0016] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention as described above or otherwise will become clearer.

[0017] Figure 1 This is a flowchart of a defect detection method for SMT solvers powered by historical defect test cases.

[0018] Figure 2 It is a flowchart for generating new test inputs using historical defect test cases.

[0019] Figure 3 This is a flowchart of differential verification of different SMT solvers and collection of corresponding results. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, this chapter provides a more detailed description of the invention in conjunction with the accompanying drawings.

[0021] Figure 1 This is a flowchart illustrating a defect detection method for an SMT solver powered by historical defect test cases, as described in this invention. It includes five steps, as follows:

[0022] Step 1: Collect historical defect use cases from the solver's defect tracker;

[0023] Step 2: Extract key information from the collected historical defect test cases, mainly including the core logical structure, and use all atomic formulas as itemsets to mine association rules using an association rule learning algorithm;

[0024] Step 3: Guided by association rules, instantiate the extracted logical structure using atomic formulas from historical defect test cases to synthesize valid SMT solver test inputs;

[0025] Step 4: Solve the generated test inputs using different SMT solvers and record the outputs obtained;

[0026] Step 5: Inconsistent results obtained by different SMT solvers for the same test input, as well as solver crashes or memory errors, are considered potential defects.

[0027] Figure 2 Step 3 is a flowchart for generating test inputs based on the historical defect test cases collected in Step 1. The process is as follows:

[0028] Step 3-1: Extract useful information for generating test input from historical defect test cases, including the core logical structure of the formulas and atomic formulas. The core logical structure refers to the sequence of logical connectives in the formula, and atomic formulas are formulas that no longer contain any logical connectives. After completing this step, proceed to step 3-2.

[0029] Step 3-2: Use the Apriori algorithm to learn the association rules that exist in the atomic formula, save the obtained association rules, and proceed to step 3-3.

[0030] Step 3-3: Select the collected atomic formulas to fill the logic structure. Before each selection, check whether the selected atomic formula conforms to a certain association rule. If it does, select the corresponding atomic formula according to the rule; otherwise, randomly select an atomic formula to fill the logic structure. Repeat the above process until the logic structure is completely filled.

[0031] Figure 3This is a flowchart of steps 4 and 5, which perform differential analysis on different SMT solvers based on the test input generated in step 3. The process is as follows:

[0032] Step 4-1: Use multiple SMT solvers to solve the same test input. Here, the two solvers under test are denoted as SMT solver one and SMT solver two, respectively. After the solution is completed, proceed to step 4-2.

[0033] Step 4-2: Record the output of different solvers, including but not limited to the following information: the solution results of the test formula (returned in standard output form), error messages from the solver (invalid model report or abnormal interruption signal), etc., as the output of the SMT solver. Proceed to step 5-1.

[0034] Step 5-1: Perform differential analysis based on the solver output obtained in Step 3-2. If the solver's solution results are the same and there is no abnormal information, end this round of operation and continue to execute the process starting from Step 4-1 for the next test input; otherwise, proceed to Step 5-2.

[0035] Step 5-2: Record relevant information about the solver defect, including but not limited to the following: the test input that triggered the defect, the command used to enable the solver, and the information returned by different solvers. After completing this step, end the current run and continue with the process starting from step 4-1 for the next test input.

[0036] This invention provides a defect detection method for SMT solvers empowered by historical defect test cases. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A defect detection method for SMT solvers empowered by historical defect test cases, characterized in that, include: This method involves collecting historical defect test cases from the SMT solver, generating test inputs using these historical defect test cases, and validating the test inputs to detect defects in the SMT solver. The method includes the following steps: 1) Collect historical defective use cases of the solver: Historical defect use cases are collected from the SMT solver defect tracking system, wherein the historical defect use cases include the logic formulas that trigger SMT solver defects; 2) Extract information from historical defect test cases: The logical formulas in the historical defect test cases are parsed, and the sequences of logical connectives and quantifiers in the logical formulas are extracted as the core logical structure. Atomic formulas in the logical formulas are also extracted. Association rules are learned for the set of atomic formulas in each historical defect test case to obtain the association rules between atomic formulas. The association rules and the extracted atomic formulas are saved for subsequent test input generation. 3) Generate test inputs using information from historical defect test cases: Select the core logic structure extracted in step 2), and fill it with atomic formulas; during the filling process, determine whether the already filled atomic formulas conform to the association rules: When the already filled atomic formulas meet a certain association rule, the corresponding atomic formulas are selected for filling according to the association rule; when there are no matching association rules, atomic formulas are selected for filling; the above process is repeated until the instantiation of the core logic structure is completed, and one or more instantiated core logic structures are combined to form test input; 4) Verify test inputs and detect SMT solver defects: The test input is provided to multiple SMT solvers, or to different solver configurations of the same SMT solver for solving; the solution results corresponding to different SMT solvers or different solver configurations are compared, and potential defects in the SMT solver are determined based on the differences in solution results, invalid models, or abnormal termination of the solver; wherein, the potential defects include sturdiness defects, invalid model defects, and crash defects.

2. The SMT solver defect detection method empowered by historical defect test cases according to claim 1, characterized in that, The collection of historical defect use cases in SMT solvers includes: collecting defect use cases in the defect tracking system of commonly used SMT solvers; the commonly used SMT solvers include at least one of Z3, cvc5 and Yisics2.

3. The SMT solver defect detection method empowered by historical defect test cases according to claim 1, characterized in that, The information extracted from the historical defect test cases in step 2) includes: the core logic structure in the logic formula; the atomic formulas in the logic formula; and the solver option information corresponding to the historical defect test cases.

4. The SMT solver defect detection method empowered by historical defect test cases according to claim 1, characterized in that, The association rule learning includes: analyzing the co-occurrence relationships among atomic formulas in multiple historical defect test cases; recording frequently co-occurring atomic formulas in historical defects as association rules to guide the generation of new test inputs; wherein, the association rule learning adopts the Apriori algorithm.

5. The SMT solver defect detection method empowered by historical defect test cases according to claim 1, characterized in that, In step 3), in addition to using atomic formulas from historical defect test cases to populate the core logic structure, other atomic formulas are also used to participate in the instantiation of the core logic structure.

Images