Test method, device and computer readable storage medium
By extracting and classifying path constraint features, using machine learning to predict the most suitable solver, the problem of inefficient solution in symbol execution technology is solved, and the efficiency and coverage of software testing is improved.
Patent Information
- Application Number
- CN202010541780.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-06-15
- Publication Date
- 2024-12-20
- Estimated Expiration
- 2040-06-15
AI Technical Summary
In the prior art, symbol execution technology has poor efficiency when solving path constraints, resulting in low software testing.
By extracting the path constraint features of the tested program, inputting them into the pre-trained classifier, obtaining the corresponding solver, and using machine learning technology to predict the most suitable solver for path constraint solutions, improving testing efficiency.
The efficiency and test coverage of software testing are improved, and the path constraint solution process is optimized by dynamically adjusting the classifier parameters to select the solver with the shortest solution time.
Smart Images
Figure CN113806207B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a testing method, a testing device and a computer-readable storage medium. Background Art
[0002] In software fuzz testing, there may be situations where the test coverage cannot meet the requirements. In related technologies, the problem of low coverage in fuzz testing can be improved through symbolic execution technology. Summary of the invention
[0003] After analysis, the inventors found that symbolic execution technology usually uses a single solver to solve constraints, but different solvers have large differences in efficiency when solving different constraints. Therefore, the related technology still has the problem of low testing efficiency.
[0004] A technical problem to be solved by the embodiments of the present invention is: how to improve testing efficiency.
[0005] According to a first aspect of some embodiments of the present invention, a testing method is provided, comprising: extracting features of a path constraint of a program under test; inputting the features of the path constraint into a pre-trained classifier to obtain a classification result of the path constraint, wherein each classification result of the classifier corresponds to a preset solver; using the solver corresponding to the classification result of the path constraint to solve the path constraint; and using the solution result to test the program under test.
[0006] In some embodiments, the testing method also includes: extracting features of the path constraints used for training; inputting the features of the path constraints used for training into the classifier to be trained to obtain classification results of the path constraints used for training, wherein each classification result of the classifier corresponds to a preset solver; inputting the path constraints used for training into each preset solver, and determining the value of the cost function of the classifier according to the solution time of each preset solver; and adjusting the parameters of the classifier according to the value of the cost function.
[0007] In some embodiments, for each path constraint used for training: when the solution is correct and not timed out, the value of the cost function is determined based on the solution time of the solver corresponding to the classification result of the path constraint used for training; when the solution is correct but times out, the value of the cost function is a first preset value; when an error occurs in the solution, the value of the cost function is a second preset value.
[0008] In some embodiments, when the solution is correct and there is no timeout, the value of the cost function is determined based on the normalized result of the solution time of the solver corresponding to the classification result of the path constraint used for training, wherein the maximum and minimum values of the solution time of the path constraint used for training are normalized according to each preset solver.
[0009] In some embodiments, the features of the path constraint include one or more of grammatical statistical features, grammatical structural features, variable features, or logical features.
[0010] In some embodiments, extracting features of the path constraints of the tested program includes: converting the acquired path constraints of the tested program into a syntax tree, wherein non-leaf nodes of the syntax tree represent operations, leaf nodes represent objects being operated on, and the objects being operated on include at least one of variables or constants; extracting features of the path constraints from the syntax tree.
[0011] In some embodiments, the characteristics of the path constraint include one or more of statistical characteristics of operation in the syntax tree, structural characteristics of the syntax tree, variable characteristics, or logical characteristics.
[0012] In some embodiments, converting the acquired path constraints into a syntax tree includes: storing each constraint in the acquired path constraints in sequence into a constraint cache table; generating a path constraint in SMT-LIB format according to the constraint cache table; and generating a syntax tree according to the path constraint in SMT-LIB format.
[0013] In some embodiments: the testing method further comprises pre-testing the tested program; and, in case of a blockage in the pre-test, extracting path constraints from statements between a start position and a blockage position of the tested program.
[0014] In some embodiments, the pre-test is a fuzz test.
[0015] In some embodiments, the path constraints of the program under test are obtained through a symbolic execution engine.
[0016] According to a second aspect of some embodiments of the present invention, a testing device is provided, comprising: an extraction module, configured to extract features of a path constraint of a program under test; a classification module, configured to input the features of the path constraint into a pre-trained classifier to obtain a classification result of the path constraint, wherein each classification result of the classifier corresponds to a preset solver; a solution module, configured to use the solver corresponding to the classification result of the path constraint to solve the path constraint; and a testing module, configured to use the solution result to test the program under test.
[0017] In some embodiments, the testing device also includes: a training module, configured to extract features of path constraints for training; input the features of the path constraints for training into the classifier to be trained to obtain classification results of the path constraints for training, wherein each classification result of the classifier corresponds to a preset solver; input the path constraints for training into each preset solver, determine the value of the cost function of the classifier according to the solution time of each preset solver; and, according to the value of the cost function, adjust the parameters of the classifier.
[0018] According to a third aspect of some embodiments of the present invention, there is provided a testing device, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute any one of the aforementioned testing methods based on instructions stored in the memory.
[0019] According to a fourth aspect of some embodiments of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein the program implements any one of the aforementioned testing methods when executed by a processor.
[0020] Some embodiments of the above invention have the following advantages or beneficial effects: the embodiments of the present invention use machine learning technology to predict the path constraint solver of symbolic execution, thereby improving the efficiency of software testing while improving the test coverage.
[0021] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0023] Figure 1 A schematic flow chart of a testing method according to some embodiments of the present invention is shown.
[0024] Figure 2 A schematic flow chart of a training method according to some embodiments of the present invention is shown.
[0025] Figure 3 A schematic structural diagram of a testing device according to some embodiments of the present invention is shown.
[0026] Figure 4 The following are schematic structural diagrams of testing devices according to other embodiments of the present invention.
[0027] Figure 5 A schematic structural diagram of a testing device according to some further embodiments of the present invention is shown. DETAILED DESCRIPTION
[0028] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present invention and its application or use. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0029] The relative arrangement of components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention unless specifically stated otherwise.
[0030] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.
[0031] Technologies, methods, and apparatus known to ordinary technicians in the relevant field may not be discussed in detail, but where appropriate, such technologies, methods, and apparatus should be considered part of the authorization specification.
[0032] In all examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.
[0033] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.
[0034] Figure 1 FIG. 2 is a flow chart showing a test method according to some embodiments of the present invention. Figure 1 As shown, the testing method of this embodiment includes steps S102 to S110.
[0035] In step S102, the path constraints of the program under test are obtained. The path constraints can be obtained by running a symbolic execution engine.
[0036] In some embodiments, path constraints of the program under test are obtained through a symbolic execution engine.
[0037] In some embodiments, the tested program is pre-tested; if the pre-test is blocked, the path constraints are extracted from the statements between the start position and the blocking position of the tested program, so that the solution can be more targeted.
[0038] In some embodiments, the pre-test is a fuzz test, which can be performed using a fuzz test engine such as libfuzzer or AFL.
[0039] For example, the fuzz testing engine is first used to instrument the program; then, the instrumented program is run. When a path is found to be blocked, the symbolic execution engine is called, and the symbolic execution engine extracts the path constraints at the time of the current blockage.
[0040] Path constraints can be stored in a constraint cache table and represented in SMT-LIB (Constraint Solver Library) format, syntax tree, etc.
[0041] In step S104, features of the path constraints are extracted.
[0042] In some embodiments, the features of the path constraint include one or more of grammatical statistical features, grammatical structural features, variable features, or logical features. For example, grammatical statistical features include operation categories, operation times, etc., grammatical structural features include operation complexity, etc., variable features include variable operation formulas, etc., and logical features include branches in the operation process, etc. Thus, the main features of the path constraint can be described from various angles, and the extracted features can be understood by the model.
[0043] In some embodiments, the obtained path constraint is converted into a syntax tree, wherein the non-leaf nodes of the syntax tree represent operations, the leaf nodes represent objects to be operated, and the objects to be operated include at least one of variables or constants. Since the tree structure can more clearly and explicitly describe the variables, operations, and the relationship between them in the path constraint, by converting the path constraint into a syntax tree, the features of the path constraint can be extracted more efficiently and accurately.
[0044] In some embodiments, the features of the path constraint include one or more of the statistical features of the operation in the syntax tree, the structural features of the syntax tree, the variable features, or the logical features. For example, the statistical features of the operation (such as AND, OR, Sum, etc.) can be obtained by counting the non-leaf nodes, the structural features of the syntax tree can be obtained by counting the depth of the syntax tree, the number of nodes, etc., the variable features can be obtained by the relationship between the leaf nodes and the non-leaf nodes directly or indirectly connected to the leaf nodes, and the logical features can be obtained by the branches of the syntax tree.
[0045] In some embodiments, a syntax tree may be generated in the following manner: storing each constraint in the acquired path constraint in sequence into a constraint cache table; generating a path constraint in SMT-LIB format according to the constraint cache table; and generating a syntax tree according to the path constraint in SMT-LIB format.
[0046] In step S106, the features of the path constraint are input into a pre-trained classifier to obtain a classification result of the path constraint, wherein each classification result of the classifier corresponds to a preset solver, such as Z3, CVC4, etc.
[0047] In step S108, the path constraint is solved using a solver corresponding to the classification result of the path constraint.
[0048] In step S110, the program to be tested is tested using the solution result, which is a test value that can guide the program to a corresponding path, so that the required test can be accurately performed.
[0049] The method of the above embodiment uses machine learning technology to predict the path constraint solver of symbolic execution, thereby improving the efficiency of software testing while improving the test coverage.
[0050] Reference below Figure 2 An embodiment of a method for training a classifier is described.
[0051] Figure 2 FIG. 2 shows a flow chart of a training method according to some embodiments of the present invention. Figure 2 As shown, the training method of this embodiment includes steps S202 to S210.
[0052] In step S202, path constraints for training are obtained.
[0053] In step S204, features of the path constraints for training are extracted.
[0054] In step S206, the features of the path constraint for training are input into the classifier to be trained to obtain the classification results of the path constraint for training, wherein each classification result of the classifier corresponds to a preset solver.
[0055] In step S208, the path constraints used for training are input into each preset solver, and the value of the cost function of the classifier is determined according to the solution time of each preset solver.
[0056] In some embodiments, for each path constraint used for training: when the solution is correct and not timed out, the value of the cost function is determined according to the solution time of the solver corresponding to the classification result of the path constraint used for training, for example, the value of the cost function is determined according to the normalized result of the solution time of the solver corresponding to the classification result of the path constraint used for training. Thus, the classifier can be fed back according to the actual solution time of the solver corresponding to the classification result, so as to prompt the classifier to adjust in the direction of selecting the solver with the shortest solution time.
[0057] In some embodiments, when the solution is correct but times out, the value of the cost function is a first preset value; when the solution is erroneous, the value of the cost function is a second preset value. Thus, special considerations can be made for situations where the solution time is too long or the solution cannot be found, so as to reduce the probability that the classifier outputs a solver that takes too long to find a solution or cannot find a solution.
[0058] The first preset value and the second preset value may be, for example, greater than the cost function value when the solution is correct and not timed out, so as to give the classifier a greater penalty when the solution takes too long or fails to be solved.
[0059] In some embodiments, formula (1) may be used as a cost function.
[0060]
[0061] In formula (1), "if solved" means when the solution is correct and not timed out; "if timeout" means when the solution is correct but timed out; "if error" means when an error occurs in the solution. m represents the preset maximum value when the solution is correct and not timed out; T represents the solver solution time corresponding to the classification result of the path constraint used for training; T max and T min Respectively represent the maximum and minimum solution time of each preset solver for the path constraint used for training; j is the first preset value; k is the second preset value; and there are m <j<k。
[0062] In step S210, the parameters of the classifier are adjusted according to the value of the cost function.
[0063] The above steps can be iteratively executed until the condition for stopping training is met.
[0064] Through the method of the above embodiment, the classifier can be iteratively adjusted according to the solution time of each solver for the same path constraint and the prediction result of the classifier, so that the trained classifier can predict the solver with the least calculation time as much as possible, thereby improving the test efficiency.
[0065] Reference below Figure 3 An embodiment of the test device of the present invention is described.
[0066] Figure 3 FIG. 2 shows a schematic diagram of a test device according to some embodiments of the present invention. Figure 3 As shown, the testing device 30 of this embodiment includes: an extraction module 310, configured to extract the characteristics of the path constraint of the program under test; a classification module 320, configured to input the characteristics of the path constraint into a pre-trained classifier to obtain a classification result of the path constraint, wherein each classification result of the classifier corresponds to a preset solver; a solution module 330, configured to use the solver corresponding to the classification result of the path constraint to solve the path constraint; and a testing module 340, configured to use the solution result to test the program under test.
[0067] In some embodiments, the testing device 30 also includes: a training module 350, configured to extract features of path constraints for training; input the features of the path constraints for training into the classifier to be trained to obtain classification results of the path constraints for training, wherein each classification result of the classifier corresponds to a preset solver; input the path constraints for training into each preset solver, determine the value of the cost function of the classifier according to the solution time of each preset solver; and, according to the value of the cost function, adjust the parameters of the classifier.
[0068] In some embodiments, for each path constraint used for training: when the solution is correct and not timed out, the value of the cost function is determined based on the solution time of the solver corresponding to the classification result of the path constraint used for training; when the solution is correct but times out, the value of the cost function is a first preset value; when an error occurs in the solution, the value of the cost function is a second preset value.
[0069] In some embodiments, when the solution is correct and there is no timeout, the value of the cost function is determined based on the normalized result of the solution time of the solver corresponding to the classification result of the path constraint used for training, wherein the maximum and minimum values of the solution time of the path constraint used for training are normalized according to each preset solver.
[0070] In some embodiments, the features of the path constraint include one or more of grammatical statistical features, grammatical structural features, variable features, or logical features.
[0071] In some embodiments, the extraction module 310 is further configured to convert the acquired path constraints of the tested program into a syntax tree, wherein the non-leaf nodes of the syntax tree represent operations, the leaf nodes represent the objects being operated, and the objects being operated include at least one of variables or constants; and extract the features of the path constraints from the syntax tree.
[0072] In some embodiments, the characteristics of the path constraint include one or more of statistical characteristics of operation in the syntax tree, structural characteristics of the syntax tree, variable characteristics, or logical characteristics.
[0073] In some embodiments, the extraction module 310 is further configured to store each of the acquired path constraints in sequence into a constraint cache table; generate a path constraint in SMT-LIB format according to the constraint cache table; and generate a syntax tree according to the path constraint in SMT-LIB format.
[0074] In some embodiments: the testing device 30 also includes a pre-test module 360, which is configured to pre-test the program under test; the extraction module 310 is further configured to extract path constraints from the statements between the start position and the blocking position of the program under test when the pre-test is blocked.
[0075] In some embodiments, the pre-test is a fuzz test.
[0076] In some embodiments, the path constraints of the program under test are obtained through a symbolic execution engine.
[0077] Figure 4 FIG. 2 shows a schematic diagram of the structure of a testing device according to some other embodiments of the present invention. Figure 4 As shown, the testing device 40 of this embodiment includes: a memory 410 and a processor 420 coupled to the memory 410 , and the processor 420 is configured to execute the testing method in any one of the aforementioned embodiments based on instructions stored in the memory 410 .
[0078] The memory 410 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, and other programs.
[0079] Figure 5 FIG. 2 shows a schematic diagram of the structure of a testing device according to some other embodiments of the present invention. Figure 5As shown, the test device 50 of this embodiment includes: a memory 510 and a processor 520, and may also include an input / output interface 530, a network interface 540, a storage interface 550, etc. These interfaces 530, 540, 550 and the memory 510 and the processor 520 may be connected, for example, via a bus 560. Among them, the input / output interface 530 provides a connection interface for input / output devices such as a display, a mouse, a keyboard, and a touch screen. The network interface 540 provides a connection interface for various networked devices. The storage interface 550 provides a connection interface for external storage devices such as SD cards and USB flash drives.
[0080] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein the program implements any one of the aforementioned testing methods when executed by a processor.
[0081] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable non-transient storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0082] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0083] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0084] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0085] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A testing method comprising: Inputting the extracted features of the path constraint for training into the classifier to be trained to obtain the classification results of the path constraint for training, wherein each classification result of the classifier corresponds to a preset solver; Input the path constraint for training into each preset solver, and determine the value of the cost function of the classifier according to the solution time of each preset solver, wherein, when the solution is correct and not timed out, the value of the cost function is determined according to the solution time of the solver corresponding to the classification result of the path constraint for training; when the solution is correct but times out, the value of the cost function is a first preset value; when an error occurs in the solution, the value of the cost function is a second preset value; According to the value of the cost function, adjusting the parameters of the classifier; Extract the characteristics of the path constraints of the program under test; Inputting the features of the path constraint into a pre-trained classifier to obtain a classification result of the path constraint, wherein each classification result of the classifier corresponds to a preset solver; Using a solver corresponding to the classification result of the path constraint to solve the path constraint; and The program under test is tested using the solution result.
2. The testing method according to claim 1, wherein: When the solution is correct and there is no timeout, the value of the cost function is determined based on the normalized result of the solution time of the solver corresponding to the classification result of the path constraint used for training, wherein the maximum and minimum values of the solution time of the path constraint used for training are normalized according to each preset solver.
3. The testing method according to claim 1, wherein: The features of the path constraint include one or more of grammatical statistical features, grammatical structural features, variable features or logical features.
4. The test method according to claim 1 or 3, wherein: The features of extracting the path constraints of the program under test include: Convert the acquired path constraints of the tested program into a syntax tree, wherein the non-leaf nodes of the syntax tree represent operations, the leaf nodes represent operated objects, and the operated objects include at least one of variables or constants; Features of the path constraint are extracted from the syntax tree.
5. The testing method according to claim 4, wherein: The characteristics of the path constraint include one or more of statistical characteristics of the operation in the syntax tree, structural characteristics of the syntax tree, variable characteristics or logical characteristics.
6. The testing method according to claim 4, wherein: The step of converting the acquired path constraint into a syntax tree comprises: Each constraint in the acquired path constraint is stored in the constraint cache table in sequence; Generate path constraints in a constraint solver library SMT-LIB format according to the constraint cache table; Generate a syntax tree according to the path constraints in the SMT-LIB format.
7. The testing method according to claim 1, wherein: The testing method further includes: performing a pre-test on the tested program; and In the case where the pre-test is blocked, path constraints are extracted from statements between the start position and the blocking position of the tested program.
8. The testing method according to claim 7, wherein: The pre-test is a fuzzy test.
9. The testing method according to claim 1, wherein: The path constraints of the program under test are obtained through a symbolic execution engine.
10. A testing device comprising: A training module is configured to input the extracted features of the path constraint for training into the classifier to be trained, and obtain the classification result of the path constraint for training, wherein each classification result of the classifier corresponds to a preset solver; input the path constraint for training into each preset solver, and determine the value of the cost function of the classifier according to the solution time of each preset solver, wherein, when the solution is correct and not timed out, the value of the cost function is determined according to the solution time of the solver corresponding to the classification result of the path constraint for training; when the solution is correct but times out, the value of the cost function is a first preset value; when an error occurs in the solution, the value of the cost function is a second preset value; and according to the value of the cost function, the parameters of the classifier are adjusted; an extraction module configured to extract features of path constraints of the program under test; A classification module, configured to input the features of the path constraint into a pre-trained classifier to obtain a classification result of the path constraint, wherein each classification result of the classifier corresponds to a preset solver; a solving module configured to solve the path constraint using a solver corresponding to the classification result of the path constraint; and The testing module is configured to use the solution result to test the tested program.
11. A testing device comprising: Memory; as well as A processor coupled to the memory, the processor being configured to execute the testing method according to any one of claims 1 to 9 based on instructions stored in the memory.
12. A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the test method according to any one of claims 1 to 9 is implemented.