Method for testing a system

Abstract

Method for testing a system. A computer-implemented method for testing a system, in particular a computer program, a hardware system or an embedded system, is presented. Here, a first part of an execution structure for the system is examined using static analysis in terms of whether the system runs error-free when executing an execution path corresponding to the first part of the execution structure. If an error-free execution is determined in the static analysis for the first part of the execution structure, an examination of the system is carried out using dynamic analysis, which skips the execution path corresponding to the first part of the execution structure.

Description

Technical Field

[0001] This invention relates to a computer-implemented method for testing systems, particularly computer programs, hardware systems, or embedded systems, as well as computer programs and testing systems established for this purpose. Background Technology

[0002] A known method for dynamically testing systems to identify weaknesses is so-called fuzzing (also known as fuzzy testing or fuzzing). In this approach, the system under test (SUT), typically a computer program or a portion thereof, is executed. This execution is based on input variables obtained from a fuzzer. During the execution of the system—specifically, the execution of software, i.e., processing by a computer—the fuzzer observes the behavior of the SUT. If the SUT terminates or exhibits other undesirable behavior during execution, the fuzzer labels the provided input variables as error-related.

[0003] Known available fuzzers include Libfuzz, AFL-fuzz, or Hongfuzz.

[0004] However, dynamic approaches used for software testing, such as fuzzing, tend to focus on surface analysis because they often fail to find deep paths within the computer system under test. Furthermore, the efficiency of fuzzing depends on the execution speed of the system under test. The more times the system under test is executed, the more input variables can be examined to determine whether erroneous system behavior exists for those input variables.

[0005] Other known methods for testing computer programs and identifying weaknesses include static methods such as symbolic execution, konkolisch execution, or concolic execution, abstract interpretation, model checking, or finite model checking. In these methods, the source code or intermediate representation is analyzed using mathematical formalization. These techniques enable guarantees regarding the types of errors present in the software.

[0006] Baldoni, Roberto et al.’s “A survey of symbolic execution techniques” (ACMComputing Surveys (CSUR) 51.3 (2018): 50) provides an overview of symbolic and concrete symbolic execution and abstract interpretation methods.

[0007] Static analysis schemes for software depend on the often undecidable problem, whether the determined computer program can be formally verified, and to what extent. To provide automated, computer-implemented analysis tools, approximations are usually necessary. Therefore, static analysis often yields many false positives. For example, static analysis may report problems in the system under test that are unlikely to occur during actual execution, or it may report input variables that should lead to the traversal of a defined program path, but which, during actual execution, lead to the traversal of a different program path. The accuracy and execution duration of static analysis, among other criteria, can depend significantly on the size of the program being analyzed. Summary of the Invention

[0008] This paper introduces a computer-implemented method for testing a system, wherein the system under test includes, in particular, a computer program, a hardware system, or an embedded system.

[0009] Here, static analysis is used to examine the first part of the execution structure used for the system in terms of whether the system runs without errors when executing the execution path corresponding to the first part of the execution structure.

[0010] Static analysis here includes examining the system, preferably in a computer-implemented manner, based on the execution structure and with the aid of formal mathematical methods, without actually executing the system. Preferably, static analysis can employ model checking, finite model checking, checking with theorem provers, type transformation, abstract interpretation, symbolic execution, or concrete symbolic execution.

[0011] If the first part of the execution structure is determined to be error-free in static analysis, then dynamic analysis is used to check the system, which skips or selectively avoids executing the execution path corresponding to the first part of the execution structure.

[0012] The system is preferably checked using dynamic analysis to ensure that it is executed error-free for at least one defined input variable. This dynamic analysis includes checks performed on the computer that controls the system during its actual execution. Preferably, the dynamic analysis includes fuzz testing, particularly dynamic analysis of at least one input variable generated by a fuzzer.

[0013] Therefore, a testing method is introduced that, when combined with static analysis, enables particularly effective dynamic testing of the system. Specifically, the execution time of dynamic analysis can be progressively reduced using evolving or iterative static analysis, by gradually reducing the number of execution paths to be tested in the dynamic analysis through static analysis.

[0014] Additionally, considering only a portion of the execution structure in static analysis is equally effective in constituting the check and improving the possible accuracy of the static checking techniques used.

[0015] In a particularly preferred design, if the first part of the execution structure is determined to be error-free in the static analysis, then the first part of the execution structure is skipped in the subsequent static analysis. The subsequent static analysis of the execution structure is therefore performed after the static analysis of the first part of the execution structure has been completed. This ensures that the static analysis also runs effectively and that the portion of the execution structure that becomes smaller with repeated static analysis must also be checked.

[0016] In another preferred design, a first input variable is calculated for the first part of the execution structure based on static analysis, and when an erroneous execution is determined for the first part of the execution structure in the static analysis, the first input variable is stored in a set of input variables to be considered for dynamic analysis. Specifically, the following input variable is calculated as the first input variable, for which the static analysis concludes that an erroneous execution has occurred.

[0017] Therefore, substructures identified as erroneous in static analysis can be specifically elucidated in dynamic analysis. If static analysis provides, for example, false positive errors, these can be detected in dynamic analysis. Accordingly, actual errors can also be verified. Thus, a favorable overall testing method is provided.

[0018] In the preferred design, the static analysis and dynamic analysis steps are performed in parallel. This allows the advantages of the method to be realized particularly quickly and efficiently. However, if, for example, insufficient parallel computing capacity is available, the advantages can also be achieved in an alternative design where the static and dynamic analysis checks are performed sequentially or sequentially. In particular, in the two variants, the dynamic and static analyses can be performed cyclically.

[0019] In a preferred design of the described method, the system under test includes a computer program with a tree-like execution structure that represents the execution path of the computer program, and dynamic analysis includes the program code that executes the computer program on the computer. The described method for automated system testing can be implemented particularly effectively for systems implemented by or containing computer programs.

[0020] In another preferred design, the static analysis employs two different analysis methods to examine the first portion of the execution structure used for the system. A first analysis method examines the first portion of the execution structure from its root to its leaves, while a second analysis method examines the first portion from its leaves to its root. These two analysis methods are preferably used simultaneously in the static analysis of the subtrees of the execution structure. Therefore, the size of the portion of the structure to be examined by each analysis method can be reduced, thereby enabling the static analysis to be performed faster and with higher accuracy.

[0021] In preferred designs, errors identified during static and / or dynamic analysis include at least one runtime exception, violated assertions, occupation of disallowed system states, system termination, faulty memory access, faulty resource access, or program stagnation (Hängenbleiben), particularly failure to continue computation. By examining these potentially critical system errors, the system's safety and smooth operation can be determined. Automated elimination of errors identified in particularly preferred designs can also depend on the results of dynamic analysis, especially for the described error types. Therefore, not only is an effective method for checking computational errors in system execution provided, but also a system that improves its performance. Attached Figure Description

[0022] Embodiments of the invention will then be explained in more detail with reference to the accompanying drawings. In the drawings:

[0023] Figure 1 This schematically illustrates the execution branches of a computer program.

[0024] Figure 2 An exemplary flow of a method for testing a computer program is illustrated schematically. Detailed Implementation

[0025] The invention will first be described below with reference to the case where the system to be tested is a computer program or contains a computer program. Here, the system to be tested can be a pure computer program or an embedded system having a computer program. However, the described method is also suitable for a pure hardware system as the system to be tested.

[0026] Automated dynamic testing can be performed by a computer-based test system at the system under test (SUT). Here, the test system generates input variables for the SUT, executes or initiates the execution of the SUT, and observes the SUT to identify erroneous behavior. The SUT receives the input variables and performs calculations based on them. In the testing of a computer program, the program is executed, in particular, by a processor handling the program.

[0027] Dynamic testing of a computer program can create records about input variables that cause the program to execute erroneously (error sets). Dynamic testing can also (e.g., partially) record information about the tests already performed, particularly about the corresponding input variables (input sets). Furthermore, dynamic testing can record information about input variables that are of interest to the test system in order to compute additional input variables for testing at the system under test.

[0028] Figure 1 An exemplary execution tree structure, or tree 1, of a computer program is schematically shown, representing different execution paths of the computer program. The illustrated tree structure 1 has nodes 10 to 19 and edges connecting the nodes along the execution paths. Here, node 10 is the root node, nodes 11, 12, and 13 are internal nodes, and nodes 14 to 19 are leaf nodes. Subtree 100 is also shown, which includes internal node 11 and leaf nodes or leaves 14 and 15. Each path in the execution tree structure from the root node 10 to the leaf nodes 14 to 19 corresponds to an execution path of the executed computer program. Exemplary execution paths can be given by successive nodes 10, 11, and 15.

[0029] Figure 2 An exemplary flow of a method for testing a computer program is schematically illustrated. Here, the method includes a dynamic analysis step shown in box 21 and a static analysis step shown in box 22.

[0030] In step 220 of the static analysis, the execution tree structure is identified or subtrees of the tree structure are selected based on the execution tree structure of the computer program. The execution tree structure can be obtained as an input variable or created based on input variables, such as source code or binary files. Depending on the hardware system to be tested, the execution tree structure can also be created based on the hardware layout.

[0031] The tree structure can be created here for the system or computer program to be tested using available techniques, such as symbolic execution, software theorem provers, or a combination of these methods.

[0032] In subsequent step 222, a static analysis performed on a computer examines the identified or selected subtree to check whether an error will occur when executing the computer program along the execution path contained in or presented within that subtree. This analysis preferably provides three different results: "No error," "Error will occur," and "Unknown," where "Unknown" is the result for situations where the test system cannot determine, based on the analysis, whether an error will occur.

[0033] The results of this error-free analysis can be performed using available techniques, in particular model checking, finite model checking, theorem provers, typology transformation, abstract interpretation, symbolic execution, concrete symbolic execution, or a combination thereof.

[0034] The method proceeds to box 223 after step 222, where steps are performed based on the results of step 222.

[0035] In response to the result "no errors will occur," the method branches from step 222 to step 224 in box 223. There, the system being checked is modified so that the subtree determined to be error-free is no longer tested.

[0036] In particular, the system under inspection is modified so that execution of the subtree determined to be error-free in dynamic analysis is no longer possible. This modification can be performed, for example, by replacing the root directory or root in the source code of the executable computer program corresponding to the portion of the subtree determined to be error-free with an exit statement or terminating statement. Alternatively, the corresponding instructions can be injected into the binary code of the computer program at the appropriate point.

[0037] In another variant for modification, path conditions can be calculated from the root of the entire tree to the root of the subtree, and a test based on said path conditions can be added as the first instruction of the computer program to be tested. In this alternative, input variables that do not satisfy the path conditions can then be omitted, or it can be ensured, in dynamic analysis, that input variables satisfying the negated path conditions are used for generation, particularly by a fuzzer. This also prevents (further) dynamic testing of that subtree or corresponding part of the computer program. This variant can be particularly advantageous if the system to be tested runs on separate hardware on which the software cannot be easily or efficiently replaced automatically. Other methods for this purpose are also possible.

[0038] The modified system is then fed to box 21 and there it is fed to step 210 of the dynamic analysis.

[0039] In step 224 of box 223, the tree used for static analysis is also modified so that subtrees determined to be error-free are no longer analyzed in the static analysis. The modified tree is then fed as a new input variable to step 220, which, as described above, corresponds to the start of the static analysis.

[0040] If it's impossible to determine error-free nature for a subtree, but it's possible to extract the condition under which the execution of the part of the computer program corresponding to the subtree results in an error, then in a variant, this result can be treated the same as the result "no error will occur." Here, instead of modifying the tree structure by removing subtrees, a guarantee statement for the negative condition can replace the subtree in the tree structure. This approach can be particularly useful if the method used to determine error-free nature provides accurate results for the computation of the condition.

[0041] For the result "unknown", step 222 branches to step 226. There, information about the corresponding subtrees that have already been checked and have a result of "unknown" can be stored. This information can also be used to prevent (always) re-checking the subtree in further static analysis. For example, if a timeout occurs during the computation of the corresponding static analysis, a result of "unknown" will occur. In step 227, it can then be stored which subtree has a result of "unknown" so that the subtree is not computed again.

[0042] In response to the result "An error may occur," step 222 branches to step 225. In this step, the testing system preferably calculates one or more input variables for the system to be tested, which cause the execution of the system to be tested, the execution leading to the root of the subtree identified as having an error. Such calculation of input variables can be performed, for example, using symbolic execution, concrete symbolic execution, model checking, finite model checking, abstract interpretation, type transformation, test case generation, or theorem prover.

[0043] At least one of the resulting input variables is preferably passed to block 216 of dynamic analysis, particularly dynamic analysis 21, in step 225, the block corresponding to the set of input variables of interest, i.e., to be considered in further dynamic analysis.

[0044] Instead of adding at least one generated input variable to the set of input variables to be considered, the alternative scheme computes path conditions for the subtrees and initiates new test instances, particularly fuzzy test instances, of the dynamic analysis in parallel. In the new test instances, only input variables that satisfy the path conditions are generated. As a result, the new test instances focus on finding errors in subtrees identified as potentially erroneous by the static analysis. For example, such new fuzzy test instances can be accepted if the already running fuzzer is restarted for a new iteration. Different fuzzy test instances can additionally share their set of input variables of interest. This scheme is particularly useful when additional computational resources are available or techniques with over-approximate statistical analysis are used in static analyses that may provide many false positives.

[0045] After the corresponding steps in box 223 are completed, static analysis 22 continues with step 220. Here, in particular for the branch from step 224 (resulting in "no error"), a restart of static analysis is performed based on the modified tree structure received as input variables. Instead of restarting static analysis with the modified tree structure, static analysis can also restart or continue with the original or unchanged tree structure in alternative variants. In this case, information about the subtrees that have already been analyzed is stored and considered for further analysis to avoid duplicate analysis of the same subtrees. This approach is particularly suitable if the calculation of the modified tree structure is time-consuming / computationally expensive.

[0046] In a preferred variant of the described static analysis, the static analysis is performed using at least two different analysis methods, wherein a first static analysis method examines the execution structure from the root to the leaves, and a second static analysis method examines the execution structure from the leaves to the root. These two analysis methods are preferably used simultaneously in the static analysis of the subtrees of the execution structure. Therefore, the size of the portion of the structure to be examined by each analysis method can be reduced, thereby enabling the static analysis to be performed faster and with higher accuracy.

[0047] The dynamic analysis in Box 21 can be implemented using a fuzzer. Fuzzing (also known as fuzzy testing or fuzzing) is a known method for dynamically testing software to find weaknesses. In this approach, a computer program, typically with input variables, is executed from the system under test (SUT), which receives these input variables from a fuzzer. During the execution of the system, i.e., particularly the software execution, i.e., during computer processing, the fuzzer observes the behavior of the SUT. If the SUT terminates or exhibits other undesirable behavior during execution, the fuzzer marks the provided input variables as error-related. Known available fuzzers include, for example, Libfuzz, AFL-fuzz, or Hongfuzz.

[0048] Available fuzzing variants include white-box fuzzing, black-box fuzzing, and binary fuzzing, the latter requiring only a compiled executable file and no source code. Hardware fuzzing can also be used if the system being tested is a hardware system that performs the function being tested.

[0049] The fuzzer used can run on the same hardware as the system being tested. Alternatively, the fuzzer can also run on other hardware, where input and output variables, results, or observations about the system under test are communicated through a suitable interface.

[0050] Figure 2 Box 21 in the diagram illustrates the steps of automated dynamic analysis of the system under test. Here, the fuzzer is activated in step 210.

[0051] In subsequent step 211, the fuzzer generates at least one input variable for executing the system under test. The generation of input variables is preferably based on a trial-and-error approach and may also consider white-box information, such that, for example, input variables that lead to new execution paths during system execution are preferred. Generation can be performed here based on a set of stored input variables that have already been applied and, if necessary, will be expanded in further analysis (interesting set). Here, the generation of new input variables is performed based on these stored input variables through changes in parameters and values.

[0052] Next, at least one input variable, or information about said input variable, is stored in a record of the executed or initiated test conditions in step 212 and used as an input variable in step 213 for testing the system under test, in order to execute the system under test based on the input variable. If the system is a computer program or an embedded system including a computer program, then calculation steps corresponding to the processor's calculations based on the input variable are executed. During execution, the system under test's behavior and output variables are observed by the test system. Here, in the case of the computer program under test, for example, the execution path can be observed by adding an observer or software monitor during program compilation.

[0053] The results of the observations are generated in subsequent step 214. Possible results include, in particular, the results of "faulty execution" and "fault-free execution". Examples of faulty execution of the tested computer program are runtime exceptions, violated assertions, unauthorized occupation of system state, system termination, faulty memory access (buffer overflow, access after memory release), faulty resource access such as due to faulty synchronization in a multithreaded program, or program stalls (e.g., livelock, deadlock, etc.).

[0054] Following step 214, the branch proceeds to step 215 for the result "Error-prone execution," where the input variables or information about them that caused the error-prone execution are stored. The goal is a set of errors or records related to these input variables.

[0055] Following step 214, the branch proceeds to step 216 for the result "error-free execution," where the input variables or information about them that led to error-free execution are stored. The target is the stored set of input variables of interest, or those to be considered by the fuzzer for further dynamic analysis. This consideration is performed as described with respect to step 211, in that the fuzzer generates input variables for further analysis based on the input variables from this set.

[0056] Steps 215 and 216 branch back into step 210, where the fuzzer generates the input variables again. In an alternative variant, the method may also terminate after the first erroneous behavior is found.

[0057] After modifying the system to be tested in step 224 of box 22, it is preferable to branch to step 210, i.e., to restart the fuzzer based on the modified system to be tested. Instead of restarting the fuzzer every time it is modified in step 224 (as a result of each subtree being determined to be error-free), fuzz testing can also be restarted only when multiple subtrees have been analyzed or deemed error-free. This approach is particularly advantageous when restarting fuzz testing means significant execution overhead or when subtree analysis (e.g., at the start of analysis) is performed very quickly.

[0058] In a preferred variant, dynamic and static analyses are performed in parallel. However, in alternative variants, the dynamic and static analyses can also be performed sequentially, for example, by stopping the dynamic analysis after a predetermined time, performing the static analysis, and then resuming the dynamic analysis based on the results of the static analysis (i.e., especially in the case of a modified system to be tested if necessary). This can be particularly useful when there is insufficient parallel computing capacity. The stopping and starting of the dynamic or static analysis can here depend on a trial-and-error approach, for example, on the elapsed time or the progress of the analysis.

[0059] The system’s use of static and dynamic analysis for computer-implemented and automated testing can reliably, and particularly time-efficiently and resource-efficiently, identify and recognize execution errors.

[0060] The stored results of the tests (especially the set of input variables that led to erroneous behavior, the set of input variables already used for dynamic analysis, the set of input variables to be considered by dynamic analysis (interesting set), or information about subtrees identified as error-free in static analysis) can be used to improve the system. Based on the stored results, for example, bug debugging can be performed, or, for example, analysis of possible consequences can be conducted for security requirements, or product or security releases can be made for the system.

Claims

1. A computer-implemented method for testing a system, wherein the system under test is a computer program, hardware system, or embedded system, characterized in that... step: - Using static analysis (22), examine (222) the first part of the execution structure for the system in the following aspects: whether the system runs without errors when executing the execution path corresponding to the first part of the execution structure, wherein errors include at least one runtime exception, a violated assertion, an unauthorized occupation of a system state, system termination, faulty memory access, faulty resource access, or a pause in the computer program. - If the first part of the execution structure is determined to be error-free in the static analysis (22), the system is checked using dynamic analysis (21), which skips the execution path corresponding to the first part of the execution structure, wherein the skipping is achieved by modifying the system being checked so that the execution path corresponding to the first part of the execution structure is no longer possible.

2. The method of claim 1, wherein the pause of the computer program includes not continuing calculation.

3. The method according to any one of claims 1 to 2, characterized in that, If the first part of the execution structure is determined to be error-free in the static analysis, then the first part of the execution structure is skipped in the further static analysis (22).

4. The method according to any one of claims 1 to 2, characterized in that, The first input variable is calculated for the first part of the execution structure according to the static analysis (22), and when an erroneous execution is determined for the first part of the execution structure in the static analysis (22), the first input variable is stored in the set of input variables to be considered for the dynamic analysis (21).

5. The method according to claim 4, characterized in that, The input variable is calculated as the first input variable, and the static analysis (22) concludes that an error has occurred.

6. The method according to any one of claims 1 to 2, characterized in that, The system under test includes a computer program, the execution structure is a tree structure, the tree structure presents the execution path of the computer program, and the dynamic analysis (21) includes the program code of the computer program executed on the computer.

7. The method according to any one of claims 1 to 2, characterized in that, The static analysis (22) and the dynamic analysis (21) are performed in parallel.

8. The method according to any one of claims 1 to 2, characterized in that, The static analysis (22) and the dynamic analysis (21) are performed sequentially.

9. The method according to any one of claims 1 to 2, characterized in that, The system is checked using the dynamic analysis (21) in terms of whether it executes without errors for at least one defined input variable.

10. The method according to claim 9, characterized in that, The dynamic analysis (21) is used to check the system for runtime exceptions, violated assertions, unauthorized occupation of system states, system termination, faulty memory access, faulty resource access, or lapses during system execution.

11. The method according to claim 9, characterized in that, The system is improved based on the dynamic analysis (21).

12. The method according to claim 11, characterized in that, The identified errors are eliminated based on the dynamic analysis (21).

13. The method according to any one of claims 1 to 2, characterized in that, The dynamic analysis (21) includes fuzz testing.

14. The method according to claim 13, characterized in that, The dynamic analysis (21) is performed on at least one input variable generated by the fuzzer.

15. The method according to any one of claims 1 to 2, characterized in that, The static analysis (22) includes model checking, finite model checking, checking with the theorem prover, type transformation, abstract interpretation, symbolic execution or concrete symbolic execution.

16. The method according to any one of claims 1 to 2, characterized in that, The static analysis (22) includes examining the system in a computer-implemented manner based on the execution structure without actually executing the system.

17. The method according to claim 16, characterized in that, The static analysis (22) includes examining the system in a computer-implemented manner based on the execution structure without actually executing the system, using formal mathematical methods.

18. The method according to any one of claims 1 to 2, characterized in that, In the static analysis (22), two different analysis methods are used to perform an examination of the first part of the execution structure for the system, wherein the first analysis method examines the first part of the execution structure from the root to the leaf of the execution structure and the second analysis method examines the first part of the execution structure from the leaf to the root of the execution structure.

19. The method according to any one of claims 1 to 2, characterized in that, The dynamic analysis (21) includes checks performed on the computer of the system during actual execution of the system.

20. A computer program product comprising a computer program configured to perform the method according to any one of claims 1 to 19.

21. A machine-readable memory storing a computer program configured to perform the method according to any one of claims 1 to 19.

22. A test system for testing a system, the test system having a memory as claimed in claim 21, a processor for executing a computer program as claimed in claim 20, and at least one interface to the system under test.

Images