Apparatus, computer-readable data carrier, computer program, and method for fuzz testing software

Abstract

A method 100 for fuzz testing software comprises obtaining 110 seeds for fuzzing, obtaining 120 a respective bitmap size change (code coverage gain) and an average bitmap size change (code coverage ga

Description

Embodiments of the present disclosure relate to an apparatus, a computer-readable data carrier, a computer program, and a method for fuzz testing software. In recent years, with the continuous efforts of researchers, fuzzing has become a powerful automatic software testing technique, which has had a great impact in both industry and academia. New effective fuzzing methods are emerging one after another, which have produced a large number of crashes in practice. Finding the root cause of a crash is much more difficult than finding a new crash, because it is based on the manual and labor-intensive efforts of software developers. So, there is a huge gap between finding a crash and identifying the actual origin of a crash. To solve this problem, researchers have proposed many techniques, such as backward taint analysis and reverse execution. But these techniques have the following shortcomings: • They can identify only certain fault types. • They can only point to the assembly instructions of this specific fault and do not provide any contextual information or explanation which is very useful. Also, some concepts relate to an automatic analysis tool, which can provide root cause information of crash input for a binary executable without source code. Not only that, but such tool can also automatically provide the context information on erroneous behavior, that is, an explanation of the misbehavior of the crash. More importantly, such tool may be applied for various bugs (even if there is no data dependence between root cause and crash). Still, fuzz testing concepts have the following shortcomings: • They utilize AFL’s (AFL = American Fuzzy Lop) original crash mode to get more crash and non-crash sets. However, this primitive crash mode mutation strategy is very conservative, resulting in very slow growth of code coverage. This will slow down the discovery of test cases that are valid for root cause analysis. • They do not cluster and streamline a crashes output by the fuzzing test. When the target program contains multiple types of bugs, the subsequent root cause analysis results will be very inaccurate, and even wrong results will be generated. • The number of crash and non-crash outputs is very unbalanced, which leads to the use of inaccurate predicate calculation formulas, which reduces the accuracy of root cause analysis. Hence, there may be a demand of an improved concept of fuzz testing software. This demand may be satisfied by the subject-matter of the appended independent claims. Dependent claims disclose beneficial embodiments thereof. Embodiments of the present disclosure provide a method for fuzz testing software. The method comprises obtaining seeds for fuzzing, obtaining a respective bitmap size change and an average bitmap size change of test cases for the seeds, and determining a number of modifications for the seeds based on the respective bitmap size change and the average bitmap size change. This allows for a more efficient root cause analysis. The proposed approach provides an improvement of a mutation strategy of fuzzing. The improved mutation strategy provides that a result of the root cause analysis is more accurate than of other concepts. Not only that, since the proposed approach can use fewer traces to obtain more streamlined crash candidates, it greatly reduces the difficulty for developers to find the root cause. Apart from that, embodiments of the proposed approach may provide for adding a cluster selection process of crashes to improve a following evaluation, as laid out in more detail later. In practice, determining the number of modifications may comprise determining a respective deviation of the respective bitmap size change from the average bitmap size change for the test cases and determining the number of modifications for the seeds based on the respective deviation of a respective test case such that a first seed with a smaller deviation than a second seed is modified more often than the second seed. In this way, seeds providing more moderate coverage change are mutated more strongly than seed providing lower or higher coverage change. This provides an appropriate growth of coverage and precision / accuracy in detecting crash causes in the software. The method may be implemented in a fuzz testing algorithm. Accordingly, the method may further comprise modifying the seeds according to the number of modifications. Some implementations provide for a so-called “performance score” indicating how often a seed should be mutated. Accordingly, determining the number of modifications may comprise adapting a performance score of the respective test cases of the seeds and determining the number of the modifications such that the first seed is modified more often than the second seed. In some embodiments, the method further comprises filtering out one or more seeds of test cases from the modification based on a respective information gain of the respective test cases. In this way, seeds providing less information gain may be discarded. The information gain, e.g., depends on a number of bits of information gained in the respective test case. Filtering out the one or more seeds, e.g., comprises filtering out the seeds of the test cases gaining less information than a lower predefined information gain threshold or more information than an upper information gain threshold from the modification. In particular, filtering out the one or more seeds may comprise filtering out the seeds of the test cases gaining the least and the most information from the modification. In this way, the computational effort may be reduced. In some embodiments, the method further comprises clustering the test cases resulting in a software crash based on their respective crash cause. This allows a more substantial and efficient evaluation of the fuzz testing. For example, this allows to focus on certain clusters to examine certain crash causes. Clustering the test cases may comprise determining similarity scores of pairs of test cases based on their execution traces, obtaining a similarity matrix based on the similarity scores, and clustering the test cases based on the similarity matrix and based on a silhouette score for different numbers of clusters. In this way, the test cases may be grouped automatically and in an automated manner such that an effort for the evaluation of the fuzz testing is reduced. In some embodiments, the method further comprises analyzing the test cases based on predicates of respective crash causes of the test cases. In this way, the fuzz testing can be evaluated more targeted and efficient. A skilled person having benefit from the present disclosure will appreciate that the proposed method may be implemented in a computer program. In practice, e.g., one, some, or all steps of the proposed method may be implemented in a computer program. So, embodiments of the present disclosure may provide a computer-implemented method including the features of the proposed method. Accordingly, embodiments of the present disclosure may provide a computer program for fuzz testing software (i.e., for fuzz testing of software), the computer program comprising instructions which, when the computer program is executed by a computer, cause the computer to carry out the proposed method. The skilled person will appreciate that the computer program may be stored technically (digitally). Accordingly, embodiments may further provide a computer-readable data carrier having stored thereon the proposed computer program. Also, the skilled person will understand that the proposed method may be implemented in an apparatus, thereby enabling the apparatus to execute the proposed method. Embodiments of such an apparatus, e.g., comprise one or more interfaces for communication and a data processing circuit configured to execute the proposed method. Further features and details are described below with reference to the appended drawings. Fig. 1 shows a flow chart schematically illustrating an embodiment of a method for fuzz testing; Fig. 2 shows a flow chart schematically illustrating another embodiment of the method; and Fig. 3 shows a block diagram schematically illustrating an embodiment of an apparatus for fuzz testing. Fuzz testing, also known as “fuzzing”, is a software testing technique that involves providing invalid, unexpected, or random inputs to a software (here synonymously used with software program or set of software programs) in order to uncover vulnerabilities, bugs, or crash causes. It is particularly effective at finding issues that might not be caught by other, traditional testing methods. Fuzz testing involves generating a wide range of inputs, using an initial seed, to probe different parts of software code and uncover potential issues. In doing so, a so-called “bitmap size” and code coverage metrics are used to help assess the thoroughness of the testing and identify areas that might require further attention. In fuzz testing, a “seed” is an initial input that serves as starting point for generating variations of inputs. It is noted that, in practice, as in context of the present disclosure, the term “seed” or “seeds” may not only refer to the very first initial input but also to mutations thereof that are used for further mutations, i.e., for mutations of already mutated inputs. The variations are created through so-called “mutations” or “transformations” making random or systematic changes to the seed input, such as altering values, adding, or removing characters, and so on. In this way, different execution variations of the code are tested for different inputs to provoke crashes and identify crash causes. In doing so, the so-called “bitmap size” is used to determine a code coverage while testing. So, the bitmap size refers to a data structure used to keep track of which parts of the software code have been executed during the fuzz testing. This mainly helps to measure the code coverage, as it helps determine how much of the code has been exercised by the generated inputs. A larger bitmap size indicates a greater coverage and vice versa. As mentioned above, fuzzing may be inaccurate, require a high computational effort, and difficult to evaluate. Hence, an improved concept of fuzz testing is proposed, as laid out in more detail with reference to Fig. 1. Fig. 1 shows a flow chart schematically illustrating an embodiment of a method 100 for fuzz testing. The method 100 comprises obtaining 110 seeds for fuzzing. For this, the seeds may be manually crafted, obtained from existing test cases, from real-world data that the software is expected to handle, through random generation, from a corpus including a collection of valid input, and / or using machine learning or heuristics. The skilled person will appreciate that the proposed method may be applied to any kind of seed and different measures to obtain the seeds. Method 100 further comprises obtaining 120 a respective bitmap size change and an average bitmap size change (in practice bitmap size gain) of test cases for the seeds. To this end, it is determined how much the bitmap size changes for each of the seeds when applied in the fuzz testing. In this way, the seeds can be rated by their bitmap size changes to determine a gain of code coverage for the seeds. In fuzz testing, generally, seeds providing a higher code coverage gain may be mutated more often than seeds providing a lower code coverage gain. However, this may lead to inaccurate results of root cause analysis. The proposed method therefore suggests determining 130 a number of modifications for the seeds based on the respective bitmap size change and the average bitmap size change. In doing so, e.g., seeds providing moderate code coverage gain may be mutated more often than other seeds providing less or more code coverage gain. It turned out that this allows more precise results while the computational effort is reduced compared to other fuzz testing concepts maintaining the seeds with the highest code coverage gain. So, it is proposed that the number of modifications for a seed is higher if the code coverage gain / bitmap size change is closer to the average code coverage gain / bitmap size change than if the code coverage gain / bitmap size change deviates more from the average code coverage gain / bitmap size change. In this way, less computational effort is used for seeds providing insufficient and less targeted information gain. At the same time seeds providing sufficient information gain to uncover crash causes are maintained. Further features and detail are stated in more detail below with reference to Fig. 2. Fig. 2 relates to another embodiment of the proposed method 100. As indicated by dashed boxes, the embodiment of fig. 2 includes three stages. It comprises a fuzzing stage 210 which provides for testing software for different test cases based on different seeds (inputs) and for classifying the test cases as “crashes” or “non-crashes”. Further, the method includes a clustering stage 220 for clustering the test cases based on their crash cause. As well, the method includes an analysis stage 230 for determining the crash cause of the test cases. Embodiments of such multi-stage process in connection with the proposed approach are referred to as “ClusteRCA” (Clustering Root Cause Analysis). Fuzzing stage 210: The original AFL uses coverage-guided fuzzing, which maximizes code coverage based on feedbacks, triggering more crashes on different paths. However, such coverage-guided fuzzing can lead to a large number of crashes unrelated to a desired target root cause. If these crash data are used for subsequent root cause analysis, the results will be inaccurate. In contrast, the present disclosure suggests a crash exploration mode of fuzzing which can efficiently generate a (large) number of crashes caused by the same bug via different code paths. These crashes are suitable as a data set for root cause analysis. For this, the fuzzing stage 210 comprises a fuzzing strategy called “MildFuzz” that provides that seeds providing moderate coverage gain are mutated more often that other seeds providing more or less coverage gain, as laid out in more detail above. In doing so, e.g., seeds which do not provide any information gain, i.e., which do not add any new bit in the bitmap, are discarded. As well, seeds providing the least and / or the most new bits in the bitmap may be discarded. In a first step 212 of the fuzzing stage 210, a crash is caused or provoked using an initial input. So, like in the crash exploration mode, ClusteRCA uses the crash input as a mutation seed, and only keeps input that can triggers the crash in the fuzzing queue. Then, in a subsequent step 214, MildFuzz is applied to the initial input to generate mutated seeds as input to the software to initiate crash and non-crash test cases. In doing so, a respective deviation of the respective bitmap size change from the average bitmap size change for the test cases is determined for the seeds. The proposed fuzzing strategy suggests that the seeds based on the respective deviation of a respective test case such that a seeds with a smaller deviation are modified more often than a second seed is modified more often than the second seed. That is, more mutations or variants are generated for seeds having or providing a smaller deviation. The variants or mutations are then used in another test and / or fuzzing cycle to provoke further crash and non-crash test cases of the tested software. In doing so, execution traces for the mutations may further increase the bitmap size by one or more bits, i.e., gain information. In comparison with other mutation / fuzzing strategies, the proposed one achieves a favorable trade-off between a sufficient code coverage and resource consumption for the processing of seeds / execution of the software for the seeds. In doing so, the proposed approach provides a fuzzing strategy that may be more efficient and targeted to a desired bug / crash cause. As the skilled person will appreciate, a metric or function for determining the number of modifications based on the deviation may be adapted, e.g., to tailor it to different use cases. To this end, the metric or function, e.g., one or more parameters of the metric or function may be adjusted accordingly. In practice, the number of modifications may be determined based on a so-called “performance score”, wherein the performance score is determined based on the bitmap size change. Accordingly, the calculation of the performance score may be adapted to adjust the determination of the number of modifications. In doing so, one or more seeds may be filtered out based on their information gain, i.e., based on the number of new bits in the bitmap after execution of the software based on the respective seed, and a information gain threshold. In practice, e.g., seeds providing less and or more information gain than the threshold are filtered out. Alternatively, the seeds providing the least and the most new bits are filtered out. The test cases are, then, categorized as “Non-crashes” 216a and “Crashes” 216b, respectively, and used as input to the clustering stage 220 Clustering stage 220: Fuzzers are highly non-deterministic in how they discover bugs and paths through the target program. The fuzzer may discover one or more crashes for a bug, but it is not known a priori how many crashes it will find. Consequently, one cannot know how many bugs are discovered during a fuzzing campaign. For example, a given set of 100 crashes may map to 100 bugs with one crash each, one bug with 100 crashes, or anything in between. The key challenges for clustering are to (a) discover the exact number of bugs, and (b) assign crashes to the correct bug. As stated above, the number of expected clusters is not known and must be inferred during clustering. The present approach determines the number of clusters by running the clustering process multiple times, refining the assumed number of bugs based on a heuristic. In the clustering stage, a control flow graph constructor 224 is applied to non-crashing execution traces 222a of the non-crashes and crashing execution traces 222b of the crashes to determine connections and / or similarities of the traces. Then, a clustering analysis 226 is applied. In doing so, ClusteRCA utilizes a WeisfeilerLehman Subtree Kernel algorithm to generate a similarity matrix based on the connections and / or similarities. Then, ClusterRCA sets the initial number of clusters to 2, and uses a Spectral Clustering algorithm to cluster the similarity matrix. After the clustering process is completed, ClusteRCA will calculate a silhouette score. In the next step, ClusteRCA cyclically increases the number of clusters by 1 and repeats the above process to obtain a new silhouette score. Finally, ClusteRCA finds the number of clusters and clustering results corresponding to the highest silhouette score. In doing so, the test cases are clustered by their crash cause. That is, ideally, test cases triggered by the same crash cause are in the same cluster and test cases triggered by different crash causes are in different clusters. Then, only the crashes in the same cluster as the initial crash are kept, and the others are discarded in step 227, e.g., to direct the analysis to a certain desired crash cause. In doing so, the clustering analysis 226 provides non-crashing execution traces 228a and crashing execution traces 228b of the same cluster for the following analysis stage 230. Analysis stage 230: ClusteRCA provides for a root cause analysis 232. For this, ClusteRCA collects three types of predicates, namely control flow predicates, register-related predicates, and memory-related predicates. Then, ClusteRCA adds up the number of traces of all crashes and non-crashes that were mispredicted, and divides by the total number of traces to get a score. Next, subtract 0.5 from this score, take the absolute value, and multiply by 2. The final score in the range [0, 1 ] is the probability of success of the predicate. Subsequently, the predicates are ranked by their probability of success in step 234. Then, in a subsequent step 236, ClusteRCA selects all the predicates with scores over 0.95, which are the root cause candidates. So, the root cause may be determined based on the candidates. For this, they may be graphically displayed to and chosen by a user. In this way, crash causes may be identified and fixed. In comparison with other concept, the proposed approach delivers a more accurate result. Thus, the proposed approach may simplify and the identification of bugs and the revision of software. The skilled person will appreciate that the above example is not limited only to the specified means, algorithms, and parameter values. In embodiments, the skilled person accordingly may choose equivalent algorithms or other parameter values, e.g., to tailor the proposed concept individually, to a desired use case. The proposed method, e.g., is applied in the inspection and / or troubleshooting of software in the production and / or afterwards, e.g., after shipment or implementation. The explained embodiments, e.g., are implemented in a computer program for fuzz testing software. For storing and / or distributing such computer program, it may be (permanently) stored on a (non-volatile) computer-readable data carrier. The computer program, e.g., enables an apparatus (including a computer) to execute the proposed method of fuzz testing. Further details of such an apparatus are stated in more detail below with reference to Fig. 3. Fig. 3 shows a block diagram schematically illustrating an embodiment of an apparatus 300 for fuzz testing. The apparatus comprises one or more interfaces 310 for communication and a data processing circuit 320 configured to execute the proposed method. In embodiments, the one or more interfaces 310 may comprise wired and / or wireless interfaces for transmitting and / or receiving communication signals in connection with the execution of the proposed concept. In practice, the interfaces, e.g., comprise pins, wires, antennas, and / or the like. As well, the interfaces may comprise means for (analog and / or digital) signal or data processing in connection with the communication, e.g., filters, samples, analog-to-digital converters, signal acquisition and / or reconstruction means as well as signal amplifiers, compressors and / or any encryption / decryption means. The data processing circuit 320 may correspond to or comprise any type of programable hardware. So, examples of the data processing circuit 320, e.g., comprise a memory, microcontroller, field programable gate arrays, one or more central, graphical processing units, and / or the like. To execute the proposed method, the data processing circuit 320 may be configured to access or retrieve an appropriate computer program for the execution of the proposed method from a memory of the data processing circuit 320 or a separate memory which is communicatively coupled to the data processing circuit 320. In the foregoing description, it can be seen that various features are grouped together in examples for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed examples require more features than are expressly recited in each claim. Rather, as the following claims reflect, subject matter may lie in less than all features of a single disclosed example. Thus, the following claims are hereby incorporated into the description, where each claim may stand on its own as a separate example. While each claim may stand on its own as a separate example, it is to be noted that, although a dependent claim may refer in the claims to a specific combination with one or more other claims, other examples may also include a combination of the dependent claim with the subject matter of each other dependent claim or a combination of each feature with other dependent or independent claims. Such combinations are proposed herein unless it is stated that a specific combination is not intended. Furthermore, it is intended to include also features of a claim to any other independent claim even if this claim is not directly made dependent to the independent claim. Although specific embodiments have been illustrated and described herein, it will be appreciated by those of ordinary skill in the art that a variety of alternate and / or equivalent implementations may be substituted for the specific embodiments shown and described without departing from the scope of the present embodiments. This application is intended to cover any adaptations or variations of the specific embodiments discussed herein. Therefore, it is intended that the embodiments be limited only by the claims and the equivalents thereof.

Claims

1. A method (100) for fuzz testing software, the method (100) comprising:obtaining (110) seeds for fuzzing;obtaining (120) a respective bitmap size change and an average bitmap size change of test cases for the seeds;determining (130) a number of modifications for the seeds based on the respective bitmap size change and the average bitmap size change.

2. The method (100) of claim 1, wherein determining (130) the number of modifications comprises:determining a respective deviation of the respective bitmap size change from the average bitmap size change for the test cases; anddetermining the number of modifications for the seeds based on the respective deviation of a respective test case such that a first seed with a smaller deviation than a second seed is modified more often than the second seed.

3. The method (100) of claim 1 or 2, wherein the method (100) further comprises modifying the seeds according to the number of modifications.

4. The method (100) of any one of the preceding claims, wherein determining (130) the number of modifications comprises adapting a performance score of the respective test cases of the seeds and determining the number of the modifications such that the first seed is modified more often than the second seed.

5. The method (100) oy any one of the preceding claims, wherein the method (100) further comprises filtering out one or more seeds of test cases from the modification based on a respective information gain of the respective test cases.

6. The method (100) of claim 5, wherein the information gain depends on a number of bits of information gained in the respective test case.

7. The method (100) of claim 5 or 6, wherein filtering out the one or more seeds comprises filtering out the seeds of the test cases gaining less information than a lower predefined information gain threshold or more information than an upper information gain threshold from the modification.

8. The method (100) of claim 5 or 6, wherein filtering out the one or more seeds comprises filtering out the seeds of the test cases gaining the least and the most information from the modification.

9. The method (100) of any one of the preceding claims, wherein the method (100) further comprises clustering the test cases resulting in a software crash based on their respective crash cause.

10. The method (100) of claim 9, wherein clustering the test cases comprises:determining similarity scores of pairs of test cases based on their execution traces;obtaining a similarity matrix based on the similarity scores; andclustering the test cases based on the similarity matrix and based on a silhouette score for different numbers of clusters.

11. The method (100) of any one of the preceding claims, wherein the method (100) further comprises analyzing the test cases based on predicates of respective crash causes of the test cases.

12. A computer program for fuzz testing software, the computer program comprising instructions which, when the computer program is executed by a computer, cause the computer to carry out the method (100) of any one of the claims 1 to 11.

13. A computer-readable data carrier having stored thereon the computer program of claim 11.

14. An apparatus (300) comprising:one or more interfaces (310) for communication; anda data processing circuit (320) configured to execute the method (100) of any one of the claims 1 to 11.

Images