Device, computer-readable data carrier, computer program and method for fully testing software

Abstract

Embodiments of the present disclosure relate to an apparatus (300), a computer-readable data carrier, a computer program and a method (100) for fuzzing software. The method (100) comprises: obtaining (110) a seed for fuzz testing; obtaining (120) a corresponding bitmap size change and an average bitmap size change of the test case of the seed; and determining (130) a number of modifications of the seed based on the respective bitmap size change and the average bitmap size change.

Description

[0001] Embodiments of the present disclosure relate to an apparatus, a computer- readable data carrier, a computer program and a method for fuzz testing software.

[0002] In recent years, fuzz testing has become a powerful automated software testing technique that has had a great impact in both industry and academia, thanks to the continuous efforts of researchers. New effective fuzzing methods are emerging, producing a large number of crashes in practice. Finding the root cause of a crash is much more difficult than finding new crashes, as it is based on manual and labor-intensive work by software developers. Therefore, there is a huge gap between finding a crash and identifying the actual origin of the crash. To address this problem, researchers have proposed many techniques, such as backward taint analysis and reverse execution. But these techniques have the following drawbacks: • They can only identify certain fault types.

[0003] • They can only point to the assembly instructions for this particular fault, but cannot provide any very useful contextual information or explanation.

[0004] In addition, some concepts involve automatic analysis tools that can provide root cause information for crash inputs for binary executables without source code. Not only that, but such tools can also automatically provide contextual information about the erroneous behavior, i.e. an explanation of the erroneous behavior of the crash. More importantly, such tools can be applied to a variety of bugs (defects) even if there is no data dependency between the root cause and the crash.

[0005] However, the fuzzing concepts have the following drawbacks: • They exploit the original crash pattern of AFL (AFL = American Fuzzy Lop) to obtain more crash and non-crash sets. However, this original crash pattern mutation strategy is very conservative, resulting in very slow growth of code coverage. This will slow down the discovery of test cases that are effective for root cause analysis.

[0006] • They do not cluster and refine the crashes output by fuzz testing. When the target program contains multiple types of bugs, the results of subsequent root cause analysis will be very inaccurate, or even produce false results.

[0007] • The number of crashes and non-crash outputs is very unbalanced, resulting in the use of inaccurate predicate calculation formulas, which reduces the accuracy of root cause analysis.

[0008] Therefore, there can be a need to improve the concept of fuzz testing software.

[0009] This requirement can be met by the subject matter of the appended independent claims. The dependent claims disclose advantageous embodiments of this disclosure.

[0010] The embodiments disclosed herein provide a method for fuzz testing software. The method includes: obtaining a seed for fuzz testing; obtaining corresponding bitmap size changes and average bitmap size changes for test cases of the seed; and determining the number of modifications to the seed based on the corresponding bitmap size changes and average bitmap size changes.

[0011] This allows for more efficient root cause analysis. The proposed method offers an improvement on the mutation strategy of fuzz testing. The improved mutation strategy makes the results of root cause analysis more accurate than those of other concepts. Moreover, since the proposed method can obtain more concise crash candidates using fewer trajectories, it significantly reduces the difficulty for developers to find root causes.

[0012] In addition, implementations of the proposed method can provide a cluster selection process with added collapse to improve subsequent evaluation, as will be elaborated in more detail later.

[0013] In practice, determining the number of modifications can include determining the deviation between the corresponding bitmap size change of the test case and the average bitmap size change, and determining the number of modifications to the seed based on the corresponding deviation of the test case, such that the first seed with a smaller deviation than the second seed is modified more frequently than the second seed. In this way, a seed that provides more moderate coverage changes is more abrupt than a seed that provides lower or higher coverage changes. This provides an appropriate increase in the coverage and precision / accuracy of detecting crash causes in software.

[0014] This method can be implemented in a fuzzing algorithm. Therefore, the method can further include modifying the seed based on the number of modifications.

[0015] Some implementations provide a so-called "performance score" that indicates how often a seed should mutate. Therefore, determining the number of modifications can include adjusting the performance score of the corresponding test cases for the seed, as well as determining the number of modifications that makes the first seed modified more frequently than the second seed.

[0016] In some implementations, the method further includes filtering out one or more seeds of test cases from the modifications based on the corresponding information gain of the test cases. In this way, seeds that provide less information gain can be discarded.

[0017] Information gain, for example, depends on the number of bits of information gained in the corresponding test case.

[0018] Filter out one or more seeds, such as seeds that filter out test cases from modifications that have less information gain than the predefined lower information gain threshold or more information gain than the upper information gain threshold.

[0019] Specifically, filtering out one or more seeds includes filtering out the seeds with the least and most informational gain from the modifications. This reduces computational overhead.

[0020] In some implementations, the method further includes clustering test cases that cause software crashes based on the corresponding crash causes. This allows for more substantial and efficient evaluation of fuzz testing. For example, it allows focusing on certain clusters to examine specific crash causes.

[0021] Clustering test cases can include: determining the similarity score of test case pairs based on their execution trajectories; obtaining a similarity matrix based on the similarity scores; and clustering the test cases based on the similarity matrix and silhouette scores of different numbers of clusters. In this way, test cases can be grouped automatically and in an automated manner, reducing the effort required for fuzz testing evaluation.

[0022] In some implementations, the method further includes analyzing test cases based on assertions of the corresponding crash causes. This allows for more targeted and efficient evaluation of fuzz tests.

[0023] Those skilled in the art will understand that the proposed method can be implemented in a computer program. In practice, for example, one, some, or all of the steps of the proposed method can be implemented in a computer program. Therefore, embodiments of the present disclosure can provide a computer-implemented method including the features of the proposed method.

[0024] Therefore, embodiments of this disclosure may provide a computer program for fuzzing software (i.e., for fuzzing software), the computer program including instructions that, when executed by a computer, cause the computer to perform the proposed method.

[0025] Those skilled in the art will understand that computer programs can be stored technically (digitally).

[0026] Therefore, the implementation scheme may further provide a computer-readable data carrier on which the proposed computer program is stored.

[0027] Furthermore, those skilled in the art will understand that the proposed method can be implemented in a device, thereby enabling the device to perform the proposed method.

[0028] Implementations of such a device may include, for example, one or more interfaces for communication and data processing circuitry configured to perform the proposed method.

[0029] Further features and details are described below with reference to the accompanying drawings.

[0030] Figure 1 A flowchart illustrating an implementation scheme for a fuzz testing method is shown; Figure 2 A flowchart illustrating another embodiment of the method is shown; and Figure 3 A block diagram schematically illustrating an implementation scheme for a device used for fuzz testing is shown.

[0031] Fuzzing, also known as "fuzzing," is a software testing technique that involves providing software with invalid, unexpected, or random input (here used implicitly with the software program or a collection of software programs) in order to discover vulnerabilities, bugs, or causes of crashes. It is particularly effective at finding problems that other traditional testing methods might miss.

[0032] Fuzzing involves generating a wide range of inputs using an initial seed to probe different parts of the software code and uncover potential problems. In doing so, metrics such as "bitmap size" and code coverage are used to help assess the thoroughness of the tests and identify areas that may require further attention.

[0033] In fuzzing, a "seed" is the initial input, serving as the starting point for generating input variants. It should be noted that in practice, as in the context of this disclosure, the term "seed" or "seeds" can refer not only to the initial input but also to variants of that initial input used for further mutations—that is, variants of an already mutated input. Variants are created by making random or systematic changes (such as changing values, adding or deleting characters, etc.) to the seed input through so-called "mutations" or "transformations." In this way, different mutations are performed on different inputs to test the code, triggering crashes and identifying the causes of the crashes.

[0034] In doing this, the so-called "bitmap size" is used to determine code coverage during testing. Therefore, bitmap size refers to the data structure used to track which parts of the software code have been executed during fuzzing. This primarily helps in measuring code coverage because it helps determine how much code has been executed from the generated input. A larger bitmap size indicates greater coverage, and vice versa.

[0035] As mentioned above, fuzz testing can be inaccurate, requires a high amount of computation, and is difficult to evaluate.

[0036] Therefore, an improved concept for fuzz testing is proposed, as referenced... Figure 1 To elaborate in more detail.

[0037] Figure 1 A flowchart illustrating an implementation scheme of method 100 for fuzz testing is shown.

[0038] Method 100 includes obtaining 110 seeds for fuzz testing. For this purpose, seeds can be created manually, obtained from existing test cases, obtained from real-world data that the software is expected to process, obtained through random generation, obtained from a corpus containing a set of valid inputs, and / or obtained using machine learning or heuristic learning. Those skilled in the art will understand that the proposed method can be applied to any kind of seed and different measures for obtaining the seeds.

[0039] Method 100 further includes obtaining the corresponding bitmap size changes and average bitmap size changes (actually bitmap size gains) for the test cases of the 120 seeds. To this end, it is determined how much the bitmap size changes for each seed when applied in fuzzing. In this way, the seeds can be rated based on their bitmap size changes to determine the code coverage gain of each seed.

[0040] In fuzz testing, seeds that provide higher code coverage gain typically mutate more frequently than seeds that provide lower code coverage gain. However, this can lead to inaccurate results from root cause analysis.

[0041] Therefore, the proposed method suggests determining the number of modifications for the 130 seeds based on the corresponding bitmap size change and the average bitmap size change. In doing so, for example, a seed providing moderate code coverage gain can mutate more frequently than other seeds providing less or more code coverage gain. This has proven to allow for more accurate results compared to other fuzzing concepts, while reducing computational effort and preserving the seed with the highest code coverage gain. Therefore, it is proposed that the number of seed modifications is higher if the code coverage gain / bitmap size change is closer to the average code coverage gain / bitmap size change than if the code coverage gain / bitmap size change deviates more from the average code coverage gain / bitmap size change. In this way, less computational effort is used for seeds that provide insufficient and less of the target information gain. Meanwhile, seeds that provide sufficient information gain to discover the cause of the crash are retained.

[0042] The following is for reference. Figure 2 Further details and features will be provided.

[0043] Figure 2 Another embodiment of the proposed method 100 is described. As indicated by the dashed box,Figure 2 The implementation scheme comprises three phases. It includes a fuzzing phase 210, which allows testing the software against different test cases based on different seeds (inputs) and classifies test cases as "crashing" or "non-crashing". Furthermore, the method includes a clustering phase 220 for clustering test cases based on their crash causes. Similarly, the method includes an analysis phase 230 for determining the collision causes of test cases.

[0044] The implementation of this multi-stage process combining the proposed method is called "ClusterRCA" (Clustering Root Cause Analysis).

[0045] Fuzzing Phase 210: The original AFL used coverage-guided fuzzing, which maximizes code coverage based on feedback, triggering more crashes on different paths. However, this coverage-guided fuzzing can lead to a large number of crashes unrelated to the intended root cause. If this crash data is used for subsequent root cause analysis, the results will be inaccurate.

[0046] In contrast, this disclosure proposes a fuzzing crash exploration pattern that can efficiently generate (a large number) of crashes caused by the same bug via different code paths. These crashes are suitable as a dataset for root cause analysis. To this end, fuzzing phase 210 includes a fuzzing strategy called “MildFuzz”, which stipulates that seeds providing moderate coverage gain are mutated more frequently than other seeds providing more or less coverage gain, as explained in more detail above. In doing so, for example, seeds that do not provide any information gain (i.e., seeds that do not add any new bits to the bitmap) are discarded. Similarly, seeds that provide the fewest and / or most new bits in the bitmap can be discarded.

[0047] In the first step 212 of the fuzzing phase 210, an initial input is used to induce or trigger a crash. Therefore, just as in crash exploration mode, ClusterRCA uses the crash input as a mutation seed and only retains inputs that can trigger a crash in the fuzzing queue.

[0048] Then, in subsequent step 214, MildFuzz is applied to the initial input to generate a mutation seed as input to the software to initialize crash and non-crash test cases.

[0049] In doing so, the corresponding bitmap size change for the test case is determined for the seed, and the corresponding deviation from the average bitmap size change is considered. The proposed fuzzing strategy suggests seeds based on the corresponding deviation for the test case, such that seeds with a smaller deviation than the second seed are modified more frequently than the second seed. That is, for seeds with or providing a smaller deviation, more mutations or variants are generated. The variants or mutations are then used in another test and / or fuzzing loop to trigger further crash and non-crash test cases for the software under test. In doing so, the execution trajectory for the variants can further increase the bitmap size by one or more times, i.e., gain information. Compared with other mutation / fuzzing strategies, the proposed strategy achieves a favorable trade-off between sufficient code coverage and resource consumption for seed processing / seed software execution. In doing so, the proposed method provides a fuzzing strategy that may be more efficient and more targeted to the expected bug / crash causes.

[0050] As those skilled in the art will understand, metrics or functions used to determine the number of modifications based on deviations can be adapted, for example, to suit different use cases. To this end, the metric or function (e.g., one or more parameters of the metric or function) can be adjusted accordingly.

[0051] In practice, the number of modifications can be determined based on a so-called "performance score," which is determined based on changes in bitmap size. Therefore, the calculation of the performance score can be adapted to adjust the determination of the number of modifications.

[0052] In doing so, one or more seeds can be filtered out based on the information gain of one or more seeds, that is, based on the number of new bits in the bitmap after the software is executed based on the corresponding seed, and an information gain threshold. In practice, for example, seeds that provide less and / or more information gain than the threshold are filtered out. Alternatively, seeds that provide the fewest and most new bits are filtered out.

[0053] The test cases were then categorized as “non-crash” 216a and “crash” 216b, respectively, and used as input for clustering phase 220.

[0054] Clustering Phase 220: Fuzz testers face a high degree of uncertainty in how they discover bugs and paths in a target program. A fuzz tester might find one or more bugs with a crash, but it doesn't know in advance how many crashes it will find. Therefore, it's impossible to know how many bugs will be found during a fuzzing campaign. For example, a given set of 100 crashes could map to 100 bugs, one bug with one crash, one bug with 100 crashes, or anything in between. The key challenges of clustering are (a) the exact number of bugs found, and (b) assigning crashes to the correct bugs. As mentioned above, the expected number of clusters is unknown and must be inferred during clustering. This method determines the number of clusters by running the clustering process multiple times, refining the hypothesized number of bugs based on heuristic learning.

[0055] During the clustering phase, a control flow graph builder 224 is applied to non-crash, non-crash execution trajectories 222a and crash execution trajectories 222b to determine the connectivity and / or similarity of the trajectories. Then, cluster analysis 226 is applied. In doing so, ClusterRCA uses the Weisfeiler-Lehman subtree kernel algorithm to generate a similarity matrix based on connectivity and / or similarity. ClusterRCA then sets the initial number of clusters to 2 and uses a spectral clustering algorithm to cluster the similarity matrix. After the clustering process is complete, ClusterRCA calculates silhouette coefficients. In subsequent steps, ClusterRCA iteratively increases the number of clusters by 1 and repeats the above process to obtain new silhouette coefficients. Finally, ClusterRCA finds the number of clusters and the clustering results corresponding to the highest silhouette coefficients. In doing so, test cases are clustered according to their crash causes. That is, ideally, test cases triggered by the same crash cause are in the same cluster, while test cases triggered by different crash causes are in different clusters.

[0056] Then, in step 227, only crashes in the same cluster as the initial crash are retained, and other crashes are discarded, for example, to guide the analysis toward a desired cause of the crash. In doing so, cluster analysis 226 provides non-crash execution trajectories 228a and crash execution trajectories 228b for the same clusters in subsequent analysis phase 230.

[0057] Analysis Phase 230: ClusterRCA provides root cause analysis232. To do this, ClusterRCA collects three types of predicates: control flow predicates, register-related predicates, and memory-related predicates. Then, ClusterRCA sums the number of all incorrectly predicted crash and non-crash trajectories and divides by the total number of trajectories to obtain a score. Next, 0.5 is subtracted from this score, the absolute value is taken, and then multiplied by 2. The final score in the range [0, 1] represents the probability of a predicate's success.

[0058] Subsequently, in step 234, the predicates are sorted by their success probability.

[0059] Then, in subsequent step 236, ClusterRCA selects all predicates with scores greater than 0.95; these predicates are root cause candidates. Therefore, root causes can be determined based on these candidates. For this purpose, they can be graphically displayed to the user for selection. In this way, the causes of crashes can be identified and fixed. Compared to other concepts, the proposed method provides a more accurate result. Therefore, the proposed method can simplify bug identification and software revision.

[0060] Those skilled in the art will understand that the above examples are not limited to the specified means, algorithms, and parameter values. In implementation, those skilled in the art can therefore choose equivalent algorithms or other parameter values, for example, to individually tailor the proposed concept to the desired use case.

[0061] The proposed methods, for example, are applied to the inspection and / or troubleshooting of software in production and / or after (e.g., after delivery or implementation).

[0062] The explained implementation scheme is, for example, implemented in a computer program used for fuzz testing of software. For storage and / or distribution of such a computer program, it can be (permanently) stored on a (non-volatile) computer-readable data carrier.

[0063] Computer programs, for example, enable devices (including computers) to perform the proposed fuzz test.

[0064] The following is for reference. Figure 3 Further details of this device will be provided.

[0065] Figure 3 A block diagram schematically illustrating an embodiment of a device 300 for fuzz testing is shown.

[0066] The device includes one or more interfaces 310 for communication and data processing circuitry 320 configured to perform the proposed method.

[0067] In the implementation, one or more interfaces 310 may include wired and / or wireless interfaces for transmitting and / or receiving communication signals while performing the proposed concept. In practice, interfaces may include, for example, pins, wires, antennas, and / or the like. Similarly, interfaces may include means for performing (analog and / or digital) signal or data processing during communication, such as filters, samplers, analog-to-digital converters, signal acquisition and / or reconstruction means, and signal amplifiers, compressors, and / or any encryption / decryption means.

[0068] The data processing circuit 320 can correspond to or include any type of programmable hardware. Therefore, examples of the data processing circuit 320 include, for example, a memory, a microcontroller, a field-programmable gate array, one or more central processing units, graphics processing units, and / or similar devices. To perform the proposed method, the data processing circuit 320 can be configured to access or retrieve an appropriate computer program from its own memory or a separate memory communicatively coupled to it to perform the proposed method.

[0069] As can be seen in the foregoing description, various features are grouped together in the examples for the purpose of rationalizing this disclosure. This approach of the disclosure should not be construed as reflecting an intention that the claimed examples require more features than are expressly recited in each claim. Rather, as reflected in the appended claims, the subject matter of the invention may lie in fewer features than all the features of a single disclosed example. Therefore, the appended claims are incorporated herein by reference, wherein each claim may itself serve as a separate example. While each claim may stand alone as a separate example, it should be noted that although dependent claims may refer to a particular combination with one or more other claims in the claim, other examples may also include combinations of dependent claims with the subject matter of each other dependent claim, or combinations of each feature with other dependent or independent claims. Such combinations are proposed herein unless it is stated that a specific combination is not contemplated. Furthermore, even if a claim is not directly dependent on any other independent claim, the features of that claim are contemplated to be included in that independent claim.

[0070] While specific embodiments have been shown and described herein, those skilled in the art will understand that various alternative and / or equivalent implementations may be substituted for the shown and described specific embodiments without departing from the scope of the present embodiments. This application is intended to cover any adaptations or variations of the specific embodiments discussed herein. Therefore, it is contemplated that the embodiments are limited only by the claims and their equivalents.

Claims

1. A method (100) for fuzz testing software, the method (100) comprising: Obtain (110) seeds for fuzz testing; Obtain the corresponding bitmap size change and average bitmap size change of the test cases for the seed described in (120); The number of times the seed is modified is determined based on the corresponding bitmap size change and the average bitmap size change.

2. The method (100) of claim 1, wherein determining (130) the number of modifications comprises: Determine the corresponding deviation of the bitmap size change for the test case relative to the average bitmap size change; as well as The number of modifications to the seed is determined based on the corresponding deviation of the corresponding test case, such that the first seed, which has a smaller deviation than the second seed, is modified more frequently than the second seed.

3. The method (100) of claim 1 or 2, wherein the method (100) further comprises modifying the seed according to the number of modifications.

4. The method (100) of any of the preceding claims, wherein determining (130) the number of modifications includes adjusting the performance score of the corresponding test case of the seed, and determining the number of modifications such that the first seed is modified more frequently than the second seed.

5. The method (100) as claimed in any of the preceding claims, wherein the method (100) further comprises filtering out one or more seeds of test cases from the modification based on the corresponding information gain of the corresponding test case.

6. The method (100) of claim 5, wherein the information gain depends on the number of bits of information gained in the corresponding test case.

7. The method (100) of claim 5 or 6, wherein filtering out the one or more seeds comprises: The modification filters out seeds from the test cases that contain information with a gain less than the predefined lower information gain threshold or more information with a gain greater than the upper information gain threshold.

8. The method (100) of claim 5 or 6, wherein filtering out the one or more seeds comprises: The seeds that filter out the least and most information about the gain of the test cases from the modifications.

9. The method (100) as claimed in any of the preceding claims, wherein the method (100) further comprises clustering the test cases that cause the software crash based on the corresponding crash causes of the test cases that cause the software crash.

10. The method (100) of claim 9, wherein clustering the test cases comprises: The similarity score of the test case pair is determined based on the execution trajectory of the test case pair; A similarity matrix is ​​obtained based on the similarity scores; as well as The test cases are clustered based on the similarity matrix and the silhouette coefficients of different numbers of clusters.

11. The method (100) as described in any of the preceding claims, wherein the method (100) further comprises analyzing the test cases based on assertions of corresponding crash causes of the test cases.

12. A computer program for performing fuzz testing on software, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method (100) as claimed in any one of claims 1 to 11.

13. A computer-readable data carrier having a computer program as described in claim 11 stored thereon.

14. An apparatus (300), comprising: One or more interfaces (310) for communication; as well as A data processing circuit (320) configured to perform the method (100) as claimed in any one of claims 1 to 11.

Images