Computer-readable data carrier, apparatus, computer program, and method for fuzz testing software

Abstract

Embodiments of the present disclosure relate to a computer-readable data carrier, an apparatus, a computer program, and a method for fuzz testing software. The method comprises obtaining information on an individual path coverage of a seed and obtaining information on an average path coverage of the seeds used for fuzz testing. Further, the method comprises determining a seed energy of the seed based on a ratio of the individual and the average path coverage. The seed energy indicates how often the seed is mutated.

Description

[0001] Description

[0002] Computer-readable data carrier, apparatus, computer program, and method for fuzz testing software

[0003] Embodiments of the present disclosure relate to a computer-readable data carrier, an apparatus, a computer program, and a method for fuzz testing software. In particular, the present disclosure relates to a concept of seed energy scheduling and.

[0004] With increasingly complex software, software testing becomes more and more important in various fields of technology.

[0005] With the help of the rapid development of automatic testing technology, software developers can get a large number of crashes in a short period of time, thus, confirming that there are vulnerabilities or bugs in the software. However, finding the root cause of the crash is a time-consuming and laborious task.

[0006] Two techniques like reverse execution and backward taint analysis can assist in locating the root cause, but do not provide context information or explanation of the underlying fault. An automated analysis approach solves the shortcomings of the above two types of techniques, which are not only identify the root cause of a given crashing input for a binary executable, but also provide the analyst with context information on the erroneous behavior that characterizes crashing inputs.

[0007] Such approach first generates a diverse set of similar inputs based on a single crashing input seed. These inputs either crash the program or induce benign behavior. After that, the program’s states will be traced and the predicates will be generated while each found input is being executed. Finally, it will run a statistical analysis of all predicates to reveal the location of the root cause and provide an explanation of the misbehavior caused by the crash. Such approach can not only handle bugs with no data dependency between crash and root cause, but also uncover root causes when many millions of instructions were executed between crashing location and developer fix. Before analyzing, it first needs to get a large number of crash and non-crash test cases using the fuzzing tool. To improve efficiency, a fuzzing strategy is designed based on the crash mode of coverage-guided AFL (American Fuzzy Loop), which takes a crash input as a seed and mutates it to generate new inputs. Said approach modifies the original AFL to record both crash and non-crash test cases. While this design provides powerful analytical capabilities, it also introduces two drawbacks.

[0008] • Root cause analysis results are not accurate enough. Since the fuzzing strategy is coverage-guided, the large number of crash test cases generated have very different paths to each other. Therefore, the probability that these crash test cases are caused by the same bug is not high. This can lead to inaccurate analysis results.

[0009] • The root cause analysis is too inefficient. It needs to increase the number of crash and non-crash test cases to improve the accuracy of the analysis results. However, with the coverage-guided fuzzing strategy, the number of crash and non-crash test cases stored is extremely unbalanced. The number of non-crash test cases is far more than the number of crashes. In the analysis process, the impact of all non-crash test cases on the analysis results is similar to that of all crash test cases. The large number of non-crash test cases does not increase the accuracy of the results and is therefore nearly useless. On the other hand, each test case consumes the same amount of time during the analysis process. The large number of non-crash test cases consumes a large amount of analysis time without much gain in analysis results, making the analysis process very inefficient.

[0010] Hence, there may be a demand for an improved concept of fuzz testing software.

[0011] This demand can be satisfied by the subject-matter of the appended independent claims. The dependent claims disclose beneficial embodiments thereof.

[0012] The present disclosure particularly discloses a more efficient fuzzing strategy for automated root cause analysis and explanation. Embodiments may be therefore referred to as “’’Root Cause Analysis Fuzz” (RCAfuzz). Implementations of the proposed approach may be also designed based on AFL’s crash mode. Therefore, the proposed approach may also keep only crash test cases when selecting seeds. The difference is that, when assigning energy values to the seeds, the seeds with smaller coverage may get higher energy values. In other words, seeds with smaller coverage may get more mutations and executions. This energy scheduling method of crash seeds can greatly increase the probability that all crashes belong to the same root cause, thereby improving the accuracy of the analysis results. On the other hand, when collecting non-crash test cases, the proposed approach may adopt a more efficient retention strategy, which may be referred to as “Hill Climbing Retention”. The core goal of hill climbing retention is to retain as few non-crash test cases as possible to obtain enough useful information. Since the number of non-crash test cases is greatly reduced, the proposed approach can greatly improve the efficiency of root cause analysis.

[0013] Generally speaking, in other words, embodiments of the present disclosure may be summarized as follows:

[0014] Embodiments of the present disclosure provide a method for fuzz testing software based on a plurality of seeds.

[0015] The method comprises obtaining information on an individual path coverage of a seed and obtaining information on an average path coverage of the seeds used for fuzz testing. Further, the method comprises determining a seed energy of the seed based on a ratio of the individual and the average path coverage. The seed energy indicates how often the seed is mutated.

[0016] This allows a more efficient mutation mechanism and, thereby, a more efficient fuzz testing of software. Apart from that, this may lead to better result of the fuzz testing.

[0017] In practice, the seed may be mutated more often the higher the seed energy, and the seed energy may be higher the lower the ratio. So, seeds with a higher ratio are, e.g., mutated less often than seeds with a lower ratio, In some embodiments, the method further comprises obtaining an information yield of a test case for multiple test cases and selecting one or more test cases for an analysis of the fuzz testing based on their information yield. This, e.g., allows to filter out less informative test cases for a more efficient and / more accurate analysis results.

[0018] In practice, e.g., the test case is selected more likely for the analysis if the information yield is higher than for previous test cases.

[0019] Obtaining the information yield, e.g., comprises obtaining a number of newly added path tuples in an execution path of the test case and obtaining a number of newly changed hit-counts for a particular path tuple in the execution path of the test case. As well it, e.g., comprises obtaining the information yield based on the number of newly added path tuples and the number of newly changed hit-counts.

[0020] In some embodiments, the number of newly added path tuples is weighted by an index of the test case, and obtaining the information yield comprises obtaining the information yield based on the weighted number of newly added path tuples and the number of newly changed hit-counts.

[0021] For example, the information yield is defined by:

[0022] A = i ■ X + i , wherein A denotes the information yield, i denotes the index, X denotes the number of newly added path tuples, and Y denotes the number of newly changed hit-counts.

[0023] A skilled person having benefit from the present disclosure will appreciate that the proposed approach may be implemented as computer-implemented method and / or a computer program.

[0024] Accordingly, embodiments of the present disclosure provide a computer program comprising instructions which, when the computer program is executed by a computer, cause the computer to carry out the proposed method. Further embodiments provide an apparatus comprising one or more interfaces for communication and a data processing circuit configured to execute the proposed method.

[0025] Other embodiments provide a computer-readable data carrier having stored thereon the computer program.

[0026] Further, embodiments are now described with reference to the attached drawings. It should be noted that the embodiments illustrated by the referenced drawings show merely optional embodiments as an example and that the scope of the present disclosure is by no means limited to the embodiments presented:

[0027] Brief description of the drawings

[0028] Fig. 1 shows a flow chart schematically illustrating an embodiment of a method of fuzz testing software;

[0029] Fig. 2 shows a flow chart of another embodiment of the method; and

[0030] Fig. 3 shows a block diagram of an apparatus according to the proposed approach.

[0031] Fuzz testing, also known as fuzzing, is a software testing technique that involves providing random, invalid, or unexpected data as input to a program in order to discover vulnerabilities, bugs, or unexpected behavior. The goal of fuzz testing is to identify security vulnerabilities and software defects that may not be easily found through traditional testing methods.

[0032] In fuzz testing, a tool called a fuzzer is used to generate and submit a large number of test inputs to the target software. The fuzzer systematically manipulates input data, introducing variations and mutations to simulate real-world scenarios where the software might encounter unexpected or malicious inputs. Fuzz testing is particularly effective in finding issues related to buffer overflows, input validation errors, and other types of security vulnerabilities.

[0033] Seeds in the context of fuzz testing refer to the initial input data or test cases used by the fuzzer to generate subsequent test inputs. Seeds serve as the starting point for the fuzzer, and from these seeds, the fuzzer generates a large number of mutated or modified inputs to explore different paths through the software.

[0034] The quality and diversity of seeds can significantly impact the effectiveness of fuzz testing. Well-chosen seeds increase the likelihood of discovering various types of vulnerabilities. Fuzzers may use different strategies for seed selection, such as random inputs, inputs extracted from real-world scenarios, or predefined inputs based on known patterns. The idea is to cover a broad spectrum of input possibilities to maximize the chances of uncovering hidden software flaws.

[0035] In summary, fuzz testing is a dynamic and automated testing technique that relies on the generation of diverse and often unexpected inputs to identify vulnerabilities in software. Seeds play a crucial role as the initial input data from which the fuzzer generates a multitude of test cases, helping to uncover potential issues in the target software.

[0036] Embodiments of the present disclosure are based on the finding that seeds with a relatively high code coverage require a high computational effort while seeds with a relatively low code coverage do not gain a lot of information. One idea of the proposed approach is to reduce a number of seed mutations of such seeds using a more efficient energy scheduling mechanism that prioritizes seeds with an average execution path coverage, as outlined below with reference to the drawings in more detail.

[0037] Fig. 1 shows a flow chart schematically illustrating an embodiment of a method 100 of fuzz testing software. The method 100 comprises obtaining 110 information on an individual path coverage (also referred to as “code coverage”) of a seed and obtaining 120 information on an average path coverage of the seeds used for fuzz testing.

[0038] For this, e.g., AFL’s appropriate code coverage analysis function may be applied. In practice, e.g., the individual code coverage of the seeds is obtained. Then, the average code coverage may be obtained from the individual code coverage of the seeds.

[0039] As well, the method 100 comprises determining 130 a seed energy of the seed based on a ratio of the individual and the average path coverage. The seed energy indicates how often the seed is mutated.

[0040] So, e.g., seeds with an individual code coverage closer to the average code coverage are mutated more often than others. In this way, seeds resulting in a higher computational effort and / or in less information gain may be mutated less often for a higher efficiency of the fuzz testing.

[0041] So, in other words, the proposed approach proposes a new fuzzing strategy, which can get crash test cases with lower overall coverage than other approaches. This fuzzing strategy is designed based on a different seed energy scheduling mechanism. Based on these crash test cases with lower coverage, the proposed approach can obtain more accurate root cause analysis results.

[0042] Additionally or alternatively, the proposed approach suggests a test case retention mechanism. In particular, it proposes a way to calculate a distance between non-crash test cases. This can effectively represent the amount of new information or an information yield in non-crash test cases. Then, based on the calculated non-crash distance mentioned above, a non-crash test case retention algorithm called “hill-climbing retention” is proposed. Hill-climbing retention can achieve uniform retention of a small number of test cases at all distance levels. It can obtain enough information for root cause analysis while only retaining a very small amount of non-crash test cases, thus greatly speeding up the efficiency of root cause analysis.

[0043] In embodiments, the proposed fuzzing or energy scheduling strategy and the test case retention mechanism may be implemented alone or in combination, as outlined in more detail with reference to Fig. 2. So, the embodiments of the present disclosure may provide a method for processing test cases in fuzz testing software. In embodiments, the method for processing test cases comprises obtaining an information yield of a test case for multiple test cases and selecting one or more test cases for an analysis of the fuzz testing based on their information yield. This method may include various features described herein with the proposed test case retention mechanism and / or the proposed seed energy scheduling.

[0044] Fig. 2 shows a flow chart of another embodiment of the proposed approach where the proposed energy scheduling strategy and the test case retention mechanism are implemented in combination.

[0045] From Fig. 2, one can see that the whole workflow may be divided into two phases, which are fuzzing (see module 210) and analysis (see module 220).

[0046] In the fuzzing phase 210, crash test cases 211 are evaluated and their seeds are mutated (see 214). In doing so, the proposed seed energy scheduling is applied (see 212).

[0047] • Seed Energy Scheduling for Reducing or Minimizing Coverage: The seed energy determines how often the seed will be fuzzed. The higher the energy, the more often it will be fuzzed, and vice versa. Therefore, seeds with higher energy will be mutated and executed more often. The proposed approach first calculates the ratio of the current seed’s path coverage area to the average coverage area of all seeds. The larger the ratio, the larger the coverage area of the seed. The seed is assigned a low energy value. Conversely, the smaller the ratio, the smaller the coverage area. This seed will be assigned a high energy value. As indicated, a feedback loop may be applied where the seed energy scheduling mechanism is iteratively applied to further crash test cases 215 resulting from the mutated seeds.

[0048] Further, the proposed test case retention mechanism (“Hill-Climb(ing) Retention”) is applied (see 213).

[0049] • Hill-Climbing Retention: During the root cause analysis, each crash or non-crash test case requires a certain amount of time for processing and computation. Too many test cases can seriously slow down the root cause analysis. On the other hand, not every test case has the same amount of new information. Low information test cases have limited impact on the accuracy of results 222 and can be approximated as noise. Therefore, how to reduce the number of test cases as much as possible while ensuring sufficient information is of great help to improve the efficiency of root cause analysis. In the design of the proposed approach, the present disclosure proposes the hill-climbing retention method. First, the proposed approach defines the formula for adding information. XLrepresents the number of newly added path tuple in the execution path of the i-th test case. Ytrepresents the number of newly changed hit-counts for a particular path tuple in the execution path of the i -th test case. / I represents the amount of new information brought by the i -th test case. According to our theoretical analysis of the amount of new information, the greater the Xtor Ytof the test case, the greater the amount of new information brought. Moreover, the influence of Xtof new test cases on the amount of new information will increase with the increase of the number of test cases, but the influence of Yton the amount of new information will not change with the increase of the number of test cases. Therefore, the formula for the amount of new information (information yield) may be as follows:

[0050] A = i ■ X + i ,

[0051] As can be seen from the above equation, the square of Atis equal to the square of Xtplus the square of Ytmultiplied by the weight (and index) i. The weight i is added to balance the influence of Xt and Yton the distance At. Based on the definition of the newly added information (At), the retention strategy / mechanisms selects (non-crash) test cases 216 which are used for the analysis 220. When the amount of new information (At) of the current test case is larger than all previous test cases G4j_i, ■ ■ ■ , li), The proposed approach will retain the test case with a high probability. If the amount of new information Atis not the largest, the probability of being retained will be very small or zero. Based on this strategy, The proposed approach always retains the test cases with the largest amount of new information with a high probability, and discards the test cases with insufficient information. In the process of fuzzing, starting from the first retained test case, the amount of new information quickly reaches the maximum value, and the number of retained test cases does not increase. From the dimension of information volume, the amount of new information added to the retained test cases increases step by step like climbing a mountain, so we named this retention strategy “hill-climbing retention”.

[0052] In the analysis 220, the obtained test cases 215 and 216 are then evaluated, e.g., to find errors and / or vulnerabilities. For this, an appropriate calculation 221 is applied to the test cases to obtain results 222 indicating such errors and / or vulnerabilities.

[0053] Such embodiments have various advantages:

[0054] First, compared with existing technologies, The proposed approach’s fuzzing strategy can output a set of crash test cases with smaller coverage. The root cause analysis results based on this set of crash test cases will be more accurate than existing technologies. Secondly, the calculation method for the distance between test cases proposed in the proposed approach can effectively represent the amount of new information of test cases. This method of representing the amount of added information is a capability that existing technologies do not have. Third, the hill-climbing retention strategy of non-crash test cases proposed by the proposed approach greatly reduces the number of non-crash test cases that need to be collected. This makes the root cause analysis process based on the proposed approach much faster than existing solutions.

[0055] The proposed approach is a new fuzzing strategy suitable for root cause analysis.

[0056] Based on the test cases obtained by the proposed approach, the efficiency and accuracy of root cause analysis can be significantly improved. Fig. 2 shows the workflow of root cause analysis.

[0057] Based on the seed energy scheduling for Reducing or Minimizing Coverage (compared to other approaches), the proposed approach can obtain crash test cases caused by the same root cause with a greater probability. Therefore, subsequent root cause analysis can get more accurate results. Based on the hill-climbing retention strategy, The proposed approach can get enough information while retaining a small number of non-crash test cases. This greatly improves the efficiency of subsequent root cause analysis.

[0058] The proposed approach has two advantages to be useful in commercial applications. First of all, the proposed approach can greatly improve the efficiency of developers’ root cause analysis of crash test case. Secondly, the proposed approach can greatly improve the accuracy of root cause analysis of developers on crash test case. Based on these two advantages, The proposed approach can locate the root cause more quickly and accurately, so as to reduce the debugging time of developers. The proposed approach can be applied in the development process of vehicle system software to improve software development efficiency and save manpower and material resources. Not only that, the proposed approach can also be applied in the software development process of cloud services to reduce the labor cost of cloud server developers.

[0059] In particular, the proposed approach may be applied in an apparatus, as laid out in more detail below with reference to Fig. 3.

[0060] Fig. 3 shows a block diagram schematically illustrating an embodiment of such an apparatus 300. The apparatus comprises one or more interfaces 310 for communication and a data processing circuit 320 configured to execute the proposed method.

[0061] In embodiments, the one or more interfaces 310 may comprise wired and / or wireless interfaces for transmitting and / or receiving communication signals in connection with the execution of the proposed concept. In practice, the interfaces, e.g., comprise pins, wires, antennas, and / or the like. As well, the interfaces may comprise means for (analog and / or digital) signal or data processing in connection with the communication, e.g., filters, samples, analog-to-digital converters, signal acquisition and / or reconstruction means as well as signal amplifiers, compressors and / or any encryption / decryption means.

[0062] The data processing circuit 320 may correspond to or comprise any type of programable hardware. So, examples of the data processing circuit 320, e.g., comprise a memory, microcontroller, field programable gate arrays, one or more central, and / or graphical processing units. To execute the proposed method, the data processing circuit 320 may be configured to access or retrieve an appropriate computer program for the execution of the proposed method from a memory of the data processing circuit 320 or a separate memory which is communicatively coupled to the data processing circuit 320.

[0063] In the foregoing description, it can be seen that various features are grouped together in examples for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed examples require more features than are expressly recited in each claim. Rather, as the following claims reflect, subject matter may lie in less than all features of a single disclosed example. Thus, the following claims are hereby incorporated into the description, where each claim may stand on its own as a separate example. While each claim may stand on its own as a separate example, it is to be noted that, although a dependent claim may refer in the claims to a specific combination with one or more other claims, other examples may also include a combination of the dependent claim with the subject matter of each other dependent claim or a combination of each feature with other dependent or independent claims. Such combinations are proposed herein unless it is stated that a specific combination is not intended. Furthermore, it is intended to include also features of a claim to any other independent claim even if this claim is not directly made dependent to the independent claim. Although specific embodiments have been illustrated and described herein, it will be appreciated by those of ordinary skill in the art that a variety of alternate and / or equivalent implementations may be substituted for the specific embodiments shown and described without departing from the scope of the present embodiments. This application is intended to cover any adaptations or variations of the specific embodiments discussed herein. Therefore, it is intended that the embodiments be limited only by the claims and the equivalents thereof.

Claims

Patent claims1 . A method (100) for fuzz testing software based on a plurality of seeds, the method (100) comprising: obtaining (110) information on an individual path coverage of a seed; obtaining (120) information on an average path coverage of the seeds used for fuzz testing; and determining (130) a seed energy of the seed based on a ratio of the individual and the average path coverage, wherein the seed energy indicates how often the seed is mutated.

2. The method (100) of claim 1 , wherein the seed is mutated more often the higher the seed energy, and wherein the seed energy is higher the lower the ratio.

3. The method (100) of claim 1 or 2, wherein the method (100) further comprises: obtaining an information yield of a test case for multiple test cases; and selecting one or more test cases for an analysis of the fuzz testing based on their information yield.

4. The method (100) of claim 3, wherein the test case is selected more likely for the analysis if the information yield is higher than for previous test cases.

5. The method (100) of claim 3 or 4, wherein obtaining the information yield comprises: obtaining a number of newly added path tuples in an execution path of the test case;obtaining a number of newly changed hit-counts for a particular path tuple in the execution path of the test case; and obtaining the information yield based on the number of newly added path tuples and the number of newly changed hit-counts.

6. The method (100) of claim 5, wherein the number of newly added path tuples is weighted by an index of the test case, and wherein obtaining the information yield comprises obtaining the information yield based on the weighted number of newly added path tuples and the number of newly changed hit-counts.

7. The method (100) of claim 6, wherein the information yield is defined by:A = i ■ X + T2, wherein A denotes the information yield, i denotes the index, X denotes the number of newly added path tuples, and Y denotes the number of newly changed hit-counts.

8. A computer program comprising instructions which, when the computer program is executed by a computer, cause the computer to carry out the method (100) of any one of the claims 1 to 7.

9. An apparatus (300) comprising: one or more interfaces (310) for communication; and a data processing circuit (320) configured to execute the method (100) of any one of the claims 1 to 7.

10. A computer-readable data carrier having stored thereon the computer program of claim 8.

Images