Method for generating fuzz test framework

Abstract

The invention relates to a method (1) for generating a fuzz test framework (10). According to the method (1), a software source code (2) is provided. And selecting a target function (3) to be fuzzed from the software source code (2) and calling a sample program (5) of the target function (3). The sample program (5) is then compiled to generate a bit code (8) and the objective function (3) is fragmented from the bit code (8) based on the objective function (3) to obtain the fuzz test framework (10). The invention also relates to a computer program product comprising instructions which, when executed by said computer, cause said computer to execute the above method (1).

Description

Technical Field

[0001] The present invention relates to a method of generating a fuzzing harness using a sample program that calls a target function to be fuzz tested. The present invention also relates to a computer program that, when executed by a computer, causes the computer to generate a fuzzing harness. Background Art

[0002] Software analysis is an important task in automotive software development due to the high standards set by the automotive industry to ensure the quality of its safety-critical software. In addition to safety requirements, the recent introduction of safety compliance requirements in automotive software also requires a thorough analysis of automotive software to eliminate potential security vulnerabilities.

[0003] Software analysis has been introduced into the software development process since a long time ago, and software analysts keep improving the effectiveness and efficiency of the analysis process. Among the categories of software analysis, dynamic analysis is becoming more and more popular in the software community. Instrumented fuzzing is one of the methods in dynamic analysis. Instrumented fuzzing has a fuzz testing engine that generates random inputs to the software to be fuzz tested, trying to reveal vulnerabilities in the software. In order to adopt instrumented fuzzing as a software analysis method, it is necessary to create a fuzz testing framework that connects the fuzz testing engine to the fuzz testing target. Software developers or testers with little fuzz testing background need a lot of improvement efforts to understand how to write a fuzz testing framework. Therefore, automation of framework generation is needed to reduce this improvement effort.

[0004] In particular, a fuzz testing framework is a test case or a specific test target. It is an "executable entry point" to the fuzz testing target and connects random inputs from the input generation engine to the fuzz testing target. When a tester wants to pass random data to a specific part of the fuzz testing target, he or she needs to create a fuzz testing framework. In other words, the fuzz testing framework can determine the analysis coverage, that is, to enable random inputs to reach every part of the fuzz testing target.

[0005] For example, if one wants to assess the security of a server application in a client-server architecture, the tester needs a testing framework to perform various setup tasks, such as starting the server process before sending network input (i.e., random input) from a client connection. Another example would be testing a dynamically loaded library (DLL). Since a DLL itself cannot run independently, a fuzz testing framework needs to be written, in this case, using an executable program of the DLL that passes input from the command line into the DLL function.

[0006] In order to prepare a fuzz testing framework, the source code of the fuzz test target must be obtained from the software version control system. In addition, a sample program or consumer program that uses the fuzz test target must be obtained. Then, it is necessary to check how the fuzz test target is used in the sample program, and the result is called trace information. Creating trace information can be done via dynamic tracing (e.g., function execution tracing via running the program) or static tracing (e.g., function call tracing via scanning the source code). Then, the trace information is used to create a test framework. The tasks include obtaining call trace information of the target function, parsing the target function, and creating test logic from the collected information. If all is done manually, the above process may be cumbersome and time-consuming. An alternative is to convert the trace information into test logic semi-automatically, where the user needs to select the appropriate framework from a given selection generated from the framework.

[0007] An example of automated test logic generation in fuzz testing is given in D. Bucur et al.’s article “FUDGE: Fuzz Driver Generation at Scale” (https: / / doi.org / 10.1145 / 3338906.3340456). In K. Ispoglou et al.’s article “FuzzGen: Automatic Fuzzer Generation” (Proceeding of the 29 th Another example of automatic fuzz tester generation is given in the USEIX Security Symposium (Proceedings of the 29th USEIX Security Conference, 2020). Summary of the invention

[0008] It is an object of the present invention to provide an improved method for generating a fuzz testing framework. Another object of the present invention is to provide a computer program comprising instructions which, when executed by a computer, implement the improved method for generating a fuzz testing framework.

[0009] The objects of the invention are solved by the subject-matter of the independent claims, wherein further embodiments are incorporated in the dependent claims.

[0010] According to one aspect of the present invention, a method for generating a fuzz testing framework is provided. In this context, fuzz testing may also be referred to as robustness testing, fuzzy testing or negative testing.

[0011] According to the method, a piece of software source code is provided. The software source code includes source code of the software to be tested. A target function to be fuzz tested is selected from the software source code. The selection of the target function can be performed manually by a user. The selection of the target function can also be performed automatically, wherein the target function can be selected based on the appearance of the target function in the source code or based on the date when the target function was created.

[0012] Furthermore, a sample program is selected from the software source code, wherein the sample program calls the target function. The sample program may be part of a test suite for the software, but it may also be a non-test consuming program. If more than one sample program that calls the target function can be identified, the method may be repeated for more than one sample program, or the sample program may be selected, for example, based on the size of the sample program, where smaller sample programs will be preferred.

[0013] Once the target function and sample program have been selected, the sample program is compiled to generate bit code, for example, with a file extension of ".bc".

[0014] Based on the bitcode, the target function is fragmented. For the fragmentation process, the desired target function is provided to a fragmentor. The result of the fragmentation process is a fuzz testing framework that can be used to test the target function. When the target function is fragmented from the bitcode, portions of the bitcode that are not relevant to the target function are removed, thereby requiring less computing resources (such as memory and / or CPU time).

[0015] Using the proposed method for generating a fuzz testing framework, the quality of the generated fuzz testing framework is ensured, since the method does not depend on the user's skills. Moreover, time is saved by applying the method, since the tedious work of generating a fuzz testing framework is now automated. Finally, it is sufficient for a developer who does not have special software analysis skills to generate a fuzz testing framework using the above method, so this task no longer requires highly skilled developers.

[0016] According to an embodiment, the software source code is the source code of software for a vehicle, in particular the source code of software for a vehicle with an advanced driver assistance system (ADAS), and more particularly the source code of software for an autonomous vehicle. In these cases, accurately testing the software and removing errors and vulnerabilities may save lives, because errors in ADAS or autonomous vehicles may lead to casualties in a vehicle collision in the worst case.

[0017] According to an embodiment, after selecting the target function, external dependencies of the target function are identified. External dependencies are functions used by the target function and are functions that are not of interest to the analysis. The target function is then further fragmented based on the external dependencies.

[0018] According to an embodiment, the data flow of at least one of the external dependencies is modeled. That is, fragmentation can be performed by using the modeled data flow rather than the external dependencies themselves. This enables more accurate and efficient fragmentation of the program.

[0019] According to an embodiment, at least one of the external dependencies is replaced by a stub implementation of the external dependency. In particular, external dependencies that require a lot of integration work to include may be replaced by a stub implementation. Then, fragmenting the target function from the bit code also depends on the stub implementation.

[0020] According to an embodiment, the method further comprises identifying an entry point of the sample program that will lead to the call of the target function. This step is performed after selecting the sample program and before compiling the sample program. The compiled sample program is then executed starting from the entry point. Therefore, when compiling the sample program, the program code before the entry point that does not affect the call of the target function has been omitted, thereby further saving computing resources.

[0021] According to an embodiment, the sample program is a multi-threaded program. In this case, identifying the entry point includes identifying other functions in multiple threads that need to be called before entering the entry point. Therefore, for each thread, there may be a separate thread-specific entry point, and compiling the sample program may start from each of the specific entry points of each thread.

[0022] According to an embodiment, the bitcode is LLVM bitcode, i.e. virtual machine bitcode. LLVM bitcode is very versatile and very suitable for generating fuzz testing frameworks. In this case, the compiler wllvm can be used, in particular replacing the standard compiler used for the sample program, such as gcc or clang. As a fragmenter, the llvm fragmenter can be used. In addition, in order to model external dependencies, the functionModelAddUse API can be used.

[0023] According to an embodiment, the method also includes returning the fuzz test framework to a source code file that generates a framework source. Therefore, the framework source is the source code for the fragmented program. In particular, the source code file is a human-readable code, such as a c file. The framework source is then checked and modified, in particular by a user. Since the symbol information of the variable is not retained during fragmentation, the symbol is represented by "var1", "var2", etc. in the returned source code. Then, the modified framework source is compiled to generate a modified fuzz test framework, which is improved compared to the original fuzz test framework.

[0024] According to an embodiment, the method further comprises fuzz testing the target function using a fuzz testing framework. That is, random input data is provided to the target function via the fuzz testing framework, and thus potential errors in the target function can be detected. Once such an error is detected, it must be further analyzed and then patched, which is usually done by experienced developers. As a result, the target function has fewer errors, becomes more stable, more reliable and less vulnerable to attack.

[0025] According to another aspect of the present invention, a computer program product is provided. The computer program product comprises instructions which, when executed by a computer, cause the computer to perform the steps according to the above description. The technical advantages of the computer program product correspond to the technical advantages of the method and have been explained in detail above. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] These and other aspects of the invention will become apparent from and be further elucidated with reference to the embodiments described by way of example in the following description and with reference to the accompanying drawings, in which:

[0027] Figure 1 A flow chart illustrating an embodiment of a method for generating a fuzz testing framework;

[0028] Figure 2 A flow chart illustrating another embodiment of a method for generating a fuzz testing framework; and

[0029] Figure 3 A flow chart of yet another embodiment of a method for generating a fuzz testing framework is shown.

[0030] In the figures, elements corresponding to elements already described may have the same reference numerals.Examples, implementations or optional features, whether or not indicated as non-limiting, should not be construed as limiting the invention as claimed. DETAILED DESCRIPTION

[0031] Figure 1 A flow chart of an embodiment of a method 1 for generating a fuzz testing framework is shown. According to the method 1, a software source code 2 is provided. From the software source code 2, a target function 3 to be fuzz tested is selected 4. The selection can be performed manually or based on certain rules, for example, firstly a target function 3 widely used in the software source code 2 is selected. Moreover, a sample program 5 that calls the target function 3 is selected 6 from the software source code 2. The sample program 5 is then compiled 7 to generate a bit code 8. From the bit code 8, the target function 3 is fragmented 9 to obtain a fuzz testing framework 10.

[0032] By using the method 1 for generating a fuzz testing framework 10, the quality of the generated fuzz testing framework 10 is ensured, because the method does not rely on the user's skills. Moreover, time is saved by applying the method 1, because the tedious work of generating the fuzz testing framework 10 is now automated. Finally, a developer without special software analysis skills is sufficient to generate a fuzz testing framework 10 using the above-described method 1, so this task no longer requires highly skilled developers.

[0033] Figure 2 FIG. 1 is a flow chart showing another embodiment of a method 1 for generating a fuzzy testing framework 10. Figure 1 Compared to the method 1 of , the method further comprises the step of identifying 11 external dependencies 12 of the target function 3. For said external dependencies 12, the data flow may be modeled and / or the external dependencies 12 may be replaced by stub implementations. In either case, the result will be used to generate the bitcode 8 and perform the fragmentation 9. As a result, the fuzz testing framework 10 is smaller and requires less computing resources (such as memory and / or CPU time).

[0034] Figure 3 FIG. 1 is a flowchart of another embodiment of a method 1 for generating a fuzzy testing framework 10. Figure 1 Compared to method 1, the fuzz testing framework 10 is rolled back 13 to source code files that produce a framework source 14. The framework source 14 can then be inspected and modified 15, particularly by an experienced developer, to create a modified framework source 16. Finally, the modified framework source 16 is compiled 17 to generate a modified fuzz testing framework 18, which is improved compared to the original fuzz testing framework 10.

[0035] Other variations of the disclosed embodiments may be understood and implemented by those skilled in the art by studying the drawings, the disclosure and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. Any reference signs in the claims should not be construed as limiting the scope of the claims.

Claims

1. A method (1) for generating a fuzzy testing framework (10), the method (1) comprising: Provide a piece of software source code (2); Selecting a target function to be fuzz tested (3) from the software source code (2); Selecting a sample program (5) from the software source code (2) that calls the target function (3); Compiling the sample program (5) to generate bit code (8); Based on the target function (3), the target function (3) is fragmented from the bit code (8) to obtain the fuzzy testing framework (10).

2. The method (1) according to claim 1, wherein: The software source code (2) is the source code of software for a vehicle, in particular the source code of software for a vehicle with an advanced driver assistance system, and more particularly the source code of software for an autonomous driving vehicle.

3. The method (1) according to claim 1 or 2, wherein: After selecting the target function (3) and selecting the sample program (5), external dependencies (12) of the target function (3) are identified, and the target function (3) is further fragmented based on the external dependencies (12).

4. The method (1) according to claim 3, wherein: The data flow of at least one of the external dependencies (12) is modeled.

5. The method (1) according to claim 3 or 4, wherein: At least one of the external dependencies (12) is replaced by a stub implementation of the external dependency (12).

6. The method (1) according to any one of claims 1 to 5, wherein After selecting the sample program (5) and before compiling the sample program (5), the method (1) further comprises identifying an entry point of the sample program (5) that will result in a call to the target function (3), and executing the compilation of the sample program (5) starting from the entry point.

7. The method (1) according to claim 6, wherein: The sample program (5) is a multi-threaded program, and identifying the entry point includes identifying other functions in the multi-threads to be called before entering the entry point.

8. The method (1) according to any one of claims 1 to 7, wherein The bitcode (8) is the LLVM bitcode.

9. The method (1) according to any one of claims 1 to 8, wherein the method (1) further comprises: Returning the fuzz testing framework (10) to a source code file that generates a framework source (14); Checking and modifying the framework source (14); as well as The modified framework source (16) is compiled to generate a modified fuzz testing framework (18).

10. The method (1) according to any one of claims 1 to 9, wherein the method (1) further comprises fuzz testing the target function (3) using the fuzz testing framework (10).

11. A computer program product comprising instructions which, when said program is executed by a computer, cause said computer to perform the steps of the method (1) according to any one of claims 1 to 10.

Images