Method for testing computer program

Abstract

According to various embodiments, a method for testing a computer program is provided, having: training a machine learning model for predicting coverage rates of the computer program for test inputs fed to the machine learning model, said coverage rates being reached if the computer program is executed using the test inputs as inputs; testing the computer program in a plurality of iterations, where in each iteration, a test input is generated; predicting a coverage rate of the computer program by means of the trained machine learning model, the coverage rate being reached if the computer program is executed using the generated test input as an input; determining whether a total coverage rate hitherto achieved by testing the computer program increases by the predicted coverage rate; and in response to having determined that the total coverage hitherto achieved by testing the computer program increases by the predicted coverage, executing the computer program with the generated test input.

Description

Technical Field

[0001] The present disclosure relates to a method for testing a computer program. Background Art

[0002] An important part of software application development is testing. In particular, errors that cause application failures should be identified and corrected. An example for software testing is the dynamic software testing method fuzzing.

[0003] A black-box fuzzer generates input for a target program (i.e., a computer program to be tested) without knowing its internal behavior or its implementation, i.e., from the fuzzer's point of view, the computer program is a black-box computer program. Thus, in this case, the source code of the target program is not available (to the fuzzer), which prevents the fuzzer from obtaining feedback about the coverage achieved when testing (e.g., each type of code coverage, such as path, line, or branch coverage), which guides the generation of additional test cases. A black-box fuzzer thus tests in practice on a random basis and has no chance to improve over time. This is common when testing embedded devices and their software.

[0004] Grey box fuzzers, ie fuzzers that also do not have the program code but have coverage information and can select test inputs accordingly, will be significantly more effective. However, obtaining this coverage information, for example by using simulators or stubs, is complex and not possible, particularly in the case of embedded systems.

[0005] Therefore, a low-cost approach that enables gray-box fuzz testing of black-box computer programs, in particular computer programs for embedded systems, is desirable. Summary of the invention

[0006] According to various embodiments, a method for testing a computer program is provided, comprising: training a machine learning model for predicting coverage of the computer program for test inputs fed to the machine learning model, respectively, the coverage being achieved if the computer program is executed using the test inputs as input; testing the computer program in a plurality of iterations, wherein in each iteration,

[0007] Generate test inputs (e.g. via mutation of previous test inputs or seeds from a corpus);

[0008] predicting, with the aid of the trained machine learning model, the coverage of the computer program that would be achieved if the computer program were executed using the generated test inputs as inputs (i.e., the generated test inputs are fed to the trained machine learning model and the output of the trained machine learning model in response to the inputs is taken as an estimate of the coverage);

[0009] determine whether the total coverage achieved to date by testing the computer program is increased by the predicted coverage; and

[0010] In response to having determined that the total coverage achieved to date by testing the computer program is increased by the predicted coverage, executing the computer program using the generated test inputs.

[0011] The above method enables grey-box fuzz testing of black-box computer programs, especially in the case of fuzz testing of software for embedded systems (or devices), because in this case the possibility of using other ways to implement grey-box fuzz testing (such as static instrumentation or using emulators) is greatly limited.

[0012] Coverage is, for example, code coverage, such as path, line or branch coverage, or information about which functions or other program parts, such as functions, are implemented when the computer program is executed with corresponding test inputs as input.

[0013] According to various embodiments, a machine learning model (e.g., a neural network) is trained separately for a computer program to be tested. However, this can occur starting from a machine learning model that has been pre-trained for one or more other computer programs.

[0014] If it is determined that the total coverage achieved so far by testing the computer program is not increased by the predicted coverage, the generated test inputs are discarded, for example (ie no test run is carried out for them).

[0015] Various embodiments are described below.

[0016] Embodiment 1 is a method for testing a computer program as described above.

[0017] Embodiment 2 is a method according to embodiment 1, wherein in each iteration, the test input is generated from another test input based on whether the coverage already predicted by the trained machine learning model for the other test input is a total coverage, and the total coverage increases the total coverage achieved by the test before the computer program is executed using the other test input as input.

[0018] Thus, the oracle's estimate can be used not only to decide whether to execute the corresponding test case for the test input, but also to decide whether the test input is used as a basis for generating other test inputs (i.e., for example, added to a corpus). In other words, the method can therefore also have, in response to determining that the total coverage achieved so far by testing the computer program is increased by the predicted coverage, adding the generated test input to the set of test inputs from which test inputs for further iterations are generated (e.g., by mutation). The efficiency of fuzz testing can thereby be improved.

[0019] Embodiment 3 is a method according to embodiment 1 or 2, further comprising: checking whether the total coverage achieved when testing the computer program has been increased by executing the computer program using the generated test inputs, and in response to the total coverage achieved when testing the computer program having been increased by executing the computer program using the generated test inputs, adding the generated test inputs to the set of following test inputs, and generating test inputs for further iterations from the test inputs.

[0020] Therefore, the predictions of the machine learning model (oracle) can be verified so that only test inputs that have actually led to an increase in total coverage are added to the corpus. The verified coverage can also be used as benchmark truth information for further training of the machine learning model. Thereby, the efficiency of fuzz testing can be improved.

[0021] Embodiment 4 is a method according to any one of embodiments 1 to 3, comprising: generating test inputs on a test system, determining the coverage of the computer program on the test system, and in response to the test system having determined that the total coverage achieved to date by testing the computer program is increased by the predicted coverage, executing the computer program using the generated test inputs on a system embedded with the test system.

[0022] Thus, the method enables grey-box fuzz testing of black-box computer programs for embedded systems.

[0023] Embodiment 5 is a software testing system, which is configured to implement the method according to any one of embodiments 1 to 4.

[0024] Embodiment 6 is a computer program having instructions, which, when executed by a processor, cause the processor to implement the method according to any one of embodiments 1 to 4.

[0025] Embodiment 7 is a computer-readable medium storing instructions, which, when executed by a processor, causes the processor to implement the method according to any one of embodiments 1 to 4. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In the accompanying drawings, like reference numerals generally refer to the same parts in completely different views. The accompanying drawings are not necessarily drawn to scale, with emphasis instead generally being placed on depicting the principles of the invention. In the following description, various aspects are described with reference to the following drawings.

[0027] Figure 1 A computer used to develop and / or test software applications is shown.

[0028] Figure 2 Explain the data collection used to train a machine learning model to predict coverage.

[0029] Figure 3 Explain by Figure 2 The collected training data is used to train the machine learning model.

[0030] Figure 4 Explain by Figure 3 Controlled fuzz testing of trained machine learning models.

[0031] Figure 5 A flow chart is shown which illustrates a method for testing a computer program according to one embodiment. DETAILED DESCRIPTION

[0032] The following detailed description relates to the attached drawings, which illustrate the particular details and aspects of the present disclosure for the purpose of illustration, in which the present invention may be performed. Other aspects may be used, and structural, logical and electrical changes may be implemented without departing from the scope of protection of the present invention. Because some aspects of the present disclosure may be combined with one or more other aspects of the present disclosure to form new aspects, the various aspects of the present disclosure are not necessarily mutually exclusive.

[0033] Various examples are described in more detail below.

[0034] Figure 1 A computer 100 is shown for use in developing and / or testing software applications.

[0035] The computer 100 has a CPU (Central Processing Unit) 101 and a working memory (RAM) 102. The working memory 102 is used to load a program code from a hard disk 103, for example, and the CPU 101 executes the program code.

[0036] In this example, it is assumed that a user intends to utilize computer 100 to develop and / or test a software application.

[0037] To this end, the user executes the software development environment 104 on the CPU 101 .

[0038] The software development environment 104 enables a user to develop and test applications (i.e. software) 105 for various devices 106, i.e. target hardware, such as embedded systems for controlling robotic devices including robot arms and autonomous vehicles, or also for mobile (communication) devices. For this purpose, the CPU 101 can execute an emulator as part of the software development environment 104 in order to simulate the behavior of the corresponding device 106 for which the application is or has been developed. The software development environment 104 can also be regarded as or designed as a software testing environment if the software development environment is only used to test software from other sources.

[0039] The user can distribute the finished application to the corresponding device 106 via the communication network 107. Instead of via the communication network 107, this can also be done in a different way, for example by means of a USB stick.

[0040] Before this happens, however, the user should test the application 105 in order to avoid distributing an application that does not function as intended to the device 106. This may be the case even if the user did not write the application 105 himself on the computer 100. In particular, it may be the case that the user does not have the source code of the application but only its executable code (i.e. binary program), i.e. if the application (computer program) 105 is a black box computer program from the tester's perspective.

[0041] The testing method is so-called fuzz testing. Fuzz testing (or Fuzz-Testing) is an automated software testing method in which invalid, unexpected or random data is fed as input to the computer program to be tested. The program is then monitored for exceptions, such as abnormal termination, missing failed integrated code assertions or potential memory leaks.

[0042] Fuzzers (i.e., test programs using fuzz testing) are typically used to test programs that process structured inputs. The structure is specified, for example, in a file format or file format or protocol and distinguishes between valid and invalid inputs. Effective fuzzers produce semi-valid inputs that are "valid enough" so as not to be rejected outright by the input parser of the program under test, but are "invalid enough" to reveal edge cases and unexpected behavior that are not handled correctly in the program under test.

[0043] The following describes the terminology used in conjunction with fuzz testing:

[0044] Fuzz testing (or Fuzz-Testing) is an automated testing process to send randomly generated inputs to the target program (the program to be tested) and observe its response.

[0045] A fuzzer or fuzz testing engine is a program that automatically generates inputs. Therefore, the program is not associated with the software to be tested (e.g., through instrumentation). However, the fuzzer has the ability to instrument the code, generate test cases, and execute the program to be tested. Known examples are AFL and libfuzzer.

[0046] A fuzz target is a software program or a functionality that should be tested by fuzz testing. The main feature of a fuzz target should be that it accepts potentially untrusted inputs generated by the fuzzer during the fuzz testing process.

[0047] A fuzz test is a combined version of a fuzzer and a fuzz target. Thus, the fuzz target can be instrumented code, with the fuzzer associated with its input (i.e., providing the input). Fuzz tests are executable. The fuzzer can also start, observe, and stop multiple fuzz tests, i.e., execute multiple test passes (typically hundreds or thousands per second), each with slightly different inputs generated by the fuzzer.

[0048] A test case is a specific test input and a specific test traversal from the fuzz test. (Correspondingly, (at least) one test case belongs to each test input and one test input belongs to each test case). Typically, test traversals that are of interest for reproducibility (discovery of new code paths or abnormal termination) are stored. In this way, a specific test case with the corresponding input can also be executed on a fuzz target that is not connected to the fuzzer, such as a released version of the program.

[0049] Coverage-guided fuzzing uses code coverage information as feedback during fuzz testing in order to identify whether an input has caused the execution of new code paths or blocks.

[0050] Generator-based fuzz testing (English: generation-based fuzzing) uses prior knowledge about the target program (fuzz target) in order to create test inputs. An example for such prior knowledge is a grammar corresponding to the input specification of the fuzz target, i.e. the input grammar of the fuzz target (i.e. the program to be tested).

[0051] • Mutation-based fuzz testing generates new test inputs by making small changes to the initial test inputs that continue to make the test inputs valid but trigger new behaviors.

[0052] A seed is an initial test input that is used as a starting point for mutation-based fuzz testing. The seed is usually provided by the user.

[0053] The energy of a seed is the number of test cases generated from the seed via mutation.

[0054] Power schedule is the importance that a mutation-based fuzzer assigns to seeds, which directly affects the order in which seeds are placed into the queue for test input generation for mutation.

[0055] A corpus is a collection of test inputs generated during the fuzz testing process. These test inputs are usually derived based on a seed and / or other test inputs that have already been generated (typically through mutations). The corpus grows during the course of fuzz testing as more and more test inputs are generated and added.

[0056] Static instrumentation is the insertion of commands into a program (to be tested) in order to obtain feedback about the execution. The static instrumentation is mostly implemented by a compiler and can, for example, describe the code blocks reached during execution.

[0057] Dynamic instrumentation is controlling the execution of a program (to be tested) during runtime in order to generate feedback from the execution. The dynamic instrumentation is mostly implemented by operating system functionality or by using an emulator.

[0058] A debugger is a device or program that can control a target device or target program and can provide functions such as for recalling register or memory values ​​and for pausing and single-stepping the execution of the target program.

[0059] Next, a black-box fuzzer setup is considered, particularly for testing software for embedded systems, i.e. a software development environment 104 possesses a seed set and implements a black-box fuzzer for testing a computer program 105 for a target device 106, which is an embedded system.

[0060] Black-box fuzz testing is a common testing method when source code is not available. However, most of the fuzz testing research focuses on improving fuzz testing in gray-box or white-box configurations, because black-box configurations are the most difficult to improve. Coverage-oriented fuzzers (such as AFL) typically use static source code instrumentation to obtain feedback about target coverage during processing of test inputs. AFL implements coverage-oriented fuzz testing for ARM-based microcontrollers through the hardware trace interface Embedded Trace Macrocell (ETM). Instead, in the case of closed-source target programs, dynamic instrumentation can be used, that is, the binary to be tested is instrumented during runtime, and the instrumentation is not compiled into the target.

[0061] In the case of embedded systems, static instrumentation for fuzz testing is more difficult to implement compared to non-embedded systems for the following reasons:

[0062] Typically the fuzzer runs on a different computer than the embedded system. Therefore, code coverage information must be transferred.

[0063] Testing the entire system, which is usually composed of multiple components. The software may contain libraries from third-party vendors and software components from other vendors or customers. If these components are provided as binaries, the components may be considered closed-source components, which cannot be modified. Therefore, closed-source components cannot be instrumented by the compiler.

[0064] • Static instrumentation increases code size, which may be critical on embedded systems with constrained resources, i.e. there is not enough memory for instrumentation or to support the additional functionality of fuzz testing.

[0065] Dynamic instrumentation is only possible with certain operating systems, and even in this case, code coverage information must be transmitted (which significantly slows down the testing). Furthermore, the program code may also be located in a read-only memory of the execution system, making changes to the program code impossible.

[0066] Another alternative for plugging is to execute the software of the embedded system in a system simulator (such as QEMU). Here, the transparency of the simulator is used to collect feedback for fuzz testing. Unfortunately, setting up a simulator for a specific target may mean a huge amount of work. The reason for this is that the software for embedded systems usually depends on the availability of external components such as sensors and actuators. If these components are missing in the simulator, the software (executed by the simulator) is very likely to use different approaches and therefore cannot be compared with the real process. Therefore, not only must the command set of the corresponding embedded system be simulated, but all expected hardware peripherals must also be simulated.

[0067] Hardware-in-the-loop (HiL) solutions such as Avatar2 deal with this difficulty by forwarding all IO requests from the emulator to the hardware and transmitting the results back (peripheral proxy). However, HiL is a big bottleneck.

[0068] In view of the above-mentioned difficulties in the case of black-box fuzz testing, according to various embodiments, an oracle for estimating the coverage that can be achieved with the help of test inputs is trained with the help of machine learning and used to guide fuzz testing (i.e., specifically select test inputs fed to the computer program to be tested).

[0069] Thus, according to various embodiments, fuzz testing of software for embedded devices is improved by using an oracle that is able to estimate the code coverage of test inputs (and therefore test cases) on an execution system without actually measuring the code coverage. This enables gray-box fuzz testing (i.e., black-box fuzz testing, but with information about coverage) to be performed on embedded devices without the expense of accurate coverage measurements. For example, a machine learning model implements a software development environment 104 (the machine learning model also trains the software development environment itself if necessary).

[0070] Figure 2 Illustrates the (training) data collection used to train a machine learning model to predict coverage (also referred to herein as a “(coverage) oracle”).

[0071] Training data may be collected by executing test cases on the embedded device 202 and observing code coverage 203. For this purpose, existing test inputs from the corpus 201 (particularly seeds) may be used, or a standard black-box or grey-box fuzzer may be used to generate new test inputs from existing test inputs. For each of the executed test cases, the coverage on the embedded system 202 is observed.

[0072] For example, to call coverage information from embedded system 202 (and thus observe coverage), embedded system 202 (i.e., computer 100 sends test inputs to corresponding embedded system 106 and receives coverage information from embedded system 106) may be used in the HiL structure mentioned above.

[0073] For example, this is done with the aid of a debugger, i.e. the computer 100 (or "host") sends test inputs and debug commands in order to control the embedded system 106, execute the computer program to be tested with the test inputs (via a debug interface) and receive coverage information from the embedded system 106 (e.g. by a debug response via the debug interface). In this case, the communication network 107 therefore includes a debug interface and a channel for the test input (e.g., via WiFi, Bluetooth or a serial connection). Thus, for example, the debugger is connected to a microcontroller of the embedded system 106 and can be controlled by the microcontroller for execution. A simple possibility for determining coverage in such a setting is to traverse the program code of the computer program to be tested in a single step and simultaneously record the program addresses reached. Alternatively, an emulator with a peripheral agent technology can be used for collecting coverage data, or a plugging pile can be used. It should be noted that these techniques for collecting coverage drastically increase the runtime overhead and are therefore not suitable for coverage-oriented fuzz testing. However, in the case of the method provided here, the techniques only appear for collecting training data (or when verifying the predictions of the oracle), and not during actual fuzz testing.

[0074] Figure 3 Explain by Figure 2 The collected training data is used to train the oracle.

[0075] Will be based on Figure 2 The determined training test input 301 and the coverage 302 observed for the training test input (as Ground-Truth, i.e., "reference truth") are used as training data for training an oracle 303, so that a trained oracle 304 is generated, which can predict the code coverage achieved when the computer program to be tested is executed on an embedded target system based on the test case (i.e., test input).

[0076] In practice, oracles 303, 304 may be implemented by a machine learning model (e.g., a neural network) that is trained using test cases as input and coverage information (for each test case) as a target task (i.e., as a benchmark truth value).

[0077] Figure 4 Explain by Figure 3 Oracle-directed fuzz testing with trained oracles.

[0078] In the case of oracle-guided fuzz testing, test inputs are generated by a test input generator 402 based on test inputs from a corpus 401 .

[0079] Each of these test inputs is fed to a trained oracle 403 which predicts the coverage for the test input, ie, is used to obtain a coverage estimate 404 .

[0080] Based on the coverage estimate, the test cases (which correspond to the test inputs) are executed on the target system 405 (i.e., using the test inputs as inputs to the (black box) target program and executing the (black box) target program) to observe errors and abnormal terminations 406. This correlation is Figure 4 This is illustrated by dashed arrow 407 .

[0081] Here, if the coverage estimation means an enlargement of the total coverage achieved so far during the fuzz testing (ie, for example, the oracle 403 predicts that the test case will be used to implement functions that have not been implemented in the previous process of fuzz testing), the test case is executed on the target system 405.

[0082] If a test input increases the (total) coverage (at least according to the oracle), said test input can be added to the corpus 401 (as is common in fuzz testing). For test cases for which the oracle predicts an increase in the (total) coverage, it can also be checked whether this is actually the case when executing the test case. This can be done in one of the above ways (e.g. HiL settings), which is complex as described above, but is only performed in the case of test inputs that increase the total test coverage according to the oracle.

[0083] A common grey-box fuzzer can be used that uses the coverage estimate created by the oracle for each test case for selecting the test cases to be executed, without having to resort to expensive coverage acquisition mechanisms of the embedded device. Thus, the oracle achieves the goal of providing a fast mechanism for determining coverage and transforming from black-box fuzz testing to grey-box fuzz testing.

[0084] Optionally, Figure 2 , 3 The process of 4 can also be repeated multiple times. This means that at a later point in time additional training data can be collected to update the oracle for more accurate coverage predictions.

[0085] In summary, according to various embodiments, there is provided Figure 5 The method shown in .

[0086] Figure 5 A flow chart 500 is shown which represents a method for testing a computer program according to one specific embodiment.

[0087] In 501, a machine learning model is trained to predict the coverage of a computer program for test inputs fed to the machine learning model, respectively, so that if the computer program is executed using the test inputs as inputs, the coverage is achieved (i.e., a corresponding coverage is predicted for each test input fed to the machine learning model).

[0088] At 502, a computer program is tested in a plurality of iterations, wherein in each iteration,

[0089] At 503, generate test inputs (e.g., through mutation of previous test inputs or seeds from a corpus);

[0090] At 504, by means of the trained machine learning model, predicting a coverage of the computer program that would be achieved if the computer program were executed using the generated test inputs as inputs (i.e., the generated test inputs are fed to the (trained) machine learning model, and outputs of the machine learning model in response to the inputs are adopted as estimates of the coverage);

[0091] At 505, determining whether the total coverage achieved so far by testing the computer program (ie, by testing with previous test inputs) is increased by the predicted coverage; and

[0092] In 506, in response to having determined that the total coverage achieved so far by testing the computer program is increased by the predicted coverage, executing the computer program using the generated test inputs (and otherwise not executing, but generating new test inputs; however, it is also possible to intersperse other iterations that do not use the machine learning model into the iterations, i.e., the selection of the test inputs is not made in relation to the predictions of the machine learning model).

[0093] Figure 5 The method can be implemented by one or more computers having one or more data processing units. The term "data processing unit" can be understood as any type of entity that enables processing of data or signals. For example, data or signals can be processed according to at least one (i.e., one or more than one) specific function implemented by a data processing unit. The data processing unit may include an analog circuit, a digital circuit, a logic circuit, a microprocessor, a microcontroller, an integrated circuit of a central unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a programmable gate array (FPGA), or any combination thereof or constructed thereof. Any other means for implementing the corresponding functions described in more detail here can also be understood as a data processing unit or a logic circuit device. One or more of the method steps described in detail here can be performed (e.g., implemented) by a data processing unit through one or more specific functions, and the specific functions are implemented by the data processing unit.

[0094] For example, Figure 5 The control software can be used in a manner similar to that used for testing programs, for example control software for robotic devices. The term "robotic device" can be understood to refer to any technical system, such as a computer-controlled machine, a vehicle, a household appliance, a power tool, a manufacturing machine, a personal assistant or an access control system. The control software can also be used for data processing systems, such as navigation devices.

[0095] For example, by means of a test device (e.g. Figure 1 The computer 100 and the target device 106) are implemented Figure 5 method.

[0096] Although specific embodiments have been shown and described herein, it will be appreciated by those skilled in the art that the specific embodiments shown and described may be replaced with a variety of alternatives and / or equivalent implementations without departing from the scope of protection of the present invention. The present application should encompass any adaptation or variation of the specific embodiments discussed herein. It is therefore intended that the present invention be limited only by the claims and their equivalents.

Claims

1. A method for testing a computer program (105), comprising: training a machine learning model (303) for predicting coverage of the computer program (105) for test inputs (301) fed to the machine learning model, respectively, such coverage being achieved if the computer program (105) is executed using the test inputs as input; The computer program (105) is tested in a plurality of iterations, wherein in each iteration, Generate test input; Predicting, by means of the trained machine learning model (304, 403), a coverage of the computer program (105), the coverage being achieved if the computer program (105) is executed using the generated test inputs as input; determining whether the total coverage achieved so far by testing the computer program (105) is increased by the predicted coverage (404); and In response to having determined that the total coverage achieved to date by testing the computer program (105) is increased by the predicted coverage (404), the computer program (105) is executed using the generated test inputs.

2. A method according to claim 1, wherein in each iteration, the test input is generated from another test input based on whether the coverage (404) already predicted by the trained machine learning model (304, 403) for the other test input is a total coverage, and the total coverage increases the total coverage achieved by the test before the computer program (105) is executed using the other test input as input.

3. The method according to claim 1 or 2 further comprises: checking whether the total coverage achieved when testing the computer program (105) has been increased by executing the computer program (105) using the generated test input, and in response to the total coverage achieved when testing the computer program (105) having been increased by executing the computer program (105) using the generated test input, adding the generated test input to the set of following test inputs, and generating test inputs for further iterations from the test inputs.

4. The method according to any one of claims 1 to 3, comprising: Generate test input on the test system, determining coverage (404) of the computer program (105) on the test system, and In response to the test system having determined that total coverage achieved to date by testing the computer program (105) is increased by the predicted coverage (404), the computer program (105) is executed on a system (106, 405) in which the test system is embedded using the generated test inputs.

5. A software testing system (100) configured to carry out the method according to any one of claims 1 to 4. 6 . A computer program comprising instructions which, when executed by a processor, cause the processor to carry out the method according to claim 1 .

7. A computer readable medium storing instructions which, when executed by a processor, cause the processor to implement the method according to any one of claims 1 to 4.

Images