Method for testing computer program

Abstract

According to various embodiments, a method for testing a computer program is described, the method comprising performing a plurality of test iterations, where for each test iteration, a reference test input and a variation for the reference test input are selected according to a selection policy (201), and generating a test input for the computer program by applying the selected variation to the selected reference test input, where for each test iteration, a test traversal of the computer program is performed with the corresponding generated test input for the computer program, wherein a test coverage increase by test traversal is determined for each test iteration (209), and wherein a selection policy is trained by means of reinforcement learning (201) wherein the test coverage increase (209) is used as a reward (204).

Description

Technical Field

[0001] The present disclosure relates to a method for testing a computer program. Background Art

[0002] An important part of developing software applications is testing. In particular, errors that cause application failures should be identified and corrected. An example of software testing is the dynamic software testing method fuzz testing.

[0003] Mutation-based fuzzers (such as AFL, Honggfuzz or Libfuzzer) use mutation-based, coverage-controlled fuzz testing to test software with low cost. For each fuzz testing iteration (i.e. each test pass), a "seed" test input (also referred to as reference test input in the following) is selected from a so-called (input) corpus, which is randomly mutated and sent as input to the computer program to be tested. If a newly generated input in this way triggers previously unseen behavior (such as execution in a code path that has not been executed so far or a crash), the input is added to the corpus. In this way, the input space of a computer program can be explored with little or no knowledge of the input format, based only on the initially available reference test inputs.

[0004] In the case of the "Havoc" mutation strategy, for each fuzz testing iteration, the mutations applied to the corresponding reference test input are randomly composed of basic mutation sequences (such as inverting individual bits for different bit cells). Good results can be obtained using the Havoc mutation strategy. However, a way is desired to further improve the efficiency and / or effectiveness of testing of computer programs, especially by means of fuzz testing. Summary of the invention

[0005] According to various embodiments, a method for testing a computer program is provided, the method comprising performing a plurality of test iterations, wherein for each test iteration, reference test inputs and mutations for the reference test inputs are selected according to a selection strategy, and test inputs for the computer program are generated by applying the selected mutations to the selected reference test inputs, wherein for each test iteration, a test run of the computer program is performed using the corresponding generated test inputs for the computer program, wherein for each test iteration, an increase in test coverage achieved by the test run is determined, and wherein the selection strategy is trained by means of reinforcement learning, wherein, for example, an increase in test coverage is used as a reward. The coverage (or its increase) can be evaluated according to various metrics for detecting test coverage, such as code coverage (branch coverage, line coverage, number of triggered program crashes (i.e., coverage in terms of processing paths leading to crashes)), memory coverage, etc.

[0006] The above method can achieve higher efficiency and / or effectiveness of coverage-controlled fuzz testing compared to randomly selecting a reference test input (also called a "seed") to mutate for the current fuzz testing iteration and randomly selecting mutations (e.g., in the form of a basic mutation sequence).

[0007] Any RL method can be used for reinforcement learning (RL), which is based on, for example, the optimization of an objective function that includes the efficiency and / or effectiveness of fuzz testing (e.g., in the form of evaluating the achieved coverage). Various machine learning models can be used for reinforcement learning, for example, the selection policy can be represented by a neural network trained during reinforcement learning. The elements that implement the selection policy can be considered as RL agents (i.e., the training or learning of the selection policy can also be considered as the training of the RL agent).

[0008] The training of the selection strategy can take place individually for each of the plurality of computer programs to be tested. For example, the following basic selection strategy is respectively assumed, which is trained (and thus improves its suitability for testing the corresponding computer program) during the fuzz testing of the corresponding computer program.

[0009] It should be noted that a variation of a reference test input may also consist in combining the reference test input with other reference test inputs. The selection of other reference test inputs may then be considered as part of the variation selection.

[0010] Various embodiments are described below.

[0011] Embodiment 1 is a method for testing a computer program as described above.

[0012] Embodiment 2 is a method according to embodiment 1, wherein for at least some testing iterations, selecting variants includes selecting base variant sequences.

[0013] Thus, the agent can generate more complex mutations by arranging together various basic mutations (from a pre-given set of, for example, "simple" mutations).

[0014] Embodiment 3 is a method according to embodiment 1, wherein for at least some test iterations, selecting mutations includes selecting a base mutation sequence that is a continuation of a base mutation sequence used to generate test inputs for a previous test iteration.

[0015] Thus, the agent may, for example, continually continue to mutate reference test inputs in successive testing iterations (eg, fuzz testing iterations) (until the agent, for example, decides to transition to other reference test inputs).

[0016] Embodiment 4 is the method described in embodiment 2 or 3, wherein the basic variation is one or more of the following: inverting a single bit for a specific bit position, setting a data element at a specific position to a predetermined value, adding a predetermined number to / subtracting a predetermined number from a data element at a specific position, setting a data element at a specific position to a random value, removing a data element at a specific position, and copying a data element at a specific position.

[0017] These basic variations allow the agent to achieve a high degree of flexibility in choosing variations.

[0018] Embodiment 5 is a method according to any one of embodiments 1 to 4, wherein for each test iteration, reference test inputs and mutations are selected according to a selection strategy based on test inputs of a previous test iteration.

[0019] In other words, the state entering the RL agent contains the test inputs of the previous (e.g., last) test iteration. Thus, the RL agent (or selection policy) learns, among other things, which test inputs are worth pursuing further (e.g., which test inputs should be further mutated).

[0020] Embodiment 6 is a method according to any one of embodiments 1 to 5, wherein the computer program is a control program for a robotic device, and the robotic device is controlled using the computer program based on a result of a test of the computer program.

[0021] Embodiment 7 is a testing device configured to perform the method according to any one of embodiments 1 to 6.

[0022] Embodiment 8 is a computer program having instructions, which, when executed by a processor, cause the processor to perform the method according to any one of embodiments 1 to 6.

[0023] Embodiment 9 is a computer-readable medium storing instructions, and when the instructions are executed by a processor, the instructions cause the processor to perform the method according to any one of embodiments 1 to 6. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In the accompanying drawings, like reference numerals generally refer to the same parts in the various views. The accompanying drawings are not necessarily drawn to scale, with emphasis instead generally being placed on illustrating the principles of the invention. In the following description, various aspects are described with reference to the following drawings.

[0025] Figure 1 A computer used to develop and / or test software applications is shown.

[0026] Figure 2 A selection strategy for fuzz testing trained according to one embodiment is described.

[0027] Figure 3 A flow chart is shown which illustrates a method for testing a computer program according to one embodiment. DETAILED DESCRIPTION

[0028] The following detailed description relates to the attached drawings, which illustrate specific details and aspects of the present disclosure that can perform the present invention for the purpose of illustration. Other aspects may be used and structural, logical and electrical changes may be made without departing from the scope of protection of the present invention. Because some aspects of the present disclosure may be combined with one or more other aspects of the present disclosure to form new aspects, the various aspects of the present disclosure are not necessarily mutually exclusive.

[0029] Various examples are described in more detail below.

[0030] Figure 1 A computer 100 is shown for use in developing and / or testing software applications.

[0031] The computer 100 has a CPU (Central Processing Unit) 101 and a work memory (RAM) 102. The work memory 102 is used to load a program code from a hard disk 103, for example, and the CPU 101 executes the program code.

[0032] In the present example it is assumed that a user intends to utilize computer 100 to develop and / or test a software application.

[0033] To this end, the user executes the software development environment 104 on the CPU 101 .

[0034] The software development environment 104 enables users to develop and test applications (i.e., software) 105 for various devices 106, i.e., for target hardware (such as embedded systems for controlling robotic devices including mechanical arms and autonomous vehicles) or for mobile (communication) devices. To this end, the CPU 101 can execute an emulator as part of the software development environment 104 to simulate the behavior of the following corresponding devices 106 for which the application will be or has been developed. If the software development environment 104 is only used to test software from other sources, the software development environment 104 can also be regarded as or designed as a software testing environment.

[0035] The user can distribute the finished application to the corresponding device 106 via the communication network 107. Instead of via the communication network 107, this can also be done in other ways, for example by means of a USB stick.

[0036] Before this occurs, however, the user should test the application 105 to avoid distributing an application to the device 106 that does not function as specified.

[0037] One testing method is so-called fuzz testing. Fuzz testing (Fuzzing or Fuzz-Testing) is an automated software testing method in which invalid, unexpected or random data is fed as input to the computer program to be tested. The program is then monitored for anomalies such as crashes, lack of failed built-in code assertions or potential memory leaks.

[0038] Typically, a fuzzer (i.e., a test program using fuzz testing) is used to test a program that processes structured inputs. The structure is specified, for example, in a file format or file format or protocol and a distinction is made between valid and invalid inputs. An effective fuzzer produces semi-valid inputs that are "valid enough" to not be directly rejected by the input parser of the program under test, but "invalid enough" to reveal unexpected behavior and edge cases that are not properly handled in the program under test.

[0039] The following describes the terminology used in conjunction with fuzz testing:

[0040] Fuzz testing (Fuzzing or Fuzz-Testing) is an automated testing process that sends randomly generated inputs to a target program (the program under test) and observes its responses.

[0041] A fuzzer or fuzz testing engine is a program that automatically generates input. Therefore, the fuzzer or fuzz testing engine is not associated with the software to be tested (e.g., through instrumentation). However, the fuzzer or fuzz testing engine has the ability to instrument code, generate test cases, and execute the program to be tested. Known examples are afl and libfuzzer.

[0042] A fuzz target is a software program or function that should be tested by fuzz testing. The main feature of a fuzz target should be that it takes potentially untrusted inputs generated by the fuzzer during the fuzz testing process.

[0043] A fuzz test is a combined version of a fuzzer and a fuzz target. Thus, the fuzz target can be instrumented code, where the fuzzer is associated with its input (i.e., provides the input). Fuzz tests are executable. The fuzzer can also start, observe, and stop multiple fuzz tests, i.e., execute multiple test passes (typically hundreds or thousands per second), each with slightly different inputs generated by the fuzzer.

[0044] A test case is a specific (test) input and a specific test run from a fuzz test. Typically, runs that are of interest for reproducibility (finding new code paths or crashes) are stored. In this way, a specific test case with the corresponding input can also be executed on a fuzz target that is not connected to the fuzzer, such as a released version of the program.

[0045] Coverage-guided fuzzing uses code coverage information as feedback during fuzz testing to identify whether an input causes the execution of new code paths or blocks.

[0046] Generator-based fuzz testing (English: generation-based fuzzing) uses prior knowledge about the target program (fuzz target) to create test inputs. An example of such prior knowledge is the grammar corresponding to the input specification of the fuzz target, i.e. the input grammar of the fuzz target (i.e. the program to be tested).

[0047] Static instrumentation is the insertion of commands into a program (to be tested) to obtain feedback about the execution. The static instrumentation is mostly implemented by the compiler and can, for example, describe the code blocks reached during execution.

[0048] Dynamic instrumentation is controlling the execution of a program (to be tested) during runtime to generate feedback from the execution. The dynamic instrumentation is mostly implemented by operating system functionality or by using an emulator.

[0049] Test inputs for a test case (ie, test passes or test iterations) may be generated from test inputs of previous test cases by mutation, ie, a mutator generates current test inputs from previous test cases from a set of reference test cases, referred to as a corpus.

[0050] For example, a coverage-controlled fuzzer may include a random search mechanism called Havoc. Instead of using only one "basic" mutator (i.e., a mutator that performs a basic mutation, such as flipping (i.e., inverting) a bit) to generate test inputs for fuzz testing iterations, Havoc randomly selects a plurality of such basic mutators, each of which performs a basic mutation, i.e., flipping (i.e., inverting) a single bit for different bit units, and sequentially applies the basic mutations to test cases (from a corpus), i.e., selecting a sequence of basic mutations that constitutes the mutations of the current fuzz testing operation.

[0051] Decisive for the efficiency and / or effectiveness of fuzz testing, i.e. for the achieved coverage of the test program, in particular after a certain number of fuzz test iterations (i.e. test iterations), is which mutations are applied for each fuzz test iteration (i.e. for example which and how many basic mutations are applied to the test input of the test case to produce the current test input) and to which reference test case the mutations are applied. The selection of mutations and reference test cases can be carried out randomly. It can also be provided that a reference test case is selected randomly and a fixed number of basic mutations (which in turn can be selected randomly) are applied to the test input of the reference test case. The selection of reference test cases - determining how many mutations should be applied to the reference test case respectively - is also called "power scheduling".

[0052] According to various embodiments, the selection of reference test cases to be mutated for the current fuzz testing iteration (i.e., "power scheduling") and which mutations are applied to the selected reference test cases (i.e., which basic mutations, their number and / or their order) are learned by the agent with the help of reinforcement learning (RL) (or synonymously: training the selection strategy implemented (or pursued) by the agent). The agent is trained here to optimize the efficiency and / or effectiveness of fuzz testing.

[0053] Examples of basic mutations are as follows:

[0054] - Single bit flip, that is, flip (invert) a single bit at a specific bit position of the test input

[0055] - Setting "interesting" (i.e. pre-given) blocks in the test input, such as bytes, words or double words (in big endian or little endian). What is "interesting" here can be based on expert knowledge or on the results of fuzz testing to date.

[0056] - add or subtract a small integer to or from a specific byte, word, or doubleword (in big or little endian) in the test input,

[0057] -Set random bytes at specific parts of the test input

[0058] -Remove blocks (e.g. bytes, words, or double words) at specific positions from the test input

[0059] -Copy a block (e.g., byte, word, or doubleword) at a specific location in the test input by either overwriting a subsequent block or inserting (i.e., moving a subsequent block).

[0060] For example, a block can be set via block memset.

[0061] By linking (ie sequencing) elementary variations, variations and thus test inputs can be generated which may not result from a single elementary variation.

[0062] With each basic variation of the application, the mutated test input deviates increasingly from the reference test input from which it was generated. This is illustrated by Table 1 for simple (short) test inputs.

[0063]

[0064]

[0065] Table 1

[0066] The efficiency and / or effectiveness of fuzz testing may involve different fuzzer evaluation metrics, such as different measures of code coverage or the number of program bugs found.

[0067] For reinforcement learning with respect to this optimization goal (efficiency and / or effectiveness), representations of rewards and subsequent states are derived from the actions selected and performed by the agent in the "environment", which are again fed to the RL agent for selecting the next action.

[0068] Figure 2 Training a selection strategy for fuzz testing according to one embodiment is shown.

[0069] The selection strategy is implemented by an agent 201. For each fuzzing iteration, the agent 201 selects an action 202 for the fuzzer 203 and obtains a reward 204 and a subsequent state 205 for doing so. The agent's goal is to maximize the reward for the actions performed by it. Here, the agent may follow an RL algorithm that achieves a tradeoff between exploration (discovering new parts of the input space about which the agent has no or little information) and making full use of the knowledge that the agent has acquired so far (prioritization and mutation of test cases that worked best in previous fuzzing iterations).

[0070] For example, for the current iteration of fuzz testing, the elements of reinforcement learning are as follows:

[0071] -The state of the environment (or the control system to be controlled) is expressed as an encoding of a reference test case selected for the last fuzz testing iteration, i.e., a reference test input from the corpus 208 that has been mutated to produce test inputs for the last fuzz testing iteration, and as a byte array representing the test input of the last fuzz testing iteration (i.e., the mutated selected reference test input). It should be noted that when using deep reinforcement learning (Deep RL, such as deep Double-Q learning), the explicit modeling of the state space can be abandoned.

[0072] Each action A comprises the selection of reference test inputs which are mutated for generating test inputs 206 for the computer program 207 for the current fuzz testing iteration, and this is done by means of mutations. The selection of mutations may here comprise the selection of a basic mutation sequence (see the example above). Thus, at each fuzz testing iteration, the agent 201 is faced with the decision whether the same reference test input as at the last fuzz testing iteration should be changed (i.e., e.g., further mutated) or whether a different reference test input should be mutated.

[0073] - Reward function R a (s, s′), a∈A, s, s′∈S are designed so that the agent 201 is rewarded if the selected action increases the total code coverage achieved so far in the test, where s is the state in which action a is performed and s′ is the subsequent state achieved. Different metrics can be used for code coverage in this case (e.g., line coverage, path coverage, edge coverage). In this case, the test can start with a test traversal for an initial set of reference test inputs (using the reference test inputs to initialize the corpus 208), so that a certain coverage is achieved thereafter, which can then be gradually increased through the actions of the agent 201.

[0074] - As RL algorithms, any standard RL learning method that can consider discrete action and state spaces can be used, such as various versions of Q-learning, SARSA (State Action Reward State Action), deep reinforcement learning, etc.

[0075] Therefore, the fuzz testing of the computer program 207 is performed, for example, as follows:

[0076] 1) Initialize the fuzzer 203 , that is, the observations of the fuzzer 203 are empty or set to the results of the test traversal for the reference test input (seed) initially contained in the corpus 208 .

[0077] 2) At each fuzz testing iteration, the RL agent 201 selects which reference test inputs from the corpus should be mutated and which mutations should be applied here to generate test inputs for the current fuzz testing iteration. As mentioned above, this includes the possibility of applying additional mutations to the already modified reference test inputs from the previous fuzz testing iteration.

[0078] 3) Perform a test pass using the test inputs generated for the current fuzz testing iteration and compute a reward 209 for this (e.g. observe and evaluate the achieved coverage (or its increase by the current fuzz testing iteration). The agent gets its reward based on this observation. When the coverage has been increased, the test input is added to the fuzzer's corpus 208 as a reference test input.

[0079] 4) Repeat steps 2) and 3).

[0080] 5) The testing process stops when, for example, the user ends this or certain termination criteria are met (eg, a certain time has passed or a certain coverage has been reached).

[0081] In summary, according to various embodiments, there is provided Figure 3 The method shown in .

[0082] Figure 3 A flow chart 300 is shown that represents a method for testing a computer program according to one embodiment.

[0083] Perform multiple executions of multiple test iterations, where

[0084] - in 301, for each test iteration, selecting a reference test input and a mutation for the reference test input according to a selection strategy, and generating a test input for the computer program by applying the selected mutation to the selected reference test input;

[0085] - at 302, for each test iteration, performing a test pass of the computer program using the corresponding generated test input for the computer program (ie executing the computer program using the corresponding generated test input);

[0086] - At 303, for each test iteration, the test coverage increase achieved by the test pass is determined

[0087] - In 304, the selection strategy is trained by means of reinforcement learning, wherein test coverage increase is used as a reward.

[0088] The plurality of test passes need not include all test passes executed for testing the computer program, for example, test passes may also be additionally executed for the reference test input itself. Furthermore, it should be noted that a training step need not be performed for each test iteration, but rather multiple test iterations may be combined in one training step (i.e., training may be performed based on batches of test iterations), even in Figure 3 This is shown.

[0089] Figure 3The method can be performed by one or more computers having one or more data processing units. The term "data processing unit" can be understood as any type of entity that enables processing of data or signals. For example, data or signals can be processed according to at least one (i.e., one or more) specific functions performed by the data processing unit. The data processing unit may include an analog circuit, a digital circuit, a logic circuit, a microprocessor, a microcontroller, a central unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a programmable gate array integrated circuit (FPGA), or any combination thereof, or may be composed of an analog circuit, a digital circuit, a logic circuit, a microprocessor, a microcontroller, a central unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a programmable gate array integrated circuit (FPGA), or any combination thereof. Any other means for implementing the corresponding functions described in more detail herein may also be understood as a data processing unit or a logic circuit device. One or more of the method steps described in detail herein may be performed (e.g., implemented) by a data processing unit through one or more specific functions performed by the data processing unit.

[0090] According to various implementations, the method is in particular computer-implemented.

Claims

1. A method for testing a computer program (105), the method comprising: Perform multiple test iterations, wherein for each test iteration, a reference test input and a variation for the reference test input are selected according to a selection strategy (201), and a test input for the computer program (105) is generated by applying the selected variation to the selected reference test input; wherein for each test iteration, a test pass of the computer program (105) is performed using the corresponding generated test input for the computer program (105); wherein for each test iteration, an increase in test coverage achieved by the test pass is determined (209); as well as The selection strategy (201) is trained by means of reinforcement learning, wherein the test coverage increase (209) is used as a reward (204).

2. The method of claim 1 , wherein for at least some of the testing iterations, selecting a variation comprises selecting a base variation sequence.

3. The method of claim 1 , wherein for at least some of the test iterations, selecting mutations comprises selecting a base mutation sequence that is a continuation of a base mutation sequence used to generate test inputs for a previous test iteration.

4. The method according to claim 2 or 3, wherein the basic mutation is one or more of the following: inverting a single bit for a specific bit position, setting a data element at a specific position to a predetermined value, adding a predetermined number to / subtracting a predetermined number from a data element at a specific position, setting a data element at a specific position to a random value, removing a data element at a specific position, and duplicating a data element at a specific position.

5. The method according to any one of claims 1 to 4, wherein for each test iteration, the reference test inputs and the mutations are selected according to the selection strategy (201) based on the test inputs of the previous test iteration.

6. The method according to any one of claims 1 to 5, wherein the computer program (105) is a control program for a robotic device, and the robotic device is controlled by means of the computer program (105) based on the result of the test of the computer program (105).

7. A testing device (100) configured to carry out the method according to any one of claims 1 to 6.

8. A computer program having instructions which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 6.

9. A computer-readable medium storing instructions, which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 6.

Images