Method for testing computer program

Abstract

According to various embodiments, a method for testing a computer program is described, comprising: setting one or more breakpoints onto one or more arithmetic or displacement operations in the computer program; executing a computer program; when one of the set breakpoints is triggered, determining whether an overflow flag is set by performing a corresponding arithmetic or displacement operation; and in response to determining that the overflow flag has been set by the corresponding arithmetic or displacement operation, triggering a display that the computer program has an error.

Description

Technical Field

[0001] The present disclosure relates to a method for testing a computer program. Background Art

[0002] An important part of developing software applications is testing and the corresponding error clearing when errors are found. In particular, errors that lead to application failures should be identified and corrected. An important aspect here is testing for undefined behavior, such as by so-called (memory) sanitizers. Compiling and testing software on common desktop and server hardware, such as x86, with the help of various sanitizers is a measure that can discover errors such as the Heartbleed bug that previously remained undiscovered for a long time.

[0003] Comprehensive testing is particularly important for computer programs on often safety-relevant embedded systems, such as control devices for vehicles. However, sanitizers for desktop and server hardware cannot be used or can only be used poorly for such systems, since embedded systems typically have restricted resources and such sanitizers have a huge resource requirement and therefore cannot be used or can even affect the execution of the computer program so that errors occur in the first place or remain undetected. Summary of the invention

[0004] Therefore, what are desired are methods for testing computer programs that allow testing for undefined behavior and are applicable to embedded systems.

[0005] According to various embodiments, a method for (automatically) testing a computer program is provided, comprising: setting one or more breakpoints in the computer program to one or more arithmetic or bit shift operations; executing the computer program; when one of the set breakpoints is triggered, determining whether an overflow flag is set by performing the corresponding arithmetic or bit shift operation; and in response to determining that the overflow flag is set by the corresponding arithmetic or bit shift operation, triggering a display that the computer program has an error.

[0006] This may be done for each or at least a number of arithmetic or bit shift operations occurring in the computer program (eg, depending on how many breakpoints are available).

[0007] The above method allows testing with overflow detection of arithmetic or bit shift operations on embedded systems with the aid of a debugger. This is particularly suitable when testing with fuzz testing, because fuzz testing can also be implemented in a debugger-controlled manner and can be effectively used on embedded systems in this way.

[0008] Sanitizers can be implemented with the aid of code instrumentation. However, for this purpose either the source code must be available or instruction set-specific instrumentation based on binaries (binary instrumentation) is required, which is very susceptible. The alternative emulator-based instrumentation is also very platform-specific, and each embedded platform requires its own emulator. The method described above enables testing with a debugger-controlled sanitizer without the need for instrumentation or emulation, and can therefore be applied in many cases.

[0009] Various embodiments are described below.

[0010] Embodiment 1 is the above-mentioned method for testing a computer program.

[0011] Embodiment 2 is a method according to embodiment 1, wherein determining whether an overflow flag is set by performing a corresponding arithmetic or bit shift operation comprises: traversing (assembler) instructions implementing the arithmetic or bit shift operation in individual steps.

[0012] In other words, the sub-operations that implement the arithmetic or bit shift operation are "stepped through" until a sub-operation that could result in setting the overflow flag has been performed.

[0013] Embodiment 3 is a method according to embodiment 1 or 2, comprising: in response to determining that an overflow flag has been set by a corresponding arithmetic or bit shift operation, triggering a display that the computer program has an error by triggering an abort (ie, a crash) of the computer program.

[0014] Thus, for example, within the scope of fuzz testing, a computer program can be marked as having an error, because its execution causes the computer program to terminate (i.e. crash), after which the fuzzer displays an error. For example, a warning can also be output on the stderr stream, or a debugger can be monitored on the host side (i.e. on a test system that tests the computer program on the execution system with the aid of a debugger).

[0015] Embodiment 4 is a method according to any one of embodiments 1 to 3, including: executing multiple test traversals (e.g., fuzzy test-test traversals, i.e., fuzzy test iterations), and setting breakpoints to different arithmetic or displacement operations for different test traversals.

[0016] Thereby, a large number of arithmetic operations and / or bit shift operations can be covered.

[0017] Embodiment 5 is a method according to any one of embodiments 1 to 4, comprising: executing a computer program (105) on an embedded system (106), and performing the following steps through a test system (100) connected to the embedded system (106) (via a debugging interface): setting breakpoints, determining whether an overflow flag is set, and triggering the display when necessary: ​​the computer program has an error.

[0018] According to various embodiments, in particular, a test of a computer program for an embedded system, including memory monitoring, can be performed on the embedded system itself.

[0019] Embodiment 6 is a method according to any one of embodiments 1 to 5, wherein the computer program (105) is a control program for a robotic device, and the computer program (105) is used to control the robotic device based on the results of a test of the computer program (105).

[0020] Embodiment 7 is a testing device, which is designed to perform the method according to any one of embodiments 1 to 6.

[0021] Embodiment 8 is a computer program having instructions, which, when executed by a processor, cause the processor to perform the method according to any one of embodiments 1 to 6.

[0022] Embodiment 9 is a computer-readable medium storing instructions, which, when executed by a processor, causes the processor to perform the method according to any one of embodiments 1 to 6. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In the accompanying drawings, similar reference numerals generally refer to the same parts throughout the various views. The accompanying drawings are not necessarily drawn to scale, with emphasis instead generally being placed on illustrating the principles of the invention. In the following description, various aspects are described with reference to the following drawings.

[0024] Figure 1 A computer used to develop and / or test software applications is shown.

[0025] Figure 2 A flow chart is shown which illustrates a method for testing a computer program according to one embodiment. DETAILED DESCRIPTION

[0026] The following detailed description relates to the attached drawings, which illustrate specific details and aspects of the present disclosure that can perform the present invention for the purpose of illustration. Other aspects may be used and structural, logical and electrical changes may be made without departing from the scope of protection of the present invention. Various aspects of the present disclosure are not necessarily mutually exclusive, because some aspects of the present disclosure may be combined with one or more other aspects of the present disclosure to form new aspects.

[0027] Various examples are described in more detail below.

[0028] Figure 1 A computer 100 is shown for use in developing and / or testing software applications.

[0029] The computer 100 has a CPU (Central Processing Unit) 101 and a work memory (RAM) 102. The work memory 102 is used to load a program code from a hard disk 103, for example, and the CPU 101 executes the program code.

[0030] In the present example it is assumed that a user intends to utilize computer 100 to develop and / or test a software application.

[0031] To this end, the user executes the software development environment 104 on the CPU 101 .

[0032] The software development environment 104 enables users to develop and test applications 105 for various devices 106, i.e., for target hardware (such as embedded systems for controlling robotic devices including robot arms and autonomous vehicles) or for mobile (communication) devices. To this end, the CPU 101 can execute an emulator as part of the software development environment 104 to simulate the behavior of the corresponding device 106 for which the application is to be or has been developed. If the software development environment 104 is only used to test software from other sources, the software development environment 104 can also be regarded as or designed as a software testing environment.

[0033] The user can distribute the finished application to the corresponding device 106 via the communication network 107. Instead of via the communication network 107, this can also be done in other ways, for example by means of a USB stick.

[0034] Before this occurs, however, the user should test the application 105 to avoid distributing an application to the device 106 that does not function as specified.

[0035] One testing method is so-called fuzz testing. Fuzz testing (Fuzzing or Fuzz-Testing) is an automated software testing method in which invalid, unexpected or random data is fed as input to the computer program to be tested. The program is then monitored for anomalies such as crashes, lack of failed built-in code assertions or potential memory leaks.

[0036] Typically, fuzzers (i.e., test programs using fuzz testing) are used to test programs that process structured inputs. The structure is specified, for example, in a file format or file format or protocol and distinguishes between valid and invalid inputs. An effective fuzzer produces semi-valid inputs that are "valid enough" to not be directly rejected by the input parser of the program under test, but "invalid enough" to reveal unexpected behavior and edge cases that are not properly handled in the program under test.

[0037] The following describes the terminology used in conjunction with fuzz testing:

[0038] Fuzz testing (Fuzzing or Fuzz-Testing) is an automated testing process that sends randomly generated inputs to a target program (the program under test) and observes its responses.

[0039] A fuzzer or fuzz testing engine is a program that automatically generates input. Therefore, the fuzzer or fuzz testing engine is not associated with the software under test and does not perform instrumentation. However, the fuzzer or fuzz testing engine has the ability to instrument code, generate test cases and execute the program under test. Known examples are afl and libfuzzer.

[0040] A fuzz target is a software program or function that should be tested by fuzz testing. The main feature of a fuzz target should be that it takes potentially untrusted inputs generated by the fuzzer during the fuzz testing process.

[0041] A fuzz test is a combined version of a fuzzer and a fuzz target. Thus, a fuzz target can be instrumented code, where the fuzzer is associated with its input (i.e., provides the input). Fuzz tests are executable. A fuzzer can also start, observe, and stop multiple fuzz tests (typically hundreds or thousands per second), each with slightly different inputs generated by the fuzzer.

[0042] A test case is a specific input and a specific test run from a fuzz test. Usually, the runs that are of interest for reproducibility (finding new code paths or crashes) are stored. In this way, a specific test case with the corresponding input can also be executed on a fuzz target that is not connected to the fuzzer, such as a released version of the program.

[0043] Coverage-guided fuzzing uses code coverage information as feedback during fuzz testing to identify whether an input causes the execution of new code paths or blocks.

[0044] Generator-based fuzz testing (English: generation-based fuzzing) uses prior knowledge about the target program (fuzz target) to create test inputs. An example of such prior knowledge is the grammar corresponding to the input specification of the fuzz target, i.e. the input grammar of the fuzz target (i.e. the program to be tested).

[0045] Static instrumentation is the insertion of commands into a program (to be tested) to obtain feedback about the execution. The static instrumentation is mostly implemented by the compiler and can, for example, describe the code blocks reached during execution.

[0046] Dynamic instrumentation is controlling the execution of a program (to be tested) during runtime to generate feedback from the execution. The dynamic instrumentation is mostly implemented by operating system functionality or by using an emulator.

[0047] A debugger is a device or a program that can control a target device or a target program and can provide functions such as for calling register values ​​or memory values ​​and for pausing and executing a target program in individual steps.

[0048] ·Breakpoints are set by the debugger on commands of the target program or device so that execution stops when they are reached and the controlling process is notified of this.

[0049] Setting a (data) watchpoint by the debugger at a memory address of the target program or target device, so that execution is halted when the memory address is accessed, and the controlling process is informed of this by triggering an interrupt.

[0050] Embedded systems usually have a microcontroller that processes inputs and responds with outputs to perform specific tasks. Although microcontrollers use the same memory model as conventional user programs and are programmed using the same programming language, microcontroller programs are significantly more difficult to test. In order to enable debugging, microcontrollers usually provide the following possibilities: interrupting the program using breakpoints (stop points), traversing the commands of the program in individual steps, and setting watchpoints on storage addresses. When accessing the corresponding storage area, the watchpoint triggers an interrupt. Hardware breakpoints and watchpoints are typically implemented as physical registers in the debugging unit of the microcontroller, and therefore their number is limited and depends on the corresponding system. For example, for a typical microcontroller, the maximum number is four breakpoints and two data watchpoints. Usually watchpoints can be distinguished between read access and write access.

[0051] In particular, breakpoints and watchpoints can be used to implement debugger-controlled fuzz testing, eliminating the need for instrumentation.

[0052] Fuzz testing (and debugger-controlled fuzz testing) is very effective at finding bugs that trigger observable behavior, such as crashes or reboots. However, not all classes of bugs can be observed because the program silently fails on them. An example is the Heartbleed vulnerability. The core of the Heartbleed vulnerability is that Heartbleed only reads out of bounds of an array, while writes result in segmentation faults that are easily observable.

[0053] The Heartbleed vulnerability was only found with the help of Address Sanitizer (ASan). ASan inserts additional commands, metadata, and checks during the compilation of a program to prevent errors when memory is corrupted. When such Sanitizer commands are available in a program, more errors can be found when debugging the program than without the Sanitizer. Automated testing (such as fuzz testing) is particularly interesting when a Sanitizer is set up in the program to be tested (i.e. in the fuzz target) to find additional errors.

[0054] For embedded systems (such as data processing devices with ARM architecture), such sanitizers are not as easy to use as for standard platforms (such as x86 platforms). There are many reasons for this:

[0055] Embedded systems are too resource-constrained to implement a Sanitizer. For example, Asan requires double the memory, MSan (MemorySanitizer) requires 2.5 times the resources, and UBSan (UndefinedBehaviorSanitizer) even requires three times the working memory of the program.

[0056] The Sanitizer increases the size of the compiled binary. In the automotive industry, the size of such binaries often almost corresponds to the available flash memory of the target hardware. Therefore, the additional instrumentation of the Sanitizer does not fit in the flash memory.

[0057] The use of Sanitizers results in slower runtime of binaries on the corresponding hardware due to the Sanitizer's additional instrumentation and metadata collection and tracking. Embedded systems strongly depend on asynchronous events, such as interrupts, and thus Sanitizers may cause timing-based false positive errors, i.e., Sanitizers may introduce new errors during runtime.

[0058] Embedded systems usually do not have a user interface for displaying runtime errors. On x86 systems, for example, segmentation faults are forwarded to STDERR so that the user sees the crash. Instead, embedded systems fail silently, i.e. without the user noticing this, and restart after such a crash.

[0059] Therefore, according to various embodiments, a method is provided that enables memory monitoring (i.e., Sanitizer functionality) to be used for embedded systems, in particular, enables memory monitoring to be used for debugger-controlled fuzz testing. Here, the memory monitoring itself can be implemented with the aid of a debugger (or a debugger for fuzz testing) (the debugger does not need to be running on the execution system).

[0060] In debugger-based fuzz testing, the interaction between the system executing the test (and corresponding to, for example, the computer 100) and the target system (target hardware, such as an embedded system, such as the target device 106) is carried out via a debug connection (i.e., a debug interface), which is provided, for example, by a dedicated debugger hardware device. The test input data is transmitted to the target system 106 in the form of an input vector, for example, via WiFi or a CAN bus (depending on the type of the target device 106), that is, the communication network 107 is such a debug connection during this test (then when distributing the tested software, the communication network can be any other communication network). The system executing the test (hereinafter also referred to as the test system 100) controls the execution of the target program (i.e., the program to be tested) in the target system via the debug connection, that is, starts the execution and resumes the execution after an interrupt (especially an interrupt triggered by a data monitoring point).

[0061] The debugger-controlled Sanitizer does not require plugging or emulation, but only requires a debug interface with the target system (e.g., an embedded system on which the software is executed) with the possibility of setting breakpoints and monitoring points. This type of debug interface and debugging capabilities are universal and also available, which leads to the wide and simple applicability of the method described below. In addition, since most or all of the information related to the Sanitizer is collected and stored on the host side of the debugger (i.e., in the test system 100), the embedded system can also be tested with its completed version (e.g., the sold version), so the memory of the target system is only slightly loaded, for example, for metadata. The size of the compiled binary file of the target program does not increase, because the binary file is used during testing as if it were set for the target system 106.

[0062] When a breakpoint is reached, the debugger halts the target system. Therefore, the approach described below leads to time-based false alarms only in rare cases. These false alarms can also be eliminated by other testing techniques, such as by subsequently verifying the errors found on the target system. Using a debugger also provides a good view of the internal affairs of the target system.

[0063] The methods described below are used to detect undefined behavior, in particular behavior caused by overflow (e.g., integer (signed or unsigned) overflow, overflow caused by invalid bit shift operations, or pointer overflow caused by arithmetic (address) operations).

[0064] Such overflows can be dangerous, because the corresponding operation that causes the overflow typically complies with the corresponding programming language standard, but the overflow may cause unexpected behavior that the programmer did not anticipate, and thus may be a common source of malfunctions in the corresponding computer program. For example, an overflow in the case of an unsigned integer typically causes the value to jump back ("wrap-around") to the beginning of the corresponding value range. Therefore, an overflow can be considered a program error.

[0065] Therefore, according to various embodiments, it can be provided that, when testing a computer program, it is checked whether an overflow has occurred based on an overflow flag (“V-Flag”), wherein the execution system 106 is designed so that the execution system sets the overflow flag in the event of an overflow (e.g. in the above-mentioned case, i.e., an integer overflow, an overflow caused by an invalid displacement operation, or a pointer overflow) (i.e., the execution system 106 has a processor that contains such a flag (e.g., in a flag register) and sets the flag in the event of an overflow). As described above, it is assumed here that the test system 106 is connected to the execution system (e.g., an embedded system) 106 by means of a debug connection, and the execution of the computer program to be tested on the execution system 106 is tested by means of a debugger.

[0066] For example, the test system 100 performs the following operations:

[0067] 1. Set breakpoints on arithmetic or displacement operations. Such operations can be easily found, for example, by decomposing the computer program to be tested. In order to select those arithmetic or displacement operations on which breakpoints are to be set, various strategies can be used (e.g., random selection or selection based on a list processed in multiple fuzz testing iterations, i.e., so that each arithmetic or displacement operation is equipped with a breakpoint, for example, in at least one fuzz testing iteration, i.e., in order to test all arithmetic or displacement operations, if there are not enough breakpoints available for all arithmetic or displacement operations, the same test case can be executed for different placements of breakpoints.).

[0068] 2. When executing the computer program, if one of the set breakpoints is triggered (i.e. an arithmetic operation or a shift operation is called), the overflow flag is checked (if necessary, one or more individual steps are performed with the aid of a debugger so that the following (assembler) instruction is executed, in response to which the overflow flag is set if necessary).

[0069] 3. In response to the overflow flag being set, display that the computer program has an error, such as outputting a warning or logging a warning, interrupting execution, triggering an exception, or crashing the computer program (so that, for example, the test program is identified as having an error when performing fuzz testing).

[0070] It should be noted that the above processing will show errors even if the overflow is the intended behavior of the programmer. To avoid this, it can be stipulated that the programmer marks the operation as correct or intended, and the test system shall not set breakpoints if the operation is so marked.

[0071] In summary, according to various embodiments, there is provided a Figure 2 The method shown in .

[0072] Figure 2 A flow chart 200 is shown which represents a method for testing a computer program according to one embodiment.

[0073] At 201, one or more breakpoints are set in a computer program on one or more arithmetic or bit shift operations.

[0074] In 202 , a computer program is executed, wherein when one of the set breakpoints is triggered, in 203 it is determined whether an overflow flag is set by performing a corresponding arithmetic or bit shift operation.

[0075] At 204, in response to determining that an overflow flag has been set by a corresponding arithmetic or bit shift operation, a display is triggered (ie, "raised") that the computer program has an error.

[0076] "One arithmetic or shift operation" should be understood as one arithmetic operation or one shift operation (possibly both). Accordingly, "multiple arithmetic or shift operations" include multiple arithmetic operations and / or shift operations.

[0077] Figure 2The method can be performed by one or more computers having one or more data processing units. The term "data processing unit" can be understood as any type of entity that enables processing of data or signals. For example, data or signals can be processed according to at least one (i.e., one or more) specific functions performed by the data processing unit. The data processing unit may include an analog circuit, a digital circuit, a logic circuit, a microprocessor, a microcontroller, a central unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a programmable gate array integrated circuit (FPGA), or any combination thereof, or may be composed of an analog circuit, a digital circuit, a logic circuit, a microprocessor, a microcontroller, a central unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a programmable gate array integrated circuit (FPGA), or any combination thereof. Any other means for implementing the corresponding functions described in more detail herein may also be understood as a data processing unit or a logic circuit device. One or more of the method steps described in detail here may be performed (e.g., implemented) by a data processing unit through one or more specific functions performed by the data processing unit.

[0078] Figure 2 The control software can be used in a test program in a manner such as control software for a robotic device. The term "robotic device" may be understood to refer to any technical system, such as a computer-controlled machine, a vehicle, a household appliance, a power tool, a manufacturing machine, a personal assistant or an access control system. The control software can also be used for data processing systems, such as navigation equipment.

[0079] Figure 2 The method is for example performed by a test device (e.g. Figure 1 computer 100 and target device 106) to execute.

[0080] Although specific embodiments have been shown and described herein, it is recognized by those skilled in the art that the specific embodiments shown and described may be replaced by various alternatives and / or equivalent implementations without departing from the scope of protection of the present invention. The present application should encompass any adaptation or variation of the specific embodiments discussed herein. Therefore, the present invention is intended to be limited only by the claims and their equivalents.

Claims

1. A method for testing a computer program (105), comprising: Setting one or more breakpoints in the computer program (105) at one or more arithmetic or bit shift operations; Executing the computer program (105); When one of the set breakpoints is triggered, determining whether an overflow flag is set by performing a corresponding arithmetic or bit shift operation; and In response to determining that the overflow flag is set by the corresponding arithmetic or bit shift operation, triggering a display that the computer program (105) has an error.

2. The method of claim 1 , wherein determining whether an overflow flag is set by performing a corresponding arithmetic or bit shift operation comprises: The instructions implementing the arithmetic or bit shift operation are traversed in individual steps.

3. The method according to claim 1 or 2, comprising: In response to determining that an overflow flag has been set by the corresponding arithmetic or bit shift operation, the display is triggered by triggering an abort of the computer program (105): the computer program (105) has an error.

4. The method according to any one of claims 1 to 3, comprising: Execute multiple test passes and set breakpoints on different arithmetic or bit shift operations for different test passes.

5. The method according to any one of claims 1 to 4, comprising: The computer program (105) is executed on an embedded system (106), and the following steps are performed by a test system (100) connected to the embedded system (106): setting a breakpoint, determining whether an overflow flag is set, and triggering a display when necessary: ​​the computer program has an error.

6. The method according to any one of claims 1 to 5, wherein the computer program (105) is a control program for a robotic device, and the computer program (105) is used to control the robotic device based on the results of the test of the computer program (105).

7. A testing device designed to carry out the method according to any one of claims 1 to 6.

8. A computer program having instructions which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 6.

9. A computer-readable medium storing instructions, which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 6.

Images