Method for testing computer program

Abstract

According to various embodiments, a method for testing a computer program is described, comprising: setting one or more breakpoints onto one or more memory release instructions in the computer program; executing the computer program, and when one of the set breakpoints is triggered, setting a pointer obtained by the memory release instruction as an explanation of the storage area to be released as a value that triggers an exception when the pointer is subsequently used in the computer program.

Description

Technical Field

[0001] The present disclosure relates to a method for testing a computer program. Background Art

[0002] An important part of developing software applications is testing and the corresponding error clearing when errors are found. In particular, errors that lead to application failures should be identified and corrected. An important aspect here is testing with the view that important memory areas are not accessed accidentally (or by an attacker), i.e. using memory monitoring such as that performed by so-called (memory) sanitizers. Compiling and testing software on common desktop and server hardware such as x86 with the help of various sanitizers is a measure that can discover errors such as the Heartbleed bug that previously remained undiscovered for a long time.

[0003] Comprehensive testing that also includes such memory monitoring is particularly important for computer programs on often safety-relevant embedded systems, such as control devices for vehicles. However, sanitizers for desktop and server hardware cannot be used or can only be used poorly for such systems, since embedded systems typically have restricted resources and such sanitizers have a huge resource requirement and therefore cannot be used or can even affect the execution of the computer program so that errors occur in the first place or remain undetected. Summary of the invention

[0004] Therefore, what are desired are methods for testing computer programs that allow memory monitoring and are applicable to embedded systems.

[0005] According to various embodiments, a method for (automatically) testing a computer program is provided, comprising: setting one or more breakpoints in the computer program to one or more memory release instructions; executing the computer program; and when one of the set breakpoints is triggered, setting a pointer obtained by the memory release instruction as a description of a storage area to be released to a value (e.g. a null value) that will trigger an exception when the pointer is subsequently used in the computer program.

[0006] This may be done for each or at least a plurality of memory release instructions occurring in the computer program (eg depending on how many breakpoints are available).

[0007] The above method enables testing with detection of dangling pointers on embedded systems with the aid of a debugger. This is particularly applicable when testing with fuzz testing, since fuzz testing can also be implemented in a debugger-controlled manner and in this way can be effectively used on embedded systems.

[0008] By triggering an exception, for example, in the context of fuzz testing, a computer program can be marked as having an error, because its execution causes an abort of the computer program (and thus an abort or a crash), followed by the fuzzer displaying an error. For example, a warning can also be output on the stderr stream, or the debugger can be monitored on the host side (i.e., on a test system that tests the computer program on the execution system with the aid of a debugger).

[0009] Sanitizers can be implemented with the aid of code instrumentation. However, for this purpose either the source code must be available or command set-specific instrumentation based on binaries (binary instrumentation) is required, which is very susceptible. The alternative emulator-based instrumentation is also very platform-specific, and each embedded platform requires its own emulator. The method described above enables testing with a debugger-controlled sanitizer without the need for instrumentation or emulation, and can therefore be applied in many cases.

[0010] Various embodiments are described below.

[0011] Embodiment 1 is the above-mentioned method for testing a computer program.

[0012] Embodiment 2 is a method according to embodiment 1, comprising: determining which values ​​a system executing a computer program will respond with an exception to, and setting the value to which the pointer is set to one of the determined values.

[0013] Therefore, the method can be used in various execution systems.

[0014] Embodiment 3 is a method according to embodiment 1 or 2, comprising: executing multiple test passes (eg, fuzzy test passes, i.e., fuzzy test iterations) and setting breakpoints to different memory release instructions for different test passes.

[0015] Thereby, a large number of memory release instructions can be covered.

[0016] Embodiment 4 is a method according to any one of embodiments 1 to 3, comprising: executing a computer program on an embedded system, and performing the following steps through a test system connected to the embedded system (via a debugging interface): setting breakpoints, and setting a pointer obtained by a memory release instruction as a description of the storage area to be released to a value that will trigger an exception when the pointer is subsequently used in the computer program.

[0017] According to various embodiments, in particular, testing of a computer program for an embedded system, including memory monitoring, is enabled on the embedded system itself.

[0018] Embodiment 5 is a method according to any one of embodiments 1 to 4, comprising: when one of the set breakpoints is triggered, checking whether a (local) variable (in the corresponding stack frame or in a register) contains the address of a storage unit of the storage area to be released, and setting the variable to the value or other value that will trigger an exception when the pointer is subsequently used in the computer program if necessary.

[0019] This makes it possible, for example, to prevent a pointer that has been calculated from an original pointer from dangling due to the release.

[0020] Embodiment 6 is a method according to any one of embodiments 1 to 5, wherein the computer program is a control program for a robotic device, and the robotic device is controlled using the computer program based on a result of a test of the computer program.

[0021] Embodiment 7 is a testing device, which is designed to perform the method according to any one of embodiments 1 to 6.

[0022] Embodiment 8 is a computer program having instructions, which, when executed by a processor, cause the processor to perform the method according to any one of embodiments 1 to 6.

[0023] Embodiment 9 is a computer-readable medium storing instructions, which, when executed by a processor, causes the processor to perform the method according to any one of embodiments 1 to 6. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In the accompanying drawings, similar reference numerals generally refer to the same parts throughout the various views. The accompanying drawings are not necessarily drawn to scale, with emphasis instead generally being placed on illustrating the principles of the invention. In the following description, various aspects are described with reference to the following drawings.

[0025] Figure 1 A computer used to develop and / or test software applications is shown.

[0026] Figure 2A flow chart is shown which illustrates a method for testing a computer program according to one embodiment. DETAILED DESCRIPTION

[0027] The following detailed description relates to the attached drawings, which illustrate specific details and aspects of the present disclosure that can perform the present invention for the purpose of illustration. Other aspects may be used and structural, logical and electrical changes may be made without departing from the scope of protection of the present invention. Because some aspects of the present disclosure may be combined with one or more other aspects of the present disclosure to form new aspects, the various aspects of the present disclosure are not necessarily mutually exclusive.

[0028] Various examples are described in more detail below.

[0029] Figure 1 A computer 100 is shown for use in developing and / or testing software applications.

[0030] The computer 100 has a CPU (Central Processing Unit) 101 and a work memory (RAM) 102. The work memory 102 is used to load a program code from a hard disk 103, for example, and the CPU 101 executes the program code.

[0031] In the present example it is assumed that a user intends to utilize computer 100 to develop and / or test a software application.

[0032] To this end, the user executes the software development environment 104 on the CPU 101 .

[0033] The software development environment 104 enables users to develop and test applications 105 for various devices 106, i.e., for target hardware (such as embedded systems for controlling robotic devices including robot arms and autonomous vehicles) or for mobile (communication) devices. To this end, the CPU 101 can execute an emulator as part of the software development environment 104 to simulate the behavior of the corresponding device 106 for which the application is to be or has been developed. If the software development environment 104 is only used to test software from other sources, the software development environment 104 can also be regarded as or designed as a software testing environment.

[0034] The user can distribute the finished application to the corresponding device 106 via the communication network 107. Instead of via the communication network 107, this can also be done in other ways, for example by means of a USB stick.

[0035] Before this occurs, however, the user should test the application 105 to avoid distributing an application to the device 106 that does not function as specified.

[0036] One testing method is so-called fuzz testing. Fuzz testing (Fuzzing or Fuzz-Testing) is an automated software testing method in which invalid, unexpected or random data is fed as input to the computer program to be tested. The program is then monitored for anomalies such as crashes, lack of failed built-in code assertions or potential memory leaks.

[0037] Typically, fuzzers (i.e., test programs using fuzz testing) are used to test programs that process structured inputs. The structure is specified, for example, in a file format or file format or protocol and distinguishes between valid and invalid inputs. An effective fuzzer produces semi-valid inputs that are "valid enough" to not be directly rejected by the input parser of the program under test, but "invalid enough" to reveal unexpected behavior and edge cases that are not properly handled in the program under test.

[0038] The following describes the terminology used in conjunction with fuzz testing:

[0039] Fuzz testing (Fuzzing or Fuzz-Testing) is an automated testing process that sends randomly generated inputs to a target program (the program under test) and observes its responses.

[0040] A fuzzer or fuzz testing engine is a program that automatically generates input. Therefore, the fuzzer or fuzz testing engine is not associated with the software under test and does not perform instrumentation. However, the fuzzer or fuzz testing engine has the ability to instrument code, generate test cases and execute the program under test. Known examples are afl and libfuzzer.

[0041] A fuzz target is a software program or function that should be tested by fuzz testing. The main feature of a fuzz target should be that it takes potentially untrusted inputs generated by the fuzzer during the fuzz testing process.

[0042] A fuzz test is a combined version of a fuzzer and a fuzz target. Thus, a fuzz target can be instrumented code, where the fuzzer is associated with its input (i.e., provides the input). Fuzz tests are executable. A fuzzer can also start, observe, and stop multiple fuzz tests (typically hundreds or thousands per second), each with slightly different inputs generated by the fuzzer.

[0043] A test case is a specific input and a specific test run from a fuzz test. Usually, the runs that are of interest for reproducibility (finding new code paths or crashes) are stored. In this way, a specific test case with the corresponding input can also be executed on a fuzz target that is not connected to the fuzzer, such as a released version of the program.

[0044] Coverage-guided fuzzing uses code coverage information as feedback during fuzz testing to identify whether an input causes the execution of new code paths or blocks.

[0045] Generator-based fuzz testing (English: generation-based fuzzing) uses prior knowledge about the target program (fuzz target) to create test inputs. An example of such prior knowledge is the grammar corresponding to the input specification of the fuzz target, i.e. the input grammar of the fuzz target (i.e. the program to be tested).

[0046] Static instrumentation is the insertion of commands into a program (to be tested) to obtain feedback about the execution. The static instrumentation is mostly implemented by the compiler and can, for example, describe the code blocks reached during execution.

[0047] Dynamic instrumentation is controlling the execution of a program (to be tested) during runtime to generate feedback from the execution. The dynamic instrumentation is mostly implemented by operating system functionality or by using an emulator.

[0048] A debugger is a device or a program that can control a target device or a target program and can provide functions such as for calling register values ​​or memory values ​​and for pausing and executing a target program in individual steps.

[0049] ·Breakpoints are set by the debugger on commands of the target program or device so that execution stops when they are reached and the controlling process is notified of this.

[0050] Setting a (data) watchpoint by the debugger at a memory address of the target program or target device, so that execution is halted when the memory address is accessed, and the controlling process is informed of this by triggering an interrupt.

[0051] Embedded systems usually have a microcontroller that processes inputs and responds with outputs to perform specific tasks. Although microcontrollers use the same memory model as conventional user programs and are programmed using the same programming language, microcontroller programs are significantly more difficult to test. In order to enable debugging, microcontrollers usually provide the following possibilities: interrupting the program using breakpoints (stop points), traversing the commands of the program in individual steps, and setting watchpoints on storage addresses. When accessing the corresponding storage area, the watchpoint triggers an interrupt. Hardware breakpoints and watchpoints are typically implemented as physical registers in the debugging unit of the microcontroller, and therefore their number is limited and depends on the corresponding system. For example, for a typical microcontroller, the maximum number is four breakpoints and two data watchpoints. Usually watchpoints can be distinguished between read access and write access.

[0052] In particular, breakpoints and watchpoints can be used to implement debugger-controlled fuzz testing, eliminating the need for instrumentation.

[0053] Fuzz testing (and debugger-controlled fuzz testing) is very effective at finding bugs that trigger observable behavior, such as crashes or reboots. However, not all classes of bugs can be observed because the program silently fails on them. An example is the Heartbleed vulnerability. The core of the Heartbleed vulnerability is that Heartbleed only reads out of bounds of an array, while writes result in segmentation faults that are easily observable.

[0054] The Heartbleed vulnerability was only found with the help of Address Sanitizer (ASan). ASan inserts additional commands, metadata, and checks during the compilation of a program to prevent errors when memory is corrupted. When such Sanitizer commands are available in a program, more errors can be found when debugging the program than without the Sanitizer. Automated testing (such as fuzz testing) is particularly interesting when a Sanitizer is set up in the program to be tested (i.e. in the fuzz target) to find additional errors.

[0055] For embedded systems (such as data processing devices with ARM architecture), such sanitizers are not as easy to use as for standard platforms (such as x86 platforms). There are many reasons for this:

[0056] Embedded systems are too resource-constrained to implement a Sanitizer. For example, Asan requires double the memory, MSan (MemorySanitizer) requires 2.5 times the resources, and UBSan (UndefinedBehaviorSanitizer) even requires three times the working memory of the program.

[0057] The Sanitizer increases the size of the compiled binary. In the automotive industry, the size of such binaries often almost corresponds to the available flash memory of the target hardware. Therefore, the additional instrumentation of the Sanitizer does not fit in the flash memory.

[0058] The use of Sanitizers results in slower runtime of binaries on the corresponding hardware due to the Sanitizer's additional instrumentation and metadata collection and tracking. Embedded systems strongly depend on asynchronous events, such as interrupts, and thus Sanitizers may cause timing-based false positive errors, i.e., Sanitizers may introduce new errors during runtime.

[0059] Embedded systems usually do not have a user interface for displaying runtime errors. On x86 systems, for example, segmentation faults are forwarded to STDERR so that the user sees the crash. Instead, embedded systems fail silently, i.e. without the user noticing this, and restart after such a crash.

[0060] Therefore, according to various embodiments, a method is provided that enables memory monitoring (i.e., Sanitizer functionality) to be used for embedded systems, in particular, memory monitoring can be used for debugger-controlled fuzz testing. Here, the memory monitoring itself can be implemented with the help of a debugger (or a debugger for fuzz testing).

[0061] In debugger-based fuzz testing, the interaction between the system executing the test (and corresponding to, for example, the computer 100) and the target system (target hardware, such as an embedded system, such as the target device 106) is carried out via a debug connection (i.e., a debug interface), which is provided, for example, by a dedicated debugger hardware device. The test input data is transmitted to the target system 106 in the form of an input vector, for example, via WiFi or a CAN bus (depending on the type of the target device 106), that is, the communication network 107 is such a debug connection during this test (then when distributing the tested software, the communication network can be any other communication network). The system executing the test (hereinafter also referred to as the test system 100) controls the execution of the target program (i.e., the program to be tested) in the target system via the debug connection, that is, starts the execution and resumes the execution after an interrupt (especially an interrupt triggered by a data monitoring point).

[0062] The debugger-controlled Sanitizer does not require plugging or emulation, but only requires a debug interface with the target system (e.g., an embedded system on which the software is executed) with the possibility of setting breakpoints and monitoring points. This type of debug interface and debugging capabilities are universal and also available, which leads to the wide and simple applicability of the method described below. In addition, since most or all of the information related to the Sanitizer is collected and stored on the host side of the debugger (i.e., in the test system 100), the embedded system can also be tested with its completed version (e.g., the sold version), so the memory of the target system is only slightly loaded, for example, for metadata. The size of the compiled binary file of the target program does not increase, because the binary file is used during testing as if it were set for the target system 106.

[0063] When a breakpoint is reached, the debugger halts the target system. Therefore, the approach described below leads to time-based false alarms only in rare cases. These false alarms can also be eliminated by other testing techniques, such as by subsequently verifying the errors found on the target system. Using a debugger also provides a good view of the internal affairs of the target system.

[0064] The method described below is used to detect dangling pointers using a debugger. A dangling pointer is a pointer that points to invalid (i.e. no longer valid) data. The following program code is an example.

[0065]

[0066] When the memory area pointed to by pointer dp is released, pointer dp will be dangling. Therefore, after calling the memory release instruction (free in this example), any reference made with the help of this pointer may lead to unwanted behavior, especially data leakage.

[0067] The problem of dangling pointers can be avoided by setting the pointer to a null value after the memory deallocation instruction:

[0068]

[0069] However, this must be programmed into the computer by a programmer.

[0070] Therefore, according to various embodiments, it is provided that such a defect (i.e. a program code, in particular a memory release instruction, which results in a dangling pointer, for example based on a binary file of a computer program) is detected (with the aid of a debugger which does not need to be run on the execution system). In the case of such a detection, a pointer indicating the memory area to be released by the memory release instruction is set to a null value (or to another value which triggers or causes an exception). Other parameters may also be set to such values, such as a variable containing the following address: this address also points to or refers to the memory area to be released (for example a pointer derived from the original pointer).

[0071] As described above, it is assumed here that the test system 106 is connected to the execution system (eg embedded system) 106 by means of a debug connection and that the execution of the computer program to be tested on the execution system is tested by means of a debugger.

[0072] For example, the test system 100 performs the following operations:

[0073] 1. Set a breakpoint on a memory release instruction (e.g., free or a function that calls free). In order to select the memory release instruction to set a breakpoint from the memory release instructions present in the computer program, various strategies may be used (e.g., random selection or selection based on a list processed in multiple fuzz testing iterations, i.e., so that each memory release instruction is equipped with a breakpoint, for example, in at least one fuzz testing iteration, i.e., in order to test all memory release instructions, if there are not enough breakpoints available for all memory release instructions, the same test case may be executed for different placements of breakpoints.).

[0074] 2. When executing the computer program, if one of the set breakpoints is triggered, i.e., a memory release instruction (e.g., a call to free) is executed, the pointer (or a variable derived from the pointer) is set to a value that triggers an exception (e.g., a null value or other specific bit pattern to which the execution system responds with an exception). Thus, if the pointer is used for reference again, an exception is triggered, e.g., if the pointer has been overwritten with a null value, a null pointer exception is triggered, so that the use of the pointer can be seen after the memory release instruction.

[0075] In summary, according to various embodiments, there is provided a Figure 2 The method shown in .

[0076] Figure 2 A flow chart 200 is shown which represents a method for testing a computer program according to one embodiment.

[0077] In 201 , one or more breakpoints are set in a computer program on one or more memory release instructions.

[0078] In 202, a computer program is executed, wherein in 203, when one of the set breakpoints is triggered, a pointer obtained by the memory release instruction as a description of the storage area to be released is set to a value (e.g. a null value) that will trigger an exception when the pointer is subsequently used in the computer program.

[0079] The anomaly can then be responded to, for example by indicating that the computer program contains an error.

[0080] Figure 2The method can be performed by one or more computers having one or more data processing units. The term "data processing unit" can be understood as any type of entity that enables processing of data or signals. For example, data or signals can be processed according to at least one (i.e., one or more) specific functions performed by the data processing unit. The data processing unit may include an analog circuit, a digital circuit, a logic circuit, a microprocessor, a microcontroller, a central unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a programmable gate array integrated circuit (FPGA), or any combination thereof, or may be composed of an analog circuit, a digital circuit, a logic circuit, a microprocessor, a microcontroller, a central unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a programmable gate array integrated circuit (FPGA), or any combination thereof. Any other means for implementing the corresponding functions described in more detail herein may also be understood as a data processing unit or a logic circuit device. One or more of the method steps described in detail here may be performed (e.g., implemented) by a data processing unit through one or more specific functions performed by the data processing unit.

[0081] Figure 2 The control software can be used in a test program in a manner such as control software for a robotic device. The term "robotic device" may be understood to refer to any technical system, such as a computer-controlled machine, a vehicle, a household appliance, a power tool, a manufacturing machine, a personal assistant or an access control system. The control software can also be used for data processing systems, such as navigation equipment.

[0082] Figure 2 The method is for example performed by a test device (e.g. Figure 1 computer 100 and target device 106) to execute.

[0083] Although specific embodiments have been shown and described herein, it is recognized by those skilled in the art that the specific embodiments shown and described may be replaced by various alternatives and / or equivalent implementations without departing from the scope of protection of the present invention. The present application should encompass any adaptation or variation of the specific embodiments discussed herein. Therefore, the present invention is intended to be limited only by the claims and their equivalents.

Claims

1. A method for testing a computer program (105), comprising: setting one or more breakpoints in the computer program (105) at one or more memory release instructions; Executing the computer program (105); as well as When one of the set breakpoints is triggered, the pointer obtained by the memory release instruction as a description of the memory area to be released is set to a value that will trigger an exception when the pointer is subsequently used in the computer program (105).

2. The method of claim 1 , comprising determining which values ​​a system executing the computer program will respond with an exception to, and setting the value to which the pointer is set to one of the determined values.

3. The method according to claim 1 or 2, comprising: Multiple test passes are executed and breakpoints are set at different memory deallocation instructions for different test passes.

4. The method according to any one of claims 1 to 3, comprising: The computer program (105) is executed on an embedded system (106), and the following steps are performed by a test system (100) connected to the embedded system (106): a breakpoint is set, and a pointer obtained by the memory release instruction as a description of the storage area to be released is set to a value that will trigger an exception when the pointer is subsequently used in the computer program (105).

5. The method according to any one of claims 1 to 4, comprising: When one of the set breakpoints is triggered, a check is performed to determine whether a variable contains the address of a memory cell of the memory area to be released and, if necessary, the variable is set to the value or another value which triggers an exception when the pointer is subsequently used in the computer program.

6. The method according to any one of claims 1 to 5, wherein the computer program (105) is a control program for a robotic device, and the computer program (105) is used to control the robotic device based on the results of the test of the computer program (105).

7. A testing device designed to carry out the method according to any one of claims 1 to 6.

8. A computer program having instructions which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 6.

9. A computer-readable medium storing instructions, which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 6.

Images