Methods for testing a computer program

Abstract

According to various embodiments, a method for testing a computer program is described, comprising executing the computer program until a memory release instruction is called for a memory area previously allocated by a memory allocation instruction of the computer program, setting, for each of one or more memory locations of the memory area released by the memory release instruction, a respective watchpoint to the memory location of the memory area and executing the memory release instruction, indicating, for each set watchpoint, that the computer program has an error if the set watchpoint is triggered, and removing, for each of the set watchpoints, the watchpoint if the memory location to which it is set is re-allocated by another memory allocation instruction in the computer program.

Description

[0001] The present disclosure relates to methods for testing a computer program.

[0002] A crucial part of software application development is testing and, if errors are found, their correction. In particular, errors that lead to application failure should be identified and corrected. An important aspect of this is testing to ensure that critical memory areas are not accessed unintentionally (or by an attacker), i.e., testing with memory monitoring, as performed by a so-called (memory) sanitizer. Compiling and testing software on common desktop and server hardware, such as x86, using various sanitizers is one way to detect errors like the Heartbleed bug, which had previously gone undetected for a long time.

[0003] Comprehensive testing, including memory monitoring, is particularly important for computer programs on embedded systems, such as vehicle control units, which are often safety-critical. Sanitizers designed for desktop and server hardware are unsuitable or poorly suited for such systems because embedded systems typically have limited resources, and these sanitizers have significant resource requirements. Therefore, they cannot be used or may even interfere with the execution of the computer program, potentially causing an error or allowing an existing error to go undetected.

[0004] Therefore, methods for testing computer programs that enable memory monitoring and are suitable for embedded systems are desirable.

[0005] According to various embodiments, a method for testing a computer program is provided, comprising executing the computer program until a memory release instruction is invoked for a memory area previously allocated by a memory allocation instruction of the computer program, setting, for each of one or more memory locations of the memory area released by the memory release instruction, a respective watchpoint to the memory location of the memory area and executing the memory release instruction, indicating, for each set watchpoint, that the computer program has an error if the set watchpoint is triggered, and removing, for each of the set watchpoints, the watchpoint if the memory location to which it is set is re-allocated by another memory allocation instruction in the computer program.

[0006] This can be done for each or at least several memory release instructions that occur in the computer program (e.g., depending on how many watchpoints are available). The watchpoints can be read watchpoints, write watchpoints, or watchpoints that trigger on both read and write operations.

[0007] The method described above enables testing with memory monitoring (i.e., with a sanitizer) regarding accesses to released memory areas on an embedded system using a debugger. This is particularly suitable for fuzzing testing, since fuzzing can also be implemented in a debugger-controlled manner and thus effectively used for embedded systems.

[0008] Sanitizers can be implemented using code instrumentation. However, this requires either access to the source code or instruction set-specific instrumentation based on the binary file (binary instrumentation), which is very vulnerable. An alternative, emulator-based instrumentation is also highly platform-specific, and each embedded platform requires its own emulator. The method described above allows testing with a debugger-driven sanitizer and requires neither instrumentation nor emulation, making it applicable in many cases.

[0009] Several exemplary implementations are given below. Example 1 is a method for testing a computer program as described above.

[0010] Exemplary embodiment 2 is the method according to exemplary embodiment 1, wherein the computer program is executed until the memory allocation instruction is called, information about which memory area is allocated by the memory allocation instruction is stored, and the one or more memory locations on which watchpoints are set are determined based on the stored information.

[0011] In other words, during a memory allocation command, execution can be paused (by setting breakpoints) and information about the allocated memory area can be saved.

[0012] Embodiment 3 is the method according to embodiment 1 or 2, comprising setting breakpoints on memory allocation instructions of the computer program and determining, for each of the set watchpoints, whether the memory location to which it is set is re-allocated by another memory allocation instruction in the computer program when one of the set breakpoints is triggered.

[0013] Thus, using a debugger (with breakpoints and watchpoints), it is possible to monitor whether released (and not yet reallocated) memory areas are being accessed (read and / or write accesses, depending on the type of watchpoints used).

[0014] Exemplary embodiment 4 is the method according to one of the exemplary embodiments 1 to 3, comprising executing the computer program on an embedded system and setting the watchpoints by a test system connected to the embedded system.

[0015] According to various embodiments, in particular the testing of a computer program for an embedded system on the embedded system itself, including memory monitoring, is made possible.

[0016] Embodiment 5 is the method according to one of embodiments 1 to 4, wherein the computer program is a control program for a robot device and the robot device is controlled depending on a result of testing the computer program with the computer program.

[0017] Exemplary embodiment 6 is an exemplary test setup configured to perform a method according to one of the exemplary embodiments 1 to 5.

[0018] Embodiment 7 is a computer program including instructions that, when executed by a processor, cause the processor to perform a method according to any one of embodiments 1 to 5.

[0019] Embodiment 8 is a computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform a method according to any one of embodiments 1 to 5.

[0020] In the drawings, like reference characters generally refer to the same parts throughout the several views. The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention. In the following description, various aspects are described with reference to the following drawings. Fig. Figure 1 shows a computer for developing and / or testing software applications. Fig. Figure 2 shows a flowchart illustrating a method for testing a computer program according to one embodiment.

[0021] The following detailed description refers to the accompanying drawings, which, by way of illustration, show specific details and aspects of this disclosure in which the invention may be practiced. Other aspects may be utilized, and structural, logical, and electrical changes may be made without departing from the scope of the invention. The various aspects of this disclosure are not necessarily mutually exclusive, as some aspects of this disclosure may be combined with one or more other aspects of this disclosure to form new aspects.

[0022] Various examples are described in more detail below.

[0023] Fig. Figure 1 shows a Computer 100 for the development and / or testing of software applications.

[0024] Computer 100 has a CPU (Central Processing Unit) 101 and a RAM 102. The RAM 102 is used to load program code, e.g., from a hard drive 103, and the CPU 101 executes the program code.

[0025] In this example, it is assumed that a user intends to develop and / or test a software application using computer 100.

[0026] To do this, the user runs a software development environment 104 on the CPU 101.

[0027] The software development environment 104 enables the user to develop and test an application 105 for various devices 106, i.e., target hardware such as embedded systems for controlling robotic devices, including robotic arms and autonomous vehicles, or for mobile (communication) devices. For this purpose, the CPU 101, as part of the software development environment 104, can run an emulator to simulate the behavior of the respective device 106 for which an application is being or has been developed. If it is used solely for testing software from another source, the software development environment 104 can also be considered or configured as a software test environment.

[0028] The user can distribute the finished application to appropriate devices 106 via a communication network 107. Instead of a communication network 107, this can also be done in other ways, for example using a USB stick.

[0029] Before this happens, however, the user should test application 105 to avoid distributing a malfunctioning application to devices 106.

[0030] One testing method is called fuzzing. Fuzzing, or fuzz testing, is an automated software testing procedure in which invalid, unexpected, or random data is fed into a computer program under test. The program is then monitored for exceptions such as crashes, missing or failed integrated code assertions, or potential memory leaks.

[0031] Typically, fuzzers (i.e., test programs that use fuzzing) are used to test programs that process structured input. This structure is specified, for example, in a file format or protocol and distinguishes between valid and invalid input. An effective fuzzer generates semi-valid input that is "valid enough" not to be directly rejected by the input parser of the program under test, but "invalid enough" to reveal unexpected behavior and edge cases that are not properly handled in the program under test.

[0032] The following describes terminology used in connection with fuzzing: • Fuzzing or fuzz testing is the automated testing process of sending randomly generated inputs to a target program (program under test) and observing its response. A fuzzer, or fuzzing engine, is a program that automatically generates input. It is not linked to the software under test, and no instrumentation is performed. However, it has the ability to instrument code, generate test cases, and execute programs under test. Well-known examples include afl and libfuzzer. • A fuzz target is a software program or function intended to be tested through fuzzing. A key characteristic of a fuzz target should be its ability to accept potentially untrusted inputs generated by the fuzzer during the fuzzing process. A fuzz test is the combined version of a fuzzer and a fuzz target. A fuzz target can then be instrumented code where a fuzzer is connected to (i.e., provides) its inputs. A fuzz test is executable. A fuzzer can also start, observe, and stop multiple fuzz tests (usually hundreds or thousands per second), each with a slightly different input generated by the fuzzer. • A test case is a specific input and a specific test run from a fuzz test. Typically, runs of interest (such as discovering new code paths or crashes) are saved for reproducibility. This allows a specific test case with the corresponding input to be executed even on a fuzz target not connected to a fuzzer, such as the release version of a program. • Coverage-guided fuzzing uses code coverage information as feedback during fuzzing to detect whether an input has caused the execution of new code paths or blocks. • Generator-based fuzzing uses prior knowledge about the target program (fuzz target) to create test inputs. An example of such prior knowledge is a grammar that corresponds to the input specification of the fuzz target, i.e., the input grammar of the fuzz target (i.e., the program under test). • Static instrumentation is the insertion of instructions into a program (under test) to obtain feedback on its execution. It is usually implemented by the compiler and can, for example, indicate the code blocks executed during runtime. Dynamic instrumentation is the control of the execution of a program (under test) during runtime in order to generate feedback from the execution. It is usually implemented through operating system functionalities or through the use of emulators. • A debugger is a device or program that can control a target device or program and provide functions such as retrieving register or memory values ​​and pausing and stepping through the target program. • A breakpoint is set via a debugger on an instruction of the target program or device to stop execution when it is reached and to inform the controlling process. • A (data) watchpoint is set via a debugger to a memory address of a target program or device to pause execution when the memory address is accessed and to inform the controlling process by triggering an interrupt.

[0033] Embedded systems typically have a microcontroller that processes inputs and responds with outputs to perform a specific task. Although microcontrollers use the same memory model and are programmed with the same programming languages ​​as regular user applications, their programs are significantly more difficult to test. To facilitate debugging, microcontrollers usually provide the ability to interrupt the program with breakpoints, step through the program's instructions one by one, and set watchpoints at memory addresses. Watchpoints trigger an interrupt when the corresponding memory locations are accessed. Hardware breakpoints and watchpoints are typically implemented as physical registers in a microcontroller's debug unit, and their number is therefore limited and depends on the specific system.For example, the maximum number of breakpoints and two data watchpoints for a typical microcontroller is four. Watchpoints can usually distinguish between read and write accesses.

[0034] Breakpoints and watchpoints can be used in particular for implementing debugger-controlled fuzzing, thus eliminating the need for instrumentation.

[0035] Fuzzing, also known as debugger-driven fuzzing, is very efficient at finding errors that trigger observable behavior, such as a crash or restart. However, entire classes of errors may go unnoticed because the program fails silently in these cases. The Heartbleed bug is one example. The core of the Heartbleed bug was that it only read beyond the boundary of an array, whereas a write operation would have caused an easily observable segmentation error.

[0036] The Heartbleed bug was only discovered with the help of the Address Sanitizer (ASan). ASan inserts additional instructions, metadata, and checks during program compilation to prevent memory corruption errors. When such sanitizer instructions are present in a program, more errors can be found during debugging than without a sanitizer. Automated tests, such as fuzzing, particularly benefit when a sanitizer is included in the program under test (i.e., in the fuzz target) to uncover additional bugs.

[0037] For embedded systems, such as a data processing device with ARM architecture, such sanitizers are not as easy to use as for standard platforms, such as x86 platforms. There are several reasons for this: • An embedded system is too resource-constrained to implement a sanitizer. For example, Asan requires twice the memory, MSan (MemorySanitizer) requires 2.5 times the resources, and UBSan (UndefinedBehaviorSanitizer) requires three times the program's memory. Sanitizers increase the size of the compiled binary file. In the automotive industry, the size of such binaries typically corresponds almost exactly to the available flash memory of the target hardware. Therefore, additional instrumentation for a sanitizer would not fit into the flash memory. • Due to the additional instrumentation required by sanitizers and the collection and tracking of metadata, using a sanitizer results in slower execution of a binary program on the respective hardware. Embedded systems are highly dependent on asynchronous events, such as interrupts, and therefore sanitizers can lead to time-based false positives; that is, a sanitizer can introduce new errors during runtime. Embedded systems typically lack a user interface to display runtime errors. On x86 systems, for example, a segmentation error is passed to STDERR, so the user sees the crash. Embedded systems, on the other hand, fail silently, i.e., without the user noticing, and restart after such a crash.

[0038] According to various embodiments, an approach is therefore provided that enables the use of memory monitoring (i.e., a sanitizer functionality) for an embedded system, in particular such that the memory monitoring can be used for debugger-controlled fuzzing. In this approach, the memory monitoring itself is enabled with the aid of a debugger (or the debugger used for fuzzing).

[0039] In debugger-based fuzzing, interactions occur between the system performing the test (e.g., computer 100) and the target system (target hardware, e.g., an embedded system, for example, a target device 106) via a debug connection (i.e., a debug interface), provided, for example, by a dedicated debugger hardware device. The test input data is transmitted to the target system 106 in the form of an input vector, e.g., via Wi-Fi or a CAN bus (depending on the type of target device 106). In this test, the communication network 107 acts as such a debug connection (when distributing the software under test, the communication network can be any other communication network). The system performing the test, hereinafter also referred to as test system 100, controls the execution of the target program (i.e., the program under test) in the target system via the debug connection.starts execution and resumes execution after an interrupt (especially an interrupt triggered by a data watchpoint).

[0040] A debugger-driven sanitizer requires no instrumentation or emulation, but only a debug interface to the target system (e.g., an embedded system on which the software is running) with the ability to set breakpoints and watchpoints. These types of debug interfaces and capabilities are generic and widely available, resulting in broad and easy applicability of the approach described below. Furthermore, the target system's memory usage is minimal, for example, for metadata, since most or all sanitizer-related information is collected and stored on the debugger's host side (i.e., in the system under test 100). This allows the embedded system to be tested even in its final, retail version (as sold, for example). The size of the target program's compiled binary is not increased, as it can be used for testing exactly as it is intended for use on the target system 106.

[0041] A debugger halts the target system when a breakpoint is reached. Therefore, the approach described below only rarely leads to time-based false alarms. These false alarms can also be eliminated by other testing techniques, such as subsequent validation of a detected error on the target system. Furthermore, using a debugger provides valuable insight into the internal workings of a target system.

[0042] The approach described below is used to detect access to already released memory areas (i.e., dynamic buffer areas in memory).

[0043] For example, test system 100 performs the following: 1. Setting breakpoints on memory allocation instructions such as malloc, realloc, and calloc 2. When one of the memory allocation instructions is called, an interrupt is triggered by the respective set breakpoint. In response, the test system 100 stores the base address (i.e., the starting address of the respective memory area to be allocated) and the size of the memory area to be allocated as allocation metadata. 3. Setting breakpoints on memory release commands such as `free` 4. In response to an interrupt being triggered by one of the breakpoints set on a memory release instruction a. Storing the pointer of the memory release instruction (i.e., the beginning of the memory area that the memory release function is to release) as release metadata b. Setting one or more watchpoints to memory locations within the memory area that the memory release function is to release (ideally to all memory locations within the memory area that the memory release function is to release, if enough watchpoints are available); the size of the memory area can be determined, for example, from the allocation metadata, which in turn can be assigned to the memory release instruction based on the memory address (i.e., the release metadata). 5. If one of the watchpoints is triggered, this is interpreted as an error (access to shared memory). In this case, test system 100 will therefore indicate that an error has occurred, which in turn can trigger the execution of a security measure. 6. If a monitored memory area, i.e., a memory area containing one or more memory locations on which watchpoints (from 4b) are set, is re-allocated (which can be detected by monitoring the memory allocation functions, e.g., by triggering the breakpoints set on it (in 1), the watchpoint(s) set on the memory area will be removed.

[0044] Since the number of breakpoints and watchpoints may be limited, it is possible that not all released (or to be released) memory areas can be fully monitored in this way. If this is the case, the test system (e.g., a fuzzer used to test computer program 105) can randomly select the subset of memory areas and also the parts of those areas (i.e., the memory locations) to be monitored, or the memory areas can be monitored sequentially in several runs of the target program (e.g., fuzz test runs).

[0045] A memory location is understood here as a unit of memory that is monitored by a watchpoint, i.e., the memory area assigned to a memory address (which can be chosen as the target of a watchpoint).

[0046] In summary, according to various embodiments, a method is provided as described in Fig. 2 shown.

[0047] Fig. Figure 2 shows a flowchart 200, which represents a method for testing a computer program according to one embodiment.

[0048] In 201, the (to be tested) computer program is executed until a memory release instruction is called for a memory area previously allocated by a memory allocation instruction of the computer program.

[0049] In 202, for each of one or more memory locations of the memory area that is freed by the memory release command, a corresponding watchpoint is set on the memory location of the memory area and the memory release command is executed.

[0050] In section 203, for each set watchpoint, it is stated that the computer program has an error if (i.e., in response to) the set watchpoint being triggered.

[0051] In section 204, for each of the set watchpoints, the watchpoint is removed if the memory location to which it is set is re-allocated by another memory allocation instruction in the computer program.

[0052] The procedure of Fig. 2. This can be performed by one or more computers with one or more data processing units. The term "data processing unit" can be understood as any type of entity that enables the processing of data or signals. The data or signals can, for example, be processed according to at least one (i.e., one or more than one) special function performed by the data processing unit. A data processing unit can include or be composed of an analog circuit, a digital circuit, a logic circuit, a microprocessor, a microcontroller, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an integrated circuit, a programmable gate array (FPGA), or any combination thereof.Any other way of implementing the respective functions described in more detail herein may also be understood as a data processing unit or logic circuit arrangement. One or more of the method steps described in detail herein may be performed (e.g., implemented) by a data processing unit through one or more specific functions performed by the data processing unit.

[0053] The approach of Fig. 2. This is used to test a program, for example, control software for a robot device. The term "robot device" can be understood as referring to any technical system, such as a computer-controlled machine, a vehicle, a household appliance, a power tool, a manufacturing machine, a personal assistant, or an access control system. The control software can also be used for data processing systems such as a navigation device.

[0054] The procedure of Fig. 2 is, for example, achieved by a test setup (e.g., computer 100 and target device 106 of Fig. 1) carried out.

[0055] Although specific embodiments have been presented and described herein, it will be recognized by those skilled in the art that the specific embodiments shown and described can be replaced by a variety of alternative and / or equivalent implementations without departing from the scope of protection of the present invention. This application is intended to cover any adaptations or variations of the specific embodiments discussed herein. Therefore, it is intended that this invention is limited only by the claims and their equivalents.

Claims

[1] A method for testing a computer program (105), comprising: Executing the computer program (105) until a memory release command is called for a memory area previously allocated by a memory allocation command of the computer program (105); Setting, for each one or more memory locations of the memory area released by the memory release instruction, a respective watchpoint to the memory location of the memory area and executing the memory release instruction; Indicating, for each set watchpoint, that the computer program (105) has an error if the set watchpoint is triggered; and For each of the set watchpoints, removing the watchpoint if the memory location to which it is set is re-allocated by another memory allocation instruction in the computer program (105). [2] Method according to claim 1, wherein the computer program (105) is executed until the memory allocation command is called, information about which memory area is allocated by the memory allocation command is stored, and the one or more memory locations to which watchpoints are set are determined based on the stored information. [3] Method according to claim 1 or 2, comprising setting breakpoints on memory allocation instructions of the computer program (105) and determining, for each of the set watchpoints, whether the memory location to which it is set is re-allocated by another memory allocation instruction in the computer program (105) when one of the set breakpoints is triggered. [4] Method according to one of claims 1 to 3, comprising executing the computer program (105) on an embedded system (106) and setting the watchpoints by a test system (100) connected to the embedded system (106). [5] Method according to one of claims 1 to 4, wherein the computer program (105) is a control program for a robot device and the robot device is controlled with the computer program (105) depending on a result of the testing of the computer program (105). [6] Test arrangement which is arranged to carry out a method according to one of claims 1 to 5. [7] A computer program comprising instructions which, when executed by a processor, cause the processor to perform a method according to any one of claims 1 to 5. [8] A computer-readable medium storing instructions which, when executed by a processor, cause the processor to perform a method according to any one of claims 1 to 5.

Images