Methods for testing a computer program

Abstract

According to various embodiments, a method for testing a computer program is described, comprising setting one or more breakpoints on one or more memory release instructions in the computer program, executing the computer program, and, upon triggering one of the set breakpoints, setting a pointer, which the memory release instruction receives as an indication of the memory area to be released, to a value that triggers an exception upon subsequent use of the pointer in the computer program.

Description

The present disclosure relates to methods for testing a computer program.An important part of the development of software applications is testing and, if errors are found, a corresponding error correction. In particular, errors that lead to the failure of an application should be identified and corrected. An important aspect in this case is testing with regard to the fact that important memory areas are not accessed unintentionally (or by an attacker), i.e. testing with memory monitoring, as is carried out by a so-called (memory) tester. Compiling and testing software on common desktop and server hardware, e.g., x86, using various Sanitizers is one way against which errors such as the Learned bug that has long been undetected can be detected.Particularly important for computer programs on embedded systems, such as controllers for a vehicle, which are often safety relevant, is extensive testing that also involves such memory monitoring. However, sanitizers used for desktop and server hardware are not usable or are only poorly usable for such systems because embedded systems typically have limited resources and such sanitizers have a considerable resource requirement and therefore cannot be used or even can influence the execution of the computer program such that an error first arises at all or an error remains undetected.Methods for testing computer programs that enable memory monitoring and are suitable for embedded systems are therefore desirable.According to various embodiments, a method for (automatically) testing a computer program is provided, comprising setting one or more breakpoints to one or more memory release commands in the computer program, executing the computer program, and upon triggering one of the set breakpoints, setting a pointer, which the memory release failure receives as an indication of the memory area to be released, to a value (e.g. zero) which triggers an exception upon a subsequent use of the pointer in the computer program.This may be done for each or at least multiple memory enable commands that occur in the computer program (e.g., depending on how many breakpoints are available).The method described above allows for hanging pointer detection testing on an embedded system using a debuggger. This is particularly suitable for testing with fuzzing, since fuzzing can also be implemented in a manner controlled by debugger and can thus be used effectively for embedded systems.By triggering an exception, the computer program can be marked as faulty, for example, as part of fuzzing, since its execution causes an exception (and thus an abort or crash), whereupon the fuzzer indicates an error. It would also be possible, for example, to issue a warning on the std err stream or on the host side (i.e. on the test system that the computer program tests on an executing system by means of a debugger), the debugger being monitored.Sanitizers can be implemented by code instrumentation. However, this requires either the source code to be available or instruction set-specific instrumentation based on the binary file (binary instrumentation) is required, which is very vulnerable. Alternative emulator-based instrumentation is also very platform specific, and each embedded platform requires a separate emulator. The method described above allows testing with a Debugger-controlled Sanger and does not require instrumentation or emulation and is therefore applicable in many cases.Various exemplary embodiments are given below.Embodiment 1 is a method for testing a computer program as described above.Embodiment 2 is the method according to Embodiment 1, comprising determining to which values the system on which the computer program is executed reacts with an exception and setting the value to which the pointer is set to one of the determined values.Thus, the method can be used for various executing systems.Embodiment 3 is the method of embodiment 1 or 2, comprising performing multiple test passes (e.g., fuzzing test passes, i.e., fuzzing iterations) and setting breakpoints to, from test pass to test pass, other memory release commands.Thus, a large number of memory enable commands can be covered.Exemplary embodiment 4 is the method according to one of exemplary embodiments 1 to 3, comprising executing the computer program on an embedded system and carrying out the setting of the breakpoints and the setting of the pointer which the memory release miss receives as an indication of the memory area to be released to a value which triggers an exception on a subsequent use of the pointer in the computer program by a test system connected to the embedded system (via a debugging interface).According to various embodiments, in particular testing of a computer program for an embedded system is made possible on the embedded system itself including a memory monitoring.Exemplary embodiment 5 is a method according to one of exemplary embodiments 1 to 4, comprising, when one of the set breakpoints is triggered, checking whether a (local) variable (in the respective stack frame or else in a register) contains an address of a memory cell of the memory area to be released and, if appropriate, setting the variable to the value or another value which triggers an exception when the pointer is used in the computer program subsequently.Thus, for example, pointers calculated from the original pointer can be prevented from becoming suspended by the release.Exemplary embodiment 6 is the method according to one of exemplary embodiments 1 to 5, wherein the computer program is a control program for a robot device and the robot device is controlled on the basis of a result of testing the computer program using the computer program.Exemplary embodiment 7 is a test arrangement which is configured to carry out a method according to one of exemplary embodiments 1 to 6.Embodiment 8 is a computer program comprising instructions which, when executed by a processor, cause the processor to perform a method according to any of embodiments 1 to 6.Embodiment 9 is a computer readable medium storing instructions that, when executed by a processor, cause the processor to perform a method according to any of Embodiments 1 to 6.In the drawings, like reference numerals generally refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention. In the following description, various aspects will be described with reference to the following drawings. Figure 1 shows a computer for the development and / or testing of software applications. FIG. 2 is a flow diagram illustrating a method for testing a computer program according to an embodiment.The following detailed description refers to the accompanying drawings, which show, by way of illustration, specific details and aspects of this disclosure in which the invention may be practiced. Other aspects may be utilized and structural, logical, and electrical changes may be made without departing from the scope of the invention. The various aspects of this disclosure are not necessarily mutually exclusive, as some aspects of this disclosure may be combined with one or more other aspects of this disclosure to form novel aspects.Various examples will be described in more detail below.FIG. 1 shows a computer 100 for the development and / or testing of software applications.The computer 100 includes a CPU (Central Processing Unit) 101 and a random access memory (RAM) 102. The working memory 102 is used for loading program code, for example, from a hard disk 103, and the CPU 101 executes the program code.In the present example, it is believed that a user intends to develop and / or test a software application with the computer 100.To this end, the user executes a software development environment 104 on the CPU 101.The software development environment 104 allows the user to develop and test an application 105 for various devices 106, that is, target hardware such as embedded systems for controlling robot devices including robot arms and autonomous vehicles, or also for mobile (communication) devices. To do so, the CPU 101 may execute, as part of the software development environment 104, an emulator to simulate the behavior of the particular device 106 for which an application is or was being developed. If it is used only for testing software from another source, the software development environment 104 can also be considered or designed as a software test environment.The user may distribute the completed application to respective devices 106 via a communication network 107. Instead of via a communication network 107, this can also be done in another manner, for example by means of a USB stick.However, before doing so, the user should test the application 105 to avoid distributing an improperly functioning application to the devices 106.One test method is fuzz. Fuzzing or fuzz testing is an automated software test method in which invalid, unexpected or random data is supplied as inputs to a computer program to be tested. The program is then monitored for exceptions such as crashes, missing failed integrated code assertions, or potential memory leaks.Typically, fuzzers (i.e., test programs that use fuzz) are used to test programs that process structured inputs. This structure is specified, for example, in a file format or a file format or protocol and distinguishes between valid and invalid entries. An effective fuzzer generates semi-valid inputs that are "valid enough" not to be rejected directly by the input parser of the program under test, but are "invalid enough" to reveal unexpected behaviors and boundaries that are not properly handled in the program under test.Terminology used in the context of fuzzing will be described below:• Fuzzing or fuzz testing is the automated test process to send randomly generated inputs to a target program (program to be tested) and observe its response.• A fuzzer or fuzzing engine is a program that automatically generates inputs. It is therefore not linked to the software to be tested, and instrumentation is also not performed. However, it has the ability to instrument code, generate test cases, and execute programs to be tested. Known examples are afl and libfuzzer.• A fuzz target is a software program or function to be tested by fuzz. A key feature of a fuzz target should be that it potentially accepts untrusted inputs generated by the fuzzer during the fuzzing process.• A fuzz test is the combined version of a fuzzer and a fuzz target. A fuzz target may then be instrumented code in which a fuzzer is associated with (i.e., provides) its inputs. A fuzz test is executable. A fuzzer may also start, observe, and stop multiple fuzz tests (typically hundreds or thousands per second), each with a slightly different input generated by the fuzzer.• A test case is a particular input and test run from a fuzz test. Normally, runs of interest for reproducibility (finding new code paths or crashes) are stored. In this way, a particular test case with the corresponding input can also be executed on a fuzz target that is not connected to a fuzzer, e.g. the release version of a program.• Coverage-guided fuzzing (cover-guided fuzzing) uses code coverage information as feedback during fuzzing to detect whether input caused new code paths or blocks to execute.• Generator-based fuzzing (generation-based fuzzing) uses prior knowledge of the target program (fuzz target) to generate test inputs. An example of such prior knowledge is a grammar that corresponds to the input specification of the fuzz target, i.e., the input grammar of the fuzz target (i.e., the program under test).• Static instrumentation is the insertion of instructions into a program (to be tested) to obtain feedback about execution. It is usually realized by the compiler and can specify, for example, the code blocks reached during execution.• Dynamic instrumentation is the control of execution of a program (to be tested) during runtime to generate feedback from the execution. It is usually implemented by operating system system functionalities or by the use of emulators.• A suggestor is an apparatus or program that can control a target device or program and provide functions, e.g., to fetch register or memory values and pause and execute the target program in single steps.• A breakpoint is set via a suggestor to an instruction of the target program or device to stop execution upon reaching and inform the controlling process.• A (data) watchpoint is set to a memory address of a target program or device via a debugger to stop execution when the memory address is accessed and inform the controlling process of triggering an interrupt.Embedded systems typically include a microcontroller that processes inputs and responds with outputs to perform a particular task. Although microcontrollers use the same memory model and are programmed with the same programming languages as normal user programs, their programs are much more difficult to test. In order to enable debugging, microcontrollers generally offer the possibility of interrupting the program with breakpoints (breakpoints), passing through the instructions of the program in individual steps and setting watchdog points to memory addresses. Watchdog points trigger an interrupt when the corresponding memory areas are accessed. Hardware breakpoints and watchdog points are typically implemented as physical registers in the debug unit of a microcontroller and their number is therefore limited and depends on the particular system. For example, the maximum number for a typical microcontroller is four breakpoints and two data watchdog points. Usually, watchdog points can distinguish between read and write accesses.Breakpoints and watchdog points can be used in particular for the realization of a debug-controlled fuzzing, so that it does not require instrumentation.Fuzzing, also debug-driven fuzz, is very efficient in finding errors that trigger observable behavior, such as crash or restart. However, entire classes of errors cannot be observed because the program fails silently in them. An example is the Curable Bug. The core of the learned bug was that it only reads beyond the boundary of an array, while a write operation would have caused an easily observed segmentation error.The heartbed bug was found only with the aid of the Address Sanitizer (ASan). ASan inserts additional instructions, metadata, and checks during program compilation to prevent errors in memory damage. If such Sanitizer instructions are available in a program, more errors can be found in debugging the program than without Sanitizer. In particular, automated tests, such as fuzzing, shine when a sanitizer is provided in the program to be tested (i.e. in the fuzz target) in order to reveal additional errors.For embedded systems, such as an ARM architecture computing device, such sanitizers are not as easy to use as for standard platforms, such as x86 platforms. There are several reasons for this:• An embedded system is too resource constrained to implement a Sanitizer. For example, Asan needs twice the memory, MSan (MemorySan) needs 2.5 times the resources, and UBSan (UndefinedBehaviorSanitizer) needs even three times the working memory of the program.• Sanitizer increase the size of the compiled binary file. In the automotive industry, the size of such bins typically closely matches the available flash memory of the target hardware. Additional instrumentation of a Sanitizer would therefore not fit into flash memory.• Due to the additional instrumentation of Sanitizers and the collection and tracking of metadata, the use of a Sanitizer results in a slower runtime of a binary program on the respective hardware. Embedded systems are highly dependent on asynchronous events such as interrupts, and therefore, reinitators may result in time-based false positive errors, i.e., a reinitator may introduce new errors during runtime.• Embedded systems typically do not have a user interface for displaying runtime errors. For example, on x86 systems, a segmentation error is passed to STD ERR so that the user sees the crash. Embedded systems, on the other hand, fail silently, i.e., without the user noticing it, and restart after such a crash.According to various embodiments, therefore, an approach is provided that enables the use of a memory monitor (i.e. a Sanger functionality) for an embedded system, in particular such that the memory monitor may be used for debug-controlled fuzz. In this case, the memory monitoring itself is made possible with the aid of a debugger (or the debugger used for fuzzing).In debug-based fuzzing, interactions between the system performing the test (e.g., corresponding to the computer 100) and the target system (target hardware, e.g., an embedded system, e.g., a target device 106) occur via a debug link (i.e., debug interface) provided, for example, by a dedicated debug hardware device. The test input data are transmitted to the target system 106 in the form of an input vector, for example via WiFi or a CAN bus (depending on the type of the target device 106), i.e. the communication network 107 is such a debug connection during this testing (in the case of distributing the tested software, the communication network can then be any other communication network). The system that performs the test, also referred to as test system 100 below, controls the execution of the target program (i.e., the program to be tested) in the target system via the debug link, i.e., starts the execution and resumes execution after an interrupt (in particular, an interrupt triggered by a data watchpoint).A debug-controlled sanger does not require instrumentation or emulation, but only requires a debug interface to the target system (e.g., an embedded system on which the software executes) with the ability to set breakpoints and watchdog points. This type of debug interfaces and capabilities are generic and widely available, resulting in broad and easy applicability of the approach described below. In addition, the storage of the target system is only lightly loaded, e.g., for metadata, as most or all of the Sanger-related information is collected and stored on the host side of the debuggger (i.e., in the testing system 100), so that the embedded system can also be tested in its final version (e.g., as sold). The size of the target program's compiled binary is not increased because it can be used as intended for the target system 106 for deployment in testing.A debugger stops the target system when a breakpoint is reached. Therefore, the approach described below only rarely results in time-based false alarms. These false alarms can also be excluded by other test techniques, for example by subsequent validation of a fault found on the target system. The use of a debugger also provides a good view into the internals of a target system.The approach described below is used for detecting suspended pointers (Dangling pointers) with a debugger. A hanging pointer is a pointer that points to invalid (e.g., no longer valid) data. An example of this is the following program code. The pointer dp becomes suspended when the memory area pointed to is released. Thus, any pointer reference after the memory enable command (free in this example) is called can result in unwanted behavior, in particular a data leak.The problem of a hanging pointer can be avoided by setting the pointer to zero after the memory enable command: However, this must be programmed into the computer program by the programmer.According to various embodiments, therefore, it is provided to detect (by means of a debugger that does not need to run on the executing system) such deficiencies (i.e. program code, especially a memory release instruction that leads to a hanging pointer, e.g. based on the binary file of the computer program). In the event of such a detection, the pointer indicating the memory area to be enabled by the memory enable command is set to zero (or else to another value which initiates or provokes an exception (exception). Another parameter may also be set to such a value, e.g. a variable containing an address which also points to or in the memory area to be released (e.g. a pointer derived from the original pointer).As described above, it is assumed that the test system 106 is connected to the executing (e.g., embedded) system 106 by means of a debug link and tests the execution of the computer program to be tested thereon by means of a debug switch.For example, test system 100 performs: 1. setting breakpoints to a memory enable command, e.g., free or a function that calls free. In order to select those memory release commands of the memory release commands present in the computer program to which breakpoints are set, different strategies can be used (e.g. random or list-wise, which is executed in a plurality of fuzzing iterations, i.e. such that each memory release command has been provided with a breakpoint, for example, in at least one fuzzing iteration, i.e. in order to test all memory release commands, the same test case can be carried out for different placements of breakpoints if not enough breakpoints are available for all memory release commands). 2. in executing the computer program, when one of the set breakpoints is triggered, i.e., a memory enable miss is executed (e.g., a free is invoked), the pointer (or even a variable derived from the pointer) is set to a value that triggers an exception (such as zero or another particular bit pattern to which the executing system responds by an exception.) This triggers an exception when the pointer is reused for referencing, such as a null pointer exception, when the pointer has been overwritten with null, so that its use is visible after the memory enable command.In summary, according to various embodiments, a method is provided as illustrated in FIG. 2.FIG. 2 shows a flow diagram 200 illustrating a method for testing a computer program according to an embodiment.At 201, one or more breakpoints are set to one or more memory enable instructions in the computer program.In 202, the computer program is executed, wherein in 203, upon triggering one of the set breakpoints, the pointer that the memory release miss receives as an indication of the memory area to be released is set to a value (e.g., zero) that triggers an exception upon a subsequent use of the pointer in the computer program.The exception can then be responded to, for example, by indicating that the computer program contains an error.The method of FIG. 2 may be performed by one or more computers having one or more computing devices. The term "data processing unit" may be understood as any type of entity that enables the processing of data or signals. The data or signals may be treated, for example, according to at least one (i.e., one or more than one) specific function performed by the data processing unit. A data processing unit may include or be formed from an analog circuit, a digital circuit, a logic circuit, a microprocessor, a microcontroller, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a programmable gate array integrated circuit (FPGA), or any combination thereof. Any other way of implementing the respective functions described in more detail herein may also be understood as a data processing unit or logic circuitry. One or more of the method steps described in detail herein may be executed (e.g., implemented) by a data processing unit through one or more special functions performed by the data processing unit.The approach of FIG. 2 is for testing a program, such as control software for a robotic device. The term "robotic device" may be understood to refer to any technical system, such as a computer controlled machine, a vehicle, a household appliance, a power tool, a manufacturing machine, a personal assistant, or an access control system. The control software can also be used for data processing systems such as a navigation device.The method of FIG. 2 is performed, for example, by a test arrangement (e.g., computer 100 and target device 106 of FIG. 1 ).Although specific embodiments have been illustrated and described herein, it will be appreciated by those skilled in the art that the specific embodiments shown and described may be substituted for a variety of alternative and / or equivalent implementations without departing from the scope of the present invention. This application is intended to cover any adaptations or variations of the particular embodiments discussed herein. Therefore, it is intended that this invention be limited only by the claims and the equivalents thereof.

Claims

A method of testing a computer program (105), comprising: setting one or more breakpoints to one or more memory enable instructions in the computer program (105); executing the computer program (105); and upon triggering one of the set breakpoints, setting a pointer that the memory enable miss receives as an indication of the memory area to be released to a value that triggers an exception upon subsequent use of the pointer in the computer program (105).The method of claim 1, comprising determining to which values the system on which the computer program is executed responds with an exception and setting the value to which the pointer is set to one of the determined values.The method of claim 1 or 2, comprising performing multiple test runs and setting breakpoints to, from test run to test run, other memory enable commands.Method according to one of Claims 1 to 3, comprising executing the computer program (105) on an embedded system (106) and carrying out the setting of the breakpoints and the setting of the pointer which the memory release fault receives as an indication of the memory area to be released to a value which triggers an exception on a subsequent use of the pointer in the computer program (105), by a test system (100) connected to the embedded system (106).Method according to one of Claims 1 to 4, comprising, when one of the breakpoints which are set is triggered, checking whether a variable contains an address of a memory cell of the memory area which is to be released and, if appropriate, setting the variable to the value or another value which triggers an exception when the pointer is used in the computer program subsequently.Method according to one of Claims 1 to 5, wherein the computer program (105) is a control program for a robot device and the robot device is controlled on the basis of a result of testing the computer program (105) using the computer program (105).Test arrangement which is configured to carry out a method according to one of Claims 1 to 6.A computer program comprising instructions which, when executed by a processor, cause the processor to perform a method according to any one of claims 1 to 6.A computer readable medium storing instructions which, when executed by a processor, cause the processor to perform a method according to any one of claims 1 to 6.

Images