Method for testing computer program

Abstract

According to various embodiments, a method for testing a computer program is described, the method comprising: executing the computer program until a storage release instruction is invoked for a storage area previously allocated by a storage allocation instruction of the computer program; for each of one or more storage positions of the storage area released by the storage release instruction, setting a corresponding monitoring point on the storage position of the storage area, and executing the storage release instruction; for each set monitoring point, if the set monitoring point is triggered, displaying that the computer program has an error; and for each of the set monitoring points, removing the monitoring point if the storage location to which the monitoring point is set is reallocated by another storage allocation instruction in the computer program.

Description

Technical Field

[0001] The present disclosure relates to a method for testing a computer program. Background Art

[0002] An important part of developing software applications is testing and the corresponding error correction when errors are found. In particular, errors that lead to application failures should be identified and corrected. An important aspect here is testing with the view that important memory areas are not accessed accidentally (or by an attacker), i.e. using memory monitoring such as that performed by so-called (memory) sanitizers. Compiling and testing software on common laptop and server hardware such as x86 with the help of various sanitizers is a measure that can discover errors such as the Heartbleed bug that previously remained undiscovered for a long time.

[0003] Comprehensive testing that also includes such memory monitoring is particularly important for computer programs on often safety-relevant embedded systems, such as control devices for vehicles. However, sanitizers for laptop and server hardware cannot be used or can only be used poorly for such systems, since embedded systems typically have restricted resources and such sanitizers have a huge resource requirement and therefore cannot be used or can even affect the execution of the computer program so that errors occur in the first place or remain undetected.

[0004] Therefore, a method for testing a computer program that can implement memory monitoring and is applicable to embedded systems is desirable. Summary of the invention

[0005] According to various embodiments, a method for testing a computer program is provided, the method comprising: executing the computer program until a storage release instruction is called for a storage area previously allocated by a storage allocation instruction of the computer program; for each of one or more storage locations of the storage area released by the storage release instruction, setting a corresponding monitoring point to the storage location of the storage area, and executing the storage release instruction; for each set monitoring point, if the set monitoring point is triggered, displaying that the computer program has an error; and for each of the set monitoring points, if the storage location to which the monitoring point is set is reallocated by another storage allocation instruction in the computer program, removing the monitoring point.

[0006] This may be performed for each or at least a plurality of storage release instructions occurring in the computer program (eg depending on how many watchpoints are available). A watchpoint may be a read watchpoint, a write watchpoint, or a watchpoint that is triggered on both read and write.

[0007] The method described above enables testing with memory monitoring (i.e., with a sanitizer) with the aid of a debugger with respect to accesses to released memory areas on an embedded system. This is particularly applicable when testing with fuzz testing, since fuzz testing can also be implemented in a debugger-controlled manner and can be used effectively for embedded systems in this way.

[0008] Sanitizers can be implemented with the aid of code instrumentation. However, for this purpose either the source code must be available or command set-specific instrumentation based on binaries (binary instrumentation) is required, which is very susceptible. The alternative emulator-based instrumentation is also very platform-specific, and each embedded platform requires its own emulator. The method described above enables testing with a debugger-controlled sanitizer without the need for instrumentation or emulation, and can therefore be applied in many cases.

[0009] Various embodiments are described below.

[0010] Embodiment 1 is a method for testing a computer program as described above.

[0011] Embodiment 2 is a method according to embodiment 1, wherein a computer program is executed until a storage allocation instruction is called, information about which storage area is allocated by the storage allocation instruction is stored, and one or more storage locations to which the monitoring point is set are determined based on the stored information.

[0012] In other words, in the case of a memory allocation instruction, execution can be suspended (by setting a breakpoint) and information about the corresponding allocated memory area can be stored.

[0013] Embodiment 3 is the method described in Embodiment 1 or 2, including setting a breakpoint on a storage allocation instruction of a computer program, and for each of the set monitoring points, if one of the set breakpoints is triggered, determining whether to reallocate the storage location to which the monitoring point is set through another storage allocation instruction in the computer program.

[0014] Thus, when using a debugger (using breakpoints and watchpoints), it is possible to monitor whether a freed (and not yet reallocated) memory area is accessed (read and / or write access, depending on the type of watchpoint used).

[0015] Embodiment 4 is a method according to any one of embodiments 1 to 3, comprising executing a computer program on an embedded system and setting a monitoring point through a test system connected to the embedded system.

[0016] According to various embodiments, in particular, testing of a computer program for an embedded system, including memory monitoring, is enabled on the embedded system itself.

[0017] Embodiment 5 is a method according to any one of Embodiments 1 to 4, wherein the computer program is a control program for a robotic device, and the robotic device is controlled using the computer program based on a result of a test of the computer program.

[0018] Embodiment 6 is an embodiment of a testing device, which is configured to perform the method according to any one of embodiments 1 to 5.

[0019] Embodiment 7 is a computer program having instructions, which, when executed by a processor, cause the processor to perform the method according to any one of embodiments 1 to 5.

[0020] Embodiment 8 is a computer-readable medium storing instructions, and when the instructions are executed by a processor, the instructions cause the processor to perform the method according to any one of embodiments 1 to 5. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In the accompanying drawings, like reference numerals generally refer to the same parts throughout the various views. The accompanying drawings are not necessarily drawn to scale, but emphasis is generally placed on illustrating the principles of the invention instead. In the following description, various aspects are described with reference to the following drawings.

[0022] Figure 1 A computer used to develop and / or test software applications is shown.

[0023] Figure 2 A flow chart is shown which illustrates a method for testing a computer program according to one embodiment. DETAILED DESCRIPTION

[0024] The following detailed description relates to the attached drawings, which illustrate specific details and aspects of the present disclosure that can perform the present invention for the purpose of illustration. Other aspects may be used and structural, logical and electrical changes may be made without departing from the scope of protection of the present invention. Because some aspects of the present disclosure may be combined with one or more other aspects of the present disclosure to form new aspects, the various aspects of the present disclosure are not necessarily mutually exclusive.

[0025] Various examples are described in more detail below.

[0026] Figure 1 A computer 100 is shown for use in developing and / or testing software applications.

[0027] The computer 100 has a CPU (Central Processing Unit) 101 and a work memory (RAM) 102. The work memory 102 is used to load a program code from a hard disk 103, for example, and the CPU 101 executes the program code.

[0028] In the present example it is assumed that a user intends to utilize computer 100 to develop and / or test a software application.

[0029] To this end, the user executes the software development environment 104 on the CPU 101 .

[0030] The software development environment 104 enables users to develop and test applications 105 for various devices 106, i.e., for target hardware (such as embedded systems for controlling robotic devices including mechanical arms and autonomous vehicles) or for mobile (communication) devices. To this end, the CPU 101 can execute an emulator as part of the software development environment 104 to simulate the behavior of the corresponding device 106 for which the application is to be or has been developed. If the software development environment 104 is only used to test software from other sources, the software development environment 104 can also be regarded as or designed as a software testing environment.

[0031] The user can distribute the finished application to the corresponding device 106 via the communication network 107. Instead of via the communication network 107, this can also be done in other ways, for example by means of a USB stick.

[0032] Before this occurs, however, the user should test the application 105 to avoid distributing an application to the device 106 that does not function as specified.

[0033] One testing method is so-called fuzz testing. Fuzz testing (Fuzzing or Fuzz-Testing) is an automated software testing method in which invalid, unexpected or random data is fed as input to the computer program to be tested. The program is then monitored for anomalies such as crashes, lack of failed built-in code assertions or potential memory leaks.

[0034] Typically, a fuzzer (i.e., a test program using fuzz testing) is used to test a program that processes structured inputs. The structure is specified, for example, in a file format or file format or protocol and a distinction is made between valid and invalid inputs. An effective fuzzer produces semi-valid inputs that are "valid enough" to not be directly rejected by the input parser of the program under test, but "invalid enough" to reveal unexpected behavior and edge cases that are not properly handled in the program under test.

[0035] The following describes the terminology used in conjunction with fuzz testing:

[0036] Fuzz testing (Fuzzing or Fuzz-Testing) is an automated testing process that sends randomly generated inputs to a target program (the program under test) and observes its responses.

[0037] A fuzzer or fuzz testing engine is a program that automatically generates input. Therefore, the fuzzer or fuzz testing engine is not associated with the software under test and does not perform instrumentation. However, the fuzzer or fuzz testing engine has the ability to instrument code, generate test cases and execute the program under test. Known examples are afl and libfuzzer.

[0038] A fuzz target is a software program or function that should be tested by fuzz testing. The main feature of a fuzz target should be that it takes potentially untrusted inputs generated by the fuzzer during the fuzz testing process.

[0039] A fuzz test is a combined version of a fuzzer and a fuzz target. Thus, a fuzz target can be instrumented code, where the fuzzer is associated with its input (i.e., provides the input). Fuzz tests are executable. A fuzzer can also start, observe, and stop multiple fuzz tests (typically hundreds or thousands per second), each with slightly different inputs generated by the fuzzer.

[0040] A test case is a specific input and a specific test run from a fuzz test. Usually, the runs that are of interest for reproducibility (finding new code paths or crashes) are stored. In this way, a specific test case with the corresponding input can also be executed on a fuzz target that is not connected to the fuzzer, such as a released version of the program.

[0041] Coverage-guided fuzzing uses code coverage information as feedback during fuzz testing to identify whether an input causes the execution of new code paths or blocks.

[0042] Generator-based fuzz testing (English: generation-based fuzzing) uses prior knowledge about the target program (fuzz target) to create test inputs. An example of such prior knowledge is the grammar corresponding to the input specification of the fuzz target, i.e. the input grammar of the fuzz target (i.e. the program to be tested).

[0043] Static instrumentation is the insertion of commands into a program (to be tested) to obtain feedback about the execution. The static instrumentation is mostly implemented by the compiler and can, for example, describe the code blocks reached during execution.

[0044] Dynamic instrumentation is controlling the execution of a program (to be tested) during runtime to generate feedback from the execution. The dynamic instrumentation is mostly implemented by operating system functionality or by using an emulator.

[0045] A debugger is a device or a program that can control a target device or a target program and can provide functions such as for calling register values ​​or memory values ​​and for pausing and executing a target program in individual steps.

[0046] ·Breakpoints are set by the debugger on commands of the target program or device so that execution stops when they are reached and the controlling process is notified of this.

[0047] Setting a (data) watchpoint by the debugger at a memory address of the target program or target device, so that execution is halted when the memory address is accessed, and the controlling process is informed of this by triggering an interrupt.

[0048] Embedded systems usually have a microcontroller that processes inputs and responds with outputs to perform specific tasks. Although microcontrollers use the same memory model as conventional user programs and are programmed using the same programming language, microcontroller programs are significantly more difficult to test. In order to enable debugging, microcontrollers usually provide the following possibilities: interrupting the program using breakpoints (stop points), traversing the commands of the program in individual steps, and setting watchpoints on storage addresses. When accessing the corresponding storage area, the watchpoint triggers an interrupt. Hardware breakpoints and watchpoints are typically implemented as physical registers in the debugging unit of the microcontroller, and therefore their number is limited and depends on the corresponding system. For example, for a typical microcontroller, the maximum number is four breakpoints and two data watchpoints. Usually watchpoints can be distinguished between read access and write access.

[0049] In particular, breakpoints and watchpoints can be used to implement debugger-controlled fuzz testing, eliminating the need for instrumentation.

[0050] Fuzz testing (and debugger-controlled fuzz testing) is very effective at finding bugs that trigger observable behavior, such as crashes or reboots. However, not all classes of bugs can be observed because the program silently fails on them. An example is the Heartbleed vulnerability. The core of the Heartbleed vulnerability is that Heartbleed only reads out of bounds of an array, while writes result in segmentation faults that are easily observable.

[0051] The Heartbleed vulnerability was only found with the help of Address Sanitizer (ASan). ASan inserts additional commands, metadata, and checks during the compilation of a program to prevent errors when memory is corrupted. When such Sanitizer commands are available in a program, more errors can be found when debugging the program than without the Sanitizer. Automated testing (such as fuzz testing) is particularly interesting when a Sanitizer is set up in the program to be tested (i.e. in the fuzz target) to find additional errors.

[0052] For embedded systems (such as data processing devices with ARM architecture), such sanitizers are not as easy to use as for standard platforms (such as x86 platforms). There are many reasons for this:

[0053] Embedded systems are too resource-constrained to implement a Sanitizer. For example, Asan requires double the memory, MSan (MemorySanitizer) requires 2.5 times the resources, and UBSan (UndefinedBehaviorSanitizer) even requires three times the working memory of the program.

[0054] The Sanitizer increases the size of the compiled binary. In the automotive industry, the size of such binaries often almost corresponds to the available flash memory of the target hardware. Therefore, the additional instrumentation of the Sanitizer does not fit in the flash memory.

[0055] The use of Sanitizers results in slower runtime of binaries on the corresponding hardware due to the Sanitizer's additional instrumentation and metadata collection and tracking. Embedded systems strongly depend on asynchronous events, such as interrupts, and thus Sanitizers may cause timing-based false positive errors, i.e., Sanitizers may introduce new errors during runtime.

[0056] Embedded systems usually do not have a user interface for displaying runtime errors. On x86 systems, for example, segmentation faults are forwarded to STDERR so that the user sees the crash. Instead, embedded systems fail silently, i.e. without the user noticing this, and restart after such a crash.

[0057] Therefore, according to various embodiments, a method is provided that enables memory monitoring (i.e., Sanitizer functionality) to be used for embedded systems, in particular, memory monitoring can be used for debugger-controlled fuzz testing. Here, the memory monitoring itself can be implemented with the help of a debugger (or a debugger for fuzz testing).

[0058] In debugger-based fuzz testing, the interaction between the system executing the test (and corresponding to, for example, the computer 100) and the target system (target hardware, such as an embedded system, such as the target device 106) is carried out via a debug connection (i.e., a debug interface), which is provided, for example, by a dedicated debugger hardware device. The test input data is transmitted to the target system 106 in the form of an input vector, for example, via WiFi or a CAN bus (depending on the type of the target device 106), that is, the communication network 107 is such a debug connection during this test (then when distributing the tested software, the communication network can be any other communication network). The system executing the test (hereinafter also referred to as the test system 100) controls the execution of the target program (i.e., the program to be tested) in the target system via the debug connection, that is, starts the execution and resumes the execution after an interrupt (especially an interrupt triggered by a data monitoring point).

[0059] The debugger-controlled Sanitizer does not require plugging or emulation, but only requires a debug interface with the target system (e.g., an embedded system on which the software is executed) with the possibility of setting breakpoints and monitoring points. This type of debug interface and debugging capabilities are universal and available to a large extent, which leads to the wide and simple applicability of the method described below. In addition, since most or all of the information related to the Sanitizer is collected and stored on the host side of the debugger (i.e., in the test system 100), the embedded system can also be tested with its completed version (e.g., the sold version), so the memory of the target system is only slightly loaded, for example, for metadata. The size of the compiled binary file of the target program does not increase, because the binary file is used during testing as if it were set for the target system 106.

[0060] When a breakpoint is reached, the debugger halts the target system. Therefore, the approach described below leads to time-based false alarms only in rare cases. These false alarms can also be eliminated by other testing techniques, such as by subsequently verifying the errors found on the target system. Using a debugger also provides a good view of the internal affairs of the target system.

[0061] The approach described below is used to detect accesses to released memory areas, ie, for example, dynamic buffer areas in the memory.

[0062] For example, the test system 100 performs the following:

[0063] 1. Set breakpoints on storage allocation instructions such as malloc, realloc, and calloc

[0064] 2. When one of these storage allocation instructions is called, an interrupt is triggered by a correspondingly set breakpoint. In response, the test system 100 stores the base address (ie, the starting address of the corresponding storage area to be allocated) and the size of the storage area to be allocated as allocation metadata.

[0065] 3. Set breakpoints on storage release instructions such as free

[0066] 4. In response to an interrupt being triggered by one of the breakpoints set on a store release instruction

[0067] a. Store the pointer to the storage release instruction (i.e. the starting point of the storage area that the storage release function should release) as release metadata

[0068] b. Setting one or more monitoring points to the storage locations of the storage area that the storage release function should release (ideally, if there are enough monitoring points available, they are set to all storage locations of the storage area that the storage release function should release); for example, the size of the storage area can be determined based on the allocation metadata, and the allocation metadata can be assigned to the storage release instruction based on the storage address (i.e., the release metadata).

[0069] 5. When one of the monitoring points is triggered, this is evaluated as an error (access to released memory). In this case, the test system 100 therefore indicates that an error has occurred, which in turn can trigger the execution of safety measures.

[0070] 6. If the monitored storage area, i.e. the storage area containing one or more storage locations to which the monitoring points (from 4b) are set, is reallocated (this can be detected by monitoring the storage allocation function, i.e., for example by triggering a breakpoint set (in 1) at the said storage location), the monitoring point / points set to the storage area are removed.

[0071] Since the number of breakpoints and monitor points can be limited, it is possible that not all released (or to be released) memory areas can be fully monitored in this way. If this is the case, the testing system (e.g., a fuzzer for testing computer program 105) can randomly select a subset of memory areas and their parts (i.e., storage locations) that should be monitored or monitor memory areas one after the other in multiple passes of the target program (e.g., fuzz testing passes).

[0072] A storage location is understood here to be a unit of a memory that is monitored by a monitoring point, ie, for example, a storage area that is assigned to a storage address that can be selected as a target of a monitoring point.

[0073] In summary, according to various embodiments, there is provided Figure 2 The method shown in .

[0074] Figure 2 A flow chart 200 is shown which represents a method for testing a computer program according to one specific embodiment.

[0075] In 201 , the computer program (to be tested) is executed until a memory release instruction is called for a memory area previously allocated by a memory allocation instruction of the computer program.

[0076] In 202, for each of the one or more storage locations of the storage area released by the storage release instruction, a corresponding monitoring point is set to the storage location of the storage area, and the storage release instruction is executed.

[0077] In 203 , for each set monitoring point, if (ie, in response to) the set monitoring point is triggered, the computer program has an error.

[0078] In 204 , for each of the set monitoring points, if the storage location to which the monitoring point is set is reallocated by another storage allocation instruction in the computer program, the monitoring point is removed.

[0079] Figure 2The method can be performed by one or more computers having one or more data processing units. The term "data processing unit" can be understood as any type of entity that enables processing of data or signals. For example, data or signals can be processed according to at least one (i.e., one or more) specific functions performed by the data processing unit. The data processing unit may include an analog circuit, a digital circuit, a logic circuit, a microprocessor, a microcontroller, a central unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a programmable gate array integrated circuit (FPGA), or any combination thereof, or may be composed of an analog circuit, a digital circuit, a logic circuit, a microprocessor, a microcontroller, a central unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a programmable gate array integrated circuit (FPGA), or any combination thereof. Any other means for implementing the corresponding functions described in more detail herein may also be understood as a data processing unit or a logic circuit device. One or more of the method steps described in detail herein may be performed (e.g., implemented) by a data processing unit through one or more specific functions performed by the data processing unit.

[0080] Figure 2 The control software can be used in a test program in a manner such as control software for a robotic device. The term "robotic device" may be understood to refer to any technical system, such as a computer-controlled machine, a vehicle, a household appliance, a power tool, a manufacturing machine, a personal assistant or an access control system. The control software can also be used for data processing systems, such as navigation equipment.

[0081] Figure 2 The method is for example performed by a test device (e.g. Figure 1 computer 100 and target device 106) to execute.

[0082] Although specific embodiments have been shown and described herein, it is recognized by those skilled in the art that the specific embodiments shown and described may be replaced by various alternatives and / or equivalent implementations without departing from the scope of protection of the present invention. The present application should encompass any adaptation or variation of the specific embodiments discussed herein. Therefore, the present invention is intended to be limited only by the claims and their equivalents.

Claims

1. A method for testing a computer program (105), the method comprising: executing the computer program (105) until a storage release instruction is called for a storage area previously allocated by a storage allocation instruction of the computer program (105); for each of the one or more storage locations of the storage area released by the storage release instruction, setting a corresponding monitoring point to the storage location of the storage area, and executing the storage release instruction; For each set monitoring point, if the set monitoring point is triggered, it is displayed that the computer program (105) has an error; as well as For each of the set monitoring points, if the storage location to which the monitoring point is set is reallocated by another storage allocation instruction in the computer program (105), the monitoring point is removed.

2. A method according to claim 1, wherein the computer program (105) is executed until the storage allocation instruction is called, information about which storage area is allocated by the storage allocation instruction is stored, and one or more storage locations to which the monitoring point is set are determined based on the stored information.

3. A method according to claim 1 or 2, the method comprising setting breakpoints on storage allocation instructions of the computer program (105), and for each of the set monitoring points, if one of the set breakpoints is triggered, determining whether the storage location to which the monitoring point is set is reallocated by another storage allocation instruction in the computer program (105).

4. The method according to any one of claims 1 to 3, comprising executing the computer program (105) on an embedded system (106) and setting the monitoring point by a test system (100) connected to the embedded system (106).

5. The method according to any one of claims 1 to 4, wherein the computer program (105) is a control program for a robotic device, and the robotic device is controlled by using the computer program (105) based on the results of the test of the computer program (105). 6 . A testing device, which is configured to carry out the method according to claim 1 .

7. A computer program having instructions which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 5.

8. A computer-readable medium storing instructions, which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 5.

Images