Method for testing computer program
By using the debugger to set observation points to monitor memory release instructions on the embedded system, the problems of large consumption of disinfector resources and undetected errors in embedded systems are solved, and the effectiveness of memory monitoring and error detection is achieved.
Patent Information
- Application Number
- CN202410687059.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-05-31
- Filing Date
- 2024-05-30
- Publication Date
- 2024-12-03
AI Technical Summary
Existing disinfectants for desktop and server hardware are not effectively used in embedded systems because they consume a lot of resources, may affect the execution of computer programs, and potential memory errors such as hard-working vulnerabilities in embedded systems cannot be detected.
Memory monitoring is implemented by using a debugger on an embedded system, setting observation points to monitor memory release instructions, skipping memory release instructions, and indicating errors when the observation points are triggered, avoiding resource consumption and potential errors not being discovered.
It realizes effective memory monitoring on embedded systems, can detect potential memory errors, avoid the problems of resource consumption and errors not being discovered, is suitable for fuzzy testing, and does not require detection or emulators.
Smart Images

Figure CN119065956A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to a method for testing a computer program. Background Art
[0002] An important component of software application development is testing and, if errors are found, the corresponding error correction. In particular, the aim is to identify and correct errors that lead to application failures. An important aspect of this is to carry out tests to ensure that important memory areas cannot be accessed accidentally (or by attackers), i.e. to carry out tests with memory monitoring, such as by so-called (memory) sanitizers. Compiling and testing software on common desktop and server hardware, such as x86, with the help of various sanitizers is a measure that can discover previously long-undiscovered errors such as the Heartbleed bug.
[0003] Especially for computer programs on embedded systems, such as control units for vehicles, which are usually safety-related, a comprehensive test is important, which also includes memory monitoring. However, for such systems, sanitizers for desktop and server hardware cannot be used or can only be used poorly, because embedded systems usually have limited resources, while such sanitizers have huge resource requirements and therefore cannot be used or can even affect the execution of the computer program, so that errors occur for the first time or go undetected.
[0004] Therefore, what is desirable is a method for testing computer programs that enables memory monitoring and is suitable for embedded systems. Summary of the invention
[0005] According to various embodiments, a method for testing a computer program is described, the method comprising: executing the computer program until a memory release instruction is called for a memory area previously allocated by a memory allocation instruction of the computer program; skipping the memory release instruction, and for each memory location in one or more memory locations of the memory area to be released by the memory release instruction, setting a corresponding observation point to the memory location of the memory area; and for each set observation point, if the set observation point is triggered, it indicates that the computer program has an error.
[0006] This can be performed for each memory deallocation instruction or at least a plurality of memory deallocation instructions that appear in the computer program (e.g. depending on how many watchpoints are available). These watchpoints can be read watchpoints, write watchpoints, or watchpoints that are triggered both when reading and when writing.
[0007] The above method can be implemented with the aid of a debugger to test using memory monitoring (i.e., using a sanitizer) on accesses to freed memory areas on an embedded system. This is particularly suitable for testing using fuzz testing, because fuzz testing can also be implemented in a debugger-controlled manner and can be effectively used on embedded systems in this way.
[0008] Sanitizers can be implemented with the help of code instrumentation. However, for this purpose either the source code must be available or command set-specific instrumentation based on binaries (binary instrumentation) is required, which is very vulnerable. An alternative, emulator-based instrumentation is also very platform-specific, and each embedded platform requires its own emulator. The above approach enables testing with a sanitizer controlled by a debugger and does not require instrumentation or emulation, and can therefore be applied in many cases.
[0009] Various embodiments are described below.
[0010] Embodiment 1 is a method for testing a computer program as described above.
[0011] Embodiment 2 is a method according to embodiment 1, wherein the computer program is executed until a memory allocation instruction is called; information about which memory area is allocated by the memory allocation instruction is stored; and one or more memory locations to which the observation point is set are determined based on the stored information.
[0012] In other words, in the case of a memory allocation instruction, the execution can be stopped (by means of the setting of a breakpoint) and information about the corresponding allocated memory area can be stored.
[0013] Embodiment 3 is a method according to embodiment 1 or 2, which comprises: releasing the memory area and removing one or more observation points based on the running time of the computer program (for example, since the arrival of the memory release instruction) (for example, when exceeding a specified threshold in the form of absolute time (for example, milliseconds) or the number of clock cycles).
[0014] This prevents the memory area from being permanently blocked (ie, no longer able to be reallocated), which could lead to problems for longer programs or such programs with a large memory requirement.
[0015] Embodiment 4 is a method according to any one of embodiments 1 to 3, the method comprising: executing the computer program on an embedded system, and setting an observation point through a test system connected to the embedded system.
[0016] According to various embodiments, a computer program for an embedded system may be tested, in particular on the embedded system itself, including memory monitoring.
[0017] Embodiment 5 is a method according to any one of Embodiments 1 to 4, wherein the computer program is a control program for a robot apparatus, and the robot apparatus is controlled using the computer program based on a result of a test on the computer program.
[0018] Embodiment 6 is a testing device configured to perform the method according to any one of embodiments 1 to 5.
[0019] Embodiment 7 is a computer program having instructions, which, when executed by a processor, cause the processor to execute the method according to any one of embodiments 1 to 5.
[0020] Embodiment 8 is a computer-readable medium storing instructions, which, when executed by a processor, cause the processor to execute the method according to any one of embodiments 1 to 5. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] In the accompanying drawings, like reference numerals generally refer to the same parts in all the different views. These drawings are not necessarily to scale, with emphasis instead generally being placed on presenting the principles of the invention. In the description below, various aspects are described with reference to the following drawings.
[0022] Figure 1 A computer for developing and / or testing software applications is shown.
[0023] Figure 2 A flow chart is shown which presents a method for testing a computer program according to an embodiment.
[0024] The detailed description below refers to the accompanying drawings, which show, for the purpose of illustration, the specific details and aspects of the present disclosure in which the present invention may be implemented. Other aspects may be used and structural changes, logical changes, and electrical changes may be performed without departing from the scope of protection of the present invention. Different aspects of the present disclosure are not necessarily mutually exclusive, as some aspects of the present disclosure may be combined with one or more other aspects of the present disclosure to form new aspects. DETAILED DESCRIPTION
[0025] Various examples are described in more detail below.
[0026] Figure 1 A computer 100 is shown for use in developing and / or testing software applications.
[0027] The computer 100 has a CPU (Central Processing Unit) 101 and a working memory (RAM) 102. The working memory 102 is used to load a program code, for example, from a hard disk 103, and the CPU 101 executes the program code.
[0028] In the present example, it is assumed that the user intends to use computer 100 to develop and / or test a software application.
[0029] To this end, the user runs the software development environment 104 on the CPU 101 .
[0030] The software development environment 104 enables users to develop and test applications 105 for various devices 106, i.e. target hardware, such as embedded systems for controlling robotic devices including robot arms and autonomous vehicles, or also for mobile (communication) devices. For this purpose, the CPU 101 can run an emulator as part of the software development environment 104 in order to simulate the behavior of the corresponding device 106 for which the application is being or has been developed. If the software development environment is only used to test software from other sources, the software development environment 104 can also be regarded as or designed as a software testing environment.
[0031] The user can distribute the finished application to the corresponding devices 106 via the communication network 107. Instead of via the communication network 107, this can also be done in another way, for example by means of a USB memory stick.
[0032] Before doing so, however, the user should test the application 105 to avoid distributing an application to the device 106 that does not function properly.
[0033] The testing method is so-called fuzz testing. Fuzz or fuzz testing is an automated software testing method in which invalid, unexpected or random data is supplied as input to the computer program to be tested. The program is then monitored for anomalies such as crashes, missing failed built-in code assertions or potential memory leaks.
[0034] Typically, a fuzzer (i.e., a test program using fuzz testing) is used to test a program that processes structured inputs. The structure is specified, for example, in a file format or file format or protocol, and distinguishes between valid and invalid inputs. An effective fuzzer produces semi-valid inputs that are "valid enough" so as not to be rejected outright by the input parser of the program under test, but "invalid enough" to reveal unexpected behavior and critical cases that are not properly handled in the program under test.
[0035] In the following, terms used in connection with fuzz testing are described:
[0036] Fuzz or fuzz testing is an automated testing process that sends randomly generated inputs to a target program (program under test) and observes the response of the target program.
[0037] A fuzzer or fuzz testing engine is a program that automatically generates inputs. Therefore, it is not associated with the software under test and does not perform any instrumentation. However, it is able to instrument the code, generate test cases, and execute the program under test. Known examples are afl and libfuzzer.
[0038] A fuzz target is a software program or function to be tested by fuzz testing. A main feature of a fuzz target should be that it accepts potentially untrusted inputs generated by the fuzzer during the fuzz testing process.
[0039] A fuzz test is a combined version of a fuzzer and a fuzz target. The fuzz target can then be instrumented code where the fuzzer is associated with its inputs (i.e., provided with those inputs). The fuzz test can be executed. The fuzzer can also start, observe, and stop multiple fuzz tests (typically hundreds or thousands per second), each with slightly different inputs generated by the fuzzer.
[0040] A test case is a specific input and a specific test run from a fuzz test. Usually, the runs that are interesting for reproducibility (finding new code paths or crashes) are stored. In this way, a specific test case with the corresponding input can also be executed on a fuzz target that is not connected to the fuzzer, such as a released version of the program.
[0041] During fuzz testing, coverage-guided fuzzing uses code coverage information as feedback to identify whether an input causes the execution of new code paths or blocks.
[0042] Generation-based fuzzing uses prior knowledge about the target program (fuzz target) in order to create test inputs. An example of such prior knowledge is a grammar corresponding to the input specification of the fuzz target, i.e., the input grammar of the fuzz target (i.e., the program to be tested).
[0043] Static instrumentation is the insertion of commands into a program (to be tested) in order to obtain feedback about the execution. Static instrumentation is usually implemented by a compiler and can, for example, indicate code blocks reached during the execution.
[0044] Dynamic instrumentation is controlling the execution of a program (to be tested) at runtime in order to generate feedback from the execution. This dynamic instrumentation is usually implemented through system functions of the operating system or by using an emulator.
[0045] A debugger is a device or program that can control a target device or a target program and can provide functions, such as for recalling register or memory values and for pausing and executing the target program in single steps.
[0046] Breakpoints are set by the debugger at instructions of the target program or device so that when they are reached, execution is stopped and the controlling process is notified.
[0047] A (data) watchpoint is set by the debugger to a memory address of the target program or target device so that when the memory address is accessed the execution is stopped and the controlling process is notified of this by triggering an interrupt.
[0048] Embedded systems usually have a microcontroller that processes inputs and responds with outputs in order to complete a specific task. Although microcontrollers use the same memory model and are programmed with the same programming language as regular user programs, microcontroller programs are much more difficult to test. In order to implement debugging, microcontrollers usually provide the option of interrupting the program using breakpoints, stepping through the commands of the program, and setting watchpoints on memory addresses. A watchpoint triggers an interrupt when the corresponding memory area is accessed. Hardware breakpoints and watchpoints are usually implemented as physical registers in the debug unit of the microcontroller, and their number is therefore limited and depends on the corresponding system. For example, the maximum number for a typical microcontroller is four breakpoints and two data watchpoints. Usually, a watchpoint can distinguish between read access and write access.
[0049] Breakpoints and watchpoints can be particularly useful for implementing debugger-controlled fuzz testing, so that no instrumentation is required.
[0050] Fuzz testing, also known as debugger-controlled fuzz testing, is very efficient in finding bugs that trigger observable behavior, such as crashes or reboots. However, the entire class of bugs cannot be observed, because the program silently fails at them. An example is the Heartbleed-Bug. The essence of the Heartbleed-Bug is that it only reads out of array bounds, while writes cause segmentation faults that are easily observable.
[0051] The Heartbleed vulnerability was only found with the help of Address Sanitizer (ASan). ASan inserts additional commands, metadata and checks during program compilation to prevent memory corruption errors. If such sanitizer commands are available in a program, more errors can be found when debugging the program than without the sanitizer. In particular, when the sanitizer is available in the program under test (i.e. in the fuzz target), automated testing, such as fuzz testing, comes into play to find additional errors.
[0052] For embedded systems, such as data processing devices with ARM architecture, such sanitizers are not as easy to use as for standard platforms, such as x86 platforms. There are several reasons for this:
[0053] The resources of embedded systems are too limited to implement sanitizers. For example, Asan requires twice the memory, MSan (MemorySanitizer) requires 2.5 times the resources, and UBSan (UndefinedBehaviorSanitizer) even requires three times the working memory of the program.
[0054] Sanitizers increase the size of compiled binaries. In the automotive industry, the size of such binaries is often close to the available flash memory corresponding to the target hardware. Therefore, the additional checks of the sanitizer will not fit in flash memory.
[0055] The use of sanitizers may result in slower runtimes of binaries on the corresponding hardware due to the sanitizer's additional instrumentation and metadata collection and tracking. Embedded systems rely heavily on asynchronous events, such as interrupts, and thus sanitizers may result in event-based false positives, i.e., sanitizers may introduce new errors at runtime.
[0056] Embedded systems usually do not have a user interface for displaying runtime errors. For example, on x86 systems, segmentation faults are forwarded to STDERR, making the user see the crash. Embedded systems fail silently, i.e. the user does not notice, and reboot after such a crash.
[0057] Therefore, according to various embodiments, a method is provided that enables the use of memory monitoring (i.e., sanitizer functionality) for embedded systems, in particular, enables the use of the memory monitoring for debugger-controlled fuzz testing. Here, the memory monitoring itself can be implemented with the aid of a debugger (or a debugger used for fuzz testing).
[0058] In debugger-based fuzz testing, interaction is performed between a system performing the test (and corresponding to, for example, a computer 100) and a target system (target hardware, such as an embedded system, such as a target device 106) via a debug connection (i.e., a debug interface), which is provided, for example, by a dedicated debugger hardware device. Test input data is transmitted to the target system 106 in the form of input vectors, for example, via WiFi or a CAN bus (depending on the type of the target device 106), i.e., the communication network 107 is such a debug connection in this test (then, when distributing the tested software, the communication network can be any other communication network). The system performing the test, also referred to below as the test system 100, controls the execution of the target program (i.e., the program to be tested) in the target system via the debug connection, i.e., starts the execution and resumes the execution after an interruption (especially an interruption triggered by a data observation point).
[0059] A sanitizer controlled by a debugger does not require any detection or simulation, but only a debug interface to the target system (e.g. an embedded system on which the software is executed) to be able to set breakpoints and watchpoints. Debug interfaces and debugging capabilities of this type are generic and widely available, which results in a wide and easy applicability of the method described below. Furthermore, the memory of the target system is subjected to only a small load, e.g. for metadata, since most or all sanitizer-related information is collected and stored on the host side of the debugger (i.e. in the test system 100), so that the embedded system can also be tested in its finished version (e.g. the version that is sold). The size of the compiled binary file of the target program is not increased, since the binary file can be used during testing as it is provided for use by the target system 106.
[0060] When a breakpoint is reached, the debugger stops the target system. Thus, the method described below only leads to time-based false alarms in rare cases. These false alarms can also be eliminated by other testing techniques, such as by subsequently verifying the errors found on the target system. The use of a debugger also provides a good understanding of the internals of the target system.
[0061] The method described below is used to detect access to a freed memory area (ie, a dynamic buffer in memory).
[0062] At this point, it should be noted that identity-based checking tools may invalidate the identity of a freed object. Subsequently, the access checking mechanism can identify stale (i.e. "dangling") pointer references as long as the freed memory or identity is not reused. However, if the corresponding program reuses the memory or identity for a new allocation, this approach can erroneously allow stale pointer references. Tools based on reuse delays reduce the probability of such identification errors by delaying the reuse of memory areas or identities until they are "stale". This results in a trade-off between accuracy and memory overhead, as longer reuse delays result in higher memory overhead, but also in a higher probability of identifying stale pointer references.
[0063] According to various embodiments, such an aging mechanism is used in conjunction with (and, for example, adapted to) a debugger.
[0064] For example, the test system 100 performs the following operations:
[0065] 1. Set breakpoints on memory allocation instructions such as malloc, realloc, and calloc
[0066] 2. When one of the memory allocation instructions is called, an interrupt is triggered by the corresponding set breakpoint. In response, the test system 100 stores the base address (i.e., the starting address of the corresponding memory area to be allocated) and the size of the memory area to be allocated as allocation metadata.
[0067] 3. Set breakpoints on memory release instructions such as free
[0068] 4. The triggering of an interrupt in response to one of the breakpoints set on the memory deallocation instruction
[0069] a. Stores the pointer to the memory release instruction (i.e. the beginning of the memory area that the memory release function should release), as release metadata
[0070] b. Set one or more observation points to the memory location of the memory area that the memory release function should release (ideally, if enough observation points are available, the observation points are set to all memory locations of the memory area that the memory release function should release); the size of the memory area can be determined, for example, based on allocation metadata, and these allocation metadata can be assigned to the memory release instruction based on the memory address (i.e., release metadata).
[0071] c. without releasing the memory area that the memory release function should release (i.e., finally skipping the release instruction), continuing the program (i.e., returning from the interrupt routine of the triggered interrupt)
[0072] 5. When one of these watchpoints is triggered, this is considered an error (accessing freed memory). Thus, in this case, the test system 100 indicates that an error has occurred, which in turn can trigger the execution of safety measures.
[0073] 6. If the memory area to be released by the skipped release instruction has aged (this can, for example, be configured as a specific runtime, which is specified as the execution time of the execution device, i.e., for example, the corresponding target system 106 (for example, in milliseconds) or specified as a specific number of clock signals of the execution device), one / more observation points set to the memory area are removed, and the memory release instruction that releases the memory area (and is skipped) is executed.
[0074] Since the number of breakpoints and watchpoints may be limited, it may not be possible to fully monitor all released (or to be released) memory regions in this way. If this is the case, the testing system (e.g., a fuzzer for testing the computer program 105) can randomly select a subset of memory regions and their parts (i.e., memory locations) to be monitored, or these memory regions can be monitored one by one in multiple passes of the target program (e.g., fuzz testing passes).
[0075] Here, a memory location is understood to be a memory unit monitored by a watchpoint, ie, for example, a memory area assigned to a memory address that can be selected as a target of a watchpoint.
[0076] In summary, according to various embodiments, there is provided a method for Figure 2 The method shown in .
[0077] Figure 2 A flow chart 200 is shown which presents a method for testing a computer program according to an embodiment.
[0078] At 201 , a computer program (to be tested) is executed until a memory release instruction is called for a memory area previously allocated by a memory allocation instruction of the computer program.
[0079] At 202 , the memory deallocation instruction is skipped, and for each memory location in the one or more memory locations of the memory region to be deallocated by the memory deallocation instruction, a corresponding watchpoint is set to the memory location of the memory region.
[0080] At 203 , for each set observation point, if (ie, in response to) the set observation point is triggered, it is indicated that the computer program has an error.
[0081] Figure 2The method can be performed by one or more computers with one or more data processing units. The term "data processing unit" can be understood as any type of entity capable of processing data or signals. For example, these data or signals can be processed according to at least one (that is, one or more than one) specific function, which is performed by a data processing unit. The data processing unit may include an analog circuit, a digital circuit, a logic circuit, a microprocessor, a microcontroller, a central unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an integrated circuit of a programmable gate array (FPGA), or any combination thereof or be constructed by these. Any other means for implementing the corresponding functions described in more detail herein may also be understood as a data processing unit or a logic circuit device. One or more of the method steps described in detail here can be implemented (e.g., realized) by a data processing unit through one or more specific functions, which are performed by the data processing unit.
[0082] Figure 2 The method is used for testing programs, for example control software for a robotic device. The term "robotic device" may be understood to refer to any technical system, such as a computer-controlled machine, vehicle, household appliance, power tool, manufacturing machine, personal assistant or access control system. The control software may also be used for data processing systems such as navigation equipment.
[0083] For example, by means of a test device (e.g. Figure 1 The computer 100 and the target device 106) are executed Figure 2 method.
[0084] Although specific embodiments are presented and described herein, it is recognized by those skilled in the art that the specific embodiments shown and described may be replaced with a variety of alternative and / or equivalent implementations without departing from the scope of protection of the present invention. The present application should encompass any adjustments or changes to the specific embodiments discussed herein. It is therefore intended that the present invention be limited only by the claims and their equivalents.
Claims
1. A method for testing a computer program (105), the method comprising: Executing the computer program (105) until a memory release instruction is called for a memory area previously allocated by a memory allocation instruction of the computer program (105); skipping the memory release instruction and, for each memory location in the one or more memory locations of the memory region to be released by the memory release instruction, setting a corresponding watch point to the memory location of the memory region; and For each set observation point, if the set observation point is triggered, it indicates that the computer program (105) has an error.
2. The method according to claim 1, wherein: Executing the computer program (105) until the memory allocation instruction is called; storing information about which memory region is allocated by the memory allocation instruction; And determining one or more memory locations to set the watchpoint to based on the stored information.
3. A method according to claim 1 or 2, wherein the method comprises: releasing the memory area and removing one or more observation points based on the running time of the computer program (for example, since the arrival of the memory release instruction) (for example, when a specified threshold in the form of absolute time (for example, milliseconds) or the number of clock cycles is exceeded).
4. The method according to any one of claims 1 to 3, comprising: executing the computer program (105) on an embedded system (106), and setting the observation point by a test system (100) connected to the embedded system (106).
5. The method according to any one of claims 1 to 4, wherein: The computer program (105) is a control program for a robot device, and the robot device is controlled using the computer program (105) according to a result of a test on the computer program (105). 6 . A testing device, which is configured to carry out the method according to claim 1 . 7 . A computer program comprising instructions which, when executed by a processor, cause the processor to execute the method according to claim 1 .
8. A computer-readable medium storing instructions, which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 5.
