Memory access exception monitoring method based on compact memory pool

Abstract

The present invention provides a memory access exception monitoring method based on a compact memory pool, and belongs to the field of computer technology. The method of the present invention optimizes the memory pool of the memory access error detection tool ASan, and improves the memory allocation method of heap variables, including: for the heap variables of the same application, memory blocks of different sizes are allocated with the same memory pool to allocate storage addresses; each allocated memory block corresponds to a node of a red-black tree, and a red-black tree is maintained by using the memory protection zone at the back end of the memory block; the first address of the memory block can be quickly retrieved through the red-black tree, and the program exception can be found when the address range of the red-black tree node is exceeded. Under the premise of ensuring the detection performance of ASan, the method of the present invention avoids the generation of multiple fragmented Shadow Memories, and reduces the waste of memory usage in the ASan memory pool.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular relates to a memory access exception monitoring method based on a compact memory pool. Background Art

[0002] The advantages of C / C++ make it extremely attractive and indispensable in some cases: it runs at an excellent speed, uses little memory and disk space, is mature, has predictable execution, and their platform applicability is almost unparalleled. You can use this language on a platform without installing any other tools. However, unlike runtime systems such as Java and Golang, C / C++ does not have a memory management mechanism provided by the runtime system for excellent program running efficiency, and users need to manage memory manually. This leads to vulnerabilities in C / C++ programs that are different from other languages ​​(languages ​​with memory management mechanisms), namely memory safety issues.

[0003] In response to the memory safety issues of C / C++ programs, a large number of memory safety detection tools have emerged in the open source community, such as Valgrind, AddressSanitizer, HWAddressSanitizer, etc. ASan (AddressSanitizer) was open sourced by Google in 2012. Due to its excellent performance and lower additional resource consumption, it is widely used in program detection and gradually replaced Valgrind. ASan captures memory safety issues such as pointer out-of-bounds and use after release in programs through source code instrumentation and dynamic link libraries. This is mainly done through Shadow Memory, Redzone, Memory Quarantine and other auxiliary data structures.

[0004] Since all memory allocated to applications can be accessed normally, there is no mechanism that can effectively distinguish between pointer out-of-bounds and use after release. Therefore, Shadow Memory is introduced to identify whether the memory is accessible. Each memory used by an application has a corresponding Shadow Memory. When the program can access this memory, the Shadow Memory is set to a natural number. When the memory is inaccessible, it is set to a negative number. Figure 1 Shown is a schematic diagram of ShadowMemory.

[0005] The introduction of Shadow Memory categorizes the program's memory. Next, memory safety issues are attributed to illegal memory, so that Shadow Memory can be effectively distinguished. Since the application needs to use the memory as fully as possible, the two variables are closely arranged without gaps. ASan introduces a memory protection area to capture pointer out-of-bounds problems. When a memory block needs to be allocated, more memory is allocated at both ends of the memory block. The size is calculated in proportion to the memory you want to allocate, which is called a memory protection area. Set the Shadow Memory of the memory protection area to inaccessible, so that when a pointer out-of-bounds occurs, the memory protection area will be accessed and the pointer out-of-bounds can be captured.

[0006] Program variables are divided into global variables, local variables and heap variables. The present invention mainly focuses on the memory allocation problem of heap variables. The program needs to maintain a memory pool to efficiently and reasonably allocate heap variables, but due to the particularity of heap variables, the memory safety detection tool uses a memory pool similar to the Slab mechanism, but this method consumes more memory.

[0007] The existing slab-like memory pool mechanism allocates a fixed size, such as a 1MB memory pool, to each of the 30 memory blocks of 16B, 32B, ..., 65536B. This is used to allocate fixed-size memory blocks. When the fixed-size memory blocks are used up, another 1MB memory pool is allocated. Figure 2 As shown in the figure, there are two 1MB memory pools, which are used to allocate 16B and 32B memory blocks respectively. This memory pool can satisfy that any address of a memory block can be located at the first address. Each memory pool is 1MB address aligned, and k-byte memory blocks are k-byte aligned in the memory pool. Since the minimum unit of physical memory allocated by the operating system is a physical page, which is generally 4KB, and a memory pool is allocated using a slab-like mechanism, such as a 1MB memory pool to allocate memory blocks of a specific size, such as 16B. If the program only allocates 16B a few times, such as 5 times, then 4KB-5*16B of memory are wasted. Not only that, the discreteness of memory blocks also leads to the discreteness of Shadow Memory, which also leads to the growth of ShadowMemory memory consumption.

[0008] In the existing Slab-like memory pool mechanism, due to the existence of the memory protection zone, the first address of the memory block (chunk_begin) is different from the first address of the memory obtained by the application (user_begin). Therefore, the memory pool can convert the first address user_begin obtained by the application to the first address chunk_begin of the memory block. When an illegal address is accessed, the corresponding information of the memory block can be obtained based on this address. Therefore, the memory pool should satisfy that any address can locate the memory block (not just the first address). Figure 3 As shown in FIG. 1 , a pseudo code for obtaining the first memory address of a Slab-like memory pool mechanism is shown. In the figure, chunk_size indicates the size of a memory block, and pool_begin indicates the starting address of the allocated memory pool. Summary of the invention

[0009] In order to solve the memory allocation problem of heap variables mentioned above, a memory pool similar to the Slab mechanism is used to allocate memory. When the memory block of a specific size is small and the number of allocations is small, there is a problem of large memory consumption. The present invention provides a memory access exception monitoring method based on a compact memory pool to solve this problem in the memory allocation of heap variables in ASan.

[0010] The present invention provides a memory access exception monitoring method based on a compact memory pool, which optimizes the memory pool of a memory access error detection tool ASan and improves the memory allocation method of heap variables. The method of the present invention comprises:

[0011] (1) For heap variables of the same application, memory blocks of different sizes are allocated storage addresses using the same memory pool;

[0012] (2) Each allocated memory block corresponds to a node of the red-black tree, and a red-black tree is maintained using the memory protection zone at the back end of the memory block; the node information is stored in the memory protection zone at the back end of the memory block; the node information includes the starting address of the memory block and the left and right nodes of the node.

[0013] The address range of the memory block represented by the node in the red-black tree is [the starting address of the memory block, the address of the back-end memory protection area].

[0014] (3) When memory needs to be released, the red-black tree is searched according to the first address of the memory to be released, user_begin, to find the corresponding node and determine whether the first address of the memory is within the address range corresponding to the node. If so, it means that the memory to be released belongs to the memory pool corresponding to the red-black tree. The node is deleted from the red-black tree and the memory is returned. If not, it means that the memory to be released does not belong to the memory pool corresponding to the red-black tree. At this time, a bug occurs in the feedback program.

[0015] The advantages and positive effects of the present invention are:

[0016] (1) By using the method of the present invention, all memory blocks are allocated using one memory pool. It is not necessary to apply for a unique memory pool for each size of memory block for allocation. This avoids the situation where a memory pool for allocating a fixed size of k is allocated only once and then not allocated again, thereby reducing memory waste.

[0017] (2) The existing solution has multiple memory pools, which will lead to multiple fragmented Shadow Memories. However, the method of the present invention only has one memory pool, and the corresponding Shadow Memories are also concentrated in one piece, so naturally the Shadow Memory consumes less memory.

[0018] (3) The method of the present invention maintains a red-black tree in the memory protection zone at the back end of the memory block. On the one hand, it can quickly retrieve the first address of the memory block. On the other hand, program anomalies can be discovered through the address range of the red-black tree nodes. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a schematic diagram of shadow mapped memory;

[0020] Figure 2 It is a schematic diagram of the existing Slab-like memory pool;

[0021] Figure 3 It is a schematic diagram of the algorithm for obtaining the first memory address of the existing Slab-like memory pool mechanism;

[0022] Figure 4 is an example diagram of a compact memory pool used by the method of the present invention;

[0023] Figure 5 It is a schematic diagram of the method of the present invention using the memory protection zone to maintain the red-black tree;

[0024] Figure 6 It is a comparison diagram of the use of the memory allocation mechanism of the present invention and the existing Slab-like memory allocation mechanism;

[0025] Figure 7 It is a comparison diagram of the memory release mechanism between the method of the present invention and the existing Slab-like method. DETAILED DESCRIPTION

[0026] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0027] The present invention proposes a memory access exception monitoring method based on a compact memory pool, which adopts a compact memory pool allocation scheme, that is, memory blocks of different sizes of the same application heap variable are allocated using one memory pool, such as Figure 4As shown, the use of this memory pool loses the unique advantage of the slab mechanism memory pool that the first address of the memory block can be obtained through a constant. Then, it is necessary to solve the problem of how to obtain the first address of this memory pool. Since each time a memory block is allocated, an additional memory protection zone is allocated at both ends to capture pointer crossings, some meta information of the memory block is saved in the front-end memory protection zone, such as: allocation stack, release stack, memory block size, etc. The memory protection zone at the back end is not fully utilized. Therefore, the present invention makes full use of the memory protection zone at the back end to maintain an efficient index structure, and adopts a red-black tree to solve the problem of obtaining the first address.

[0028] like Figure 5 As shown, the present invention uses the back-end memory protection zone to maintain a red-black tree. When a memory block is to be allocated, a node in the red-black tree will be generated accordingly. Each memory block stores the node information of the red-black tree in the back-end memory protection zone. The node information includes the starting address of the memory block, and the ending address is the address of the back-end memory protection zone. In this way, this red-black tree can meet two functions:

[0029] (1) Determine whether a memory is allocated by the memory pool;

[0030] (2) Determine the starting address chunk_begin of the memory block.

[0031] In the embodiment of the present invention, the data structure of the red-black tree node information is defined as follows:

[0032]

[0033] Among them, Node is a pointer data, left and right represent the left and right subtree nodes respectively, and addr represents the starting address of the memory block chunk_begin. The address range represented by the red-black tree node object is [addr, &object], and &object represents the address of the backend memory protection zone corresponding to the node object.

[0034] Storing the above data structure in the memory protection area at the back end of the memory block enables efficient retrieval of the memory block address.

[0035] like Figure 6 As shown, ① is a slab-like memory allocation mechanism, which takes a memory block from a memory pool of corresponding size and returns it to the application. ② is the compact memory pool mechanism of the present invention, which first takes a memory block of appropriate size from a large memory pool, and then, according to the content recorded in the back-end memory protection zone, inserts the memory block as a node into the red-black tree, and finally returns it to the application.

[0036] For example, when allocating a memory block of size k, first align k to 2 n B size, calculate the size of the memory protection zone at the left and right ends according to the memory block size, then record the starting address of the memory block in the back-end memory protection zone, and record the left and right subtree node information after the node is inserted into the red-black tree.

[0037] like Figure 7 As shown, ① is the Slab-like memory release mechanism. Due to the particularity of the memory pool, that is, each memory pool only allocates memory blocks of one size, the first address of the memory block can be determined according to the block size. Figure 7 ② is the compact memory pool memory release mechanism of the present invention, firstly, the memory block where the address is located is retrieved from the red-black tree, and then the first address of the memory block recorded in the node is determined, which is the first address to be determined, and then the red-black tree is adjusted to return the memory.

[0038] In the embodiment of the present invention, when releasing the memory, it is actually released according to the first address of the memory, as follows:

[0039] When allocating memory, void*ptr=malloc(sizeof(int));

[0040] When releasing memory, free(ptr); / / Use the pointer returned by applying for memory to release it.

[0041] Among them, malloc is used to apply for memory space, and ptr represents the first address of the memory to be released.

[0042] If the memory block to be released is a memory block requested from the heap, it will be determined at a node in the red-black tree whether the memory block to be released is allocated from the memory pool based on the starting address addr of the node retrieved from the red-black tree and the address &object of the backend memory protection zone corresponding to the node. If ptr≥addr&&ptr<&object is satisfied, it proves that the memory block is allocated from the memory pool, and the first address of the memory block is addr. Then delete the node from the red-black tree. At this time, the red-black tree may be unbalanced, and the nodes of the red-black tree are adjusted according to the normal red-black tree algorithm. If a bug occurs in the program and a memory block that does not belong to the memory pool is released, such as a global variable, ptr is not within the address range corresponding to the node. By first judging before the memory is released that the released memory is allocated by the memory pool, this program fault can be captured.

[0043] Through the above implementation process, the present invention optimizes the memory pool of the memory access error detection tool ASan. ASan captures pointer out-of-bounds by adding a memory protection zone. The memory protection zone refers to adding a certain size of memory before and after the variable. In this way, when a pointer out-of-bounds occurs, the memory protection zone is accessed, and the pointer out-of-bounds is captured. However, with the memory protection zone, the situation of user_begin! = chunk_begin appears. Therefore, the original memory pool adopts a Slab-like memory pool. This memory pool can realize the conversion of user_begin into chunk_begin in constant time, but the negative impact brought about is the increase in memory consumption. Therefore, the method of the present invention proposes to use a compact memory pool instead of a Slab-like memory pool, and proposes to use the right memory protection zone to establish a red-black tree to realize user_begin to find chunk_begin with a time complexity of O(logn). And it is found through actual testing that the performance of ASan using the method of the present invention for memory access error detection has not decreased. Therefore, the use of the method of the present invention reduces the situation of fragmented Shadow Memory and reduces the waste of memory.

[0044] Except for the technical features described in the specification, all of them are known technologies to those skilled in the art. The present invention omits the description of known components and known technologies to avoid redundancy and unnecessary limitation of the present invention. The implementation methods described in the above embodiments do not represent all implementation methods consistent with the present application. On the basis of the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative labor are still within the protection scope of the present invention.

Claims

1. A memory access anomaly monitoring method based on a compact memory pool, using ASan for memory access error detection, It is characterized in that The method optimizes the ASan memory pool, comprising the following steps: (1) For heap variables of the same application, memory blocks of different sizes are allocated from the same memory pool; (2) Each allocated memory block corresponds to a node of the red-black tree, and a red-black tree is maintained using the memory protection zone at the back end of the memory block; the node information is stored in the memory protection zone at the back end of each memory block, and the node information includes the starting address of the memory block and the left and right subtree nodes of the node; The address range of the memory block represented by the node in the red-black tree is [the starting address of the memory block, the address of the back-end memory protection zone]; (3) When the memory needs to be released, the red-black tree is searched according to the first address of the memory to be released, user_begin, to find the corresponding node, and to determine whether the first address of the memory is within the address range corresponding to the node. If so, it means that the memory to be released belongs to the memory pool corresponding to the red-black tree. The node is deleted from the red-black tree and the memory is returned. If not, it means that the memory to be released does not belong to the memory pool corresponding to the red-black tree, and the feedback program fails at this time.

2. The method according to claim 1, It is characterized in that In the step (2), the data structure of the node information includes the following: The left subtree node *left of the node and the right subtree node *right of the node are both pointer type data; The starting address addr of the memory block.

Images