Server-oriented persistent memory data management method

Abstract

The invention provides a server-oriented persistent memory data management method, which is characterized by comprising the following steps of: 1, remotely accessing persistent memory data, abstracting a persistent memory access area into a persistent object and exposing the persistent object to an application program; step 2, constructing a quickly indexed hash table for the persistent object in the DRAM; 3, dividing the persistent memory into a data node pool, a log entry pool, an object entry pool and a log content pool; and 4, performing a high-concurrency garbage collection strategy based on the persistent object. By integrating the RDMA technology and the persistent memory technology, an ingenious data structure and management method are designed, and efficient remote data access performance can be provided.

Description

Technical Field

[0001] The present invention belongs to the field of system software, especially the field of system software based on a hybrid memory architecture, and specifically relates to related technologies such as memory management of new non-volatile memory, hybrid heterogeneous memory management, and remote access. Background Art

[0002] Persistent memory PM (Persistent Memory) has become a hot topic in academia and business due to its excellent characteristics such as non-volatility, byte-addressability, low memory access latency, and high storage density.

[0003] Combining dynamic random access memory DRAM (Dynamic Random Access Memory) and PM to build a hybrid memory architecture has once been a research hotspot in academia. J.Y. Jung et al. explored in "Memorage: emerging persistent ram based malleable main memory and storage architecture. Proceedings of the 27th international ACM conference on International conference on supercomputing. ACM, 2013, pp. 115–126" that PM is used as a memory medium and accessed into the computer system in the way of a memory bus to build a hybrid memory system with DRAM, constructing a flat memory architecture. This architecture can make the best use of the excellent characteristics of non-volatile storage devices and provide a broader design space for the research and development of system software and application software. Based on this architecture, many scholars have constructed persistent memory systems and file systems on PM. For example, H. Volos et al. proposed a persistent memory system based on non-volatile memory technology in "Mnemosyne: Lightweight Persistent Memory. ACM SIGARCH Computer Architecture News, vol. 39, no. 1. ACM, 2011, pp. 91–104", which can persistently store structured data on PM, and applications can access PM like accessing DRAM.

[0004] In addition, the Remote Direct Memory Access (RDMA) technology can bypass the participation of the operating system and directly access remote memory, providing high-bandwidth and low-latency access performance for remote data access. If these two technologies are combined and a hybrid memory architecture is considered to be built on a remote server, with the help of RDMA technology, the PM can enable persistent data to be directly accessed remotely. This solution provides a new idea for remotely accessing persistent data.

[0005] The present invention aims to make the above idea feasible by involving a persistent memory data management method for servers. Firstly, through the semantic abstraction of persistent objects, the application of persistent memory can be generalized and applied to various scenarios such as persistent memory variables and persistent memory file systems. Secondly, by constructing a hash table in DRAM, dividing the persistent memory into four functional regions, and constructing an entry scale table and other technical means, good access performance such as high concurrency and low latency can be provided for remote persistent memory data.

[0006] After retrieval, no description or report of similar technologies to the present invention has been found, and no similar materials at home and abroad have been collected. Summary of the Invention

[0007] Aiming at the deficiencies in the prior art, the purpose of the present invention is to provide a general new persistent memory data management method for the server side based on a hybrid memory architecture. Through the semantic abstraction of persistent objects, this method provides an efficient method for remote applications to directly access persistent data. Combining various data structures such as hash tables, linear tables, and linked lists, this method enables applications to directly and concurrently access remote persistent data.

[0008] The persistent memory data management method for servers provided by the present invention includes:

[0009] Step 1: For the remote access of persistent memory data, abstract the persistent memory access area as a persistent object and expose it to the application program;

[0010] Step 2: Build a hash table for quickly indexing persistent objects in DRAM, implemented using the Optimistic Concurrent Cukoo Hashing (OCCH) structure to support concurrent execution of multiple read operations. After power failure and restart, the hash table can be quickly reconstructed by traversing the nodes;

[0011] Step 3: Divide the persistent memory into four parts: a data node pool, a log entry pool, an object entry pool, and a log content pool;

[0012] Step 4: A garbage collection strategy based on the high concurrency of persistent objects.

[0013] Preferably, persistent objects are maintained in the form of key-value pairs, and applications or remote programs can access persistent data through a unique "key". This abstraction is compatible with the abstract semantics of various different persistent memories and storage-class memories, such as: persistent memory regions, persistent variables, persistent memory files, etc.

[0014] Preferably, in step 2, the hash table in DRAM is implemented using the OCCH (Optimistic Concurrent Cukoo Hashing) structure to support concurrent execution of multiple read operations. After power-off and restart, the hash table can be quickly reconstructed by traversing the nodes.

[0015] Preferably, in step 3, the data node pool is used to maintain the metadata of persistent object nodes, mainly including the unique "key" value, creation time, and a pointer to a log entry;

[0016] The log entry pool is used to maintain log entries. Each log entry represents a modification to a persistent object, mainly including a valid bit, a checksum, and a pointer to a linked list containing multiple data entries;

[0017] The data entry pool is used to maintain data entries. Each data entry stores the metadata of each log data, mainly including the size of the data block, the offset of the data block in the persistent memory region, and a pointer to the persistent memory region storing the data block;

[0018] The log content pool is used to maintain all log data. Each item of log data stores the modification content for each persistent object;

[0019] The allocation and recycling of nodes in the four regions of the persistent memory are all implemented through a memory allocator built into the DRAM. The memory allocator can be quickly reconstructed or restored by using representation bits.

[0020] Preferably, in step 4, the garbage collection policy can be based on memory objects as the unit granularity to implement garbage collection. According to the concurrent access of memory objects, garbage collection can also be executed concurrently. The garbage collection policy mainly includes two items: log cleaning and log compression.

[0021] Log compression mainly merges the log content according to the offset of the data entries, modifies the metadata, and sets the old discarded nodes to invalid; log cleaning mainly recovers those entries and log content nodes that are set to invalid.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] 1) The present invention can provide efficient remote data access performance. The byte-addressable persistent memory can effectively utilize the RDMA technology to directly access persistent data; the implementation methods such as the hash table of the OCCH structure and the concurrent read and write can improve the data access performance through concurrent processing.

[0024] 2) The persistent object abstraction of the present invention provides a general interface. There have been disagreements on the semantic processing of persistent memory. This method can support multiple semantic abstractions such as variables and files. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 Diagram for direct access to remote persistent data;

[0026] Figure 2 Diagram for flat hybrid memory architecture;

[0027] Figure 3 Diagram for the organizational structure of persistent content;

[0028] Figure 4 Diagram for persistent data write operation;

[0029] Figure 5 Diagram for persistent data read operation. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0030] In order to make the objectives and technical solutions of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. This embodiment is implemented on the premise of the technical solution of the present invention, and detailed implementation manners and specific operation processes are given. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several deformations and improvements can still be made, and these all belong to the protection scope of the present invention.

[0031] This embodiment provides a persistent memory data management method for servers, as Figure 1 shown. Based on the flat hardware memory system architecture, as Figure 2 shown, in which the DRAM is mainly used to cache frequently accessed data, such as storing the hash index of the persistent memory. The PM is mainly used to store infrequently accessed data and data that needs to be persisted. The specific steps are as follows:

[0032] Step 1: The method exposes interface functions to the application program through persistent object abstraction to facilitate the access to remote persistent data. The application program can specify a unique persistent object ID for the persistent object, such as Per_obj001, to identify its persistent object. When the application program performs read and write access later, it also locates through this ID;

[0033] Step 2: The method constructs a hash table in DRAM for position indexing of object nodes of persistent objects. Here, the OCCH (Optimistic Concurrent Cukoo Hashing) structure is used to implement it, which supports concurrent execution of multiple read operations. After power failure and restart, the hash table can be quickly reconstructed by traversing the nodes. For example, for the persistent object Per_obj001 above, by calculating HASH(Per_obj001), a storage node is obtained to store the metadata of the persistent object;

[0034] Step 3: The persistent memory is divided into four parts: a data node pool, a log entry pool, an object entry pool, and a log content pool. As Figure 3 , the data node pool is used to maintain the metadata of persistent object nodes, mainly including a unique "key" value, a creation time, and a pointer to a log entry; the log entry pool is used to maintain log entries, and each log entry represents a modification to a persistent object, mainly including a valid bit, a checksum, and a pointer to a linked list containing multiple data entries; the data entry pool is used to maintain data entries, and each data entry stores the metadata of each log data, mainly including the size of the data block, the offset of the data block in the persistent memory area, and a pointer to the persistent memory area storing the data block; the log content pool is used to maintain all log data, and each log data stores the modification content for each persistent object;

[0035] Preferably, the allocation and recycling of nodes in the four regions of the persistent memory are all implemented by a memory allocator built into DRAM, and the memory allocator can be quickly reconstructed or restored through identification;

[0036] Preferably, each node is connected by a singly linked list. As Figure 3 shown, for example, when a client sends a write access request for Per_obj001 to the server, the server first retrieves the data node of Per_obj001 by retrieving the hash table. Then, from the log entry pool and the data entry pool, a log entry node and a data entry are obtained, and a log content node of a specified length is obtained from the log content pool. First, the log entry is inserted behind the data node, and the other nodes are linked as Figure 4 shown, and then its various data items are filled. Among them, the valid bit of the log entry is set to in access, and after the verification of the checksum is correct after the write is completed, it is modified to valid, and if the verification fails, it is set to invalid. Based on the byte-addressable characteristic of the persistent memory, the client can complete direct reading and writing of data through the RDMA technology. Since each write operation operates on a different log content node, the client can implement parallel operations on remote data.

[0037] Preferably, when a client sends a read access request for Per_obj001 to the server, the server first retrieves the data node of Per_obj001 by searching the hash table, then obtains the first log node (the latest) from the linked list of this data node, and retrieves the response data block from the subsequent linked list according to the offset and data length provided by the client to complete the data read operation, as Figure 5 shown;

[0038] Step 4: The garbage collection policy can use memory objects as the unit granularity to implement garbage collection. According to the concurrent access of memory objects, garbage collection can also be executed concurrently. The garbage collection policy mainly includes two items: log cleaning and log compression.

[0039] Preferably, log cleaning mainly reclaims those invalid nodes. The cleaning program scans all data nodes, traverses the linked list of each data node, and reclaims all log nodes with invalid valid bits in the linked list and all nodes in the subsequent linked list.

[0040] Preferably, log compression merges the log content according to the time of log entries and the offset of data entries, modifies the metadata, and sets the old discarded nodes to invalid.

Claims

1. A persistent memory data management method for a server, characterized in that: include: Step 1: Remotely access persistent memory data and abstract the persistent memory access area into a persistent object and expose it to the application. Step 2: Build a fast index hash table for persistent objects in DRAM; Step 3: Divide the persistent memory into four parts: data node pool, log entry pool, object entry pool, and log content pool; Step 4: High concurrency garbage collection strategy based on persistent objects.

2. The server-oriented persistent memory data management method according to claim 1, characterized in that Persistent objects are maintained in the form of key-value pairs. Applications or remote programs can access persistent data through a unique "key". This abstraction is compatible with a variety of different abstract semantics of persistent memory and storage-level memory, such as persistent memory areas, persistent variables, persistent memory files, etc.

3. The server-oriented persistent memory data management method according to claim 1, characterized in that: In step 2, the hash table in the DRAM is implemented using the OCCH (Optimistic Concurrent Cukoo Hashing) structure to support concurrent execution of multiple read operations. After power failure and restart, the hash table can be quickly reconstructed by traversing the nodes.

4. The server-oriented persistent memory data management method according to claim 1, characterized in that: In step 3, the data node pool is used to maintain the metadata of the persistent object node, which mainly includes a unique "key" value, a creation time, and a pointer to a log entry. The log entry pool is used to maintain log entries. Each log entry represents a modification to a persistent object, mainly including a valid bit, a checksum, and a linked list pointing to multiple data entries. The data entry pool is used to maintain data entries. Each data entry stores the metadata of each log data, mainly including the size of the data block, the offset of the data block in the persistent memory area, and a pointer to the persistent memory area where the data block is stored. The log content pool is used to maintain all log data. Each log data stores the modification content of each persistent object. The allocation and deallocation of nodes in the four regions of persistent memory are implemented through a memory allocator built into DRAM, which can be quickly reconstructed or restored by using representation bits.

5. The server-oriented persistent memory data management method according to claim 1, characterized in that: In step 4, the garbage collection strategy can implement garbage collection with memory objects as the unit granularity. According to the concurrent access of memory objects, garbage collection can also be performed concurrently. The garbage collection strategy mainly includes two contents: log cleaning and log compression; Log compression mainly merges log contents according to the offset of data entries, modifies metadata, and sets old obsolete nodes to invalid; log cleaning mainly recycles entries and log content nodes that are set to invalid.

Images