Method for realizing high-availability distributed memory pool based on erasure codes

Abstract

The invention discloses a high-availability distributed memory pool system based on erasure codes and a method thereof, which are designed for coping with errors such as node faults and page loss, are suitable for general computing scenes deployed on a large scale, and ensure high availability of applications such as big data machine learning, and the method does not need kernel mode code modification, and runs in a user mode through a computing node (QEMU). Physical and remote memories are managed in a unified manner, and memory management of a cache line level is realized. A memory page is divided into K data blocks and P verification blocks by using an erasure code technology, and the K data blocks and the P verification blocks are stored in different remote memory nodes in a scattered manner. In order to reduce the influence of erasure codes on the read-write performance of the memory, a background backup mechanism based on monitoring is introduced: a memory monitor periodically scans a memory page and verifies a node state, reports a fault to a computing node once the fault is found, and triggers a fault processing flow. And the stability and the reliability of the distributed memory system are effectively improved.

Description

Technical Field

[0001] The present invention relates to the fields of computer virtualization, memory systems, and distributed system architectures. Specifically, the present invention proposes an erasure code disaster recovery solution for user-mode memory pools, providing an erasure code-based backup and error recovery strategy for user-mode memory pools at the virtual machine granularity, that is, virtual machine remote memory, improving the fault tolerance of virtual machine remote memory, making it difficult for errors in some remote memories to cause the overall system to crash, thus meeting the application scenarios of highly available distributed memory pools. Background Art

[0002] A distributed memory pool is a technology that integrates the physical memory resources of multiple nodes into a logically unified memory space through a network. Each logical computing node can utilize the physical memory of remote memory nodes to expand its own memory capacity. With the development of high-memory-consuming applications such as big data and machine learning, the memory resources of a single node often cannot meet the computing power requirements. Taking HPC as an example, according to the analysis of the computing power improvement from 2012 to 2019, the vertical expansion of computing power based on Moore's law can only bring about an 8-fold performance improvement, while the horizontal expansion brought by the distributed architecture can bring a 37,500-fold improvement. Among them, the distributed memory pool plays an indispensable role.

[0003] Currently, the mainstream distributed memory system is a multi-level memory architecture based on the remote Swap mechanism. This architecture divides the data center nodes into memory nodes and computing nodes. The computing nodes swap out cold pages (i.e., pages with low access frequency) to remote memory nodes through the remote Swap mechanism, and at the same time swap in the memory nodes on the remote nodes into the local memory. Although there are already many distributed memory systems, most of these systems rely on changes at the kernel or hardware level and are difficult to achieve the goal of large-scale deployment. At the same time, the existing implementation schemes usually do not consider the situation of node failures. Once a node in the distributed memory pool fails, it is very likely to have a serious impact on the stability and availability of the entire memory pool. As the number of memory nodes increases, the probability of multiple nodes failing simultaneously will increase exponentially. Therefore, the disaster recovery capabilities of these systems are generally weak. Therefore, optimizing the disaster recovery mechanism of the distributed memory pool and improving the stability and reliability of the system are key issues that need to be solved urgently. Summary of the Invention

[0004] In view of the above deficiencies of the prior art, the present invention provides a highly available distributed memory pool method based on erasure codes, implementing an error recovery mechanism for node failures, page losses, etc. under multiple memory nodes, so that the distributed memory system can meet the large-scale deployment in general computing scenarios and at the same time have an effective fault recovery mechanism to ensure the high availability of large-scale applications such as big data machine learning.

[0005] The present invention mainly designs and implements two subsystems: a distributed memory pool and an erasure code backup. Specifically as follows:

[0006] For the distributed memory pool subsystem, aiming at the problem that mainstream distributed memory systems need to modify the operating system kernel or hardware, the present invention designs a user-state distributed memory pool system that does not require modifying the underlying kernel and hardware and is unaware to upper-layer applications. At the same time, different from the existing memory pool systems applied to physical nodes, this system is deployed at the granularity of virtual machine instances, so it is more in line with the application deployment form commonly used in data centers.

[0007] For the erasure code backup subsystem, the present invention adopts an erasure code encoding backup form of memory page partitioning and performs data recovery at the granularity of memory pages. A single memory page will be equally divided into K parts at the time points of reading, writing, and regular backup. P parity blocks are generated by erasure code encoding, and the K+P data blocks and parity blocks are stored on different nodes respectively. Through the protection of erasure code, node failures can only cause the loss of one or a few data blocks or parity blocks, and the remaining blocks can be decoded by erasure code to recover the original data. Therefore, the erasure code backup subsystem provides an error recovery mechanism for page errors, losses, and even single-node to multi-node failures, ensuring the high availability of the distributed memory pool subsystem.

[0008] In the selection of erasure code encoding algorithms, since memory encoding needs to perform overall one-time encoding on data and then distribute the data blocks to multiple nodes to successfully recover the original data during the decoding stage, the present invention adopts an erasure code with MDS (Minimum Distance Separable) characteristics. MDS encoding can ensure that the original data can be recovered when the number of lost data blocks is less than the number of parity blocks, while non-MDS type erasure codes require more redundant blocks to ensure data recoverability. The present invention adopts the Reed-Solomon code in MDS codes to obtain the highest data redundancy efficiency and optimal erasure performance.

[0009] The present invention designs a distributed memory architecture with strong generality and easy for large-scale deployment for a highly available general distributed memory pool. At the same time, based on this distributed memory system, erasure code backup is performed on remote memory pages to achieve the high availability of the multi-node distributed memory pool, thereby ensuring the correct operation of large-scale memory applications such as graph computing and machine learning in the distributed memory pool system.

[0010] The object of the present invention is achieved through the following technical solutions:

[0011] A highly available distributed memory pool system based on erasure code, characterized by including:

[0012] A memory node initialization module for initializing at least one memory node. Each memory node is a virtual machine instance of a specific template, has an independent network IP address, and joins a distributed memory pool;

[0013] A computing node virtual machine instance creation module that, when creating a virtual machine instance of a computing node, uses the userfaultfd function of Linux to forward page fault requests for subsequent physical memory pages to a specific user-space function for processing. The user-space function implements the management of a cache system composed of local memory;

[0014] A remote memory allocation module that, according to the size of the memory space requested by the user, selects and allocates a memory node with remaining memory space in the memory pool as a remote memory node for the computing node;

[0015] A cache management and memory page swapping module for managing the local memory cache during the operation of the computing node, and swapping out local memory pages to a remote memory node or swapping in the required memory pages from a remote memory node when needed.

[0016] The cache management and memory page swapping module includes:

[0017] The local memory is used as the last-level cache of the cache system, and a multi-way set-associative cache management strategy is adopted;

[0018] When the computing node accesses a memory page placed on a remote memory node, it triggers a local cache miss, requests the corresponding page from the remote memory node, and performs a cache line swap-in.

[0019] It also includes an erasure code backup module. When writing a page to a remote memory, the page is divided into K data blocks, and P parity blocks are generated by encoding using the erasure code module; the K + P backup blocks are stored on independent memory nodes respectively, and the frame number of the memory page and the numbers of the K + P memory nodes are recorded on the computing node; when the computing node requests a memory page, the backup blocks are obtained from the corresponding memory nodes according to the recorded numbers, and the original data blocks are restored by decoding using the erasure code module.

[0020] It also includes a monitoring-based background erasure code backup mechanism for periodically scanning memory nodes to detect memory loss, errors, or faults; when a memory node failure is detected, the erasure code module is called in the background thread of the computing node for page recovery.

[0021] The monitoring-based background erasure code backup mechanism also includes a remote memory monitor for periodically scanning all memory nodes connected to the computing node and verifying the alive state of the nodes by the ping method.

[0022] A method using the above-mentioned distributed memory pool system, characterized by including the following steps:

[0023] S1. Initialize memory nodes and add them to the distributed memory pool;

[0024] S2. Create a virtual machine instance of the computing node and configure the page fault handling function;

[0025] S3. Allocate remote memory nodes for the computing node according to the user request;

[0026] S4. Manage the local memory cache during the operation of the computing node and perform the swap-in and swap-out operations of memory pages.

[0027] The specific steps of step S4 include:

[0028] S4.1 Use the local memory as the last-level cache and adopt the multi-way set-associative cache management strategy;

[0029] S4.2 When a local cache miss occurs, request and swap in the required memory page from the remote memory node.

[0030] It further includes: S5. Execute the erasure code backup process, including page partitioning, encoding, storage, and recovery; S6. Use the monitoring-based background erasure code backup mechanism to detect and handle memory node failures.

[0031] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0032] It solves the problem of the performance degradation of the page read critical path caused by using erasure codes for error page recovery. Through the monitoring mechanism and background recovery operations, the performance isolation between the erasure code module and the remote memory is achieved, thereby reducing the probability of triggering data recovery during page reading. Description of the Drawings

[0033] Figure 1 . Architecture schematic diagram

[0034] Figure 2 . Erasure code backup schematic diagram

[0035] Figure 3 . Monitoring-based background erasure code backup mechanism Specific implementation method

[0037] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings. These embodiments are implemented on the premise of the technical solution architecture of the present invention, and detailed implementation methods and specific operation processes are given. However, the applicable platforms are not limited to the following examples. A specific deployment example is a cluster composed of two servers. Each server runs a computing node and a memory node respectively, and the servers are connected through a local area network. The present invention is not limited by the types and numbers of servers and can be extended to clusters of more than two servers.

[0038] This embodiment proposes a distributed memory pool method. By combining two subsystems, namely the distributed memory pool and the erasure code backup, an error recovery mechanism is implemented in the case of node failures, page losses, etc. under multiple memory nodes. This mechanism enables the distributed memory system to meet the large-scale deployment requirements in general computing scenarios and ensure the high availability of large-scale applications such as big data and machine learning.

[0039] A general-purpose distributed memory pool solution includes the initialization of the memory pool system, which involves the division and initialization of computing nodes and memory nodes, as well as the page fault and page replacement processes of the memory pool system.

[0040] The process of initializing the general-purpose distributed memory pool is as follows:

[0041] Step 1: The virtual machine management system initializes some memory nodes and adds them to the distributed memory pool. Each memory node is a virtual machine instance of a specific template and has an independent network IP.

[0042] Step 2: When creating a virtual machine instance of the computing node, the Linux userfaultfd function is used to forward the page fault requests of subsequent physical memory pages to a specific user-mode function for processing. This page fault handling function implements the management (swap-in and swap-out) of the cache system composed of local memory.

[0043] Step 3: The user sends a computing virtual machine creation command to the virtual machine management terminal, including the CPU, memory, and IO resource requirements of the virtual machine instance.

[0044] Step 4: After the computing node is set up and starts running, the user sends a remote memory allocation instruction to the management terminal. The memory pool system selects K memory nodes with remaining memory space in the memory pool for the user according to the requested memory space size and allocates these nodes to the computing node as remote memory nodes.

[0045] Step 5: The computing node sends an ACK request packet to the memory node to establish a connection. At the same time, a multi-way set-associative cache system is created using local memory, and the eviction of cache lines is redirected to the page fault handling function of the operating system.

[0046] Step 6: During the operation of the computing node, if the local capacity memory is exceeded, local memory cache line eviction is triggered, and the cache management system uses the corresponding replacement algorithm to swap out the local memory page to the remote memory node.

[0047] Step 7: During the operation of the computing node, if a memory page placed in the remote memory node is accessed, a local cache miss is triggered, a request for the corresponding page is sent to the remote memory node, and the corresponding cache line is swapped out.

[0048] So far, the distributed memory system is created and uses the remote memory as the extended memory. The distributed memory system has the following characteristics:

[0049] It is compatible with the general Linux kernel, runs on the general hardware architecture, and does not require upper-layer applications to be modified. The function is implemented in the virtual machine monitor software, ensuring the maximum generality.

[0050] Use the local memory as the last level cache of the cache system, use the multi-way set-associative cache management strategy, regard the remote memory as the program memory, greatly simplify the system design, and reduce the introduction of additional abstraction layers.

[0051] The erasure code backup process of the distributed memory system is as follows:

[0052] Step 1: When writing a page to the remote memory, the agent of the memory pool system divides the page into K equally sized blocks, called data blocks.

[0053] Step 2: The agent encodes the K data blocks using the erasure code module to generate P parity blocks.

[0054] Step 3: Store the K + P backup blocks (data blocks + parity blocks) on independent memory nodes respectively, and use a hash table on the computing node to record the page frame number of the memory page and the numbers of the K + P memory nodes.

[0055] Step 4: When the computing node requests a memory page from the remote memory node, the agent queries the record of the corresponding page frame number in the hash table and returns the obtained list of memory nodes.

[0056] Step 5: The agent requests the backup block data from the remote memory nodes in the list and terminates waiting for the data after a given time.

[0057] Step 6: Use the obtained backup blocks, call the erasure code module to decode to obtain the original data blocks, and recombine the data blocks and store them in the local memory page.

[0058] A background erasure code backup mechanism based on monitoring, as shown in the appendix Figure 3 is as follows:

[0059] Step 1: During the operation of the compute node program, the memory monitor on the memory node will periodically scan the memory pages stored in the node.

[0060] Step 2: The remote memory monitor on the compute node will periodically scan all the memory nodes it is connected to and verify the alive status of the nodes by ping.

[0061] Step 3: When the remote memory node detects memory loss, error or failure, it will report the error to the compute node to which the corresponding memory page belongs.

[0062] Step 4: When the compute node finds that a certain memory node is unreachable, it determines that the memory node has failed.

[0063] Step 5: Both Step 3 and Step 4 will cause the compute node to initiate the page recovery process and call the erasure code module in the background thread to obtain the corresponding page by decoding.

[0064] Embodiment: Each server is equipped with Ubuntu Server 16.04.1 LTS 64bit, and is equipped with an Intel(R) Xeon(R) E5-2620 v4 CPU and 128G of memory, and is specifically developed based on QEMU 5.2.0, supporting mainstream Linux guest operating systems, and is also compatible with other Linux host operating systems.

[0065] As Figure 1 shown, the compute node (QEMU) on the physical machine runs in the host user mode without any modification to the kernel code. The cache management system in the compute node uniformly manages the physical memory and the remote memory, treats the physical memory of the compute node as the last-level cache, and the remote memory as the physical memory logically. The compute node connects to multiple memory nodes through the cache management system and writes and stores the memory pages in the remote memory by means of cache line eviction and swapping. Multiple memory nodes and compute nodes can run on a single physical machine. Based on Figure 1 the architecture in, the compute node (QEMU) on the physical machine runs in the host user mode without any modification to the kernel code. The cache management system in the compute node uniformly manages the physical memory and the remote memory, treats the physical memory of the compute node as the last-level cache, and the remote memory as the physical memory logically. The compute node connects to multiple memory nodes through the cache management system and writes and stores the memory pages in the remote memory by means of cache line eviction and swapping. Multiple memory nodes and compute nodes can run on a single physical machine.

[0066] The generation and backup process of the erasure code is as follows Figure 2 As shown, the proxy program also divides the memory of the computing node into K original data blocks, generates P parity blocks through the erasure code module, and finally stores the total K + P data blocks in different remote memory nodes.

[0067] To reduce the performance impact of the erasure code module on the critical path of memory read and write, the present invention adopts a monitoring-based background erasure code backup mechanism, as Figure 3 shown. During the operation of the computing node program, the memory monitor on the memory node will regularly scan the memory pages stored in the node. The remote memory monitor on the computing node will regularly scan all the memory nodes it is connected to and verify the survival status of the nodes by means of ping. When the remote memory node detects memory loss, error or failure, it will report the error to the computing node to which the corresponding memory page belongs. When the computing node finds that a certain memory node is unreachable, it determines that the memory node has failed.

[0068] The preferred specific embodiments of the present invention have been described in detail above. It should be understood that those of ordinary skill in the art can make many modifications and variations based on the concept of the present invention without creative labor. Therefore, all technical solutions that can be obtained by those skilled in the art in the technical field of the present invention through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the protection scope determined by the claims.

Claims

1. A high-availability distributed memory pool system based on erasure coding, characterized in that: include: A memory node initialization module is used to initialize at least one memory node. Each memory node is a virtual machine instance of a specific template, has an independent network IP address, and is added to the distributed memory pool. The computing node virtual machine instance creation module uses the userfaultfd function of Linux to forward the subsequent physical memory page fault requests to a specific user state function for processing when creating a virtual machine instance of a computing node. The user state function implements cache system management composed of local memory. The remote memory allocation module selects and allocates a memory node with remaining memory space as a remote memory node for the computing node in the memory pool according to the memory space size requested by the user; The cache management and memory page swap-in and swap-out module is used to manage the local memory cache during the operation of the computing node, and swap out the local memory page to the remote memory node when necessary, or swap in the required memory page from the remote memory node.

2. The distributed memory pool system according to claim 1, characterized in that: The cache management and memory page swap-in and swap-out module includes: The local memory is used as the last level cache of the cache system, adopting a multi-way set associative cache management strategy; When a compute node accesses a memory page placed in a remote memory node, a local cache miss is triggered, and the corresponding page is requested from the remote memory node and a cache line swap is performed.

3. The distributed memory pool system according to claim 1 or 2, characterized in that: It also includes an erasure code backup module, which is used to divide the page into K data blocks when writing the page to the remote memory, and use the erasure code module to encode and generate P check blocks; store the K+P backup blocks in independent memory nodes respectively, and record the frame number of the memory page and the numbers of the K+P memory nodes on the computing node; When a computing node requests a memory page, it obtains the backup block from the corresponding memory node according to the recorded number, and uses the erasure code module to decode and restore the original data block.

4. The distributed memory pool system according to claim 1 or 2, characterized in that: It also includes a monitoring-based background erasure code backup mechanism, which is used to regularly scan memory nodes to detect memory loss, errors, or failures. When a memory node failure is detected, the erasure code module is called in the background thread of the computing node to perform page recovery.

5. The universal distributed memory pool system according to claim 4, characterized in that: The monitoring-based background erasure code backup mechanism also includes a remote memory monitor, which is used to periodically scan all memory nodes connected to the computing nodes and verify the survival status of the nodes by pinging.

6. A method for using the distributed memory pool system according to any one of claims 1 to 5, characterized in that: The steps include: S1. Initialize the memory node and join the distributed memory pool; S2. Create a virtual machine instance for the computing node and configure a page fault handling function; S3. Allocate remote memory nodes to computing nodes according to user requests; S4. Manage the local memory cache and perform memory page swapping operations during the operation of the computing node.

7. The method according to claim 6, characterized in that The step S4 specifically includes: S4.1 uses local memory as the last level cache and adopts a multi-way set associative cache management strategy; S4.2 requests and swaps in the required memory page from the remote memory node when a local cache miss occurs.

8. The method according to claim 6 or 7, characterized in that Also includes: S5. Execute the erasure code backup process, including page division, encoding, storage and recovery; S6. Use the monitoring-based background erasure code backup mechanism to detect and handle memory node failures.

Images