Anti-caching method supporting nonvolatile memory database and storage medium

Abstract

The invention provides an anti-caching method supporting a nonvolatile memory database, which comprises the following steps of: constructing a storage hierarchical architecture comprising a nonvolatile memory layer and a solid state disk layer, and introducing a data dynamic migration mechanism for migrating cold data from the nonvolatile memory layer to the solid state disk layer; a cache management algorithm is adopted, a hash table and an LRU linked list are combined, distribution of data between a nonvolatile memory layer and a solid state disk layer is determined according to the access frequency of the data, and the data are tracked and managed by using a globally unique identifier; a space management mechanism is implemented, and idle pages are managed by using an idle page linked list in a non-volatile memory layer, so that the released non-volatile memory space is quickly reutilized. According to the method, not only is a large performance fall not generated when a large amount of data is stored in the nonvolatile database, but also a remarkable effect is achieved in the aspect of optimizing storage resources, and a new solution is provided for the field of storage of the nonvolatile memory database.

Description

Technical Field

[0001] The invention belongs to the field of information technology, and in particular relates to an anti-caching method and a storage medium supporting a non-volatile memory database. Background Art

[0002] Although traditional memory (such as DRAM) is fast, data will be lost after power failure, so it relies on disks for persistent data storage. Non-volatile memory (NVM) combines the speed and byte-addressable characteristics of memory with the persistence of hard disks to retain data in the event of power failure. Taking Intel Optane persistent memory as an example, its maximum capacity will reach TB level. In addition, it is very fast and is the non-volatile memory closest to DRAM at present, while the cost is much lower than DRAM. In different application scenarios, these two advantages of persistent memory can be fully utilized through reasonable configuration.

[0003] However, the limited nature of non-volatile memory (NVM) resources remains an issue that cannot be ignored. As the amount of data continues to grow, how to efficiently utilize limited NVM resources has become a problem that needs to be solved urgently. In addition, the persistence mechanism in traditional database systems usually relies on cumbersome I / O operations, which not only increases the complexity of the system, but also further reduces performance. Summary of the invention

[0004] In view of the above problems, the present invention proposes an anti-caching method that supports non-volatile memory databases. It aims to expand the storage capacity of non-volatile memory databases by introducing a solid-state drive (NVMe SSD) layer as an auxiliary storage medium. At the same time, by accurately identifying frequently accessed hot data and infrequently accessed cold data, and realizing dynamic migration of data between the non-volatile memory (NVM) layer and the solid-state drive (NVMe SSD) layer, it is ensured that on the basis of expanding the capacity space, the efficiency of the database itself is not greatly affected. The ultimate goal is to provide an efficient, reliable, and cost-effective storage solution to meet the needs of storage systems in the big data era.

[0005] The technical solution of the present invention is as follows:

[0006] An anti-caching method supporting a non-volatile memory database, comprising:

[0007] Constructing a storage hierarchy architecture including a non-volatile memory layer and a solid-state drive layer, wherein the non-volatile memory layer is used to store frequently accessed hot data, and the solid-state drive layer is used to store infrequently accessed cold data;

[0008] Introducing a dynamic data migration mechanism to migrate cold data from the non-volatile memory layer to the SSD layer;

[0009] A cache management algorithm is used, combining hash tables and LRU linked lists, to determine the distribution of data between the non-volatile memory layer and the SSD layer based on the access frequency of the data, and a globally unique identifier is used to track and manage the data;

[0010] A space management mechanism is implemented to manage free pages using a free page list in the non-volatile memory layer so that the released non-volatile memory space can be quickly reused.

[0011] Furthermore, the non-volatile memory layer stores the data in the form of files in a file system supporting the non-volatile memory layer by means of memory mapping.

[0012] The cache management algorithm further comprises:

[0013] When data is inserted, a globally unique identifier is generated and associated with the data item, which is then mounted in the LRU list;

[0014] When data is accessed, the position in the LRU list is updated to reflect the access frequency of the data;

[0015] When the system restarts, the LRU linked list is rebuilt to restore the hot and cold information status of the data.

[0016] The data dynamic migration mechanism further includes:

[0017] Data migration is performed at the granularity of data segments, and the size of each data segment is a preset value;

[0018] During the data migration process, the data segment optimistic locking mechanism is used to isolate transaction operations from migration operations.

[0019] Furthermore, the size of the data segment is 2MB.

[0020] Furthermore, the data dynamic migration mechanism also includes:

[0021] When an X lock is required on a target data segment for migration, the system first checks whether the tuples in the data segment are being accessed by any transaction;

[0022] If the tuple in the target data segment is not accessed by any transaction, the tuple-level X lock is notified to be released. After the release is completed, the X lock of the target data segment is successfully locked, and then the target data segment is migrated;

[0023] If the tuples in the target data segment are being accessed by transactions, these transactions will be allowed to proceed according to the transaction priority policy, the migration action will be terminated, and the target data segment will not be locked. The data segment will be remounted to the head of the LRU linked list. When the LRU linked list reaches the threshold, the data segment at the end of the LRU linked list will be selected as the new migration target.

[0024] The space management mechanism further includes:

[0025] Maintain a free page list in each SSD layer to manage all free non-volatile memory pages;

[0026] When data is migrated from the non-volatile memory layer to the SSD layer, the released SSD layer pages are added to the FPL linked list;

[0027] Periodically check the FPL linked list to reclaim and reuse non-volatile memory pages that have not been used for a long time.

[0028] The present invention also provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the above anti-caching method is implemented.

[0029] Compared with the prior art, the present invention has the following advantages:

[0030] 1. The present invention rationally designs the storage structure, fully utilizes the speed and persistence of NVM and the large capacity and low cost of SSD, effectively expands the database storage capacity, and saves economic costs.

[0031] 2. The present invention proposes a cache replacement method suitable for NVM, which can better adapt to the persistence characteristics of NVM. After restarting, the system can quickly restore the original data hot and cold information. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings generally illustrate various embodiments by way of example and not limitation, and together with the description and claims, serve to illustrate the embodiments of the invention. Where appropriate, the same reference numerals are used throughout the drawings to refer to the same or similar parts. Such embodiments are illustrative and are not intended to be exhaustive or exclusive embodiments of the present apparatus or method.

[0033] Figure 1 A diagram representing the proposed storage architecture;

[0034] Figure 2 This is the flow chart for cold data storage.

[0035] Figure 3 The flowchart for data callback in SSD. DETAILED DESCRIPTION

[0036] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0037] The anti-caching method supporting non-volatile memory database provided by the present invention includes:

[0038] By optimizing the storage hierarchy architecture, the advantages of NVM and NVMe SSD can be fully utilized to ensure efficient storage access. A reasonable data layout needs to be designed to improve system performance;

[0039] Uses advanced cache management algorithms to help decide which data should remain in NVM and which can be moved to NVMe SSDs;

[0040] A locking mechanism is designed to ensure data consistency when migrating between NVM and NVMe SSD.

[0041] The data layout is divided into the NVM layer and the NVMe SSD layer. In the non-volatile memory (NVM), the persistent memory development kit (PMDK) provided by Intel is used to store data in the form of files in the file system that supports NVM through memory mapping. This method bypasses the page cache of the operating system to directly access the NVM data. Since the virtual address will be relocated each time the database is started, in order to ensure the correct access to the data, the virtual address mapped to each file needs to be recorded in a SegmentAddr array. Since the virtual addresses mapped to different files are not continuous, each file is logically divided into several 8KB pages, and by adding a layer of mapping, the data on the NVM forms a logically continuous address space. For each relational table in the database, several 2MB data segments (extents) are dynamically allocated for data storage, and the first page ID of each extent is recorded in the extentIds array. Each element in the extentIds array is a 32-bit integer, in which the first bit is used to indicate whether the extent is stored in the non-volatile memory (NVM). Specifically, if the first bit is 0, it indicates that the segment currently resides in NVM, and if it is 1, it is stored in NVMe SSD. In the data layout of the solid-state drive (NVMe SSD), it is managed with the help of the file system. Data is migrated between the two storage media, NVM and SSD, at the granularity of extent. For data located in the SSD, tableId (unique identifier of the table) is used as the file path, and the file is logically divided into several blocks of segment size in advance to ensure that each segment of each table can find the corresponding address on the SSD logically. The actual allocation and recycling of SSD physical addresses are efficiently and correctly managed by the file system.

[0042] The cache management algorithm uses the LRU (least recently used) linked list mechanism to implement an efficient cache replacement strategy. In this algorithm, the LRU linked list maintains the hot and cold order of data to ensure that it can be eliminated according to the principle of least recently used. Due to the persistence characteristics of non-volatile memory (NVM), to prevent NVM space leakage, the system must restore its cache state after restart. The specific implementation steps are as follows:

[0043] 1. Read persistent data: Every time the system restarts, the extentIds array of each table in the database is read from NVM. The array contains the identification information of the extent, and each identification is a 32-bit unsigned integer.

[0044] 2. Parse the storage location: Parse each entry in the extentIds array and check the most significant bit to determine whether the extent resides in NVM.

[0045] 3. Rebuilding the LRU linked list: For the segments identified as residing in the NVM, they are inserted into the LRU linked list in sequence according to their order in the array.

[0046] Through the above steps, the system can quickly restore its cache management mechanism after restart, thereby ensuring the continuity of cache data and ensuring that NVM space will not leak. This design not only optimizes the performance of the cache strategy in a persistent memory environment, but also significantly improves the stability of the system.

[0047] In the process of data migration from NVM to SSD, consistency is a core challenge, especially in a multi-transaction concurrent environment. Specifically, when data is being migrated from NVM to SSD, if other transactions are allowed to modify the extent being written to disk, the data on the SSD may become outdated or inconsistent. This inconsistency will destroy the correctness of the data, resulting in the reading of the wrong data version, which in turn affects the reliability and stability of the system. At the same time, since the locking operation of the transaction is a finer tuple-level lock, a circular wait chain may be generated, causing a deadlock. In order to solve a series of problems in the concurrency control of the disk operation and transaction access, we have designed a strict locking mechanism to ensure the consistency of data during the migration process. If the system successfully adds an X lock to the target extent, it will start to migrate the data of the extent from NVM to SSD, and automatically release the X lock after the migration is completed. During the locking process, if other transactions are operating on the extent, these transactions will be processed first, and the data writing behavior will be blocked. During the process of extent flushing, the X lock has been successfully added to the extent, and the operations of reading and writing the target extent will be blocked. After the lock is released, the blocked transaction can continue to execute, and the updated data distribution status can be seen, thereby ensuring data consistency. In order to further deal with possible deadlock risks, the system implements a deadlock detection algorithm. When a deadlock is detected, the system will forcibly release the x lock on the extent to resolve the deadlock. Subsequently, the flushing behavior of the target extent will be canceled, and it will be remounted to the head of the LRU linked list and regarded as hot data. At the same time, the system will select a new extent from the LRU linked list to continue data migration to release limited NVM resources and ensure system reliability and concurrency performance.

[0048] Storage hierarchy architecture Figure 1As shown in the figure, in non-volatile memory (NVM), data storage follows the row storage principle. Each table is equipped with a root extent to store the metadata of the table and other extents for storing data. In the process of creating a table, the system automatically allocates a root extent for it, and its size is set to 2MB. As the rows of data are inserted, the system will dynamically allocate additional extents to the table to store data. The size of these extents is also 2MB. When the data storage of an extent reaches the capacity limit, the system will automatically allocate new extents to meet the storage needs. Each 2MB extent is further subdivided into multiple 8KB pages. Each page has a globally unique identifier in NVM, called pageId, which corresponds to a starting address in NVM. In the root extent, an array named extentIds is maintained to record the first pageId (data stored in NVM) and SSDId (data stored in SSD) corresponding to each extent. The corresponding data obtained by querying the extenIds array according to the extentId is used to distinguish whether the extent is stored in NVM or SSD. If the data is in NVM, the pageId with the first digit of 0 is used. If the data is in SSD, the SSDId with the first digit of 1 is used (block number on SSD, 2MB per block). Through these IDs, the first address of each extent can be quickly accessed, or the corresponding extent data on SSD can be accessed according to the SSDId. In NVM, the system will use a global least recently used (LRU) linked list to determine the access frequency of each extent, that is, the hot and cold status. The most recently accessed extent will be marked as a hot extent. Since the extentId is not globally unique, we re-encode the tableId and extentId to generate a globally unique globextentId and place it at the head of the LRU linked list. Conversely, less frequently accessed extents are marked as cold extents. In an SSD, each table corresponds to a file on the SSD, which is divided into multiple 2MB data blocks. Each data block stores only one extent-sized data. When data is written, the offset calculated by multiplying the SSDId obtained in the extentIds array by the extent size is used as the corresponding position where the data is written to the file. Since each table corresponds to a file and the ExtentId number increases from 0, there will be no overlap in the position of the extent area data when it is written to the file. It is completely feasible to use only the ExtentId number as the offset of the extent area in the file.

[0049] The specific process of cold data storage is as follows: Figure 2 The specific process is as follows:

[0050] 1) The globextentId encoded by the accessed extent is added to the LRU list, and the number of globextentIds in the LRU list is checked to see if it reaches the preset threshold. If it does not reach the threshold, normal data read and write operations are continued without executing the disk write process.

[0051] 2) When the number of globextentIds in the LRU linked list reaches the threshold, the system takes out the least recently used globextentId from the end of the linked list, decodes the globextentId, and obtains the corresponding tableId and extentId.

[0052] 3) For the X lock on the target extent, notify the tuple-level X lock in the extent to be released. If a deadlock occurs, release the X lock on the extent, mount the extent to the head of the LRU linked list, and select the new tail extent to be written to the disk.

[0053] 4) According to the pageId corresponding to the extentId recorded in the extentIds array, obtain the starting address of the extent in NVM.

[0054] 5) Open the SSD file corresponding to tableId. Calculate the offset of the data in the SSD file: multiply extentId by extent size to get the write position. Call the file system API to refresh the extent data in NVM to the specified offset position of the SSD file.

[0055] 6) The corresponding position of the extentId recorded in the extentIds array is modified to SSDId, marking that the data corresponding to the extentId has been stored in the SSD.

[0056] 7) Update the NVM management module to release the extent space in NVM that has been migrated to the SSD to ensure that the space can be used by new data.

[0057] The callback process is as follows Figure 3 As shown, the specific process is:

[0058] 1) Query the extentIds array, obtain the element corresponding to the extentId, and determine whether the extent currently exists in NVM or SSD. If the data exists in NVM, the process is the same as the cold data disk process. If the data exists in SSD, a callback action is required.

[0059] 2) Allocate space in NVM for callback data storage. If there is a free page in FPL, allocate the free page in FPL and call back 2.1 in the flowchart. Otherwise, reallocate the space of extent size from NVM and call back 2.2 in the flowchart.

[0060] 3) SSDId obtained from the extentIds array

[0061] 4) According to SSDId, calculate the offset position of the data in the SSD, read the corresponding extent data from the SSD, and call it back from the SSD to the NVM.

[0062] 5) Change the corresponding element of the extentId in the extentIds array to pageId. Mark the extent data in NVM with pageId as the marker.

[0063] 6) Encode the tableId and extentId of the extent into a globally unique globextentId and mount it into the LRU linked list to track and manage the access frequency of data. If the LRU reaches the threshold, the cold data is dropped to disk.

[0064] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for anti-caching supporting non-volatile memory database, characterized in that: include: Constructing a storage hierarchy architecture including a non-volatile memory layer and a solid-state drive layer, wherein the non-volatile memory layer is used to store frequently accessed hot data, and the solid-state drive layer is used to store infrequently accessed cold data; Introducing a dynamic data migration mechanism to migrate cold data from the non-volatile memory layer to the SSD layer; A cache management algorithm is used, combining hash tables and LRU linked lists, to determine the distribution of data between the non-volatile memory layer and the SSD layer based on the access frequency of the data, and a globally unique identifier is used to track and manage the data; Implement a space management mechanism and use a free page list in the non-volatile memory layer to manage free pages and reuse the released non-volatile memory space.

2. The anti-caching method supporting non-volatile memory database according to claim 1, characterized in that: The non-volatile memory layer stores the data in the form of files in a file system supporting the non-volatile memory layer by means of memory mapping.

3. The anti-caching method supporting non-volatile memory database according to claim 1, characterized in that: The cache management algorithm further comprises: When data is inserted, a globally unique identifier is generated and associated with the data item, which is then mounted in the LRU list; When data is accessed, the position in the LRU list is updated to reflect the access frequency of the data; When the system restarts, the LRU linked list is rebuilt to restore the hot and cold information status of the data.

4. The anti-caching method supporting non-volatile memory database according to claim 1, characterized in that: The data dynamic migration mechanism further includes: Data migration is performed at the granularity of data segments, and the size of each data segment is a preset value; During the data migration process, the data segment optimistic locking mechanism is used to isolate transaction operations from migration operations.

5. The anti-caching method supporting non-volatile memory database according to claim 1, characterized in that: The size of the data segment is 2MB.

6. The anti-caching method supporting non-volatile memory database according to claim 1, characterized in that: The data dynamic migration mechanism also includes: When an X lock is required on a target data segment for migration, the system first checks whether the tuples in the data segment are being accessed by any transaction; If the tuple in the target data segment is not accessed by any transaction, the tuple-level X lock is notified to be released. After the release is completed, the X lock of the target data segment is successfully locked, and then the target data segment is migrated; If the tuples in the target data segment are being accessed by transactions, these transactions will be allowed to proceed according to the transaction priority policy, the migration action will be terminated, and the target data segment will not be locked. The data segment will be remounted to the head of the LRU linked list. When the LRU linked list reaches the threshold, the data segment at the end of the LRU linked list will be selected as the new migration target.

7. The anti-caching method supporting non-volatile memory database according to claim 1, characterized in that: The space management mechanism further includes: Maintain a free page list in each SSD layer to manage all free non-volatile memory pages; When data is migrated from the non-volatile memory layer to the SSD layer, the released SSD layer pages are added to the FPL linked list; Periodically check the FPL linked list to reclaim and reuse non-volatile memory pages that have not been used for a long time.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the anti-caching method according to any one of claims 1 to 7 is implemented.

Images