A method for anti-caching index of hybrid storage of non-volatile memory and solid state disk

Abstract

The application discloses a kind of non-volatile memory and solid state disk hybrid storage anti-cache index method, belong to database index technical field;Including: the layered index architecture of mixed storage is constructed, layered index architecture includes non-volatile memory layer and solid state disk layer;Index node in non-volatile memory layer is divided into internal node and leaf node;The metadata of leaf node is separated from load, and metadata is kept in non-volatile memory layer as anchor, and address pointer format of non-volatile memory layer and solid state disk layer is unified;According to the hot and cold state of leaf node of leaf node access frequency and dynamic expiration time policy, the ownership of leaf node between two layers of storage and the migration between different storage layers are dynamically managed in combination with candidate queue.The application combines non-volatile memory and solid state disk, uses the persistence and low delay of non-volatile memory and the low cost and large capacity of solid state disk, so that it meets the scene of mass data, while improving resource utilization and reducing cost.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of database indexing, and more particularly to a non-volatile memory and solid state disk hybrid storage anti-caching index method. BACKGROUND

[0002] In a database system, as a core technology, index has a profound and significant impact on improving system performance and efficiency, and reasonable design and application of index is the key to ensuring efficient operation of the storage system. On the one hand, with the rapid development of informatization and digitization, the data processing demand faced by the database is rising, and the sharp expansion of massive data makes the role of index more prominent; on the other hand, the bottleneck of traditional DRAM technology in capacity expansion and cost control gradually appears, which has been difficult to keep up with the growing memory demand.

[0003] The rise of non-volatile memory (NVM) technology, such as phase change memory (PCM) and 3DXPoint technology, provides a new possibility for building efficient indexes in NVM. For example, Intel Optane can not only meet the demand for large-capacity storage, but also has the advantages of byte addressing, low latency, and no data loss after power failure, which can well make up for the shortcomings of DRAM with high cost and small capacity. However, as the amount of data increases, the space occupied by the index also increases, and the problem of insufficient NVM resources still exists. When facing massive data, how to more efficiently use limited NVM resources is a problem, which includes the following problems:

[0004] The existing non-volatile memory space management mechanism faces a general space management scenario, so it will occupy a large amount of DRAM at runtime, and because it needs to manage non-volatile memory fragments, the released space cannot be effectively reused, resulting in a large waste of resources; the existing hot and cold identification of index data is at the page level (i.e., multiple nodes exist in the same page, and the node types in the page may not be consistent), which causes the hot and cold identification of index data to be inaccurate, such as determining a page with a small number of hot data nodes as a cold page, resulting in frequent swapping in and out of a large amount of data, causing a large I / O overhead.

[0005] Therefore, how to solve the problems in the prior art is a problem that those skilled in the art need to solve. SUMMARY

[0006] In view of the above problems, the present application provides a non-volatile memory and solid state disk hybrid storage anti-caching index method for overcoming the above problems or at least partially solving the above problems;

[0007] In order to achieve the above purpose, the present application adopts the following technical solutions:

[0008] The embodiment of the present application provides a non-volatile memory and solid state disk hybrid storage anti-caching index method, comprising the following steps:

[0009] S1, constructing a hierarchical index architecture of hybrid storage, the hierarchical index architecture comprising a non-volatile memory layer and a solid state disk layer, the non-volatile memory layer being used for storing frequently accessed hot index nodes, and the solid state disk layer being used for storing infrequently accessed cold index nodes;

[0010] S2, dividing the index nodes in the non-volatile memory layer into internal nodes and leaf nodes, wherein the leaf nodes comprise two parts of metadata and payload;

[0011] S3, separating the metadata and the payload of the leaf nodes, storing the metadata in the non-volatile memory layer as an anchor, and unifying the address pointer formats of the non-volatile memory layer and the solid state disk layer;

[0012] S4, acquiring an access feature based on the anchor point in the non-volatile memory layer, judging the hot and cold states of the leaf nodes according to the access frequency and a dynamic expiration time strategy of the leaf nodes, and combining a candidate queue to dynamically manage the belonging of the leaf nodes between the two layers of storage and the migration between different storage layers.

[0013] Further, the hierarchical index architecture adopts a space management mechanism, divides the non-volatile memory layer into different types of segments according to the node types of the index, adopts a partition management strategy in the segments, uses an offset to allocate space and a logical deletion manner to release space in the partition, and records log information corresponding to the operation in the metadata of the partition;

[0014] The solid state disk is divided into a plurality of files with the same capacity, is managed by using a bitmap stored in the non-volatile memory layer, and is allocated to the solid state disk layer by using a producer-consumer mode and setting a state bit for the solid state disk layer block.

[0015] Further, the producer-consumer mode comprises:

[0016] A producer thread: obtaining the offset of a free block by accessing the bitmap, setting the block state to a first state, and putting the block into the tail of the producer-consumer queue;

[0017] A consumer thread: taking out the offset of the head of the producer-consumer queue by a thread requesting space, and modifying the corresponding block state to a second state after writing data is completed;

[0018] The corresponding block state is modified to a third state when the space is released,

[0019] wherein the first state indicates that the block has been allocated but not used, the second state indicates that the block has been used, and the third state indicates that the block is free.

[0020] Further, the non-volatile memory layer is segmented, and each segment is allocated with a type of node space;

[0021] The segment offset is used to locate the index node position in the segment, and the nextOffset variable is used to mark the next free position in the partition. When logical deletion, the corresponding offset chain is linked to the free list in the partition;

[0022] The key steps of the log are recorded when allocating and releasing space in the partition.

[0023] Further, when allocating non-volatile memory layer space, determine the type of index data node that needs to be allocated;

[0024] First, check if there is free space in the free list. If not, allocate unused space, return nextOffset, and move it to the next free position. If it cannot be allocated, check if there is logically released space in the free list. If there is, take the offset at the end of the list and encode the segment number and offset into a unified address pointer. If not, a new non-volatile memory segment needs to be allocated;

[0025] When releasing space, parse the corresponding non-volatile memory segment number and offset according to the unified address pointer, and take logical release. The offset corresponding to the released space is linked to the free list.

[0026] Further, the unified address pointer structure has n bits, where the n-1 bit is used to distinguish between internal nodes and leaf nodes, the n-2 bit is used to distinguish the storage hierarchy to which the node belongs, the n-4 bit to n-16 bit represents the non-volatile memory segment number or solid state disk file number, and the n-17 bit to 0 bit represents the offset.

[0027] Further, according to the access characteristic data stored in the leaf node metadata, the hot and cold of the leaf node is identified and judged;

[0028] The access characteristic data stored in the leaf node metadata includes the latest access time, the access frequency, and the address pointer information;

[0029] The latest access time is the time of the last access to the leaf node, the access frequency is the number of times the leaf node is accessed in NVM, and the average access frequency of the leaf node is the ratio of the total number of accesses to the number of non-volatile memory layer leaf nodes.

[0030] Further, a fine-grained cache management algorithm is used;

[0031] Set up a candidate queue to store nodes to be migrated to the solid state disk layer;

[0032] When reaching the set non-volatile memory layer space usage threshold, a traversal thread is enabled, all leaf metadata parts are scanned, and leaf load nodes meeting cold data are put into a candidate queue as candidate nodes, and if a leaf load node being accessed is encountered, it is skipped.

[0033] Further, at the same time of starting the traversal thread, a migration thread is enabled for processing nodes in the candidate node queue, the candidate nodes are taken from the head of the queue in turn, and their cold and hot states are judged again, if they still meet the cold data condition, they are migrated to the solid state disk layer, and the corresponding non-volatile memory layer space is released, and the state of the corresponding block in the solid state disk layer space is changed;

[0034] When accessing a leaf load node in the solid state disk layer, it is migrated to the non-volatile memory layer, and the corresponding metadata and the state of the corresponding solid state disk layer block are modified.

[0035] Through the above technical solution, compared with the prior art, the present application provides a non-volatile memory and solid state disk hybrid storage anti-cache index method, which has the following beneficial effects:

[0036] 1、The present application combines non-volatile memory and solid state disk, uses the persistence and low latency of non-volatile memory and the low cost and large capacity of solid state disk, expands the index capacity of non-volatile memory, meets the scene of massive data, and improves resource utilization and reduces cost.

[0037] 2、The present application proposes a fine space management mechanism for non-volatile memory, divides the non-volatile memory into different types of segments according to the node type of the index, uses a partition management strategy in the segment, uses an offset to allocate space in the partition, and uses a logical deletion method to release space, so as to efficiently allocate and release non-volatile memory, and records log information corresponding to the operation in the metadata of the partition to prevent space leakage after failure.

[0038] 3、The present application separates the metadata and load of the index leaf node, stores the metadata in the non-volatile memory as an anchor, and unifies the address pointer format of the non-volatile memory and the solid state disk, realizes fast positioning and transparent access of nodes across storage layers, improves index query efficiency and simplifies system implementation complexity.

[0039] 4、The present application proposes a fine-grained cache management algorithm, judges the cold and hot state of the leaf node according to the index leaf node access frequency and dynamic expiration time strategy, and combines a candidate queue to dynamically manage the ownership of the leaf load node between the two layers of storage and the migration between different storage layers, a kind of cache management algorithm, which more accurately judges the cold and hot state of the data load node and decides the storage layer of its ownership, reduces the amount of data in and out, thereby reducing the computing overhead. BRIEF DESCRIPTION OF DRAWINGS

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0041] Figure 1 This is a flowchart of the decaching indexing method provided in the embodiments of the present invention;

[0042] Figure 2 This is an overall architecture diagram of the non-volatile memory index provided in this embodiment of the invention;

[0043] Figure 3 This is a flowchart illustrating the allocation and release of non-volatile memory space provided in this embodiment of the invention.

[0044] Figure 4 This is a flowchart illustrating the allocation and release of solid-state drive space provided in an embodiment of the present invention.

[0045] Figure 5 This is a diagram of the unified address pointer structure provided in the embodiments of the present invention;

[0046] Figure 6 This is a flowchart of the cache replacement process provided in an embodiment of the present invention. Detailed Implementation

[0047] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] This invention discloses an anti-caching indexing method for hybrid storage of non-volatile memory and solid-state drive; such as... Figure 1 As shown, it includes the following steps:

[0049] S1. Construct a hybrid storage hierarchical index architecture, which includes a non-volatile memory layer and a solid-state drive layer. The non-volatile memory layer is used to store frequently accessed hot index nodes, and the solid-state drive layer is used to store infrequently accessed cold index nodes.

[0050] S2. Divide the index nodes in the non-volatile memory layer into internal nodes and leaf nodes, wherein the leaf nodes include metadata and payload.

[0051] S3. Separate the metadata of leaf nodes from the payload, store the metadata in the non-volatile memory layer as an anchor, and unify the address pointer format of the non-volatile memory layer and the solid-state drive layer.

[0052] S4. Based on the anchor points in the non-volatile memory layer, access characteristics are obtained. At the same time, the hot and cold status of leaf nodes is judged according to the access frequency of leaf nodes and the dynamic expiration time strategy. Combined with the candidate queue, the affiliation of leaf nodes between the two storage layers and the migration between different storage layers are dynamically managed.

[0053] This invention combines non-volatile memory and solid-state drives (SSDs), leveraging the persistence and low latency of non-volatile memory and the low cost and large capacity of SSDs to expand the index capacity of non-volatile memory, making it suitable for scenarios with massive amounts of data, while improving resource utilization and reducing costs.

[0054] The above methods will now be described in detail.

[0055] like Figure 2 As shown, during index construction, nodes are allocated and data is stored in the NVM layer. At the same time, the nodes are organized into a structured form with high query performance to achieve fast data retrieval.

[0056] As the amount of index data continues to grow, when the storage capacity of NVM approaches saturation, a data migration mechanism is activated: cold leaf nodes with low access frequency in NVM are migrated to the solid-state drive layer. At the same time, in order to maintain the overall access performance of the index, only leaf payload nodes are migrated to SSD, while all internal nodes remain in NVM.

[0057] In SSD, storage space is allocated for the migrated cold data nodes. The node location is efficiently managed through metadata information such as bitmaps stored in NVM. A unified address pointer is used to ensure that the index can quickly determine the storage layer and physical location of the node, thereby supporting fast and transparent access to cross-layer data.

[0058] Cold-load nodes that have been migrated to SSDs may still be accessed again. When this happens, the system triggers a reverse migration mechanism to migrate the corresponding nodes back from SSDs to the NVM layer to ensure subsequent access performance.

[0059] The space management mechanism of the non-volatile memory as a whole adopts a management strategy based on segmentation. Within each segment, a partition strategy is used to improve concurrency. Allocation and recovery of the NVM are performed using an offset in the partition and a logical deletion method. To prevent space leakage after a failure, logs are recorded before allocation and release operations, and it is specified that only the same type of node can be allocated within the same segment. Compared with existing third-party general non-volatile memory allocation libraries, this method effectively reduces the overhead of memory management metadata, overcomes the fixed limitation that general memory allocators are difficult to effectively reuse released memory space due to adaptation to multiple scenarios, and significantly improves the utilization efficiency and space reusability of the NVM.

[0060] In an implementation, the system records the starting virtual address allocated by each segment. By combining the starting address with the offset in the segment, the physical location of the data node can be accurately located. To further improve the concurrency performance and avoid a single segment from becoming a bottleneck for thread competition during allocation and release, a partition strategy is used to divide a segment into multiple partitions. Each partition maintains independent metadata information for managing the internal space state, thereby realizing multi-thread parallel allocation and release operations. The partition metadata information is uniformly stored in the header of the segment. The metadata content includes a partition lock for concurrent control, an offset of the next free position in the partition, a partition free list for storing the released space in the partition, a phase operation type log for recovery, and a global allocation node number for quickly judging whether the space of the segment is exhausted. The free list is operated by recording the tail index.

[0061] To ensure that the allocator does not deadlock due to the error state of the lock before the failure during failure recovery, the metadata of each partition is traversed during restart, and the state of the lock is reset to no lock. The log is divided into two categories: allocation and release. The offset and the number of free nodes in the free list are recorded before allocation and release. A step execution completion flag is recorded after the free list is modified, which is used to restore the space to the original correct state during log recovery to avoid leakage of the NVM. An independent free segment set is maintained for each type of segment.

[0062] When the new segment creation or segment space release operation in a segment is completed, the segment number of the segment is added to the free segment set of the corresponding type, effectively avoiding the problem that the system always unconditionally creates a new segment when the current segment is insufficient.

[0063] As shown in Figure 3 the specific process of NVM space allocation is as follows:

[0064] After receiving the memory allocation request, the allocator first determines the starting attempt partition: add the last accessed partition number to a maximum prime number smaller than the total number of partitions, and then take the total number of partitions as the modulus to obtain the first partition number to be attempted;

[0065] A mutex is applied to the selected partition to achieve concurrent access control;

[0066] The operation log is recorded, and the log will be updated in the subsequent execution steps;

[0067] The length of the free list of the partition is checked first to determine whether there is an allocatable node: if there is, the corresponding offset of the node is returned from the list, the index pointing to the tail of the list is reduced by 1, and the corresponding operation log is recorded;

[0068] If there is no available node in the free list of the current partition, the next free offset recorded in the partition is queried to determine whether the offset is within the maximum range of the current partition: if it is not out of bounds, the offset is returned, and the free offset pointer in the partition is updated to the start position of the next node, and the relevant log is recorded, otherwise the allocation fails;

[0069] The lock of the current partition is released. If the allocation is successful, the log is marked as completed, and the allocated offset is returned; if the allocation fails, the global allocation node number of the current segment is used to determine whether the current segment has no available space: if the space is insufficient, a new segment is allocated by the allocator, and step 1 is returned to try to allocate again until the allocation is successful, and the global allocation node number is updated.

[0070] The specific process of releasing NVM space is as follows:

[0071] A space release request is parsed to obtain the partition corresponding to the space to be released;

[0072] A mutex is applied to the partition to achieve concurrent access control;

[0073] The operation log is recorded, and the log will be updated in the subsequent execution steps;

[0074] The tail index of the free list is recorded, the release offset is written in the tail of the free list, the tail index is increased by 1, the global allocation node number of the current segment is updated, and the current segment number is added to the corresponding free segment set;

[0075] The mutex of the current partition is released.

[0076] The SSD space management is built on the traditional file system, and the bitmap mechanism is introduced to finely manage the state of each SSD file internal storage block in view of the deficiency of the traditional file system in fine-grained space management. Specifically, a two-bit encoded bitmap identifies three states of each block: "00" represents idle, "01" represents allocated but not yet put into use, and "11" represents used. The bitmap is the core metadata structure in SSD space allocation.

[0077] To improve its access and update efficiency, the bitmap is managed by partition to support concurrent operation, and the state of the block in the corresponding partition is quickly determined and modified through efficient bit operation. Each partition is equipped with a partition lock, which provides not only concurrent control but also intergenerational information, i.e. the start-up number of the current index system. When locking, the intergenerational information in the lock is verified first to see if it is consistent with the current index intergenerational; if not, it is determined that it is the first access after restart, and the corresponding recovery logic needs to be executed.

[0078] When the index is restarted, the error correction and recovery of the bitmap state are completed in the index log playback process. Therefore, during this recovery process, only the blocks with state "01" in the current partition need to be reset to "00", and the intergenerational information in the lock needs to be updated synchronously. To optimize the allocation performance, the producer-consumer mode is adopted, i.e. an independent dedicated allocation producer thread is set up to be responsible for scanning and pre-allocating idle SSD blocks, and the blocks are inserted into the tail of the free block queue after being constructed into address pointers of a unified format.

[0079] The maximum length of the queue can be configured, and the blocks inserted into the queue have their state marked as "01". When the consumer thread requesting to allocate SSD blocks arrives, the pre-constructed address pointer can be directly obtained from the head of the queue, and the state of the block is modified to "11", which greatly reduces the real-time allocation delay.

[0080] In addition, similar to the non-volatile memory management mechanism, the SSD layer also maintains a free file set. Since the nodes migrated to SSDs only contain a single type, only a unified free file set needs to be maintained to efficiently manage the reuse and allocation of SSD files.

[0081] The specific process of SSD space allocation and release is shown in Figure 4

[0082] The specific process of allocating SSD space is as follows, and the process is always looped:

[0083] Check if the queue is full, if not, determine the starting attempt partition: add the last access partition number to a maximum prime number smaller than the total number of partitions, and take the modulus of the total number of partitions to get the first partition number to be tried;

[0084] Apply a mutual exclusion lock to the selected partition to achieve concurrent access control;​

[0085] Find the free block in the current partition, if exists, add to the end of the queue, if not, return to step 1;

[0086] Modify the state of the corresponding block to "01";

[0087] Unlock the corresponding partition.

[0088] The specific process of releasing the SSD space is as follows:

[0089] Get the partition number and block number by parsing the uniform address pointer, and lock the corresponding partition;

[0090] Modify the state of the corresponding block to "00";

[0091] Unlock the corresponding partition.

[0092] The structure of the uniform format address pointer is as shown in Figure 5 .

[0093] First, separate the metadata of the index leaf node from the payload, and store the metadata in the NVM as an anchor. On this basis, a uniform address pointer is proposed. The structure is 64 bits in total, of which the 63rd bit is used to distinguish between internal nodes and leaf nodes, the 62nd bit is used to distinguish the storage hierarchy to which the node belongs ("0" represents belonging to the non-volatile memory layer, and "1" represents belonging to the solid state disk layer), the 60th to 48th bits represent the non-volatile memory segment number or the solid state disk file number, and the 47th to 0th bits represent the offset. First, determine the hierarchy to which it belongs, and then determine the segment or file to which the node belongs. SSD file allocation uses the same type of node, so the block number can be calculated through the offset.

[0094] Among them, the fine-grained cache management strategy combines the access frequency of the leaf node and the dynamic expiration time to determine whether the payload node should belong to the NVM layer or the SSD layer. The candidate strategy is used to put the first cold node into the candidate queue, and only when it is judged as a cold node in the candidate queue again do we consider it a real cold node. At the same time, the migration of cold and hot data between the two layers is managed.

[0095] In specific implementation, the latest access timestamp and access frequency of each leaf metadata node are recorded as the basis for cold and hot state judgment. The leaf node sets a dynamic expiration time, which is updated in units of minutes. The value is estimated by estimating the time required to access all leaf nodes according to the rate of accessing leaf nodes in the current minute, and the minimum value of the expiration time is limited to 360 seconds and the maximum value is limited to 720 seconds.

[0096] A monitoring thread is set to acquire real-time key information such as NVM usage, total access times of leaf nodes, and access rate. Meanwhile, a traversal thread is set to scan the leaf nodes in the NVM, and the starting time stamp is recorded when the traversal starts. During the traversal, the system judges the hot and cold states of each node: if the node has expired or its access times are lower than the average access times currently recorded, the node is put into the candidate queue.

[0097] The migration thread continuously monitors the candidate queue and takes out the nodes to be processed for secondary screening. Only when the node meets the conditions of "expired" and "access times lower than average access times" at the same time, the load node is migrated to the SSD layer. To improve the accuracy of hot and cold judgment and reduce the jitter caused by frequent migration, a two-time screening mechanism is adopted.

[0098] In the expiration time judgment, the time difference between the two traversal starting times is taken as a complete time period. If the latest access time stamp of a leaf node plus its expiration time is earlier than the 80% time point of the time period, the node is determined to be an expired node. In particular, in the first traversal, the time difference is the interval from the initial time 0 to the current time.

[0099] The specific process of the traversal thread and the migration thread in cache replacement is shown in Figure 6

[0100] When the monitoring thread detects that the non-volatile memory space usage reaches the preset threshold, the traversal thread is awakened to start traversing the leaf nodes stored in the NVM. The traversal thread attempts to lock the leaf node. If the locking fails, it indicates that the node is in the accessed state, and it is determined to be a hot data node and is skipped. If the locking is successful, it is further determined whether the node meets one of the following conditions: the node has expired, or its access frequency is lower than the average access frequency currently recorded by the system. If any of the conditions is met, the node is determined to be a cold data node.

[0101] All identified cold data nodes are added to the candidate cold data node queue for subsequent processing.

[0102] The specific process of the migration-dump thread is as follows:

[0103] The node is taken out from the head of the candidate cold data node queue, and secondary judgment is performed thereon. Only when the node meets the two conditions of "expired" and "access frequency lower than average access frequency" at the same time, it is finally confirmed to be a cold data node.

[0104] An SSD storage space is allocated for the confirmed cold data node, and an SSD block that has been pre-formatted and whose address pointer is ready is obtained from the head of the pre-allocated queue of the SSD space manager.

[0105] ​The cold data node data is migrated to the allocated SSD space, and the state of the corresponding SSD block is modified to "11". Meanwhile, detailed migration logs are recorded, including migration stage identification and SSD space address pointer, to ensure that the system can be recovered after a failure;

[0106] After migration is completed, the NVM space originally occupied by the cold data node is released.

[0107] The specific process of the migration-callback thread is as follows:

[0108] When a data node is accessed, it is first determined whether the target leaf node has been migrated to the SSD layer;

[0109] If the node has been migrated to the SSD, the callback process is triggered, and the node is reallocated space in the NVM layer;

[0110] The node data is migrated from the SSD space to the newly allocated NVM space;

[0111] After the data callback is completed, the SSD space originally occupied by the node is immediately released, that is, the state of the corresponding SSD block is modified to "00".

[0112] The present application proposes a fine space management mechanism for non-volatile memory, which divides the non-volatile memory into different types of segments according to the node types of the index, adopts a partition management strategy in the segment, uses an offset to allocate space in the partition, and uses a logical deletion method to release space, so as to efficiently allocate and release the non-volatile memory, and records the log information of the corresponding operation in the metadata of the partition to prevent space leakage after a failure; meanwhile, the metadata and the load of the index leaf node are separated, the metadata is retained in the NVM as an anchor, and the NVM and SSD address pointer formats are unified, realizing fast positioning and transparent access of the node across the storage layer, improving the index query efficiency and simplifying the system implementation complexity.

[0113] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts of each embodiment can be referred to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0114] The above description of the disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for anti-caching index of hybrid storage of non-volatile memory and solid state disk, characterized in that, The method comprises the following steps: S1, constructing a hierarchical index architecture of hybrid storage, the hierarchical index architecture comprising a non-volatile memory layer and a solid state disk layer, the non-volatile memory layer being used for storing frequently accessed hot index nodes, and the solid state disk layer being used for storing infrequently accessed cold index nodes; The hierarchical index architecture adopts a space management mechanism, the non-volatile memory layer is divided into different types of segments according to the types of the index nodes, a partition management strategy is adopted in the segments, an offset is used to allocate space in the partition, and a logical deletion manner is used to release space, and log information of the corresponding operation is recorded in the metadata record of the partition; The solid state disk is divided into a plurality of files with the same capacity, the bitmap stored in the non-volatile memory layer is used for management, and a state bit is set for the solid state disk layer block and a producer-consumer mode is adopted to allocate the solid state disk layer; S2, dividing the index nodes in the non-volatile memory layer into internal nodes and leaf nodes, wherein the leaf nodes comprise metadata and payload; S3, separating the metadata and the payload of the leaf nodes, storing the metadata in the non-volatile memory layer as an anchor, and unifying the address pointer formats of the non-volatile memory layer and the solid state disk layer; S4, obtaining an access feature based on the anchor in the non-volatile memory layer, judging the hot and cold states of the leaf nodes according to the access frequency and the dynamic expiration time strategy of the leaf nodes, and combining a candidate queue to dynamically manage the belonging of the leaf nodes between the two layers of storage and the migration between different storage layers.

2. The anti-caching index method of hybrid storage of non-volatile memory and solid state disk according to claim 1, wherein, The producer-consumer mode comprises: A producer thread: obtaining the offset of a free block by accessing the bitmap, setting the block state to a first state, and putting it into the tail of the producer-consumer queue; A consumer thread: the thread requesting space takes the offset at the head of the producer-consumer queue, and modifies the corresponding block state to a second state after writing data is completed; When releasing space, the corresponding block state is modified to a third state, Wherein, the first state indicates that the block has been allocated but not used, the second state indicates that the block has been used, and the third state indicates that the block is free.

3. The anti-caching index method of claim 2, wherein, Segmenting the non-volatile memory layer comprises: Allocating one type of node space for each segment; The offset in the segment is used to locate the position of the index node in the segment, the nextOffset variable in the partition is used to mark the next free position in the partition, and the corresponding offset is linked to the free linked list in the partition when logically deleting; Log the key steps when allocating and releasing space in the partition.

4. The anti-caching index method of claim 3, wherein, Allocating non-volatile memory layer space comprises: determining the type of index data node that needs to be allocated; Preferably, whether there is free space in the free linked list is found from the free linked list, if there is no free space in the free linked list, then unused space is allocated, the nextOffset is returned, and it is moved to the next free position; if it cannot be allocated, then whether there is logically released space in the free linked list is checked, if there is, then the offset at the tail of the linked list is taken out, and the segment number and the offset are encoded into a unified address pointer, and if there is not, then a new non-volatile memory segment needs to be allocated; When releasing space, the corresponding non-volatile memory segment number and offset are obtained by parsing the unified address pointer, and logical release is adopted, the offset corresponding to the released space is linked to the free linked list.

5. The anti-caching index method of hybrid storage of non-volatile memory and solid state disk according to claim 4, wherein, The unified address pointer format of the non-volatile memory layer and the solid state disk layer includes: a unified address pointer structure of n bits, wherein the (n-1)th bit is used to distinguish the internal node and the leaf node, the (n-2)th bit is used to distinguish the storage layer to which the node belongs, the (n-4)th bit to the (n-16)th bit represents the non-volatile memory segment number or the solid state disk file number, and the (n-17)th bit to the 0th bit represents the offset.

6. The anti-caching index method of hybrid storage of non-volatile memory and solid state disk according to claim 5, wherein, The cold and hot state judgment of the leaf node includes: judging the cold and hot of the data leaf node according to the access characteristics stored in the leaf node metadata; The access characteristic data stored in the leaf node metadata includes the latest access time, the access frequency and the address pointer information; The latest access time is the time of the last access to the leaf node, the access frequency is the number of times that the leaf node is accessed in the NVM, and the average access frequency of the leaf node is the ratio of the total access frequency to the number of non-volatile memory layer leaf nodes.

7. The anti-caching index method of claim 6, wherein, The hierarchical index architecture adopts a fine-grained cache management algorithm; A candidate queue is set to store the nodes to be migrated to the solid state disk layer; When the set non-volatile memory layer space usage threshold is reached, a traversal thread is started to scan all leaf metadata parts, and the leaf nodes that meet the cold data are put into the candidate queue as candidate nodes, and if a leaf node being accessed is encountered, it is skipped.

8. The anti-caching index method of hybrid storage of non-volatile memory and solid state disk according to claim 7, characterized in that, At the same time of starting the traversal thread, a migration thread is started to process the nodes in the candidate node queue, and the candidate nodes are taken from the head of the queue in turn, and the cold and hot state of the candidate nodes is judged again, and if the candidate nodes still meet the cold data condition, the candidate nodes are migrated to the solid state disk layer, the corresponding non-volatile memory layer space is released, and the state of the corresponding block in the solid state disk layer is changed; When the leaf node in the solid state disk layer is accessed, it is migrated to the non-volatile memory layer, and the corresponding metadata and the state of the corresponding solid state disk layer block are modified.

Images