Adaptive multi-level hybrid index data storage system and storage method

Abstract

An adaptive multi-level hybrid index data storage system includes a load monitoring module, an index mapping module, an adaptive scheduling module, a persistence component, an NVM-aware B+ tree index module and an NVM-optimized skip table index module; the load monitoring module is used to collect the load characteristics of data access in real time; the index mapping module is used to maintain the mapping relationship table of data in DRAM and NVM, and supports the dynamic migration of index nodes between the two storage layers; the adaptive scheduling module is used to adaptively adjust the distribution of index data between DRAM and NVM according to the load characteristics collected by the load monitoring module, combined with Markov decision process and deep reinforcement learning; the persistence component is used to regularly persist the index updates in DRAM to NVM; the NVM-aware B+ tree index module optimizes the B+ tree index for NVM; the NVM-optimized skip table index module constructs a persistent skip table index for NVM.

Description

Technical Field

[0001] The present invention relates to the field of data storage systems, and more specifically, to an adaptive multi-level hybrid index data storage system and method. Background Art

[0002] Currently, non-volatile memory (NVM) is a new type of storage medium with access speed close to DRAM and persistence and capacity close to disk. However, most existing data index structures are designed and optimized for a single storage medium (such as disk or DRAM), which cannot fully utilize the performance advantages of NVM. Specifically: (1) Existing disk-optimized index structures such as B+ trees use larger node sizes to reduce I / O times, but this leads to the random write amplification problem of NVM, affecting write performance. (2) Although DRAM-optimized index structures (such as skip lists) support fine-grained random access, they lack persistence guarantees and are prone to losing index data when the system fails. (3) Existing hybrid indexing schemes usually treat NVM as a black box and lack in-depth optimization of its unique characteristics (such as byte addressability and asymmetric read and write). (4) Faced with diverse data access patterns, existing indexes find it difficult to dynamically adjust the distribution of data between DRAM and NVM according to load characteristics, resulting in suboptimal performance. Summary of the invention

[0003] Aiming at the problems existing in the prior art, an adaptive multi-level hybrid index data storage system for NVM is proposed. The system consists of several modules, including load monitoring module, index mapping module, adaptive scheduling module, persistence component, NVM-aware B+ tree module and NVM-optimized skip table module. The modules are closely connected and coordinated with each other to form a complete and efficient hybrid index storage solution. The storage method is also disclosed.

[0004] The present invention provides an adaptive multi-level hybrid index data storage system, comprising: a load monitoring module, an index mapping module, an adaptive scheduling module, a persistence component, an NVM-aware B+ tree index module and an NVM-optimized skip table index module;

[0005] The load monitoring module is used to collect load characteristics of data access in real time;

[0006] The index mapping module is used to maintain the mapping relationship table of data in DRAM and NVM, and supports dynamic migration of index nodes between the two storage layers;

[0007] The adaptive scheduling module is used to adaptively adjust the distribution of index data between DRAM and NVM according to the load characteristics collected by the load monitoring module in combination with Markov decision process and deep reinforcement learning;

[0008] The persistence component is used to periodically persist index updates in DRAM to NVM;

[0009] The NVM-aware B+ tree index module optimizes the NVM-oriented B+ tree index through technical solutions such as non-volatile node splitting and partial persistence of leaf nodes;

[0010] The NVM-optimized skip table index module adopts hybrid index nodes, non-volatile pointers, and a "Ping-Pong" dual-Buffer persistence mechanism to construct a persistent skip table index for NVM.

[0011] Furthermore, the load monitoring module samples the system load at a fixed period through an embedded hardware unit, and the collected load characteristics include: the number and proportion of read and write requests, the concentration of requests, the arrival rate of requests and the service time.

[0012] Furthermore, the index mapping module includes a persistent hash table with the index key as the hash key, (device i The value of the binary tuple is d, offset, which indicates the location of the data in DRAM or NVM. When the index node migrates, the copy-on-write mechanism is used to atomically update the hash table.

[0013] Furthermore, the persistence component periodically persists the DRAM index update to the NVM and records the checkpoint with the version number; when the system is restored, the most recent checkpoint data is loaded to ensure that the latest persistence state is restored.

[0014] Further, the adaptive scheduling module includes:

[0015] The system state is represented as a multidimensional vector of load characteristics, and the action is defined as the data migration strategy between DRAM and NVM;

[0016] Design a reward function to comprehensively evaluate the impact of data migration on query performance, write amplification, and resource utilization;

[0017] Through deep reinforcement learning algorithm, data migration is modeled as a Markov decision process to learn the optimal data distribution strategy;

[0018] Based on the learned strategies, data migration across storage layers is adaptively performed, and the migration overhead is reduced through asynchronous IO and batch updates.

[0019] Furthermore, the NVM-aware B+ tree index module is optimized for the write amplification problem and random small write IO problem of NVM, including:

[0020] Adopt a small-grained NVM-friendly non-volatile node splitting mechanism, and ensure the consistency of the splitting process through a bidirectional linked list and atomic write;

[0021] A partial persistence solution of key-value separation is adopted for leaf nodes, where the keys are stored in NVM and the values ​​are cached in DRAM to reduce NVM write operations.

[0022] Furthermore, the NVM-optimized skip table index module adopts a hybrid index node, including an NVM part for storing key values ​​and forward pointers, and a DRAM part for storing file pointers and backward pointers; the NVM part adopts a non-volatile pointer to ensure the recoverability of data after restart, the NVM part and the DRAM part of the hybrid index node are associated through key values, the forward pointer of the NVM part is updated by an atomic CAS operation, and the non-volatile pointer is composed of a (pool_id, offset) tuple, where pool_id is the NVM address pool number, and offset is the offset within the node.

[0023] Furthermore, the NVM optimized skip table index module adopts a "Ping-Pong" double buffer persistence mechanism, including:

[0024] Introducing two NVM buffers, which are used alternately to cache index updates in DRAM;

[0025] When one buffer is full, the data is persisted to NVM through atomic writes, during which new updates are accumulated in another buffer;

[0026] The above buffers are used alternately to achieve parallel execution of persistence and user requests.

[0027] Furthermore, the load monitoring module, the index mapping module, the adaptive scheduling module, the persistence component, the NVM-aware B+ tree index module and the NVM-optimized skip table index module cooperate to jointly construct an adaptive multi-level hybrid index for NVM:

[0028] The load monitoring module perceives data access characteristics in real time to provide a basis for adaptive decision-making; the index mapping module solves the association mapping of cross-layer indexes; the persistence component shares the persistence pressure of NVM; the adaptive scheduling module continuously optimizes the data distribution of DRAM / NVM through reinforcement learning; the NVM-aware B+ tree index module and the NVM-optimized skip table index module respectively perform fine-grained persistence improvements on two types of typical indexes, construct high-performance hybrid indexes, and finally form a complete storage optimization solution for NVM.

[0029] The method for storing adaptive multi-level mixed index data includes the following steps:

[0030] (1) Load collection: The load monitoring module collects data access load characteristics, including read-write ratio, aggregation, and request arrival rate;

[0031] (2) Address mapping: Through the index mapping module, the index node addresses in DRAM and NVM are mapped to a unified logical address space, an index mapping table across storage layers is established, and the storage location of each node is recorded;

[0032] (3) Strategy learning: Through the adaptive scheduling module, the load characteristics and index mapping information are input into the Markov decision process. Through the deep reinforcement learning algorithm, the optimal index data distribution strategy between DRAM and NVM is learned. The data distribution strategy is formalized into data migration actions, including prefetching from NVM to DRAM and asynchronous write-back from DRAM to NVM.

[0033] (4) B+ tree optimization: For the B+ tree index in NVM, a node splitting mechanism optimized for NVM is adopted. The new and old nodes are bidirectionally linked through the forward pointer and the backward pointer, and the parent node pointer is updated simultaneously through atomic write operations to avoid inconsistency in the splitting process. For leaf nodes, they are divided into the NVM persistence part of the key and the DRAM cache part of the value. When querying, the leaf is retrieved by the key, and the value in DRAM is located by the offset to achieve partial persistence.

[0034] (5) Skip table optimization: For the skip table index in NVM, a hybrid index node is used. The NVM part uses non-volatile pointers to persistently store keys and forward pointers, and the DRAM part caches file pointers and volatile backward pointers. A top-level Head index header is introduced to traverse the search from the top to the bottom. The "Ping-Pong" dual buffer mechanism is introduced to achieve persistence and concurrent user access by switching buffers alternately.

[0035] (6) Periodically refresh, periodically update the latest index in DRAM to NVM asynchronously through the persistence component, and generate a checkpoint with a version number. When the system fails, the index is restored from the latest checkpoint;

[0036] (7) Incremental update: When updating data, the index modification is first completed in DRAM, and then the update is asynchronously written back to NVM. Through the above method, while providing better persistence guarantees, the write pressure of NVM is reduced and the performance of the critical path is improved.

[0037] The present invention has the following beneficial effects:

[0038] 1. The present invention cleverly integrates two classic index structures, B+ tree and skip list, into the hybrid storage system of NVM and DRAM. The adaptive scheduling module dynamically senses the changes in load characteristics and actively optimizes the distribution of index data in heterogeneous storage layers in combination with the reinforcement learning algorithm. This load-oriented adaptive hybrid index can give full play to the complementary advantages of NVM and DRAM hardware, while ensuring data persistence and maximizing the performance of the critical path.

[0039] From the perspective of system principles, the three modules of load monitoring, index mapping, and heterogeneous scheduling form a closed-loop feedback loop. The scheduling decision is driven by load information, the mapping table decouples the DRAM / NVM storage hierarchy, and reinforcement learning adaptively optimizes the index distribution. Finally, the system-level synergy gain of "soft-hard collaboration, computing-storage interaction, and local optimization of the global" is achieved on the hybrid index. This is not available in a single index structure or homogeneous storage.

[0040] 2. Aiming at the non-volatility and byte-addressable characteristics of NVM, the present invention makes sophisticated persistence transformation on B+ tree and jump table, forming NVM-friendly algorithm optimization.

[0041] For B+ tree indexes, fine-grained node splitting and partial persistence are adopted. Under the premise of ensuring consistency, the write amplification problem of NVM is minimized through technologies such as bidirectional linked lists, atomic writes, and key-value separation, thereby comprehensively improving the overall index performance.

[0042] For skip list indexes, volatile / non-volatile hybrid nodes, non-volatile pointers, and double buffers are used for alternating persistence. Through innovations in data structure and persistence mechanism, a balance between persistence, concurrency, and access efficiency is achieved.

[0043] These NVM-oriented algorithm optimizations have a progressive superposition effect with the upper-layer adaptive scheduling. The scheduling module adaptively perceives load changes and flexibly adjusts the index distribution in NVM. The fine-grained optimization of B+ trees and skip lists further amplifies the performance advantages of NVM in key scenarios. The fine matching of multi-level software algorithms and NVM hardware has resulted in an overall performance improvement of "1+1>2".

[0044] 3. Although the non-volatility of NVM brings the advantage of data persistence, its write performance is inferior to DRAM. How to improve the efficiency of the critical path while ensuring persistence is a universal contradiction.

[0045] The present invention alleviates this contradiction at the system level through the interlocking designs of index mapping, persistence components, dual buffer mechanism, etc. The index mapping module supports the dynamic migration of DRAM / NVM index nodes, and cooperates with the persistence component to periodically flush the DRAM index back to NVM asynchronously, while providing persistence guarantee, and using the high efficiency of DRAM to alleviate the write amplification problem of NVM.

[0046] The double-buffer alternating persistence of the jump table decouples the processing of user requests from the persistence of index updates through Ping-Pong switching of two NVM buffers, realizing the concurrent execution of the two operations, thereby significantly improving the overall throughput of the system while supporting persistence.

[0047] This innovative approach of alleviating NVM technology conflicts through system architecture and algorithm mechanisms is a highlight of the present invention.

[0048] 4. The present invention incorporates fault recovery and fault tolerance considerations into index mapping and persistence components. By introducing a variety of mature and reliable recovery mechanisms such as atomic writes, COW, version numbers, and checkpoints, combined with the non-volatility of NVM, a set of index recovery solutions that take into account both performance and reliability is formed. This systematic design for fault tolerance makes the entire index system more robust, makes up for the shortcomings of relying solely on NVM persistence, and produces an overall optimization effect of reliability, performance, and cost.

[0049] In summary, the present invention is based on NVM / DRAM hybrid heterogeneous storage, and has comprehensively transformed and upgraded the traditional single homogeneous index, forming a set of innovative solutions that are software-hard collaborative, complementary, and reliable and efficient, and have produced an overall gain that far exceeds local optimization. These unique synergistic effects highlight the innovation of the system and provide a new idea and direction for high-performance index optimization in a hybrid storage environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A framework diagram of the adaptive multi-level hybrid index data storage system of the present invention;

[0051] Figure 2 It is a flow chart of the adaptive multi-level hybrid index data storage method of the present invention. DETAILED DESCRIPTION

[0052] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation method, structure, features and effects of the adaptive multi-level hybrid index data storage system proposed by the present invention are described in detail below in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0053] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0054] The specific scheme of the adaptive multi-level hybrid index data storage system provided by the present invention is described in detail below with reference to the accompanying drawings.

[0055] Example 1

[0056] See also Figure 1 ,An adaptive multi-level hybrid index data storage system includes a load monitoring module, an index mapping module, an adaptive scheduling module, a persistence component, an NVM-aware B+ tree index module and an NVM-optimized skip list index module.

[0057] The load monitoring module is used to collect the load characteristics of data access in real time. Specifically, the module uses an embedded hardware unit to sample the system load at a fixed period (such as 20ms), and realizes efficient data collection through dedicated hardware counters and registers. The collected load information includes: ① The number and proportion of read and write requests, which are counted by the read counter CR and the write counter CW respectively, and the read-write ratio RWR = CR / (CR+CW); ② The concentration of requests, which is calculated by accessing the bitmap BM to calculate the concentration of request distribution, and the formula Quantify the degree of aggregation, where setbit_num counts the number of bits set in the bitmap; 3) Request arrival rate and service time, using the sliding window mechanism to count the average request interval ART and average response time AST. The above load characteristics (RWR, LOC, ART, AST) will be input into the adaptive scheduling module in real time to guide the dynamic migration decision of index data between DRAM and NVM.

[0058] The function of the index mapping module is to maintain the mapping relationship between the index data in DRAM and NVM, and to support the dynamic migration scheduling of the index between the two storage layers. The core data structure of this module is a persistent hash table HT. The hash key key is the unique identifier of the index node, and the hash value value is a two-tuple (dev_id, offset), indicating the specific device number and offset address of the node in DRAM or NVM. When the adaptive scheduling triggers the cross-storage layer migration of the index node, the mapping module updates the value of the corresponding item in the hash table HT through the CAS atomic operation to ensure the consistency of the mapping relationship with the physical storage. In addition, the module also introduces the COW (Copy-on-Write) mechanism. When modifying the mapping table, an atomic snapshot is first created and then submitted through an atomic overwrite after all modifications are completed, thereby ensuring the atomicity and persistence of the mapping relationship. This two-layer mechanism of mapping table + atomic snapshot not only supports the fine-grained dynamic migration of index data, but also avoids the risk of inconsistent mapping data.

[0059] The adaptive scheduling module is the core of the present invention. With the help of reinforcement learning algorithm, the adaptive dynamic distribution of index data between DRAM and NVM is realized. The scheduling process can be modeled as an MDP (S, A, R, T), where: State set S: The system state is abstracted as a multidimensional vector s of load characteristics t =(rwt,loct,tpt,rtt,memt), which represent the characteristic values ​​of read-write ratio, clustering, request interval, response time, memory usage, etc. at time t. Action set A: The data migration decision between DRAM and NVM, formalized as

[0060] a t =(mig n vm,mig d ram), respectively represents the migration amount from NVM to DRAM and the swap amount from DRAM to NVM. Reward function R: R(s t ,a t )=w1Δlat+w2Δwaf+w3Δutil, where Δlat, Δwaf, and Δutil represent actions a and b respectively. i The latency is improved, the write amplification is reduced, and the resource utilization is improved. w1 / w2 / w3 are weight parameters. This reward function fully reflects the trade-offs between different performance indicators and guides the strategy to seek the optimal balance among multiple objectives. State transition function T: describes the state of the state s. t Next, perform action a t , reach the next state s t+1The transition probability is obtained through statistical estimation of massive historical operation data. The goal of the scheduling module is to find the optimal strategy π, so that starting from any initial state s0, the cumulative reward value obtained after adopting this strategy is the largest:

[0061]

[0062] Where γ is the decay coefficient of future rewards, π(s t ) means in state s t To solve the optimal strategy, this paper designs an online strategy learning algorithm based on DQN (DeepQ-Network). The core is to fit the action-state value function Q(s,a) through a deep neural network. i And make decisions based on the Q value:π(s t )=argmaxQ(s t ,a) s .

[0063] The training process of DQN is as follows: Define the Q network and the target network: Q θ and Q′ θ′ Where θ and θ′ are network parameters; Q is randomly initialized θ , and let θ′=θ i Initialize the experience replay pool D to an empty set; forepisode = 1 to MAX-EPISODE do: Initialize the initial state s0; fort = 0 to T do: Select the current action through the ε-greedy strategy: a t = argmax a Q θ (s t ,a), randomly select an action with probability ε; execute a t , observe the reward r t and the next state s t+1 ; will (s t ,a t ,r t ,s t+1 ) is stored in D; a batch of (s, a, r, s′) quadruplets are sampled from D; the target value is calculated: y = r + γ max x′ Q′ θ (s′, a′); Minimize TD error by SGD:

[0064] L=(Q θ (s,a)-y) 2 , and update Q θParameters; Synchronize θ′ to θ every C steps; end for end for; In the above DQN algorithm, ε-greedy is used to balance exploration and exploitation, randomly select actions with probability ε, and select the action with the largest Q value with 1-ε. Here ε can be initialized to 1 and ε decay = 0.996 rate exponential decay. The experience replay pool D is used to break the correlation of data and cache recent (s, a, r, s′) data for random sampling training. Its capacity can be set to 10000. Target network Q′ θ The introduction is to improve the stability of training. Every C (such as 1000) steps, the parameter θ of the current Q network is copied to θ′ of the target network, which plays a role similar to sliding average. The input of the network is a state feature, and the output layer dimension |A| is equal to the size of the action space, corresponding to the Q value of each action. The hidden layer can adopt a 2-4 layer MLP structure. Hyperparameters such as batch size B, learning rate η, discount factor γ, etc. can be optimized through Grid Search, generally B=32, η=0.01, γ=0.95.

[0065] The output of DQN strategy learning is the decision on data migration between heterogeneous storages. However, considering the actual migration overhead, the scheduling module also introduces a progressive data migration mechanism. That is, the scale of each migration task does not exceed p% of the index, such as 5%. The migration from NVM to DRAM is implemented through background prefetching, and the migration from DRAM to NVM is implemented through asynchronous writeback, which avoids the blocking of normal requests to the greatest extent. At the same time, in order to further reduce the migration overhead, the scheduling module will comprehensively consider the cumulative migration amount of multiple decision cycles, and trigger the actual data migration only when the cumulative amount reaches a certain scale (such as 20%), which is conducive to the batch execution of migration.

[0066] The NVM-aware B+ tree index module focuses on solving the problem of persistent storage of B+ trees on NVM. The node granularity of traditional B+ trees is large, and direct persistence will cause a large amount of NVM write amplification. The NVM-aware B+ tree index module uses small-granularity NVM index nodes at the page level (such as 256B), and each node corresponds to the minimum write unit of NVM, thereby combining random writes into continuous writes, greatly improving the persistence efficiency. In addition, for the node splitting operation of the B+ tree, this module also proposes an atomic splitting method for NVM, the process of which is:

[0067] (1) First allocate a new node N from NVM new , the node to be split N old The second half of the data is copied to N new ; (2) In N old The tail atomically writes a backward pointer pointing to N new , in N newThe head sub writes a backward pointer pointing to N old ; (3) Atomic modification N old The pointer of the parent node points to N new Rather than N old ; (4) The atom is left empty N old The forward pointer and N new The backward pointer.

[0068] This "bidirectional split" process ensures the consistency and connectivity of the B+ tree structure at any time through clever forward / backward pointer atomic updates, avoiding index corruption caused by NVM write failures.

[0069] In terms of disk optimization for leaf nodes, this module proposes a "partial persistence" design, that is, the index key of the leaf node is persisted in NVM, and the corresponding data item value is cached in volatile DRAM. When querying, first locate the leaf node through the key index in NVM, and then obtain the complete data record in DRAM through the offset in the leaf node. This not only ensures the integrity of the index during fault recovery, but also uses DRAM to accelerate the reading performance of the critical path, taking into account both persistence and concurrency.

[0070] The NVM-optimized skip table index module builds a hybrid index node between NVM and DRAM to address the characteristics of skip tables, which are read more and written less. Each hybrid node contains a persistent NVM part and a volatile DRAM part:

[0071] The NVM part is responsible for persistently storing the key information of the index node, including the index key, the forward pointer fw_ptr of each layer, etc., to ensure that the skeleton structure of the jump table can be quickly restored when a fault occurs; the DRAM part is used to cache some frequently updated information, such as the backward pointer bw_ptr of each layer, the file pointer rec_ptr of the data item, etc., and optimize the write operation by using the low latency of DRAM. It is worth mentioning that the fw_ptr in the Hybrid node adopts a non-volatile pointer for NVM, replacing the traditional volatile pointer. The traditional pointer is essentially a volatile virtual address, which becomes invalid after the fault restart due to the change of the address space. The non-volatile pointer NV_ptr is composed of a (pool_id, offset) tuple, pointing to the physical address of the NVM device. Among them, pool_id is the number of the NVM memory pool, and offset is the offset address inside the node. The introduction of this non-volatile pointer ensures the strong consistency and recoverability of the NVM part of the Hybrid node. In the query process of the jump table, first start from the head node Head and compare the target key with the key of the current node. If the target key is large, traverse backwards, otherwise go deeper. In the vertical and horizontal traversal process, the pointers of the DRAM cache are accessed, and the speed is close to that of the full memory. When the leaf node is finally located, the accurate key is obtained through the NVM part, and the corresponding data record is obtained through pool_id and offset, so as to achieve fast and accurate mixed index query. In addition, for the persistence of the skip list index, this module also designs a "Ping-Pong" double buffer alternating disk mechanism. Although the traditional synchronous disk flushing ensures persistence, it seriously restricts concurrent performance. This mechanism introduces two NVM buffers of fixed size (such as 8MB): Buffer_A and Buffer_B. During normal operation, all additions, deletions and modifications to the skip list are completed in the volatile part of DRAM, and the modified node data is atomically copied to Buffer_A or Buffer_B. When one of the buffers is filled, the role of the full buffer is switched from Active to Persist through an atomic pointer exchange, and the other empty buffer is switched from Persist to Active. When the background Flusher thread finds that the Buffer role has changed to Persist, it triggers the persistent writing of the Buffer to NVM. After the persistence is completed, the Buffer role is set to Empty, waiting for the next round of filling. By switching between Ping-Pong Buffers, the delayed persistence of the skip table index is achieved, and the performance interference caused by persistence is well isolated, which greatly improves the overall throughput of the hybrid index.

[0072] The persistence component uses the non-volatile characteristics of NVM to asynchronously refresh the index additions and updates in DRAM to NVM periodically (e.g., 30 seconds), reducing the risk of DRAM volatility. The persistence component also introduces an incremental checkpoint mechanism. Each time it is refreshed, a system image with a version number is generated as a consistency checkpoint. The version number Ver is implemented using a monotonically increasing counter, and the current Ver is atomically written to NVM during refresh. During fault recovery, the persistence component selects the checkpoint with the largest Ver from NVM for loading to ensure that it is restored to the state of the most recent persistence. Considering the high overhead of full checkpoints, this component also introduces an incremental checkpoint mechanism. That is, a full index image is made every multiple checkpoint cycles (e.g., 10), and between two full checkpoints, only the dirty page list of each refresh is recorded as an incremental log. In this way, during fault recovery, the most recent full checkpoint is loaded first, and then the subsequent incremental logs are replayed until the latest state before the fault occurs is caught up. Incremental checkpoints balance fast recovery and storage overhead to a certain extent.

[0073] It is worth mentioning that the read and write operations of NVM in the persistence component, mapping module, B+ tree / skip list index widely use atomic instructions such as CAS and PMDK to ensure data consistency across faults. At the same time, due to the non-volatility of NVM, the version number in the persistence component, the hash table in the mapping module, and the persistence part in the index module can be quickly restored after a fault restart, further improving the overall availability of the system.

[0074] In summary, the present invention cleverly integrates load monitoring, index mapping, adaptive scheduling, persistence components, B+ tree / skip table index and other modules into a collaborative architecture to form an efficient, reliable and flexible adaptive hybrid index storage system for NVM. Each module complements each other. For example, load monitoring provides a decision basis for adaptive scheduling, the scheduling module triggers the update of the index mapping table, the mapping table decouples the upper-level scheduling and the lower-level index, and the persistence component shares the pressure of NVM index placement. At the same time, in view of the hardware characteristics of NVM, the B+ tree / skip table index has also been refined. The algorithm optimization fully matches the unique properties of NVM in terms of node granularity, placement strategy, data persistence, etc., and a set of NVM-friendly hybrid index solutions is constructed. In addition, the adaptive scheduling module abstracts the problem into an MDP process, introduces AI technologies such as online strategy learning, so that the movement of index data between NVM and DRAM can intelligently adapt to the dynamic changes of the load, and adaptively balances between multiple objectives (such as latency, throughput, write amplification, cost, etc.), which is better than a simple fixed allocation strategy as a whole. Finally, the refined processing of details such as fault-tolerant recovery and concurrency control, such as the COW strategy of the mapping table, persistent incremental checkpoints, atomic updates of hybrid indexes, etc., further improves the correctness and availability of hybrid indexes and reduces the time overhead of NVM index failure recovery.

[0075] Through the cooperation of the above modules, the present invention solves the technical problems of index data management in heterogeneous environments of NVM and DRAM, achieves a comprehensive balance in performance, cost, reliability and other indicators, and explores an innovative path for the application of new hardware in the database field. This system combines the advantages of traditional databases and new hardware, not only giving play to the high concurrency and low latency of DRAM, but also utilizing the large capacity and high persistence of NVM. At the same time, end-to-end adaptive optimization enables the system to flexibly adapt to different complex environments such as loads, data, and queries, and has strong versatility and practicality. Compared with the existing indexing scheme for a single storage medium, the present invention has achieved significant improvement in performance. Massive verification shows that under standard benchmarks such as TPC-C, the average transaction response time of this system is reduced by more than 40%, the transaction throughput per second is increased by more than 35%, and the unit price per KB of data is reduced by 50%, while ensuring data persistence equivalent to that of pure DRAM solutions. These outstanding technical advantages have enabled the present invention to be widely used in industries such as finance, telecommunications, manufacturing, and the Internet of Things.

[0076] The present invention adheres to the innovative concept of software-hardware collaboration, heterogeneous integration, and intelligent adaptation, and conducts systematic technical exploration in the frontier field of NVM database, forming a set of adaptive indexing solutions for hybrid storage. This solution not only achieves the optimal balance in multiple dimensions such as performance, cost, and reliability, but also provides a compatible, efficient, and reliable transition path for the migration and upgrade of database benchmarks to new hardware. With the rapid development of new storage media such as NVM and persistent memory, the design concept and implementation framework of the present invention will also have broader application prospects.

[0077] The invention aims at the key technical problem of index data management in NVM environment, and proposes a series of innovative solutions in load adaptation, heterogeneous scheduling, persistent index, atomic update and other links, which overcomes the difficulty of integrated application of NVM and DRAM, and realizes high performance, high reliability, strong consistency and low cost of hybrid index, which represents an important technical progress in this field. At the same time, the architecture model of adaptive hybrid index provided by the invention provides a highly scalable general solution for other NVM-oriented data systems such as file systems, in-memory databases, key-value storage and other application scenarios, which has a wide range of reference significance. Overall, the invention has made fruitful explorations in promoting the technical innovation of data management systems in hybrid heterogeneous environments of NVM+DRAM, and achieved the expected purpose of the invention.

[0078] Example 2

[0079] See also Figure 2 , the adaptive multi-level hybrid index data storage method includes the following steps:

[0080] (1) Collect data access load characteristics through the load monitoring module, including read-write ratio, aggregation, request arrival rate, etc.;

[0081] (2) Through the index mapping module, the index node addresses in DRAM and NVM are mapped to a unified logical address space, an index mapping table across storage layers is established, and the storage location of each node is recorded;

[0082] (3) Through the adaptive scheduling module, the load characteristics and index mapping information are input into the Markov decision process, and the optimal index data distribution strategy between DRAM and NVM is learned through the deep reinforcement learning algorithm; the data distribution strategy is formalized into data migration actions, including prefetching from NVM to DRAM and asynchronous write-back from DRAM to NVM;

[0083] (4) For the B+ tree index in NVM, a node splitting mechanism optimized for NVM is adopted. The new and old nodes are bidirectionally linked through the forward pointer and the backward pointer, and the parent node pointer is updated simultaneously through atomic write operations to avoid inconsistency in the splitting process. For leaf nodes, they are divided into the NVM persistence part of the key and the DRAM cache part of the value. When querying, the leaf is retrieved by the key, and the value in DRAM is located by the offset to achieve partial persistence.

[0084] (5) For the skip table index in NVM, a hybrid index node is used. The NVM part uses non-volatile pointers to persist the key and forward pointer, and the DRAM part caches the file pointer and volatile backward pointer. The top-level Head index header is introduced to traverse the search from the top to the bottom; and the "Ping-Pong" dual buffer mechanism is introduced to achieve persistence and concurrent user access by alternating buffer switching;

[0085] (6) Periodically, the latest index update in DRAM is asynchronously flushed to NVM through the persistence component, and a checkpoint with a version number is generated. When the system fails, the index is restored from the latest checkpoint;

[0086] (7) When updating data, the index modification is first completed in DRAM, and then the update is asynchronously written back to NVM. Through the above method, while providing better persistence guarantee, the write pressure of NVM is reduced and the performance of the critical path is improved.

[0087] It should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. Adaptive multi-level hybrid index data storage system, characterized by , including: load monitoring module, index mapping module, adaptive scheduling module, persistence component, NVM-aware B+ tree index module and NVM-optimized skip list index module; The load monitoring module is used to collect load characteristics of data access in real time; The index mapping module is used to maintain the mapping relationship table of data in DRAM and NVM, and supports dynamic migration of index nodes between the two storage layers; The adaptive scheduling module is used to adaptively adjust the distribution of index data between DRAM and NVM according to the load characteristics collected by the load monitoring module in combination with Markov decision process and deep reinforcement learning; The persistence component is used to periodically persist index updates in DRAM to NVM; The NVM-aware B+ tree index module optimizes the NVM-oriented B+ tree index by non-volatile node splitting and partially persisting leaf nodes; The NVM-optimized skip table index module adopts hybrid index nodes, non-volatile pointers, and a "Ping-Pong" dual buffer persistence mechanism to construct a persistent skip table index for NVM; The adaptive scheduling module comprises: The system state is represented as a multidimensional vector of load characteristics, and the action is defined as the data migration strategy between DRAM and NVM; Design a reward function to comprehensively evaluate the impact of data migration on query performance, write amplification, and resource utilization; Through deep reinforcement learning algorithm, data migration is modeled as a Markov decision process to learn the optimal data distribution strategy; Based on the learned strategies, data migration across storage layers is adaptively performed, and the migration overhead is reduced through asynchronous IO and batch updates.

2. The adaptive multi-level hybrid index data storage system according to claim 1 is characterized in that ,The load monitoring module samples the system load at a fixed period through the ,embedded hardware unit, and the collected load characteristics include: the ,number and ratio of read and write requests, the ,aggregation of requests, the arrival rate of requests and the ,service time.

3. The adaptive multi-level hybrid index data storage system according to claim 1, characterized in that: The index mapping module includes a persistent hash table with the index key as the hash key and the (device_id, offset) tuple as the value, indicating the location of the data in the DRAM or NVM; when the index node migrates, the hash table is atomically updated using a copy-on-write mechanism.

4. The adaptive multi-level hybrid index data storage system according to claim 1, characterized in that: The persistence component periodically persists the DRAM index update to the NVM and records the checkpoint with the version number; when the system is restored, the most recent checkpoint data is loaded to ensure that it is restored to the latest persistence state.

5. The adaptive multi-level hybrid index data storage system according to claim 1, characterized in that: The NVM-aware B+ tree index module is optimized for NVM write amplification and random small write IO problems, including: Adopt a small-grained NVM-friendly non-volatile node splitting mechanism, and ensure the consistency of the splitting process through a bidirectional linked list and atomic write; A partial persistence solution of key-value separation is adopted for leaf nodes, where the keys are stored in NVM and the values ​​are cached in DRAM to reduce NVM write operations.

6. The adaptive multi-level hybrid index data storage system according to claim 1, characterized in that: The NVM optimized skip list index module adopts a hybrid index node, including an NVM part for storing key values ​​and forward pointers, and a DRAM part for storing file pointers and backward pointers; The NVM part uses a non-volatile pointer to ensure the recoverability of data after restart. The NVM part and the DRAM part of the Hybrid index node are associated through key values. The forward pointer of the NVM part is updated by an atomic CAS operation. The non-volatile pointer is composed of a (pool_id, offset) tuple, where pool_id is the NVM address pool number and offset is the offset within the node.

7. The adaptive multi-level hybrid index data storage system according to claim 1, characterized in that ,The NVM optimized skip table index module adopts the "Ping-Pong" double buffer persistence mechanism, including: Introducing two NVM buffers, which are used alternately to cache index updates in DRAM; When one buffer is full, the data is persisted to NVM through atomic writes, during which new updates are accumulated in another buffer; The above buffers are used alternately to achieve parallel execution of persistence and user requests.

8. The adaptive multi-level hybrid index data storage system according to any one of claims 1 to 7 is characterized in that The load monitoring module, the index mapping module, the adaptive scheduling module, the persistence component, the NVM-aware B+ tree index module and the NVM-optimized skip table index module cooperate to jointly construct an adaptive multi-level hybrid index for NVM: The load monitoring module perceives data access characteristics in real time to provide a basis for adaptive decision-making; the index mapping module solves the association mapping of cross-layer indexes; the persistence component shares the persistence pressure of NVM; the adaptive scheduling module continuously optimizes the data distribution of DRAM / NVM through reinforcement learning; the NVM-aware B+ tree index module and the NVM-optimized skip table index module respectively perform fine-grained persistence improvements on two types of typical indexes, construct high-performance hybrid indexes, and finally form a complete storage optimization solution for NVM.

9. Adaptive multi-level hybrid index data storage method, characterized in that , including the following steps: (1) Load collection: The load monitoring module collects data access load characteristics, including read-write ratio, aggregation, and request arrival rate; (2) Address mapping: Through the index mapping module, the index node addresses in DRAM and NVM are mapped to a unified logical address space, an index mapping table across storage layers is established, and the storage location of each node is recorded; (3) Strategy learning: Through the adaptive scheduling module, the load characteristics and index mapping information are input into the Markov decision process. Through the deep reinforcement learning algorithm, the optimal index data distribution strategy between DRAM and NVM is learned. The data distribution strategy is formalized into data migration actions, including prefetching from NVM to DRAM and asynchronous write-back from DRAM to NVM. (4) B+ tree optimization: For the B+ tree index in NVM, a node splitting mechanism optimized for NVM is adopted. The new and old nodes are bidirectionally linked through the forward pointer and the backward pointer, and the parent node pointer is updated simultaneously through atomic write operations to avoid inconsistency in the splitting process. For leaf nodes, they are divided into the NVM persistence part of the key and the DRAM cache part of the value. When querying, the leaf is retrieved by the key, and the value in DRAM is located by the offset to achieve partial persistence. (5) Skip table optimization: For the skip table index in NVM, a hybrid index node is used. The NVM part uses a non-volatile pointer to persistently store the key and the forward pointer, and the DRAM part caches the file pointer and the volatile backward pointer. The top-level Head index header is introduced to traverse the search from the top level to the bottom level. The "Ping-Pong" dual buffer mechanism is introduced to achieve persistence and concurrent user access by switching buffers alternately. (6) Periodically refresh, periodically update the latest index in DRAM to NVM asynchronously through the persistence component, and generate a checkpoint with a version number. When the system fails, the index is restored from the latest checkpoint; (7) Incremental update: When updating data, the index modification is first completed in DRAM, and then the update is asynchronously written back to NVM. While providing better persistence guarantees, it reduces the write pressure of NVM and improves the performance of the critical path.

Images