A database index construction method and apparatus

Abstract

The application provides a database index construction method and device, belonging to the technical field of information storage, and the method comprises the following steps: in the case of receiving a data insertion request, inserting the to-be-inserted data into a to-be-inserted position; if the detection result of first detection is that a node of a global index is added, allocating the required address space for the added node based on the address space of a dynamic random access memory; if the detection result of second detection is that the parent node of the added node is located in a nonvolatile memory, inserting the added node into a quick index; if the used space proportion of the dynamic random access memory reaches a preset proportion, migrating the added node from the address space of the dynamic random access memory to the address space of the nonvolatile memory; wherein the global index and the quick index both adopt a skip list data structure. The application adopts the skip list data structure, optimizes the index structure of the in-memory database, and reduces the space consumption of the index on the DRAM under the premise of ensuring high data response speed.

Description

Technical Field

[0001] This invention relates to the field of information storage technology, and in particular to a database index construction method and apparatus. Background Technology

[0002] As the scale of data in cloud computing scenarios expands, the total amount of memory required for in-memory databases to run also increases. Since the performance of in-memory databases is directly limited by the amount of memory space they can use, when the data scale exceeds the available memory capacity of the in-memory database, database administrators can only choose to increase storage costs by purchasing additional hardware to increase DRAM (Dynamic Random Access Memory) capacity, or sacrifice performance by evicting some data to disk.

[0003] To mitigate database system performance degradation caused by data volume exceeding available memory capacity, much work revolves around partitioning data storage using the locality of reference characteristic of data access, moving cold data from DRAM to secondary storage media. However, even after cold data is removed from DRAM, the database system still needs to retain all indexes in memory. The DRAM space consumed by indexes can sometimes exceed half of the total database space consumption. Therefore, reducing DRAM space consumption by indexes while maintaining high data response speed is a current hot research topic. Summary of the Invention

[0004] In view of this, the present invention provides a database index construction method and apparatus to solve the problem of excessive DRAM space occupied by indexes, resulting in DRAM resource waste, while ensuring data response speed.

[0005] To address the aforementioned technical problems, in a first aspect, the present invention provides a database index construction method, applied to a hybrid memory system including dynamic random access memory and non-volatile memory, the method comprising:

[0006] Upon receiving a data insertion request, the system locates the insertion position of the data in the global index, inserts the data at the insertion position, and performs a first detection to check whether a node in the global index has been added.

[0007] If the first detection result indicates that a node in the global index has been added, the required address space is allocated to the new node based on the address space of the dynamic random access memory. The position of the parent node of the new node is then detected. If the second detection result indicates that the parent node of the new node is located in the non-volatile memory, the new node is inserted into the shortcut index.

[0008] The percentage of used space in the dynamic random access memory is obtained. If the percentage of used space in the dynamic random access memory reaches a preset ratio, the required address space is allocated to the new node based on the address space of the non-volatile memory, so as to migrate the new node from the address space of the dynamic random access memory to the address space of the non-volatile memory.

[0009] Both the global index and the shortcut index use a skip list data structure.

[0010] Optionally, locating the position of the data to be inserted in the global index upon receiving a data insertion request includes:

[0011] Set the height of the current node based on the height of the starting node, and use the starting node as the initial current node;

[0012] Acquisition steps: Based on the current node height, obtain the next node as the new current node, starting from the current node;

[0013] If the current node is not a null pointer and the corresponding key value is less than the search code of the data to be inserted, then the height of the current node is reduced by a preset value, and the acquisition step is executed until the current node is a null pointer or the corresponding key value is less than the search code of the data to be inserted, and the first verification is performed on whether the number of parent node groups of the current node is a null pointer.

[0014] If the first verification result indicates that the number of parent node groups of the current node is not a null pointer, a second verification is performed to check whether the height of the current node is zero. If the second verification result indicates that the height of the current node is zero, the current node is returned and the positioning ends. If the second verification result indicates that the height of the current node is not zero, the height of the current node is subtracted by a preset value, and the acquisition step is executed until the height of the current node is zero.

[0015] Optionally, if the detection result of the second detection is that the parent node of the newly added node is located in the non-volatile memory, inserting the newly added node into the shortcut index includes:

[0016] Set the current node as the jump head and set the current node height to a random height;

[0017] If the current node height exceeds the maximum height allowed by the non-volatile memory, obtain the portion of the current node height exceeding the maximum height allowed by the non-volatile memory, set each item of the parent node array index corresponding to the excess portion as the skip list header, and set the maximum height allowed by the non-volatile memory as the current node height.

[0018] Optionally, allocating the required address space for the new node based on the address space of the non-volatile memory, so as to migrate the new node from the address space of the dynamic random access memory to the address space of the non-volatile memory, includes:

[0019] If the parent node of the newly added node has a successor node in the dynamic random access memory, then the successor node is determined as the target node, the required address space is allocated to the target node based on the address space of the non-volatile memory, and the target node is migrated from the address space of the dynamic random access memory to the address space of the non-volatile memory.

[0020] Optionally, allocating the required address space for the new node based on the address space of the non-volatile memory, so as to migrate the new node from the address space of the dynamic random access memory to the address space of the non-volatile memory, includes:

[0021] Progressive steps: Obtain each successor node as the current node;

[0022] If the height of the current node is less than the height of the parent node of the newly added node in the non-volatile memory, then the current node is migrated to the non-volatile memory, and the progressive steps are executed until the height of the current node is greater than or equal to the height of the parent node in the non-volatile memory.

[0023] In a second aspect, the present invention provides a database index building apparatus applied to a hybrid memory system including dynamic random access memory and non-volatile memory, the apparatus comprising:

[0024] The first insertion module is used to locate the position where the data to be inserted is to be inserted in the global index when a data insertion request is received, insert the data to be inserted at the position where it is to be inserted, and perform a first detection on whether the node of the global index has been added.

[0025] The second insertion module is used to allocate the required address space for the newly added node based on the address space of the dynamic random access memory if the detection result of the first detection is that the node of the newly added node is added to the global index, perform a second detection on the position of the parent node of the newly added node, and insert the newly added node into the shortcut index if the detection result of the second detection is that the parent node of the newly added node is located in the non-volatile memory.

[0026] The migration module is used to obtain the percentage of used space of the dynamic random access memory. If the percentage of used space of the dynamic random access memory reaches a preset ratio, the module allocates the required address space for the new node based on the address space of the non-volatile memory, so as to migrate the new node from the address space of the dynamic random access memory to the address space of the non-volatile memory.

[0027] Both the global index and the shortcut index use a skip list data structure.

[0028] Optionally, the first insertion module includes:

[0029] The first acquisition unit is used to set the height of the current node according to the height of the starting node, and to use the starting node as the initial current node;

[0030] The second acquisition unit is used to acquire the following steps: based on the height of the current node, acquire the next node as the new current node starting from the current node;

[0031] The positioning unit is used to subtract a preset value from the height of the current node if the current node is not a null pointer and the corresponding key value is less than the search code of the data to be inserted, and to execute the acquisition step until the current node is a null pointer or the corresponding key value is less than the search code of the data to be inserted, and to perform a first verification on whether the number of parent node groups of the current node is a null pointer.

[0032] The positioning unit is further configured to perform a second verification on whether the height of the current node is zero if the verification result of the first verification is that the number of parent node groups of the current node is not a null pointer; if the verification result of the second verification is that the height of the current node is zero, return to the current node and end the positioning; if the verification result of the second verification is that the height of the current node is not zero, subtract a preset value from the height of the current node and execute the acquisition step until the height of the current node is zero.

[0033] Optionally, the first insertion module includes:

[0034] Set the current node as the jump head and set the current node height to a random height;

[0035] If the current node height exceeds the maximum height allowed by the non-volatile memory, obtain the portion of the current node height exceeding the maximum height allowed by the non-volatile memory, set each item of the parent node array index corresponding to the excess portion as the skip list header, and set the maximum height allowed by the non-volatile memory as the current node height.

[0036] Optionally, the migration module includes:

[0037] The migration unit is configured to, if the parent node of the newly added node has a successor node in the dynamic random access memory, determine the successor node as the target node, allocate the required address space for the target node based on the address space of the non-volatile memory, and migrate the target node from the address space of the dynamic random access memory to the address space of the non-volatile memory.

[0038] Optionally, the migration module includes:

[0039] The third acquisition unit is used for the progressive step: acquiring each successor node as the current node;

[0040] A stopping unit is configured to migrate the current node to the non-volatile memory if the height of the current node is less than the height of the parent node of the newly added node in the non-volatile memory, and execute the progressive steps until the height of the current node is greater than or equal to the height of the parent node in the non-volatile memory.

[0041] Thirdly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in any of the database index construction methods of the first aspect described above.

[0042] The beneficial effects of the above-described technical solution of the present invention are as follows:

[0043] In this embodiment of the invention, the nodes of the skip list data structure are composed of multiple pointer fields. During insertion, there are no splitting or merging operations; only the pointer fields are modified. Furthermore, there is no free space within the nodes. Therefore, this embodiment optimizes the index structure of a hybrid memory system including dynamic random access memory (DRAM) and non-volatile memory (NDRAM) by employing a skip list data structure, ensuring high data response speed. When the parent node of a new node stored in the address space of DRAM is located in NDRAM, the new node is inserted into the shortcut index. Since the skip list in the shortcut index structure can use the node's search code as the description information of the current node, indicating that all child nodes of the current node are larger than the current node, inserting the new node into the shortcut index speeds up the response speed to new data. When the used space ratio of DRAM reaches a preset proportion, the required address space is allocated to the new node based on the address space of NDRAM, and the new node is migrated from the address space of DRAM to the address space of NDRAM. This embodiment reduces the DRAM space consumption of the index while ensuring high data response speed. Attached Figure Description

[0044] Figure 1 This is a flowchart illustrating a database index construction method provided in Embodiment 1 of the present invention;

[0045] Figure 2 This is a schematic diagram of a skip list structure provided in Embodiment 1 of the present invention;

[0046] Figure 3 One of the schematic diagrams of the experimental results provided in Embodiment 1 of the present invention;

[0047] Figure 4 The second schematic diagram illustrates the experimental results provided in Embodiment 1 of the present invention;

[0048] Figure 5 The third schematic diagram of the experimental results provided in Embodiment 1 of the present invention;

[0049] Figure 6 The fourth schematic diagram of the experimental results provided in Embodiment 1 of the present invention;

[0050] Figure 7 Fifth schematic diagram of the experimental results provided in Embodiment 1 of the present invention;

[0051] Figure 8 Sixth schematic diagram of the experimental results provided in Embodiment 1 of the present invention;

[0052] Figure 9 Seventh schematic diagram of the experimental results provided in Embodiment 1 of the present invention;

[0053] Figure 10 Eighth schematic diagram of the experimental results provided in Embodiment 1 of the present invention;

[0054] Figure 11 This is a schematic diagram of a database index building device provided in Embodiment 2 of the present invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention are within the scope of protection of the present invention.

[0056] The relevant technologies are explained below.

[0057] As the scale of data in cloud computing scenarios expands, the total amount of memory required for in-memory databases to run also increases. Since the performance of in-memory databases is directly limited by the amount of memory space they can use, when the data scale exceeds the available memory capacity of the in-memory database, database administrators can only choose to increase storage costs by purchasing additional hardware to increase DRAM (Dynamic Random Access Memory) capacity, or sacrifice performance by evicting some data to disk.

[0058] To mitigate database system performance degradation caused by data volume exceeding available memory capacity, much work revolves around leveraging the locality of access to partition data storage and moving cold data from DRAM to secondary storage media. For example, H-Store's anti-caching architecture uses the LRU (Least Recently Used) mechanism to track data access patterns and asynchronously evict cold data to the disk's debuffer. VoltDB (an in-memory database that provides the scalability of NoSQL databases and the ACID consistency of traditional relational database systems) utilizes the operating system's virtual memory mechanism to migrate cold pages to disk. In HANA (High-Performance Analytic Appliance) systems, all new data is inserted into a row storage engine designed for OLTP (On-Line Transaction Processing) workloads, and as data ages, it is gradually moved to an optimized dictionary-compressed columnar storage. Hyper further categorizes data based on its access frequency, classifying it into hot data, warm data, cold data, and unchanging frozen data. Hot data is placed in small pages for quick location and querying, while frozen data is compressed and moved to large pages in secondary storage to achieve efficient and memory-friendly snapshots.

[0059] However, even though cold data has been removed from DRAM, the database system still needs to keep all indexes in memory. The DRAM space consumed by indexes can sometimes account for more than half of the total database space consumption. Therefore, how to reduce the DRAM space consumption of indexes is a hot research issue.

[0060] SILT storage system is a flash-based key-value store that achieves high performance and low memory footprint through a multi-level storage hierarchy index structure with different data structures. SILT's first layer is a LogStructured Structure supporting fast read and write operations; the second layer uses a traditional hash table for buffering. The hash table does not store the complete key, but rather stores a flag bit of the key to save memory. When querying a key, it first checks if the flag bit matches; if a match is found, it retrieves the key value from disk for comparison; the final layer is a prefix tree, which achieves very low memory overhead through compression algorithms. A non-relational dynamic hybrid indexing method dynamically partitions the index based on the frequency of field access. By analyzing the field access history within a statistical period, the weights of non-primary key fields are updated, dividing fields into high-frequency and low-frequency fields. Detailed indexes are built for high-frequency fields, and summary indexes are built for low-frequency fields, reducing the space overhead of indexing low-frequency fields. Another method saves memory space by building a two-stage index structure. The first-stage index is used to index new data, and a migration algorithm is periodically triggered to move the data from the first stage to the second-stage, read-optimized, memory-efficient index structure. The difference between these studies lies in how they determine what needs to be migrated and the mechanisms for migrating the data.

[0061] The above technical solution has the following limitations:

[0062] 1. Although cold data has been deleted from DRAM, the database system still needs to retain all indexes in memory. The DRAM space consumed by indexes can sometimes reach more than half of the total database space consumption, resulting in a significant waste of DRAM resources.

[0063] Second, partitioned storage of the index structure mainly refers to creating different indexes for data located on different storage media. This leads to increased costs in managing the index structure, data migration requires the maintenance of multiple indexes, and lacks scalability, making it difficult to apply this model to various database index structures.

[0064] Third, few existing technologies divide the index structure into regions based on the hotness or coldness of the data. This is because the read / write latency of traditional secondary storage media such as SSDs (Solid State Disks or Solid State Drives) is significantly different from that of DRAM. If the index portion is placed in an SSD, it will inevitably cause serious performance loss.

[0065] The emergence of new hardware such as non-volatile memory (NVM) has made indexed partitioned storage schemes possible.

[0066] Therefore, please refer to Figure 1 , Figure 1 This is a flowchart illustrating a database index construction method according to Embodiment 1 of the present invention. The method is applied to a hybrid memory system including dynamic random access memory and non-volatile memory, and includes the following steps:

[0067] Step 11: Upon receiving a data insertion request, locate the position where the data to be inserted is to be inserted in the global index, insert the data to be inserted at the position, and perform a first detection on whether the node in the global index has been added;

[0068] Step 12: If the detection result of the first detection is that a node of the global index has been added, the required address space is allocated to the new node based on the address space of the dynamic random access memory, and the position of the parent node of the new node is detected in the second detection. If the detection result of the second detection is that the parent node of the new node is located in the non-volatile memory, the new node is inserted into the shortcut index.

[0069] Step 13: Obtain the percentage of used space in the dynamic random access memory. If the percentage of used space in the dynamic random access memory reaches a preset ratio, allocate the required address space for the new node based on the address space of the non-volatile memory, so as to migrate the new node from the address space of the dynamic random access memory to the address space of the non-volatile memory.

[0070] Both the global index and the shortcut index use a skip list data structure.

[0071] In this embodiment of the invention, the database index construction method includes two stages. The first stage is the data insertion receiving stage. When a data insertion request is received, the target node for insertion can be searched downwards from the root node of the global index. When the insertion of the data to be inserted triggers an increase in the number of nodes in the global index, the address space required for the new node is allocated in DRAM. If the parent node of the new node is located in the NVM address space, the new node is inserted into the shortcut index for maintenance to speed up the response to new data. The second stage is the migration stage. That is, when the DRAM space occupancy reaches a certain proportion, a migration thread is triggered to migrate the new node (i.e., the target node) to the NVM space, thereby reducing DRAM space usage.

[0072] In this embodiment, both the global index and the shortcut index adopt a skip list data structure, which supports efficient insertion, query and deletion operations.

[0073] In this embodiment, optionally, two index data structures are used: a global index and a shortcut index. The global index is used to maintain all the data in the database and is built in the NVM address space, while the shortcut index is used to maintain newly added data and nodes in order to speed up the response speed of new data query updates.

[0074] In this embodiment, optionally, for DRAM nodes and NVM nodes with global indexes, the nodes of the skip list data structure are composed of multiple pointer fields. There is no splitting or merging operation during the insertion process, only the pointer fields are modified, and there is no empty space inside the node.

[0075] In this embodiment, optionally, for a shortcut index using a skip list data structure, the search key of a node can be used as the descriptive information of the current node, indicating that all child nodes of the current node are larger than the current node. Specifically, the skip list maintains the order of the linked list at different levels, and the search key values ​​of subsequent nodes are all larger than the search key value of the current node. Therefore, the search key of a node can be used as the descriptive information of the current node to indicate that all child nodes of the current node are larger than the current node.

[0076] In this embodiment of the invention, by employing a skip list data structure, the index structure of a hybrid memory system including dynamic random access memory (DRAM) and non-volatile memory (NDRAM) is optimized. When the parent node of a new node stored in the address space of DRAM is located in NDRAM, the new node is inserted into the shortcut index. When the used space ratio of DRAM reaches a preset proportion, the required address space is allocated to the new node based on the address space of NDRAM, and the new node is migrated from the address space of DRAM to the address space of NDRAM. This embodiment of the invention reduces the space consumption of the index on DRAM while ensuring high data response speed.

[0077] The following example illustrates the database index construction method described above.

[0078] In this embodiment of the invention, the NVM_SkipList (non-volatile memory skip list) used contains members key, value, and height, representing the search code, data value, and node height of the node, respectively. The next array represents successor nodes at different levels, and the getNext and setNext functions are used to get and set the successor node value at a certain height, respectively. NVM_SKipList has a global index root node member nvmRoot and a shortcut index root node member dramRoot. In addition, maxNVMHeight (maximum height of non-volatile memory) and maxDRAMHeight (maximum height of non-volatile memory) record the maximum height of the current two indices (global index and shortcut index), respectively.

[0079] Please refer to Figure 2 , Figure 2 This is a schematic diagram of a skip list structure provided in Embodiment 1 of the present invention. Figure 2 As shown, in Figure 2 When performing a query operation in a skip list, the following process is included:

[0080] Step 1.1: Let the current node be the head node, and the current level be the highest height of the skip list (corresponding to level=4 in the diagram);

[0081] Step 1.2: Check the successor node of the current node at level 1. If the successor node is a null pointer or the search key of the successor node is greater than the target search key, subtract one from level 1 and return to step 1.2. If the search key value of the successor node is less than the target search key, set the current node as the successor node and return to step 1.2. If the search key of the successor node is equal to the target search key, return the result directly.

[0082] The process of performing an insertion operation on a skip list is similar to the query process. When searching for the insertion position, the set of parent nodes is recorded. Figure 2 Taking node5 as an example, the set of parent nodes of node5 is {node4, node2, head}. Then, a random height value is assigned to the new node as the number of pointer fields, and then the pointer of the successor node of the corresponding parent node is modified to the newly added node.

[0083] In one optional implementation, locating the position of the data to be inserted in the global index upon receiving a data insertion request includes:

[0084] Set the height of the current node based on the height of the starting node, and use the starting node as the initial current node;

[0085] Acquisition steps: Based on the current node height, obtain the next node as the new current node, starting from the current node;

[0086] If the current node is not a null pointer and the corresponding key value is less than the search code of the data to be inserted, then the height of the current node is reduced by a preset value, and the acquisition step is executed until the current node is a null pointer or the corresponding key value is less than the search code of the data to be inserted, and the first verification is performed on whether the number of parent node groups of the current node is a null pointer.

[0087] If the first verification result indicates that the number of parent node groups of the current node is not a null pointer, a second verification is performed to check whether the height of the current node is zero. If the second verification result indicates that the height of the current node is zero, the current node is returned and the positioning ends. If the second verification result indicates that the height of the current node is not zero, the height of the current node is subtracted by a preset value, and the acquisition step is executed until the height of the current node is zero.

[0088] Specifically, since each node in a skip list can only store one piece of data, the insertion of new data triggers the creation of a new node. When constructing a new node, the first step is to locate the target position, which can be done using the `findGreaterOrEqual` method, and includes the following steps:

[0089] Step 2.1: Based on the height of the starting node, subtract one from the height of the starting node and set it as the current height;

[0090] Step 2.2: Based on the current height, obtain the next node starting from the starting node;

[0091] Step 2.3: If the next node is not a null pointer and the corresponding key value is less than the search code of the data to be inserted, then jump to step 2.4; otherwise, jump to step 2.5.

[0092] Step 2.4: Set x as the next node and jump to step 2.2, where x is the node pointer assignment in the pseudocode;

[0093] Step 2.5: If the parent node count is not a null pointer, then set the index of the parent node array corresponding to the current height to x;

[0094] Step 2.6: If the current height is 0, return to the next node and the positioning ends; otherwise, decrease the current height by one and jump to step 2.2.

[0095] In other embodiments of the present invention, if the detection result of the second detection is that the parent node of the newly added node is located in the non-volatile memory, inserting the newly added node into the shortcut index includes:

[0096] Set the current node as the jump head and set the current node height to a random height;

[0097] If the current node height exceeds the maximum height allowed by the non-volatile memory, obtain the portion of the current node height exceeding the maximum height allowed by the non-volatile memory, set each item of the parent node array index corresponding to the excess portion as the skip list header, and set the maximum height allowed by the non-volatile memory as the current node height.

[0098] In this embodiment, as mentioned earlier, a skip list node has a parent node array (parent node set) consisting of multiple parent nodes. Therefore, in the first stage of database construction, it is necessary to consider how to select nodes from the parent node set to add to the shortcut index. Figure 2 Taking node5 as an example, there are three options for the parent node combination to be added to the shortcut index. If node4 is added, then when node5 is migrated to NVM and its address changes, node2 and node head cannot modify the corresponding successor node pointers, causing an error in the program. Therefore, only parent nodes at the same level as the node's height can be selected. That is, for node5, node head needs to be added to the shortcut index, and for node3, node2 needs to be added to the shortcut index.

[0099] Specifically, inserting a new node into the shortcut index includes the following steps:

[0100] Step 3.1: Set the current node as the head of the skip list;

[0101] Step 3.2: Set x to the node located based on the search code of the data to be inserted, the previous node, and the jump table header;

[0102] Step 3.3: Set the current height to a random height;

[0103] Step 3.4: If the current height exceeds the maximum height allowed by NVM, set the flag to true; otherwise, set it to false.

[0104] Step 3.5: If the current height exceeds the maximum height allowed by NVM, set each item of the parent node array index corresponding to the part of the current height that exceeds the maximum height allowed by NVM as the head of the skip list, and set the maximum height allowed by NVM as the current height;

[0105] Step 3.6: If flag is true, or the first item in the parent node array index is the skip list head, or the last item in the parent node array index is the skip list head, then proceed to step 3.7; otherwise, proceed to step 3.8.

[0106] Step 3.7: Set x as the new NVM node, then proceed to step 3.9:

[0107] Step 3.8: Set x as the new node, then proceed to step 3.9;

[0108] Step 3.9: For indices starting from 0 up to the current height, set the next value of x to the next value of the corresponding item in the parent node array, and then set the next value of the corresponding item in the parent node array to x.

[0109] Step 3.10: If flag is true, or the first item of the parent node array index is a skip list header, or the last item of the parent node array index is a skip list header, set the parent node to the last item of the parent node array index, and call the Insert Shortcut function to insert the search code of the parent node.

[0110] In some embodiments of the present invention, allocating the required address space for the newly added node based on the address space of the non-volatile memory, so as to migrate the newly added node from the address space of the dynamic random access memory to the address space of the non-volatile memory, includes:

[0111] If the parent node of the newly added node has a successor node in the dynamic random access memory, then the successor node is determined as the target node, the required address space is allocated to the target node based on the address space of the non-volatile memory, and the target node is migrated from the address space of the dynamic random access memory to the address space of the non-volatile memory.

[0112] In this embodiment, the second stage of database construction needs to consider when to stop the migration iteration of the current node. For each NVM parent node that needs to be migrated, it is first determined whether it has a DRAM successor node. If it does not exist, the process can be returned directly, i.e., the migration is stopped; otherwise, the migration of the successor node is performed.

[0113] In other embodiments of the present invention, allocating the required address space for the newly added node based on the address space of the non-volatile memory, so as to migrate the newly added node from the address space of the dynamic random access memory to the address space of the non-volatile memory, includes:

[0114] Progressive steps: Obtain each successor node as the current node;

[0115] If the height of the current node is less than the height of the parent node of the newly added node in the non-volatile memory, then the current node is migrated to the non-volatile memory, and the progressive steps are executed until the height of the current node is greater than or equal to the height of the parent node in the non-volatile memory.

[0116] For example, let `height` be the height of the NVM parent node, and initialize the `prev` array. The `getNext(0)` function is called to continuously obtain successor nodes. If the height of a successor node exceeds the `height` value, or if the successor node's height is equal to `height` and it is an NVM node, the iteration terminates. For successor nodes located in the DRAM address space, the `allocateNVMNode` function is called to migrate the node to the NVM address space, then the value of the `prev` array is modified, and the iterative migration of nodes continues.

[0117] In this embodiment, optionally, the iterative migration of the skip list structure includes the following steps:

[0118] Step 4.1: Initialize the height of the current node and declare the array of parent nodes;

[0119] Step 4.2: Determine if the current node has a DRAM successor node. If it does, proceed to step 4.11; otherwise, proceed to step 4.3.

[0120] Step 4.3: Initialize the parent node set;

[0121] Step 4.4: Obtain the successor node of the current node. If it is empty, set each item of the parent node array index corresponding to the part of the current height that exceeds the maximum height allowed by NVM as the head of the skip list, and set the maximum height allowed by NVM as the current height. If it is not empty, go to step 4.5.

[0122] Step 4.5: Obtain the height of the successor node. If it exceeds the height of the current node, proceed to step 4.6. If it is equal to the height of the current node and the successor node is an NVM node, proceed to step 4.11.

[0123] Step 4.6: Determine if a DRAM successor node exists. If it exists, proceed to step 4.7; otherwise, proceed to step 4.11.

[0124] Step 4.7: Insert a shortcut index for the successor node and proceed to step 4.11;

[0125] Step 4.8: Determine whether the successor node is an NVM node. If it is an NVM node, proceed to step 4.9; otherwise, proceed to step 4.10.

[0126] Step 4.9: Update the parent node set, set the successor node as the current node, and repeat step 4.4;

[0127] Step 4.10: Migrate the successor node to the NVM address space, update the parent node set, and repeat step 4.4;

[0128] Step 4.11: The iterative migration operation ends.

[0129] Please refer to Figures 3 to 10 , Figure 3 This is one of the schematic diagrams of the experimental results provided in Embodiment 1 of the present invention. Figure 4 This is the second schematic diagram of the experimental results provided in Embodiment 1 of the present invention. Figure 5 This is the third schematic diagram of the experimental results provided in Embodiment 1 of the present invention. Figure 6 The fourth schematic diagram of the experimental results provided in Embodiment 1 of the present invention Figure 7The fifth schematic diagram of the experimental results provided in Embodiment 1 of the present invention Figure 8 This is the sixth schematic diagram of the experimental results provided in Embodiment 1 of the present invention. Figure 9 The seventh schematic diagram of the experimental results provided in Embodiment 1 of the present invention. Figure 10 This is the eighth illustration of the experimental results provided in Embodiment 1 of the present invention. In this embodiment, the Yahoo! Cloud Serving Benchmark (YCSB) test benchmark program is used to evaluate the performance of the index structure in this embodiment. YCSB is a tool for basic testing of cloud services, providing six different OLTP workloads with varying read / write ratios. In this embodiment, a read-write balanced workload, a read-only workload, and a read-intensive workload are selected to test the index structure. The data initialization phase of the workload is tested and analyzed as a write-only workload. For each workload, two basic search codes, 64-bit monotonically increasing integers and 64-bit random integers, are used for the experiment, with the index value being a 64-bit integer. The data locality of the workload follows a Latest distribution pattern, with a distribution amplitude of 10%, meaning that 90% of the accesses are concentrated on 10% of the data. The workload models mainly include four types: insert-only, balanced, read-only, and read-heavy. The search code types include 64-bit monotonically increasing integers (Mono) and 64-bit random integers (Rand). In this embodiment, the comparison schemes include the original index structure and a publicly disclosed two-stage index compression scheme. For ease of distinction, the original index structure scheme, the two-stage index compression scheme, and the scheme in this embodiment are respectively referred to as SkipList, Hybrid, and NVM.

[0130] like Figures 3 to 10 As shown in the figures, the performance metric in each figure is throughput. The figure title indicates the configuration of the test workload. The horizontal axis represents the number of threads (threands), and the vertical axis represents throughput (throughput). Curve ① represents the original index structure scheme, curve ② represents the two-stage index compression scheme, and curve ③ represents the scheme in this embodiment of the invention. Through throughput testing, it can be seen that when the search code is an auto-incrementing integer, the scheme in this embodiment of the invention can achieve a throughput comparable to the original index structure scheme while reducing DRAM resource usage. Compared with the two-stage index compression scheme, the scheme in this embodiment of the invention exhibits superior throughput performance. However, when the search code is a random integer, the scheme in this embodiment of the invention suffers some performance loss. This is because random integers lead to uncertainty in the position of newly added nodes, resulting in frequent quick index insertions and thus a decrease in overall performance.

[0131] Write-only workloads: The two-stage index compression scheme mainly requires periodic node migration. Experimental results show that both the scheme in this embodiment and the two-stage index compression scheme have some performance loss for write-only workloads. However, under different workload scenarios, the scheme in this embodiment outperforms the two-stage index compression scheme. This is because the two-stage index compression scheme blocks normal database requests during migration until the migration is complete, while the scheme in this embodiment only engages in lock contention with requests involving the migration nodes during migration, resulting in a smaller performance impact than the two-stage index compression scheme.

[0132] Read / write balanced workload: When running a read / write balanced workload, the scheme in this embodiment is superior to the two-stage index compression scheme under different search keys and index structure settings. This is because the two-stage index compression scheme inserts new data into the dynamic index structure for update requests and then merges it with the static structure, leading to a higher migration frequency and performance degradation. Comparison with the original index structure requires distinguishing between different search keys. When the search key is an auto-incrementing integer, the scheme in this embodiment achieves a throughput similar to the original, while performance decreases when the search key is a random integer.

[0133] Read-only workloads: The problem with the two-stage index compression scheme is that a single query request may traverse two index structures, which will reduce the overall query performance. At the same time, since most of the nodes of the global index are located in the NVM address space, the read and write latency of NVM will further cause performance loss.

[0134] Read-intensive workloads: The solution in this embodiment of the invention is superior to the two-stage index compression solution on all workloads.

[0135] To more intuitively compare the storage cost reduction effect of the solution in this embodiment of the invention compared with the two existing solutions, Table 1 below summarizes the memory consumption of each solution. Table 1 compares the storage costs of the three solutions. When calculating the storage cost, the DRAM storage cost per GB (Gigabyte) is 1, and the NVM storage cost per GB is 0.3.

[0136] Table 1: Comparison of Database Memory and Storage Costs

[0137]

[0138] As can be seen from Table 1 above, for six different index structures and workloads, compared with the original index structure, the solutions in the embodiments of the present invention can reduce storage costs by 59% and 29%, respectively; compared with the two-stage index compression solution, the solutions in the embodiments of the present invention can reduce storage costs by 59% and 29%, respectively.

[0139] The above experimental results demonstrate that the database index construction method based on DRAM-NVM hybrid memory and skip list structure in the embodiments of the present invention can provide higher index performance in databases supported by hybrid memory systems, improve the effective utilization rate of high-speed DRAM resources in hybrid memory, and reduce the memory cost of the database system from the overall design perspective compared with the prior art.

[0140] In summary, this embodiment of the invention optimizes the index structure of the memory-based database by employing a skip list data structure. It places hot data and its index in DRAM, which has a faster read and write speed, and places cold data with low access frequency and its index in NVM. This allows the database to accommodate more data while maintaining high throughput, saving a significant amount of DRAM space, improving the scalability of the index structure, and enhancing database performance and resource utilization.

[0141] Please see Figure 11 , Figure 11 This is a schematic diagram of a database index building device according to Embodiment 2 of the present invention, applied to a hybrid memory system including dynamic random access memory and non-volatile memory. The device 110 includes:

[0142] The first insertion module 111 is used to locate the position of the data to be inserted in the global index when a data insertion request is received, and if the data to be inserted causes the number of nodes in the global index to increase, allocate the address space required for the new node in the dynamic random access memory.

[0143] The second insertion module 112 is used to insert the new node into the shortcut index if the parent node of the new node is located in the non-volatile memory.

[0144] Migration module 113 is used to migrate the target node to the non-volatile memory when the used space ratio of the dynamic random access memory reaches a preset ratio.

[0145] Both the global index and the shortcut index use a skip list data structure.

[0146] In this embodiment of the invention, by adopting a skip list data structure, the index structure of the memory-based database is optimized. Hot data and its index are placed in DRAM with faster read and write speeds, while cold data with low access frequency and its index are placed in NVM. This allows the database to accommodate more data while maintaining high throughput, saving a significant amount of DRAM space, improving the scalability of the index structure, and enhancing database performance and resource utilization.

[0147] Optionally, the first insertion module includes:

[0148] The first acquisition unit is used to set the height of the current node according to the height of the starting node, and to acquire the next node as the current node starting from the starting node;

[0149] The second acquisition unit is used to reduce the height of the current node by one and continue to acquire the next node as the current node if the current node is not a null pointer and the corresponding key value is less than the search code of the data to be inserted, until the current node is a null pointer or the corresponding key value is less than the search code of the data to be inserted, and the parent node array of the current node is not a null pointer.

[0150] The positioning unit is configured to, if the current node is a null pointer or the corresponding key value is not less than the search code of the data to be inserted, and the parent node array of the current node is not a null pointer, return the current node and end the positioning when the current node height is zero, and decrement the current node height by one when the current node height is not zero, and continue to obtain the next node as the current node until the current node height is zero.

[0151] Optionally, the second insertion module includes:

[0152] The first setting unit is used to set the current node as the jump table header and set the current node height to a random height;

[0153] The second setting unit is configured to, if the current node height exceeds the maximum height allowed by the non-volatile memory, set each item of the parent node array index corresponding to the portion of the current node height exceeding the maximum height allowed by the non-volatile memory as the head of the skip list, and set the maximum height allowed by the non-volatile memory as the current node height.

[0154] The insertion unit is used to set the parent node of the newly added node as the last item of the parent node array index, and call the insertion shortcut function to insert the search code of the parent node of the newly added node into the shortcut index.

[0155] Optionally, the migration module includes:

[0156] The migration unit is configured to, if the parent node of the newly added node has a successor node in the dynamic random access memory, determine the successor node as the target node and migrate it to the non-volatile memory.

[0157] Optionally, the migration module includes:

[0158] The third acquisition unit is used to acquire each successor node as the current node;

[0159] The stopping unit is configured to migrate the current node to the non-volatile memory if the height of the current node is less than the height of the parent node of the newly added node in the non-volatile memory, and to stop acquiring the successor node and stop migrating the current node to the non-volatile memory if the height of the current node is greater than or equal to the height of the parent node in the non-volatile memory.

[0160] The embodiments of the present invention are product embodiments corresponding to the method embodiment one described above, and therefore will not be repeated here. For details, please refer to the embodiment one described above.

[0161] Embodiment 3 of the present invention provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the steps of any of the database index construction methods in Embodiment 1 above. For details, please refer to the description of the method steps in the corresponding embodiments above.

[0162] The aforementioned computer-readable storage media include both permanent and non-permanent, removable and non-removable media, which can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0163] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A database index construction method, characterized in that, Applied to a hybrid memory system including dynamic random access memory and non-volatile memory, the method includes: Upon receiving a data insertion request, the system locates the insertion position of the data in the global index, inserts the data at the insertion position, and performs a first detection to check whether a node in the global index has been added. If the first detection result indicates that a node in the global index has been added, the required address space is allocated to the new node based on the address space of the dynamic random access memory. The position of the parent node of the new node is then detected. If the second detection result indicates that the parent node of the new node is located in the non-volatile memory, the new node is inserted into the shortcut index. Obtain the percentage of used space in the dynamic random access memory. If the percentage of used space in the dynamic random access memory reaches a preset ratio, allocate the required address space for the new node based on the address space of the non-volatile memory, so as to migrate the new node from the address space of the dynamic random access memory to the address space of the non-volatile memory. Both the global index and the shortcut index use a skip list data structure.

2. The method according to claim 1, characterized in that, The step of locating the position of the data to be inserted in the global index upon receiving a data insertion request includes: Set the height of the current node based on the height of the starting node, and use the starting node as the initial current node; Acquisition steps: Based on the current node height, obtain the next node as the new current node, starting from the current node; If the current node is not a null pointer and the corresponding key value is less than the search code of the data to be inserted, then the height of the current node is reduced by a preset value, and the acquisition step is executed until the current node is a null pointer or the corresponding key value is less than the search code of the data to be inserted, and the first verification is performed on whether the number of parent node groups of the current node is a null pointer. If the first verification result indicates that the number of parent node groups of the current node is not a null pointer, a second verification is performed to check whether the height of the current node is zero. If the second verification result indicates that the height of the current node is zero, the current node is returned and the positioning ends. If the second verification result indicates that the height of the current node is not zero, the height of the current node is subtracted by a preset value, and the acquisition step is executed until the height of the current node is zero.

3. The method according to claim 2, characterized in that, If the detection result of the second detection is that the parent node of the newly added node is located in the non-volatile memory, inserting the newly added node into the shortcut index includes: Set the current node as the jump head and set the current node height to a random height; If the current node height exceeds the maximum height allowed by the non-volatile memory, obtain the portion of the current node height exceeding the maximum height allowed by the non-volatile memory, set each item of the parent node array index corresponding to the excess portion as the skip list header, and set the maximum height allowed by the non-volatile memory as the current node height.

4. The method according to claim 1, characterized in that, Allocating the required address space for the new node based on the address space of the non-volatile memory, so as to migrate the new node from the address space of the dynamic random access memory to the address space of the non-volatile memory, includes: If the parent node of the newly added node has a successor node in the dynamic random access memory, then the successor node is determined as the target node, the required address space is allocated to the target node based on the address space of the non-volatile memory, and the target node is migrated from the address space of the dynamic random access memory to the address space of the non-volatile memory.

5. The method according to claim 4, characterized in that, Allocating the required address space for the new node based on the address space of the non-volatile memory, so as to migrate the new node from the address space of the dynamic random access memory to the address space of the non-volatile memory, includes: Progressive steps: Obtain each successor node as the current node; If the height of the current node is less than the height of the parent node of the newly added node in the non-volatile memory, then the current node is migrated to the non-volatile memory, and the progressive steps are executed until the height of the current node is greater than or equal to the height of the parent node in the non-volatile memory.

6. A database index building apparatus, characterized in that, The device, applicable to a hybrid memory system including dynamic random access memory and non-volatile memory, comprises: The first insertion module is used to locate the position where the data to be inserted is to be inserted in the global index when a data insertion request is received, insert the data to be inserted at the position where it is to be inserted, and perform a first detection on whether the node of the global index has been added. The second insertion module is used to allocate the required address space for the newly added node based on the address space of the dynamic random access memory if the detection result of the first detection is that the node of the newly added node is added to the global index, perform a second detection on the position of the parent node of the newly added node, and insert the newly added node into the shortcut index if the detection result of the second detection is that the parent node of the newly added node is located in the non-volatile memory. The migration module is used to obtain the percentage of used space of the dynamic random access memory. If the percentage of used space of the dynamic random access memory reaches a preset ratio, the module allocates the required address space for the new node based on the address space of the non-volatile memory, so as to migrate the new node from the address space of the dynamic random access memory to the address space of the non-volatile memory. Both the global index and the shortcut index use a skip list data structure.

7. The apparatus according to claim 6, characterized in that, The first insertion module includes: The first acquisition unit is used to set the height of the current node according to the height of the starting node, and to use the starting node as the initial current node; The second acquisition unit is used to acquire the following steps: based on the height of the current node, acquire the next node as the new current node starting from the current node; The positioning unit is used to subtract a preset value from the height of the current node if the current node is not a null pointer and the corresponding key value is less than the search code of the data to be inserted, and to execute the acquisition step until the current node is a null pointer or the corresponding key value is less than the search code of the data to be inserted, and to perform a first verification on whether the number of parent node groups of the current node is a null pointer. The positioning unit is further configured to perform a second verification on whether the height of the current node is zero if the verification result of the first verification is that the number of parent node groups of the current node is not a null pointer; if the verification result of the second verification is that the height of the current node is zero, return to the current node and end the positioning; if the verification result of the second verification is that the height of the current node is not zero, subtract a preset value from the height of the current node and execute the acquisition step until the height of the current node is zero.

8. The apparatus according to claim 7, characterized in that, The first insertion module includes: Set the current node as the jump head and set the current node height to a random height; If the current node height exceeds the maximum height allowed by the non-volatile memory, obtain the portion of the current node height exceeding the maximum height allowed by the non-volatile memory, set each item of the parent node array index corresponding to the excess portion as the skip list header, and set the maximum height allowed by the non-volatile memory as the current node height.

9. The apparatus according to claim 6, characterized in that, The migration module includes: The migration unit is configured to, if the parent node of the newly added node has a successor node in the dynamic random access memory, determine the successor node as the target node, allocate the required address space for the target node based on the address space of the non-volatile memory, and migrate the target node from the address space of the dynamic random access memory to the address space of the non-volatile memory.

10. The apparatus according to claim 9, characterized in that, The migration module includes: The third acquisition unit is used for the progressive step: acquiring each successor node as the current node; A stopping unit is configured to migrate the current node to the non-volatile memory if the height of the current node is less than the height of the parent node of the newly added node in the non-volatile memory, and execute the progressive steps until the height of the current node is greater than or equal to the height of the parent node in the non-volatile memory.

Images