A graph data heterogeneous hierarchical storage structure based on persistent memory and a method thereof

Abstract

The application discloses a kind of graph data heterogenous layered storage structure and method based on persistent memory, the application uses layered hash table heterogenous storage structure, stores the equivalent query data structure in topology data and index data in graph data, simultaneously using a layered storage B+Tree storage structure, stores the range query data structure in index data, solves the problem of query speed and limited memory space, simultaneously by converting the random write of persistent memory into sequential write, reduce write delay, by constructing the architecture of layered storage, reduce the write times of data structure in persistent memory, improve query efficiency.

Description

Technical Field

[0001] This invention belongs to the field of computer software technology, specifically relating to a heterogeneous hierarchical storage structure and method for graph data based on persistent memory. Background Technology

[0002] In existing graph storage and query technologies, data persistence and retrieval typically involve persisting data to disk and then loading it into memory using lazy loading or full loading methods for computation before returning the result to the client. Storing data on disk inevitably incurs significant waiting time when reading it. While lazy loading distributes I / O time across each query, which is beneficial for memory management and limiting memory usage, the nature of graph computation means that the number of edges accessed during the Expand operation increases exponentially. After only a few Expand operations, traversing the entire graph may become necessary, severely slowing down query speeds if lazy loading is used. Full loading is a crucial method for addressing graph query latency, but it requires loading all data into memory during program initialization, significantly slowing down startup and potentially leading to memory exhaustion due to limited memory space. Therefore, migrating data to persistent devices with higher read / write speeds is a growing trend. However, simply copying data structures from memory to persistent memory can create significant performance bottlenecks due to the asymmetry between read and write operations in persistent memory. Summary of the Invention

[0003] To address the problems of slow query speed, limited memory space, or severe performance bottlenecks caused by numerous writes to persistent memory in existing technologies, this invention provides a heterogeneous hierarchical storage structure and method for graph data based on persistent memory. This invention employs a hierarchical hash table heterogeneous storage structure to store topological data and equality query data structures in the index data of the graph data. Simultaneously, it uses a hierarchical B+Tree storage structure to store range query data structures in the index data, thus solving the problems of query speed and limited memory space. By constructing a hierarchical storage architecture, the number of writes to data structures in persistent memory is reduced.

[0004] This invention is achieved through the following technical solution:

[0005] A heterogeneous hierarchical storage structure for graph data based on persistent memory, the heterogeneous hierarchical storage structure for graph data includes a hierarchical heterogeneous hash table and a hierarchical heterogeneous B+ tree;

[0006] Among them, the topological data and index data of the graph data are stored using hierarchical heterogeneous hash tables.

[0007] The range query data structure in the index data of the graph data uses a hierarchical heterogeneous B+ tree for storage.

[0008] In existing graph storage and query technologies, lazy loading significantly reduces query speed; full loading requires loading all data into memory during program initialization, severely slowing down startup and potentially leading to memory exhaustion due to limited memory space; furthermore, numerous writes to persistent memory can create serious performance bottlenecks. The heterogeneous hierarchical storage structure proposed in this invention, based on the characteristics of graph data, constructs a hierarchical heterogeneous hash table to store the equality query data structure in the topology and index data, and a hierarchical heterogeneous B+ tree to store the range query data structure in the index data, reducing the number of writes to persistent memory data structures and improving query speed.

[0009] In a preferred embodiment, the hierarchical heterogeneous hash table of the present invention is divided into an upper layer and a lower layer, with the size of the lower layer being half the size of the upper layer, that is, every two buckets in the upper layer correspond to one bucket in the lower layer.

[0010] The hierarchical heterogeneous hash table uses WAL logs for add, delete, and modify operations.

[0011] In a preferred embodiment, each bucket of the present invention contains 8 key-value pairs. When a key is written to the hash table, the system first checks whether the key exists in the current bucket. If it does not exist, the system continues to check whether the key exists in the next lower level. If it does not exist, the key is written directly. If both the upper and lower buckets are full, the lower level is set as a temporary storage level, the upper level is set as the lower level, and the rehashing operation begins. The new upper level is allocated twice the size of the original upper level, and the keys in the temporary storage level are rehashed to the upper level.

[0012] As a preferred embodiment, the hierarchical heterogeneous B+ tree of the present invention uses DRAM and persistent memory for hierarchical storage. The leaf layer of B+ trees, that is, all leaf nodes, is stored in persistent memory, while the upper-level nodes are stored in DRAM.

[0013] Each leaf node stores 16 key-value pairs of 256 bytes each, meaning each leaf node is 4KB in size.

[0014] The hierarchical heterogeneous B+ tree uses WAL logs for add, delete, and modify operations.

[0015] In a preferred embodiment, when performing add, delete, or modify operations, the present invention first writes the modified leaf node into DRAM, points the pointer of its parent node to DRAM, then records the operation in the WAL log in persistent memory, starts a thread that applies the log periodically, and applies the operation recorded in the log to the leaf node stored in persistent memory through transactions. When a log is committed to persistent memory, the pointer of the parent node in DRAM is pointed to the updated node in persistent memory.

[0016] In a preferred embodiment, the WAL log of the present invention is 256 bytes in size, and the fields include: 2 bytes of CRC16 checksum, 1 byte of operation type OPTYPE, 1 byte of value length VALUE_LEN, 4 bytes of sequence code SERIAL_NUM, 8 bytes of key content KEY_CONTENT, and a maximum of 240 bytes of value content VALUE_CONTENT; each log entry is appended to the end of the log.

[0017] Secondly, the present invention also proposes a data storage method based on the above-mentioned heterogeneous hierarchical storage structure for graph data, the data storage method comprising:

[0018] When the graph database process starts, the hierarchical heterogeneous B+ tree first reads the leaf nodes in persistent memory and constructs the upper-level nodes of the B+ tree in DRAM, while the lower-level pointers point to the leaf nodes in persistent memory.

[0019] When a graph topology and its node attributes are imported, two hierarchical heterogeneous hash tables are first constructed to store the mapping between node IDs and their source and destination edges, and the mapping between edge IDs and their source and destination nodes, respectively.

[0020] Based on the node's attribute type, either create a hierarchical heterogeneous hash table to store the equal-value index or create a hierarchical heterogeneous B+ tree to store the range query index.

[0021] In a preferred embodiment, when writing data to a hierarchical heterogeneous hash table, the present invention:

[0022] Write key-value pairs to the DRAM hash table for temporary storage;

[0023] At the same time, the key-value pairs are written to the WAL log stored in persistent memory;

[0024] A background thread periodically parses the WAL log and records the changes in a persistent in-memory hash table.

[0025] Release the space temporarily used by the DRAM hash table for key-value pairs;

[0026] When writing data to a hierarchical heterogeneous B+ tree:

[0027] If there is enough space in the leaf nodes and no node splitting or merging is triggered, the corresponding leaf node written in the persistent memory is copied into the DRAM, the value is written to the leaf node, and the upper-level pointer is set to point to the leaf node; if there is not enough space in the leaf nodes, two new leaf nodes are generated in the DRAM, and the upper-level pointer is set to point to these two new leaf nodes.

[0028] Write the key-value pairs to the WAL log stored in persistent memory;

[0029] A background thread periodically parses the WAL log and moves the changes to the leaf nodes in persistent memory.

[0030] Release the space of the leaf nodes temporarily stored in the B+ tree in DRAM.

[0031] Thirdly, the present invention also proposes a data query method based on the above-mentioned heterogeneous hierarchical storage structure for graph data, the data query method comprising:

[0032] When you need to access a hierarchical heterogeneous hash table:

[0033] Its hash_key is calculated using a hash function;

[0034] Search the DRAM hash table. If the hash_key does not exist, continue to the next step. If the hash_key exists, determine whether it is marked as empty. If it is, return empty; otherwise, return the value.

[0035] The system searches the persistent in-memory hash table. If the hash key exists, it returns the value; otherwise, it returns null.

[0036] As a preferred embodiment, the data query method of the present invention further includes:

[0037] When accessing a hierarchical heterogeneous B+ tree is required:

[0038] In the root node, perform a binary search for the first value greater than the key value. If it exists, access the corresponding child node of that node; otherwise, access the pointer to the last child node of the root node.

[0039] The child node is used as the current node. A binary search is performed in the current node to find the first value greater than the key value. If it exists, the corresponding child node of the current node is accessed. Otherwise, the pointer of the last child node of the current node is accessed. The current operation is repeated until the leaf node is accessed.

[0040] When a leaf node pointer is obtained, if the pointer is in DRAM, the leaf node is read from DRAM; otherwise, the leaf node is read from persistent memory. A binary search is used to determine if the key-value pair exists. If it exists, the value is returned; otherwise, null is returned.

[0041] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0042] 1. This invention utilizes heterogeneous characteristics to construct a hierarchical heterogeneous storage hash table and a hierarchical heterogeneous storage B+ tree, reducing the write latency and number of writes in persistent memory, and accelerating queries by utilizing the read speed of DRAM and persistent memory.

[0043] 2. This invention uses DRAM to record modified content and writes persistent WAL logs sequentially in 256-byte units. Modifications to persistent memory are flushed in a background thread. By utilizing the data recorded in DRAM, the blocking caused by simultaneous reading and writing to persistent memory is greatly reduced. At the same time, by utilizing the properties of hierarchical hashing, the number of persistent memory flushes is reduced, thereby improving the lifespan of persistent memory. Attached Figure Description

[0044] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and form part of this application, do not constitute a limitation thereof. In the drawings:

[0045] Figure 1 This is a schematic diagram of graph topology and topology data hash representation in an embodiment of the present invention;

[0046] Figure 2 This is a schematic diagram of the startup of a hierarchical heterogeneous B+ tree according to an embodiment of the present invention;

[0047] Figure 3 This is a schematic diagram illustrating the writing of data to a hierarchical heterogeneous hash table according to an embodiment of the present invention;

[0048] Figure 4 This is a schematic diagram illustrating the writing of data to a hierarchical heterogeneous B+ tree according to an embodiment of the present invention;

[0049] Figure 5 This is a schematic diagram illustrating a hierarchical heterogeneous hash table query in an embodiment of the present invention.

[0050] Figure 6 This is a flowchart illustrating the hierarchical heterogeneous hash table query process according to an embodiment of the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0052] Example:

[0053] In existing graph storage and query technologies, lazy loading spreads data I / O time across each query, which, while beneficial for memory management and limiting memory usage, significantly reduces query speed. Full loading requires loading all data into memory during program initialization, severely slowing down startup and potentially leading to memory exhaustion due to limited memory space. Furthermore, numerous writes to persistent memory can create serious performance bottlenecks. Therefore, this embodiment proposes a heterogeneous hierarchical storage structure for graph data based on persistent memory. This structure, tailored to the characteristics of graph data, employs a hierarchical hash table to store topological data and equality query data structures from the index data. It also utilizes a hierarchical B+Tree storage structure to store range query data structures from the index data. Attribute data is still persisted to disk using an LSM-Tree, addressing the issues of query speed and limited memory space. By converting random writes to persistent memory into sequential writes, write latency is reduced. Finally, by constructing a hierarchical storage architecture, the number of writes to data structures in persistent memory is reduced, improving query speed.

[0054] Graph data queries mainly fall into three categories. The first is topology data queries, which require computation using graph topology information. This topology data is stored in a hash table, and the topology data for a node / edge is retrieved via its ID. The second is querying the attribute information of graph nodes. Since attribute information occupies a significant amount of storage space and is only accessed at the endpoint of the graph query, this embodiment uses an LSM-Tree to persist the graph attribute information to disk. The last category is index queries, which are divided into two types: equality queries, stored in a hash table, and range queries, stored in a B+ tree. Because index and topology queries are frequently used in graph queries, and the number of edges and nodes involved increases exponentially with the number of expand operations, the space occupied by hash tables and B+ trees is suitable for storage in persistent memory. Therefore, this embodiment primarily optimizes the storage of hash tables and B+ trees in persistent memory.

[0055] To address the high latency of random writes to persistent memory, this embodiment employs a hierarchical heterogeneous storage hash table to reduce random writes. This hash table uses a log to further reduce the latency of writing persistent content. Specifically, when performing add, delete, or modify operations on the persistent memory hash table, a Write-Ahead Log (WAL) is used. The operations are first recorded in the WAL log, and then the newly added data is written to the DRAM hash table (i.e., DRAM). In the HashMap, each WAL log entry is 256 bytes in size, containing the following fields: CRC16 checksum (2 bytes), operation type OPTYPE (1 byte), value byte length VALUE_LEN (1 byte), sequence number SERIAL_NUM (4 bytes), key content KEY_CONTENT (8 bytes), and value content VALUE_CONTENT (side length, maximum 240 bytes). Each log entry is appended to the end of the log. Since persistent memory (PMEM) guarantees atomic writes of 8 bytes, the CRC16 checksum + operation type + value byte length + sequence number (8 bytes in total) can be atomically written to persistent memory. Therefore, the CRC16 checksum can be used to determine whether the log was written successfully and completely. The 256-byte size of each log entry maximizes the sequential write performance of persistent memory. Simultaneously, a thread is started to periodically apply the logs, using transactions to apply the logs to the persistent memory hash table (i.e., PMEM HashMap). Furthermore, this hierarchical heterogeneous storage hash table utilizes the concept of hierarchical hashing (Pengfei Zuo et al.) to reduce the data that needs to be modified during hash table rehashing. Specifically, the hash table is divided into an upper layer and a lower layer, with the lower layer being half the size of the upper layer. Every two buckets in the upper layer correspond to one bucket in the lower layer, and each bucket contains eight key-value pairs. When a key is written to the hash table, the system first checks if the key exists in the current bucket. If it does not exist, the system continues to check the lower layer to see if the key exists. If it does not exist, the key is written directly. If both the upper and lower layer buckets are full, the lower layer is set as a temporary storage layer, and the upper layer is set as the lower layer. The rehashing operation then begins, with the new upper layer being twice the size of the original upper layer. The keys in the temporary storage layer are rehashed to the upper layer. Using this method, the number of key-value pairs changed by rehashing can be reduced by one-third of the entire hash table.

[0056] Because traditional B+ trees require extensive modifications to the upper-level structure when adding or deleting nodes, resulting in numerous write operations, this embodiment employs a hierarchical heterogeneous storage B+ tree to reduce write operations in persistent memory. Specifically, this embodiment uses DRAM and persistent memory for hierarchical storage. The leaf layer of the B+ tree, i.e., all leaf nodes, is stored in persistent memory (i.e., the B+ tree PMEM part), while the upper-level nodes are stored in DRAM (i.e., the B+ tree DRAM part). Each leaf node stores 16 256-byte key-value pairs, meaning each leaf node is 4KB in size. When there are write, modify, or delete operations, the modified leaf node is first written to DRAM, and its parent node's pointer is set to DRAM. Then, the operation is recorded in the WAL log in persistent memory. The WAL log format is the same as the hierarchical heterogeneous storage hash table described above. A thread is started to periodically apply the log, applying the log-recorded operations to the leaf nodes stored in persistent memory through transactions. When a log is committed to persistent memory, the parent node's pointer in DRAM is set to the updated node in persistent memory.

[0057] The process of data storage and retrieval in this embodiment based on the proposed hierarchical heterogeneous storage structure is as follows:

[0058] The data storage (data writing) process is as follows:

[0059] When the machine boots up (i.e., when the graph database process starts), the hierarchical heterogeneous B+ tree first needs to read the leaf nodes in persistent memory and then construct the upper-level nodes of the B+ tree in DRAM. The lower-level pointers point to the leaf nodes in persistent memory, such as... Figure 2 As shown;

[0060] When importing a graph topology and its node attributes, two hierarchical heterogeneous hash tables need to be constructed first. These tables store the mappings between node IDs and their source and destination edges, and between edge IDs and their source and destination nodes, respectively. Figure 1 As shown.

[0061] For node attributes, indexes need to be created according to requirements. Based on the attribute type, they are divided into equality indexes and range query indexes. Hierarchical heterogeneous hash tables and hierarchical heterogeneous B+ trees can be created to store equality indexes and range query indexes respectively.

[0062] Specifically, such as Figure 3 As shown, when writing data to a hierarchical heterogeneous hash table:

[0063] Write key-value pairs to the DRAM hash table for temporary storage;

[0064] At the same time, the key-value pairs are written to the WAL log stored in persistent memory;

[0065] A background thread periodically parses the WAL log and records the changes in a persistent in-memory hash table.

[0066] Release the space temporarily used by the DRAM hash table to store key-value pairs.

[0067] Specifically, such as Figure 4 As shown, when writing data to a hierarchical heterogeneous B+ tree:

[0068] If there is enough space in the leaf nodes and no node splitting or merging is triggered, the corresponding leaf node written in the persistent memory is copied into the DRAM, the value is written to the leaf node, and the upper-level pointer is set to point to the leaf node; if there is not enough space in the leaf nodes, two new leaf nodes are generated in the DRAM, and the upper-level pointer is set to point to these two new leaf nodes.

[0069] Write the key-value pairs to the WAL log stored in persistent memory;

[0070] The background thread periodically parses the WAL log and updates the changes to the leaf nodes in persistent memory;

[0071] Release the space of the leaf nodes temporarily stored in the B+ tree in DRAM.

[0072] Data query process:

[0073] When performing an expand operation, to obtain the set of destination nodes for all outgoing or incoming edges from a set of nodes, a hierarchical heterogeneous hash table needs to be accessed, specifically as follows: Figure 5-6 :

[0074] The hash_key is calculated using the hash function hash(x);

[0075] Search the DRAM hash table. If the hash_key does not exist, continue to the next step. If the hash_key exists, determine whether it is marked as empty. If it is, return empty; otherwise, return the value.

[0076] The system searches the persistent in-memory hash table. If the hash key exists, it returns the value; otherwise, it returns null.

[0077] When it is necessary to look up a node by attribute, the hierarchical heterogeneous B+ tree of the attribute index needs to be accessed:

[0078] In the root node, perform a binary search to find the first value greater than the key. If it exists, access the corresponding child node of that node; otherwise, access the pointer to the last child node of the root node.

[0079] The child node (i.e. the child node determined in the previous step) is used as the current node. A binary search is performed in the current node to find the first value greater than the key value. If it exists, the corresponding child node of the current node is accessed. Otherwise, the pointer of the last child node of the current node is accessed. The current operation is repeated until the leaf node is accessed.

[0080] When a leaf node pointer is obtained, if the pointer is in DRAM, the leaf node is read from DRAM; otherwise, the leaf node is read from persistent memory. A binary search is used to determine if the key value exists. If it exists, the value is returned; otherwise, null is returned.

[0081] The following specific examples illustrate the data storage process and querying of graph databases:

[0082]

[0083]

[0084]

[0085] Specifically, the stored procedure (write procedure) includes:

[0086] When the machine boots up (i.e. when the graph database process starts), the hierarchical heterogeneous B+ tree first needs to read the leaf nodes in persistent memory and build the upper-level nodes.

[0087] When importing a graph topology and its node attributes, two hierarchical heterogeneous hash tables need to be constructed first. These tables distribute the mapping between node IDs and their source and destination edges, as well as the mapping between edge IDs and their source and destination nodes. For node attributes, indexes need to be created according to requirements. Based on attribute type, these can be categorized as equality indexes and range query indexes. Hierarchical heterogeneous hash tables and hierarchical heterogeneous B+ trees can be built to meet these requirements. For example, let's assume a range query index is created for the `age` attribute.

[0088] For the two nodes, key-value pairs {78562323330406344,{"out_edge_id":1}} and {43455993040678557,{"in_edge_id":1}} need to be added to the topology data hash table, and key-value pairs {19,78562323330406344} and {25,43455993040678557} need to be added to the age attribute index data.

[0089] When writing data to the topology data hash table:

[0090] (1) Write key-value pairs {78562323330406344, {"out_edge_id":1}} to the DRAM hash table for temporary storage;

[0091] (2) Write operation to WAL log, set opcode OPTYE to 1 (write), KEY to 8562323330406344, VALUE to the serialized string {"out_edge_id":1}, and get a log with a maximum length of 256 bytes, and write it to the WAL log stored in persistent memory (PMEM);

[0092] (3) The background thread will periodically parse the WAL log and update the changes to the hash table in persistent memory;

[0093] (4) Release the space of the DRAM hash table temporary key pair {78562323330406344,{"out_edge_id":1}}.

[0094] When writing {19,78562323330406344} into the B+ tree indexed by the age attribute:

[0095] (1) If there is enough space in the leaf node and no node splitting or merging is triggered, the corresponding leaf node written in the persistent memory is copied into the DRAM, the value is written to the leaf node, and the upper-level pointer is pointed to the leaf node.

[0096] If there is insufficient space for leaf nodes, two new leaf nodes are generated in DRAM, and the upper-level pointer is set to point to these two leaf nodes.

[0097] (2) Write operation to WAL log, set opcode OPTYE to 1 (write), KEY to 19, VALUE to 78562323330406344 serialized string to get log with a maximum length of 256 bytes, and write to WAL log stored in persistent memory.

[0098] (3) The background thread will periodically parse the WAL log, change the changes to the leaf nodes of the B+ tree in persistent memory, and change the DRAM upper-level pointer to point to the leaf node in persistent memory.

[0099] (4) Release the space of the leaf nodes temporarily stored in the B+ tree in DRAM.

[0100] When writing {25,43455993040678557} into the B+ tree indexed by the age attribute:

[0101] (1) If there is enough space in the leaf node and no node splitting or merging is triggered, the corresponding leaf node written in the persistent memory is copied into the DRAM, the value is written to the leaf node, and the upper-level pointer is pointed to the leaf node.

[0102] If there is insufficient space for leaf nodes, two new leaf nodes are generated in DRAM, and the upper-level pointer is set to point to these two leaf nodes.

[0103] (2) Write operation to WAL log, set opcode OPTYE to 1 (write), KEY to 25, VALUE to 43455993040678557 serialized string to get log with a maximum length of 256 bytes, and write it to WAL log stored in persistent memory.

[0104] (3) The background thread will periodically parse the WAL log, change the changes to the leaf nodes of the B+ tree in persistent memory, and change the DRAM upper-level pointer to point to the leaf node in persistent memory.

[0105] (4) Release the space of the two leaf nodes temporarily stored in the DRAM+ tree.

[0106] The query process specifically includes:

[0107] When performing an expand operation, which involves obtaining the set of destination points for all outgoing or incoming edges from a set of nodes, it is necessary to access the topology data hash table.

[0108] When node 78562323330406344 searches for its outgoing edge 1, and outgoing edge 1 searches for its destination node 43455993040678557, the topological hash table is accessed:

[0109] (1) Calculate its hash_key using the hash function hash(x).

[0110] (2) Search in the DRAM hash table. If the hash_key does not exist, continue to the next step. If the hash_key exists, determine whether it is marked as empty. If it is, return empty; otherwise, return the value.

[0111] (3) Search the persistent memory hash table. If the hash_key exists, return the value; otherwise, return null.

[0112] When you need to find a node by an attribute, such as the node where age = 19, you need to access the B+ tree index of the age attribute:

[0113] (1) In the root node, perform a binary search for the first value greater than 19. If it exists, access the child node corresponding to that node; otherwise, access the pointer of the last child node of the root node.

[0114] (2) The child node is used as the current node. A binary search is performed in the current node to find the first value greater than 19. If it exists, the corresponding child node of the current node is accessed. Otherwise, the pointer of the last child node of the current node is accessed. The current operation is repeated until the leaf node is accessed.

[0115] (3) When a leaf node pointer is obtained, if the pointer is in DRAM, the leaf node is read from DRAM; otherwise, the leaf node is read from persistent memory. A binary search is used to determine if key=19 exists. If it exists, the value is returned, i.e., node id78562323330406344; otherwise, null is returned.

[0116] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0117] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0118] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0119] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0120] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A heterogeneous hierarchical storage structure for graph data based on persistent memory, characterized in that, The graph data heterogeneous hierarchical storage structure includes a hierarchical heterogeneous hash table and a hierarchical heterogeneous B+ tree; Among them, the topological data and index data of the graph data are stored using hierarchical heterogeneous hash tables. The range query data structure in the index data of the graph data adopts a hierarchical heterogeneous B+ tree storage; the hierarchical heterogeneous hash table is divided into an upper layer and a lower layer, the size of the lower layer is half the size of the upper layer, that is, every two buckets in the upper layer correspond to one bucket in the lower layer; The hierarchical heterogeneous hash table uses WAL logs for add, delete, and modify operations. Each bucket contains 8 key-value pairs. When a key is written to the hash table, the system first checks if the key exists in the current bucket. If it does not exist, it continues to the next lower level to check if the key exists. If it does not exist, the key is written directly. If both the upper and lower buckets are full, the lower level is set as a temporary storage level, the upper level is set as the lower level, and the rehashing operation begins. The new upper level is allocated twice the size of the original upper level, and the keys in the temporary storage level are rehashed to the upper level. The hierarchical heterogeneous B+ tree uses DRAM and persistent memory for hierarchical storage. The leaf layer of the B+ tree, i.e., all leaf nodes, is stored in persistent memory, while the upper-level nodes are stored in DRAM. When performing add, delete, or modify operations on the hierarchical heterogeneous B+ tree, the modified leaf node is first written to DRAM, and its parent node pointer is pointed to DRAM. Then, the add, delete, or modify operations are recorded in the WAL log in persistent memory. A thread that applies the log periodically is started, and the operations recorded in the log are applied to the leaf nodes stored in persistent memory through transactions. When a log is committed to persistent memory, the parent node pointer in DRAM is pointed to the updated node in persistent memory.

2. The heterogeneous hierarchical storage structure for graph data based on persistent memory according to claim 1, characterized in that, Each leaf node stores 16 key-value pairs of 256 bytes each, meaning each leaf node is 4KB in size. The hierarchical heterogeneous B+ tree uses WAL logs for add, delete, and modify operations.

3. A heterogeneous hierarchical storage structure for graph data based on persistent memory according to any one of claims 1-2, characterized in that, The WAL log is 256 bytes in size, and its fields include: a 2-byte CRC16 checksum, a 1-byte operation type OPTYPE, a 1-byte value length VALUE_LEN, a 4-byte sequence number SERIAL_NUM, an 8-byte key content KEY_CONTENT, and a maximum of 240 bytes of value content VALUE_CONTENT; each log entry is appended to the end of the log.

4. A data storage method based on a heterogeneous hierarchical storage structure for graph data with persistent memory, as described in any one of claims 1-3, characterized in that, The data storage method includes: When the graph database process starts, the hierarchical heterogeneous B+ tree first reads the leaf nodes in persistent memory and constructs the upper-level nodes of the B+ tree in DRAM, while the lower-level pointers point to the leaf nodes in persistent memory. When a graph topology and its node attributes are imported, two hierarchical heterogeneous hash tables are first constructed to store the mapping between node IDs and their source and destination edges, and the mapping between edge IDs and their source and destination nodes, respectively. Based on the node's attribute type, either create a hierarchical heterogeneous hash table to store the equal-value index or create a hierarchical heterogeneous B+ tree to store the range query index.

5. The data storage method according to claim 4, characterized in that, When writing data to a hierarchical heterogeneous hash table: Write key-value pairs to the DRAM hash table for temporary storage; At the same time, the key-value pairs are written to the WAL log stored in persistent memory; A background thread periodically parses the WAL log and records the changes in a persistent in-memory hash table. Release the space temporarily used by the DRAM hash table for key-value pairs; When writing data to a hierarchical heterogeneous B+ tree: If there is enough space in the leaf node and no node splitting or merging is triggered, the corresponding leaf node written in the persistent memory is copied into the DRAM, the value is written to the leaf node, and the upper-level pointer is pointed to the leaf node. If there is insufficient space for the leaf nodes, two new leaf nodes are generated in DRAM, and the upper-level pointer is set to point to these two new leaf nodes. Write the key-value pairs to the WAL log stored in persistent memory; A background thread periodically parses the WAL log and moves the changes to the leaf nodes in persistent memory. Release the space of the leaf nodes temporarily stored in the B+ tree in DRAM.

6. A data query method based on a heterogeneous hierarchical storage structure for graph data with persistent memory, as described in any one of claims 1-3, characterized in that, The data query method includes: When you need to access a hierarchical heterogeneous hash table: Its hash_key is calculated using a hash function; Search the DRAM hash table. If the hash_key does not exist, continue to the next step. If the hash_key exists, determine whether it is marked as empty. If it is, return empty; otherwise, return the value. The system searches the persistent in-memory hash table. If the hash key exists, it returns the value; otherwise, it returns null.

7. The data query method according to claim 6, characterized in that, The data query method also includes: When accessing a hierarchical heterogeneous B+ tree is required: In the root node, perform a binary search for the first value greater than the key value. If it exists, access the corresponding child node of that node; otherwise, access the pointer to the last child node of the root node. The child node is used as the current node. A binary search is performed in the current node to find the first value greater than the key value. If it exists, the corresponding child node of the current node is accessed. Otherwise, the pointer of the last child node of the current node is accessed. The current operation is repeated until the leaf node is accessed. When a leaf node pointer is obtained, if the pointer is in DRAM, the leaf node is read from DRAM; otherwise, the leaf node is read from persistent memory. A binary search is used to determine if the key-value pair exists. If it exists, the value is returned; otherwise, null is returned.

Images