A storage method, reading method and system of graph data

Abstract

The application relates to the technical field of data storage, and provides a graph data storage method, a reading method and a system, which comprises the following steps: a primary key is established for each vertex and edge of graph data respectively, the primary key can be sorted, and the primary keys of each vertex and the adjacent edges of the vertex are continuously sorted; the data values of the vertices and edges of the graph data are stored in a persistent memory to obtain a storage address, each storage address corresponds to a primary key; a specific data structure is used to form an index to record the relationship between the primary keys and the corresponding storage addresses, the specific data structure is a data structure supporting range query, the data structure is sorted according to the primary keys, and the specific data structure is stored in a dynamic random access memory. The graph topology access performance and the attribute graph query performance are improved. The system not only supports an analytical load on an attribute graph, but also can support a transaction load, and has good universality and expansibility.

Description

Technical Field

[0001] This invention relates to the field of data storage technology, and specifically to methods and systems for storing and retrieving graph data. Background Technology

[0002] With the popularization of digitalization, graph data is widely used in computer science. Graph data is a data structure that uses nodes and edges to represent entities and relationships, and it can effectively simulate complex networks and systems.

[0003] Applications of graph data include:

[0004] Social networks: Graph data can be used to represent user relationships, interests, and dynamic information, thereby providing more personalized services.

[0005] Knowledge graphs can use graph data to represent knowledge entities and relationships in various fields, thereby supporting intelligent question answering, semantic search, natural language understanding, and more.

[0006] E-commerce: Graph data can be used to represent the similarity between products, user purchasing behavior, review information, etc., thereby providing more accurate recommendations and marketing.

[0007] Reading and storing graph data is a very important application scenario in digital life and production, and the efficiency of reading and storing graph data is crucial.

[0008] A common approach to storing attribute graphs is to use a key-value store based on an LSM-Tree. In these systems, each vertex and each edge is typically represented as a unique key-value pair. While LSM-Tree structures are write-friendly, they suffer from read amplification, especially during graph topology traversal. This is because the basic operation of graph traversal involves scanning the neighbors from a given vertex, which corresponds to a range query in an LSM-Tree, resulting in low efficiency.

[0009] Another common approach is to use an existing relational database to store the attribute graph. However, relational databases are not friendly to topology queries and require expensive table join operations.

[0010] None of the methods mentioned above can reflect the topological structure of a graph. Finding a vertex or edge usually involves a large number of graph traversal operations, which consumes a lot of computing resources and network bandwidth, leading to performance degradation. This problem is particularly serious when the graph data is too large. Summary of the Invention

[0011] To address the above problems, this invention provides a method for storing graph data.

[0012] The method includes:

[0013] A primary key is created for each vertex and edge of the graph data. The primary keys are sortable, and the primary keys of each vertex and its adjacent edges are sorted consecutively. The data values ​​of the vertices and edges of the graph data are stored in persistent memory to obtain a storage address, and each storage address corresponds to a primary key. An index is formed using a specific data structure to record the relationship between the primary key and the corresponding storage address. The specific data structure is a data structure that supports range queries and is sorted by the primary key. The specific data structure is stored in dynamic random access memory.

[0014] Furthermore, storing the vertex and edge data values ​​of the graph data in persistent memory includes: allocating a contiguous storage space in persistent memory, and storing the data values ​​of each vertex and its adjacent edges of the graph data in the contiguous storage space.

[0015] Preferably, storing the vertex and edge data values ​​of the graph data in persistent memory includes: storing the vertex and edge data values ​​of the graph data in persistent memory in pages.

[0016] Preferably, for attribute graph data, data values ​​with the same label are stored on the same page.

[0017] Preferably, the primary key is represented by a numerical value, and the primary key is sorted using a numerical sorting rule.

[0018] Preferably, the specific data structure is a B+ tree, and the leaf nodes of the B+ tree record the storage address.

[0019] The present invention also discloses a method for reading graph data stored by the above-described storage method, comprising: calculating a key value of a vertex or edge of a graph data; querying a storage address in persistent memory based on the key value in a dynamic random access memory, wherein the storage address stores the data value of a vertex or edge of a graph data; and reading the data value from the storage address in persistent memory.

[0020] To address the above problems, the present invention also provides a graph data storage system.

[0021] The system includes:

[0022] A primary key generation module is used to create a primary key for each vertex and edge of the graph data. The primary keys are sortable, and the primary keys of each vertex and its adjacent edges are sorted consecutively. A persistent memory storage module is used to store the data values ​​of the vertices and edges of the graph data in persistent memory, obtaining a storage address, and each storage address corresponds to a primary key. A matching module is used to use a specific data structure to form an index to record the relationship between the primary key and the corresponding storage address. The specific data structure is a data structure that supports range queries and is sorted by the primary key. A dynamic storage module is used to store the specific data structure in dynamic random access memory.

[0023] The system and method in this invention correspond to each other, and the specific preferred solutions of the method are also applicable to the system.

[0024] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0025] By storing the graph's topology and attributes in dynamic random access memory and persistent memory respectively, the advantages of these two physical storage media are fully utilized. By combining the topology and index into one, the operation of retrieving neighbors in the graph is transformed into a range scan of the index, improving the performance of graph topology access without increasing additional storage overhead or operational complexity. Furthermore, strategies such as organizing attribute data according to labels improve the performance of attribute graph queries. This system supports not only analytical workloads on attribute graphs but also transactional workloads, exhibiting good versatility and scalability. Attached Figure Description

[0026] Figure 1 A flowchart illustrating a method for storing graph data according to an embodiment of the present invention;

[0027] Figure 2 A block diagram of a graph data storage system provided in an embodiment of the present invention;

[0028] Figure 3 This is a logical diagram illustrating how to query a storage address from a B+ tree, as provided in an embodiment of the present invention.

[0029] Figure 4 A graph showing the correspondence between points, edges, and primary keys of an attribute graph provided in an embodiment of the present invention;

[0030] Figure 5 This is a structural diagram of paginated data storage provided in an embodiment of the present invention;

[0031] Figure 6 This is a diagram of the dynamic random access memory and persistent memory data structure provided in an embodiment of the present invention. Detailed Implementation

[0032] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. Before describing the technical solutions of each embodiment of the present invention in detail, the terms and terms involved will be explained. In this specification, components with the same name or the same reference numerals represent similar or the same structures and are limited to illustrative purposes.

[0033] As is well known, Dynamic Random Access Memory (DRAM) offers slightly faster read / write speeds compared to persistent memory (PM), but is more expensive and has lower capacity. Furthermore, data in DRAM may be lost, while data in PM is less prone to loss. This invention constructs a hybrid architecture that leverages the characteristics of both PM and DRAM. Specifically, it designs an index-based topology strategy and a label-based page partitioning strategy to address the topology and attribute access characteristics of graphs, achieving efficient support for both transactional and analytical workloads in graphs.

[0034] See Figure 1 The method for storing graph data provided by the present invention includes:

[0035] Step 102: Create a primary key for each vertex and edge of the graph data. The primary key is sortable, and the primary keys of each vertex and its adjacent edges are sorted consecutively.

[0036] The primary key can be any data type that supports sorting, such as numeric or string types. Sorting of numeric or string types is common knowledge in this field and will not be described in detail here. The primary keys of each vertex and its adjacent edges are consecutively sorted; that is, according to the sorting rules, there will be no other vertex or edge primary keys between the primary keys of each vertex and its adjacent edges. This way, when searching for a vertex or edge, only a certain range needs to be searched, which can improve search efficiency.

[0037] Step 104: Store the vertex and edge data values ​​of the graph data in persistent memory to obtain a storage address, and each storage address corresponds to a primary key;

[0038] Each vertex and edge has a unique primary key, and each vertex and edge also has a data value. For simple graphs, this data value can be just the primary key itself; for attribute graphs, the data value can be both the primary key and the attribute value. This data value is stored in persistent memory, recording its storage address, thus creating a one-to-one correspondence between the primary key of each vertex and edge and the storage address of its data value.

[0039] Step 106: Use a specific data structure to form an index to record the relationship between the primary key and the corresponding storage address. The specific data structure is a data structure that supports range queries and is sorted by the primary key.

[0040] Step 108: Store the specific data structure in a dynamic random access memory.

[0041] This data structure reflects the correspondence between the primary key of each vertex and edge and the storage address of its data value. Because this data structure supports range queries and is sortable, it can quickly find the corresponding storage address based on the primary key. Since dynamic random access memory (DRAM) has high read efficiency, finding the storage address based on the primary key in DRAM is also very fast.

[0042] The above technical solution employs a layered architecture, storing graph data in persistent memory and indexes in dynamic random access memory (DRAM) to fully utilize the physical characteristics of both. Furthermore, through careful index design, the primary index and graph topology are integrated: the index not only maintains the mapping between primary keys and records for fast lookup of vertex or edge data values, but also represents the graph topology. Therefore, graph topology operations can be implemented using only the index.

[0043] The data structure mentioned in this invention is a sortable data structure that supports range queries, such as an array or a segment tree. A B+ tree is a preferred data structure because it supports range queries and offers good performance for single-point reads, deletions, and updates. The leaf nodes of a B+ tree record the storage addresses of the vertex or edge data values ​​of the graph data.

[0044] When storing graph data in persistent memory, a contiguous block of storage can be allocated to store the data values ​​of each vertex and its adjacent edges. This improves retrieval efficiency when searching for the data values ​​of vertices and their adjacent edges.

[0045] The methods described above can effectively support transactional workloads for graph data. However, in everyday applications, the demand for accessing graph data includes not only transactional workloads but also analytical workloads. Transactional workloads refer to operations such as adding, deleting, modifying, and querying graph data, typically involving a small number of vertices and edges. For example, in social networks, users can create, update, or delete their personal information, friend relationships, and updates, all of which fall under transactional workloads. Analytical workloads, on the other hand, refer to complex analyses and mining of graph data, usually involving a large number or even the entire set of vertices and edges. For example, in social networks, users can make recommendations and categories based on their interests, location, age, and other criteria.

[0046] To support analytical workloads with graph data, this invention further proposes using persistent memory in an interleaved mode. In interleaved mode, persistent memory storage space is organized in fixed-size blocks, such as 4KB. Each fixed-size contiguous storage space is called a page. The system organizes data according to labels. For attribute graphs, only data with the same label is written within the same page. For simple graphs, all vertices and edges can be treated as having the same label. Each record represents a vertex or edge, and the format of each record is fixed. The design of this format only needs to ensure that each record can be correctly segmented and that the various fields of the data value can be accurately located within each record. For example, each record can consist of three parts: a primary key, an offset table, and attributes. The offset table is used to track the offset of each variable-length field in the data value.

[0047] By storing data in pages using a specific format, based on vertices or labels, it becomes easy to find a specific field within a data value by label or vertex when retrieving data. This improves the performance of attribute graph queries and effectively supports transactional workloads in attribute graphs. For example, because the data is organized by labels, when a query only needs data for a particular type of label, only the corresponding data pages need to be read sequentially. When a query only needs a specific field of an attribute (such as a person's age), the offset table allows direct location of the corresponding field, reducing read amplification.

[0048] For data stored according to the graph data storage method proposed in this invention, this invention also provides a reading method, namely, when reading, first calculate the primary key of a vertex or edge according to the primary key generation rule, then query the storage address of the corresponding vertex or edge data value in persistent memory according to the primary key in the dynamic random access memory, and finally read the data value from persistent memory according to the storage address.

[0049] The present invention also provides a graph data storage system, such as Figure 2 As shown, it includes:

[0050] The primary key generation module is used to create a primary key for each vertex and edge of the graph data. The primary keys are sortable, and the primary keys of each vertex and its adjacent edges are sorted consecutively.

[0051] A persistent memory storage module is used to store the vertex and edge data values ​​of the graph data in persistent memory, obtaining a storage address, and each storage address corresponds to a primary key;

[0052] The matching module is used to record the relationship between the primary key and the corresponding storage address using a specific data structure, wherein the specific data structure is a sortable data structure that supports range queries;

[0053] A dynamic storage module is used to store the specific data structure in a dynamic random access memory.

[0054] The present invention will now be explained with reference to a specific embodiment.

[0055] When storing graph data in persistent memory, the data is organized according to labels, meaning that only data with the same label is written to the same page, and arrays are used to track these pages.

[0056] Figure 5 The diagram illustrates the structure of a page: each page begins with a one-byte lock to ensure the safety of concurrent access; next is a tail pointer, which records the starting position for the next write; finally, the actual data to be stored. Each record represents a vertex or edge and consists of three parts: a primary key, an offset table, and attributes. The offset table tracks the offset of each variable-length field, with the length of each field recorded in one byte. If a field's length is less than 255 bytes, its data is stored directly after the offset table. For particularly large fields, such as photos, they are stored in a different location. The offset table allows direct access to specific fields when reading attribute graph properties, thus supporting analytical workloads for attribute graphs.

[0057] For an attribute graph, each vertex has a unique identifier, represented by a VID. In this example, each VID is no more than 4 bits. The number of labels in each attribute graph is usually limited, and each label is also assigned a unique LID. Since a label only requires a few bytes of memory to store its name and LID, the overhead is minimal.

[0058] Figure 4 This demonstrates a property graph and its corresponding primary key design for vertices and edges. Labels are represented by 4 bits, and a maximum of 16 labels can be represented. Figure 4 There are four types of tags: Person, Post, Know, and hasCreator, with their assigned LIDs represented in binary as (0001, 0010, 0011, and 0100). The primary key is represented by 16 bits. Specifically, for a vertex, the highest 4 bits represent its LID, the next 4 bits represent the vertex's VID, and the lowest 8 bits are set to 0 to align with the edge's primary key. For an edge, the highest 8 bits represent the source vertex's LID and VID, and the next 8 bits represent the edge's LID and the destination vertex's VID.

[0059] In this example graph, the primary key is an unsigned 16-bit integer. In practical applications, the number of bits in the key can be set according to the size of the graph. For example, when the key length is 64 bits, it can encode a number containing 2... 32 Vertex and 2 64 A diagram with a single edge.

[0060] The primary key is designed using a combination of LID and vertex / edge ID. This design not only enables efficient topology-based access but also implicitly indicates the labels of vertices and edges.

[0061] Subsequently, each vertex or edge of the attribute graph is inserted into a B+ tree with its storage address and primary key in persistent memory. This B+ tree is sorted by primary key, and the storage addresses are all stored in the leaf nodes. This design ensures that the primary key of a vertex and its adjacent edges are stored contiguously in the index, allowing graph traversal by scanning the index and facilitating label-based access patterns.

[0062] For example, to retrieve the people known by a vertex labeled "Person" with a VID of 1, a range query is performed, with the interval [0001000100110000, 0001000101000000]. Since Person's LID is 0001, its vertex and base VIDs are 1, and the LID of "KNOW" (knowledge relationship) is 0011, the starting interval is 0001 0001 0011 0000. Only after all edges labeled "KNOW" for that vertex have been read can the next type of edge be retrieved. Because the LID of "KNOW" is 0011, the next LID is 0011 plus 1, which is 0100. This determines the termination key of the range query: 0001 0001 01000000.

[0063] Finally, the B+ tree is stored in dynamic random access memory. Figure 6 It demonstrates the data storage structures in dynamic random access memory and persistent memory.

[0064] When reading the attribute graph data, the primary key of a point or edge is first obtained by the pattern of primary key generation. Then, according to the structural characteristics of the B+ tree, the storage address corresponding to the data value of the point or edge is found, and the data is read from the persistent memory based on this storage address. Figure 3 The diagram illustrates the logic of searching for the corresponding storage relationship in a B+ tree based on the primary key.

[0065] The above uses attribute graphs as an example to illustrate the implementation of this invention. For simple graphs, the logic is similar. If an 8-bit unsigned number is used to represent the primary key, and each VID does not exceed 4 bits, the primary key generation rule for a simple graph can be that the primary key of each vertex is formed by concatenating the high 4 bits of VID and the low 4 bits of 0. The primary key of an edge is composed of the VIDs of the two vertices connected by the edge, and each vertex's VID occupies 4 bits.

[0066] Furthermore, in this embodiment, the data values ​​of the attribute graph are stored in pages in persistent memory, and arrays are used to track these pages. The data structure of each page is fixed, and the primary keys of all points and edges are also stored in these pages. When data in the dynamic random access memory is lost, all pages can be found directly from these arrays. The storage addresses of all primary keys and their corresponding data values ​​can be retrieved from each page, and the B+ tree can be reconstructed based on this. This B+ tree is then stored in the dynamic random access memory.

[0067] This invention fully utilizes the advantages of both physical storage media—dynamic random access memory and persistent memory—by storing the graph's topology and attributes separately in dynamic random access memory and persistent memory. By combining the topology and index into one, the operation of retrieving neighbors in the graph is transformed into a range scan of the index, improving the performance of graph topology access without increasing additional storage overhead or operational complexity. Furthermore, strategies such as organizing attribute data according to labels improve the performance of attribute graph queries. This system supports not only analytical workloads on attribute graphs but also transactional workloads, exhibiting good versatility and scalability.

[0068] The above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for storing graph data, characterized in that, include: For each vertex and edge of the graph data, a primary key is created. The primary keys are sortable, and the primary keys of each vertex and its adjacent edges are sorted consecutively. The vertex and edge data values ​​of the graph data are stored in persistent memory to obtain a storage address, and each storage address corresponds to a primary key; wherein, storing the vertex and edge data values ​​of the graph data in persistent memory includes: allocating a contiguous storage space in persistent memory, and storing the data values ​​of each vertex of the graph data and its adjacent edges in the contiguous storage space; An index is formed using a specific data structure to record the relationship between the primary key and its corresponding storage address. The specific data structure is a data structure that supports range queries and is sorted by the primary key. The specific data structure is a B+ tree, and the leaf nodes of the B+ tree record the primary key and its corresponding storage address. The specific data structure is stored in dynamic random access memory; Storing the vertex and edge data values ​​of the graph data in persistent memory includes: The vertex and edge data values ​​of the graph data are stored in persistent memory in pages; For attribute graph data, store data values ​​with the same label on the same page; The primary key is represented by a numerical value, and the primary key is sorted using numerical sorting rules; For graph data of simple graphs, each vertex is assigned a unique numerical ID, and each vertex generates a unique primary key based on its own numerical ID. Each edge generates a unique primary key based on the numerical IDs of the two vertices that make up the edge. For attribute graph data, a unique numerical ID is generated for each label and each vertex. Each vertex generates a unique primary key based on the numerical ID of its label and its own numerical ID. Each edge generates a unique primary key based on the numerical ID of the label of the edge's source vertex, the numerical ID of the edge's source vertex, the numerical ID of the edge's label, and the numerical ID of the edge's destination vertex.

2. A method for reading graph data stored by the storage method of claim 1, characterized in that, include: Calculate the key value of a vertex or edge in a graph dataset; The key value is used to query the persistent memory for a storage address, where the storage address stores the data value of a vertex or edge of a graph; and Read the data value from the storage address in persistent memory.

3. A graph data storage system for implementing the method as described in claim 1, characterized in that, include: The primary key generation module is used to create a primary key for each vertex and edge of the graph data. The primary keys are sortable, and the primary keys of each vertex and its adjacent edges are sorted consecutively. A persistent memory storage module is used to store the vertex and edge data values ​​of the graph data in persistent memory, obtaining a storage address, and each storage address corresponds to a primary key; The matching module is used to form an index using a specific data structure to record the relationship between the primary key and the corresponding storage address. The specific data structure is a data structure that supports range queries and is sorted by the primary key. A dynamic storage module is used to store the specific data structure in a dynamic random access memory.

Images