A graph data storage method and apparatus

Abstract

The application relates to the technical field of databases, and provides a graph data storage method and device.The method comprises the following steps: storing data records into a partition data table; storing attribute index records into an attribute index table, so as to find corresponding data records through attributes; wherein the attribute index records comprise attribute values and ROWIDs of corresponding data records; storing graph topology data into an adjacency table, so as to find corresponding data records through edges or vertices; wherein the graph topology data comprises edge IDs, vertex IDs, topological structures between edges and vertices, and ROWIDs of corresponding data records.The application separately stores various types of data in graph data, stores each vertex and edge and corresponding attributes, relations and labels into a partition data table, and separately stores graph topology data through an adjacency table, so that the storage scale of the adjacency table is not excessively large, and the query performance of the graph topology data is prevented from being reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of database, in particular to a graph data storage method and device. BACKGROUND

[0002] A graph is a data structure composed of a number of points and edges connecting the points. Graphs are increasingly used in the fields of computer and artificial intelligence due to their intuitive representation of relationships between things and their easily expandable structure.

[0003] There are many methods for storing this data structure of graph, which can be generally divided into two categories: native graph storage and non-native graph storage. Native graph storage can well solve the correlation analysis between a large number of entities by maintaining the original adjacency relationship internally, and has good performance in processing the relationship between entities. Non-native graph storage generally uses a KV key-value pair-based data structure to represent the adjacency relationship between vertices, and a KV key-value pair can only represent one layer of adjacency relationship. When performing multi-layer adjacency query, the KV structure needs to be frequently accessed, and in some graph storage architectures where computing and storage are separated, multiple network communications are required, resulting in low query performance and failing to reflect the advantages of graph storage in relationship mining.

[0004] Common native graph storage methods mainly include adjacency matrix and adjacency list. The adjacency matrix uses a one-dimensional array to store all vertices in the graph, and a two-dimensional array to store the adjacency relationship (edge) between vertices in the graph. Even if there is no edge between two points, a storage location in the two-dimensional array is also occupied, and the space complexity is O(V2). The adjacency list also uses a one-dimensional array to store all vertices in the graph, but when storing edges, a single linked list is established for each vertex to record all edges of the vertex, and the space complexity is O(V+E). Comparing the two storage methods, the adjacency matrix has the advantage of being more suitable for the storage of dense graphs, and it is easy to determine whether there is an edge between any two vertices in the graph, and it is also easy to insert and delete edges. However, when storing sparse graphs, there is a lot of space waste, and in general, multiple edges cannot be stored, and it takes a lot of effort to scan the entire row or column to determine how many associated edges a vertex has. The storage efficiency of the adjacency list is higher than that of the adjacency matrix, multiple edges can be stored, and querying all adjacent edges of a vertex only requires querying the single linked list of the vertex. However, to determine whether there is an edge between two vertices, the entire single linked list needs to be traversed, and the query efficiency is lower than that of the adjacency matrix.

[0005] The above native graph storage method mainly considers the topological structure of the graph, and in actual graph applications, the vertices and edges in the graph need to be grouped according to categories, and a simple adjacency matrix or adjacency list cannot be classified and queried according to categories. Meanwhile, the vertices and edges usually have their own attribute information, and if the attribute data and the topological data are stored together, the storage size of the topological data of the graph will be expanded, thereby causing the query performance of most adjacency relationships to be sharply reduced.

[0006] The non-native graph storage based on the KV structure has low performance in multi-layer adjacency query, and the traditional native graph storage structures such as the adjacency matrix and the adjacency list cannot store and query different categories of vertices and edges, and there is no good solution for the storage of attribute graphs, and they cannot meet the requirements of modern large graph storage.

[0007] Therefore, it is urgent to overcome the defects of the prior art in the technical field. SUMMARY

[0008] The technical problem to be solved by the present application is that in the prior art, attribute data and topological data are stored together, which expands the storage size of the topological data of the graph, thereby sharply reducing the query performance of most adjacency relationships.

[0009] The present application adopts the following technical solutions:

[0010] In a first aspect, the present application provides a graph data storage method, and a data storage structure includes a plurality of partition data tables, an adjacency list, an attribute index table and a vertex index table, and the storage method includes:

[0011] storing data records into corresponding partition data tables; wherein, based on relationships and labels, the plurality of partition data tables are further divided into relationship partition data tables and label partition data tables, and based on the difference between vertices and edges, the data records are further divided into vertex records and edge records, each relationship corresponds to a relationship partition data table for storing edge records with the corresponding relationship, and each label corresponds to a label partition data table for storing vertex records with the corresponding label, the vertex records include vertex IDs and all attributes of the corresponding vertices, and the edge records include edge IDs, vertex IDs of starting vertices of the corresponding edges, vertex IDs of terminal vertices of the corresponding edges and all attributes of the corresponding edges;

[0012] storing attribute index records into the attribute index table to facilitate finding the corresponding data records through attributes; wherein, the attribute index records include attribute values and ROWIDs of the corresponding data records;

[0013] The graph topology data is stored into an adjacency list, so as to find the corresponding data record through an edge or a vertex; wherein, the graph topology data comprises an edge ID, a vertex ID, a topology structure between the edge and the vertex, and a ROWID of the corresponding data record.

[0014] Preferably, the storing of the data record into the corresponding partition data table comprises:

[0015] Each data file is divided into a plurality of pages, and the data record is written into a corresponding page; wherein, each partition data table comprises one or more partitions;

[0016] and a relative offset of the data record in the page, to generate a corresponding ROWID, so as to determine the location of the data record;

[0017] The partition index data of each partition is stored in the first partition of the partition data table, so as to access each partition in the partition data table; wherein, the partition index data of each partition at least comprises a tail page address of the partition and a data record quantity of the partition.

[0018] Preferably, the method further comprises an insertion operation of the data record, and specifically comprises:

[0019] According to the label or the relationship corresponding to the data record, a target partition data table is found;

[0020] According to the partition index data, a target partition with the least data record quantity and without being locked in the target partition data table is found, and a tail page of the target partition is accessed;

[0021] According to the tail page address in the partition index data, it is judged whether the remaining storage space of the tail page is sufficient, if the storage space of the tail page is sufficient, then the partition lock of the target partition is locked, the data record is written into the tail page, and the partition index data of the target partition is updated, and after the data is written successfully, the partition lock of the target partition is unlocked;

[0022] If the remaining storage space of the tail page is insufficient, a new page is allocated for the target partition after the tail page, the new page is used as a new tail page of the target partition, the partition lock of the target partition is locked, the data record is written into the new tail page, after the data is written successfully, the partition lock of the target partition is unlocked, and the partition index data of the target partition is updated.

[0023] Preferably, the method further comprises an update operation of the data record, and specifically comprises:

[0024] determining the partition and storage location of the old data record according to the ROWID of the old data record;

[0025] comparing the data size of the old data record with the data size of the new data record, if the data size of the new data record is not greater than the data size of the old data record, directly overwriting the new data record into the storage location of the old record;

[0026] if the data size of the new data record is greater than the data size of the old data record, writing the new data record into the tail page of the partition in the form of data insertion, and overwriting the pointer pointing to the new data record into the storage location of the old data record.

[0027] Preferably, the method further comprises a data record deletion operation, specifically comprising:

[0028] adding a deletion mark to the data record to be deleted, and updating the number of data records in the partition where the data record to be deleted is located, when a data record with a deletion mark is found, considering that the data record has been deleted, and recycling the storage space of the data record to be deleted by a defragmentation program asynchronously when idle.

[0029] Preferably, the method further comprises a data record deletion operation, specifically comprising:

[0030] each graph corresponds to an adjacency list, and the adjacency list comprises a vertex array, a relationship array, a starting relationship chain table, a terminating relationship chain table, a same relationship edge chain table, a starting edge chain table, and a terminating edge chain table;

[0031] the vertex array stores vertex IDs, entry pointers corresponding to starting relationship chain tables, entry pointers corresponding to terminating relationship chain tables, and ROWIDs corresponding to vertex records;

[0032] each vertex ID in the vertex array corresponds to a starting relationship chain table and a terminating relationship chain table, the starting relationship chain table stores relationship IDs with the corresponding vertex as the starting vertex and entry pointers corresponding to starting edge chain tables, and the terminating relationship chain table stores relationship IDs with the corresponding vertex as the terminating vertex and entry pointers corresponding to terminating edge chain tables;

[0033] each relationship ID in the starting relationship chain table corresponds to a starting edge chain table, and the starting edge chain table stores edge IDs with the corresponding vertex as the starting vertex and having the corresponding relationship, and ROWIDs of corresponding edge records;

[0034] each relationship ID in the terminating relationship chain table corresponds to a terminating edge chain table, and the terminating edge chain table stores edge IDs with the corresponding vertex as the terminating vertex and having the corresponding relationship, and ROWIDs of corresponding edge records;

[0035] The relationship ID, the entry pointer corresponding to the same relationship edge chain table and the ROWID corresponding to the relationship record are stored in the relationship array, and each relationship ID in the relationship array corresponds to a same relationship edge chain table, and the same relationship edge chain table stores the edge ID with the corresponding relationship and the ROWID corresponding to the edge record.

[0036] Preferably, the adjacency list further comprises a label array and a vertex chain table, and specifically comprises:

[0037] The label array stores the label ID, the ROWID corresponding to the label record and the entry pointer of the vertex chain table, and each label ID in the label array corresponds to a vertex chain table, and the vertex chain table stores the vertex ID with the corresponding label.

[0038] Preferably, the starting edge chain table, the ending edge chain table and the same relationship edge chain table are stored in the form of a same edge array, and specifically comprises:

[0039] The edge ID and the ROWID corresponding to the edge record are stored in the edge array, and the first subscript, the second subscript and the third subscript are stored in the edge array; wherein the first subscript is the array subscript corresponding to the next edge in the starting edge chain table, the second subscript is the array subscript corresponding to the next edge in the ending edge chain table, and the third subscript is the array subscript corresponding to the next edge in the same relationship edge chain table.

[0040] Preferably, the method further comprises generating a corresponding data dictionary for each type of graph data, and specifically comprises:

[0041] The graph ID and the graph name are stored as a piece of graph metadata in the data dictionary.

[0042] The attribute ID, the attribute name and the attribute data type are stored as a piece of attribute metadata in the data dictionary.

[0043] The label ID, the label name and the corresponding attribute metadata list are stored as a piece of label data in the data dictionary, and the relationship ID, the relationship name and the corresponding attribute metadata list are stored as a piece of relationship metadata in the data dictionary; wherein the attribute metadata list stores the attribute ID of multiple attribute metadata.

[0044] In order to facilitate the storage of the adjacency list, the partition data table and the attribute index table, the corresponding ID is used for storage.

[0045] In a second aspect, the application further provides a graph data storage device for implementing the graph data storage method of the first aspect, and the device comprises:

[0046] At least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, the instructions are executed by the processor to execute the graph data storage method of the first aspect.

[0047] In a third aspect, the present application further provides a non-volatile computer storage medium, the computer storage medium stores computer executable instructions, the computer executable instructions are executed by one or more processors to complete the graph data storage method of the first aspect.

[0048] The present application stores multiple types of data in the graph data separately, stores each vertex and edge and corresponding attributes, relationships and labels into the partition data table, and stores the graph topology data separately through the adjacency list, so as to avoid the storage size of the adjacency list being too large and causing the query performance of the graph topology data to decline, and at the same time, stores the ROWID of the data record in the adjacency list and the attribute index table, so that the corresponding data record can be quickly found through the adjacency list and the attribute index table, that is, the adjacency list is used as the index for finding the data record according to the graph topology data, and the attribute index table is used as the index for finding the data record according to the attribute, thereby providing powerful graph query function. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0050] Figure 1 is a flowchart of a graph data storage method provided by the embodiments of the present application;

[0051] Figure 2 is a schematic diagram of a partition data table in a graph data storage method provided by the embodiments of the present application;

[0052] Figure 3 is a schematic diagram of an attribute index table in a graph data storage method provided by the embodiments of the present application;

[0053] Figure 4 is a flowchart of a graph data storage method provided by the embodiments of the present application;

[0054] Figure 5 is a schematic diagram of a partition data table of a graph data storage method provided by the embodiments of the present application;

[0055] Figure 6is a flow diagram of a graph data storage method provided by an embodiment of the present application;

[0056] Figure 7 is a schematic diagram of an adjacency list in a graph data storage method provided by an embodiment of the present application;

[0057] Figure 8 is a partial schematic diagram of an adjacency list in a graph data storage method provided by an embodiment of the present application;

[0058] Figure 9 is a partial schematic diagram of an adjacency list in a graph data storage method provided by an embodiment of the present application;

[0059] Figure 10 is a partial schematic diagram of an adjacency list in a graph data storage method provided by an embodiment of the present application;

[0060] Figure 11 is a schematic diagram of an architecture of a graph data storage system provided by an embodiment of the present application;

[0061] Figure 12 is a schematic diagram of an architecture of a graph data storage device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0063] In addition, the technical features involved in each of the embodiments of the present application described below can be combined with each other as long as there is no conflict.

[0064] Embodiment 1:

[0065] Embodiment 1 of the present application provides a graph data storage method, and the data storage structure used by the method includes a plurality of partition data tables, an adjacency list, an attribute index table and a vertex index table, as shown in Figure 1 The storage method includes:

[0066] In step 201, the data records are stored into corresponding partition data tables; wherein, based on the relations and the labels, the plurality of partition data tables are further divided into relation partition data tables and label partition data tables, based on the difference of the vertices and the edges, the data records are further divided into vertex records and edge records, each relation corresponds to a relation partition data table, for storing the edge records with the corresponding relation, each label corresponds to a label partition data table, for storing the vertex records with the corresponding label, the vertex record includes the vertex ID and all attributes of the corresponding vertex, the edge record includes the edge ID, the vertex ID of the starting vertex of the corresponding edge (such as the starting vertex ID shown in Figure 2 ), the vertex ID of the terminal vertex of the corresponding edge (such as the terminal vertex ID shown in Figure 2 ), and all attributes of the corresponding edge, as shown in Figure 2 .

[0067] The data storage structure further includes a partition data table directory, for each label or each relation, there is a partition data table directory, which points to the corresponding partition data table, so as to find the corresponding data record according to the label or the relation. The corresponding partition data table directory can be retrieved according to the label ID or the relation ID. As an embodiment of the partition data table directory, the label ID or the relation ID can be carried in the name of the corresponding partition data table, so that the corresponding partition data table can be obtained through the label ID or the relation ID.

[0068] This embodiment adopts the attribute graph model, that is, the graph data is composed of graphs, vertices, edges, labels (which can be understood as vertex types), relations (which can be understood as edge types) and attributes, that is, a graph is composed of a plurality of vertices and edges, different types of vertices are classified according to labels, different types of edges are classified according to relations, and the vertices and edges can correspond to one or more attributes.

[0069] The partition data table does not need to consider sorting, and only sequential writing when inserting, so the performance is higher than that of traditional random writing, especially when loading large amounts of data, the performance is improved more obviously. And the data is organized according to multiple partitions, in the case of concurrency, different users can insert on different partitions, the writing efficiency can be greatly improved, thereby the storage performance can be greatly improved.

[0070] In step 202, the attribute index record is stored into the attribute index table, so as to find the corresponding data record through the attribute; wherein, the attribute index record includes the attribute value and the ROWID of the corresponding data record. The ROWID represents the storage position of the data record in the partition data table. Through the ROWID, the corresponding data record can be quickly accessed.

[0071] One vertex or one edge can correspond to one or more attributes, and when multiple vertices or edges correspond to attributes with the same attribute value, multiple attribute index records with the same attribute value are stored in the attribute index table. For example, there is a vertex record with vertex ID vertex1 and attribute property1, and ROWID is 12, and there is an edge record with edge ID edge1 and attribute property1, and ROWID is 5. Therefore, there are two attribute index records <property1, 12> and <property1, 5> in the attribute index table.

[0072] The attribute index record is implemented using a B+ tree. Specifically, a Key-Value key-value pair data structure is implemented based on a B+ tree, that is, the attribute value is taken as the Key value, and other data is taken as the Value value, so that the index value can be quickly found through the attribute value.

[0073] The Value (i.e., Property_value) in the key-value pair of the attribute index record has three storage modes:

[0074] The first mode is to store ROWID and vertex ID at the same time or to store ROWID and edge ID at the same time, as shown in FIG. 1. Figure 3 When the adjacency relationship needs to be queried, adjacency query is performed in the grouped adjacency table based on the vertex / edge ID. When the point / edge attribute data needs to be queried, the point / edge detailed record is queried in the partitioned data table based on the ROWID.

[0075] The second mode is to store only the vertex ID or the edge ID. When the point / edge attribute data needs to be queried, the point / edge record is found in the grouped adjacency table based on the vertex / edge ID, and then the ROWID is obtained, and the point / edge attribute is queried in the partitioned data table based on the ROWID.

[0076] The third mode is to store only the ROWID of the data record. When the point / edge adjacency relationship needs to be queried, the point / edge detailed record is queried in the partitioned data table based on the ROWID, and then the point / edge ID is obtained, and adjacency query is performed in the grouped adjacency table based on the point / edge ID.

[0077] In step 203, the graph topology data is stored in the adjacency table, so that the corresponding data record can be found through the edge or vertex; wherein the graph topology data includes edge ID, vertex ID, topology structure between edges and vertices, and ROWID of the corresponding data record.

[0078] The topology structure between the edges and the vertices is specifically the relationship between the edges and the starting vertices of the edges and the ending vertices of the edges.

[0079] The embodiment separates and stores multiple types of data in the graph data, stores each vertex and edge and corresponding attributes, relations and labels into a partition data table, and separately stores graph topology data through an adjacency table, so as to avoid that the storage size of the adjacency table is too large and the query performance of the graph topology data is reduced, and meanwhile, the ROWIDs of data records are stored in the adjacency table and the attribute index table, so that the corresponding data records can be quickly found through the adjacency table and the attribute index table, that is, the adjacency table is used as an index for finding data records according to graph topology data, and the attribute index table is used as an index for finding data records according to attributes, thereby providing powerful graph query functions.

[0080] In actual use, when storing graphs, attributes, labels or relations, direct storage of corresponding names may cause repeated storage of the same name, and further cause too large storage space occupation. In view of this problem, the embodiment provides the following preferred implementation manner, that is, the method further comprises generating a corresponding data dictionary for each type of graph data, specifically comprising:

[0081] The graph ID and the graph name are stored as a piece of graph metadata in the data dictionary.

[0082] The attribute ID, the attribute name and the attribute data type are stored as a piece of attribute metadata in the data dictionary.

[0083] The label ID, the label name and a corresponding attribute metadata list are stored as a piece of label data in the data dictionary, and the relation ID, the relation name and a corresponding attribute metadata list are stored as a piece of relation metadata in the data dictionary, wherein the attribute metadata list stores attribute IDs of multiple attribute metadata.

[0084] In order to facilitate storage of the adjacency table, the partition data table and the attribute index table, corresponding IDs are used for storage.

[0085] Each metadata in the data dictionary formed by the embodiment is defined in a strong mode, the graph metadata includes the graph ID and the graph name, and the data model thereof can be represented as <graph ID, graph name>, the label and relation metadata are respectively used to constrain attributes corresponding to vertices or edges, and a same data model is used for storage, including the label or relation ID, the label or relation name and multiple attribute metadata, and the data model thereof can be represented as <label / relation ID, label / relation name, attribute metadata list>, the label and the relation share a same data model, wherein the attribute metadata includes the attribute ID, the attribute name and the attribute data type, and the data model thereof can be represented as <attribute ID, attribute name, attribute data type>, and the corresponding IDs of the above metadata can be automatically sorted and generated by the system. Thus, when storing the partition data table, the attribute index table and the adjacency table, the relation ID, the label ID and the attribute ID are used for alternative storage, thereby reducing storage space occupation.

[0086] This embodiment also provides an optional implementation method, namely, storing the data records in the corresponding partition data table, such as... Figure 4 As shown, it specifically includes:

[0087] In step 301, each data file is treated as a partition, and each data file is divided into multiple pages. Data records are then written to the corresponding pages. Each partition data table contains one or more partitions, and each partition corresponds to one data file.

[0088] Specifically, vertex records are stored in the corresponding label partitioned data table according to their corresponding labels, and edge records are stored in the corresponding relation partitioned data table according to their corresponding relations.

[0089] The partition table in which a data record is stored is determined by its corresponding tag or relationship. The corresponding partition table directory is found based on its tag or relationship, and the partition table pointed to by the directory is the partition table used to store the data record.

[0090] In step 302, a corresponding ROWID is generated based on the file number of the partition where the data record is located, the page number of the page where the data record is located, and the relative offset of the data record within the page, in order to determine the location of the data record.

[0091] In step 303, the partition index data of each partition is stored in the first partition of the partition data table to facilitate access to each partition in the partition data table; wherein, the partition index data of each partition includes at least the last page address of the partition and the number of data records of the partition.

[0092] The number of partitions in the partition data table is determined by those skilled in the art based on the size of the data to be stored and the different concurrency levels of data reading and writing, and the number of partitions in the partition data table is specified when the partition data table is created.

[0093] The first partition does not refer to a specific partition, but rather to a single partitioned data table. That is, each partitioned data table has a first partition used to store the partition index data of all partitions within that table. As an optional implementation, the first partition can be the first partition in the partitioned data table. For example, if there are two partitioned data tables, table_1 and table_2, then table_1 has a first partition, called the first partition of table_1, used to store the partition index data of each partition in table_1. Similarly, table_2 also has a first partition, called the first partition of table_2, used to store the partition index data of each partition in table_2.

[0094] The overall storage architecture of each partition data table is shown in Figure 5 As shown in the figure, a partition index data is maintained for each partition, which is stored in a partition linked list saved in the file header of the first partition (i.e. the first partition). The number of pages occupied by the partition file header varies according to the number of partitions. Each partition index data in the partition linked list saves a tail page pointer (i.e. a pointer to the tail of the last record in the tail page) pointing to the corresponding partition file, so that the insertion position of data can be obtained through the pointer, and the number of data records of the corresponding partition is also saved, which can provide a basis for selecting a partition for insertion operation according to the size of the number of data records.

[0095] Each data in the partition data table has a unique ROWID. As an optional implementation, the data structure of the ROWID can be represented as <FILE_ID, PAGE_ID, PAGE_OFFSET>, where FILE_ID is the file number of the partition where the data is located, PAGE_ID is the page number of the page where the data is located, and PAGE_OFFSET is the relative offset of the data in the page. Therefore, if the ROWID of a data record is known, the file number, page number and page offset of the record can be obtained through the ROWID, so that the record can be quickly found.

[0096] For the storage of the adjacency list, the embodiment also provides an optional implementation, i.e. the storage of the graph topology data into the adjacency list, which specifically includes:

[0097] Each graph corresponds to an adjacency list, and the adjacency list includes a vertex array, a relationship array, a starting relationship linked list, a terminating relationship linked list, a same relationship edge linked list, a starting edge linked list and a terminating edge linked list, as shown in Figure 6

[0098] In step 401, the vertex ID, the entry pointer of the corresponding starting relationship linked list, the entry pointer of the corresponding terminating relationship linked list and the ROWID of the corresponding vertex record are stored in the vertex array.

[0099] In step 402, each vertex ID in the vertex array corresponds to a starting relationship linked list and a terminating relationship linked list. The starting relationship linked list stores the relationship ID with the corresponding vertex as the starting vertex and the entry pointer of the corresponding starting edge linked list, and the terminating relationship linked list stores the relationship ID with the corresponding vertex as the terminating vertex and the entry pointer of the corresponding terminating edge linked list.

[0100] ​In step 403, each relationship ID in the starting relationship chain table corresponds to a starting edge chain table, and the starting edge chain table stores the edge ID with the corresponding relationship and the ROWID of the corresponding edge record with the corresponding vertex as the starting vertex; each relationship ID in the ending relationship chain table corresponds to an ending edge chain table, and the ending edge chain table stores the edge ID with the corresponding relationship and the ROWID of the corresponding edge record with the corresponding vertex as the ending vertex.

[0101] In step 404, the relationship ID, the entry pointer of the corresponding same-relationship edge chain table and the ROWID of the corresponding relationship record are stored in the relationship array, each relationship ID in the relationship array corresponds to a same-relationship edge chain table, and the same-relationship edge chain table stores the edge ID with the corresponding relationship and the ROWID of the corresponding edge record.

[0102] The formed adjacency list is shown in Figure 7 The starting relationship chain table entry represents the starting relationship chain table entry pointer, the edge chain table entry of the relationship in the starting relationship chain table represents the entry pointer of the starting edge chain table, and the like, in the starting relationship chain table, one relationship ID corresponds to one starting edge chain table, Figure 7 In the actual use, as many starting relationship chain tables as relationship IDs exist, as many starting edge chain tables as starting relationship chain tables exist.

[0103] All edges in one starting edge chain table have the same starting vertex and the same relationship; all edges in one ending edge chain table have the same ending vertex and the same relationship; all edges in one same-relationship edge chain table have the same relationship, but the starting vertex or the ending vertex can be different.

[0104] It should be noted that the starting edge chain table, the ending edge chain table and the same-relationship edge chain table are only a description form, and do not necessarily represent the storage structure as a data chain table, in the subsequent optional embodiments, the storage can also be in the form of an array, which will be described in detail in the subsequent content, and is not described here.

[0105] It should be noted that, for the sake of clarity and simplicity of description, the embodiment takes the “relationship ID with the corresponding vertex as the starting vertex” as the abbreviation of the “relationship ID of the edge with the corresponding vertex as the starting vertex”, and takes the “relationship ID with the corresponding vertex as the ending vertex” as the abbreviation of the “relationship ID of the edge with the corresponding vertex as the ending vertex”.

[0106] The embodiment also has a preferred embodiment, as shown in Figure 8 The adjacency list further includes a label array and a vertex chain table, and specifically includes:

[0107] The label array stores label IDs, ROWIDs of corresponding label records, and entry pointers of vertex chain tables.

[0108] Each label ID in the label array corresponds to a vertex chain table, which stores vertex IDs having the corresponding label.

[0109] The adjacency list formed by the above two embodiments has both the tables as shown in Figure 7 and the tables as shown in Figure 8 All vertices in a graph are stored in an array.

[0110] In actual graph applications, there are many grouping queries based on labels and relationships, such as querying all vertices with the label "person", or querying all adjacent vertices with the adjacent relationship "follow" of the vertex "Zhang San". In order to meet the needs of such grouping queries, the vertices in the vertex array are first grouped according to labels, and vertices with the same label are concatenated into a vertex chain table. The entry pointer of each chain table is saved in each label element of the label array.

[0111] The adjacent edges of each vertex are grouped according to the relationship to which the edge belongs. All relationships of adjacent edges are divided into two chain tables according to the outgoing edges and incoming edges of the vertex, namely the starting relationship chain table (hereinafter also referred to as the relationship chain table of the same starting vertex) and the terminating relationship chain table (hereinafter also referred to as the relationship chain table of the same terminating vertex). The starting relationship chain table is used to store all relationships of edges with the corresponding vertex as the starting vertex. The terminating relationship chain table is used to store all relationships of edges with the corresponding vertex as the terminating vertex. In the starting relationship chain table, the entry pointer of the corresponding starting edge chain table is also stored for each relationship. The starting edge chain table stores edges having the relationship and with the vertex as the starting vertex. In the terminating relationship chain table, the entry pointer of the corresponding terminating edge chain table is also stored for each relationship. The terminating edge chain table stores edges having the relationship and with the vertex as the terminating vertex.

[0112] In actual use, since a directed edge corresponds to a starting vertex and a terminating vertex, when stored according to the above embodiments, a directed edge must be stored once in the corresponding starting edge chain table and once in the corresponding terminating edge chain table, resulting in the occupation of additional storage space due to repeated storage. In view of this problem, the present embodiment provides the following preferred embodiments, specifically including:

[0113] The starting edge chain table, the ending edge chain table and the same relation edge chain table are stored in the form of a multi-head edge chain table, i.e. the edges are stored in the multi-head edge chain table, and the multi-head edge chain table further stores a first pointer, a second pointer and a third pointer corresponding to each edge. The starting edge chain table is composed of multiple edges and corresponding first pointers, and the first pointer of a target edge points to the next edge of the target edge in the starting edge chain table. The ending edge chain table is composed of multiple edges and corresponding second pointers, and the second pointer of a target edge points to the next edge of the target edge in the ending edge chain table. The same relation edge chain table is composed of each edge and a corresponding third pointer, and the third pointer of a target edge points to the next edge of the target edge in the same relation edge chain table. The formed multi-head edge chain table is as shown in Figure 9 .

[0114] In order to further optimize the storage structure and reduce the storage space occupation, the embodiment further provides the following preferred implementation, i.e. the starting edge chain table, the ending edge chain table and the same relation edge chain table are stored in the form of an edge array, as shown in Figure 10 .

[0115] The ROWID of the corresponding edge record is stored in the edge array, and a first subscript, a second subscript and a third subscript are stored in the edge array. The first subscript is the array subscript corresponding to the next edge in the starting edge chain table, the second subscript is the array subscript corresponding to the next edge in the ending edge chain table, and the third subscript is the array subscript corresponding to the next edge in the same relation edge chain table.

[0116] The next edge is the next edge corresponding to the current record, and the corresponding edge ID can also be stored in the edge array. For example, the edge ID of an edge is edge1, the array subscript of edge1 is 1, the edge ID of the next edge in the starting edge chain table containing edge1 is edge2, the array subscript of edge2 is 3, the edge ID of the next edge in the ending edge chain table containing edge1 is edge3, the array subscript of edge3 is 3, the edge ID of the next edge in the same relation edge chain table containing edge1 is edge4, the array subscript of edge4 is 4, the first subscript of edge1 is 2, the second subscript of edge1 is 3, and the third subscript of edge1 is 4.

[0117] It should be noted that since one edge corresponds to only one starting vertex and one ending vertex, and only corresponds to one relationship, that is, one edge only exists in one starting edge chain table, one ending edge chain table and one same relationship edge chain table, and each has a determined next edge, therefore, the plurality of starting edge chain tables, the plurality of ending edge chain tables and the plurality of same relationship edge chain tables in the adjacency list can be stored in the same edge array, and when the corresponding edge has no next edge, the corresponding subscript is set to be a null value or a default value.

[0118] In actual use, the graph data storage method further includes insertion, update and deletion operations of each data, and the embodiment further takes the insertion, update and deletion operations of attributes as examples to provide the following optional implementation, that is, the method further includes an insertion operation of a data record, specifically including:

[0119] According to the label or relationship corresponding to the data record, a corresponding target partition data table is found. Specifically, according to the label ID or relationship ID, a corresponding partition data table directory is found, and a corresponding partition data table, that is, the target partition data table, is obtained according to the partition data table directory.

[0120] According to the partition index data, a target partition with the least number of data records and not being locked in the target partition data table is found, and a tail page of the target partition is accessed.

[0121] According to the tail page address in the partition index data, it is judged whether the remaining storage space of the tail page is sufficient, if the storage space of the tail page is sufficient, the partition lock of the target partition is locked, the data record is written into the tail page, the number of data records corresponding to the target partition is updated, and after the data is written successfully, the partition lock of the target partition is unlocked.

[0122] If the remaining storage space of the tail page is insufficient, a new page is allocated for the target partition after the tail page, the new page is used as a new tail page of the target partition, the partition lock of the target partition is locked, the data record is written into the new tail page, after the data is written successfully, the partition lock of the target partition is unlocked, and the partition index data of the target partition is updated.

[0123] For the insertion operation, in the actual implementation, a corresponding insertion interface is usually used to implement the following process: the system first selects a suitable partition (a partition that is not locked and has a smaller tail page pointer address) from the partition linked list, and locks the partition, and obtains the tail page pointer of the partition. If the remaining space of the page is greater than the size of the to-be-inserted record, the record is directly inserted, and the tail page pointer position and the number of data records of the partition are updated. If the tail page space is insufficient, a new page is allocated after the tail page, and the newly allocated page is referred to as a new tail page. If the remaining space of the file is insufficient, automatic expansion is performed. After successful insertion, the partition lock is released, and the insertion interface returns a ROWID composed of a file number, a page number, and an offset in the page, which is the physical address corresponding to the record. The ROWID is written into the attribute index table and the adjacency table when corresponding data is inserted into the attribute index table and the adjacency table, so as to facilitate subsequent data record searching according to attributes, vertices, or edges, and the tail page pointer and the number of data records of the partition are updated.

[0124] The method further includes an update operation of a data record, and specifically includes the following steps.

[0125] According to the ROWID of the old data record, the partition and the storage position of the old data record are determined.

[0126] The data size of the old data record is compared with the data size of the new data record. If the data size of the new data record is not greater than the data size of the old data record, the new data record is directly overwritten and written into the storage position of the old record.

[0127] If the data size of the new data record is greater than the data size of the old data record, the new data record is written into the tail page of the partition in the form of data insertion, and a pointer pointing to the new data record is overwritten and written into the storage position of the old data record.

[0128] For the update operation, because the ROWID of the old data record is referenced by other indexes, in order not to affect the data reference of the record by other indexes, the physical address of the record cannot be changed, and the mode of first deleting and then inserting adopted in some other embodiments is not suitable. The embodiment of the present application adopts an in-place update mode. First, the record is found according to the ROWID, and the record length after update is compared with the record length before update. If the record length after update is less than or equal to the record length before update, the record is modified in place. If the record length after update is greater than the record length before update, the original record space cannot accommodate the record after update, and a record after update is inserted at the tail of the same partition, and the original record is deleted and a pointer pointing to the record after update is added at the original record.

[0129] The preferred embodiment avoids the increase of data volume caused by data update by updating the old and new data in the same storage location, thereby saving storage space.

[0130] For the same reason, the method further comprises a data record deletion operation, specifically comprising:

[0131] A deletion mark is added to the data record to be deleted, and the number of data records in the partition where the data record to be deleted is located is updated, when a data record with a deletion mark is found, it is considered that the data record has been deleted, and the storage space of the data record to be deleted is asynchronously recycled by a defragmentation program when it is idle.

[0132] For the deletion operation, only a deletion mark is added to the record to quickly respond to user operations, and the recycling of the space is performed asynchronously by a defragmentation program subsequently. Meanwhile, the number of data records in the partition is updated, for example, when a deletion mark is added to a record, the number of data records in the partition where the record is located is reduced by one.

[0133] There is also a preferred embodiment that the adjacency relationship of the same vertex is preferably preserved in the same partition, thereby improving the cache hit rate of disk and memory access, and further improving the access performance of the adjacency relationship.

[0134] Embodiment 2:

[0135] On the basis of the above-mentioned embodiment 1, the embodiment of the application further provides a graph data storage system for implementing the graph data storage method described in embodiment 1, as shown in Figure 11 The system comprises:

[0136] A data dictionary management module is configured to create, modify, delete and query metadata such as graphs, labels and relationships.

[0137] A partition data table management module is configured to insert, update, delete and query metadata and vertex records or edge records, and manage partitions, files, pages and records.

[0138] A group adjacency table management module is responsible for managing the adjacency index of vertices and edges, assigning IDs to vertices and edges, inserting vertices and edges into appropriate positions in arrays and linked lists, maintaining index structures, and performing multi-hop adjacency queries.

[0139] An attribute index management module is configured to create, delete and update attribute indexes, and perform index queries based on attributes.

[0140] An executor module is configured to receive user requests, determine the data structure to be accessed according to the request type and parameters, and call the corresponding module for execution.

[0141] The partition data table management module further comprises:

[0142] The file automatic expansion module is configured to automatically expand the file when the partition data file space is insufficient.

[0143] The file fragment recovery module is configured to arrange and recover the space where the records marked with the deletion mark are located when the partition data is subjected to multiple deletion operations.

[0144] Embodiment 3:

[0145] As Figure 12 shown in FIG. 1, which is a schematic diagram of the architecture of the graph data storage device according to an embodiment of the present application. The graph data storage device according to the embodiment comprises one or more processors 21 and a memory 22. In the embodiment, the processor 21 is taken as an example. Figure 12

[0146] The processor 21 and the memory 22 can be connected through a bus or other means, Figure 12 for example, through a bus.

[0147] The memory 22 is a non-volatile computer readable storage medium, which can be used to store non-volatile software programs and non-volatile computer executable programs, such as the graph data storage method in Embodiment 1. The processor 21 executes the graph data storage method by running the non-volatile software programs and instructions stored in the memory 22.

[0148] The memory 22 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 22 can optionally include a memory remotely arranged with respect to the processor 21, and these remote memories can be connected to the processor 21 through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0149] The program instructions / modules are stored in the memory 22, and when executed by the one or more processors 21, the graph data storage method in Embodiment 1 is executed.

[0150] It is worth noting that the information interaction, execution process, and the like between the modules and units in the above-mentioned device and system are based on the same concept as the processing method embodiments of the present application, and the specific content can be referred to the description in the method embodiments of the present application, which will not be described here.

[0151] ​Those skilled in the art can understand that all or part of the steps in the various methods of the embodiments can be completed by instructing the related hardware with a program, and the program can be stored in a computer readable storage medium, which can include Read Only Memory (ROM), Random Access Memory (RAM), a magnetic disk or an optical disk, etc.

[0152] The above description is merely the preferred embodiments of the present application, and is not used to limit the present application. Any modification, equivalent replacement and improvement made in the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A graph data storage method characterized by comprising: The data storage structure comprises a plurality of partition data tables, an adjacency table, an attribute index table and a vertex index table, and the storage method comprises: storing data records into corresponding partition data tables; wherein, based on relations and tags, the plurality of partition data tables are further divided into relation partition data tables and tag partition data tables, and based on different vertices and edges, the data records are further divided into vertex records and edge records, each relation corresponds to a relation partition data table for storing edge records with the corresponding relation, and each tag corresponds to a tag partition data table for storing vertex records with the corresponding tag, the vertex records comprise vertex IDs and all attributes of the corresponding vertex, and the edge records comprise edge IDs, vertex IDs of starting vertices of the corresponding edge, vertex IDs of terminal vertices of the corresponding edge and all attributes of the corresponding edge; storing attribute index records into the attribute index table so as to find corresponding data records through attributes; wherein, the attribute index records comprise attribute values and ROWIDs of the corresponding data records; storing graph topology data into the adjacency table so as to find corresponding data records through edges or vertices; wherein, the graph topology data comprises edge IDs, vertex IDs, topologies between edges and vertices and ROWIDs of the corresponding data records; the storing of the graph topology data into the adjacency table comprises: each graph corresponds to an adjacency table, the adjacency table comprises a vertex array, a relation array, a starting relation linked list, a terminal relation linked list, a same-relation edge linked list, a starting edge linked list and a terminal edge linked list; the vertex array stores vertex IDs, entry pointers of corresponding starting relation linked lists, entry pointers of corresponding terminal relation linked lists and ROWIDs of corresponding vertex records; each vertex ID in the vertex array corresponds to a starting relation linked list and a terminal relation linked list, the starting relation linked list stores relation IDs with the corresponding vertex as a starting vertex and entry pointers of corresponding starting edge linked lists, and the terminal relation linked list stores relation IDs with the corresponding vertex as a terminal vertex and entry pointers of corresponding terminal edge linked lists; each relation ID in the starting relation linked list corresponds to a starting edge linked list, the starting edge linked list stores edge IDs with the corresponding vertex as a starting vertex and having the corresponding relation and ROWIDs of corresponding edge records; each relation ID in the terminal relation linked list corresponds to a terminal edge linked list, the terminal edge linked list stores edge IDs with the corresponding vertex as a terminal vertex and having the corresponding relation and ROWIDs of corresponding edge records; the relation array stores relation IDs, entry pointers of corresponding same-relation edge linked lists and ROWIDs of corresponding relation records, each relation ID in the relation array corresponds to a same-relation edge linked list, and the same-relation edge linked list stores edge IDs having the corresponding relation and ROWIDs of corresponding edge records.

2. The graph data storage method of claim 1, wherein, the storing of the data records into corresponding partition data tables specifically comprises: dividing each data file into a plurality of pages and writing data records into corresponding pages, taking each data file as a partition in the corresponding partition data table; wherein, each partition data table comprises one or more partitions. According to the file number of the partition where the data record is located, the page number of the page where the data record is located, and the relative offset of the data record in the page, a corresponding ROWID is generated to determine the location of the data record; The partition index data of each partition is stored in the first partition of the partition data table to facilitate access to each partition in the partition data table; wherein the partition index data of each partition at least includes the tail page address of the partition and the number of data records in the partition.

3. The graph data storage method of claim 2, wherein, The method further includes an insertion operation of a data record, specifically comprising: According to the label or relationship corresponding to the data record, find the corresponding target partition data table; According to the partition index data, find the target partition with the least number of data records and not locked in the target partition data table, and access the tail page of the target partition; According to the tail page address in the partition index data, determine whether the remaining storage space of the tail page is sufficient, if the storage space of the tail page is sufficient, lock the partition lock of the target partition, write the data record to the tail page, and update the partition index data of the target partition, and after the data is written successfully, unlock the partition lock of the target partition; If the remaining storage space of the tail page is insufficient, allocate a new page for the target partition after the tail page, use the new page as the new tail page of the target partition, lock the partition lock of the target partition, write the data record to the new tail page, and after the data is written successfully, unlock the partition lock of the target partition, and update the partition index data of the target partition.

4. The graph data storage method of claim 2, wherein, The method further includes an update operation of a data record, specifically comprising: According to the ROWID of the old data record, determine the location and storage location of the old data record; Compare the data size of the old data record with the data size of the new data record, if the data size of the new data record is not greater than the data size of the old data record, directly overwrite the new data record to the storage location of the old record; If the data size of the new data record is greater than the data size of the old data record, write the new data record in the form of data insertion to the tail page of the partition, and overwrite the pointer pointing to the new data record to the storage location of the old data record.

5. The graph data storage method of claim 2, wherein, The method further includes a deletion operation of a data record, specifically comprising: Add a deletion mark to the data record to be deleted, and update the number of data records in the partition where the to-be-deleted record is located, when the corresponding data record is found to have a deletion mark, it is considered that the data record has been deleted, and the storage space of the to-be-deleted data record is asynchronously recycled by the defragmentation program when idle.

6. The graph data storage method of claim 1, wherein, The adjacency list further includes a label array and a vertex chain table, specifically comprising: The label array stores label ID, corresponding label record ROWID, and entry pointer of vertex chain table; Each label ID in the label array corresponds to a vertex chain table, and the vertex chain table stores vertex ID with the corresponding label.

7. The graph data storage method of claim 1, wherein, The starting edge chain table, the ending edge chain table, and the same relationship edge chain table are stored in the form of an edge array, specifically comprising: The ROWIDs of the corresponding edges are stored in an edge array, and a first index, a second index and a third index are stored in the edge array; the first index is an array index corresponding to a next edge in a starting edge linked list, the second index is an array index corresponding to a next edge in a terminal edge linked list, and the third index is an array index corresponding to a next edge in a same relation edge linked list.

8. The graph data storage method according to any one of claims 1-7, wherein, The method further comprises generating a corresponding data dictionary for each type of graph data, specifically comprising: storing a graph ID and a graph name as a piece of graph metadata in the data dictionary; storing an attribute ID, an attribute name and an attribute data type as a piece of attribute metadata in the data dictionary; storing a label ID, a label name and a corresponding attribute metadata list as a piece of label data in the data dictionary, and storing a relation ID, a relation name and a corresponding attribute metadata list as a piece of relation metadata in the data dictionary; the attribute metadata list stores attribute IDs of multiple pieces of attribute metadata; so as to facilitate storage of the adjacency list, the partition data table and the attribute index table using the corresponding IDs.

9. A graph data storage apparatus, characterized by comprising: The apparatus comprises: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the graph data storage method of any one of claims 1-8.

Images