A distributed native graph implementation method, system, device and storage medium

Abstract

The application discloses an implementation method, system and device of a distributed original graph and a storage medium. Access to vertices / edges of the whole distributed system is directly based on global array subscripts, is fast, avoids calculation, and automatically maintains adjacency relations. When N-hop query is performed, access is directly based on global array subscripts instead of keys, various comparison operations required in various tree structures are avoided, and the performance of array subscript access is improved by up to 190 times compared with key-based access.

Description

Technical Field

[0001] This invention relates to the field of database technology, and in particular to a method, system, device, and storage medium for implementing a distributed native graph. Background Technology

[0002] Graph databases can be divided into native graph databases and non-native graph databases, depending on their underlying storage implementation.

[0003] Native graph databases use graph models for data storage and are designed and optimized for storing and processing graphs. They support fast traversal of various graph algorithms, resulting in better performance. Index-free adjacency is key to native graph processing. During data writing, index-free adjacency accelerates processing by directly storing each site and its adjacent sites and relationships. During data query processing, index-free adjacency enables fast retrieval.

[0004] Non-native graph databases typically use non-graph model storage such as relational databases for underlying physical storage, encapsulating graph semantics on top of the storage for graph processing. To represent graph adjacency relationships, new relational tables are usually introduced into the relational database. When querying adjacent vertices / edges, both the entity table and the relational table need to perform a join query. To improve performance, indexes (typically B-trees / B+ trees / LSM trees / skiplists, etc.) are created on both the entity table and the relational table. If multi-hop adjacent vertices / edges are to be queried, the number of joined tables increases significantly. Relational databases often struggle to find the optimal table join order within a reasonable timeframe. Even if the optimal join order is found, repeated comparisons on tree indexes can lead to poor query performance.

[0005] On the other hand, as graph data applications become increasingly complex, single-machine graph databases struggle to meet the storage, query, and analysis needs of large-scale graph data. This has led to the development of distributed graph databases whose underlying storage is based on non-native graphs. When performing adjacency queries, non-native graphs access data based on keys, a method limited by the underlying storage model, especially in multi-level adjacency queries across different sites, where performance degrades significantly. Currently, the mainstream graph database Neo4j uses native graph storage, but it does not support distributed functionality and cannot scale horizontally, making it unsuitable for large-scale graph data applications. Summary of the Invention

[0006] The technical problem to be solved by this invention is to provide a method, system, device and storage medium for implementing distributed native graphs, which can directly perform distributed access based on adjacency arrays and global array indices, better support large-scale graph data and significantly improve the performance of various relational analysis queries.

[0007] To address the aforementioned technical problems, this invention provides a method for implementing a distributed native graph, comprising the following steps:

[0008] Step 1: During system operation, an array index structure is automatically established on each data node;

[0009] Step 2: During the operation of the distributed native graph system, worker threads / processes parse the received operation requests and determine the operation type;

[0010] Step 3: Perform the corresponding operation according to the different operation types.

[0011] Preferably, in step 1, the automatic creation of array index structures on each data node during system operation is as follows: the system automatically creates global vertex array indexes and edge array indexes during initialization; when creating entity tables and relational tables, vertex array indexes and edge array indexes are automatically created for specific tables by specifying options; and existing entity tables and relational tables are converted into vertex tables and edge tables by using SQL statements, thereby automatically creating vertex array indexes and edge array indexes.

[0012] Preferably, in step 3, the corresponding operation is performed according to the different operation types, specifically as follows:

[0013] (1) When the operation type is to insert vertex data, the worker thread / process calculates the station number to which the vertex needs to be sent, and then assigns a ROWID, where Graph_id indicates which graph the vertex belongs to, Node_id indicates the station number obtained by calculation, and also contains a monotonically increasing vertex ID, which indicates the index of the vertex array file of the graph corresponding to Graph_id at station Node_id; In addition to inserting the physical record corresponding to the attribute data of the vertex into the data file, for the main index of the table where the vertex is located, the physical record address corresponding to the attribute data and the vertex ROWID are stored in the leaf node at the same time; then the address information Address of the physical record of the vertex in the data file is filled into the ID-th cell of the vertex array, and last_out_edge_rowid and last_in_edge_rowid are set to an invalid value (such as -1|-1|-1|-1|, because there is no edge associated with the vertex yet), and the user is returned the edge ROWID and the success result;

[0014] (2) When the operation type is to update vertex data, it is necessary to first determine the station where the vertex is located based on the vertex ROWID, and then perform the vertex update operation on this station; if the data storage location remains unchanged, the array item corresponding to the vertex will not be changed; if the data storage location changes, the address in the vertex array index item corresponding to the record will be updated.

[0015] (3) When the operation type is to delete a vertex, it is necessary to first determine the station where the vertex is located based on the vertex ROWID, and then perform the deletion operation on this station; set the Address of the vertex array index item corresponding to the vertex to a specific flag (such as the corresponding maximum unsigned value 0xffffffffffffffff, etc.) to indicate that the vertex has been deleted;

[0016] (4) When the operation type is inserting edge data, the worker thread / process calculates the station number to which the edge needs to be sent, and then assigns a ROWID, where Graph_id indicates which graph the edge belongs to, Node_id indicates the station number of the station where the starting vertex of the edge is located, and also contains a monotonically increasing edge ID, which represents the index of the edge array file of the graph corresponding to Graph_id at station Node_id; in addition to inserting the physical record corresponding to the attribute data of the edge into the data file, for the main index of the table where the edge is located, the physical record address corresponding to the attribute data and the edge ROWID are stored in the leaf node; then the corresponding ID-th cell in the edge array is... Enter the address information of the physical record corresponding to the edge in the data file, the ROWID of the starting vertex, the ROWID of the previous outgoing edge of the starting vertex, the ROWID of the ending vertex, and the ROWID of the previous incoming edge of the ending vertex. Then set the ROWID of the previous outgoing edge of the starting vertex to the ROWID of the inserted edge. If the ending vertex and the starting vertex are on the same site, set the ROWID of the previous incoming edge of the ending vertex to the ROWID of the inserted edge. If the ending vertex and the starting vertex are not on the same site, the edge information needs to be inserted at the site where the ending vertex is located. After completing the above operations, return the edge ROWID and success result to the user.

[0017] (5) When the operation type is to update edge data, it is necessary to determine the station where the edge is located based on the edge ROWID, and then perform the edge update operation on this station. If the location where the data is stored remains unchanged, the edge array item corresponding to the edge will not be changed. If the location where the data is stored changes, the address in the edge array index item corresponding to the record will be updated.

[0018] (6) When the operation type is to delete an edge, it is necessary to determine the station where the edge is located based on the edge ROWID, and then perform the update operation on this station; set the Address of the edge array index item corresponding to the edge to a specific flag (such as the corresponding maximum unsigned value 0xffffffffffffffff, etc.) to indicate that the edge has been deleted;

[0019] (7) When the operation type is to look up vertex / edge information based on the ROWID of the vertex / edge, the site where the vertex / edge is located and the vertex / edge ID are found according to the ROWID. The address of the record can be found directly based on the corresponding ID item in the array at this site, and then the detailed information of the vertex / edge can be found directly at this address.

[0020] (8) When the operation type is to find the nth hop adjacent vertex / edge information based on ROWID, obtain the nth hop adjacent vertex / edge information based on ROWID;

[0021] (9) When the operation type is to find the nth hop adjacent vertex / edge information based on the edge ROWID, obtain the table type (Table_type), graph number (Graph_id), site number (Node_id), and edge ID index of the edge based on the vertex ROWID. On the site with site number Node_id, find the corresponding graph based on Graph_id, find the edge array based on the table type, and then obtain the ROWID of the starting vertex and the ending vertex from the ID item of the edge array. Then, obtain the nth hop adjacent vertex / edge information based on the starting vertex and the ending vertex, respectively.

[0022] Preferably, in step (4), when the operation type is inserting edge data, the specific steps include the following:

[0023] (a) The worker thread / process will obtain the station number to be sent based on the ROWID of the starting vertex of the edge. In a distributed environment, the attribute data of the edge is inserted into the same station as the starting vertex of the edge by default, and then an edge ROWID is assigned. The Graph_id indicates which graph the edge belongs to, the Node_id is the same as the station number of the starting vertex, and also contains an edge ID that is monotonically increasing on the Node_id station. This ID represents the index of the edge array file of the graph corresponding to Graph_id on the station Node_id.

[0024] (b) On the Node_id site, in addition to the physical record corresponding to the attribute data of the inserted edge, the leaf node stores the physical record address corresponding to the attribute data and the edge ROWID for the main index of the table where the edge is located.

[0025] (c) Then, at the Node_id site, fill in the address of the physical record of the edge in the data file in the ID-th cell of the edge array. The ROWID of the starting vertex: v1 and the ROWID of the previous outgoing edge of v1: v1_last_out_edge_rowid are obtained from the last_out_edge_rowid of the v1-th cell in the vertex array. The ROWID of the ending vertex: v2 and the ROWID of the previous incoming edge of v2: v2_last_in_edge_rowid. If the starting vertex and the ending vertex are on the same site, they are directly obtained from the last_in_edge_rowid of the v2-th cell in the vertex array; if they are not on the same site, continue to step (d).

[0026] (d) Then, at the Node_id site, set the last_out_edge_rowid of the v1-th cell in the vertex array to the ROWID of the currently inserted edge; if vertex v2 is also at the Node_id site, directly set the last_in_edge_rowid of the v2-th cell in the vertex array to the ROWID of the currently inserted edge, and then jump directly to step (f); if vertex v2 is not at the Node_id site, then execute step (e);

[0027] (e) If the starting vertex and the ending vertex are not on the same station, the system needs to obtain the station number based on the ROWID of the ending vertex, and then send the ROWID of the starting vertex, the ROWID of the previous outgoing edge, the ROWID of the currently inserted edge, and the ROWID of the ending vertex to the ending station to notify the ending station to insert the edge; then, the ending station receives the operation request, finds the position of the vertex in the vertex array based on the ROWID of the ending vertex, returns the last_in_edge_rowid to the station where the starting vertex is located, and fills the corresponding position in the edge array of the starting station with the edge ROWID of the previous incoming edge of v2 in step (c): v2_last_in_edge_rowid; then, the ending station continues to set last_in_edge_rowid to the ROWID of the currently inserted edge;

[0028] (f) The edge insertion operation is now complete. Return the edge ROWID and success result to the user.

[0029] Preferably, in step (9), when the operation type is to find the nth hop adjacent vertex / edge information based on ROWID, obtaining the nth hop adjacent vertex / edge information based on ROWID specifically includes the following steps:

[0030] (a) If the current hop is the 1st hop, record the current hop count as level 1. If it is not the 1st hop, the current hop count is the hop count of the previous level plus 1. Obtain the table type (Table_type), graph number (Graph_id), site number (Node_id), and vertex ID index of the vertex based on the vertex ROWID. On the site with site number Node_id, find the corresponding graph based on Graph_id, find the vertex array based on the table type, and then obtain the last_out_edge_rowid and last_in_edge_rowid of the ID item in the vertex array. Record them as cur_out_edge_rowid and cur_in_edge_rowid. If only the outgoing vertex / edge information needs to be obtained, only last_out_edge_rowid needs to be obtained. If only the incoming vertex / edge information needs to be obtained, only last_in_edge_rowid needs to be obtained. cur_out_edge_rowid is the edge ROWID of an outgoing edge that can be reached by the first hop of the vertex ROWID, and cur_in_edge_rowid is the edge ROWID of an incoming edge that can be reached by the first hop of the vertex ROWID.

[0031] (b) Find the cur_out_edge_rowid entry in the edge array, where v2 is the ROWID of the first hop's outgoing vertex and v1_last_out_edge_rowid is the ROWID of the other outgoing edge of the first hop; find the cur_in_edge_rowid entry in the edge array, where v1 is the ROWID of the first hop's incoming vertex and v2_last_in_edge_rowid is the ROWID of the other incoming edge of the first hop;

[0032] (c) Set cur_out_edge_rowid to v1_last_out_edge_rowid and cur_in_edge_rowid to v2_last_in_edge_rowid;

[0033] (d) Repeat step (b) until both cur_out_edge_rowid and cur_in_edge_rowid are invalid values, indicating that there are no more outgoing and incoming edges; all collected v2 values ​​are the ROWIDs of all outgoing vertices of the target vertex, and all collected v1 values ​​are the ROWIDs of all incoming vertices of the target vertex; all collected cur_out_edge_rowid values ​​are the ROWIDs of all outgoing edges of the target vertex, and all collected cur_in_edge_rowid values ​​are the ROWIDs of all incoming edges of the target vertex;

[0034] (e) For all the collected v1 and v2, if only the outgoing vertex / edge information is needed, then only v2 is collected; if only the incoming vertex / edge information is needed, then only v1 is collected. They are divided into different groups according to the Node_id in the ROWID. For each group, the operation command to obtain the (level+1)th hop adjacent vertex / edge information is sent to the corresponding station in sequence. At the same time, all ROWIDs in the corresponding group and the hop value (level+1) of the next hop are sent. At each station, for each received ROWID, the above steps (a)-(d) are continued to be executed to obtain all outgoing / incoming vertices / edges of the current level hop. Then, step (e) is executed to continue the search for the next hop until level equals n. Finally, all outgoing / incoming vertices / edges of the nth hop can be collected.

[0035] Accordingly, a distributed native graph implementation system includes: an operation parsing module, which parses received operation requests and determines the operation type; an operation execution module, which performs insertion, update, and deletion operations on entity-relationship data; and an adjacency management module, which manages adjacency relationships. When inserting, updating, or deleting vertices / edges, the system maintains the ROWID, vertex array index, and edge array index at each site according to the method mentioned in the first aspect. When performing association queries based on vertex / edge ROWIDs, the query is performed according to the method provided in the first aspect. This module maintains the current maximum ID for each array type at each site to pre-determine the ROWID of vertices / edges when inserting data; and an adjacency buffer module, which caches vertex array index and edge array index data at each site. When data is modified, the modified array data is written to disk when the checkpoint / buffer space is insufficient.

[0036] Preferably, the operation execution module performs the insertion, update, and deletion operations on entity-relationship data as follows:

[0037] (a) When the operation to be performed is to insert a vertex, the system needs to first calculate the Node_id of the site where the operation will be performed based on the vertex information, and then perform the vertex insertion operation at that site. In addition to the regular operation process, when operating the main index, the leaf nodes need to record the vertex ROWID information, which is automatically maintained internally by the system. At each data site, the vertex ID in the ROWID is monotonically increasing;

[0038] (b) When the operation to be performed is edge insertion, the system needs to obtain the Node_id of the station where the operation will be performed based on the ROWID of the starting vertex, and then perform the edge insertion operation at that station. The attribute information of the edge is stored at the starting station. In addition to the regular operation process, when operating the main index, the leaf nodes need to record the ROWID information of the edges. The system automatically maintains this information internally. At each data station, the edge ID in the ROWID is monotonically increasing. After the main index operation is completed, the adjacency relationship module is called to complete the insertion operation. In addition, the edge insertion operation needs to be divided into two cases: In the first case, the starting vertex and the ending vertex of the edge are at the same station, and the edge only needs to be inserted at that station; If the starting vertex and the ending vertex of the edge are not at the same station, in addition to inserting the edge and storing the attribute information of the edge at the starting station, it is also necessary to find the station where the ending vertex is located and insert the edge into the edge array of the ending station.

[0039] (c) When the operation to be performed is to update a vertex / edge, the system needs to obtain the Node_id of the site where the operation is to be performed based on the vertex / edge ROWID, and then perform the vertex / edge update operation at that site. In addition to the normal process, it is necessary to determine whether the update operation causes the physical record address corresponding to the vertex / edge to change. If it changes, the Address information of the vertex / edge array index needs to be modified.

[0040] (d) When the operation to be performed is to delete a vertex / edge, the system needs to obtain the Node_id of the site where the operation will be performed based on the vertex / edge ROWID, and then perform the operation to update the vertex / edge at that site. Outside of the normal process, the address of the vertex / edge needs to be set to a specific flag value to indicate that the corresponding record has been deleted.

[0041] Correspondingly, a device for implementing a distributed native graph includes: one or more processors;

[0042] Storage device for storing one or more programs or user data;

[0043] When the one or more programs are executed by one or more processors, the one or more processors implement a distributed native graph implementation method as described in any of the embodiments of the present invention.

[0044] Accordingly, a distributed native graph implementation storage medium stores a computer program that, when executed by a processor, implements a distributed native graph implementation method as described in any of the embodiments of the present invention.

[0045] The beneficial effects of this invention are as follows: access to vertices / edges in the entire distributed system is directly based on global array indices, which is fast and avoids computation; the system automatically maintains adjacency relationships, and when performing N-hop queries, access is directly based on global array indices instead of keys, avoiding a large number of comparison operations required in various tree structures; the performance of access based on array indices is up to 190 times better than access based on keys. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of a method for implementing a distributed native graph in Embodiment 1 of the present invention.

[0047] Figure 2 This is a schematic diagram of a method for implementing a distributed native graph in Embodiment 2 of the present invention.

[0048] Figure 3 This is a schematic diagram of the system structure of the present invention. Detailed Implementation

[0049] Example 1:

[0050] Figure 1 This is a flowchart illustrating a method for creating a distributed native graph based on global ROWID adjacency vertices / edges, as provided in Embodiment 1 of the present invention. This embodiment can be used to implement systems for distributed native graphs, such as... Figure 1 As shown, the method specifically includes the following steps:

[0051] S101, the system automatically establishes the array index structure during runtime: Optionally, the system can automatically create global vertex array indexes and edge array indexes during initialization; it can also automatically create vertex array indexes and edge array indexes for specific tables by specifying options (such as create vertex table, etc.) when creating entity tables and relational tables. These tables are called vertex tables and edge tables; it can also convert existing entity tables and relational tables into vertex tables and edge tables through SQL statements (such as alter table t1 as vertex table, etc.), thereby automatically creating vertex array indexes and edge array indexes.

[0052] Assume the system has three data stations and a graph. Each station has a vertex table (students), a class table (classes), and an edge table (learn). The table definitions and initial data are as follows:

[0053] Create a vertex table named students with the following attributes: sno varchar(20), name varchar(10), age int, and primary key(sno).

[0054] Insert three vertices into the vertex table students:

[0055] The attribute value of vertex 1 ('S001', 'Zhang San', 21) is stored in site NODE1.

[0056] The attribute values ​​of vertex 2 ('S002', 'Li Si', 22) are stored in site NODE2.

[0057] The attribute values ​​of vertex 3 ('S003', 'Zhao Wu', 23) are stored in the site NODE3.

[0058] Create a vertex table named classes with the following table attributes: cno varchar(20), name varchar(10), teacher varchar(10), and primary key(cno).

[0059] Insert four vertices into the vertex table classes:

[0060] The attribute values ​​of vertex 1 ('C001', 'Chinese', 'Teacher Wang') are stored on site NODE1.

[0061] The attribute values ​​of vertex 2 ('C002', 'Mathematics', 'Teacher Huang') are stored on the site NODE2.

[0062] The attribute values ​​of vertex 3 ('C003', 'English', 'Teacher Liu') are stored on the site NODE3.

[0063] The vertex 4 attribute values ​​('C004', 'Physics', 'Teacher Wu') are stored on the site NODE1.

[0064] Create an edge table named learn with the following table attributes: lno varchar(20), sno varchar(20), cno varchar(20), score int, level int, foreign key(sno)reference students(sno), foreign key(cno)reference classes(cno);

[0065] Insert nine edges into the edge list learn:

[0066] The attribute values ​​of edge 1 ('L001', 'S001', 'C001', 95, 1) are stored in site NODE1.

[0067] The attribute values ​​of edge 2 ('L002', 'S001', 'C002', 70, 3) are stored in site NODE1.

[0068] The attribute values ​​of edge 3 ('L003', 'S001', 'C003', 82, 2) are stored in site NODE1.

[0069] The attribute values ​​of edge 4 ('L004', 'S002', 'C001', 97, 1) are stored in site NODE2.

[0070] The attribute values ​​of edge 5 ('L005', 'S002', 'C002', 89, 2) are stored in site NODE2.

[0071] The attribute values ​​of edge 6 ('L006', 'S002', 'C003', 92, 1) are stored in site NODE2.

[0072] The attribute values ​​of edge 7 ('L007', 'S003', 'C001', 87, 2) are stored in site NODE3.

[0073] The attribute values ​​of edge 8 ('L008', 'S003', 'C002', 63, 4) are stored in the site NODE3.

[0074] The attribute values ​​of edge 9 ('L009', 'S003', 'C003', 78, 3) are stored in the site NODE3.

[0075] If a vertex is being inserted, n is calculated by the system based on the vertex's attribute information. If an edge is being inserted, n is calculated from the edge's starting vertex. Typically, a hash algorithm is used to evenly distribute all vertices across the sites.

[0076] The automatically generated vertex array indexes and their corresponding ROWIDs are as follows (using a global array, where Table_type in ROWID is 0 for a regular table, 1 for a vertex table, and 2 for an edge table; Graph_id values ​​are globally monotonically increasing, starting from 0; Node_id values ​​are globally monotonically increasing, starting from 0; and the array corresponding to each ID at each site is globally monotonically increasing, starting from 0):

[0077] NODE1 (on):

[0078] [0]|307|(2|0|0|2|)|(-1|-1|-1|-1|)| ROWID 1|0|0|0|

[0079] [1]|437|(-1|-1|-1|-1|)|(2|0|2|0|)| ROWID 1|0|0|1|

[0080] [2]|560|(-1|-1|-1|-1|)|(-1|-1|-1|-1|)| ROWID 1|0|0|2|

[0081] NODE2 (on):

[0082] [0]|350|(2|0|1|2|)|(-1|-1|-1|-1|)| ROWID 1|0|1|0|

[0083] [1]|472|(-1|-1|-1|-1|)|(2|0|2|1|)| ROWID 1|0|1|1|

[0084] On NODE3:

[0085] [0]|392|(2|0|2|2|)|(-1|-1|-1|-1|)| ROWID 1|0|2|0|

[0086] [1]|515|(-1|-1|-1|-1|)|(2|0|2|2|)| ROWID 1|0|2|1|

[0087] The automatically generated edge array indices and corresponding ROWIDs are as follows (using a global array, where the v1_last_out_edge_rowid member is -1|-1|-1|-1| indicating that vertex v1 has no previous outgoing edge, and the v2_last_in_edge_rowid member is -1|-1|-1|-1| indicating that vertex v2 has no previous incoming edge):

[0088] NODE1 (on):

[0089] [0]|584|(1|0|0|0|)|(-1|-1|-1|-1|)|(1|0|0|1|)|(-1|-1|-1|-1|)|ROWID 2|0|0|0|

[0090] [1]|630|(1|0|0|0|)|(2|0|0|0|)|(1|0|1|1|)|(-1|-1|-1|-1|)|ROWID 2|0|0|1|

[0091] [2]|682|(1|0|0|0|)|(2|0|0|1|)|(1|0|2|1|)|(-1|-1|-1|-1|)|ROWID 2|0|0|2|

[0092] NODE2 (on):

[0093] [0]|735|(1|0|1|0|)|(-1|-1|-1|-1|)|(1|0|0|1|)|(2|0|0|0|)|ROWID 2|0|1|0|

[0094] [1]|789|(1|0|1|0|)|(2|0|1|0|)|(1|0|1|1|)|(2|0|0|1|)|ROWID 2|0|1|1|

[0095] [2]|841|(1|0|1|0|)|(2|0|1|1|)|(1|0|2|1|)|(2|0|0|2|)|ROWID 2|0|1|2|

[0096] On NODE3:

[0097] [0]|895|(1|0|2|0|)|(-1|-1|-1|-1|)|(1|0|0|1|)|(2|0|1|0|)|ROWID 2|0|2|0|

[0098] [1]|944|(1|0|2|0|)|(2|0|2|0|)|(1|0|1|1|)|(2|0|1|1|)|ROWID 2|0|2|1|

[0099] [2]|991|(1|0|2|0|)|(2|0|2|1|)|(1|0|2|1|)|(2|0|1|2|)|ROWID 2|0|2|2|

[0100] S102, the distributed native graph database and other data processing systems receive the processing request insert into learnvalues('L010','S002','C004',85,2) and determine that the operation type is edge insertion.

[0101] S103: Obtain the ROWID of the starting vertex as 1|0|1|0| and the ROWID of the ending vertex as 1|0|0|2| from the information in the request. Therefore, the ROWID of the edge can be determined to be 2|0|1|3|. Perform the insertion operation in S102 on the corresponding site NODE2. Assuming the physical record's address is 1050, the vertex and edge array indices will then undergo the following operations:

[0102] At site NODE2, the last_out_edge_rowid of the starting vertex is obtained from cell 0 of the vertex array as 2|0|1|2|. At site NODE1, the last_in_edge_rowid of the ending vertex is obtained from cell 2 of the vertex array as -1|-1|-1|-1|, indicating that there is no previous incoming edge. At site NODE2, cell 3 of the edge array is filled with Address: 1050, v1: 1|0|1|0|, v1_last_out_edge_rowid: 2|0|1|2|, v2: 1|0|0|2|, v2_last_in_edge_rowid: -1|-1|-1|-1|. Then, modify cell 0 of the vertex array at site NODE2, filling in `last_out_edge_rowid: 2|0|1|3|`; modify cell 2 of the vertex array at site NODE1, filling in `last_in_edge_rowid: 2|0|1|3|`. This completes the modification, and the success result and edge ROWID are returned to the user. The worker thread / process can then write the modified array indices in memory to the array index file when needed, completing the persistence.

[0103] Example 2:

[0104] Figure 2 This is a flowchart illustrating a method for implementing a distributed native graph based on global ROWID access to adjacent vertices / edges, as provided in Embodiment 1 of the present invention. This embodiment can be used to implement systems for distributed native graphs, such as... Figure 2 As shown, the method specifically includes the following steps:

[0105] S201, like S101, automatically establishes the array index structure. In this example, it is assumed that the same initial data as S101 is used.

[0106] S202, a distributed native graph database and other data processing systems receive a processing request: `select L.score match(students S, learn L, classes C|S-(L)->C)where ROWID="1|0|0|0|"`. This request is used to find the names of students with ROWID 1|0|0|0| and their course scores. `S-(L)->C` represents the relationship between student S (starting vertex) learning L and course C (ending vertex), indicating that the operation type is an adjacency query. `ROWID="1|0|0|0|"` only represents a pseudo-SQL statement querying by ROWID and does not represent the actual query syntax.

[0107] S203, regarding the query in S202, given that the vertex ROWID is 1|0|0|0|, we can find the last_out_edge_rowid as 2|0|0|2| in cell 0 of the vertex array on site NODE1. Then, we search for the item with ID 2 in the edge array, and find that its v1_last_out_edge_rowid is 2|0|0|1|. Continuing to search for the item with ID 1 in the edge array, we find that its v1_last_out_edge_rowid is 2|0|0|0|. Continuing to search for the item with ID 0 in the edge array, we find that its v1_last_out_edge_rowid is -1|-1|-1|-1|, indicating the end of the search. This allows us to find all the courses S001 has taken, and then locate the corresponding edge array containing ROWIDs with IDs 0, 1, and 2 that meet the specified conditions. By accessing the Address of the item with IDs 0, 1, and 2 in the edge array index, we can find the corresponding grade. The query results that meet the conditions are then returned to the client.

[0108] Example 3:

[0109] Figure 3 This is a schematic diagram of a system architecture for implementing a distributed native graph based on global ROWID, as provided in Embodiment 1 of the present invention. This embodiment can be used to implement a system for distributed native graphs. The method can be executed by the distributed native graph system in this invention example, which can be implemented in software and / or hardware, such as... Figure 3 As shown, the system includes: an operation parsing module 301, an operation execution module 302, an adjacency management module 303, and an adjacency caching module 304.

[0110] Operation parsing module: This module is responsible for parsing the received operation requests and determining the operation type.

[0111] Operation Execution Module: This module is used to execute received operation requests. If it involves adjacency relationship processing, the operation needs to be transferred to the adjacency relationship management module for further execution after completion.

[0112] The adjacency management module handles adjacency-related operations, including global ROWID maintenance, vertex array index maintenance, edge array index maintenance, and distributed adjacency query.

[0113] Adjacency Cache Module: This module completes the data reading and writing operations for vertex array indices and edge array indices.

Claims

1. A method for implementing a distributed native graph, the method comprising: Comprising the following steps: Step 1, automatically establishing array index structure on each data node in system runtime; Step 2, in the process of distributed native graph system running, the working thread / process parses the received operation request and determines the operation type; Step 3, performing corresponding operation according to the operation type; when the operation type is inserting edge data, it specifically comprises the following steps: (a) the working thread / process will get the site number according to the starting vertex ROWID of the edge, and insert the attribute data of the edge into the same site as the starting vertex in the distributed environment, and then assign an edge ROWID, wherein Graph_id indicates which graph the edge belongs to, Node_id and the site number of the starting vertex are the same, and it also contains a monotonically increasing edge ID in Node_id site, which indicates the subscript of the edge array file of the graph corresponding to Graph_id in Node_id site; (b) in Node_id site, in addition to inserting the physical record corresponding to the attribute data of the edge, the primary index of the edge table is stored in the leaf node at the same time, and the physical record address corresponding to the attribute data and the edge ROWID are stored in the leaf node at the same time; (c) then in Node_id site, fill the address Address of the physical record corresponding to the edge in the data file into the first ID unit in the edge array, the ROWID of the starting vertex: v1 and the edge ROWID of the last out edge of v1: v1_last_out_edge_rowid, which is obtained from the last_out_edge_rowid of the first v1 unit in the vertex array, the ROWID of the terminal vertex: v2 and the edge ROWID of the last in edge of v2: v2_last_in_edge_rowid, if the starting vertex and the terminal vertex are in the same site, directly get v2_last_in_edge_rowid from the last_in_edge_rowid of the first v2 unit in the vertex array; if they are not in the same site, continue to perform step (d); (d) then in Node_id site, set the last_out_edge_rowid of the first v1 unit in the vertex array to the currently inserted edge ROWID; if the vertex v2 is also in Node_id site, directly set the last_in_edge_rowid of the first v2 unit in the vertex array to the currently inserted edge ROWID, and then directly jump to step (f); if the vertex v2 is not in Node_id site, perform step (e); (e) if the starting vertex and the terminal vertex are not in the same site, the system needs to get the site number according to the ROWID of the terminal vertex, and then send the starting vertex ROWID, the last out edge ROWID, the currently inserted edge ROWID and the terminal vertex ROWID to the terminal site to inform the terminal site to insert the edge; Then, the receiving station receives the operation request, finds the position of the vertex in the vertex array according to the vertex ROWID, returns the last_in_edge_rowid in the vertex to the station where the starting vertex is located, fills the edge ROWID of the last in-edge of v2 in the edge array in the starting station with v2_last_in_edge_rowid, and then sets the last_in_edge_rowid in the receiving station to the edge ROWID of the current insertion; (f) The edge insertion operation is completed, and the edge ROWID and success result are returned to the user.

2. The method of claim 1, wherein, In step 1, the system automatically establishes the array index structure on each data node at runtime, specifically: the system automatically creates a global vertex array index and an edge array index at initialization, automatically creates a vertex array index and an edge array index for a specific table by specifying options when creating an entity table and a relationship table, and automatically creates a vertex array index and an edge array index by converting an existing entity table and a relationship table into a vertex table and an edge table through an SQL statement.

3. The method of claim 1, wherein, In step 3, the corresponding operation is performed according to the operation type, specifically: (1) When the operation type is to insert vertex data, the working thread / process calculates the station number to which the vertex needs to be sent, and then assigns a ROWID, where Graph_id represents which graph the vertex belongs to, Node_id represents the station number calculated, and a monotonically increasing vertex ID is included, which represents the subscript of the vertex array file of the graph corresponding to Graph_id on the station Node_id; in addition to inserting the physical record of the attribute data of the vertex in the data file, the attribute data corresponding to the physical record address is also stored in the leaf node of the primary index of the vertex table; then, the address information Address of the physical record of the vertex corresponding to the IDth unit is filled in the vertex array, last_out_edge_rowid and last_in_edge_rowid are set to an illegal value, and the vertex ROWID and success result are returned to the user; (2) When the operation type is to update vertex data, the vertex needs to be determined according to the vertex ROWID, and then the vertex update operation is performed on the station; if the data storage location is unchanged, the vertex array index item of the vertex is not changed, and if the data storage location is changed, the address Address in the vertex array index item of the record is updated; (3) When the operation type is to delete a vertex, the vertex needs to be determined according to the vertex ROWID, and then the deletion operation is performed on the station; the Address of the vertex array index item of the vertex is set to a specific marker, indicating that the vertex has been deleted. (4) When the operation type is inserting edge data, the worker thread / process calculates the site number to which the edge needs to be sent, and then allocates a ROWID, wherein Graph_id indicates which graph the edge belongs to, Node_id indicates the site number of the site where the starting vertex of the edge is located, and a monotonically increasing edge ID is contained, which indicates the subscript of the edge array file of the graph corresponding to Graph_id on the site Node_id; in addition to inserting the physical record corresponding to the attribute data of the edge in the data file, for the primary index of the edge table, the physical record address corresponding to the attribute data and the edge ROWID are simultaneously stored on the leaf node; then the address information Address of the physical record corresponding to the edge in the data file, the ROWID of the starting vertex, the last outgoing edge ROWID of the starting vertex, the ROWID of the ending vertex, and the last incoming edge ROWID of the ending vertex are filled in the IDth unit in the edge array; then the last outgoing edge ROWID of the starting vertex is set as the ROWID of the inserted edge, if the ending vertex and the starting vertex are on the same site, the last incoming edge ROWID of the ending vertex is set as the ROWID of the inserted edge, and if the ending vertex and the starting vertex are not on the same site, the information of the edge needs to be inserted on the site where the ending vertex is located; after the above operations are completed, the edge ROWID and the success result are returned to the user; (5) When the operation type is updating edge data, the site where the edge is located needs to be determined according to the edge ROWID, and then the edge updating operation is performed on the site; if the data storage position does not change, the edge array index item corresponding to the edge does not change, and if the data storage position changes, the address Address in the edge array index item corresponding to the record is updated; (6) When the operation type is deleting an edge, the site where the edge is located needs to be determined according to the edge ROWID, and then the updating operation is performed on the site; the Address of the edge array index item corresponding to the edge is set as a specific marker, indicating that the edge has been deleted; (7) When the operation type is looking up vertex / edge information based on the ROWID of the vertex / edge, the site where the vertex / edge is located and the vertex / edge ID are found according to the ROWID, the Address of the record is found based on the array ID item on the site, and then the detailed information of the vertex / edge is directly queried at the address; (8) When the operation type is looking up the nth-hop adjacent vertex / edge information based on the ROWID, the nth-hop adjacent vertex / edge information is obtained based on the ROWID; (9) When the operation type is based on edge ROWID to find the nth hop adjacent vertex / edge information, the Table_type, Graph_id, Node_id and edge ID index of the edge to which the vertex ROWID belongs are obtained; the corresponding graph is found on the site with Node_id according to Graph_id, the edge array is found according to the Table_type, then the first ID item of the edge array obtains the ROWID of the starting vertex and the ending vertex, and then the nth hop adjacent vertex / edge information is obtained based on the starting vertex and the ending vertex respectively.

4. The method of claim 3, wherein, In step (9), when the operation type is based on ROWID to find the nth hop adjacent vertex / edge information, the nth hop adjacent vertex / edge information based on the ROWID includes the following steps: (a) If it is the first hop, record the current hop number level as 1, if it is not the first hop, the current hop number is the last hop number plus 1; the Table_type, Graph_id, Node_id and vertex ID index of the vertex to which the vertex ROWID belongs are obtained; the corresponding graph is found on the site with Node_id according to Graph_id, the vertex array is found according to the Table_type, then the first ID item of the vertex array obtains last_out_edge_rowid and last_in_edge_rowid; record as cur_out_edge_rowid and cur_in_edge_rowid, if only the out vertex / edge information is obtained, only last_out_edge_rowid is needed, if only the in vertex / edge information is obtained, only last_in_edge_rowid is needed; cur_out_edge_rowid is the edge ROWID of the first hop that can reach an out edge of the vertex ROWID, cur_in_edge_rowid is the edge ROWID of the first hop that can reach an in edge of the vertex ROWID; (b) find the first cur_out_edge_rowid item in the edge array, v2 in the item is the first hop out vertex ROWID, v1_last_out_edge_rowid is the edge ROWID of the first hop out edge; find the first cur_in_edge_rowid item in the edge array, v1 in the item is the first hop in vertex ROWID, v2_last_in_edge_rowid is the edge ROWID of the first hop in edge; (c) set cur_out_edge_rowid as v1_last_out_edge_rowid, cur_in_edge_rowid as v2_last_in_edge_rowid; (d) Repeat step (b) until cur_out_edge_rowid and cur_in_edge_rowid are both illegal values, indicating that there are no more out-edges and in-edges; all v2s collected are all out-vertex ROWIDs of the target vertex, and all vls are all in-vertex ROWIDs of the target vertex; all cur_out_edge_rowids are all out-edge ROWIDs of the target vertex, and all cur_in_edge_rowids are all in-edge ROWIDs of the target vertex; (e) For all the collected vls and v2s, if only out-vertex / edge information is obtained, only v2s are obtained, and if only in-vertex / edge information is obtained, only vls are obtained, and the vls and v2s are divided into different groups according to the node number Node_id in the ROWID; for each group, an operation command for obtaining (level+1)-hop adjacent vertex / edge information is sent to the corresponding site, and all ROWIDs in the corresponding group and the hop number value (level+1) of the next hop are also sent; on each site, for each received ROWID, the above steps (a)-(d) are continued to be executed to obtain all out / in-vertices / edges of the current level, and then step (e) is executed to continue the search of the next hop until level is equal to n, and finally all out / in-vertices / edges of the nth hop are collected.

5. A system for implementing a method based on the distributed native graph as claimed in claim 1, characterized by, Comprise: an operation analysis module for analyzing a received operation request and determining an operation type; an operation execution module for completing insertion, update, and deletion operations of entity-relation data; an adjacent relation management module for completing adjacent relation management, maintaining ROWIDs in the system, vertex array indexes and edge array indexes on each site, and performing association query based on vertex / edge ROWIDs according to the method provided above; the module maintains the current maximum ID of each array on each site to determine the vertex / edge ROWID in advance when data is inserted; an adjacent relation buffer module for caching vertex array indexes and edge array index data on each site, and writing modified array data to the disk when there is data modification and the checkpoint / buffer free space is insufficient.

6. The distributed native graph implementation system of claim 5, wherein, The operation execution module completes the insertion, update, and deletion operations of entity-relation data, specifically as follows: (a) When the operation to be executed is to insert a vertex, the system needs to calculate the site number Node_id of the operation execution according to the vertex information, and then execute the operation of inserting the vertex on the site; In addition to the conventional operation process, the leaf node needs to record vertex ROWID information when the main index is operated, which is automatically maintained internally, and the vertex ID in the ROWID is monotonically increasing on each data site; (b) When the operation to be performed is edge insertion, the system needs to obtain the site number Node_id of the operation execution according to the starting vertex ROWID, and then perform the operation of inserting the edge on the site, and the attribute information of the edge is stored on the starting site; when the main index is operated, the leaf node needs to record the ROWID information of the edge, which is automatically maintained by the system internally, and the edge ID in the ROWID is monotonically increasing on each data site, and after the main index operation is completed, the adjacent relationship module is called to complete the insertion operation; in addition, the insertion edge operation needs to be divided into two cases: the first case is that the starting vertex and the ending vertex of the edge are on the same site, and only the edge needs to be inserted on the site; if the starting vertex and the ending vertex of the edge are not on the same site, in addition to inserting the edge and storing the attribute information of the edge on the starting site, the site where the ending vertex is located also needs to be found, and the edge is inserted into the edge array of the ending site; (c) When the operation to be performed is vertex / edge update, the system needs to obtain the site number Node_id of the operation execution according to the vertex / edge ROWID, and then perform the operation of updating the vertex / edge on the site; In addition to the conventional process, it is necessary to judge whether the update operation causes the change of the physical record address corresponding to the vertex / edge, and if the change occurs, the Address information of the vertex / edge array index needs to be modified; (d) When the operation to be performed is vertex / edge deletion, the system needs to obtain the site number Node_id of the operation execution according to the vertex / edge ROWID, and then perform the operation of updating the vertex / edge on the site; In addition to the conventional process, the Address of the vertex / edge needs to be set to a specific marker value, indicating that the corresponding record has been deleted.

7. An apparatus for implementing a distributed native graph, the apparatus comprising: Comprise: One or more processors; Storage device for storing one or more programs, user data; When the one or more programs are executed by one or more processors, the one or more processors implement the implementation method of the distributed native graph as claimed in any one of claims 1-5.

8. An implementation storage medium of a distributed native graph, which stores a computer program, and the program is executed by a processor to implement the implementation method of the distributed native graph as claimed in any one of claims 1-5.

Images