An efficient query method for large-scale graph data

Abstract

The present invention discloses a method for efficiently querying large-scale graph data, comprising: partitioning graph data to obtain initial partitions of the graph data; wherein the initial partitions include the binary signature of each vertex vector and the corresponding hash bucket; adjusting the initial partitions of the graph data to obtain new partitions; converting the graph data in RDF format into a property graph format and storing it; obtaining a join plan based on a SPARQL query statement; and obtaining query results in a distributed database based on the join plan and the SPARQL query mapping algorithm. The present invention saves graph data query time.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method for efficiently querying large-scale graph data. Background Art

[0002] With the ever-increasing scale of RDF data, designing high-performance, efficient query methods for large-scale graph data to support complex business needs is an urgent problem to be solved. RDF data is logically based on a graph structure of nodes and relationships. The semantic structure and clustering information in the graph are very critical. When storing closely related data in a distributed manner, if closely related data is stored in the same partition, the number of cross-partition queries can be reduced, thereby improving query efficiency. A reasonable storage model can not only provide fast query services for upper-layer applications, but also facilitate data and system maintenance, such as load balancing and dynamic updates. In addition, query and storage are closely related and influence each other. When considering how to distribute storage, the goal should be to maximize query efficiency. SPARQL is an effective tool for querying RDF data. Therefore, to solve the problem of efficient RDF query, it is necessary to combine the storage model and design an efficient SPARQL query engine to handle various complex query requests.

[0003] Existing technical solutions:

[0004] Anhui Huayun'an Technology Co., Ltd.'s patent application "Data query method and device based on knowledge graph cluster" (patent application number: CN202211617863.4) is characterized by including: in response to receiving a current data query request, adding the current data query request to the query request message queue; according to the target data query request, determining the corresponding index in the graph database; according to the determined index, determining the corresponding data ontology in the relational database, and adding the data ontology to the message output queue. The disadvantages are: the query method and device require a large amount of computing resources and storage resources, and therefore require a large amount of capital and manpower costs; the cluster needs to be maintained and updated regularly, otherwise there will be problems such as inaccurate data or incomplete query results, which requires professional technicians to maintain.

[0005] Douyin Vision Co., Ltd.'s patent application "A graph data query method and device" (patent application number: CN202210991887.X) is characterized by including: in response to a data query request, determining the query statement indicated by the data query request; parsing each query field in the query statement in sequence according to a predefined order, and if it is a target field of the set entry type, calling the target query instruction of the target field parser; performing data query from the graph data according to the query instruction, and generating query results corresponding to the query statement. The disadvantage is that the graph database needs to store a large amount of node and relationship information, which takes up a lot of storage space. For large-scale data queries, larger storage space may be required, which increases cost and difficulty;

[0006] Hangzhou Yueshu Technology Co., Ltd.'s patent application, "A method, system, device, and medium for accelerating graph database data queries" (patent application number: CN202210957310.7), is characterized in that the method includes: after the graph data is written to the storage node in the graph database, synchronizing the graph data to the topology node; storing the graph topology relationship of the graph data through the topology node, and completing the query request based on the graph topology relationship stored in the topology node. The disadvantage is that it requires reliance on specialized hardware devices, such as graph processors or accelerator cards. These devices are expensive and require professional technicians for maintenance and management; limited by the data size, when the data set is very large, the query speed may still be very slow. Summary of the Invention

[0007] In view of this, the present invention provides an efficient query method for large-scale graph data to solve the above technical problems.

[0008] The present invention discloses a method for efficiently querying large-scale graph data, which is applied to querying graph data in a distributed database, and includes:

[0009] Step 1: Divide the graph data to obtain initial partitions of the graph data; wherein the initial partitions include the binary signature of each vertex vector and the corresponding hash bucket;

[0010] Step 2: Adjust the initial partition of the graph data to obtain a new partition;

[0011] Step 3: Convert the graph data in RDF format into property graph format and store it;

[0012] Step 4: Get the join plan based on the SPARQL query statement;

[0013] Step 5: Get the query results in the distributed database according to the connection plan and SPARQL query mapping algorithm.

[0014] Furthermore, the step 1 specifically includes:

[0015] Word2Vec is selected as the vectorization tool, each attribute of each vertex is represented as a word vector, and the vertex vector is calculated using the weighted average model;

[0016] Use all vertex vectors as input to the locality-sensitive hashing algorithm to obtain the binary signature of each vertex vector and the corresponding hash bucket;

[0017] The step 2 includes:

[0018] Assume that the initial partition of the existing graph data includes k partition results P = {p1, p2, ..., pk}, and the maximum load of each partition is MaxSize; the existing graph data is G = (V, E), V = {v1, ..., vn} is the vertex set, E = {e1, ..., em} is the edge set, |V| = n, |E| = m;

[0019] Calculate v separately i Score in pi adjacent Score;

[0020] will p i All v i According to Score adjacent Sort by size;

[0021] when From Score adjacent The smallest score v i First, calculate v i Score in other partitions move (v i ,p) score, sort all scores and take out v i The partition with the largest score p j ;

[0022] when Take out v i The partition with the second largest score p j Otherwise, v i Assigned to p j And update p i and cache tables;

[0023] Finally, the adjusted partitions are output.

[0024] Furthermore, the step 3 includes:

[0025] Convert the RDF format to the property graph format, and then use the vertex as the storage granularity, as each row of HBase, to store the properties and relationships related to the vertex.

[0026] Furthermore, the step 3 specifically includes:

[0027] For each triple data<s,p,o> , determine the type of vertex based on Resource and use URI as an attribute of the vertex;

[0028] If o is a text, p is used as the vertex attribute name and o is used as the vertex attribute value;

[0029] If o is a resource, map p to an edge, assign a unique number to p, and use the value of p as the attribute value; use the URIs of s and o as attributes of p, representing the two vertices connected by p.

[0030] Furthermore, the step 4 includes:

[0031] Step 41: Parse the SPARQL query statement to obtain the triple pattern set Plan = {T1, T2, ... T p}; The SPARQL query statement contains n triple patterns T = {t1, t2, ... t n};

[0032] Step 42: Start traversing from the set containing the most triple patterns, delete the triple patterns that have been traversed in sequence, and obtain a connection plan, which contains m sets Plan = {T1, T2, ... T m}, each set represents a subset of triple patterns.

[0033] Furthermore, the step 42 specifically includes:

[0034] In Plan={T1,T2,…T p} traverse T in sequence i :

[0035] If T i The triple pattern t containing the known subject j , then t j Put it in the first place and put T i Marked as Type-S;

[0036] If T i The triple pattern t contains the known predicate and object j , then t j Put it in the first place and put T i Marked as Type-PO;

[0037] If T i It does not contain the subject and the known triple pattern of predicate and object. j , T jThe clauses in the first order are sorted by the number of variables from small to large; if t j and t k The number of variables is the same, then the known predicate is ranked first; if t j and t k If the predicates are known, scan the secondary index table and sort the smaller result sets first;

[0038] After the traversal is completed, the connection plan is obtained.

[0039] Furthermore, Spark's parallel computing capabilities are used to generate the connection plan:

[0040] After the task of generating the join plan is submitted to Spark, assuming a query statement consists of two triple patterns {t1, t2}, after analysis, the two clauses are executed in parallel:

[0041] First, the data corresponding to the two triple patterns is queried in HBase, and the variable values ​​are extracted to form a new result set. Then, the new result set is joined with the extracted variable values ​​to form the final result set.

[0042] Then extract the required key or value from the final result set again;

[0043] Finally convert the required keys or values ​​into an in-memory array.

[0044] Furthermore, the step 5 includes:

[0045] Initialize the current RDD to the result of the first loop; RDD is a key-value pair, where the key represents the variable to be connected and the value represents the non-connected variable during the iteration;

[0046] In the connection plan, there are m sets Plan={T1,T2,…T m}Start traversal in:

[0047] If T i With the identifier Type-SorType-PO, the SPARQL query mapping algorithm is used to execute T i The first statement in the result Rows; according to Rows, T i Bind the remaining clause variables in T; merge all results to form newRDD and save it in memory; otherwise, i Middle pair t j Traverse: Get the result set through the SPARQL query mapping algorithm; adjust the key-value according to the query conditions and convert the result into a newRDD; concatenate the newRDD with the RDD of the previous step to form a new newRDD and save it in memory;

[0048] After all traversals are completed, the required values ​​in the result set are retrieved to obtain the query results.

[0049] Furthermore, the SPARQL query mapping algorithm is:

[0050] When inputting triple data<s,p,o> after,

[0051] If s is already bound, query in the HBase main table;

[0052] When s is unbound, if p is bound and s is a variable, look up in the index table;

[0053] If both s and p are not bound, scan the main data table;

[0054] Finally, the scan result set is obtained.

[0055] Furthermore, when s is already bound, in the query in the HBase main table:

[0056] When p is bound, if p is an attribute, then the attribute value is searched in row; otherwise, the other end vertex of the relationship is searched in row;

[0057] When p is unbound and o is bound, if o is an attribute value, filter out the attribute name in the attribute value column family; otherwise, filter out the relationship in the relationship column family;

[0058] When both p and o are not bound, take out all p and o related to s in row and filter them;

[0059] When s is unbound, if p is bound and s is a variable, then look up in the index table:

[0060] If o is already bound, locate the row corresponding to the predicate and then filter by the object.

[0061] Due to the adoption of the above technical solution, the present invention has the following advantages:

[0062] 1. This invention proposes a massive knowledge distributed storage model based on graph partitioning, and on the basis of distributed storage, proposes a SPARQL query optimization method to improve the access efficiency of massive semantic data.

[0063] 2. To address the problem that large-scale RDF data partition storage does not fully consider the semantic association of data, resulting in high communication overhead during query, an RDF data partitioning algorithm that combines semantic information and topological structure is proposed to improve the partitioning speed and optimize the partitioning results.

[0064] 3. To further improve execution efficiency, Spark is used to build a SPARQL query engine, and a SPARQL query mapping algorithm is proposed to submit the triple patterns in the SPARQL query statement as HBase row key queries to efficiently execute query tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments described in the embodiments of the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0066] Figure 1 Schematic diagram of the RowKey structure of an embodiment of the present invention;

[0067] Figure 2 This is a schematic diagram of the Spark iterative process of an embodiment of the present invention;

[0068] FIG3( a ) is a schematic diagram showing a comparison of query times of H2RDF, SRDF, Koral, and the method of the present invention on the LUBM10k test dataset according to an embodiment of the present invention;

[0069] FIG3( b ) is a schematic diagram showing a comparison of query times of H2RDF, SRDF, Koral, and the method of the present invention on the LUBM20k test dataset according to an embodiment of the present invention;

[0070] Figure 4 The figure is a flow chart of an efficient query method for large-scale graph data according to an embodiment of the present invention. DETAILED DESCRIPTION

[0071] The present invention will be further described with reference to the accompanying drawings and embodiments. Obviously, the embodiments described are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those skilled in the art should fall within the scope of protection of the embodiments of the present invention.

[0072] See also Figure 4 This invention provides an embodiment of an efficient query method for large-scale graph data. In this embodiment, a distributed storage model for semantic data based on graph partitioning is proposed, using an existing big data processing framework as the underlying storage container. In conjunction with the designed storage model, a join plan generation algorithm based on secondary indexes is proposed to maximize query optimization and reduce the computational burden of the query engine. To further enhance the performance of the distributed RDF management system, Spark's parallel computing capabilities are leveraged to improve execution efficiency and reduce query response time.

[0073] This embodiment includes the following steps:

[0074] This paper proposes a graph partitioning algorithm based on semantic information and topological structure. To balance efficiency and quality, a hashing algorithm is used to reduce complexity and further optimize the partitioning results. Therefore, the algorithm is divided into two phases: Phase 1, Step 1: Partitioning the graph data to obtain initial partitions; these initial partitions include the binary signature of each vertex vector and the corresponding hash bucket; Phase 2, Step 2: Adjusting the initial partitions to obtain new partitions.

[0075] In the first phase, Word2Vec is used as a vectorization tool to represent each vertex attribute as a word vector. A weighted average model is then used to calculate the vertex vector. Using all vertex vectors as input to the locality-sensitive hashing algorithm, a binary signature for each vertex vector and its corresponding hash bucket are obtained. The specific steps are shown in Algorithm 1. The algorithm ultimately outputs the hash-mapped signature vector for each vertex vector and its corresponding hash bucket number, where the hash bucket number represents the initial partitioning result.

[0076]

[0077]

[0078] The initial partitioning results have fully considered the semantic similarity of vertices. To ensure the efficiency of the algorithm and the effectiveness of the initial partitioning results, the second stage adjusts the local data based on the topological structure of the graph.

[0079] Definition 1: Consider a graph G = (V, E), where the vertex set V = {v1,...,vn}, the edge set E = {e1,...,em}, |V| = n, |E| = m. Also, define a set of partitions P = {p1,p2,...,pk}, where each partition pi contains a subset of V and E. This paper adopts the concept of edge cutting. Vertices in each partition have no duplicates, but some edges may have multiple copies in different partitions after cutting. Therefore, this paper defines a scoring function.

[0080]

[0081] Definition 2: The present invention adopts edge cutting. The more copies of an edge, the lower the efficiency of inter-partition communication. Therefore, the number of copies of an edge needs to be minimized. Let D p (v i ) is the vertex v i The in-degree and out-degree on partition p represent the number of edges connected to the vertex, and the present invention defines a scoring function.

[0082]

[0083] Definition 3: When adjusting vertices in each partition, it is necessary to consider the number of adjacent vertices in other partitions and the number of cut edges caused. Therefore, the present invention defines a scoring function formula to calculate the score of the vertex in each partition during adjustment.

[0084] Score move (v i ,p)=Score adjacent (v i ,p)+Score copy (v i ,p) (3

[0085] Based on the above analysis, the present invention proposes a partition adjustment algorithm based on the number of adjacent vertices, as shown in Algorithm 2.

[0086]

[0087]

[0088] Step 3: Convert the graph data in RDF format into property graph format and store it.

[0089] Based on HBase, this paper proposes a bidirectional adjacency list storage model based on a property graph model. This model first converts the RDF format into a property graph format. Then, using vertices as the storage granularity, each HBase row stores the attributes and relationships associated with each vertex. The conversion from the RDF model to the property graph model is shown in Algorithm 3.

[0090]

[0091]

[0092] The RowKey structure of the RDF data designed by the present invention in HBase is as follows Figure 1 As shown in the figure, corresponding to the graph partitioning results of the first phase of the present invention, the vertex ID is stored as the RowKey in HBase. The vertex ID contains a total of 64 bits, consisting of the partition ID, the serial number, and a reserved bit. The partition ID occupies 6 bits, the middle few bits are the serial number, and the maximum value is 2 to the power of 56, which can fully accommodate massive vertices. The last two bits are reserved for expansion.

[0093] The RDF data storage structure is shown in Table 1. Based on the RowKey designed above, the storage granularity of each row in HBase is defined as a vertex. The RowKey is the vertex ID. Each attribute of the vertex and its attribute value corresponds to a column and column value. All attributes form a column family.

[0094] Table 1 RDF data storage structure

[0095]

[0096] Table 2 shows the storage structure of properties. In the Properties column family, Column represents the property name, and Value represents the property value. Property values ​​are stored in a serialized manner. Table 3 shows the storage structure of edges. In the Edges column family, Column consists of four parts: LabelID represents the edge type, Direction indicates the direction of the edge, 0 for outgoing edges and 1 for incoming edges, EdgeID is the unique ID of each edge, and AdjacentID is the ID of the vertex to which the other end of the edge is connected. Value consists of two parts: KeysID, which is equivalent to an array that stores all the property names of the edge, and PropertiesValue, which is the serialized result of all the property values.

[0097] Table 2 Attribute and attribute value storage structure

[0098]

[0099] Table 3 Edge Storage Structure

[0100]

[0101] Step 4: Get the join plan based on the SPARQL query statement.

[0102] This paper proposes a secondary index model. The vertex attribute index structure is shown in Table 4, where the RowKey of the secondary index table consists of the attribute name. The attribute index table has only one column family. When the predicate is a vertex attribute, the first bit of the Column is 0. The rest of the Column consists of the VertexID to distinguish between unique and non-unique indexes. If the index is non-unique, the VertexID can be used to distinguish different vertex IDs corresponding to multiple identical index values. The Value consists of the vertex ID and the vertex attribute value.

[0103] Table 4 HBase attribute predicate secondary index structure

[0104]

[0105]

[0106] The index structure for edge predicates is shown in Table 5. The row key is the edge name. The first digit of the column is 1, identifying it as an edge attribute, and the remaining digits are the edge ID. Similarly, the column name is constructed using EdgeID. If the index is non-unique, multiple edges with the same attributes can be distinguished.

[0107] Table 5 HBase edge predicate secondary index structure

[0108]

[0109] The present invention proposes a SPARQL query join plan generation algorithm, as shown in Algorithm 4, which is divided into two stages. The first stage partitions all triple patterns, and the second stage sorts them within the partitioned subsets.

[0110]

[0111]

[0112] Step 5: Get the query results in the distributed database according to the connection plan and SPARQL query mapping algorithm.

[0113] Based on the designed storage model and the characteristics of HBase, this paper proposes an algorithm for converting SPARQL queries into HBase queries, as shown in Algorithm 5.

[0114]

[0115]

[0116] The process of executing iterative tasks in parallel in triple mode is as follows Figure 2 As shown. First, after the task is submitted to Spark, the execution process of the entire program is abstracted into a directed acyclic graph through dependency analysis. Suppose a query statement consists of two triple patterns {t1, t2}. After analysis, the two clauses can be executed in parallel. First, the data corresponding to the two triple patterns are queried in HBase and initialized as RDD-A and RDD-B respectively. At this time, the format of the result set is the row format in HBase. The values ​​of the variables need to be extracted from the query results to form new result sets RDD-C and RDD-D. Then, RDD-C and RDD-D are connected to form RDD-E. Then, the required keys or values ​​are extracted again through the map function. Finally, the action operation is called to convert it into a memory array to complete the entire process.

[0117] This paper proposes a parallel execution algorithm for SPARQL queries, as shown in Algorithm 6. First, the RDD must be initialized. The algorithm uses the result of the first loop to initialize the RDD. In subsequent loops, since the execution order of the triple pattern has been determined, for the triple pattern clauses in each subset, it is only necessary to convert the result set into an RDD. The format of the RDD in the entire iteration process is a key-value pair, where the key represents the variable to be connected and the value represents the non-connected variable in the iteration process. The intermediate result key is directly used as the input to connect to the next iteration. If a new connection variable appears in the next iteration, the key in the key-value is replaced with the new iteration variable. In the last iteration, the final query result will be extracted from a part of the RDD value.

[0118]

[0119]

[0120] The rationality and effectiveness of the distributed storage system in this invention are mainly verified by the response time of SPARQL queries. Since this invention uses HBase to store data and Spark to execute query statements, the mainstream methods H2RDF and SparkRDF with similar ideas are selected.

[0121] Table 6 Comparison of query time among H2RDF, SRDF, Koral, and the method of the present invention

[0122]

[0123]

[0124] Combining Figures 3(a) and 3(b), the query results show that H2RDF has the shortest query time, followed by the method proposed in this invention. Koral and SRDF have relatively long response times. This is mainly because low-selectivity queries do not require high connection performance. Both H2RDF and the method proposed in this invention use HBase as the underlying storage model, which can fully utilize HBase's indexing performance. SRDF stores all index files on HDFS, which is inferior to HBase in terms of low-latency data access. Koral creates a query execution tree and a coordinator for each query statement, which is more expensive for simple queries. H2RDF performs better than the method proposed in this invention, mainly because H2RDF builds three index tables (SPO, POS, OSP) based on the triple pattern, which is very suitable for queries with small result sets and high selectivity. Although the method proposed in this invention establishes a predicate-based index, the index cannot directly match the data rows. It is necessary to first match the row key in the index row and then return to the main data table for matching. Compared with H2RDF, it consumes more time. However, compared with the other two methods, the row key-based query method of the present invention is more efficient for this type of highly selective query.

[0125] In summary, the method proposed in the present invention has a good effect on low-selectivity queries with high intermediate join costs and star queries. Compared with mainstream distributed systems based on HBase, Spark, and graph partitioning, it performs better in terms of query efficiency. In actual query scenarios, the proportion of queries with known subjects and known predicates in all queries is as high as 90%. The present invention also proposes a corresponding join plan optimization algorithm, which performs better than existing mainstream methods in terms of the efficiency of queries with high join costs and star queries, and has a positive effect on the distributed management of massive knowledge.

[0126] The present invention may also use other graph partitioning algorithms, such as spectral clustering algorithm, overlapping community partitioning algorithm and genetic algorithm, etc. Other index structures may also be used to optimize the query of graph data, such as R-tree, kd-tree, Hash index, etc.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. A large-scale efficient query method for graph data, applied to querying graph data in a distributed database, characterized by: include: Step 1: Divide the graph data to obtain initial partitions of the graph data; wherein the initial partitions include the binary signature of each vertex vector and the corresponding hash bucket; Step 2: Adjust the initial partition of the graph data to obtain a new partition; Step 3: Convert the graph data in RDF format into property graph format and store it; Step 4: Get the join plan based on the SPARQL query statement; Step 5: Get the query results in the distributed database according to the join plan and SPARQL query mapping algorithm; The step 1 specifically includes: Word2Vec is selected as the vectorization tool, each attribute of each vertex is represented as a word vector, and the vertex vector is calculated using the weighted average model; Use all vertex vectors as input to the locality-sensitive hashing algorithm to obtain the binary signature of each vertex vector and the corresponding hash bucket; The step 2 includes: Assume that the initial partition of the existing graph data includes k partition results P = {p1, p2, ..., pk}, and the maximum load of each partition is MaxSize; the existing graph data is G = (V, E), V = {v1, ..., vn} is the vertex set, E = {e1, ..., em} is the edge set, |V| = n, |E| = m; Calculate v separately i In p i Score adjacent Score; will p i All v i According to Score adjacent Sort by size; when From Score adjacent The smallest score v i First, calculate v i Score in other partitions move (v i ,p) score, sort all scores and take out v i The partition with the largest score p j ; when Take out v i The partition with the second largest score p j Otherwise, v i Assigned to p j And update p i and cache tables; Finally output the adjusted partition; The step 3 comprises: Convert the RDF format to the property graph format, and then use the vertex as the storage granularity, as each row in HBase, to store the attributes and relationships related to the vertex; The step 3 specifically includes: For each triple data<s,p,o> , determine the type of vertex based on Resource and use URI as an attribute of the vertex; If o is a text, p is used as the vertex attribute name and o is used as the vertex attribute value; If o is a resource, map p to an edge, uniquely number p, and use the value of p as the attribute value; the URIs of s and o are also used as attributes of p, representing the two vertices connected by p; The step 4 comprises: Step 41: Parse the SPARQL query statement to obtain the triple pattern set Plan = {T1, T2, ... T p }; The SPARQL query statement contains n triple patterns T = {t1, t2, ... t n }; Step 42: Start traversing from the set containing the most triple patterns, delete the triple patterns that have been traversed in sequence, and obtain a connection plan, which contains m sets Plan = {T1, T2, ... T m }, each set represents a subset of triple patterns; The step 42 specifically includes: In Plan={T1,T2,…T p } traverse T in turn i : If T i The triple pattern t containing the known subject j , then t j Put it in the first place and put T i Marked as Type-S; If T i The triple pattern t contains the known predicate and object j , then t j Put it in the first place and put T i Marked as Type-PO; If T i It does not contain the subject and the known triple pattern of predicate and object. j , T j The clauses in the first order are sorted by the number of variables from small to large; if t j and t k The number of variables is the same, then the known predicate is ranked first; if t j and t k If the predicates are known, scan the secondary index table and sort the smaller result sets first; After the traversal is completed, the connection plan is obtained; The step 5 comprises: Initialize the current RDD to the result of the first loop; RDD is a key-value pair, where the key represents the variable to be connected and the value represents the non-connected variable during the iteration; In the connection plan, there are m sets Plan={T1,T2,…T m }Start traversal in: If T i Marked Type-S or Type-PO , then execute T through the SPARQL query mapping algorithm i The first statement in the result Rows; according to Rows, T i Bind the remaining clause variables in T; merge all results to form newRDD and save it in memory; otherwise, i Middle pair t j Traverse: Get the result set through the SPARQL query mapping algorithm; adjust the key-value according to the query conditions and convert the result into a newRDD; concatenate the newRDD with the RDD of the previous step to form a new newRDD and save it in memory; After all traversals are completed, the required values ​​in the result set are retrieved to obtain the query results.

2. The method according to claim 1, characterized in that Generate a connection plan by using Spark's parallel computing capabilities: After the task of generating the join plan is submitted to Spark, assuming a query statement consists of two triple patterns {t1, t2}, after analysis, the two clauses are executed in parallel: First, the data corresponding to the two triple patterns is queried in HBase, and the variable values ​​are extracted to form a new result set. Then, the new result set is joined with the extracted variable values ​​to form the final result set. Then extract the required key or value from the final result set again; Finally convert the required keys or values ​​into an in-memory array.

3. The method according to claim 1, characterized in that The SPARQL query mapping algorithm is: When inputting triplet data<s,p,o> after, If s is already bound, query in the HBase main table; When s is unbound, if p is bound and s is a variable, look up in the index table; If both s and p are not bound, scan the main data table; Finally, the scan result set is obtained.

4. The method according to claim 3, characterized in that When s is already bound, query the HBase main table: When p is bound, if p is an attribute, then the attribute value is searched in row; otherwise, the other end vertex of the relationship is searched in row; When p is unbound and o is bound, if o is an attribute value, filter out the attribute name in the attribute value column family; otherwise, filter out the relationship in the relationship column family; When both p and o are not bound, take out all p and o related to s in row and filter them; When s is unbound, if p is bound and s is a variable, then look up in the index table: If o is already bound, locate the row corresponding to the predicate and then filter by the object.

Images