An RDF knowledge graph data partitioning and distributed query method

Abstract

The application provides an RDF knowledge graph data division and distributed query method, proposes minimum schema cut as a novel RDF graph workload-aware division technology, and then decomposes a query into a group of independent executable sub-queries based on an internal schema, so that the number of cross-partition schemas is minimum and the number of query connections between partitions is minimized under a given workload, which will lead to significant improvement of query execution performance.

Description

Technical Field

[0001] This invention mainly relates to the field of knowledge graphs, specifically to an RDF knowledge graph data partitioning and distributed query method. Background Technology

[0002] The Resource Description Framework (RDF) is a general model that uses sets of triples to describe the relationships between different entities, denoted as (subject, predicate, object). An RDF dataset can be represented by a directed edge-labeled graph, where the subject and object are vertices, and the triples are edges labeled with attribute names. To retrieve and manipulate RDF graphs, an RDF query language, SPARQL, is proposed. A fundamental building block of SPARQL is the Basic Graph Schema (BGP), which is a triple schema with variables. A BGP query consists of a finite number of BGPs and can also be represented as a query graph with variables. Therefore, answering a BGP query is equivalent to finding a subgraph match (using homomorphism) on the RDF graph.

[0003] As RDF datasets grow in size, typical performance issues arise in managing and querying RDF data on a single machine, sparking interest in distributed solutions. In this invention, we focus on optimizing specialized distributed RDF systems built specifically for SPARQL query evaluation by integrating multiple centralized RDF systems across different sites using a custom physical layout. This approach is widely used due to its high efficiency. In this environment, an RDF graph G is divided into a set of subgraphs, called partitions, and then distributed across a set of compute nodes, called sites. If a query matches data across multiple partitions, joins between partitions are involved to compute the results.

[0004] There have been many studies discussing how to partition RDF graphs, and these studies can be divided into two categories: workload-agnostic and workload-aware.

[0005] The workload-agnostic approach ignores the query workload and only considers the RDF data graph. Ignoring the query workload means missing opportunities to optimize queries for frequent query patterns within the workload. In fact, analysis of well-known SPARQL actual workload LSQ shows that a small number of frequent query patterns can cover most SPARQL queries within the workload.

[0006] Workload-aware methods typically mine a set of frequent query patterns and partition the RDF graph, thereby replicating graph portions that match the frequent patterns across multiple partitions. A key assumption of these methods is that they can avoid interval joins when a query is isomorphic to one of the common query patterns. These replication methods add space overhead and data consistency maintenance overhead because these copies need to be maintained as the RDF graph is updated.

[0007] The method we propose in this invention belongs to the workload-aware category, but it does not copy data. Instead, we follow the vertex-disjoint method, assigning each vertex to a single partition, a common approach in workload-agnostic partitioning methods. In vertex-disjoint partitioning techniques, certain edges are "cut" between partitions, resulting in inter-partition joins. Methods such as 1-hop replication can reduce these joins by replicating the endpoints of the cut edges in the two partitions at their endpoints, while the space overhead of 1-hop replication is much smaller than replicating frequent query pattern matches. This is called an internal edge. An edge (ternary) between two vertices within the same partition is called an internal edge, while an edge connecting two vertices in different partitions is called a cross edge.

[0008] The traditional objective of vertex-disjoint partitioning is to minimize edge cuts while balancing partition size; this is known as minimum edge cut. It's expected that fewer intersecting edges will result in fewer intersecting matches. This is a reasonable expectation, but it doesn't sufficiently eliminate or reduce the number of joins within partitions. Even at the cost of increased edge cuts, it's generally best to minimize the number of unique attributes cut. The resulting technique, called minimum attribute cut, is a workload-independent technique with significant advantages over its ability to independently execute larger SPARQL query sets.

[0009] For minimum edge cut and minimum attribute cut, since we cannot determine whether query Q has no cross match before execution, even if Q has no cross match during execution, interval joins cannot be avoided. Therefore, its execution involves interval joins in the minimum edge cut and minimum attribute cut partitions.

[0010] If workload is taken into account, the query class can be expanded to avoid range joins. If the workload containing cross attributes has a non-star schema with frequent queries, the overall performance overhead of the workload is low. Summary of the Invention

[0011] The focus of this invention is a partitioning technique that simultaneously considers workload characteristics and RDF graph structure. We propose Minimum Schema Cut as a novel workload-aware partitioning technique for RDF graphs, and then decompose queries into a set of independent executable subqueries based on internal schemas. This minimizes the number of partition schemas spanned and the number of queries connecting intervals under a given workload, resulting in a significant improvement in query execution performance.

[0012] Based on workload characteristics and RDF graph structure, this invention proposes Minimum Pattern Cut as a new workload-aware graph partitioning technique for RDF graphs. The result is a set of partitions. Then, based on the internal patterns, the query is decomposed into a set of independent executable subqueries. The subqueries are executed in parallel within each partition to obtain matching results. This minimizes the number of partition patterns and the number of queries connecting partitions under a given workload, thereby improving query execution performance.

[0013] According to a first aspect of the present invention, a method for partitioning and distributing RDF knowledge graph data is provided, comprising:

[0014] Step 1: For a given RDF knowledge graph data graph G, workload W, and its corresponding schema set M, divide graph G into n subgraph partitions P = {G1, G2, ..., G...} based on the minimum schema partitioning method. i ,…,G n}, to meet the preset conditions.

[0015] The workload W includes multiple queries on graph G. For any query Q, a pattern m corresponding to Q is generated. The patterns corresponding to all queries in the workload W form a pattern set M.

[0016] The preset conditions include: internal pattern set M in Number of patterns | M in Maximize, and for each subgraph partition G i All satisfy | V i |≤(1+ε)×|V| / n, where |V i |Partition G for each subgraph i The number of vertices, |V| is the number of vertices in the original graph G, and ε is the maximum imbalance ratio of the user-defined partition.

[0017] The minimum pattern partitioning method in step 1 includes steps 1.1 to 1.5.

[0018] Step 1.1: Use a greedy algorithm to select the internal pattern set M based on weakly connected components. in Among them, the internal pattern set M inInitially, the set is empty. The candidate pattern set M' is initially the pattern set M. Traverse the candidate pattern set M', and read one candidate pattern m from M' at a time. op Execute step 1.1.1 once.

[0019] Step 1.1.1: For each read of m op Calculate WCC(G[Min∪{mop}]), where WCC(·) represents the set of weakly connected components of a graph set, and G[Min∪{mop}] is the induced sub-graph set of graph G based on Min∪{mop}. Calculate the candidate cost. Where c is the vertex set of any weakly connected component in WCC(G[Min∪{mop}]), and |c| is the number of vertices in vertex set c.

[0020] Step 1.2: For each iteration of the candidate pattern set M', extract the minimum candidate cost that satisfies Cost[Min∪{mop}]≤(1+ε)×|V| / n, and assign the minimum candidate cost to m. op Join M in and the m op Remove from M'; repeatedly traverse the candidate pattern set M' until M' is empty or there are no candidate patterns that can satisfy the minimum candidate cost, thus obtaining the optimal internal pattern set M. in .

[0021] Step 1.3: Based on the internal pattern set M in The generated graph G is based on M. in The inducible set G[M in ], G[M in Each induced subgraph in the graph G is coarsened into a supervertex. Any two supervertexes s1 and s2 that satisfy the supervertex coarsening condition are further coarsened into a single supervertex, until no further coarsening is possible, thus transforming graph G into a coarse graph G. c .

[0022] The coarsening condition for supervertices is that two supervertices have maximum similarity and satisfy |c(s1)|+|c(s2)|≤(1+ε)×|V| / n, where |c(s1)| and |c(s2)| are WCC(G[M in The number of vertices in the weakly connected components corresponding to s1 and s2 in the equation; the similarity between two supervertices. IM(s1) and IM(s2) represent the internal pattern sets corresponding to s1 and s2.

[0023] Step 1.4: Apply the coarse image G c Using a vertex-disjoint partitioning algorithm, the coarse graph G is divided into... cThe graph is divided into n coarse subgraph partitions. Each coarse subgraph partition is decoarsened to obtain n subgraph partitions P of graph G. Decoarsening is the process of restoring the supervertices to the induced subgraphs in their pre-coarsening state.

[0024] M in Each internal pattern m in in For one or more subgraph partitions, m is encoded using DFS. in Convert to sequence encoding.

[0025] Step 1.5: Query user to build a hash table and store it locally on the query user's end. in The sequence encoding information includes the number of partitions n and the IP address of the node where the subgraph partition is located; where the key value of the hash table is m. in The encoding, the hash table value is m in The IP address of the node where one or more subgraph partitions are located.

[0026] Step 2: For a query Q initiated on the user's local end, perform distributed query processing to obtain the matching result set MS(Q,G) of query Q on graph G. The distributed query processing includes: determining whether query Q is an independently executable query. If query Q is an independently executable query, then execute step 2.1. If query Q is not an independently executable query, then execute step 2.2.

[0027] The independently executable queries include the first type of independently executable queries, the second type of independently executable queries, and the third type of independently executable queries. Queries that do not belong to these three types are not independently executable queries.

[0028] Determining whether query Q is an independently executable query includes: for query Q, generating the corresponding pattern m; ignoring vertex constants, if m is M in one m in or multiple m inIf Q is isomorphic to m, then Q is a first-class independent executable query. If for query Q, there exists a subgraph Q' that is a first-class independent executable query, and for all edges e belonging to E(Q)\E(Q'), both endpoints of e belong to V(Q'), then Q is a second-class independent executable query. If for query Q, there exists a subgraph Q' that is a first-class independent executable query, and for all edges e belonging to E(Q)\E(Q'), at least one endpoint of e belongs to V(Q'), and there exists at least one edge e* belonging to E(Q)\E(Q'), one endpoint of e* belongs to V(Q'), and the other endpoint does not belong to V(Q'), then Q is a third-class independent executable query. Here, E(Q) and E(Q') are the sets of edges of Q and Q', V(Q') is the set of vertices of Q', and E(Q)\E(Q') represents the set of edges in E(Q) that do not belong to E(Q').

[0029] Step 2.1: Locate the corresponding partition for query Q, use the DFS encoding of pattern m corresponding to Q to retrieve the hash table stored locally, and take the intersection of the partitions corresponding to pattern m as the partition corresponding to query Q; perform the matching process on the partition corresponding to query Q, and take the union of the matching results on all partitions as the matching result set MS(Q,G) of query Q.

[0030] Step 2.2: Based on the internal pattern set M in Decompose query Q into multiple subqueries q. Then, for each subquery q, retrieve the locally stored hash table according to the DFS encoding of its corresponding pattern mq. Take the intersection of the partitions corresponding to pattern mq as the partition corresponding to subquery q. Perform the matching process of q on the partition corresponding to subquery q. Take the union of the matching results of subquery q on all corresponding partitions as the matching result of subquery q. Then, perform a join operation on the matching results of subquery q to obtain the matching result set MS(Q,G) of query Q.

[0031] Step 2.2 includes: In order to decompose query Q into multiple subqueries q, firstly, remove the cross edges in query Q that do not belong to any internal pattern, obtaining weakly connected components that only contain internal pattern edges, and store the removed cross edges in a set. Remove query Q from the middle. After considering all edges, the resulting weakly connected components containing only internal pattern edges are used as subqueries {q1,q2,…,qx}. For each subquery, its corresponding partition is determined based on its internal pattern. Let each edge vi-vj in the query q(vi) be a vertex and vj be a subquery containing vi and vj. Let |q(vi)| and |q(vj)| be the number of vertices in q(vi) and q(vj) respectively. If |q(vi)| = |q(vj)|, then add edge vi-vj to q(vi) to make query q(vi) a second-class independent executable query. Otherwise, add edge vi-vj to the subquery with more vertices in q(vi) and q(vj). After traversing... Among the subqueries {q1,q2,…,qx} obtained after all edges in the query, the subqueries containing more than one vertex are taken as the result of decomposing query Q.

[0032] In steps 2.1 and 2.2, the matching process on different partitions is executed in parallel.

[0033] Furthermore, the method provided by the present invention also includes: for generating the pattern m corresponding to Q, m is generated by deleting the single-degree vertices of Q and replacing all constants on the subject and object in Q with variables, the constants including literals and URIs.

[0034] Furthermore, the method provided by the present invention further includes: for the internal pattern set M in The internal mode m in in The matching result set MS(m) corresponding to it on graph G in All matches μ in G) in All are complete internal matches, i.e., μ in All edges are in some partition G i The internal edges in the middle.

[0035] Furthermore, the method provided by the present invention also includes: for a pattern m in M, the induced subgraph of graph G based on pattern m is G[m]: a subgraph consisting of all edges in ∪μ∈MS(m,G)E(μ) corresponding to m, where E(μ) is all edges in matching μ; the induced subgraph set G[M] of graph G based on M is the set of G[m].

[0036] According to a second aspect of the present invention, a computer device is provided, comprising: a memory for storing instructions; and a processor for invoking the instructions stored in the memory to execute the method of the first aspect.

[0037] According to a third aspect of the present invention, a computer-readable storage medium is provided, which stores instructions that, when executed by a processor, perform the method of the first aspect.

[0038] Compared with the prior art, the above-mentioned technical solution conceived in this invention has at least the following beneficial effects:

[0039] We propose a minimal pattern-cutting partitioning strategy that considers workload characteristics and maximizes the number of queries that can be executed independently within the workload. Patterns are generated by removing vertices of degree 1 and replacing all constants (strings and URIs) in the subject and object with variables from the workload (sub)queries. The goal of partitioning is defined as minimizing the number of queries with interval joins. This transforms the problem into minimizing the number of cross patterns in the workload (i.e., maximizing the number of internal patterns). Interval joins involve high communication and computational costs, which minimal pattern-cutting significantly reduces.

[0040] Compared to existing vertex-disjoint partitioning methods, minimum pattern cut avoids more interval joins, thus significantly improving distributed SPARQL query processing. The proposed minimum pattern cut technique shares the same design goal as minimum attribute cut: maximizing the number of independently executable queries that can be evaluated without interval joins. However, minimum pattern cut is workload-aware and maximizes the number of independently executable queries within a given workload, while minimum attribute cut is workload-independent and only considers the graph structure. Considering workload, minimum pattern cut offers better query execution performance. For example, in workloads with frequently queried non-star schemas, where interval joins are involved in minimum attribute cut, minimum pattern cut can execute queries independently within partitions, avoiding interval joins.

[0041] Minimal pattern cutting addresses the data locality problem by providing opportunities for data localization. It stores a small amount of data locally, such as the schema in DFS-encoded format and its corresponding partition information. This allows schema-related queries to directly locate the relevant partitions. Queries are executed independently and in parallel within each partition. Localization filters out irrelevant partitions, reducing the number of remote requests and increasing overall throughput and query performance. Minimal pattern cutting coarsens some non-maximum weak join components in the induced subgraphs of all internal schemas, ensuring that all matches of certain internal matches fall into a few partitions. Minimal pattern cutting expands the set of independently executable queries compared to its predecessor, as its query processing method decomposes queries into subqueries, which can then further locate relevant positions.

[0042] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0043] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0044] Figure 1 This is a pattern example shown according to an exemplary embodiment.

[0045] Figure 2 This is an example of a pattern-induced subgraph illustrated according to an exemplary embodiment.

[0046] Figure 3 This is an example of an RDF graph illustrated according to an exemplary embodiment.

[0047] Figure 4 This is an example of a coarsened graph and its partitions, illustrated according to an exemplary embodiment.

[0048] Figure 5 This is an example of a high-level coarse map and its partitions, illustrated according to an exemplary embodiment.

[0049] Figure 6 This is an example of advanced coarse graph decoarsening results, i.e., RDF graph partitioning results, illustrated according to an exemplary embodiment.

[0050] Figure 7 This is a workload example illustrated according to an exemplary embodiment. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0052] This invention proposes a novel workload-aware RDF graph partitioning scheme, called Minimum Schema Cut (MMC), aimed at maximizing the number of Independent Executable Queries (IEQs) in a given workload. We first define the schemas in the workload and transform the problem of maximizing the number of IEQs into maximizing the number of internal schemas. Then, to find the optimal MMC partitioning result, we propose two coarsening-based greedy algorithms that iteratively select some internal schemas and coarsen the matched induced subgraphs, placing more matches with the same internal schema into the same partition for data localization. Finally, based on the schemas, the input query is decomposed into three extended types of IEQs. The decomposed subqueries are localized and executed in parallel on their relevant partitions, and the matches are concatenated to form the final result.

[0053] This invention proposes a method for RDF knowledge graph data partitioning and distributed querying, comprising:

[0054] Step 1: For a given RDF knowledge graph data graph G, workload W, and its corresponding schema set M, divide graph G into n subgraph partitions P = {G1, G2, ..., G...} based on the minimum schema partitioning method. i ,…,G n}, to meet the preset conditions.

[0055] The workload W includes multiple queries on graph G. For any query Q, a pattern m corresponding to Q is generated. The patterns corresponding to all queries in the workload W form a pattern set M.

[0056] The preset conditions include: internal pattern set M in Number of patterns in M in Maximize, and for each subgraph partition G i All satisfy | V i |≤(1+ε)×|V| / n, where |V i |Partition G for each subgraph i The number of vertices, |V| is the number of vertices in the original graph G, and ε is the maximum imbalance ratio of the user-defined partition.

[0057] The minimum pattern partitioning method in step 1 includes steps 1.1 to 1.5.

[0058] Step 1.1: Use a greedy algorithm to select the internal pattern set M based on weakly connected components. in Among them, the internal pattern set M in Initially, the set is empty. The candidate pattern set M' is initially the pattern set M. Traverse the candidate pattern set M', and read one candidate pattern m from M' at a time. op Execute step 1.1.1 once;

[0059] Step 1.1.1: For each read of m op Calculate WCC(G[Min∪{mop}]), where WCC(·) represents the set of weakly connected components of a graph set, and G[Min∪{mop}] is the induced sub-graph set of graph G based on Min∪{mop}. Calculate the candidate cost. Where c is the vertex set of any weakly connected component in WCC(G[Min∪{mop}]), and |c| is the number of vertices in vertex set c;

[0060] Step 1.2: For each iteration of the candidate pattern set M', extract the minimum candidate cost that satisfies Cost[Min∪{mop}]≤(1+ε)×|V| / n, and assign the minimum candidate cost to m. op Join M in and the m opRemove from M'; repeatedly traverse the candidate pattern set M' until M' is empty or there are no candidate patterns that can satisfy the minimum candidate cost, thus obtaining the optimal internal pattern set M. in ;

[0061] Step 1.3: Based on the internal pattern set M in The generated graph G is based on M. in The inducible set G[M in ], G[M in Each induced subgraph in the graph G is coarsened into a supervertex. Any two supervertexes s1 and s2 that satisfy the supervertex coarsening condition are further coarsened into a single supervertex, until no further coarsening is possible, thus transforming graph G into a coarse graph G. c ;

[0062] The coarsening condition for supervertices is that two supervertices have maximum similarity and satisfy |c(s1)|+|c(s2)|≤(1+ε)×|V| / n, where |c(s1)| and |c(s2)| are WCC(G[M in The number of vertices in the weakly connected components corresponding to s1 and s2 in the equation; the similarity between two supervertices. IM(s1) and IM(s2) represent the internal pattern sets corresponding to s1 and s2;

[0063] Step 1.4: Apply the coarse image G c Using a vertex-disjoint partitioning algorithm, the coarse graph G is divided into... c The graph is divided into n coarse subgraph partitions. Each coarse subgraph partition is decoarsened to obtain n subgraph partitions P of graph G. Decoarsening is the process of restoring the hypervertices to the induced subgraphs of the state before coarsening.

[0064] M in Each internal pattern m in in For one or more subgraph partitions, m is encoded using DFS. in Convert to sequence encoding;

[0065] Step 1.5: Query user to build a hash table and store it locally on the query user's end. in The sequence encoding information includes the number of partitions n and the IP address of the node where the subgraph partition is located; where the key value of the hash table is m. in The encoding, the hash table value is m in The IP address of the node where one or more subgraph partitions are located;

[0066] Step 2: For a query Q initiated on the user's local end, perform distributed query processing to obtain the matching result set MS(Q,G) of query Q on graph G. The distributed query processing includes: determining whether query Q is an independently executable query. If query Q is an independently executable query, then execute step 2.1. If query Q is not an independently executable query, then execute step 2.2.

[0067] The independently executable queries include the first type of independently executable queries, the second type of independently executable queries, and the third type of independently executable queries. Queries that do not belong to these three types are not independently executable queries.

[0068] Determining whether query Q is an independently executable query includes: for query Q, generating the corresponding pattern m; ignoring vertex constants, if m is M in one m in or multiple m in If a combination of elements is such that Q is isomorphic to m, then Q is a first-type independent executable query. If for a query Q, there exists a subgraph Q' that is a first-type independent executable query, and for all edges e belonging to E(Q)\E(Q'), both endpoints of e belong to V(Q'), then Q is a second-type independent executable query. If for a query Q, there exists a subgraph Q' that is a first-type independent executable query, and for all edges e belonging to E(Q)\E(Q'), at least one endpoint of e belongs to V(Q'), and there exists at least one edge e* belonging to E(Q)\E(Q'), with one endpoint belonging to V(Q') and the other endpoint not belonging to V(Q'), then Q is a third-type independent executable query. Here, E(Q) and E(Q') are the sets of edges of Q and Q', V(Q') is the set of vertices of Q', and E(Q)\E(Q') represents the set of edges in E(Q) that do not belong to E(Q').

[0069] Step 2.1: Locate the corresponding partition for query Q, use the DFS encoding of pattern m corresponding to Q to retrieve the hash table stored locally, and take the intersection of the partitions corresponding to pattern m as the partition corresponding to query Q; perform the matching process on the partition corresponding to query Q, and take the union of the matching results on all partitions as the matching result set MS(Q,G) of query Q.

[0070] Step 2.2: Based on the internal pattern set M in Decompose query Q into multiple subqueries q. Then, for each subquery q, retrieve the hash table stored locally according to the DFS encoding of its corresponding pattern mq. Take the intersection of the partitions corresponding to pattern mq as the partition corresponding to subquery q. Perform the matching process of q on the partition corresponding to subquery q. Take the union of the matching results of subquery q on all corresponding partitions as the matching result of subquery q. Then, perform a join operation on the matching results of subquery q to obtain the matching result set MS(Q,G) of query Q.

[0071] Step 2.2 includes: In order to decompose query Q into multiple subqueries q, firstly, remove the cross edges in query Q that do not belong to any internal pattern, obtaining weakly connected components that only contain internal pattern edges, and store the removed cross edges in a set. Remove query Q from the middle. After considering all edges, the resulting weakly connected components containing only internal pattern edges are used as subqueries {q1,q2,…,qx}. For each subquery, its corresponding partition is determined based on its internal pattern. Let each edge vi-vj in the query q(vi) be a vertex and vj be a subquery containing vi and vj. Let |q(vi)| and |q(vj)| be the number of vertices in q(vi) and q(vj) respectively. If |q(vi)| = |q(vj)|, then add edge vi-vj to q(vi) to make query q(vi) a second-class independent executable query. Otherwise, add edge vi-vj to the subquery with more vertices in q(vi) and q(vj). After traversing... Among the subqueries {q1,q2,…,qx} obtained after all edges in the query, the subqueries containing more than one vertex are taken as the result of decomposing query Q;

[0072] In steps 2.1 and 2.2, the matching process on different partitions is executed in parallel.

[0073] For generating the pattern m corresponding to Q, m is generated by deleting the single-degree vertices of Q and replacing all constants in the subject and object of Q with variables. These constants include literals and URIs. Our defined pattern summarizes a specific query in the workload; it is an abstract query template. A query is an instance of the pattern. By minimizing the partitioning of patterns into different partitions—that is, maximizing the internal pattern set—we reduce the number of partition joins in distributed queries based on the workload, thereby improving the execution efficiency of distributed queries.

[0074] For the internal pattern set M in The internal mode m in in The matching result set MS(m) corresponding to it on graph G in All matches μ in G) in All are complete internal matches, i.e., μ in All edges are in some partition G i The internal edges in the middle.

[0075] For a pattern m in M, the induced subgraph of graph G based on pattern m is G[m]: the subgraph consisting of all edges in ∪μ∈MS(m,G)E(μ) corresponding to m, where E(μ) is all edges in matching μ; the induced subgraph set G[M] of graph G based on M is the set of G[m].

[0076] A weakly connected component (WCC) is a maximal subgraph of a directed graph, in which any two vertices can be reached from each other by a directed edge (regardless of direction). Using weakly connected components simplifies the representation of a graph and reduces computational complexity.

[0077] In one embodiment, such as Figures 1-7 As shown, for an RDF graph as follows Figure 3 Due to workload such Figure 7 Acquisition mode such as Figure 1 ,Will Figure 1 The induced subgraph of the middle pattern is as follows Figure 2 The matching is coarsened, and the coarsened graph is partitioned as follows: Figure 4 As shown, both supervertex1 and supervertex3 originate from the contained Figure 1 Some matches in pattern m1 are coarsened, so they can be further coarsened into a supervertex4, and this further coarsened graph can be as follows: Figure 5 The partitioning is shown. Uncoarsening is then performed on the partitions of the high-level coarsened map to obtain the partitions of the original map, as shown. Figure 6 As shown. Here, since all matches of m1 have been coarsened to Supervertex4, and Supervertex4 belongs to G1, queries related to m1 in the workload can be executed independently in G1.

[0078] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the following claims.

[0079] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A method for partitioning and distributed querying RDF knowledge graph data, characterized in that, include: Step 1: For a given RDF knowledge graph data graph G, workload W, and its corresponding schema set M, divide graph G into n subgraph partitions P = {G1, G2, ..., G...} based on the minimum schema partitioning method. i ,…,G n }, to meet the preset conditions; The workload W includes multiple queries on graph G. For any query Q, a pattern m corresponding to Q is generated. The patterns corresponding to all queries in the workload W form a pattern set M. The preset conditions include: internal pattern set M in Number of patterns | M in Maximize, and for each subgraph partition G i All satisfy | V i |≤(1+ε)×|V| / n, where |V i |Partition G for each subgraph i The number of vertices, |V| is the number of vertices in the original graph G, and ε is the maximum imbalance ratio of the user-defined partition; The minimum pattern partitioning method in step 1 includes steps 1.1-1.5; Step 1.1: Use a greedy algorithm to select the internal pattern set M based on weakly connected components. in Among them, the internal pattern set M in Initially, the set is empty. The candidate pattern set M' is initially the pattern set M. Traverse the candidate pattern set M', and read one candidate pattern m from M' at a time. op Execute step 1.1.1 once; Step 1.1.1: For each read of m op Calculate WCC(G[Min∪{mop}]), where WCC(·) represents the set of weakly connected components of a graph set, and G[Min∪{mop}] is the induced sub-graph set of graph G based on Min∪{mop}. Calculate the candidate cost. Where c is the vertex set of any weakly connected component in WCC(G[Min∪{mop}]), and |c| is the number of vertices in vertex set c; Step 1.2: For each iteration of the candidate pattern set M', extract the minimum candidate cost that satisfies Cost[Min∪{mop}]≤(1+ε)×|V| / n, and assign the minimum candidate cost to m. op Join M in and the m op Remove from M'; repeatedly traverse the candidate pattern set M' until M' is empty or there are no candidate patterns that can satisfy the minimum candidate cost, thus obtaining the optimal internal pattern set M. in ; Step 1.3: Based on the internal pattern set M in The generated graph G is based on M. in The inducible set G[M in ], G[M in Each induced subgraph in the graph G is coarsened into a supervertex. Any two supervertexes s1 and s2 that satisfy the supervertex coarsening condition are further coarsened into a single supervertex, until no further coarsening is possible, thus transforming graph G into a coarse graph G. c ; The coarsening condition for supervertices is that two supervertices have maximum similarity and satisfy |c(s1)|+|c(s2)|≤(1+ε)×|V| / n, where |c(s1)| and |c(s2)| are WCC(G[M in The number of vertices in the weakly connected components corresponding to s1 and s2 in the equation; the similarity between two supervertices. IM(s1) and IM(s2) represent the internal pattern sets corresponding to s1 and s2; Step 1.4: Apply the coarse image G c Using a vertex-disjoint partitioning algorithm, the coarse graph G is divided into... c The graph is divided into n coarse subgraph partitions. Each coarse subgraph partition is decoarsened to obtain n subgraph partitions P of graph G. Decoarsening is the process of restoring the hypervertices to the induced subgraphs of the state before coarsening. M in Each internal pattern m in in For one or more subgraph partitions, m is encoded using DFS. in Convert to sequence encoding; Step 1.5: Query user to build a hash table and store it locally on the query user's end. in The sequence encoding information includes the number of partitions n and the IP address of the node where the subgraph partition is located; where the key value of the hash table is m. in The encoding, the hash table value is m in The IP address of the node where one or more subgraph partitions are located; Step 2: For a query Q initiated on the user's local end, perform distributed query processing to obtain the matching result set MS(Q,G) of query Q on graph G. The distributed query processing includes: determining whether query Q is an independently executable query. If query Q is an independently executable query, then execute step 2.

1. If query Q is not an independently executable query, then execute step 2.

2. The independently executable queries include the first type of independently executable queries, the second type of independently executable queries, and the third type of independently executable queries. Queries that do not belong to these three types are not independently executable queries. Determining whether query Q is an independently executable query includes: for query Q, generating the corresponding pattern m; ignoring vertex constants, if m is M in one m in or multiple m in If a combination of elements is such that Q is isomorphic to m, then Q is a first-type independent executable query. If for a query Q, there exists a subgraph Q' that is a first-type independent executable query, and for all edges e belonging to E(Q)\E(Q'), both endpoints of e belong to V(Q'), then Q is a second-type independent executable query. If for a query Q, there exists a subgraph Q' that is a first-type independent executable query, and for all edges e belonging to E(Q)\E(Q'), at least one endpoint of e belongs to V(Q'), and there exists at least one edge e* belonging to E(Q)\E(Q'), with one endpoint belonging to V(Q') and the other endpoint not belonging to V(Q'), then Q is a third-type independent executable query. Here, E(Q) and E(Q') are the sets of edges of Q and Q', V(Q') is the set of vertices of Q', and E(Q)\E(Q') represents the set of edges in E(Q) that do not belong to E(Q'). Step 2.1: Locate the corresponding partition for query Q, use the DFS encoding of pattern m corresponding to Q to retrieve the hash table stored locally, and take the intersection of the partitions corresponding to pattern m as the partition corresponding to query Q; perform the matching process on the partition corresponding to query Q, and take the union of the matching results on all partitions as the matching result set MS(Q,G) of query Q. Step 2.2: Based on the internal pattern set M in Decompose query Q into multiple subqueries q. Then, for each subquery q, retrieve the hash table stored locally according to the DFS encoding of its corresponding pattern mq. Take the intersection of the partitions corresponding to pattern mq as the partition corresponding to subquery q. Perform the matching process of q on the partition corresponding to subquery q. Take the union of the matching results of subquery q on all corresponding partitions as the matching result of subquery q. Then, perform a join operation on the matching results of subquery q to obtain the matching result set MS(Q,G) of query Q. Step 2.2 includes: In order to decompose query Q into multiple subqueries q, firstly, remove the cross edges in query Q that do not belong to any internal pattern, obtaining weakly connected components that only contain internal pattern edges, and store the removed cross edges in a set. Remove query Q from the middle. After considering all edges, the resulting weakly connected components containing only internal pattern edges are used as subqueries {q1,q2,…,qx}. For each subquery, its corresponding partition is determined based on its internal pattern. Let each edge vi-vj in the query q(vi) be a vertex and vj be a subquery containing vi and vj. Let |q(vi)| and |q(vj)| be the number of vertices in q(vi) and q(vj) respectively. If |q(vi)| = |q(vj)|, then add edge vi-vj to q(vi) to make query q(vi) a second-class independent executable query. Otherwise, add edge vi-vj to the subquery with more vertices in q(vi) and q(vj). After traversing... Among the subqueries {q1,q2,…,qx} obtained after all edges in the query, the subqueries containing more than one vertex are taken as the result of decomposing query Q; In steps 2.1 and 2.2, the matching process on different partitions is executed in parallel.

2. The method according to claim 1, characterized in that: For generating the pattern m corresponding to Q, m is generated by deleting the single-degree vertices of Q and replacing all constants in the subject and object of Q with variables, including literals and URIs.

3. The method according to claim 1, characterized in that, For the internal pattern set M in The internal mode m in in The matching result set MS(m) corresponding to it on graph G in All matches μ in G) in All are complete internal matches, i.e., μ in All edges are in some partition G i The internal edges in the middle.

4. The method according to claim 1, characterized in that, For a pattern m in M, the induced subgraph of graph G based on pattern m is G[m]: the subgraph consisting of all edges in ∪μ∈MS(m,G)E(μ) corresponding to m, where E(μ) is all edges in matching μ; the induced subgraph set G[M] of graph G based on M is the set of G[m].

5. A computer device, characterized in that, include: Memory, used to store instructions; A processor for invoking instructions stored in the memory to execute the method as described in any one of claims 1-4.

6. A computer-readable storage medium, characterized in that, The device stores instructions that, when executed by a processor, perform the method as described in any one of claims 1-4.

Images