A SPARQL query optimization method and system in a parallel computing environment

Abstract

The application provides a SPARQL query optimization method and system in a parallel computing environment, and belongs to the technical field of RDF query optimization.The scheme splits the candidate edge set corresponding to the first query edge of the query graph structure, performs task decomposition optimization, realizes semantic independence of parallel tasks, and avoids dependence blocking in traditional pipeline parallelism.Meanwhile, the scheme first integrates query semantics into local sensitive hash design, constructs a hash sketch vector including query attribute constraints, effectively quantifies the attribute neighborhood similarity of the candidate edge, and realizes task-level redundant calculation elimination.The scheme significantly reduces redundant calculation in the query process, effectively improves the query efficiency of large-scale RDF graphs, and simultaneously combines the "worst-case optimal connection" feature of the Generic Join algorithm with parallel task decomposition for the first time, proposes an "expanding by connected subgraph vertex by vertex" execution paradigm, and breaks through the performance bottleneck of traditional algorithms in large-scale RDF graphs.

Description

Technical Field

[0001] This invention belongs to the field of RDF query optimization technology, and in particular relates to a SPARQL query optimization method and system in a parallel computing environment. Background Technology

[0002] The Resource Description Framework (RDF), as the foundational framework for representing knowledge graph data, provides a structured approach to effectively capture relationships between resources. This approach is achieved through several triples, each consisting of a subject (the focus resource in the description), a predicate (the attribute or relation described), and an object (the value or other resource).

[0003] SPARQL (SPARQL Protocol and RDF Query Language) is a proficient query language and protocol specifically designed for querying RDF knowledge graph data. The inventors discovered that existing SPARQL query methods have the following problems:

[0004] (1) SPARQL query performance bottleneck: Traditional SPARQL queries face the problem of low efficiency in complex join queries in large-scale RDF graphs. SPARQL queries usually involve join operations of multiple triples. When processing different subtasks of a single query or multiple concurrent queries, traditional methods cannot effectively identify the common computational parts between tasks, resulting in a large number of redundant subgraph matching computations. For example, different query tasks may share the exploration space of the same RDF subgraph, but existing frameworks lack a mechanism to utilize this locality, resulting in a serious waste of computing resources.

[0005] (2) Insufficient Parallelism: Early parallel SPARQL query frameworks mostly adopted simple data sharding strategies (such as hash-based random sharding) without combining the structural features of the RDF graph with query semantics. Such methods cannot guarantee load balancing between tasks and are difficult to avoid data transmission overhead across nodes. For example, task allocation based on random hashing may lead to similar query tasks being assigned to different nodes, increasing unnecessary communication costs.

[0006] (3) Efficiency bottleneck of join algorithm: Traditional join algorithms (such as nested loop join) have high time complexity, especially when dealing with SPARQL basic graph schema (BGP) queries containing a large number of variables, which cannot meet the real-time requirements. Although worst-case optimal join algorithms (such as Generic Join) theoretically provide better complexity guarantees, how to effectively parallelize them is still an unsolved problem. Summary of the Invention

[0007] This invention provides a SPARQL query optimization method and system in a parallel computing environment, which significantly reduces redundant calculations in the query process and effectively improves the query efficiency of large-scale RDF graphs.

[0008] According to a first aspect of the present invention, a SPARQL query optimization method in a parallel computing environment is provided, comprising:

[0009] Convert the SPARQL query to be executed into a query graph;

[0010] Based on the first query edge determined from the query graph, and combined with the RDF graph to be queried, a set of candidate edges corresponding to the first query edge is generated; wherein, each candidate edge corresponds to a triple in the RDF graph that matches the predicate of the first query edge.

[0011] For candidate edges in the candidate edge set, they are distributed to different computing nodes for parallel execution of query tasks through consistent hashing. Specifically, in each computing node, the query task is executed based on the worst-case optimal connection algorithm. For candidate edges distributed to different computing nodes, a hash sketch vector containing query attribute constraints is constructed based on the minimum hash value of the neighborhood vertex corresponding to each query attribute. Candidate edges with the same hash sketch vector are merged into the same query task for sequential execution.

[0012] The query results from different computing nodes are aggregated and deduplicated to obtain the final query result.

[0013] Furthermore, the execution of the query task based on the worst-case optimal join algorithm in each computing node specifically involves: generating the query vertex matching order corresponding to the query graph through a heuristic strategy; splitting the query process into consecutive query tasks based on the query vertex matching order; and executing the query tasks using the Generic Join algorithm to obtain the query results.

[0014] Furthermore, the construction of the hash sketch vector containing query attribute constraints is specifically as follows:

[0015] Get the query attribute set of the first query edge;

[0016] For each attribute in the query attribute set, iterate through the neighboring vertices of the current candidate edge and calculate the minimum hash value of the neighboring vertices;

[0017] Based on the minimum hash value corresponding to the neighboring vertices of the current candidate edge under each attribute, obtain the hash sketch vector representation of the current candidate edge.

[0018] Furthermore, the process of merging candidate edges with the same hash sketch vector into the same query task for sequential execution involves integrating multiple candidate edges into a single task for serial processing.

[0019] Furthermore, the consistent hashing adopts a hybrid hashing strategy, specifically: first, the predicates in the candidate edges are hashed to obtain the initial computing node number; then, the main vertex URIs in the candidate edges are hashed to obtain the subdivided hash values; and the target computing node of the candidate edge is determined based on the combination of the initial computing node number and the subdivided hash values.

[0020] According to a second aspect of the present invention, a SPARQL query optimization system for a parallel computing environment is provided, comprising:

[0021] The query graph conversion module is used to convert the SPARQL query to be executed into a query graph;

[0022] The candidate edge set construction module is used to generate a set of candidate edges corresponding to the first query edge based on the first query edge determined from the query graph and combined with the RDF graph to be queried; wherein each candidate edge corresponds to a triple in the RDF graph that matches the predicate of the first query edge.

[0023] The task allocation and execution module is used to allocate candidate edges in the candidate edge set to different computing nodes for parallel execution of query tasks through consistent hashing calculation. Specifically, in each computing node, the query task is executed based on the worst-case optimal connection algorithm. For candidate edges allocated to different computing nodes, a hash sketch vector containing query attribute constraints is constructed based on the minimum hash value of the neighborhood vertex corresponding to each query attribute of the candidate edge. Candidate edges with the same hash sketch vector are merged into the same query task for sequential execution.

[0024] The query results output module is used to aggregate and deduplicate the query task execution results from different computing nodes to obtain the query results.

[0025] According to a third aspect of the present invention, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and running on the memory, wherein the processor executes the program to implement the SPARQL query optimization method in a parallel computing environment.

[0026] According to a fourth aspect of the present invention, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the SPARQL query optimization method in a parallel computing environment.

[0027] The above one or more technical solutions have the following beneficial effects:

[0028] This invention provides a SPARQL query optimization method and system in a parallel computing environment. The scheme optimizes the task by splitting the candidate edge set corresponding to the first query edge of the query graph structure, achieving semantic independence of parallel tasks and avoiding dependency blocking in traditional pipelined parallelism. Simultaneously, the scheme integrates query semantics into locality-sensitive hashing design for the first time. By constructing a hash sketch vector including query attribute constraints, it effectively quantifies the attribute neighborhood similarity of candidate edges, achieving task-level redundant computation elimination. The scheme significantly reduces redundant computation during the query process, effectively improving the query efficiency of large-scale RDF graphs.

[0029] The solution described in this invention combines the "worst-case optimal connection" characteristic of the Generic Join algorithm with parallel task decomposition for the first time, and proposes an execution paradigm of "expanding vertex by vertex according to connected subgraphs", which breaks through the performance bottleneck of traditional algorithms in large-scale RDF graphs.

[0030] The solution described in this invention combines task priority and node load feedback to ensure efficient utilization of computing resources in heterogeneous clusters, breaking through the performance bottleneck of traditional static allocation.

[0031] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0032] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0033] Figure 1 This is a flowchart of the SPARQL query optimization method in a parallel computing environment as described in this embodiment of the invention;

[0034] Figure 2 This is an example of SPARQL query processing based on parallel joins and optimized by QPMH as described in this embodiment of the invention;

[0035] Figures 3(a) and 3(b) are examples of RDF graphs and SPARQL query graphs described in the embodiments of the present invention, respectively.

[0036] Figure 4 This is an example of SPARQL query processing based on parallel connections as described in this embodiment of the invention;

[0037] in, Figures 2 to 4 The number at the top left of each vertex is its unique identifier. Detailed Implementation

[0038] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0039] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0040] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0041] In one or more embodiments, such as Figure 1 As shown, a SPARQL query optimization method is provided in a parallel computing environment, including the following processing steps:

[0042] Step 1: Convert the SPARQL query to be executed into a query graph;

[0043] Step 2: Based on the first query edge determined from the query graph, and combined with the RDF graph to be queried, generate a set of candidate edges corresponding to the first query edge; wherein, each candidate edge corresponds to a triple in the RDF graph that matches the predicate of the first query edge;

[0044] Step 3: For candidate edges in the candidate edge set, they are distributed to different computing nodes for parallel execution of query tasks through consistent hashing. In each computing node, the query task is executed based on the worst-case optimal connection algorithm. For candidate edges distributed to different computing nodes, a hash sketch vector containing query attribute constraints is constructed based on the minimum hash value of the neighborhood vertex corresponding to each query attribute. Candidate edges with the same hash sketch vector are merged into the same query task for sequential execution.

[0045] Step 4: Aggregate and deduplicate the query results from different computing nodes to obtain the query results.

[0046] In one or more implementations, step 1: convert the SPARQL query to be executed into a query graph, specifically including the following processing steps:

[0047] Receive SPARQL queries from users;

[0048] Transform the received SPARQL query into a query graph Q = {V} Q E Q, L Q , f Q}; where the vertex set V QIncludes entities and variables, edge set E Q Representing triplet relations, edge label set L Q f represents all attributes in the RDF graph G. Q f is a mapping function. Q Its core function is to "define the labels of query edges" and it is a "bridge function" that connects the "query graph" and the "RDF graph".

[0049] In practice, the SPARQL query transformation uses the Apache Jena project (https: / / jena.apache.org / ), an open-source framework for handling RDF and SPARQL. Its SPARQL parser supports converting queries into graph structures. The processing steps for query transformation using this framework are as follows:

[0050] (1) Parse the SPARQL query into an abstract syntax tree using the SPARQL parser;

[0051] (2) Use the QueryFactory of the Jena project to parse the SPARQL string into a Query object and extract the graph pattern, specifically including:

[0052] Vertex set V Q Extract entities and variables from triples, such as ?city (variable, where '?' is the prefix used to represent variables in RDF (Resource Description Framework)).<http: / / dbpedia.org / ontology / City> (entity);

[0053] Edge set E Q The predicate of a triple forms an edge, connecting the subject and object vertices, such as ?city → <rdf:type> → <city>;

[0054] (3) Tag and filter mapping:

[0055] Assign labels to vertices: the label for a variable vertex (such as ?city) is the variable name, and the label for an entity vertex is its URI;

[0056] Assigning labels to edges: The label of an edge is a predicate URI (Uniform Resource Identifier) ​​(e.g., ...).<http: / / www.w3.org / 1999 / 02 / 22-rdf-syntax-ns#type> ), corresponding to L Q .

[0057] Furthermore, if the query contains a FILTER clause (such as FILTER (strlen(?city) > 5)), it is converted into a filter function F for filtering the results.

[0058] In other implementations, the query graph can be transformed using the SPARQL processor of RDF4J (source: https: / / rdf4j.org / ) through the following steps:

[0059] (1) Lexical and syntactic analysis: convert SPARQL into an internal query model.

[0060] (2) Graph pattern extraction: Parse the triplet pattern and generate V Q (Variable + Entity) and E Q (Predicate edge).

[0061] (3) Label and Filter Mapping: Assigning predicate labels (L) to edges Q ), and process FILTER as F.

[0062] Figures 3(a) and 3(b) show examples of the RDF graph and SPARQL query graph described in this embodiment, respectively.

[0063] In one or more embodiments, step 2: Based on the first query edge determined from the query graph, and combined with the RDF graph to be queried, generate a set of candidate edges corresponding to the first query edge; wherein each candidate edge corresponds to a triple in the RDF graph that matches the predicate of the first query edge; specifically including the following processing steps:

[0064] (1) Determine the first query edge e in the query graph r ∈E Q (Usually the first edge in the query execution order);

[0065] Specifically, the first query edge e r ∈ E Q The selection is usually based on the following strategies (in descending order of priority):

[0066] 1) Minimum Variable Binding Principle: Select the edge that contains the fewest unbound variables.

[0067] For example, if there exists an edge ( <dbpedia:berlin> , <rdf:type>,?type) and (?city, <dbo:capitalof>Since the former only has one variable (?type), the former is preferred.

[0068] 2) Fixed Attribute Priority: If multiple edges have the same number of variables, prioritize edges whose predicates are fixed attributes (non-variables). For example, prioritize edges with (?x, ?x, ?x) as the predicate. <rdf:type> , <dbo:city>The reason why the predicate of the former is (?x, ?p, ?y) instead of (?x, ?p, ?y) is that the predicate of the former is <rdf:type>This is a fixed attribute.

[0069] 3) Predefined execution order: If the query graph has an explicit execution plan (such as the order in which the query optimizer generates the plan), then the first edge is selected according to the plan.

[0070] It should be noted that in practical applications, choosing either edge as the first edge is feasible.

[0071] (2) Generate its candidate edge set C(e r ), where each candidate edge corresponds to an edge in the RDF graph with e r Tag-matching triples.

[0072] Specifically, the process for generating the candidate edge set is shown in Algorithm 1 below.

[0073] Algorithm 1:

[0074] Input: Query graph Q = {V Q E Q , L Q , f Q }, RDF graph G;

[0075] Output: The set of candidate edges C(er) for the first query edge er;

[0076] Step 1: Determine the first query edge e r ∈ E Q :

[0077] - Initialize min_vars = +∞, er = null;

[0078] - Traverse each edge e ∈ E Q :

[0079] - Calculate the number of unbound variables v_count in edge e.

[0080] - If v_count < min_vars, then update min_vars = v_count, e r = e.

[0081] - If v_count == min_vars and the predicate of e is a fixed attribute, then update e. r = e.

[0082] - Return e r .

[0083] Step 2: Generate the candidate edge set C(er):

[0084] - Let e r The structure is (s, p, o), where:

[0085] - s is the subject (entity or variable) of the edge.

[0086] - p is the predicate (fixed attribute or variable) of the edge.

[0087] - o is the object (entity or variable) of the edge.

[0088] - According to e r Different matching logic is executed for different types:

[0089] a) If p is a fixed attribute (e.g.) <rdf:type>):

[0090] - Find all triples with predicate p in the RDF graph G.

[0091] - If s is an entity (such as...) <dbpedia:berlin>If the subject is s, then the triplet will be further filtered.

[0092] - If o is an entity (such as...) <dbo:city>If the object is o, then the triplet will be further filtered.

[0093] - The set of triples generated is C(er).

[0094] b) If p is a variable (e.g., ?p):

[0095] - Find all triples in RDF graph G with subject s and object o (ignore the corresponding constraints if s or o are variables).

[0096] - In the generated set of triples, the predicate of each triple is a candidate value for the variable ?p.

[0097] Step 3: Apply the filter function F:

[0098] - If e r If an associated filtering condition (such as FILTER(?x > 10)) is applied, then after generating candidate edges, edges that do not meet the condition are further filtered.

[0099] To facilitate understanding, the following explains the matching relationship between the candidate edges obtained above and the triples in the RDF graph:

[0100] Explanation of core concepts:

[0101] Candidate edge: Query an edge e in the graph r Possible matching triples in an RDF graph.

[0102] Label matching: The predicate of a candidate edge must be matched with e. r The labels (i.e., predicates) are consistent. If e r If the predicate is a variable (such as ?p), then all possible predicates are matched.

[0103] Example explanation:

[0104] The first query edge e in the query graph r :

[0105] (person, <foaf:knows>, ?friend)

[0106] Relevant triples in an RDF diagram:

[0107] <ex:alice> <foaf:knows> <ex:bob> ；

[0108] <ex:alice> <foaf:knows> <ex:charlie> ；

[0109] <ex:bob> <foaf:age>"30"^^xsd:integer; # Predicate mismatch, exclude

[0110] Candidate edge set C (e r ):

[0111] {

[0112] ( <ex:alice> , <foaf:knows> , <ex:bob> ),

[0113] ( <ex:alice> , <foaf:knows> , <ex:charlie>)

[0114] }

[0115] Special cases of variable predicates:

[0116] The first query edge e in the query graph r :

[0117] ( <ex:alice>, ?p, ?object) # The predicate is the variable ?p

[0118] Relevant triples in an RDF diagram:

[0119] <ex:alice> <foaf:knows> <ex:bob> .

[0120] <ex:alice> <foaf:name>"Alice" .

[0121] <ex:alice> <dbo:birthplace> <dbr:london>.

[0122] Candidate edge set C (e r ):

[0123] {

[0124] ( <ex:alice> , <foaf:knows> , <ex:bob> ),

[0125] ( <ex:alice> , <foaf:name>, "Alice"),

[0126] ( <ex:alice> , <dbo:birthplace> , <dbr:london>)

[0127] }

[0128] At this point, the possible bound values ​​for the variable ?p are: { <foaf:knows> , <foaf:name> , <dbo:birthplace>}

[0129] In one or more implementations, the scheme described in this embodiment also performs pruning optimization on the obtained candidate edges using prior knowledge, reducing the amount of subsequent matching calculations by filtering out invalid candidate edges. Its core relies on two types of data:

[0130] RDF graph statistical metadata includes vertex in-degree distribution, attribute frequency, entity type hierarchy, etc.

[0131] Query graph structure features: such as edge predicate type (fixed attribute / variable), variable constraints (FILTER clause), etc.

[0132] Specifically, the general pruning process is as follows (executed in four stages):

[0133] Phase 1: Metadata Collection and Index Building;

[0134] Phase 2: Query graph feature extraction and pruning condition generation;

[0135] Phase 3: Multi-dimensional pruning of candidate edges;

[0136] Phase 4: Verification and dynamic adjustment of pruning results;

[0137] like Figure 2 As shown, the arrows illustrate both data processing and query logic, such as filtering edges that do not have the foaf:knows attribute; and demonstrate the scan-expansion process of Generic Join, first matching ? x→?y, then expanding ? y→?z.

[0138] Furthermore, by combining statistical information from the RDF graph (such as vertex in-degree distribution and attribute frequency), candidate edges with extremely low degrees or mismatched attributes can be filtered out. For example, if the first query edge attribute is "director", then only edges with the attribute "director" in the RDF graph will be retained as candidates, reducing subsequent invalid calculations.

[0139] In one or more embodiments, step 3: For candidate edges in the candidate edge set, they are distributed to different computing nodes for parallel execution of query tasks through consistent hashing; wherein, in each computing node, the query task is executed based on the worst-case optimal connection algorithm, and for candidate edges distributed to different computing nodes, a hash sketch vector containing query attribute constraints is constructed according to the minimum hash value of the neighborhood vertex corresponding to each query attribute, and candidate edges with the same hash sketch vector are merged into the same query task for sequential execution. The following is a detailed description:

[0140] (I) Relevant Definitions and Examples

[0141] (1) Candidate edges refer to the actual triple edges in the RDF graph that match the tag (attribute) of the query edge (such as edge er in the query graph) during the query graph processing. Specifically:

[0142] The query edge is an edge er∈E in the query graph Q. Q Its tag is f Q (er) Determined (may be a specific attribute or variable). Candidate edges are those in the RDF graph that satisfy label (i.e., the predicate in the triple) = f Q The triples of (er) can be considered as a candidate edge, thus forming the candidate edge set C(er).

[0143] Example explanation:

[0144] Suppose there is a query edge er in the query graph Q, with label f. Q (er) = "hasChild" (representing the attribute "has children"). In an RDF graph, the following triple exists:

[0145] (Alice, hasChild, Bob)

[0146] (Bob, hasChild, Charlie)

[0147] (David, hasFriend, Eve) (Tag: hasFriend, does not match)

[0148] The candidate edge set C(er) contains the first two triples, i.e.: C(er) = { (Alice, hasChild, Bob), (Bob, hasChild, Charlie)}, because the labels (i.e., predicates) in the first two triples match the label (i.e., predicate: hasChild) of the first query edge er, while the label of the third triple is different and is not included in the candidate edge set.

[0149] (2) Neighboring vertices refer to vertices directly connected to candidate edges, and are divided into two types:

[0150] Starting vertex (Subject): The subject of the candidate edge triple (the first element of the triple).

[0151] Terminating vertex (Object): The object of the candidate edge triple (the third element of the triple).

[0152] Among them, the neighboring vertex information is used for subsequent query expansion (such as connecting other query edges), and usually includes the vertex's identifier (entity or variable) and type (entity / variable).

[0153] Example explanation:

[0154] Take the candidate edge (Alice, hasChild, Bob) as an example:

[0155] Starting vertex (subject): Alice;

[0156] Terminating vertex (object): Bob;

[0157] Then the neighborhood vertices of the candidate edge are {Alice, Bob};

[0158] If the query graph subsequently contains edge er2 with connection variables ?x and ?y, and the label (i.e., predicate) is hasParent, then the query scope needs to be expanded by matching the neighboring vertices of the candidate edge (such as Bob) with the connection variables of edge er2.

[0159] (II) Parallel Execution Process of Query Tasks

[0160] Taking the query "Find directors born in the United States and their science fiction films" as an example, the parsed query graph contains the following edges:

[0161] ?director - a - Person (First query edge, predicate 'a' represents a type assertion, which is the standard abbreviation for rdf:type)

[0162] `?director - birthPlace - "USA"` (attribute constraint)

[0163] Director - Creator - Movie (The relationship between director and film)

[0164] ?movie - a - Movie (Movie genre assertion)

[0165] ?movie - genre - "Science Fiction" (film genre constraint)

[0166] The first query edge is selected as: ?director - a - Person, because it contains the variable "?director" and the predicate "a" is a high-frequency type assertion. A candidate edge set is generated by scanning all triples of type Person in the RDF graph. After the candidate edge set is generated, pruning optimization is performed based on vertex degree statistics. For example, if the out-neighborhood of the main vertex director of a candidate edge does not contain the predicate birthPlace, the candidate edge is directly filtered to reduce invalid tasks.

[0167] Based on the above example, the parallel execution process for query tasks includes the following steps:

[0168] Step 301: Distribute candidate edges to n computing nodes {W0, W1, ..., Wn-1} using a consistent hash function H: E → {0, 1, ..., n-1}. Specifically, candidate edges e' ∈ C(er) are assigned to node Wi, where i = H(e') mod n. Each computing node initializes an independent query task for the assigned candidate edge, which includes information about the candidate edge and its neighboring vertices.

[0169] In one or more embodiments, the hash function employs a hybrid hashing strategy: first, the candidate edge predicate (e.g., predicate a) is hashed (using the FNV-1a hash function) to obtain the initial node number; then, the first 16 bytes of the URI of the candidate edge's main vertex are hashed to obtain the subdivided hash value; finally, the target storage node of the candidate edge is determined by the combination of "initial node number + subdivided hash value", ensuring that candidate edges of different main bodies under the same predicate are evenly distributed, while taking into account both predicate correlation and load balancing.

[0170] The first hash (predicate hashing) hashes the predicates of candidate edges, and the output value serves as the "initial node number". Its core function is to initially group candidate edges by predicate: candidate edges with the same predicate will be assigned to the same "initial node pool". The advantage of this design is that subsequent queries targeting specific predicates can directly locate the corresponding initial node pool, reducing the cross-node scanning range and improving query efficiency.

[0171] The second hash (URI hash of the main vertex): The first 16 bytes of the URI of the candidate edge's main vertex are hashed (taking the first 16 bytes balances computational efficiency and distinguishability), and the output value is used as the "sub-hash value". Its core function is to further subdivide within the initial node pool of the same predicate: by distributing the sub-hash values, candidate edges of different subjects under the same predicate are assigned to different specific nodes in the initial node pool, avoiding excessive load on a single node.

[0172] For example, if the URIs of the main vertices of candidate edges (s1, a, Person) and (s2, a, Person) are significantly different, they will be assigned to different nodes to avoid overloading a single node.

[0173] Each candidate edge received by a compute node is initialized as an independent task. Each task contains the following metadata:

[0174] The subject and object vertex URIs of the candidate edges (e.g., s1 and Person);

[0175] First query the edge predicate (e.g., predicate a) and query the attribute set NP(e) r = {birthPlace, creator};

[0176] Query attribute set NP(e) r The first query edge is associated with an "extended set of attribute constraints". In the query logic, it is necessary to further verify the associated set of attribute predicates around the vertices matched by the first query edge. These attribute predicates determine "which adjacency relationships need to be extended from the vertex of the first query edge", which is the core basis for realizing "depth traversal and constraint verification after matching the first query edge".

[0177] The current matching depth k=2 (meaning that 2 vertices have been determined after the first query edge matching);

[0178] Neighborhood prefetching instruction: Preload the birthPlace and creator neighbor vertices of the main vertex s1 and store them in the node's local memory cache.

[0179] Step 302: For candidate edges assigned to different computing nodes, construct a hash sketch vector containing query attribute constraints based on the minimum hash value of the neighborhood vertex corresponding to each query attribute of the candidate edge, and merge candidate edges with the same hash sketch vector into the same query task for sequential execution.

[0180] Specifically, the scheme described in this embodiment provides a Locality Sensitive Hash (QPMH) algorithm that integrates query attribute constraints. Its core lies in achieving intelligent task merging by quantifying the structural and attribute similarity of candidate edges, wherein:

[0181] 1) Query attribute constrained neighborhood similarity (QPNS)

[0182] Define the similarity between candidate edges e1' and e2' as its similarity in the query attribute set NP(e r The product of the proportions of the neighborhood intersections under the following conditions:

[0183]

[0184] Here, N(e', p) represents the set of neighboring vertices with attribute p in candidate edge e'. This similarity metric considers both neighborhood structure and query attribute constraints, and can accurately characterize the local associations of candidate edges.

[0185] 2) Generation of QPMH hash sketch

[0186] Based on the minimum hashing technique, a hash sketch vector containing attribute constraints is generated for each candidate edge. The specific steps are as follows:

[0187] For query attribute set NP(e) r ) = {p1, p2, ...., p m }, initialize sketch vectors ;

[0188] For each attribute p i ∈ NP(e r ), traverse the neighborhood vertices v'∈N(e', p) of candidate edge e'. i ), calculate the minimum hash value s i = min{h1(v')}, where h1 is a pseudo-random hash function;

[0189] It should be noted here that the neighborhood vertices of the candidate edge are "all vertices extended by the starting vertex of the neighborhood (such as the known vertex A) through attribute p" determined according to the query direction; the minimum hash value is calculated as "the minimum hash value of these extended vertices";

[0190] Core logic: By using the minimum hash of the neighborhood vertex set constrained by attributes, the features of candidate edges are condensed for subsequent tasks such as merging or matching.

[0191] Final sketch vector As a feature representation of candidate edges.

[0192] In this embodiment, QPNS (Query Attribute Constrained Neighborhood Similarity) is the theoretical basis for QPMH hash sketch generation, while QPMH is the engineering implementation of QPNS. The two have a collaborative relationship of "theoretical measurement → engineering implementation". Specifically, QPNS defines the quantitative standard of candidate edge "similarity", and QPMH transforms this similarity into features that can be efficiently calculated and compared through hash sketch vectors, ultimately realizing the merging of candidate edges.

[0193] Furthermore, it should be noted that:

[0194] The essence of task execution after merging: The merged candidate edges are no longer executed in parallel, but are integrated into a single task for serial processing. For example, after candidate edges e1 and e2 are merged, the task will process the neighboring vertices of e1 and e2 in sequence, but the redundant operations are reduced by sharing computing resources (such as adjacency data caching and intermediate result reuse).

[0195] The core objective of merging is to reduce redundant computations, rather than to increase parallelism.

[0196] Traditional approach: e1 and e2 are executed as independent tasks in parallel. If they share a neighboring vertex v (e.g., e1=(A,p,B), e2=(A,p,C)), then each task needs to load the adjacency data of vertex A separately, resulting in two network requests.

[0197] After merging: the task processes e1 and e2 at once, only requiring the loading of the adjacency data of vertex A once, and the calculation results can be reused for the two edges, reducing redundant operations by 50%.

[0198] Furthermore, for the candidate edges to be merged, the execution order follows the principle of maximizing shared computation. The specific strategies are as follows (it should be noted that the following four execution order strategies are not selected one by one, but rather work collaboratively according to the priority hierarchy of "core constraints → optimization objectives → dynamic adjustment". That is, the underlying core constraints are satisfied first, then optimization strategies are superimposed on them, and finally, dynamic adjustment is used to adapt to the actual execution environment. The priority is 4) > 1) > 2) > 3)):

[0199] 1) Priority sorting based on neighborhood intersection

[0200] Core logic: Prioritize edges that share the most neighboring vertices with other candidate edges to establish reusable intermediate results as early as possible.

[0201] Implementation steps:

[0202] For the candidate edge set E={e1,e2,...,e...} to be merged m }, calculate the size of the neighborhood intersection of any two edges |N(e i )∩ N(e j )|;

[0203] Construct a neighborhood intersection matrix and find the edge with the largest intersection with other edges as the starting edge e_start;

[0204] The remaining edges are sorted in descending order of "intersection size with e_start" to form the execution order [e_start, e2, e3,...].

[0205] Example: Candidate edges: e1=(A,p,B), e2=(A,p,C), e3=(D,p,E);

[0206] Neighborhood intersection: e1 and e2 share vertex A (intersection size 1), e1 and e3 do not share vertices, and e2 and e3 do not share vertices;

[0207] Execution order: e1→e2→e3 (process e1 and e2 first, then process e3).

[0208] 2) Dependency sorting based on query attribute constraints

[0209] Core logic: If the candidate edge involves the query attribute set NP (e r In the key attributes of the edge, edges containing "high constraint attributes" are processed first.

[0210] Constraint attribute priority definition:

[0211] Fixed attributes (such as rdf:type) have higher priority than variable attributes;

[0212] Properties appearing in the FILTER clause have higher priority than ordinary properties;

[0213] Attributes with low degree (corresponding to low frequency of occurrence in the RDF graph) have higher priority than attributes with high degree (to avoid processing time due to large data volume).

[0214] Priority: Fixed attributes > Attributes in the FILTER clause > Low-degree attributes > Ordinary variable attributes > High-degree attributes

[0215] Implementation steps:

[0216] For each candidate edge e, compute the NP(e) involved. r The attribute set P(e);

[0217] Edges are sorted by the sum of the priorities of the attributes in P(e), and edges with higher priorities are executed first.

[0218] 3) Adjustment of the execution order of dynamic load balancing

[0219] Core logic: Dynamically adjust the execution order of edges within the merging task based on the real-time load of the computing nodes to avoid blocking of long-running tasks.

[0220] Implementation mechanism:

[0221] The estimated processing time for each candidate edge is T(e) = number of neighboring vertices × average attribute query time;

[0222] The "Shortest Job First (SJF)" strategy is adopted, and the candidate edges are arranged in ascending order of T(e);

[0223] If the processing time for a certain edge exceeds a threshold (e.g., twice the average processing time of a node), then the candidate edge is split into an independent subtask to avoid blocking the execution of other candidate edges.

[0224] 4) Coordination between execution order and the Generic Join algorithm

[0225] The execution order of the merge tasks must be compatible with the vertex expansion order φ of the Generic Join:

[0226] If the neighboring vertices of a candidate edge to be merged contain the first k vertices of the query graph (in φ order), then these edges are processed first to ensure that the extension process of Generic Join can utilize the intermediate results of the merged edges.

[0227] Example: If the vertex matching order φ is ? v1→? v2→? v3, and the candidate edges e1=(A,p,? v1) and e2=(? v1,q,B), then the execution order should be e1→e2 to ensure that the candidate edges processed later depend on the matching results of the candidate edges processed earlier.

[0228] Furthermore, the query vertex matching order ϕ is generated based on heuristic strategies (such as vertex degree and variable constraint density) to ensure that the subgraph formed by the first k vertices is connected, so as to support the vertex-by-vertex expansion matching of the Generic Join algorithm, that is, to extract the vertex set V generated in the query graph construction step. Q (Including entity vertices and variable vertices) and edge set E Q Then, the vertex sorting is performed to generate a matching order φ. (ϕ is mainly used in conjunction with the Generic Join algorithm to perform query tasks.)

[0229] The specific processing steps for querying vertex sorting are as follows:

[0230] 1. Heuristic strategy computation

[0231] Vertex degree calculation:

[0232] Count the number of adjacent edges for each vertex in the query graph;

[0233] Example: In a graph, if vertex x is connected to 2 edges and vertex y is connected to 3 edges, then y has a higher degree.

[0234] Variable constraint density calculation:

[0235] Count the number of times the vertex of the variable appears in the FILTER clause;

[0236] Example: If the variable ?name appears in two filter conditions and ?age appears once, then ?name has a higher constraint density.

[0237] 2. Sorting process

[0238] a. Initialize the priority queue:

[0239] Sort by vertex degree in descending order; if the degrees are the same, sort by constraint density in descending order.

[0240] Example: Vertex degree is 3 > Vertex degree is 2, and when the degrees are the same, the constraint density is 2 > the constraint density is 1.

[0241] b. Generation of connected subgraphs:

[0242] Choose the head vertex of the queue as the starting point (such as the variable vertex with the highest degree).

[0243] Each time, the highest priority vertex connected to the selected vertex set is selected from the queue, until all vertices are enqueued.

[0244] Objective: To ensure that the first k vertices form a connected subgraph and avoid isolated vertices during matching.

[0245] Furthermore, for ease of understanding, the following provides an example of the vertex sorting results of the query graph:

[0246] Query graph vertices: ?x, ?y, ?z, <entitya>;

[0247] Degrees: ?y(3) > ?x(2) > ?z(1) > <entitya>(1) # The number in parentheses represents the vertex degree;

[0248] Constraint density: ?x(2) > ?y(1) # The number in parentheses indicates the constraint density;

[0249] Sorting result φ: ?y,?x,?z, <entitya>.

[0250] Furthermore, the reason for sorting the query vertices in the scheme described in this embodiment is as follows:

[0251] (1) Optimize query execution efficiency

[0252] Avoid Cartesian product: Match vertices in connected order to reduce the size of intermediate result sets.

[0253] Example: If we match ?y first and then ?x, we can use the already matched values ​​of ?y to constrain the matching range of ?x, thus avoiding unconstrained full matching.

[0254] (2) Adapting to the Generic Join algorithm

[0255] Algorithm requirements: Generic Join requires expanding vertices sequentially, with each expansion depending on the matching results of the preceding vertices.

[0256] Connectivity guarantee: Connecting the first k vertices ensures that there are usable intermediate results at each step of expansion, avoiding algorithm failure.

[0257] In practice, the vertex sorting result is used as the input order for the Generic Join algorithm;

[0258] 1) The algorithm flow is as follows:

[0259] Process vertices sequentially by φ, starting the matching process from the first vertex.

[0260] For each vertex processed, its adjacent edges are connected to the already matched result set;

[0261] Filter candidate edges using the constraints of previous vertices to narrow down the matching range;

[0262] 2) Specific examples are as follows:

[0263] Query graph: ?x -p1-> ?y, ?y -p2-> ?z, ?x -p3-> ?z;

[0264] The sorting result φ is: ?y, ?x, ?z;

[0265] The execution process is as follows:

[0266] First, match ?y to generate a candidate edge set C(?y);

[0267] Based on the matching result of ?y, match ?x (connected via edge p1);

[0268] Finally, match ?z (connected by edges p2 and p3), and filter out invalid candidates using the preceding results.

[0269] Step 303: Each computing node executes its task based on the worst-case optimal join algorithm (Generic Join algorithm is used in this embodiment). It obtains the initial match through the Scan operation and gradually constructs the complete subgraph match using the Extend / Intersect operation.

[0270] Furthermore, for ease of understanding, the Generic Join algorithm will be explained in detail below:

[0271] In traditional SPARQL query processing, the time complexity of algorithms such as nested loop joins and hash joins is usually polynomial or exponential with the data size (e.g., O(n²)). When faced with large-scale RDF graphs or queries with high variable complexity, performance crashes are likely to occur. The core advantage of worst-case optimal join algorithms (such as Generic Join) lies in:

[0272] Theoretical complexity controllability: Its time complexity is only related to the size of the query result and graph structure parameters (such as tree width), not the size of the original data. For example, the complexity of Generic Join is O(|Q|・|G|^w), where |Q| represents the relevant size measure of the query graph, usually referring to the number of vertices (nodes) in the query graph, |G| represents the relevant size measure of the data graph (i.e., the original large-scale graph data, such as the entire knowledge graph, social relationship graph, etc.), generally referring to the number of vertices (nodes) in the data graph, and w is the tree width of the query graph. When the tree width is small, the algorithm can still remain efficient even if the amount of data increases dramatically.

[0273] Robustness in extreme scenarios: In the "worst case" of uneven distribution of RDF graph data and complex query patterns (such as multi-hop associations and a large number of variable constraints), traditional algorithms may time out due to Cartesian product explosion. However, Generic Join ensures that the intermediate result set of each matching step is controllable by expanding vertex by vertex and constraining subgraph connectivity, thus avoiding performance avalanche.

[0274] Based on the above problems, the solution described in this embodiment breaks down the query processing into a "continuous task + vertex-by-vertex expansion" mode, which is highly consistent with the execution logic of the worst-case optimal join algorithm (i.e., the Generic Join algorithm), specifically reflected in:

[0275] (1) The "incremental matching" characteristic of task decomposition

[0276] Traditional methods usually try Figure 1 The global join is completed in one step. However, the scheme described in this embodiment splits the query graph into k tasks according to vertex order φ (the first k vertices form a connected subgraph). Here, k represents the vertex composition of the query graph portion corresponding to each task when splitting the tasks, corresponding to the "scan-expand / intersection" steps of Generic Join:

[0277] Scan: The first task is to obtain the initial match based on the candidate edge set corresponding to the first query edge (such as the triples that match the label (i.e., predicate) of the first query edge in the RDF graph).

[0278] Extend / Intersect: Subsequent tasks connect the previous matching results with the adjacent edges of the current vertex in φ order, gradually building a complete subgraph matching.

[0279] This decomposition method can effectively adapt to the "incremental expansion" logic of Generic Join, where each task only depends on the previous result, avoiding global computational pressure.

[0280] (2) Consistency between connectivity constraints and algorithm execution

[0281] The scheme described in this embodiment generates a matching order φ using a heuristic strategy (vertex degree, constraint density), ensuring that the first k vertices form a connected subgraph. This is consistent with the core requirement of the Generic Join algorithm (if the subgraph is not connected, the algorithm cannot deduce subsequent results through prior matching, leading to an explosion of intermediate result sets).

[0282] For example, if the first k vertices in the query graph are isolated, the expansion process cannot utilize the matched data to filter candidate edges, degenerating into an inefficient full search.

[0283] Furthermore, the Generic Join algorithm supports cross-node data transmission. When matching expansion requires access to vertex data of other nodes (e.g., the current node lacks adjacency information for the target vertex), it dynamically requests and caches neighboring vertices and their attributes via a high-speed network. For example, the adjacent vertex u' of vertex u is stored in node W. j Then node W i To W j Request the adjacency list of u' to avoid redundant data storage while ensuring matching integrity.

[0284] The Generic Join algorithm relies on the following data when processing RDF graph queries:

[0285] (1) Basic input data

[0286] Candidate edge set: Candidate edges selected by previous pruning or assignment steps (such as edges with matching attributes), each edge contains triple information (subject-verb-object, i.e.)<s, p, o> ).

[0287] Vertex adjacency list: Adjacency relationship data for each vertex, including the attributes of its outgoing and incoming edges and the vertices it connects to, used to expand the matching path.

[0288] (2) Query constraint data

[0289] Query attribute set NP (er): Predicate constraints (such as "director") specified in the user query are used to filter edges that do not conform to the attributes.

[0290] Query graph structure: The RDF graph pattern of user queries, which defines the connection relationships and attribute conditions between vertices (such as multi-hop query paths).

[0291] (3) Distributed execution auxiliary data

[0292] Node allocation information: The mapping relationship of candidate edges to each computing node through hashing or other methods (e.g., hash value modulo number of nodes).

[0293] Cross-node vertex position index: Records which computing node the vertex data is stored in, used for cross-node data requests (such as information about vertex u' being stored in computing node Wj).

[0294] The goal of the Generic Join algorithm is to generate complete query matching results, specifically including:

[0295] (1) Initial matching results

[0296] By scanning the triples that directly match the candidate edges and query attributes, initial matching items that satisfy the single-step constraints (such as matching a single edge) are generated.

[0297] (2) Expanded complete match

[0298] Based on the initial matching, a complete path conforming to the query graph structure is constructed through the expansion operation (recursively finding vertices that satisfy the attribute constraints along adjacent edges) and the intersection operation (merging the results of multiple path matching).

[0299] For example, when querying "the types of movies directed by a director", the initial match finds the director edge, which is then expanded to connect to the movie and its type edges, forming a complete triple chain.

[0300] (3) Integration of cross-node matching results

[0301] When matching expansion requires cross-node data (such as the adjacency information of vertex u' in node Wj), the data is obtained through network requests and cached, and finally integrated into a complete match on the local node, avoiding data redundancy while ensuring the integrity of the result.

[0302] In one or more implementations, step 4: Aggregate and deduplicate the query task execution results from different computing nodes to obtain the query results; specifically, the following processing steps are performed:

[0303] (1) Distributed result collection:

[0304] After each computing node completes its task, it will generate the matching results MS. i Transmitted to a central node or aggregated via a distributed protocol (such as a P2P network). The result contains a set of variable bindings (e.g., (μ = {(?x → 013), (?a → 015)}).

[0305] (2) Deduplication and verification:

[0306] The aggregated results are globally deduplicated, filtering duplicates based on unique identifiers of the matches (such as vertex URI combinations). Simultaneously, the matching is verified to ensure all constraints of the query graph (such as attribute consistency and variable scope) to guarantee accuracy.

[0307] (3) High-efficiency output:

[0308] The deduplicated results are formatted according to the SPARQL specification and returned to the user or downstream system. For large-scale result sets, paginated output or streaming is supported to improve the user experience.

[0309] like Figure 4 The image shows an example of SPARQL query processing based on parallel connections as described in this embodiment.

[0310] Furthermore, in parallel computing environments, large-scale RDF graphs require distributed processing, but the parallelization of traditional algorithms lacks theoretical support (e.g., random sharding leads to fragmented dependencies between tasks). The solution described in this embodiment decomposes the query into "parallelizable independent subtasks" (each task processes the matching extension of one vertex) through the vertex-by-vertex expansion logic of the Generic Join algorithm. At the same time, it effectively avoids the dual problems of "load imbalance + redundant computation" after parallelization by utilizing the theoretical complexity guarantee of the Generic Join algorithm.

[0311] SPARQL queries are essentially "graph pattern matching," while the core of Generic Join is to leverage the "connectivity of the graph" to progressively constrain variable binding, breaking down the query process into "continuous tasks in the order φ of matching query vertices." This ensures that each task explicitly follows the semantic relationships of the query graph (e.g., first matching the main entity, then expanding attribute relationships). Compared to the traditional approach of "treating graph queries as multi-table joins," this approach better aligns with the graph structure characteristics of RDF data, reducing invalid computations caused by semantic biases.

[0312] Purely theoretical worst-case optimal algorithms (such as strictly calculating the optimal order based on tree width) may be infeasible in engineering due to excessive computational overhead. The solution described in this embodiment generates an approximately optimal query vertex matching order φ through heuristic strategies (vertex degree, constraint density), significantly reducing computational costs while ensuring the connectivity of the first k vertices, thus achieving a balance between theoretical optimality and engineering feasibility.

[0313] For example, the time complexity of heuristic sorting is O(m log m) (where m is the number of vertices), while the time complexity of strict tree width calculation is O(2^w・n) (where w is the tree width and n is the number of data). When w is large, the former is significantly more efficient.

[0314] Assuming a SPARQL query exists:

[0315] SELECT ?x ?y ?z WHERE {

[0316] ?x <rdf:type> <dbo:person> .

[0317] ?x <foaf:knows>?and .

[0318] ?and <dbo:bornin>?With .

[0319] ?With <rdf:type> <dbo:city>.

[0320] FILTER (strlen(str(?x)) > 5)}

[0321] Traditional method: Directly perform multiple triple joins. If the number of dbo:Person entities in the RDF graph is 10^6, the initial matching set is 10^6. Subsequent joins may produce intermediate results at the 10^12 level, leading to memory overflow.

[0322] The method described in this embodiment:

[0323] The query vertex matching order φ is determined using a heuristic strategy: ?x (degree 2, including FILTER constraint) → ?y (degree 1) → ?z (degree 1);

[0324] The first task is to scan the candidate edges of ? x using Generic Join (entities with rdf:type dbo:Person and string length > 5), assuming that 10^4 results are obtained after filtering;

[0325] The subtask expands y based on the matching results of ? x (via foaf:knows edges), reducing the intermediate result set to 10^4 × average number of adjacencies (e.g., 10) = 10^5;

[0326] End-task extension? z, the final result set is controllable.

[0327] Throughout the process, the intermediate result set of each task is filtered by the preceding constraints to avoid Cartesian product explosion, and similar tasks (such as tasks that share ? x adjacent vertices) are merged through QPMH, which effectively reduces duplicate scans.

[0328] In summary, the solution described in this embodiment does not simply introduce a worst-case optimal connection algorithm, but rather deeply integrates theoretical algorithms with engineering practice through a three-layer optimization of "task decomposition + semantic hashing + dynamic balancing".

[0329] The innovations of the scheme described in this embodiment include, but are not limited to: combining the "worst-case optimal connection" characteristic of the Generic Join algorithm with parallel task decomposition for the first time, proposing an execution paradigm of "expanding vertex by vertex according to connected subgraphs", and breaking through the performance bottleneck of traditional algorithms in large-scale RDF graphs;

[0330] Technical feasibility: The solution described in this embodiment effectively solves the problems of "task association identification" and "load balancing" in the parallelization of the GenericJoin algorithm through QPMH hashing and heuristic sorting, enabling the theoretical algorithm to run efficiently on a distributed cluster.

[0331] Application value: In scenarios with high requirements for real-time performance and robustness, such as financial knowledge graph risk control query and medical knowledge graph association analysis, it can reduce redundant computing by more than 50% and cross-node communication overhead by 70% compared with existing technologies.

[0332] In one or more embodiments, corresponding to the above method, this embodiment provides a SPARQL query optimization system in a parallel computing environment, including:

[0333] The query graph conversion module is used to convert the SPARQL query to be executed into a query graph;

[0334] The candidate edge set construction module is used to generate a set of candidate edges corresponding to the first query edge based on the first query edge determined from the query graph and combined with the RDF graph to be queried; wherein each candidate edge corresponds to a triple in the RDF graph that matches the predicate of the first query edge.

[0335] The task allocation and execution module is used to allocate candidate edges in the candidate edge set to different computing nodes for parallel execution of query tasks through consistent hashing calculation. Specifically, in each computing node, the query task is executed based on the worst-case optimal connection algorithm. For candidate edges allocated to different computing nodes, a hash sketch vector containing query attribute constraints is constructed based on the minimum hash value of the neighborhood vertex corresponding to each query attribute of the candidate edge. Candidate edges with the same hash sketch vector are merged into the same query task for sequential execution.

[0336] The query results output module is used to aggregate and deduplicate the query task execution results from different computing nodes to obtain the query results.

[0337] In one or more embodiments, the following is a specific application example of a SPARQL query optimization scheme in a parallel computing environment as described in this embodiment, which specifically includes the following processing steps:

[0338] (1) System deployment and environment configuration

[0339] The solution described in this embodiment is based on a distributed cluster architecture. The hardware environment consists of eight high-performance servers, each equipped with two 12-core CPUs, 256GB of memory, and a 20TB high-speed disk, interconnected via a 100Gbps InfiniBand network to ensure low-latency data transmission and high throughput. At the software level, a self-developed parallel query engine is employed, integrating a SPARQL parser, a distributed task scheduler, and an RDF storage module. The underlying architecture is based on C++20 and leverages the process scheduling and memory management mechanisms of the Linux operating system to achieve efficient utilization of computing resources.

[0340] The RDF dataset constructs a multi-level index during the preprocessing stage: First, triples are partitioned and stored according to predicates, with each predicate corresponding to a disk partition to accelerate scanning; second, in-neighborhood and out-neighborhood indexes are built for each vertex, using a skip list structure to support fast range queries and intersection calculations. For example, the out-neighborhood index of vertex http: / / dbpedia.org / resource is grouped and stored according to predicates such as birthPlace and nationality, and vertices within each group are sorted by URI, facilitating neighborhood intersection calculations in extension operations.

[0341] (2) Implementation of the entire query processing process

[0342] 1) Query parsing and semantic analysis

[0343] SPARQL queries are converted into an abstract syntax tree by the ANTLR parser, and then the triple patterns are extracted to construct a query graph. Taking the query "find directors born in the United States and their science fiction films" as an example, the parsed query graph contains the following edges:

[0344] ?director - a - Person (First query edge, predicate 'a' represents a type assertion, which is the standard abbreviation for rdf:type)

[0345] `?director - birthPlace - "USA"` (attribute constraint)

[0346] Director - Creator - Movie (The relationship between director and film)

[0347] ?movie - a - Movie (Movie genre assertion)

[0348] ?movie - genre - "Science Fiction" (film genre constraint)

[0349] The first edge is chosen as ?director - a - Person, because it contains the variable ?director and the predicate 'a' is a high-frequency type assertion. A candidate edge set is generated by scanning all triples of type Person in the RDF graph. After the candidate set is generated, pruning is performed based on vertex degree statistics: if the birthPlace predicate does not exist in the out-neighborhood of the main vertex director of a candidate edge, the candidate edge is directly filtered to reduce invalid tasks.

[0350] 2) Task decomposition and distributed allocation

[0351] Candidate edges are distributed to 8 computing nodes using a consistent hash function. The hash function employs a hybrid hashing strategy: first, the predicate 'a' is hashed (e.g., using the FNV-1a algorithm) to obtain initial node numbers; then, the first 16 bytes of the subject vertex URI are hashed to ensure that candidate edges from different subjects under the same predicate are evenly distributed. For example, if the subject URIs of candidate edges (s1, a, Person) and (s2, a, Person) differ significantly, they will be assigned to different nodes to avoid overloading a single node.

[0352] Each compute node receives approximately 100,000 candidate edges and initializes them as an independent task. The task contains the following metadata:

[0353] The subject and object vertex URIs of the candidate edges (e.g., s1 and Person);

[0354] First query edge predicate a and query attribute set NP(e) r = {birthPlace, creator};

[0355] The current matching depth k=2 (two vertices have been determined after the first edge matching).

[0356] Neighborhood prefetching instruction: Preload the birthPlace and creator neighbor vertices of the main vertex s1 and store them in the node's local memory cache.

[0357] 3) QPMH Task Merging Optimization

[0358] After task initialization, the compute nodes perform QPMH merging on the candidate edge set. First, they compute the attribute neighborhood sketch vector for each candidate edge:

[0359] For the attribute birthPlace, extract the vertices with the predicate birthPlace from the out-neighborhood of the main vertex s, and calculate the minimum hash value (e.g., take the minimum value after applying SHA-256 hash to the vertex URI).

[0360] For the attribute creator, extract the vertices with the predicate creator in the out-neighborhood of the main vertex s, and calculate the minimum hash value.

[0361] The minimum hash value of the two attributes is combined to form a two-dimensional sketch vector (s_birthPlace, s_creator).

[0362] Candidate edges with the same sketch vector are merged into a single task group. For example, if the main vertices s1 and s2 of candidate edges e1 and e2 were both born in the United States and directed the same type of film, and their birthPlace neighborhoods both contain the vertex USA, and their creator neighborhoods both contain the vertex http: / / example.org / movie1, then their sketch vectors are identical, and they are merged into a single task group, sharing the neighborhood vertex set {USA, movie1} and subsequent extended computations. Experiments show that this merging operation can reduce the number of tasks by approximately 25% and the number of neighborhood scans by 40%.

[0363] 4) Parallel execution and load balancing

[0364] Compute nodes execute task groups based on the Generic Join algorithm:

[0365] Scanning Phase: Perform a batch scan of the first candidate edges, quickly obtain matching triples using the predicate partition index, and generate an initial 2-match set. For example, a node scans the partition file of predicate 'a', reads all triples of type 'Person', and stores them in a local memory queue.

[0366] Expansion Phase: Matches are expanded sequentially according to the query vertex matching order [?director, birthPlace, creator, ?movie,genre]. Taking the expansion of the birthPlace attribute as an example, for each "2-match" (director, Person), its birthPlace neighboring vertices are queried and matched against the query constraint "USA". The neighbor index is used to quickly locate vertices that meet the condition, generating a "3-match" (director, Person, USA). In "n-matches", "n" represents the number of vertices that have been successfully matched in the current query.

[0367] Intersection phase: When expanding the vertices of ?movie, the constraints of creator (the relationship between the director and the movie) and genre (movie genre) must be satisfied at the same time. At this time, the intersection of the creator neighborhood and the genre neighborhood is taken to ensure that only movie vertices that satisfy both conditions are retained.

[0368] Load balancing is achieved through a dynamic task stealing mechanism: the master node periodically collects the task completion rate and CPU utilization of each node. If a node's task completion rate is below average and its CPU utilization is above 80%, 20% of the unexecuted tasks are stolen from the task queue and distributed to nodes with lower loads. For example, if node W0 experiences excessive load due to processing high-frequency predicate tasks, the master node migrates some of its tasks to node W7, reducing the difference in task completion rates between the two from 30% to 5%, and shortening the overall query time by 18%.

[0369] 5) Results aggregation and quality assurance

[0370] Each compute node pushes matching results to the master node in real time during task execution, employing a streaming aggregation model to avoid memory overflow. The master node maintains a global deduplication hash table, calculates a hash value for each matching result, and uses a Bloom filter to quickly eliminate duplicates, ensuring that the final result is free of redundancy. For queries containing ORDER BY clauses, the master node collects the pre-sorted results from each node, merges them using a merge sort algorithm, and supports pagination to handle large-scale result sets.

[0371] During the result verification phase, semantic validation is performed on the matching results: for example, checking whether the ?movie vertex simultaneously satisfies the Movie type assertion and the Science Fiction type constraint, avoiding invalid matches caused by outdated neighborhood data or transmission errors. Experiments show that this validation mechanism can control the result error rate to below 0.03%.

[0372] (3) Performance optimization and implementation effect

[0373] Neighborhood caching optimization: Compute nodes maintain a cache of the 200 most recently accessed neighboring vertices for each task group, using an LRU eviction policy. The cache hit rate reaches 68% on the YAGO2 dataset, reducing the number of cross-node data requests and decreasing the average latency of expansion operations from 12ms to 4.5ms.

[0374] Vectorized computation: The neighborhood intersection operation is processed in parallel using the AVX2 instruction set. The hash value of the vertex URI is stored as a 256-bit vector. Bitwise operations are used to accelerate the set intersection computation, reducing the processing time for the intersection of 100,000 vertex neighborhoods from 89ms to 32ms.

[0375] Compressed transmission: Neighborhood data transmitted across nodes is compressed using Snappy, with an average compression ratio of 3:1, reducing the transmission time of 1GB of neighborhood data from 210ms to 70ms and improving network bandwidth utilization by 40%.

[0376] On the LUBM 300M triple dataset, the solution described in this embodiment achieves a response time of 4.2 seconds for typical five-variable queries, a 53% improvement over the gStore distributed version (8.9 seconds). On the YAGO2 dataset, when processing complex queries containing nested subqueries, the response time is reduced from 12.7 seconds to 6.1 seconds, with redundant computation reduced by 48%. When the cluster is expanded to 16 nodes, the throughput increases linearly to 14.2 times that of the single-machine version, verifying the scalability of the solution.

[0377] (4) Implementation points and patent value

[0378] The core implementation points of this plan include:

[0379] Semantic-driven task decomposition: The selection of the first edge and the pruning of the candidate set should be closely integrated with the query semantics, giving priority to edges with strong constraints and few variables to reduce the initial matching space.

[0380] QPMH parameter tuning: The choice of hash function (e.g., xxHash64 is better than MD5) and the dimension of the sketch vector (dynamically adjusted according to the number of query attributes) directly affect the task merging efficiency, and the optimal configuration needs to be determined through experiments.

[0381] Load balancing granularity: The frequency and amount of task stealing need to be balanced between network overhead and load balancing effect. It is recommended to trigger once every 5 seconds, and the amount of stealing in a single instance should not exceed 20% of the node's task queue.

[0382] In further embodiments, the following is also provided:

[0383] An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor. When executed by the processor, the computer instructions perform the methods described in the above embodiments. For brevity, further details are omitted here.

[0384] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0385] Memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of memory may also include non-volatile random access memory. For example, memory may also store information about the device type.

[0386] A computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the methods described in the above embodiments.

[0387] The methods described in the above embodiments can be directly executed by a hardware processor, or executed by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are omitted here.

[0388] Those skilled in the art will recognize that the units, i.e., algorithm steps, of the various examples described in connection with this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0389] The above description is merely a preferred embodiment of this disclosure and is not intended to limit this disclosure. Various modifications and variations can be made to this disclosure by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.< / dbo:city> < / rdf:type> < / dbo:bornin> < / foaf:knows> < / dbo:person> < / rdf:type> < / entitya> < / entitya> < / entitya> < / dbo:birthplace> < / foaf:name> < / foaf:knows> < / dbr:london> < / dbo:birthplace> < / ex:alice> < / foaf:name> < / ex:alice> < / ex:bob> < / foaf:knows> < / ex:alice> < / dbr:london> < / dbo:birthplace> < / ex:alice> < / foaf:name> < / ex:alice> < / ex:bob> < / foaf:knows> < / ex:alice> < / ex:alice> < / ex:charlie> < / foaf:knows> < / ex:alice> < / ex:bob> < / foaf:knows> < / ex:alice> < / foaf:age> < / ex:bob> < / ex:charlie> < / foaf:knows> < / ex:alice> < / ex:bob> < / foaf:knows> < / ex:alice> < / foaf:knows> < / dbo:city> < / dbpedia:berlin> < / rdf:type> < / rdf:type> < / dbo:city> < / rdf:type> < / dbo:capitalof> < / rdf:type> < / dbpedia:berlin> < / city> < / rdf:type>

Claims

1. A SPARQL query optimization method in a parallel computing environment, characterized in that, include: Convert the SPARQL query to be executed into a query graph; Based on the first query edge determined from the query graph, and combined with the RDF graph to be queried, a set of candidate edges corresponding to the first query edge is generated; wherein, each candidate edge corresponds to a triple in the RDF graph that matches the predicate of the first query edge. For candidate edges in the candidate edge set, they are distributed to different computing nodes for parallel execution of query tasks through consistent hashing. In each computing node, the query task is executed based on the worst-case optimal connection algorithm. For candidate edges distributed to different computing nodes, a hash sketch vector containing query attribute constraints is constructed based on the minimum hash value of the neighborhood vertices corresponding to each query attribute. Candidate edges with the same hash sketch vector are merged into the same query task and executed sequentially. The query results from different computing nodes are aggregated and deduplicated to obtain the query results; The execution of the query task based on the worst-case optimal join algorithm in each computing node specifically involves: generating the query vertex matching order corresponding to the query graph through a heuristic strategy; splitting the query process into consecutive query tasks based on the query vertex matching order; and executing the query tasks using the Generic Join algorithm to obtain the query results. The construction of the hash sketch vector containing query attribute constraints is specifically as follows: Get the query attribute set of the first query edge; For each attribute in the query attribute set, iterate through the neighboring vertices of the current candidate edge and calculate the minimum hash value of the neighboring vertices; Based on the minimum hash value corresponding to the neighboring vertices of the current candidate edge under each attribute, obtain the hash sketch vector representation of the current candidate edge.

2. The SPARQL query optimization method in a parallel computing environment as described in claim 1, characterized in that, The process of merging candidate edges with the same hash sketch vector into the same query task for sequential execution involves integrating multiple candidate edges into a single task for serial processing.

3. The SPARQL query optimization method in a parallel computing environment as described in claim 1, characterized in that, The consistent hashing adopts a hybrid hashing strategy, specifically: first, the predicates in the candidate edges are hashed to obtain the initial computation node number; then, the main vertex URIs in the candidate edges are hashed to obtain the subdivided hash values. The target computing node for candidate edges is determined based on the combination of the initial computing node number and the subdivision hash value.

4. A SPARQL query optimization system for a parallel computing environment, characterized in that, include: The query graph conversion module is used to convert the SPARQL query to be executed into a query graph; The candidate edge set construction module is used to generate a set of candidate edges corresponding to the first query edge based on the first query edge determined from the query graph and combined with the RDF graph to be queried; wherein each candidate edge corresponds to a triple in the RDF graph that matches the predicate of the first query edge. The task allocation and execution module is used to allocate candidate edges in the candidate edge set to different computing nodes for parallel execution of query tasks through consistent hashing calculation. Specifically, in each computing node, the query task is executed based on the worst-case optimal connection algorithm. For candidate edges allocated to different computing nodes, a hash sketch vector containing query attribute constraints is constructed based on the minimum hash value of the neighborhood vertices corresponding to each query attribute of the candidate edge. Candidate edges with the same hash sketch vector are merged into the same query task for sequential execution. The query results output module is used to aggregate and deduplicate the query task execution results from different computing nodes to obtain the query results. The execution of the query task based on the worst-case optimal join algorithm in each computing node specifically involves: generating the query vertex matching order corresponding to the query graph through a heuristic strategy; splitting the query process into consecutive query tasks based on the query vertex matching order; and executing the query tasks using the Generic Join algorithm to obtain the query results. The construction of the hash sketch vector containing query attribute constraints is specifically as follows: Get the query attribute set of the first query edge; For each attribute in the query attribute set, iterate through the neighboring vertices of the current candidate edge and calculate the minimum hash value of the neighboring vertices; Based on the minimum hash value corresponding to the neighboring vertices of the current candidate edge under each attribute, obtain the hash sketch vector representation of the current candidate edge.

5. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and running thereon, characterized in that, When the processor executes the program, it implements the SPARQL query optimization method in a parallel computing environment as described in any one of claims 1-3.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the SPARQL query optimization method in a parallel computing environment as described in any one of claims 1-3.

Images