Distributed knowledge graph query method, device and storage medium
Knowledge graph query in a distributed environment through the BFS access mode, and parallel processing and global connection technology are used to solve the problems of low efficiency of large-scale data query and insufficient parallel processing in the existing technology, and efficient knowledge graph query is achieved.
Patent Information
- Application Number
- CN202111440843.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-30
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2041-11-30
AI Technical Summary
The existing knowledge graph query methods are inefficient in processing large-scale data and are not processed in full in parallel, especially in the merge stage, where a large amount of data transfer overhead and logical associations cannot be effectively utilized, resulting in a decrease in query efficiency.
The access mode of BFS is used to perform distributed knowledge graph query, and the target query statement is linearly divided through the first device and allocated to multiple second devices for parallel search and filtering and matching, combining with global connections to improve query efficiency.
In the case of communication delay overhead, through sufficient parallel connection and global connection, the efficiency of knowledge graph query is significantly improved, and the scale of results to be merged and the cost of computing communication is reduced.
Smart Images

Figure CN114153987B_ABST
Abstract
Description
[Technical field]
[0001] The present application relates to the query field, and in particular to a distributed knowledge graph query method, device and storage medium. [Background technology]
[0002] The query demand of knowledge graph has important practical significance. The existing knowledge retrieval method focuses on keyword query, which often lacks important structural information in semantics. For example, to search for "people whose hometown is Fujian and who work in Hunan", the query is in the form of keywords "hometown", "Fujian", "Hunan", "work", "personnel", etc. The loss of the relationship between different keywords introduces inevitable ambiguity. For example, the above keyword set can also be understood as "people whose hometown is Hunan and who work in Fujian", which causes actual errors. The query of knowledge graph not only focuses on the keyword content, but also on the logical association and interaction between different keywords.
[0003] With the continuous development of the big data era, the scale of knowledge data that needs to be processed is huge. The single-machine model is difficult to handle the query of large-scale knowledge graph data, so a distributed cluster is needed to improve the processing capacity. The core of the existing methods is to divide the large-scale graph data into multiple parts, distribute each part to different machines for processing, and finally merge the results. The problems with these methods are two points: ① The result of the knowledge graph merger is huge, far exceeding the final result itself, and the distributed merger stage will bring a lot of data transmission overhead; ② Parallelism is limited to each machine, and good parallelism is not achieved in the merger stage. [Summary of the invention]
[0004] The present application provides a distributed knowledge graph query method, device and storage medium, which allows the second device to be fully connected in parallel through the BFS access mode while the first device is globally connected, so that the connection process is sufficient, thereby maintaining sufficiently high query efficiency in the presence of communication delay overhead.
[0005] The first aspect of the present application provides a distributed knowledge graph query method, comprising:
[0006] The first device obtains a target query sentence input by a user;
[0007] The first device processes the target query statement to obtain a target query graph;
[0008] The first device linearly partitions the target query graph to obtain N query subtrees, where N is an integer greater than or equal to 2;
[0009] The first device allocates the N query subtrees to multiple second devices, so that the second devices retrieve and filter from a database according to a target query subtree to obtain a target candidate result set, and obtain corresponding matching candidate result sets from multiple second devices corresponding to the first device according to the target candidate result set, and match the target candidate result set with the matching candidate result set to obtain a pairing result, and return the pairing result, wherein the second device is any second device among the multiple second devices, and the target query subtree is a query subtree allocated to the second device;
[0010] The first device merges and connects the pairing results corresponding to each second device in the plurality of second devices to obtain a final result set.
[0011] The second aspect of the present application provides a distributed knowledge graph query method, including:
[0012] The second device receives a target query subtree assigned by the first device, where the target query subtree is any query subtree among N query subtrees obtained by linearly segmenting the target query graph by the first device, and the target query graph is obtained by the first device by processing the target query statement input by the user, where N is an integer greater than or equal to 2;
[0013] The second device performs retrieval and filtering from a database according to the target query subtree to obtain a target candidate result set;
[0014] The second device obtains a corresponding matching candidate result set from a plurality of second devices corresponding to the first device according to the target candidate result set;
[0015] The second device matches the target candidate result set with the matching candidate result set to obtain a pairing result;
[0016] The second device sends the pairing result to the first device, so that the first device merges and connects the pairing results obtained from the plurality of second devices to obtain a final result set.
[0017] A third aspect of the present application provides a device, which is a first device, including:
[0018] A receiving unit, used to obtain a target query sentence input by a user;
[0019] A processing unit, used to process the target query statement to obtain a target query graph;
[0020] A segmentation unit, configured to linearly segment the target query graph to obtain N query subtrees, where N is an integer greater than or equal to 2;
[0021] an allocating unit, configured to allocate the N query subtrees to a plurality of second devices, so that the second devices perform retrieval and filtering from a database according to a target query subtree to obtain a target candidate result set, and obtain a corresponding matching candidate result set from a plurality of second devices corresponding to the first device according to the target candidate result set, and match the target candidate result set with the matching candidate result set to obtain a pairing result, and return the pairing result, wherein the second device is any second device among the plurality of second devices, and the target query subtree is a query subtree allocated to the second device;
[0022] The merging unit is used to merge and connect the pairing results corresponding to each second device in the multiple second devices to obtain a final result set.
[0023] A fourth aspect of the present application provides a device, which is a second device, including:
[0024] A receiving unit, configured to receive a target query subtree assigned by a first device, wherein the target query subtree is any query subtree among N query subtrees obtained by linearly segmenting a target query graph by the first device, and the target query graph is obtained by the first device by processing a target query statement input by a user, wherein N is an integer greater than or equal to 2;
[0025] A retrieval unit, used to perform retrieval and filtering from a database according to the target query subtree to obtain a target candidate result set;
[0026] an acquisition unit, configured to acquire, according to the target candidate result set, a corresponding matching candidate result set from a plurality of second devices corresponding to the first device;
[0027] A matching unit, used for matching the target candidate result set with the matching candidate result set to obtain a pairing result;
[0028] A sending unit is used to send the pairing result to the first device, so that the first device merges and connects the pairing results obtained from the multiple second devices to obtain a final result set.
[0029] A fifth aspect of an embodiment of the present application provides a computer device comprising at least one connected processor, a memory, and a transceiver, wherein the memory is used to store program code, and the processor is used to call the program code in the memory to execute the steps of the distributed knowledge graph query method described in any of the above aspects.
[0030] A fourth aspect of an embodiment of the present application provides a computer storage medium, which includes instructions, which, when executed on a computer, enables the computer to execute the steps of the distributed knowledge graph query method described in any of the above aspects.
[0031] Compared with the related art, in the embodiment provided by the present application, the access mode of BFS is used to allow the second device to be fully connected in parallel while the first device is globally connected, so that the connection process is sufficient, thereby maintaining sufficiently high query efficiency even in the presence of communication delay overhead.
Brief Description of the Drawings
[0032] Figure 1 A schematic diagram of RDF data provided in an embodiment of the present application;
[0033] Figure 2 A schematic diagram of a query subgraph provided in an embodiment of the present application;
[0034] Figure 3 A schematic diagram of an embodiment of a distributed knowledge graph query method provided in an embodiment of the present application;
[0035] Figure 4 A schematic diagram of the MapReduce process for ID allocation provided in an embodiment of the present application;
[0036] Figure 5 A schematic diagram of a process for generating a vertex base code according to an embodiment of the present application;
[0037] Figure 6 A schematic diagram of one-step neighbor environment characteristics provided in an embodiment of the present application;
[0038] Figure 7 A schematic diagram of the two-step neighbor environment characteristics provided in an embodiment of the present application;
[0039] Figure 8 A schematic diagram of constructing a bitmap tree by distributed hierarchical clustering provided in an embodiment of the present application;
[0040] Fig. 9 A schematic diagram of a process flow of a distributed knowledge graph query method provided in an embodiment of the present application;
[0041] Fig.10 Another schematic diagram of the process of the distributed knowledge graph query method provided in the embodiment of the present application;
[0042] Fig.11 Another schematic diagram of the process of the distributed knowledge graph query method provided for the implementation of this application;
[0043] Fig.12 Another schematic diagram of the process of the distributed knowledge graph query method provided in the embodiment of the present application;
[0044] Fig.13 A schematic diagram of a virtual structure of a first device provided in an embodiment of the present application;
[0045] Fig.14 A schematic diagram of a virtual structure of a second device provided in an embodiment of the present application;
[0046] Fig.15 A schematic diagram of the hardware structure of the server provided in an embodiment of the present application. [Specific implementation method]
[0047] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.
[0048] With the development of the Internet, a huge amount of data has emerged rapidly and is complex. In this context, a model that can more flexibly express large amounts of complex data, the Resource Description Framework (RDF), came into being. The RDF data model was proposed by W3C. It provides a unified standard to represent things, that is, using a set of triples to describe resources on the Web and the relationship between resources. The triple is the basic unit of knowledge representation in the knowledge graph, and each piece of knowledge constitutes the knowledge graph, which drives the management of knowledge in the form of a graph and realizes knowledge data modeling, storage, and query mining. Subsequently, W3C defined the SPARQL query language for RDF datasets to further retrieve and operate data stored in RDF format.
[0049] In recent years, various query systems on RDF have emerged, such as gStore, SW-Store, RDF-3X, etc. However, as the amount of RDF data increases, many existing SPARQL query systems for RDF data are mostly stand-alone algorithms. When faced with larger-scale RDF data, these algorithms inevitably have problems such as low time efficiency and poor spatial scalability.
[0050] In view of this, the present application provides a distributed knowledge graph query method and related equipment, which integrates the topological structure. For the vertices that meet the matching requirements, their neighbor topology must completely cover the query topology. Therefore, the topological coding is integrated into the query point coding. When performing knowledge graph queries, candidate solutions that do not meet the results can be filtered out more efficiently, reducing the scale of results to be merged and reducing computing and communication costs.
[0051] The following is an explanation of the terms and concepts involved in this application, as follows:
[0052] RDF is often referred to as a "language", but more accurately, it is a data model introduced by the W3C as part of the Semantic Web movement. RDF consists of a series of statements, namely: subject, predicate and object, where the subject is an entity or category, the predicate is the attribute of the corresponding entity or category, and the object can be an entity or category, or a string. In this way, RDF data is a series of triples. RDF data can also be organized through a graph structure. If the entities in the statements are regarded as vertices and the attributes are regarded as directed edges connecting two vertices (this application refers to the Literal edge of the corresponding entity when the object is a string), then the RDF data set becomes a directed graph, that is, it can be like Figure 1 In the RDF data graph shown in , 101 Shen, 102 Hunan, 103 Zhang, 104 Writer and 105 City are vertices, the attribute between 101 Shen and 102 Hunan is the birthplace, and the attribute between 101 Shen and 104 Writer is the occupation, which means "Shen's birthplace is Hunan and his occupation is writer".
[0053] SPARQL, short for SPARQL Protocol and RDF Query Language, is a query language and protocol designed by the W3C RDF Data Working Group for querying RDF data. It is a query statement similar to Structured Query Language (SQL), and contains four query formats, namely: SELECT (returns variable values that match the query pattern), CONSTRUCT (returns an RDF graph), ASK (queries whether there are results that match the pattern), and DESCRIBE (returns an RDF graph that describes the matching resources). Among them, SPARQL queries can be converted into subgraph queries to enhance the ability to express data requirements. Suppose the query requirement is: "Who are the PhDs who work in X company and are from Anhui?" Existing query understanding methods often only extract keywords from the query, while the subgraph method is more accurate, such as Figure 2 As shown, 201 is the SPARQL query statement corresponding to "Who are the PhDs working in X Company and whose hometown is Anhui?", and 202 is the SAPRQL query graph corresponding to the SPARQL query statement. The query graph can understand that the query target is a vertex, and the vertex has three associated edges, which are the employment relationship to "X Company", the hometown location relationship to "Anhui", and the academic relationship to "doctoral degree".
[0054] Distributed processing consists of two parts: an offline part and an online part. The offline part constructs an index in a distributed manner based on the MapReduce computing framework on Hadoop and stores it in the distributed index Hbase; the online part accesses Hbase in parallel through the C parallel library message passing interface (MPI) to perform candidate filtering and parallel merging.
[0055] Hadoop is an open source distributed computing framework from Apache. It uses a simple computing model to perform cluster computing on large-scale data. Hadoop is built on the distributed file system HDFS. Its data management is done through the distributed file system HDFS, which is very suitable for applications on large-scale data sets. MapReduce highly abstracts the complex parallel computing process running on large-scale clusters into two functions: Map and Reuduce, and adopts a "divide and conquer" strategy.
[0056] HBase is an open source distributed database system of Apache. It is a column-oriented distributed database system developed on HDFS. It organizes read and write in the form of Key-Value and is suitable for real-time random read and write of ultra-large-scale data sets. HBase has good scalability. It is built from the bottom up and can achieve linear expansion of space by adding nodes. It can also manage ultra-large-scale sparse tables well on cheap hardware.
[0057] MPI is an information transmission application program interface, including protocols and semantic descriptions, which can indicate how to use its characteristics in various implementations. The goal of MPI is high performance, large scale and portability. In view of the program framework of MPI's message communication, in addition to the initialization and termination configuration functions, the most core functions of MPI are the sending (MPI_Send) and receiving (MPI_Recv) functions. These two functions also have four different modes to optimize for different application situations, so that they can meet most parallel needs and give developers very flexible choices.
[0058] See also Figure 3 , Figure 3A schematic diagram of an embodiment of a distributed knowledge graph query method provided for an embodiment of the present application, wherein, after receiving a SPARQL query statement 302 input by a user, the first device 301 processes the SPARQL query statement 302 to obtain a corresponding query graph, and linearly splits the query graph to obtain multiple query subgraphs, and then distributes the multiple query subgraphs to the second device 303. The second device 303 obtains a candidate result set by retrieving part of the query subgraphs loaded into itself from an open source distributed database system 304. The second device 303 connects the edges of the access query graph through the access order of BFS to obtain a one-step connection result, and sends the one-step connection result to the first device 301. The first device 301 performs a global connection on the one-step connection result to obtain a final result set 305, and feeds the final result set 305 back to the user.
[0059] It should be noted that the distributed knowledge graph query method in this application includes two modules, an offline module and an online module. The offline module involves the construction of an index, including the encoding of point-edge information, the encoding of neighbor topology, and the hierarchical clustering of the global encoding set based on Hadoop's MapReduce calculation to form an index of the encoding BitSet tree structure; in the online module, the first device allocates the bitmap tree corresponding to the query statement to the second device according to the user's query statement, and the second device performs query filtering from the database according to the allocated bitmap tree and returns the query results. The first device displays the query results to the user for viewing, wherein the first device is the master node in the distributed architecture, and the second device is the slave node in the distributed architecture.
[0060] First, the encoding of the vertex edge information and the encoding of the neighbor topology in the offline module are explained as follows:
[0061] Step A1: Obtain the RDF dataset to be encoded;
[0062] Step A2: deduplicate the RDF dataset to obtain the target RDF dataset.
[0063] In this embodiment, the first device may first deduplicate the RDF data set to be encoded by using the sorting capability of Hadoop's MapReduce. Of course, other methods may also be used to deduplicate the result set to be constructed, without any specific limitation, as long as the result to be constructed can be deduplicated.
[0064] Step A3: Process each RDF data in the target RDF data set through MapReduce to obtain the entity, edge and graph ID corresponding to each RDF data.
[0065] In this embodiment, the MapReduce Job for the graph ID allocation of the RDF data set by the first device is mainly completed in two stages, namely the Map stage and the Reduce stage. Figure 4 , taking the subject object as an example, first read the input RDF data set 401 (here taking the subject as an example), then start the MapReduce Job, and then enter the map phase: the first device divides the RDF data set to be processed into multiple segments 402 (such as Figure 2 split0, split1 and split2 in the above example), and assign multiple fragments 402 to multiple map tasks 403, the number of splits corresponding to the number of map tasks, Figure 4 The example of dividing into three segments is used to illustrate that each segment is assigned a map task, and each subsequent call to the map function will result in 0 or more<subID,subject> The intermediate result of the form 404, such as Figure 4 As shown, each map task processes the fragments assigned to it to obtain its own intermediate results; then enters the sort stage 405, reads the intermediate results generated in the map stage and sorts them to separate the fragments with the same subID.<subID,subject> The pairs are merged into a subID and a corresponding subject list, that is,<subID,subject list> ; Then enter the reduce phase, in the reduce phase each reducetask406 will get multiple<subID,subject list> Data, then a single reduce task for each<subID,subject list> The data calls the reduce function, and 0 or more<subID,subject> 407 pairs are output as the result. Finally, all the outputs of each reduce task, namely entities, edges and corresponding IDs, will be saved in HBase.
[0066] Step A4: Encode the entity corresponding to each RDF data to obtain the initial basic code corresponding to each RDF data.
[0067] In this embodiment, the edge corresponding to the RDF triple is marked with predict, and the weight is mapped one by one to a fixed-length bitset through predict coding as the initial basic coding of the edge, which is specifically encoded using the BKDR string hash function.
[0068] Since a pair of entities may exist in multiple RDF triples and their predictions are different, when encoding the target entity (the target entity is any entity corresponding to each RDF data), the basic codes of multiple edges from subject to object corresponding to the target entity are ORed as the final subject to object edge basic code, and then the edge basic codes of all adjacent edges of the target entity and the bitset of the string code of the neighbor node corresponding to the target entity are ORed, and the bitset obtained after ORing is used as the initial basic code of the target entity.
[0069] Since the basic coding calculation of the triples is completely independent, the MapReduce computing framework used in this application is very suitable for processing the calculation of the basic coding of the corresponding entity based on the triples. Figure 5 As shown, by splitting the RDF data set and performing the corresponding initial basic encoding on each triple encoding, and then aggregating the basic encodings of the same entity to the Reduce node for OR operation, the final initial basic encoding of the entity is obtained.
[0070] Step A5: Determine the neighbor topology code of each RDF data according to the edge corresponding to each RDF data and the initial basic code of the entity corresponding to each RDF data.
[0071] In this embodiment, the neighbor topology encoding of the entity corresponding to each RDF data consists of two sections: a one-step topology section and a two-step topology section. The one-step topology section and the two-step topology section are described below respectively:
[0072] The one-step topological segment records the distribution of the vertex (that is, the entity) one-step neighbor environment feature (the feature in which the longest simple path length from any non-central vertex to the central vertex is 1, which can be obtained by Figure 6The label topology structure is derived from the 9 topological structures in the , where derivation means labeling each vertex). The frequency of the one-step neighbor environment feature can be represented by the frequency of the vertex labels in the vertex's outgoing, incoming, and double neighbors. Therefore, there is no need to store the combination and frequency of neighbor vertex labels, but the neighbor vertex labels and frequencies can be directly stored. The k_1step labels with the highest frequency are selected from all vertex labels in the vertex's ingoing, outgoing, and double neighbors, and these labels and corresponding frequencies are encoded to obtain a one-step topological segment: in_cnt(out_cnt, bi_cnt) represents the number of labels of the incoming neighbors (outgoing neighbors, double neighbors) in the encoded vertex label, labelIDi represents the label identifier, cnti represents the frequency, and the encoding is in_cnt<labelIDi,cnti> …out_cnt<labelIDi,cnti> …bi_cnt<labelIDi,cnti> ….
[0073] The two-step topological segment records the distribution of the two-step neighbor environment features of the vertex (the features contain features with a simple path length of 2 from the non-central vertex to the central vertex, which can be obtained by Figure 7 The label topology derived from the 24 topological structures in the feature. The definition of the two paths in the feature is as follows: if there is only one simple path starting from the central vertex and the length is 2 in the feature, then the two paths are this path; if there is more than one such path in the feature, then there must be only two such simple paths in the feature. Figure 2 In the representation, it is represented as two paths starting from the central vertex, one in the clockwise direction and the other in the counterclockwise direction. The second path is defined as the one in the counterclockwise direction. The feature type coding kind_code is encoded as follows: For a two-step neighbor environment feature, code is used to represent its topological structure after removing the vertex label. Figure 2 In the number, labelID1 represents the vertex label of the first non-central vertex of the two paths, and labelID2 represents the label of the second non-central vertex of the two paths. Then its kind code kind_code is<code,labelID1,labelID2> Select the k_2step features with the highest frequency from all the two-step neighbor environment features of the vertex, encode these features and the corresponding frequencies to obtain the two-step topological segment: use kind_codei to represent the feature type code, cnti to represent the frequency, then the code is<kind_codei,cnti> ….
[0074] Thus, the basic coding of the entity corresponding to each result and the neighbor topological coding of the entity corresponding to each result can be obtained.
[0075] The following describes how to construct an index of the encoding BitSet tree structure by hierarchical clustering of the global encoding set based on Hadoop's MapReduce calculation.
[0076] Compared with the current top-down BitSet-Tree bitmap tree construction algorithm through classification, it is difficult to construct it in a distributed manner due to the organizational structure. Therefore, when the amount of data is relatively large, constructing the BitSet-Tree on a single machine will face challenges in terms of space. The distributed BitSet-Tree construction algorithm proposed in this application first performs Canopy clustering on the initial basic code to determine the initial cluster center value, and then uses K-means clustering to take the cluster center obtained by each cluster as the parent node of the current cluster, thereby realizing the bottom-up construction of the BitSet-Tree, such as Figure 8 As shown, the core of the bitmap tree construction algorithm in this application is to implement Canopy clustering and K-means clustering of the initial basic codes through MapReduce, and group the more similar initial basic codes together, as follows:
[0077] Step B1: cluster the entities corresponding to each RDF data to obtain a bitmap tree BitSet-Tree corresponding to the target RDF data set.
[0078] The following describes how to construct the BitSet-Tree corresponding to the target RDF dataset by clustering:
[0079] Step B11: Determine K entities randomly selected from the entities corresponding to the target RDF data set as initial cluster centers.
[0080] In this embodiment, the algorithm for constructing a distributed BitSet-Tree takes entity as a vertex, and the Hamming distance between the initial basic codes corresponding to the entity is the distance between the two vertices. The selection of the value of K is described below:
[0081] The value of K is defined according to the size N of the vertex set and the maximum number m of child nodes of each node of the BitSet-Tree to be constructed: K = [N / m]. The initial K points are selected through Canopy clustering.
[0082] Step B12: Cluster the i-th layer of the BitSet-Tree according to the initial basic code of each entity in the K entities and the neighbor topology code of each entity to obtain Ki cluster centers. The BitSet-Tree includes L layers, L is an integer greater than or equal to 2, i is an integer greater than or equal to 0, and i is less than L.
[0083] Step B13, determine the graph vertices of the i-th layer using Ki centers;
[0084] Step B14: Clustering is performed based on the initial basic code of the entity corresponding to each vertex of the K(i+1) vertices randomly selected from the i-th layer and the neighbor topological code of the entity corresponding to each vertex to obtain the K(i+1) cluster centers of the i+1th layer of the BitSet-Tree.
[0085] Step B15: Perform an OR operation on all the initial basic codes of the clusters corresponding to the vertices in each layer of the BitSet-Tree to obtain the final basic codes corresponding to the vertices in each layer of the BitSet-Tree.
[0086] In this embodiment, assuming that the number of layers of the final BitSet-Tree is L, the bottom layer is marked as layer 0, and the top layer is layer L-1, then a total of (L-1) K-means algorithms need to be executed, and the Ki centers obtained by the i-th clustering will become all the vertices of the i-th layer, that is, the input vertices of the i+1-th clustering. When constructing from the i-th layer to the i+1-th layer, first randomly find K(i+1) vertices from the vertices of the i-th layer, and then obtain the final K(i+1) centers through multiple iterations as all the vertices of the i+1-th layer.
[0087] In each clustering process, each map task matches the input vertex with the center closest to the input vertex (the distance between two vertices is obtained by calculating the Hamming distance between the basic codes of the two vertices), takes the center as the key and the current input vertex as the value output, and then clusters the vertices corresponding to the same center into the same Reduce to form a cluster, and finds a new center in the cluster. The new center will have a new basic code, that is, the final basic code obtained by ORing all the initial basic codes of the current cluster.
[0088] It should be noted that although the size of each cluster is not easy to control, the size of each node of BitSet-Tree is limited for query efficiency. Therefore, when the clusters gathered by Reduce exceed a certain size, they can be split by hierarchical clustering. Since the number of clusters and the upper limit of the size of a single cluster in hierarchical clustering are easy to control, it can solve the problem of K-means cluster size being difficult to control. The final hierarchical clustering results will be distributed and stored in HBase.
[0089] Step B2: constructing the index corresponding to each result in the target result set in a distributed manner.
[0090] In this embodiment, the construction of the edge table plays a vital role in the filtering in the algorithm index part and the connection in the query part, and is also very critical to the reading efficiency of Hbase. Since HBase caches a whole row of data in the data table each time, based on this feature, the edge table structure of the index part is designed as a series of<entity,OutEdgeList,InEdgeList> The OutEdgeList edge list corresponds to the entity's out-edge list, the InEdgeList variable list corresponds to the entity's in-edge list, and stores the final basic encoding of the corresponding edges.
[0091] The construction of the edge table only requires one MapReduce Job. For outgoing edges, only the subID of the six-tuple is used as the key, and the pair consisting of the corresponding objID and the final basic code of the edge obtained by the current six-tuple encoding is used as the value. That is, all the outgoing edges of the entity are gathered to the same Reduce node, and the same is true for the incoming edges. The construction of the edge table except the bottom layer depends on the upper and lower relationship of the nodes in the BitSet-Tree. If two nodes in the 0th layer of the BitSet-Tree have an edge, the corresponding weight is also added between the parent nodes of the two nodes. The specific implementation is that in the Map task, in addition to outputting the outgoing and incoming edges of the current entity, the outgoing and incoming edges of the parent node corresponding to the current entity are also output, and all output pairs are marked with the corresponding level, so that the outgoing and incoming edges of all entities at each level of the BitSet-Tree are obtained in the Reduce task. After the same edges in the outgoing edge set and the incoming edge set associated with the entity are ORed at the Reduce node, they are loaded into HBase according to the HBase storage structure of the above edge table.
[0092] The following is an explanation of the online module of the distributed knowledge graph query method from the perspective of the first device. Please refer to Fig. 9 , Fig. 9 A schematic flow chart of a distributed knowledge graph query method provided in an embodiment of the present application includes:
[0093] 901. The first device obtains a target query statement input by a user.
[0094] In this embodiment, the first device can obtain the target query statement input by the user. The specific user can enter the target query statement in the display interface corresponding to the first device according to his own needs, and then the first device can receive the query statement. For example, the user's query requirement is "Who are the doctors who work in X company and whose hometown is Anhui?", then the SPARQL query statement corresponding to the target is:
[0095] ?x1<Education Background>“Doctor”
[0096] ?x1<Work Unit><the Central Committee and State Council>
[0097] ?x1 <hometown><Anhui Province>
[0098] 902. The first device processes the target query statement to obtain a target query graph.
[0099] In this embodiment, after receiving the target query statement input by the user, the first device may process the target query statement to obtain a target query graph corresponding to the target query statement, that is, after receiving the target query statement input by the user, the first device may first convert the target query statement into a SPARQL query statement, and then organize and encode the SPARQL query statement in the form of a bitmap to obtain a target query graph. The following is a specific description:
[0100] The first device processes the target query statement to obtain a target query graph including:
[0101] The first device processes the target query statement through MapReduce to obtain entities and edges corresponding to the target query statement;
[0102] The first device encodes the entity corresponding to the target query statement to obtain an initial basic code of the entity corresponding to the target query statement;
[0103] The first device determines a neighbor topology code of an entity corresponding to the target query statement according to an edge corresponding to the target query statement;
[0104] The first device constructs the target query graph according to the initial basic coding and the neighbor topology coding.
[0105] It should be noted that the method of processing the target query statement to obtain the target query graph is similar to that of the offline module, which has been described in detail above and will not be repeated here.
[0106] 903. The first device linearly partitions the target query graph to obtain N query subtrees.
[0107] 904. The first device distributes N query subtrees to multiple second devices, so that the second devices perform retrieval and filtering according to the target query subtree to obtain a target candidate result set, and obtain corresponding matching candidate result sets from the multiple second devices corresponding to the first device according to the target candidate result set, and match the target candidate result set with the matching candidate result set to obtain a pairing result, and return the pairing result.
[0108] In this embodiment, at the beginning of query processing, the first device distributes the target query graph BitSet-Tree to multiple second devices according to a certain distribution strategy. If the established BitSet-Tree contains 10 subtrees and there are 5 second devices as the second devices in the distributed cluster, then each second device will be assigned two subtrees for parallel processing according to the standard specification of the message passing function library (Massage Passing Interface, MPI), obtain a candidate result set, and determine the pairing result based on the candidate result set, and finally return the pairing result, so that the retrieval of the candidate result set in the query processing part can be carried out quickly and in parallel. The decomposition of the task by the query algorithm is the distribution of tree branches, which further illustrates that the SPARQL query in this application is load-balanced and has good scalability.
[0109] It should be noted that, for the sake of space expansion, each second device only loads a certain proportion of the BitSet-Tree, and then the BitSet-Tree needs to be linearly split. In this application, all nodes of a certain layer of the BitSet-Tree are split, and the subset obtained by the split and all subtrees with it as the root will be used as part of the BitSet-Tree split, where all subtrees will be evenly loaded into the memory of each second device after the split. The number of nodes in the split layer should be more than the number of second devices, which can ensure full utilization of computing resources, and it needs to be the highest layer among all layers that meet this condition, so that the higher the indexed subtree, the better the pruning effect, and thus a better retrieval and filtering effect. In addition to the subtree of the BitSet-Tree, the pre-loaded data also loads the one-step neighbors associated with all nodes in the loaded subtree in the current second device, which is used to pre-filter the candidate result set. The pre-filtering is after the retrieval and before the Shuffle. After the BitSet-Tree is split, each second device will load all the edges associated with the nodes in the subtree of the BitSet-Tree split to itself.
[0110] 905. The first device merges and connects the pairing results corresponding to each second device in the plurality of second devices to obtain a final result set.
[0111] In this embodiment, the first device merges and joins the pairing results corresponding to each device to obtain the final result set. The first device merges and joins the candidate result sets corresponding to each second device to mainly achieve parallel processing in two dimensions: first, the communication between different machines in the cluster is parallel, which is achieved through the access structure of breadth first search (Breath First Search, BFS); second, each second device performs a one-step connection, and at the same time, the first device receives the one-step connection results sent by the second device and performs a global merge connection to obtain the final result set. The parallel processing of these two dimensions greatly improves the processing time.
[0112] It should be noted that the second device chooses BFS access mainly because under the BFS access structure, all outgoing edges of the same vertex can be processed in parallel. The currently accessed second device sends its own candidate result set to all visited neighboring nodes. After receiving, all neighbors will perform a one-step connection in parallel, that is, pair the candidate result sets of the two candidate variables to determine whether there is an edge, and the edge basis coding can cover the edge basis coding between the two vertices of the target query graph. Candidate results that do not match are filtered out.
[0113] The codes run by all second devices under the MPI framework are consistent, and all second devices will run BFS, but only the second device associated with the currently accessed edge will perform the corresponding sending, receiving, and one-step connection. The global connection after the one-step connection will be processed by the first device, that is, each second device sends its own one-step connection result to the first device, and the first device saves the intermediate result, which can not only reduce the communication overhead of multiple transmissions of the connection intermediate result between each second device, but also enable the second device to perform a one-step connection in BFS while the first device performs a global connection, thereby executing in parallel and improving time efficiency. The final query result is fed back to the user by the first device.
[0114] In the BFS process, when a second device accesses an edge of a query graph, if the candidate result sets corresponding to the exit and entry points of the current edge are not local, the second device does not communicate with other second devices. The execution time of this code is almost negligible, so it can be assumed that each second device will run fast enough to access an edge related to itself. Therefore, in addition to the parallel processing of all edges of the same access point, the edges without associated vertices in BFS are also processed in parallel, so the overall query parallelism is very sufficient. Fig.10 As shown, when the second device performs a one-step connection, the first device will also integrate the one-step connection results at the same time, and in the one-step connection between the second devices, the three outgoing edges of x1 are parallel, and the second device 5 will also execute the processing of the edges related to it: (x2, x5) at an almost negligible time cost. Therefore, the parallelism is high enough, which can improve the efficiency of the query and reduce the query time.
[0115] In summary, in the embodiments provided in the present application, the BFS access mode is used to allow the second device to be fully connected in parallel while the first device is globally connected, so that the connection process is sufficient, thereby maintaining sufficiently high query efficiency even in the presence of communication delay overhead.
[0116] See also Fig.11 , Fig.11 Another schematic diagram of a distributed knowledge graph query method provided in an embodiment of the present application includes:
[0117] 1101. The second device receives a target query subtree allocated by the first device.
[0118] In this embodiment, when the user has a query requirement, the query requirement can be input on the display interface corresponding to the first device. After receiving the query requirement, the first device can process the target query statement corresponding to the query requirement to obtain a target query graph, and linearly split the target query graph to obtain N query subtrees, where N is an integer greater than or equal to 2, and the target query subtree is allocated to the second device.
[0119] It should be noted that the construction of the target query graph and the linear segmentation have been described in detail in the above steps 901 to 903, and the details will not be repeated here.
[0120] 1102. The second device performs retrieval and filtering from the database according to the target query subtree to obtain a target candidate result set.
[0121] In this embodiment, after receiving the target query subtree assigned by the second device, the second device can retrieve and filter from the database according to the target query subtree to obtain the target candidate result set. That is, the second device filters the node neighbors in the database that do not match the first candidate variable to obtain the initial candidate result set, and the first candidate variable is the candidate variable corresponding to any layer in the target query subtree; then, if there is a second candidate variable corresponding to the Literal edge in the candidate variable corresponding to the initial candidate result set, the second device filters the initial candidate result set according to the Literal edge to obtain the target candidate result set. The following is a specific explanation:
[0122] When the second device searches the target query subtree, each time a new layer is retrieved, it can filter through the one-step neighbors of the candidate nodes in the current layer. The objects of the filter are all nodes in the node neighbors that do not match the candidate variables corresponding to the current layer (specifically, the matching is performed according to the final basic code of each node). For example, x1 and x2 are two candidate variables with edges in the query subtree. Both have their own corresponding final basic codes after encoding. For a candidate node node1 of x1, if none of the neighbors of node1 has a final basic code that can cover the basic code of x2, then node1 obviously will not match the final result set, so it is filtered; the execution is repeated to finally obtain the initial candidate result set;
[0123] When the second device retrieves the bottom layer of the target query subtree, it will obtain the candidate result set of each candidate variable, and the candidate result sets of these candidate variables have been filtered by the first-step neighbor. After that, the second device will further filter the initial candidate result set, that is, Literal filtering. The edges in the query subtree are not only the edges between variables, but also Literal edges (the Literal edge refers to the case where one end of the edge is a string). At this time, the second device only needs to check whether there is a corresponding Literal edge for each candidate variable to meet the Literal edge constraint while satisfying the edge constraint of the candidate variable. If the Literal edge corresponds to an entity, the ID of the corresponding entity is queried and the corresponding first-step neighbor is found in the candidate point to find out whether it is the entity. If the Literal edge corresponds to a string, the Literal edge is encoded into a basic code of an edge, and whether there is an edge covering the final basic code in the candidate point edge is found. If the above corresponding edges do not exist in the candidate point, the candidate point is also filtered to finally obtain the target candidate result set.
[0124] It should be noted that this part is called Literal filtering. Obviously, Literal filtering is performed in parallel by each second device. The filtering of one-step neighbors satisfies the constraints between candidate variables, and Literal filtering satisfies the requirements of Literal edges of candidate variables in the query graph. The filtering of the overall basic code matching is to fully filter the final connection through the basic code index.
[0125] 1103. The second device obtains a corresponding matching candidate result set from multiple second devices corresponding to the first device according to the target candidate result set.
[0126] In this embodiment, after obtaining the target candidate result set, the second device can send the first candidate variable distribution corresponding to the target candidate result set to the first device, so that the first device returns the second candidate variable distribution corresponding to the candidate result sets of other second devices except the second device among the multiple second devices, and determines the node where the candidate variable that is the same as the candidate variable corresponding to the target candidate result set is located based on the second candidate variable distribution, and obtains the candidate result of the candidate variable from the node, thereby obtaining a matching candidate result set.
[0127] It should be noted that after the retrieval and filtering, the state of the multiple second devices corresponding to the first device is that each second device may contain multiple candidate result sets of candidate variables, and the candidate result set of each candidate variable may be distributed on multiple second devices. In this state, connecting will bring a lot of communication overhead, because the endpoints of the same edge may be distributed on multiple second devices. In the connection phase, when the first device processes a certain edge, multiple sources will send candidate result sets to multiple targets, which will be a lot of communication overhead. Therefore, the second device will perform the Shuffle process after the retrieval and filtering, that is, the candidate result sets of the same candidate variables are gathered into the same second device, and the candidate result sets of different candidate variables are gathered into different second devices. In this way, for a pair of connected candidate variables in the connection phase, only one candidate result set needs to be transmitted. The Shuffle process is mainly divided into two parts. In the first part, all the second devices tell the first device which candidate result sets of candidate variables they contain. After the first device collects the global candidate result set distribution, it feeds back the candidate result set distribution to all second devices. MPI has special functions for this operation, namely MPI_gather and MPI_Bcast, and it is optimized in efficiency. In the second part, all nodes send or receive candidate result sets according to their own candidate result sets, and finally complete the Shuffle to obtain the target candidate result set.
[0128] 1104. The second device matches the target candidate result set with the matching candidate result set to determine a pairing result.
[0129] In this embodiment, after the second device determines the same variables in the candidate variables corresponding to the target candidate result set from other second devices and obtains the corresponding matching candidate result set, it can match the target candidate result set with the matching candidate result set to obtain a pairing result. That is, the second device pairs the candidate result sets of two candidate variables (the candidate result sets of the same candidate variables obtained from the target candidate result set and the matching candidate result set, respectively): whether there is an edge and the basic coding of the edge can cover the basic coding of the edge between the two points of the query graph (because the basic coding mapping of predict is one-to-one, so the coverage of the edge can ensure the existence of the corresponding query graph edge), the candidate points that do not match are filtered, and the ones retained are the pairing results.
[0130] 1105. The second device sends the pairing result to the first device, so that the first device merges and connects the pairing results received from multiple second devices to obtain a final result set.
[0131] The merge join has been described in detail in the above step 905, and will not be described in detail here.
[0132] In summary, it can be seen that in the embodiment provided by the present application, the access mode of BFS allows the second device to be fully connected in parallel while the first device is globally connected, so that the connection process is sufficient, thereby maintaining sufficiently high query efficiency even in the presence of communication delay overhead.
[0133] The distributed knowledge graph query method is described above from the perspectives of the first device and the second device respectively. The distributed knowledge graph query method is described below from the perspective of the interaction between the first device and the second device.
[0134] See also Fig.12 , Fig.12 Another schematic diagram of a distributed knowledge graph query method provided in an embodiment of the present application includes:
[0135] 1201. The first device obtains a target query statement input by a user.
[0136] 1202. The first device processes the target query statement to obtain a target query graph.
[0137] 1203. The first device linearly partitions the target query graph to obtain N query subtrees.
[0138] 1204. The first device distributes N query subtrees to multiple second devices.
[0139] It should be noted that steps 1201 to 1204 are Fig. 9 Steps 901 to 904 in the above are similar. Fig. 9 The details have been explained in detail in the article, so I will not repeat them here.
[0140] 1205. The second device performs retrieval and filtering from the database according to the target query subtree to obtain a target candidate result set.
[0141] 1206. The second device obtains a corresponding matching candidate result set from multiple second devices corresponding to the first device according to the target candidate result set.
[0142] 1207. The second device matches the target candidate result set with the matching candidate result set to determine a pairing result.
[0143] 1208. The second device sends the pairing result to the first device.
[0144] It should be noted that steps 1205 to 1208 are Fig.10 Steps 1002 to 1005 in the above are similar. Fig.10 The details have been explained in detail in the article, so I will not repeat them here.
[0145] 1209. The first device merges and connects the pairing results corresponding to each second device in the multiple second devices to obtain a final result set.
[0146] Step 1209 and Fig. 9 Similar to step 905 in the above Fig. 9 The details have been explained in detail in the article, so I will not repeat them here.
[0147] In summary, in the embodiments provided in the present application, the BFS access mode is used to allow the second device to be fully connected in parallel while the first device is globally connected, so that the connection process is sufficient, thereby maintaining sufficiently high query efficiency even in the presence of communication delay overhead.
[0148] The above describes the present application from the perspective of a distributed knowledge graph query method, and the following describes the present application from the perspective of a first device and a second device.
[0149] See also Fig.13 , Fig.13 A virtual structure diagram of a first device provided in an embodiment of the present application, wherein the first device 1300 includes:
[0150] The receiving unit 1301 is used to obtain a target query sentence input by a user;
[0151] A processing unit 1302 is used to process the target query statement to obtain a target query graph;
[0152] A segmentation unit 1303 is used to perform linear segmentation on the target query graph to obtain N query subtrees, where N is an integer greater than or equal to 2;
[0153] An allocating unit 1304 is used to allocate the N query subtrees to multiple second devices, so that the second devices perform retrieval and filtering from a database according to a target query subtree to obtain a target candidate result set, and obtain a corresponding matching candidate result set from multiple second devices corresponding to the first device according to the target candidate result set, and match the target candidate result set with the matching candidate result set to obtain a pairing result, and return the pairing result, wherein the second device is any second device among the multiple second devices, and the target query subtree is a query subtree allocated to the second device;
[0154] The merging unit 1305 is configured to merge and connect the pairing results corresponding to each second device in the plurality of second devices to obtain a final result set.
[0155] In a possible design, the processing unit 1302 is specifically configured to:
[0156] Processing the target query statement through MapReduce to obtain entities and edges corresponding to the target query statement;
[0157] Encoding the entity corresponding to the target query statement to obtain an initial basic code of the entity corresponding to the target query statement;
[0158] The first device determines a neighbor topology code of an entity corresponding to the target query statement according to an edge corresponding to the target query statement;
[0159] The target query graph is constructed according to the initial basic coding and the neighbor topology coding.
[0160] In a possible design, the processing unit 1302 constructs the target query graph according to the initial basic code and the neighbor topology code, including:
[0161] Step 1: The first device randomly selects K entities from the entities corresponding to the target query statement as initial cluster centers, where K is an integer greater than or equal to 2;
[0162] Step 2: The first device clusters the i-th layer according to the initial basic code of each entity in the K entities and the neighbor topology code of each entity to obtain Ki cluster centers, where i is an integer greater than or equal to 1, and i is less than the number of layers of the target query graph;
[0163] Step 3, the first device determines the graph vertices of the i-th layer with the Ki centers;
[0164] Step 4, the first device performs clustering according to the initial basic code of the entity corresponding to each vertex of the K(i+1) vertices randomly selected from the i-th layer and the neighbor topological code of the entity corresponding to each vertex, so as to obtain the K(i+1) cluster centers of the i+1th layer of the target query graph;
[0165] Step 5: The first device determines the K(i+1) cluster centers as the graph vertices of the i+1th layer;
[0166] Step 4 and step 5 are repeated until the graph vertices of all layers in the target query graph are determined.
[0167] In a possible design, the processing unit 1302 is further configured to:
[0168] The first device obtains an initial basic code of an entity corresponding to a target vertex, the target vertex is any cluster in a target layer, and the target layer is any layer in the target query graph;
[0169] The first device performs an OR operation on an initial basic code of an entity corresponding to the target vertex to obtain a final basic code of the target vertex.
[0170] In a possible design, the processing unit 1302 is further configured to:
[0171] The first device obtains an RDF dataset;
[0172] The first device deduplicates the RDF dataset to obtain a target RDF dataset.
[0173] The first device processes each RDF data in the target RDF data set through MapReduce to obtain an entity, an edge, and a graph identifier corresponding to each RDF data;
[0174] The first device encodes the entity corresponding to each RDF data to obtain an initial basic code of the entity corresponding to each RDF data;
[0175] The first device determines a neighbor topology code of each RDF data according to an edge corresponding to each RDF data and an initial basic code of an entity corresponding to each RDF data;
[0176] The first device clusters according to the basic code of the entity corresponding to each RDF data and the neighbor topology code of the entity corresponding to each RDF data to construct a bitmap tree corresponding to each RDF data;
[0177] The first device constructs an index corresponding to the entity corresponding to each RDF data in a distributed manner;
[0178] The first device stores the index, the bitmap tree, and the graph identifier in association with each other.
[0179] See also Fig.14 , Fig.14 A virtual structure diagram of a second device provided in an embodiment of the present application, wherein the second device 1400 includes:
[0180] A receiving unit 1401 is configured to receive a target query subtree allocated by a first device, wherein the target query subtree is any query subtree among N query subtrees obtained by linearly segmenting a target query graph by the first device, and the target query graph is obtained by the first device by processing a target query statement input by a user, wherein N is an integer greater than or equal to 2;
[0181] A retrieval unit 1402 is used to perform retrieval and filtering from a database according to the target query subtree to obtain a target candidate result set;
[0182] An acquiring unit 1403 is configured to acquire, according to the target candidate result set, a corresponding matching candidate result set from a plurality of second devices corresponding to the first device;
[0183] A matching unit 1404 is used to match the target candidate result set with the matching candidate result set to obtain a pairing result;
[0184] The sending unit 1405 is configured to send the pairing result to the first device, so that the first device merges and connects the pairing results obtained from the plurality of second devices to obtain a final result set.
[0185] In a possible design, the retrieval unit 1402 is specifically used for:
[0186] The second device filters node neighbors in the database that do not match the first candidate variable to obtain an initial candidate result set, where the first candidate variable is a candidate variable corresponding to any layer in the target query subtree;
[0187] If the candidate variables corresponding to the initial candidate result set include a second candidate variable corresponding to the Literal edge, the second device filters the initial candidate result set according to the Literal edge to obtain a target candidate result set.
[0188] Fig.15 This is a schematic diagram of the structure of the application server, such as Fig.15 As shown, the server 1500 of this embodiment includes at least one processor 1501, at least one network interface 1504 or other user interface 1503, a memory 1505, and at least one communication bus 1502. The server 1500 optionally includes a user interface 1503, including a display, a keyboard or a clicking device. The memory 1505 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 1505 stores execution instructions. When the server 1500 is running, the processor 1501 communicates with the memory 1505, and the processor 1501 calls the instructions stored in the memory 1505 to execute the above-mentioned distributed knowledge graph query method. The operating system 1506 includes various programs for implementing various basic businesses and processing tasks based on hardware.
[0189] The server provided in the embodiment of the present application can execute the technical solution of the embodiment of the above-mentioned distributed knowledge graph query method. Its implementation principle and technical effects are similar and will not be repeated here.
[0190] An embodiment of the present application also provides a computer-readable medium comprising computer execution instructions, which enable a server to execute the distributed knowledge graph query method described in the above embodiment. The implementation principle and technical effects are similar and will not be repeated here.
[0191] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.
[0192] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.< / hometown>
Claims
1. A distributed knowledge graph query method, characterized in that: include: The first device obtains a target query sentence input by a user; The first device processes the target query statement to obtain a target query graph; The first device linearly partitions the target query graph to obtain N query subtrees, where N is an integer greater than or equal to 2; The first device allocates the N query subtrees to multiple second devices, so that the second devices retrieve and filter from a database according to a target query subtree to obtain a target candidate result set, and obtain corresponding matching candidate result sets from multiple second devices corresponding to the first device according to the target candidate result set, and match the target candidate result set with the matching candidate result set to obtain a pairing result, and return the pairing result, wherein the second device is any second device among the multiple second devices, and the target query subtree is a query subtree allocated to the second device; The first device merges and connects the pairing results corresponding to each second device in the plurality of second devices to obtain a final result set, The first device processes the target query statement to obtain a target query graph including: The first device processes the target query statement through MapReduce to obtain entities and edges corresponding to the target query statement; The first device encodes the entity corresponding to the target query statement to obtain an initial basic code of the entity corresponding to the target query statement; The first device determines a neighbor topology code of an entity corresponding to the target query statement according to an edge corresponding to the target query statement; The first device constructs the target query graph according to the initial basic coding and the neighbor topology coding.
2. The method according to claim 1, characterized in that The first device constructs the target query graph according to the initial basic code and the neighbor topology code, including: Step 1: The first device randomly selects K entities from the entities corresponding to the target query statement as initial cluster centers, where K is an integer greater than or equal to 2; Step 2: The first device clusters the i-th layer according to the initial basic code of each entity in the K entities and the neighbor topology code of each entity to obtain Ki cluster centers, where i is an integer greater than or equal to 1, and i is less than the number of layers of the target query graph; Step 3, the first device determines the graph vertices of the i-th layer with the Ki centers; Step 4, the first device performs clustering according to the initial basic code of the entity corresponding to each vertex of the K(i+1) vertices randomly selected from the i-th layer and the neighbor topological code of the entity corresponding to each vertex, so as to obtain the K(i+1) cluster centers of the i+1th layer of the target query graph; Step 5: The first device determines the K(i+1) cluster centers as the graph vertices of the i+1th layer; Step 4 and step 5 are repeated until the graph vertices of all layers in the target query graph are determined.
3. The method according to claim 2, characterized in that The method further comprises: The first device obtains an initial basic code of an entity corresponding to a target vertex, the target vertex is any cluster in a target layer, and the target layer is any layer in the target query graph; The first device performs an OR operation on an initial basic code of an entity corresponding to the target vertex to obtain a final basic code of the target vertex.
4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: The first device obtains an RDF dataset; The first device deduplicates the RDF dataset to obtain a target RDF dataset; The first device processes each RDF data in the target RDF data set through MapReduce to obtain an entity, an edge, and a graph identifier corresponding to each RDF data; The first device encodes the entity corresponding to each RDF data to obtain an initial basic code of the entity corresponding to each RDF data; The first device determines a neighbor topology code of each RDF data according to an edge corresponding to each RDF data and an initial basic code of an entity corresponding to each RDF data; The first device clusters according to the basic code of the entity corresponding to each RDF data and the neighbor topology code of the entity corresponding to each RDF data to construct a bitmap tree corresponding to each RDF data; The first device constructs an index corresponding to the entity corresponding to each RDF data in a distributed manner; The first device stores the index, the bitmap tree, and the graph identifier in association with each other.
5. A distributed knowledge graph query method, characterized in that: include: The second device receives a target query subtree assigned by the first device, where the target query subtree is any query subtree among N query subtrees obtained by linearly segmenting the target query graph by the first device, and the target query graph is obtained by the first device by processing the target query statement input by the user, where N is an integer greater than or equal to 2; The second device performs retrieval and filtering from a database according to the target query subtree to obtain a target candidate result set; The second device obtains a corresponding matching candidate result set from a plurality of second devices corresponding to the first device according to the target candidate result set; The second device matches the target candidate result set with the matching candidate result set to obtain a pairing result; The second device sends the pairing result to the first device, so that the first device merges and connects the pairing results obtained from the plurality of second devices to obtain a final result set. The first device processes the target query statement to obtain the target query graph, which includes: The first device processes the target query statement through MapReduce to obtain entities and edges corresponding to the target query statement; The first device encodes the entity corresponding to the target query statement to obtain an initial basic code of the entity corresponding to the target query statement; The first device determines a neighbor topology code of an entity corresponding to the target query statement according to an edge corresponding to the target query statement; The first device constructs the target query graph according to the initial basic coding and the neighbor topology coding.
6. The method according to claim 5, characterized in that The second device retrieves and filters from the database according to the target query subtree to obtain a target candidate result set including: The second device filters node neighbors in the database that do not match the first candidate variable to obtain an initial candidate result set, where the first candidate variable is a candidate variable corresponding to any layer in the target query subtree; If the candidate variables corresponding to the initial candidate result set include a second candidate variable corresponding to the Literal edge, the second device filters the initial candidate result set according to the Literal edge to obtain a target candidate result set.
7. A device, characterized in that: The device is a first device, comprising: A receiving unit, used to obtain a target query sentence input by a user; A processing unit, used to process the target query statement to obtain a target query graph; A segmentation unit, configured to linearly segment the target query graph to obtain N query subtrees, where N is an integer greater than or equal to 2; an allocating unit, configured to allocate the N query subtrees to a plurality of second devices, so that the second devices perform retrieval and filtering from a database according to a target query subtree to obtain a target candidate result set, and obtain a corresponding matching candidate result set from a plurality of second devices corresponding to the first device according to the target candidate result set, and match the target candidate result set with the matching candidate result set to obtain a pairing result, and return the pairing result, wherein the second device is any second device among the plurality of second devices, and the target query subtree is a query subtree allocated to the second device; a merging unit, configured to merge and connect the pairing results corresponding to each second device in the plurality of second devices to obtain a final result set, Wherein, the processing unit is specifically used for: Processing the target query statement through MapReduce to obtain entities and edges corresponding to the target query statement; Encoding the entity corresponding to the target query statement to obtain an initial basic code of the entity corresponding to the target query statement; The first device determines a neighbor topology code of an entity corresponding to the target query statement according to an edge corresponding to the target query statement; The target query graph is constructed according to the initial basic coding and the neighbor topology coding.
8. A device, characterized in that: The device is a second device, comprising: A receiving unit, configured to receive a target query subtree assigned by a first device, wherein the target query subtree is any query subtree among N query subtrees obtained by linearly segmenting a target query graph by the first device, and the target query graph is obtained by the first device by processing a target query statement input by a user, wherein N is an integer greater than or equal to 2; A retrieval unit, used to perform retrieval and filtering from a database according to the target query subtree to obtain a target candidate result set; an acquisition unit, configured to acquire, according to the target candidate result set, a corresponding matching candidate result set from a plurality of second devices corresponding to the first device; A matching unit, used for matching the target candidate result set with the matching candidate result set to obtain a pairing result; a sending unit, configured to send the pairing result to the first device, so that the first device merges and connects the pairing results obtained from the plurality of second devices to obtain a final result set, The first device processes the target query statement to obtain the target query graph, which includes: The first device processes the target query statement through MapReduce to obtain entities and edges corresponding to the target query statement; The first device encodes the entity corresponding to the target query statement to obtain an initial basic code of the entity corresponding to the target query statement; The first device determines a neighbor topology code of an entity corresponding to the target query statement according to an edge corresponding to the target query statement; The first device constructs the target query graph according to the initial basic coding and the neighbor topology coding.
9. A computer storage medium, characterized in that include: Instructions, when the instructions are executed on a computer, enable the computer to execute the distributed knowledge graph query method described in any one of claims 1 to 6.
Citation Information
Patent Citations
Real-time query method and system for large-scale knowledge map under memory constraint
CN109033314A
Distributed storage and visual query processing method oriented to large-scale financial knowledge graph
CN113157943A
