Knowledge graph path reasoning method and system based on distributed database system

Abstract

The present invention belongs to the field of knowledge graph technology, and relates to a knowledge graph path reasoning method, system and application based on a distributed database system. The present method is a method for realizing knowledge graph path reasoning based on a GreenPlum distributed database system and a Madlib environment. The present method uses a GreenPlum distributed database system as a storage system for knowledge graph data, and constructs a feature matrix with real physical meaning through text classification, unified similarity relationships, machine learning and logistic regression of graph relationships, thereby realizing parallel search of all paths and reasoning analysis of deep path relationships. As a new knowledge graph path reasoning method compatible with big data and graph computing, the present method can realize reasoning analysis of deep path relationships of knowledge graphs. After verification by a data set, the present method can realize efficient calculation and improve the accuracy of path reasoning.

Description

Technical Field

[0001] The present invention belongs to the field of knowledge graph technology, and in particular relates to a knowledge graph path reasoning method, system and application based on a distributed database system. Background Art

[0002] With the development of big data and artificial intelligence technology, the idea of ​​the Internet of Everything has been gradually applied in many fields, and knowledge graph technology has emerged. Knowledge graph is essentially a semantic network that reveals the relationship between entities. It is composed of pieces of knowledge, each of which is represented by a SPO triple (Subject-Predicate-Object). In other words, the knowledge graph is a data set with a graph structure. In terms of logical structure, it is divided into two levels: the model layer and the data layer. The data layer is mainly composed of a series of facts, and knowledge will be stored in units of facts. At present, most facts are stored in the form of triples (es, r, eo), where es and eo represent the subject and object entities respectively, and r represents the relationship between them, such as (entity 1, relationship, entity 2), (entity, attribute, attribute value), etc. Generally speaking, the facts stored in the knowledge graph are usually incomplete. With the development and update of the knowledge system, the knowledge graph needs to be supplemented and improved. However, it is time-consuming and labor-intensive to manually complete the knowledge graph, and it is difficult to achieve. Therefore, it is necessary to develop an algorithm for automatically completing the knowledge graph.

[0003] In earlier path reasoning schemes, the knowledge graph can be represented as a third-order binary tensor, in which each element represents a triple, 1 represents a true triple, and 0 represents an unknown triple (or error or missing). Therefore, many completion models based on tensor decomposition have been proposed. PRA (Path-Ranking Algorithm) path reasoning uses tensor decomposition for knowledge reasoning and proposes a theoretical method. In practice, it often uses path random walks and third-order tensor calculations to generate third-order matrices. However, in actual data, there may be a large number of complex relationship scenarios that can only be reached by multi-level paths. Therefore, the existing PRA scheme is difficult to have practical significance and can only remain at the theoretical level. In this context, how to develop a new reasoning method for knowledge graph path reasoning to achieve efficient calculations and improve the accuracy of path reasoning has become an urgent problem to be solved in the field of knowledge graph technology. Summary of the invention

[0004] In order to solve the problem that the existing knowledge graph path reasoning scheme has unsatisfactory implementation effect, the present invention proposes a set of solutions.

[0005] Terminology explanation:

[0006] GreenPlum: A commonly used distributed database system with a Postgresql cluster as the underlying layer.

[0007] Triple: An organizational form of graph data, including vertices, relationship directions, and respective attributes.

[0008] Full path: the sum of all relationship paths between source data and target data in the atlas library sample.

[0009] Madlib: An AI engine based on GreenPlum. This application uses Madlib to perform full path search, and uses it to perform logistic regression training and reasoning on the final feature data matrix.

[0010] NLP (Natural Language Processing) analysis: Natural language processing methods are a general term for all technologies related to computer processing of natural language. Its purpose is to enable computers to understand and accept instructions input by humans in natural language and complete the translation function from one language to another. In this application, it is used to merge and unify similar relationships and realize the calculation of relationship similarity.

[0011] Elimination of abnormal and rare relationships: Set some common rules to eliminate abnormal interference paths and improve the credibility of training features.

[0012] Existing samples in the graph library: all the relationship data in the existing knowledge graph library (such as a person’s multi-level relationship path).

[0013] Samples with a known dependent variable: It is known that a person is either a writer or not a writer.

[0014] The purpose of the present invention is to provide a knowledge graph path reasoning method based on a distributed database system. The method is a method for realizing knowledge graph path reasoning based on the GreenPlum distributed database system and the Madlib environment. The method constructs a feature data matrix with real physical meaning through text classification, unified similarity relationship, machine learning and logistic regression of graph relationships, and realizes parallel search of full paths and reasoning analysis of deep path relationships. As a knowledge graph path reasoning method compatible with big data and graph computing, the method can realize reasoning analysis of deep path relationships of knowledge graphs.

[0015] In a first aspect, the present invention provides a knowledge graph path reasoning method based on a distributed database system, the method comprising:

[0016] Determine the source data and target data, enumerate all reachable relationship paths from the source data to the target data, record each relationship path data to form an indefinite length array A[n], and perform NLP analysis on the elements in the indefinite length array A[n];

[0017] Select a comparison sample in the atlas library, enumerate all reachable relationship paths between each data in the comparison sample and other data in the comparison sample, record each relationship path data to form an indefinite length array B[n], compare and calculate the path similarity of each A[n] and B[n], and obtain the calculation result;

[0018] Using the calculation results obtained in the previous step to form a data matrix, the data matrix is ​​combined with the known dependent variables to construct a training data table;

[0019] Use Madlib to perform logistic regression training on the training data table constructed in the previous step to obtain a trained model;

[0020] Record the relationship path data to be predicted to form an indefinite length array C[n], compare and calculate the path similarity of each A[n] and C[n], obtain the calculation results and form a data matrix to be predicted, combine the data matrix to be predicted with the known dependent variables to construct a training data table to be predicted, and use the trained model in the previous step to perform logistic regression prediction on the training data table to be predicted, that is, obtain the probability that the relationship between the data to be predicted from the source data to the target data is established.

[0021] According to some embodiments of the present invention, the elements in the variable-length array A[n] are the relationship contents contained in each relationship path between the source data and the target data.

[0022] According to some embodiments of the present invention, the NLP analysis of the elements in the indefinite-length array A[n] includes unifying similar relationship categories (such as: father, dad; write, work, etc.) and eliminating abnormal relationships and very few relationships.

[0023] According to some embodiments of the present invention, the NLP analysis includes text classification, text analysis, semantic analysis, sentiment analysis, and intent recognition.

[0024] Furthermore, the semantic analysis includes lexical analysis, syntactic analysis, pragmatic analysis, and contextual analysis.

[0025] According to some embodiments of the present invention, the comparison samples in the atlas library are samples of known dependent variables already in the atlas library.

[0026] According to some embodiments of the present invention, the path similarity calculation index of A[n] and B[n] includes the text distance and cosine distance between the two.

[0027] According to some embodiments of the present invention, the training data table is stored in a GreenPlum distributed database, the training data table contains complete trainable features, and the training data table and field naming are standardized during the storage process of the training data table.

[0028] In a second aspect, the present invention provides a knowledge graph path reasoning system based on a distributed database system, the system comprising:

[0029] Recording module: used to record various relationship path data and form an array of indefinite length;

[0030] NLP analysis module: used to perform NLP analysis on elements in variable-length arrays;

[0031] Path similarity calculation module: used to calculate the path similarity between various indefinite length arrays;

[0032] Data matrix construction module: used to construct the path similarity calculation results between various indefinite length arrays into a data matrix;

[0033] Training data table construction module: used to combine the data matrix with the known dependent variables to construct a training data table;

[0034] Madlib training module: used to perform logistic regression training on the training data table and obtain the trained model;

[0035] Relationship prediction module: used to perform logistic regression prediction on the training data table to be predicted through the trained model, and obtain the probability that the relationship between the source data and the target data to be predicted is established.

[0036] In a third aspect, the present invention provides a knowledge graph path reasoning device based on a distributed database system, comprising a processor and a memory, wherein the memory stores at least one executable instruction, and the processor executes the above-mentioned knowledge graph path reasoning method according to the instruction.

[0037] In summary, the knowledge graph path reasoning method based on the distributed database system of the present invention has the following advantages:

[0038] (1) As a new knowledge graph path reasoning method that is compatible with big data and graph computing, this method can realize the reasoning analysis of the deep path relationship of the knowledge graph. After verification by the data set, this method can achieve efficient computing and improve the accuracy of path reasoning.

[0039] (2) The method of the present invention can realize parallel search of all paths and can effectively analyze the relationship graph that can only be reached by deeper paths. By unifying similar relationships, the method makes the data more reliable, thereby constructing a feature data matrix with real physical meaning.

[0040] (3) The path reasoning scheme of the present invention does not adopt the traditional random walk combined with tensor decomposition strategy, but integrates the whole process method of complete design, making the reasoning process closer to the actual situation. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments recorded in the present invention, not all embodiments. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0042] Figure 1 It is a schematic diagram of the flow chart of the knowledge graph path reasoning method of the present invention.

[0043] Figure 2 It is a schematic diagram of the structure of the knowledge graph path reasoning system of the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with specific embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments, and the present invention can also be implemented or applied through other different specific implementation methods. The details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention.

[0045] At the same time, it should be understood that the protection scope of the present invention is not limited to the specific embodiments described below; it should also be understood that the terms used in the embodiments of the present invention are for describing specific embodiments rather than for limiting the protection scope of the present invention.

[0046] Example 1: A knowledge graph path reasoning method based on a distributed database system (see Figure 1 ), the method comprises:

[0047] Determine the source data and target data, enumerate all reachable relationship paths from the source data to the target data, and record each relationship path data to form an indefinite length array A[n]. The elements in the indefinite length array A[n] are the relationship contents contained in each relationship path between the source data and the target data.

[0048] Perform NLP analysis on the elements in the indefinite-length array A[n]. NLP analysis includes unifying similar relationship categories (such as father, dad; writing, writing, etc.) and eliminating abnormal relationships and very few relationships. NLP analysis includes text classification, text analysis, and semantic analysis.

[0049] Select a comparison sample (a sample with a known dependent variable) in the atlas library, enumerate all reachable relationship paths between each data in the comparison sample and other data in the comparison sample, record each relationship path data to form an indefinite length array B[n], compare and calculate the path similarity of each A[n] and B[n], and the path similarity calculation indicators include the text distance and cosine distance between the two to obtain the calculation results.

[0050] The calculation results obtained in the previous step are used to form a data matrix, and the data matrix is ​​combined with the known dependent variables to construct a training data table. The training data table is stored in the GreenPlum distributed database. The training data table contains complete trainable features. The training data table and field naming are standardized during the storage process of the training data table.

[0051] Use Madlib to perform logistic regression training on the training data table constructed in the previous step to obtain a trained model.

[0052] Record the relationship path data to be predicted to form an indefinite length array C[n], compare and calculate the path similarity of each A[n] and C[n], obtain the calculation results and form a data matrix to be predicted, combine the data matrix to be predicted with the known dependent variables to construct a training data table to be predicted, and use the trained model in the previous step to perform logistic regression prediction on the training data table to be predicted, that is, to obtain the probability that the relationship between the data to be predicted from the source data to the target data is established.

[0053] Example 2: A knowledge graph path reasoning system based on a distributed database system (see Figure 2 ), this system includes:

[0054] Recording module: used to record various relationship path data and form an array of indefinite length;

[0055] NLP analysis module: used to perform NLP analysis on elements in variable-length arrays;

[0056] Path similarity calculation module: used to calculate the path similarity between various indefinite length arrays;

[0057] Data matrix construction module: used to construct the path similarity calculation results between various indefinite length arrays into a data matrix;

[0058] Training data table construction module: used to combine the data matrix with the known dependent variables to construct a training data table;

[0059] Madlib training module: used to perform logistic regression training on the training data table and obtain the trained model;

[0060] Relationship prediction module: used to perform logistic regression prediction on the training data table to be predicted through the trained model, and obtain the probability that the relationship between the source data and the target data to be predicted is established.

[0061] The above modules are implemented according to the technical solution of Example 1.

[0062] The various embodiments of the present invention are described in a progressive manner, and the same or similar parts between the various embodiments can refer to the relevant description of the method embodiment.

[0063] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, replacement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A knowledge graph path reasoning method based on a distributed database system, characterized in that: The method comprises: Determine the source data and target data, enumerate all reachable relationship paths from the source data to the target data, record each relationship path data to form an indefinite length array A[n], and perform NLP analysis on the elements in the indefinite length array A[n]; Select a comparison sample in the atlas library, enumerate all reachable relationship paths between each data in the comparison sample and other data in the comparison sample, record each relationship path data to form an indefinite length array B[n], compare and calculate the path similarity of each A[n] and B[n], and obtain the calculation result; Using the calculation results obtained in the previous step to form a data matrix, the data matrix is ​​combined with the known dependent variables to construct a training data table; Use Madlib to perform logistic regression training on the training data table constructed in the previous step to obtain a trained model; Record the relationship path data to be predicted to form an indefinite length array C[n], compare and calculate the path similarity of each A[n] and C[n], obtain the calculation results and form a data matrix to be predicted, combine the data matrix to be predicted with the known dependent variables to construct a training data table to be predicted, and use the trained model in the previous step to perform logistic regression prediction on the training data table to be predicted, that is, obtain the probability that the relationship between the data to be predicted from the source data to the target data is established.

2. The knowledge graph path reasoning method according to claim 1, characterized in that: The elements in the variable-length array A[n] are the relationship contents contained in each relationship path between the source data and the target data.

3. The knowledge graph path reasoning method according to claim 1, characterized in that: The NLP analysis of the elements in the indefinite-length array A[n] includes unifying similar relationship categories and eliminating abnormal relationships and very few relationships.

4. The knowledge graph path reasoning method according to claim 1, characterized in that: The NLP analysis includes text classification, text analysis, semantic analysis, sentiment analysis, and intent recognition.

5. The knowledge graph path reasoning method according to claim 4 is characterized in that: The semantic analysis includes lexical analysis, syntactic analysis, pragmatic analysis, and contextual analysis.

6. The knowledge graph path reasoning method according to claim 1, characterized in that: The comparison samples in the atlas library are samples of known dependent variables already in the atlas library.

7. The knowledge graph path reasoning method according to claim 1, characterized in that: The path similarity calculation index of A[n] and B[n] includes the text distance and cosine distance between the two.

8. The knowledge graph path reasoning method according to claim 1, characterized in that: The training data table is stored in a GreenPlum distributed database. The training data table contains complete trainable features. The training data table and field naming are standardized during the storage process of the training data table.

9. A knowledge graph path reasoning system based on a distributed database system for implementing the knowledge graph path reasoning method of claim 1, characterized in that: The system comprises: Recording module: used to record various relationship path data and form an array of indefinite length; NLP analysis module: used to perform NLP analysis on elements in variable-length arrays; Path similarity calculation module: used to calculate the path similarity between various indefinite length arrays; Data matrix construction module: used to construct the path similarity calculation results between various indefinite length arrays into a data matrix; Training data table construction module: used to combine the data matrix with the known dependent variables to construct a training data table; Madlib training module: used to perform logistic regression training on the training data table and obtain the trained model; Relationship prediction module: used to perform logistic regression prediction on the training data table to be predicted through the trained model, and obtain the probability that the relationship between the source data and the target data to be predicted is established.

10. A knowledge graph path reasoning device based on a distributed database system, comprising a processor and a memory, characterized in that: The memory stores at least one executable instruction, and the processor executes the knowledge graph path reasoning method described in any one of claims 1-8 according to the instruction.

Images