A method and apparatus for accuracy analysis of knowledge graphs

Abstract

This application provides a method and apparatus for accuracy analysis of knowledge graphs, belonging to the field of computer technology. It includes: randomly selecting at least two sample triples from the knowledge graph; obtaining a model training set containing subgraphs related to the head and tail entities based on the topological relationship between the head and tail entities of the sample triples in the knowledge graph; training an initial model based on the triples in the subgraphs to obtain a prediction model for analyzing the accuracy of the knowledge graph; then evaluating the accuracy of the sample triples in the knowledge graph using the prediction model, obtaining an evaluation score, and identifying target triples that do not meet preset conditions based on the evaluation score; finally, verifying the content of the target triples using external open-source information to obtain the accuracy analysis results for the knowledge graph.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, electronic device, and computer-readable storage medium for accuracy analysis of knowledge graphs. Background Technology

[0002] Knowledge graphs are visual information storage tools with wide applications in semantic search, recommendation systems, and other fields. Their concise and clear layout allows readers to easily understand the attribute relationships between content elements. However, in the context of big data, as the scale of knowledge graphs continues to expand, the quality of their content has become an increasingly prominent issue.

[0003] In related technologies, the assessment of the accuracy of existing knowledge graph content mainly relies on methods such as manual sampling and comparative evaluation.

[0004] However, existing technical solutions are significantly less efficient in terms of manual sampling and testing when dealing with knowledge graphs with excessively large content. Furthermore, they are limited by relatively fixed evaluation rules, resulting in poor versatility for evaluating the correctness of different types of knowledge graphs. Overall, the methods for evaluating the accuracy of knowledge graphs are not perfect. Summary of the Invention

[0005] This application provides a method, apparatus, electronic device, and computer-readable storage medium for analyzing the accuracy of knowledge graphs, in order to address the problem that existing methods for evaluating the accuracy of knowledge graphs are not sufficiently comprehensive.

[0006] In a first aspect, embodiments of this application provide a method for accuracy analysis of knowledge graphs, the method comprising:

[0007] At least two sample triples are randomly selected from the knowledge graph, and the sample triples include at least a head entity and a tail entity;

[0008] Based on the topological relationship between the head entity and the tail entity in the knowledge graph, a model training set is obtained. The model training set includes: a first type of knowledge subgraph for the head entity and a second type of knowledge subgraph for the tail entity; wherein, the first type of knowledge subgraph is composed of a first type of triples associated with the head entity, and the second type of knowledge subgraph is composed of a second type of triples associated with the tail entity.

[0009] Based on the first type of triples of the first type of knowledge subgraph and the second type of triples of the second type of knowledge subgraph in the model training set, the initial model is trained to obtain a prediction model for analyzing the accuracy of knowledge graphs.

[0010] The accuracy of the sample triples in the knowledge graph is evaluated based on the prediction model to obtain an evaluation score.

[0011] Based on the evaluation scores, target triplets in the sample triplets whose scores do not meet the preset conditions are identified;

[0012] The content of the target triple is verified by external open-source information to obtain the accuracy analysis results for the knowledge graph.

[0013] Secondly, embodiments of this application provide a method for generating a prediction model, the method comprising:

[0014] At least two sample triples are randomly selected from the knowledge graph, and the sample triples include at least a head entity and a tail entity;

[0015] Based on the topological relationship between the head entity and the tail entity in the knowledge graph, a model training set is obtained. The model training set includes: a first type of knowledge subgraph for the head entity and a second type of knowledge subgraph for the tail entity; wherein, the first type of knowledge subgraph is composed of a first type of triples associated with the head entity, and the second type of knowledge subgraph is composed of a second type of triples associated with the tail entity. Both the first type of triples and the second type of triples include: a head entity, a tail entity, and a relation entity used to represent the relationship between the head entity and the tail entity.

[0016] Based on the first type of triples of the first type of knowledge subgraph and the second type of triples of the second type of knowledge subgraph in the model training set, the initial model is trained to obtain a prediction model for analyzing the accuracy of knowledge graphs.

[0017] Thirdly, embodiments of this application provide a method for evaluating the accuracy of a knowledge graph, the method comprising:

[0018] The method involves obtaining a prediction model for analyzing the accuracy of a knowledge graph and at least two sample triples from the knowledge graph, wherein the prediction model is trained by the prediction model generation method described in the first aspect, and the knowledge graph is the knowledge graph used in the prediction model generation method described in the first aspect.

[0019] The accuracy of the sample triples in the knowledge graph is evaluated based on the prediction model to obtain an evaluation score.

[0020] Based on the evaluation scores, target triplets in the sample triplets whose scores do not meet the preset conditions are identified;

[0021] The content of the target triple is verified by external open-source information to obtain the accuracy analysis results for the knowledge graph.

[0022] Fourthly, embodiments of this application provide an accuracy analysis device for knowledge graphs, the device comprising:

[0023] The first acquisition module is used to randomly acquire at least two sample triples from the knowledge graph, wherein the sample triples include at least a head entity and a tail entity;

[0024] The training set generation module is used to obtain a model training set based on the topological relationship between the head entity and the tail entity in the knowledge graph. The model training set includes: a first type of knowledge subgraph for the head entity and a second type of knowledge subgraph for the tail entity; wherein, the first type of knowledge subgraph is composed of a first type of triples associated with the head entity, and the second type of knowledge subgraph is composed of a second type of triples associated with the tail entity.

[0025] The model training execution module is used to train the initial model based on the first type of triples of the first type of knowledge subgraph and the second type of triples of the second type of knowledge subgraph in the model training set, so as to obtain a prediction model for analyzing the accuracy of knowledge graphs.

[0026] The triplet evaluation module is used to evaluate the accuracy of sample triplets in the knowledge graph based on the prediction model and obtain an evaluation score.

[0027] The triplet screening module is used to determine, based on the evaluation scores, target triplets in the sample triplets whose scores do not meet preset conditions.

[0028] The triple verification module is used to verify the content of the target triple using external open-source information to obtain accuracy analysis results for the knowledge graph.

[0029] Fifthly, embodiments of this application provide a predictive model generation apparatus, the apparatus comprising:

[0030] The first acquisition module is used to randomly acquire at least two sample triples from the knowledge graph, wherein the sample triples include at least a head entity and a tail entity;

[0031] A training set generation module is used to obtain a model training set based on the topological relationship between the head entity and the tail entity in the knowledge graph. The model training set includes: a first type of knowledge subgraph for the head entity and a second type of knowledge subgraph for the tail entity; wherein, the first type of knowledge subgraph is composed of a first type of triples associated with the head entity, and the second type of knowledge subgraph is composed of a second type of triples associated with the tail entity. Both the first type of triples and the second type of triples include: a head entity, a tail entity, and a relation entity used to represent the relationship between the head entity and the tail entity.

[0032] The model training execution module is used to train the initial model based on the first type of triples of the first type of knowledge subgraph and the second type of triples of the second type of knowledge subgraph in the model training set, so as to obtain a prediction model for analyzing the accuracy of knowledge graphs.

[0033] Sixthly, embodiments of this application provide an accuracy evaluation device for knowledge graphs, the device comprising:

[0034] The second acquisition module is used to acquire a prediction model for analyzing the accuracy of the knowledge graph and at least two sample triples in the knowledge graph, wherein the prediction model is trained by the prediction model generation device described in the fourth aspect; and the knowledge graph is the knowledge graph used in the prediction model generation device described in the fourth aspect.

[0035] The triplet evaluation module is used to evaluate the accuracy of sample triplets in the knowledge graph based on the prediction model and obtain an evaluation score.

[0036] The triplet screening module is used to determine, based on the evaluation scores, target triplets in the sample triplets whose scores do not meet preset conditions.

[0037] The triple verification module is used to verify the content of the target triple using external open-source information to obtain accuracy analysis results for the knowledge graph.

[0038] In a seventh aspect, embodiments of this application provide an electronic device, including: a processor;

[0039] Memory used to store the processor's executable instructions;

[0040] The processor is configured to execute the instructions to implement the method.

[0041] Eighthly, embodiments of this application provide a computer-readable storage medium that, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the method.

[0042] In this embodiment, at least two sample triples are first randomly selected from the knowledge graph. Each sample triple includes at least a head entity and a tail entity. Then, based on the topological relationship between the head and tail entities in the knowledge graph, a model training set is obtained. This training set includes a first-class knowledge subgraph for the head entity and a second-class knowledge subgraph for the tail entity. The initial model is trained based on the triples in the first and second-class knowledge subgraphs in the training set to obtain a predictive model for analyzing the accuracy of the knowledge graph. Next, the accuracy of the sample triples in the knowledge graph is evaluated using the predictive model, and an evaluation score is obtained. Based on the evaluation score, target triples that do not meet preset conditions are identified. Finally, the content of the target triples is verified using external open-source information to obtain the accuracy analysis results for the knowledge graph. By extracting a portion of the triples from the knowledge graph as sample triples and obtaining subgraphs for the head and tail nodes of the sample triples, the initial model is trained to obtain a predictive model for analyzing the accuracy of the knowledge graph. Then, the trained model is used to back-evaluate all sample triples. Based on statistical theory, the evaluation results obtained by randomly selecting a portion of the complete knowledge graph for evaluation can reflect the correctness of the entire knowledge graph content. This can effectively shorten the evaluation time for the correctness of the knowledge graph. Furthermore, the evaluation model obtained by training on the subgraphs of different types of knowledge graphs in advance has higher specific applicability, further improving the evaluation method for the accuracy of knowledge graphs.

[0043] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0044] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0045] Figure 1 This is a simplified implementation flowchart of a knowledge graph accuracy analysis method provided in this application embodiment;

[0046] Figure 2 This is an exemplary knowledge graph provided in the embodiments of this application;

[0047] Figure 3 This is a simplified implementation flowchart of a prediction model generation method provided in an embodiment of this application;

[0048] Figure 4 This is a simplified flowchart of the implementation steps of a knowledge graph accuracy evaluation method provided in this application embodiment;

[0049] Figure 5 This is a flowchart illustrating the complete implementation steps of a knowledge graph accuracy analysis method provided in this application embodiment;

[0050] Figure 6 This is a framework diagram of a knowledge graph accuracy evaluation method provided in an embodiment of this application;

[0051] Figure 7 This is a distribution diagram showing the relationship between triple support and evaluation score provided in an embodiment of this application;

[0052] Figure 8 This is a schematic diagram of the functional module structure of a knowledge graph accuracy analysis device provided in an embodiment of this application;

[0053] Figure 9 This is a schematic diagram of the functional module structure of a predictive model generation device provided in an embodiment of this application;

[0054] Figure 10 This is a schematic diagram of the functional module structure of a knowledge graph accuracy evaluation device provided in an embodiment of this application;

[0055] Figure 11 This is a functional component relationship diagram of an electronic device provided in an embodiment of this application;

[0056] Figure 12 This is a functional component relationship diagram of another electronic device provided in the embodiments of this application. Detailed Implementation

[0057] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.

[0058] Reference Figure 1 , Figure 1 This is a simplified flowchart illustrating the implementation steps of a knowledge graph accuracy analysis method provided in this application. For example... Figure 1 As shown, the steps of the method include:

[0059] Step 101: Randomly obtain at least two sample triples from the knowledge graph. The sample triples shall include at least a head entity and a tail entity.

[0060] This application provides a predictive model generation method aimed at obtaining a predictive model for analyzing the accuracy of knowledge graphs. A knowledge graph is a visualization of a knowledge domain or a mapping map of that domain. It is a series of various graphics that display the development process and structural relationships of knowledge. It uses visualization techniques to describe knowledge resources and their carriers, and to mine, analyze, construct, draw, and display knowledge and the interrelationships between them.

[0061] Reference Figure 2 This illustrates an exemplary knowledge graph provided in an embodiment of this application. For example... Figure 2 As shown, this includes multiple circular content nodes and directed line segments representing the content relationships between nodes, along with relationship descriptions. Specifically, taking "Zhang San" as the central node in the graph, it points to the "Company B" node, and the relationship on this pointing line segment is labeled "Employed at"; furthermore, the pointing relationship between "Zhang San" and "Li Si" is labeled "Spouse". In this knowledge graph, since the pointing relationship lines between nodes are unidirectional, the order of content relationships between nodes can be distinguished. In the above node relationships, (Zhang San, Employed at, Company B) can form a basic triplet. According to the directed pointing relationship of the middle "Employed at" in the graph, "Zhang San", as the starting node pointed to by the arrow, can be called the head entity, and "Company B", as the ending node pointed to by the arrow, can be called the tail entity.

[0062] In this embodiment, the prediction model is generated by acquiring elements from an existing knowledge graph. This is achieved by training the model using a small training set, consisting only of samples from their respective subgraphs. Statistical theory has shown that the accuracy of randomly sampled data is the same as the overall accuracy of the knowledge graph. This approach reduces the training time while ensuring the accuracy of the prediction model. Because the number of samples is fixed and the subgraph size is limited, the evaluation process can have a stable and bounded duration.

[0063] Specifically, a simple random sampling method is first used to obtain a certain number of sample triples from the existing knowledge graph as initial samples. Refer to the above... Figure 2 The triples obtained from the knowledge graph are used to extract three samples from 10 triples in the example: (Zhang San, address, Shanghai), (Wang Er, friend, Zhang San), and (Wang Er, spouse, Zhao Wu). Generally, when extracting approximately 900 triples from a large-scale knowledge graph, the evaluation error is less than 3.2% with a 95% probability. Therefore, using this sample size for evaluation is acceptable for this order of magnitude.

[0064] Step 102: Obtain a model training set based on the topological relationship between the head entity and the tail entity in the knowledge graph. The model training set includes: a first type of knowledge subgraph for the head entity and a second type of knowledge subgraph for the tail entity; wherein, the first type of knowledge subgraph is composed of a first type of triple associated with the head entity, and the second type of knowledge subgraph is composed of a second type of triple associated with the tail entity.

[0065] After obtaining multiple triples through random sampling, a model training set is obtained based on the topological relationships of the head / tail entities in the triples within the knowledge graph. Specifically, a breadth-first search is performed on the knowledge graph, starting with the head entity and the tail entity respectively, to obtain subgraphs for the head / tail entities. The subgraph is defined as the graph containing only a portion of the triples associated with the head or tail entity. (Continue to...) Figure 2 The subgraph obtained by traversing to the left with Zhang San as the head entity is: (Zhang San, spouse, Li Si) + (Li San, friend, Li Si) + (Li Si, friend, company A), which is a simple subgraph containing three triplets.

[0066] In this process, the knowledge subgraph obtained by traversing the head entity in the randomly sampled triples is called the first type of knowledge subgraph, and the knowledge subgraph obtained by traversing the tail entity in the sample triples is called the second type of knowledge subgraph. The triples in the subgraph still include: a head entity, a tail entity, and a relation entity representing the relationship between the head and tail entities. In the triple (Li San, Friend, Li Si), Li San is the head entity, Li Si is the tail entity, and Friend is the relation entity representing the relationship between the two.

[0067] The model's score for the extracted triplet samples depends directly on the subgraph in which they are located. Using subgraphs to form the training set allows for the filtering of irrelevant data. Meanwhile, although the embedding result of an entity depends only on its neighboring nodes, the embedding result of a relation in the knowledge graph depends on the joint learning of multiple related triples in the knowledge graph. Selecting subgraphs for training reduces the amount of data used for this part of the training.

[0068] Step 103: Based on the first type of triples of the first type of knowledge subgraph and the second type of triples of the second type of knowledge subgraph in the model training set, train the initial model to obtain a prediction model for analyzing the accuracy of the knowledge graph.

[0069] After performing a breadth-first search traversal based on the head and tail entities in all extracted sample triples to obtain the first and second type knowledge subgraphs, the set of the first type triples of the first type knowledge subgraph and the set of the second type triples of the second type knowledge subgraph are used as the training set of triples to train the initial model and obtain a predictive model for the evaluation capability of the current knowledge graph.

[0070] The prediction model generation method provided in this application can initialize the embedding of relations using pre-training results, avoiding overfitting of the embedding model due to a small number of training triples. The initial vectors used in the initial model training process can be the embedding results of semantically similar relations from other knowledge graphs.

[0071] Specifically, in this embodiment, the TransE algorithm is used to train the initial model. The basic idea of ​​TransE is to represent each entity and each relation as a low-dimensional vector, using distributed representation to describe the triples in the knowledge base. The idea is that for each fact, the relation is regarded as a translation operation from the head entity to the tail entity.

[0072] Step 104: Evaluate the accuracy of the sample triples in the knowledge graph based on the prediction model and obtain an evaluation score.

[0073] The prediction model obtained after steps 101-103 above can be used to formally evaluate the accuracy of the knowledge graph.

[0074] Based on statistical theory, the accuracy of randomly sampled data is the same as the overall accuracy of the knowledge graph. Therefore, the input to the prediction model can directly use the finite number of sample triples randomly selected in step 101. After inputting the sample triples into the prediction model, the model will assign an evaluation score based on embedded learning to each triple.

[0075] Step 105: Based on the evaluation scores, identify the target triplets in the sample triplets whose scores do not meet the preset conditions.

[0076] The scores obtained after evaluating the prediction model can be greater than 0 for any given triplet. There is no upper bound on the score, but it is generally below 25.0. The relative magnitude of the score reflects the degree to which the model judges the triplet as correct. In the TransE model, a smaller score indicates that the model is more likely to judge the triplet as correct. Therefore, to determine whether a triplet is correct from the evaluation score, incorrect examples can be manually generated, and triplets with scores lower than those of incorrect examples can be considered correct triplets. In this embodiment, triplets with scores in the middle range of all evaluation results are considered target triplets that do not meet the preset conditions.

[0077] It should be noted that the set of triples input to the evaluation method used in the prediction model of this application embodiment also includes artificially constructed negative triples (i.e., triples with obvious errors in their content). The prediction model includes an embedding layer and an embedding function. The embedding layer maps entities or relations to a vector space. The embedding function, based on the embedding results of entities / relations, maps triples to vectors between the embedding results and calculates the result, which is generally a real number used to measure whether the triples are reasonable. In this process, the model embedding layer is first initialized according to a Gaussian random distribution or existing pre-training results, and then trained according to the training set. Negative triples are generated by replacing a part of the triples in the training set.

[0078] Introducing artificially constructed negative triplets allows the prediction model to correctly identify error factors within triplets when scoring selected sample triplets, thereby more accurately evaluating the correctness of the knowledge graph.

[0079] Step 106: Verify the content of the target triple using external open-source information to obtain the accuracy analysis results for the knowledge graph.

[0080] Following step 105, after obtaining the scoring results for the sample triples of the selected samples, further verification analysis of the triple content is required. Generally, triples whose content descriptions conform to the facts or corroborate most other facts in the knowledge graph are considered correct. However, in this process, due to the insensitivity of the prediction model to semantic representation, it is necessary to verify the facts represented by the triples at the semantic level.

[0081] Fact-checking has always been a long-standing and important problem in the field of natural language processing, but even large-scale language models currently suffer from hallucinations and insufficient domain knowledge. In this application's embodiments, from the perspective of simulating human behavior, obtaining external information text related to the current triple and using it as evidence to verify the correctness of the triple is a more natural and objectively realistic approach. Available external information includes human experience, querying manually maintained small-scale knowledge graphs, and searching using open-source information.

[0082] It should be noted that in this embodiment of the application, open-source information from the Internet is used to search for disputed triples, and word frequency statistics and word co-occurrence methods are used to judge them as a supplement to the semantic verification. Search results are provided for disputed triples as evidence to facilitate manual verification, rather than relying entirely on external information for verification.

[0083] In summary, the accuracy analysis method for knowledge graphs provided in this application firstly randomly selects at least two sample triples from the knowledge graph. Each sample triple includes at least a head entity and a tail entity. Then, based on the topological relationship between the head and tail entities in the knowledge graph, a model training set is obtained. This training set includes a first-type knowledge subgraph for the head entity and a second-type knowledge subgraph for the tail entity. The initial model is trained based on the triples in the first and second-type knowledge subgraphs in the training set to obtain a predictive model for analyzing the accuracy of the knowledge graph. By extracting a portion of the triples from the knowledge graph as sample triples and obtaining subgraphs for the head and tail nodes of these sample triples to train the initial model, a predictive model for analyzing the accuracy of the knowledge graph is obtained. Finally, the trained model is used to back-evaluate all sample triples. Based on statistical theory, the evaluation results obtained by randomly selecting a portion of the complete knowledge graph for evaluation can reflect the correctness of the entire knowledge graph content. This can effectively shorten the evaluation time for the correctness of the knowledge graph. Furthermore, the evaluation model obtained by training on the subgraphs of different types of knowledge graphs in advance has higher specific applicability, further improving the evaluation method for the accuracy of knowledge graphs.

[0084] Reference Figure 3 , Figure 3 This is a simplified implementation flowchart of a prediction model generation method provided in an embodiment of this application. For example... Figure 3 As shown, the steps of the method include:

[0085] Step 201: Randomly obtain at least two sample triples from the knowledge graph. The sample triples include at least a head entity and a tail entity.

[0086] Step 202: Based on the topological relationship between the head entity and the tail entity in the knowledge graph, obtain a model training set. The model training set includes: a first type of knowledge subgraph for the head entity and a second type of knowledge subgraph for the tail entity. The first type of knowledge subgraph is composed of a first type of triples associated with the head entity, and the second type of knowledge subgraph is composed of a second type of triples associated with the tail entity. Both the first type of triples and the second type of triples include: a head entity, a tail entity, and a relation entity used to represent the relationship between the head entity and the tail entity.

[0087] Step 203: Based on the first type of triples of the first type of knowledge subgraph and the second type of triples of the second type of knowledge subgraph in the model training set, train the initial model to obtain a prediction model for analyzing the accuracy of the knowledge graph.

[0088] Steps 201-203 can be referred to in detail to the content described in steps 101-103 above, and will not be repeated here in this embodiment.

[0089] Reference Figure 4 , Figure 4 This is a simplified flowchart illustrating the implementation steps of a knowledge graph accuracy analysis method provided in this application. Figure 4 As shown, the steps of the method include:

[0090] Step 301: Obtain the prediction model used to analyze the accuracy of the knowledge graph, as well as at least two sample triples in the knowledge graph.

[0091] The prediction model obtained after steps 201-203 above can be used to formally evaluate the accuracy of the knowledge graph.

[0092] Based on statistical theory, the accuracy of randomly sampled data is the same as the overall accuracy of the knowledge graph. Therefore, the input to the prediction model can directly use the finite number of sample triples randomly selected in step 201. After inputting the sample triples into the prediction model, the model will assign an evaluation score based on embedded learning to each triple.

[0093] Step 302: Evaluate the accuracy of the sample triples in the knowledge graph based on the prediction model and obtain an evaluation score.

[0094] Step 303: Based on the evaluation scores, determine the target triplets in the sample triplets whose scores do not meet the preset conditions.

[0095] Step 304: Verify the content of the target triple using external open-source information to obtain the accuracy analysis results for the knowledge graph.

[0096] Steps 302-304 can be referred to the contents described in steps 104-106 above, and will not be repeated here in this embodiment.

[0097] In summary, the accuracy analysis method for knowledge graphs provided in this application evaluates and scores a subset of triples in the knowledge graph using a prediction model trained on randomly sampled triples. Subsequently, a manually set evaluation threshold is used to select target triples with disputed scores for correctness verification using external information. By using evaluation results obtained from randomly sampled data to reflect the overall correctness of the knowledge graph, the evaluation time for knowledge graph accuracy can be effectively shortened. Furthermore, the method of training targeted evaluation models on different types of knowledge graphs using their subgraphs beforehand provides higher specific applicability.

[0098] Reference Figure 5 , Figure 5 This is a complete implementation flowchart of a knowledge graph accuracy evaluation method provided in this application embodiment. For example... Figure 5 As shown, the steps of the method include:

[0099] Step 401: Randomly obtain at least two sample triples from the knowledge graph. The sample triples include at least a head entity and a tail entity.

[0100] For details of this step, please refer to step 101 above. This embodiment will not repeat the details here.

[0101] Step 402: Starting from the head entity, perform a breadth-first traversal search on the knowledge graph. If the breadth of the traversal search meets the preset conditions, obtain a first type of knowledge subgraph for the head entity; starting from the tail entity, perform a breadth-first traversal search on the knowledge graph. If the breadth of the traversal search meets the preset conditions, obtain a second type of knowledge subgraph for the tail entity.

[0102] Starting with the first and last entities in randomly selected sample triples, a breadth-first search is performed on the knowledge graph to obtain the subgraphs corresponding to the nodes represented by the first and last entities in the knowledge graph. Breadth-first search is a traversal algorithm for graph data structures. It is a search strategy that expands the state layer by layer and is often used to solve connectivity problems and shortest path problems. It starts from a given starting vertex and searches the nodes in the graph in a breadth-first manner until the target node is found or the entire graph is traversed.

[0103] For details, please refer to... Figure 2 In such Figure 2 In the knowledge subgraph shown, Zhang San is the head entity in the sample triplet, and Wang Er is the tail entity in the sample triplet. Assuming the subgraph has a maximum size of 3 and a depth of 1, Zhang San starts a breadth-first search to (Zhang San, Employment, Company B) → (Company B, Address, Shanghai) and (Zhang San, Spouse, Li Si) → (Li Si, Employment, Company A). The above two sets of triples can form the first type of knowledge subgraph associated with the head entity Zhang San.

[0104] Correspondingly, starting with the tail entity Wang Er, the search traverses to (Wang Er, Employment, Company B) → (Company B, Address, Shanghai), (Wang Er, Address, Beijing); these three triples can also form a second type of knowledge subgraph associated with the tail entity Wang Er. It's worth noting that for a tail entity, traversing the edges containing it (e.g., Shanghai via (Company B, Address, Shanghai)) also yields the subgraph containing Shanghai.

[0105] It should be noted that in this embodiment of the application, there is only an upper limit to the depth of knowledge subgraph traversal. That is, for a knowledge subgraph with a traversal depth of 3, if the node with a depth of 2 is reached during the traversal process and there are no other nodes that can be traversed afterward, then the subgraph with a depth of 2 will also be used as the subgraph of the current head entity (or tail entity), rather than limiting the traversal depth to 3 to be a knowledge subgraph.

[0106] Step 403: If the first type of knowledge subgraph contains the sample triplet, remove the relation entity of the sample triplet from the first type of knowledge subgraph; if the second type of knowledge subgraph contains the sample triplet, remove the relation entity of the sample triplet from the second type of knowledge subgraph, and save the contents of the first type of knowledge subgraph and the second type of knowledge subgraph after removing the sample triplet as different subgraph files in the form of triplet text, to obtain the model training set.

[0107] Following step 402, delete all content related to sample triples from the obtained knowledge subgraphs, and finally retain all relevant subgraphs in the form of triples to obtain the model training set for use as the prediction model.

[0108] Specifically, in such Figure 2 The knowledge subgraphs obtained from the knowledge graph shown are as follows: For the first type of knowledge subgraph, the relation entities of the sample triples that are also in the sample set are deleted. For example, if we use the subgraph to verify (Zhang San, employment, company B) in the sample, the relation entity corresponding to the triple in the subgraph is deleted; For the second type of knowledge subgraph, the part of the relation entity associated with the tail entity represented by Wang Er that is also covered by the sample triples is deleted.

[0109] Step 404: Based on the first type of triples of the first type of knowledge subgraph and the second type of triples of the second type of knowledge subgraph in the model training set, train the initial model to obtain a prediction model for analyzing the accuracy of the knowledge graph.

[0110] It should be noted that the model training method used in this application embodiment has various forms of model training sets obtained from the first type of knowledge subgraph and the second type of knowledge subgraph. In this application embodiment, the TransE algorithm is used to train the initial model. The method of using all out-of-sample triples as the training set is denoted as TransE. origin The method of training by using subgraphs to construct a training set is called TransE. sample The method of initializing the embedding results corresponding to the relations in the subgraph using a vectorization method (e.g., using the BERT model to vectorize the relations in the triples as strings) and then training is TransE. trainedThe training results obtained using different types of training sets differ in terms of the result threshold.

[0111] Referring to Table (1), Table (1) shows the experimental results of different preprocessing methods based on the TransE training method provided in the embodiments of this application.

[0112] Table (1)

[0113] TransE FB15K237-SYN10 FB15K237-SYN20 FB15K237-SYN40 Best AUC Thres Best AUC Thres Best AUC Thres origin 0.926 0.825 0.910 0.858 0.822 0.856 0.786 0.842 0.632 sample 0.931 0.865 0.92 0.85 0.863 0.856 0.795 0.853 0.697 trained 0.931 0.848 0.924 0.863 0.864 0.859 0.791 0.851 0.682

[0114] As shown in Table (1), the initial model was trained on the FB15K-237 dataset, which includes 900 randomly selected sample triples. We manually replaced the triples in the dataset to form a simulated dataset FB15K-237-SYNX, where X is selected from 10, 20, and 40, representing the proportion of incorrect triples. In terms of comparison metrics, Best represents the classification accuracy of triples at the optimal threshold (triples scored below the threshold are considered correct, while those above the threshold are considered incorrect; the choice of threshold determines the evaluation error. Here, the goal is to select the threshold that minimizes the evaluation error, thereby comparing whether the three methods' scores for triples reflect whether the triples are correct and which method is better under ideal conditions). AUC represents the ROC-AUC metric (ROC, Receiver Operating Characteristic curve, measures the change in a scoring model's classification result with the choice of threshold; AUC, Area Under Curve, refers to the area under the ROC curve), which measures the quality of the scoring system. It is calculated by estimating the probability that the score for a correct sample is better than the score for an incorrect sample. Thres is the classification accuracy after selecting the threshold using the validation set.

[0115] It should be noted that since the optimal threshold cannot be obtained directly in the actual environment, the method of constructing negative example triples is adopted to form the validation set used to select the threshold. At this time, the threshold is selected by classifying the negative examples in the validation set and the original triples in the test set according to the model score. The threshold with the smallest classification error is the threshold selected through the validation set (but since the threshold only needs to be applied to the test set, the threshold selected in this way is not necessarily the optimal threshold).

[0116] As shown in Table (1), on the simulated dataset FB15K-237, although the training set size decreased, the evaluation performance was similar to that of the original model on the training set because irrelevant data was filtered out when extracting subgraphs for training. It can also be seen that initialization did not significantly affect the results for these three knowledge graphs.

[0117] Meanwhile, on the relatively sparse YAGO3-10 dataset, Table (2) shows the experimental results of different preprocessing methods under another TransE training method provided in the embodiments of this application. It can be seen that during extraction, due to the filtering of the training data of relations, the performance of classification and evaluation after selecting subgraphs is worse than before selecting subgraphs. At the same time, it can be seen that the pre-trained model reduces its impact due to the assistance of existing semantics.

[0118] Table (2)

[0119] TransE YAGO3-10-SYN10 YAGO3-10-SYN20 YAGO3-10-SYN40 Best AUC Thres Best AUC Thres Best AUC Thres origin 0.906 0.839 0.904 0.827 0.842 0.789 0.758 0.810 0.600 sample 0.910 0.847 0.850 0.800 0.840 0.780 0.717 0.780 0.648 trained 0.912 0.843 0.849 0.804 0.849 0.800 0.727 0.801 0.672

[0120] In addition, due to the limitations of the number of samples and the size of the subgraph, there is an upper bound on the size of the training set for knowledge graphs of different sizes, and thus an upper bound on the training time. Refer to Table (3), which shows the experimental results of another preprocessed dataset provided in the embodiments of this application.

[0121] Table (3)

[0122]

[0123]

[0124] TransE sample With TransE trained The training set is a set consisting of subgraphs of sample triples, TransE origin To use all out-of-sample triples as the training set, the training time is the time TransE takes to learn the embeddings on the training set, and the evaluation time is the time taken to evaluate 900 samples. It is evident that the optimal threshold training parameters and training time differ depending on the sample size. Those skilled in the art can choose according to the actual situation; this embodiment does not impose any limitations.

[0125] In an optional embodiment, step 404 may specifically include:

[0126] Sub-step A1: Based on the model training set, train the initial model using the TransE training model to obtain the prediction model.

[0127] In an optional embodiment, sub-step A1 may specifically include:

[0128] Sub-step A11: Obtain at least one first-class triplet and at least one second-class triplet from the model training set, and replace the content in the first-class triplet and the second-class triplet with semantically incorrect content to obtain erroneous triplets.

[0129] For reference Figure 2 If the tail entity in the triplet shown is replaced to (Zhang San, Employment, Company B), then since Zhang San's employment has changed and there is an obvious error, the triplet can be considered an incorrect triplet.

[0130] Sub-step A12: Obtain the preset model loss function.

[0131] A loss function is a function that maps an event (an element in a sample space) to a real number that expresses the economic cost or opportunity cost associated with that event. In layman's terms, in statistics, a loss function is a function that measures the degree of loss and error (loss that is related to "incorrect" estimation, such as loss of expenses or equipment). The loss function describes the loss of a system under different parameter values.

[0132] In this embodiment, the initial model includes an embedding layer and an embedding function. The embedding layer maps entities or relations to a vector space. The embedding function, based on the embedding results of entities and relations, maps triples to vectors between the embedding results and calculates the result, typically a real number, used to measure the validity of the triples. First, the model's embedding layer is initialized based on a Gaussian random distribution or existing pre-training results. Second, training is performed on the training set. Incorrect triples are generated by replacing a portion of the triples in the training set; the model's loss function is then defined as the margin loss. Finally, parameters are adjusted according to the loss function value.

[0133] Sub-step A13: Input the model training set containing the erroneous triples into the initial model for iterative training, and adjust the parameters of the initial model according to the function value of the model loss function.

[0134] Sub-step A14: When the number of iterations has been exhausted or the value of the loss function meets the preset conditions, stop the iterative training and obtain the prediction model.

[0135] For sub-steps A13-A14: Following sub-step A12, after obtaining the preset loss function, the initial model is trained. The training set triples are used to iterate multiple times until the number of iterations is exhausted or the loss function converges to a stable state (i.e., the change in the loss function does not exceed x%). The specific value of x can be adjusted according to the actual situation. This embodiment does not limit it here.

[0136] Step 405: Evaluate the accuracy of the sample triples in the knowledge graph based on the prediction model to obtain an evaluation score.

[0137] Reference Figure 6 , Figure 6This is a framework diagram of a knowledge graph accuracy evaluation method provided in an embodiment of this application. For example... Figure 6 As shown, the knowledge graph accuracy evaluation method based on embedded learning and fact verification first evaluates the embedded learning triples of the initial model in the following order: random sampling → extracting subgraphs → model training → obtaining scores → summarizing and processing.

[0138] After steps 401-405, the initial model has been trained, resulting in a predictive model that can be used to evaluate the task. After evaluating randomly selected triplet samples, a score is obtained for each triplet. For example... Figure 6 As shown, the next step is to select the triples with disputes in the score results, including: reading in the model results → sorting and filtering the scores → searching the path → calculating the RPI algorithm → summarizing and processing. For details, please refer to the following sub-steps.

[0139] In an optional embodiment, step 405 may specifically include:

[0140] Sub-step B1: Obtain the first score threshold and the second score threshold for the evaluation score;

[0141] Among the sample triplets, those whose evaluation scores are greater than or equal to the first score threshold and less than or equal to the second score threshold are selected as a class of target triplets.

[0142] As seen from the embedding learning results in the previous section (e.g., the ROC-AUC metric), embedding learning can distinguish between correct and incorrect triples to some extent. However, research indicates that the effectiveness of embedding learning is affected by the sparsity of the knowledge graph, and there are also cases where the scores of some triples contradict their actual correctness. Therefore, we define two types of controversial triples here: one type consists of triples whose embedding learning scores are in the middle range, making it difficult for the model to determine whether they are correct; the other type consists of triples that are affected by the sparsity of the knowledge graph and lack path verification. Screening out controversial triples can be used for subsequent semantic verification based on external information.

[0143] For triples with scores in the middle range, the model is more likely to make a direct classification error. Therefore, it is difficult to directly determine whether such triples are correct and further verification is required.

[0144] Sub-step B2: Obtain the topological path of all the sample triples in the knowledge graph through traversal search.

[0145] Sub-step B3: Determine the path support of the topological path and its corresponding sample triples according to the reachable path reasoning algorithm.

[0146] Sub-step B4: Select the sample triples in the training set whose path support is less than or equal to the first support threshold as the binary target triples.

[0147] Sub-step B5: Take the set of the first type of target triplets and the second type of target triplets as the set of target triplets.

[0148] Regarding sub-steps B2-B5: It should be noted that, in this embodiment, the upper and lower bounds of the midstream data can be freely chosen. In one optional embodiment (using the TransE model, where correct triplet scores are low), the lower bound of the score is chosen as the median score. Assuming that the number of correct triplets in the knowledge graph is greater than that of incorrect triplets, triplets below the median are likely to be correct triplets.

[0149] Furthermore, since sparse components in knowledge graphs often perform poorly in embedding models, for a given entity node with an in-degree of 1, if its triple is in the training set, the embedding result of this isolated entity is uniquely determined by the embedding results of the head entity and relation in the triple, failing to reflect effective information. Conversely, if its triple is in the test set, the entity embedding is not learned at all during training, resulting in a completely random and unfounded score. The situation is similar for entity nodes with lower degrees; their triples are also prone to evaluation errors.

[0150] Due to the long-tail effect in knowledge graphs (a few nodes connected to a large number of edges), knowledge graphs contain a large number of nodes with in-degree and out-degree values ​​far below the average. This makes it almost inevitable to handle sparse subgraphs in knowledge graphs in practice. Reducing the impact of sparsity is difficult, so we attempt to measure whether the embedding model's results are supported by a sufficient number of paths by analyzing the local support for the triple. We then filter out controversial triples for further validation. Since the search process requires multiple calls to the search engine, it generally takes a long time. Filtering out controversial triples also reduces the number of searches and improves efficiency.

[0151] Therefore, to handle sparse subgraphs in knowledge graphs and samples with difficult-to-distinguish evaluation scores, it is necessary to calculate the degree to which a reasoning path supports the triple. Specifically, after obtaining sample triples with mid-range scores, a search traversal is used to find all paths from the head entity to the tail entity of the triple on the knowledge graph for all samples. For example... Figure 2 In the example (Zhang San, address, Shanghai), there is a path from the beginning entity "Zhang San" to the end entity "Shanghai" (Zhang San, employment, company B), (company B, address, Shanghai), which supports the triple to some extent.

[0152] Subsequently, based on the Reachable Paths Inference (RPI) algorithm, the relevance of the aforementioned paths to the triplet samples is calculated. A threshold is set to select the triples with lower support as another group of disputed triples. For example, for... Figure 2 Triples with a support value less than 0.5 can be selected as the controversial first type of triples.

[0153] Reference Figure 7 , Figure 7 This is a distribution diagram illustrating the relationship between triplet support and evaluation scores, provided in an embodiment of this application. For example... Figure 7 As shown, both correct and incorrect triples exhibit insufficient support. Some incorrect triples are rated as true by the model embedding (the pure black data points on the left side of the figure) but have low support, indicating that although they have a better score, they have few actual support paths and belong to a set of controversial triples.

[0154] By further testing controversial triples, evaluation error can be reduced without requiring excessive validation based on external information, thus achieving a balance between evaluation error and evaluation time.

[0155] By combining all the determined sample triples with the first type of triples, we can obtain the set of the target triples.

[0156] Step 406: Verify the content of the target triple using external open-source information to obtain the accuracy analysis results for the knowledge graph.

[0157] Continue to refer to Figure 6 After selecting the controversial triples, verification can be performed based on external information. This includes: open-source information search → word segmentation statistics to be verified → triple verification → summary processing.

[0158] This step can be referred to in detail in step 304 above, and will not be repeated here in this embodiment.

[0159] In an optional embodiment, step 406 may specifically include:

[0160] Sub-step C1: For the target triple, construct search keywords, which include: strings consisting of head entities and relation entities, and / or strings consisting of head entities and tail entities.

[0161] Sub-step C2: Obtain the word segment to be verified based on the search keywords. The word segment to be verified is the string retained after removing the search keywords from the target triplet.

[0162] Sub-step C3: Search the word segment to be verified using an open-source search engine to obtain search results.

[0163] Sub-step C4: Calculate the frequency of the word segment to be verified in the search results.

[0164] Sub-step C5: Determine the accuracy of the knowledge graph based on the word frequency and the preset word frequency threshold.

[0165] For sub-steps C1-C5: For disputed triples involving factual and semantic verification, the steps of manual verification can be simulated. Following the standards of applications such as DeFacto, based on word frequency statistics and keyword co-occurrence, a search engine is used to search for the triples to be verified and match web page content with the triples. The search results can not only be used to complete the verification of the triples, but also to search for relevant evidence online, facilitating subsequent manual verification and evaluation.

[0166] In this application's embodiments, methods such as manual detection, question answering of other knowledge graphs, and fact verification based on network information can be employed. Based on the results of random sampling, it is estimated that some correct triples can find corresponding evidence, while most incorrect triples cannot. According to simulation experiments, validation was performed on samples drawn from the FB15K-237 and YAGO3 datasets, with an experimental error of 6%. However, its classification accuracy is relatively low compared to manual evaluation, illustrating the uncertainty of network information and the limitations of fact verification methods. With the continuous development of fact verification, intelligent search, and other methods, the accuracy of this evaluation will continue to improve.

[0167] Specifically, based on open-source search information, for controversial triples (h,r,t), search keywords are constructed and searched using a search engine. The search keywords correspond to strings consisting of head entities and relations, or strings consisting of head entities and tail entities.

[0168] Next, the unverified word segments in the string are counted. The unverified word segments are the strings after removing the search keywords from the disputed triples.

[0169] Next, based on the web page content obtained from the search, the frequency of the word segment to be verified in each search result web page is statistically calculated (the frequency of the word segment and its appearance in the browsed web page).

[0170] For triple verification, a word frequency threshold can be set. This determines whether a disputed triple can be verified using a webpage as evidence; if the frequency of the word segment to be verified is less than the threshold, it is considered insufficient evidence.

[0171] Since the knowledge graphs currently used are all relatively well-known open-source knowledge graphs, and evidence for these open-source information is relatively easy to find on the internet, for triples such as (Barack_Obama, is Married to, Michelle_Obama), after string processing, searching for the keywords "Barack_Obama" and "married" reveals that "Michelle" appears 30 times in each of the first 6 documents, thus verifying the correctness of the triple. After randomly sampling 50 sets of triples from knowledge graphs such as FB15K-237 and YAGO3-10, it can be found that the method can find corresponding evidence for most genuine triples (close to 60%). However, for most erroneous triples, this method struggles to find evidence.

[0172] In summary, the accuracy analysis method for knowledge graphs provided in this application firstly randomly selects at least two sample triples from the knowledge graph. Each sample triple includes at least a head entity and a tail entity. Then, based on the topological relationship between the head and tail entities in the knowledge graph, a model training set is obtained. This training set includes a first type of knowledge subgraph for the head entity and a second type of knowledge subgraph for the tail entity. The initial model is trained based on the triples in the first and second type of knowledge subgraphs in the training set to obtain a prediction model for analyzing the accuracy of the knowledge graph. Next, the accuracy of the sample triples in the knowledge graph is evaluated using the prediction model, and an evaluation score is obtained. Based on the evaluation score, target triples that do not meet preset conditions are identified. Finally, the content of the target triples is verified using external open-source information to obtain the accuracy analysis results for the knowledge graph. By extracting a subset of triples from the knowledge graph as sample triples and training an initial model on a subgraph representing the head and tail nodes of these sample triples, a predictive model for analyzing the accuracy of the knowledge graph is obtained. This trained model is then used to back-evaluate all sample triples. Based on statistical theory, evaluating the accuracy of the entire knowledge graph by randomly selecting a subset of samples reflects the overall content correctness of the knowledge graph. This effectively shortens the evaluation time for knowledge graph accuracy. Furthermore, training targeted evaluation models on different types of knowledge graphs using their subgraphs beforehand provides greater specificity and applicability, further refining the methods for evaluating the accuracy of knowledge graphs.

[0173] Reference Figure 8 , Figure 8 This is a schematic diagram of the functional module structure of a knowledge graph accuracy analysis device 50 provided in an embodiment of this application. For example... Figure 8 As shown, the device includes:

[0174] The first acquisition module 501 is used to randomly acquire at least two sample triples from the knowledge graph, wherein the sample triples include at least a head entity and a tail entity;

[0175] The training set generation module 502 is used to obtain a model training set based on the topological relationship between the head entity and the tail entity in the knowledge graph. The model training set includes: a first type of knowledge subgraph for the head entity and a second type of knowledge subgraph for the tail entity; wherein, the first type of knowledge subgraph is composed of a first type of triples associated with the head entity, and the second type of knowledge subgraph is composed of a second type of triples associated with the tail entity.

[0176] The model training execution module 503 is used to train the initial model based on the first type of triples of the first type of knowledge subgraph and the second type of triples of the second type of knowledge subgraph in the model training set, so as to obtain a prediction model for analyzing the accuracy of knowledge graphs.

[0177] The triplet evaluation module 504 is used to evaluate the accuracy of sample triplets in the knowledge graph according to the prediction model and obtain an evaluation score.

[0178] The triplet screening module 505 is used to determine, based on the evaluation score, target triplets in the sample triplets whose scores do not meet preset conditions.

[0179] The triple verification module 506 is used to verify the content of the target triple using external open-source information to obtain the accuracy analysis results for the knowledge graph.

[0180] In summary, the knowledge graph accuracy analysis device provided in this application first randomly selects at least two sample triples from the knowledge graph. Each sample triple includes at least a head entity and a tail entity. Then, based on the topological relationship between the head and tail entities in the knowledge graph, a model training set is obtained. This model training set includes a first type of knowledge subgraph for the head entity and a second type of knowledge subgraph for the tail entity. The initial model is trained based on the triples in the first and second type of knowledge subgraphs in the model training set to obtain a predictive model for analyzing the accuracy of the knowledge graph. By extracting a portion of the triples from the knowledge graph as sample triples and obtaining subgraphs for the head and tail nodes of the sample triples to train the initial model, a predictive model for analyzing the accuracy of the knowledge graph is obtained. Finally, the trained model is used to back-evaluate all sample triples. Based on statistical theory, the evaluation results obtained by randomly selecting a portion of the complete knowledge graph for evaluation can reflect the correctness of the entire knowledge graph content. This can effectively shorten the evaluation time for the correctness of the knowledge graph. Furthermore, the evaluation model obtained by training on the subgraphs of different types of knowledge graphs in advance has higher specific applicability, further improving the evaluation method for the accuracy of knowledge graphs.

[0181] Reference Figure 9 , Figure 9 This is a schematic diagram of the functional module structure of a prediction model generation device 80 provided in an embodiment of this application. For example... Figure 9 As shown, the device includes:

[0182] The first acquisition module 801 is used to randomly acquire at least two sample triples from the knowledge graph, wherein the sample triples include at least a head entity and a tail entity;

[0183] The training set generation module 802 is used to obtain a model training set based on the topological relationship between the head entity and the tail entity in the knowledge graph. The model training set includes: a first type of knowledge subgraph for the head entity and a second type of knowledge subgraph for the tail entity; wherein, the first type of knowledge subgraph is composed of a first type of triples associated with the head entity, and the second type of knowledge subgraph is composed of a second type of triples associated with the tail entity. Both the first type of triples and the second type of triples include: a head entity, a tail entity, and a relation entity used to represent the relationship between the head entity and the tail entity.

[0184] The model training execution module 803 is used to train the initial model based on the first type of triples of the first type of knowledge subgraph and the second type of triples of the second type of knowledge subgraph in the model training set, so as to obtain a prediction model for analyzing the accuracy of knowledge graphs.

[0185] Optionally, the training set generation module 802 may further include:

[0186] The first type of knowledge subgraph acquisition submodule is used to perform a breadth-first traversal search on the knowledge graph starting from the head entity, and obtain the first type of knowledge subgraph for the head entity when the breadth of the traversal search meets the preset conditions.

[0187] The second type of knowledge subgraph acquisition submodule is used to perform a breadth-first traversal search on the knowledge graph starting from the tail entity. If the breadth of the traversal search meets the preset conditions, a second type of knowledge subgraph for the tail entity is obtained.

[0188] The sample triplet removal submodule is used to remove the relation entity of the sample triplet from the first type of knowledge subgraph when the sample triplet is contained in the first type of knowledge subgraph; and to remove the relation entity of the sample triplet from the second type of knowledge subgraph when the sample triplet is contained in the second type of knowledge subgraph.

[0189] The model training set determination submodule is used to save the contents of the first type of knowledge subgraph and the second type of knowledge subgraph after removing relation entities as different subgraph files in the form of triple text, thereby obtaining the model training set.

[0190] Optionally, the model training execution module 803 may further include:

[0191] The initial model training submodule is used to train the initial model using the TransE training model based on the model training set, thereby obtaining the prediction model.

[0192] Optionally, the initial model training submodule may further include:

[0193] An error triple generation unit is used to obtain at least one first-class triple and at least one second-class triple from the model training set, and replace the contents of the first-class triple and the second-class triple with semantically incorrect content to obtain error triples.

[0194] The loss function acquisition unit is used to acquire the preset model loss function;

[0195] An iterative training execution unit is used to input the model training set containing erroneous triples into an initial model for iterative training, and to adjust the parameters of the initial model according to the function value of the model loss function;

[0196] The prediction model determination unit is used to stop iterative training and obtain the prediction model when the number of iterations has been exhausted or the function value of the loss function meets the preset conditions.

[0197] In summary, the prediction model generation device provided in this application addresses the problem of uncontrollable runtime in embedding learning models by combining a pre-trained model with the subgraph containing the extracted samples. It evaluates and scores a subset of triples in a knowledge graph using a prediction model trained on randomly sampled triples. Subsequently, a manually set evaluation threshold selects target triples with disputed scores for correctness verification using external information. By using evaluation results obtained from randomly sampled data to reflect the overall correctness of the knowledge graph, the evaluation time for knowledge graph correctness can be effectively shortened. Furthermore, pre-training targeted evaluation models on different types of knowledge graphs using their subgraphs provides higher specific applicability.

[0198] Reference Figure 10 , Figure 10 This is a schematic diagram of the functional module structure of a knowledge graph accuracy evaluation device 90 provided in an embodiment of this application. For example... Figure 10 As shown, the device includes:

[0199] The second acquisition module 901 is used to acquire a prediction model for analyzing the accuracy of the knowledge graph and at least two sample triples in the knowledge graph, wherein the prediction model is trained by the prediction model generation method according to any one of claims 1-4; and the knowledge graph is the knowledge graph used in the prediction model generation method according to any one of claims 1-4.

[0200] The triplet evaluation module 902 is used to evaluate the accuracy of sample triplets in the knowledge graph according to the prediction model and obtain an evaluation score.

[0201] The triplet screening module 903 is used to determine, based on the evaluation score, target triplets in the sample triplets whose scores do not meet preset conditions.

[0202] The triple verification module 904 is used to verify the content of the target triple using external open-source information to obtain the accuracy analysis results for the knowledge graph.

[0203] Optionally, the triplet screening module 903 may further include:

[0204] The score threshold acquisition submodule is used to obtain the first score threshold and the second score threshold for the evaluation score;

[0205] A target triplet determination submodule is used to select sample triplets whose evaluation scores are greater than or equal to a first score threshold and less than or equal to a second score threshold as target triplets.

[0206] The topology path determination submodule is used to obtain the topology path of all the sample triples in the knowledge graph through traversal search;

[0207] The path support calculation submodule is used to determine the path support between the topological path and the corresponding sample triples according to the reachable path inference algorithm.

[0208] The binary target triplet determination submodule is used to select the sample triplets in the training set whose path support is less than or equal to a first support threshold as binary target triplets.

[0209] The target triple determination submodule is used to take the set of the first type of target triples and the second type of target triples as the set of target triples.

[0210] Optionally, the triplet verification module 904 may further include:

[0211] The keyword construction submodule is used to construct search keywords for the target triple, wherein the search keywords include: a string consisting of a head entity and a relation entity, and / or a string consisting of a head entity and a tail entity;

[0212] The submodule for obtaining the segmentation to be verified is used to obtain the segmentation to be verified based on the search keywords. The segmentation to be verified is the string retained after removing the search keywords from the target triple.

[0213] An open-source retrieval execution submodule is used to retrieve the words to be verified through an open-source search engine and obtain retrieval results;

[0214] The search result determination submodule is used to count the frequency of the word segment to be verified in the search results;

[0215] The knowledge graph accuracy determination submodule is used to determine the accuracy of the knowledge graph based on the word frequency and a preset word frequency threshold.

[0216] In summary, the knowledge graph accuracy evaluation device provided in this application evaluates and scores a subset of triples in the knowledge graph using a prediction model trained on randomly sampled triples. Subsequently, a manually set evaluation threshold selects target triples with disputed scores for correctness verification using external information. Evaluating the entire knowledge graph content using randomly selected samples reflects the overall content correctness, effectively shortening the evaluation time. Furthermore, training specific evaluation models on different types of knowledge graphs using their subgraphs beforehand provides higher applicability. The use of fact-checking semantically supplements and validates the current method, effectively supporting knowledge graph accuracy evaluation.

[0217] Figure 11 This is a block diagram illustrating an electronic device 600 according to an exemplary embodiment. For example, the electronic device 600 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0218] Reference Figure 11 The electronic device 600 may include one or more of the following components: a processing component 602, a memory 604, a power supply component 606, a multimedia component 608, an audio component 610, an input / output (I / O) interface 612, a sensor component 614, and a communication component 616.

[0219] Processing component 602 typically controls the overall operation of electronic device 600, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 602 may include one or more processors 620 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 602 may include one or more modules to facilitate interaction between processing component 602 and other components. For example, processing component 602 may include a multimedia module to facilitate interaction between multimedia component 608 and processing component 602.

[0220] Memory 604 is used to store various types of data to support the operation of electronic device 600. Examples of such data include instructions for any application or method operating on electronic device 600, contact data, phonebook data, messages, pictures, multimedia, etc. Memory 604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0221] Power supply component 606 provides power to various components of electronic device 600. Power supply component 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 600.

[0222] Multimedia component 608 includes a screen that provides an output interface between the electronic device 600 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 608 includes a front-facing camera and / or a rear-facing camera. When the electronic device 600 is in an operating mode, such as a shooting mode or a multimedia mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0223] Audio component 610 is used to output and / or input audio signals. For example, audio component 610 includes a microphone (MIC) used to receive external audio signals when electronic device 600 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 604 or transmitted via communication component 616. In some embodiments, audio component 610 also includes a speaker for outputting audio signals.

[0224] I / O interface 612 provides an interface between processing component 602 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0225] Sensor assembly 614 includes one or more sensors for providing state assessments of various aspects of electronic device 600. For example, sensor assembly 614 can detect the on / off state of electronic device 600, the relative positioning of components such as the display and keypad of electronic device 600, changes in position of electronic device 600 or a component of electronic device 600, the presence or absence of user contact with electronic device 600, orientation or acceleration / deceleration of electronic device 600, and temperature changes of electronic device 600. Sensor assembly 614 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 614 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 614 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0226] Communication component 616 facilitates wired or wireless communication between electronic device 600 and other devices. Electronic device 600 can access wireless networks based on communication standards, such as WiFi, carrier networks (such as 2G, 3G, 4G, or 5G), or combinations thereof. In one exemplary embodiment, communication component 616 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 616 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0227] In an exemplary embodiment, the electronic device 600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to implement a knowledge graph accuracy analysis method provided in this application embodiment.

[0228] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 604 including instructions, which can be executed by a processor 620 of an electronic device 600 to perform the above-described method. For example, the non-transitory storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0229] Figure 12This is a block diagram illustrating an electronic device 700 according to an exemplary embodiment. For example, the electronic device 700 may be provided as a server. (Refer to...) Figure 12 The electronic device 700 includes a processing component 722, which further includes one or more processors, and memory resources represented by a memory 732 for storing instructions, such as application programs, that can be executed by the processing component 722. The application programs stored in the memory 732 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 722 is configured to execute instructions to perform a knowledge graph accuracy analysis method provided in embodiments of this application.

[0230] Electronic device 700 may also include a power supply component 726 configured to perform power management of electronic device 700, a wired or wireless network interface 750 configured to connect electronic device 700 to a network, and an input / output (I / O) interface 758. Electronic device 700 may operate on an operating system stored in memory 732, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or similar.

[0231] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0232] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for evaluating the accuracy of a knowledge graph, characterized in that, The method includes: Obtain a prediction model for analyzing the accuracy of a knowledge graph, and at least two sample triples from the knowledge graph; the prediction model is trained using a prediction model generation method. The accuracy of the sample triples in the knowledge graph is evaluated based on the prediction model to obtain an evaluation score. Based on the evaluation scores, target triplets in the sample triplets whose scores do not meet the preset conditions are identified; The content of the target triple is verified by external open-source information to obtain the accuracy analysis results for the knowledge graph. The step of determining the target triplet whose score does not meet the preset conditions in the sample triplet based on the evaluation score includes: Obtain the first and second score thresholds for the evaluation scores; Select the sample triplets whose evaluation scores are greater than or equal to the first score threshold and less than or equal to the second score threshold as a target triplet class. By traversing and searching, the topological paths of all the sample triples in the knowledge graph are obtained; Based on the reachable path reasoning algorithm, determine the path support of the topological path and its corresponding sample triples; Select the sample triplets in the training set whose path support is less than or equal to the first support threshold as binary target triplets; The set of the first type of target triplet and the second type of target triplet is taken as the set of target triplets; The prediction model generation method includes: At least two sample triples are randomly selected from the knowledge graph, and the sample triples include at least a head entity and a tail entity; Based on the topological relationship between the head entity and the tail entity in the knowledge graph, a model training set is obtained. The model training set includes: a first type of knowledge subgraph for the head entity and a second type of knowledge subgraph for the tail entity; wherein, the first type of knowledge subgraph is composed of a first type of triples associated with the head entity, and the second type of knowledge subgraph is composed of a second type of triples associated with the tail entity. Both the first type of triples and the second type of triples include: a head entity, a tail entity, and a relation entity used to represent the relationship between the head entity and the tail entity. Based on the first type of triples of the first type of knowledge subgraph and the second type of triples of the second type of knowledge subgraph in the model training set, the initial model is trained to obtain a prediction model for analyzing the accuracy of knowledge graphs.

2. The method according to claim 1, characterized in that, The target triple includes: a head entity, a tail entity, and a relation entity used to characterize the relationship between the head entity and the tail entity; The step of verifying the content of the target triples using external open-source information to obtain accuracy analysis results for the knowledge graph includes: For the target triple, search keywords are constructed, including: strings consisting of head entities and relation entities, and / or strings consisting of head entities and tail entities; The word segment to be verified is obtained based on the search keywords, and the word segment to be verified is the string retained after removing the search keywords from the target triple; The word segmentation to be verified was retrieved using an open-source search engine to obtain search results; Statistically analyze the word frequency of the segmented words to be verified in the search results; The accuracy of the knowledge graph is determined based on the word frequency and a preset word frequency threshold.

3. The method according to claim 1, characterized in that, The sample triplet also includes: a relation entity used to characterize the relationship between the head entity and the tail entity; The step of obtaining the model training set based on the topological relationship between the head entity and the tail entity in the knowledge graph includes: Starting from the head entity, a breadth-first traversal search is performed on the knowledge graph. If the breadth of the traversal search meets the preset conditions, a first type of knowledge subgraph for the head entity is obtained. Starting from the tail entity, a breadth-first traversal search is performed on the knowledge graph. If the breadth of the traversal search meets the preset conditions, a second type of knowledge subgraph for the tail entity is obtained. If the first type of knowledge subgraph contains the sample triplet, remove the relation entity of the sample triplet from the first type of knowledge subgraph; if the second type of knowledge subgraph contains the sample triplet, remove the relation entity of the sample triplet from the second type of knowledge subgraph. The contents of the first type of knowledge subgraph and the second type of knowledge subgraph after removing relation entities are saved as different subgraph files in the form of triple text, thus obtaining the training set of the model.

4. The method according to claim 1, characterized in that, The step of training an initial model using an embedded learning model based on the model training set to obtain the prediction model used to evaluate the accuracy of the knowledge graph includes: Based on the model training set, the initial model is trained using the TransE training model to obtain the prediction model.

5. The method according to claim 4, characterized in that, The step of training the initial model using an embedded learning model via the TransE training model to obtain the prediction model includes: Obtain at least one first-class triplet and at least one second-class triplet from the model training set, and replace the contents of the first-class triplet and the second-class triplet with semantically incorrect contents to obtain erroneous triplets; Obtain the preset model loss function; The training set of the model containing the erroneous triples is input into the initial model for iterative training, and the parameters of the initial model are adjusted according to the function value of the model loss function; When the number of iterations is exhausted or the value of the loss function meets the preset conditions, the iterative training is stopped and the prediction model is obtained.

6. An accuracy evaluation device for a knowledge graph, characterized in that, The device includes: The second acquisition module is used to acquire a prediction model for analyzing the accuracy of the knowledge graph and at least two sample triples in the knowledge graph; the prediction model is obtained by training a prediction model generation method. The triplet evaluation module is used to evaluate the accuracy of sample triplets in the knowledge graph based on the prediction model and obtain an evaluation score. The triplet screening module is used to determine, based on the evaluation scores, target triplets in the sample triplets whose scores do not meet preset conditions. The triple verification module is used to verify the content of the target triple using external open-source information to obtain the accuracy analysis results for the knowledge graph. The triplet screening module includes: The score threshold acquisition submodule is used to obtain the first score threshold and the second score threshold for the evaluation score; A target triplet determination submodule is used to select sample triplets whose evaluation scores are greater than or equal to a first score threshold and less than or equal to a second score threshold as target triplets. The topology path determination submodule is used to obtain the topology path of all the sample triples in the knowledge graph through traversal search; The path support calculation submodule is used to determine the path support between the topological path and the corresponding sample triples according to the reachable path inference algorithm. The binary target triplet determination submodule is used to select sample triplets in the training set whose path support is less than or equal to a first support threshold as binary target triplets. The target triplet determination submodule is used to take the set of the first type of target triplet and the second type of target triplet as the set of target triplet; The prediction model generation method includes: At least two sample triples are randomly selected from the knowledge graph, and the sample triples include at least a head entity and a tail entity; Based on the topological relationship between the head entity and the tail entity in the knowledge graph, a model training set is obtained. The model training set includes: a first type of knowledge subgraph for the head entity and a second type of knowledge subgraph for the tail entity; wherein, the first type of knowledge subgraph is composed of a first type of triples associated with the head entity, and the second type of knowledge subgraph is composed of a second type of triples associated with the tail entity. Both the first type of triples and the second type of triples include: a head entity, a tail entity, and a relation entity used to represent the relationship between the head entity and the tail entity. Based on the first type of triples of the first type of knowledge subgraph and the second type of triples of the second type of knowledge subgraph in the model training set, the initial model is trained to obtain a prediction model for analyzing the accuracy of knowledge graphs.

7. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is enabled to perform the method as described in any one of claims 1 to 5.

Images