Data blood relationship analysis method based on knowledge graph expansion

Abstract

The invention discloses a data blood relationship analysis method based on knowledge graph expansion, relates to the technical field of knowledge graphs, and solves the problem that new information cannot be explored and added through an automatic means because the content of the knowledge graph is limited in a data range on which the knowledge graph depends when the knowledge graph is initially constructed. The method comprises the following steps: collecting metadata related to data consanguinity and constructing a knowledge graph; obtaining an expanded knowledge graph by using an inference engine according to a structure and a relationship in the power grid knowledge graph; based on the expanded knowledge graph, data consanguinity extraction and data consanguinity analysis are carried out, and the data consanguinity extraction specifically refers to query of data consanguinity information in the knowledge graph; the data consanguinity analysis specifically determines whether data loss or conversion errors occur. Through continuous expansion and optimization, the knowledge graph can better serve stable operation of the power grid and optimal configuration of resources, and powerful data support is provided for intelligent management and decision making of the power grid.

Description

Technical Field

[0001] The present invention relates to the field of knowledge graph technology, and in particular to a data kinship analysis method based on knowledge graph expansion. Background Art

[0002] The construction of knowledge graphs provides machines with rich background knowledge by converting massive amounts of information into structured entities and relationships, thereby enhancing their understanding and reasoning capabilities. In this process, data lineage plays a vital role. It not only records the source, flow, and transformation of data, but also provides a basis for data governance, quality control, and decision support. Through data lineage, we can trace the history of data changes, evaluate the quality and consistency of data, and understand the impact of data on business processes. This transparency and traceability are essential to ensuring the reliability and security of data, especially in critical infrastructure areas such as power grids, where the accuracy of data is directly related to the stable operation of the power grid and the optimal allocation of resources.

[0003] Existing knowledge graph construction technologies often lack a key link after completing the initial graph creation, that is, dynamically expanding the knowledge graph through the reasoning engine. This deficiency means that the content of the knowledge graph is limited to the data it relies on when it was originally built, and it is impossible to discover and add new information through automated means. This limitation seriously hinders the potential of knowledge graphs in revealing deep data relationships and patterns. For example, in the application of power grid knowledge graphs, if the reasoning engine cannot be used to identify and predict potential failure modes in the power grid or optimize the path of power flow, the practicality and foresight of the knowledge graph will be greatly reduced. Therefore, integrating the reasoning engine into the construction and maintenance process of the knowledge graph is crucial to achieving the continued growth and in-depth application of the knowledge graph.

[0004] In view of this, a data kinship analysis method based on knowledge graph expansion is proposed. Summary of the invention

[0005] In view of the problem that the content of the knowledge graph in the prior art is limited to the data range on which it was originally built, and it is impossible to discover and add new information through automated means, the present invention provides a data kinship analysis method based on knowledge graph expansion, which can improve the intelligence level of power grid monitoring and enhance the ability to predict the operation status of the power grid, thereby ensuring the efficient and stable operation of the power grid. The specific technical solution is as follows:

[0006] A data kinship analysis method based on knowledge graph expansion includes the following steps:

[0007] Collect metadata related to data lineage, including data sources, data tables, fields, conversion logic, business rules, and build a knowledge graph;

[0008] Utilize the structure and relationship in the power grid knowledge graph, apply reasoning rules and logic to derive new knowledge, and use the reasoning engine to obtain the expanded knowledge graph;

[0009] Based on the expanded knowledge graph, data lineage extraction and data lineage analysis are performed. The data lineage extraction specifically involves querying the data lineage information in the knowledge graph, including the extraction of data flow; the data lineage analysis specifically involves comparing the data in the source table and the target table to determine whether data is lost or converted incorrectly.

[0010] Preferably, the specific steps for constructing a knowledge graph are as follows:

[0011] Collect metadata related to data lineage, including data sources, data tables, fields, conversion logic, business rules, and other information;

[0012] Convert unstructured text data into structured knowledge representation;

[0013] Integrate and fuse knowledge from different sources, including entity disambiguation and coreference resolution;

[0014] Define entity types and their attributes based on entity modeling results, define relationship types and their constraints based on relationship modeling results, and then integrate these modeling results to build an ontology;

[0015] Map the constructed ontology into the graph database and create corresponding nodes, edges and their attributes to effectively store and manage knowledge data;

[0016] The structured, semi-structured and unstructured data after entity alignment are imported into the ontology using graph query language to obtain standard knowledge representation.

[0017] Preferably, entity disambiguation specifically includes the following steps:

[0018] Candidate entity generation: Based on the textual form of the entity reference, a set of candidate entities is generated from the knowledge graph or other data sources. These candidate entities may be related to the entity reference.

[0019] Context feature extraction: Analyze the context of entity reference, including contextual vocabulary, syntactic structure, entity relationship, etc., and extract features that help distinguish candidate entities;

[0020] Feature matching and similarity calculation: Match the context features with the features of the candidate entities and calculate the similarity to determine which candidate entity best matches the entity reference;

[0021] Disambiguation decision: Based on the results of matching and similarity calculation, select the best candidate entity as the disambiguation result of the entity reference.

[0022] Preferably, coreference resolution specifically includes associating entity references having a coreference relationship to the same entity representation.

[0023] Preferably, before performing data lineage extraction and data lineage analysis, and after obtaining the expanded knowledge graph, the following steps are also included:

[0024] Obtain the knowledge graph before and after the inference engine;

[0025] The quality of the knowledge graph is calculated by the following function:

[0026]

[0027] Where C is the confidence level, which ranges from 0 to 1; Ps is the precision; R S is the recall rate; T S is the score, which is the harmonic mean of precision and recall, T S It can be calculated by the following formula:

[0028]

[0029] Set a confidence threshold, compare the calculated confidence with the confidence threshold, and determine whether data lineage extraction and data lineage analysis are necessary for the knowledge graph.

[0030] Preferably, the data lineage extraction includes obtaining data flow information, and the specific operations are as follows:

[0031] Use the graph query language Cypher provided by the graph database to track the flow of data from the source entity to the target entity. Based on this query language, generate a data flow report, including the starting point, end point, and various processing steps of the data.

[0032] Preferably, the data lineage analysis includes analyzing whether data is lost, and the criterion for determining whether data is lost is: data that exists in the source table does not exist in the target table.

[0033] Preferably, the data lineage analysis includes whether an erroneous conversion occurs, and a judgment criterion for an erroneous conversion is that the attributes stored in the source table are inconsistent with the attributes at the corresponding location in the target table.

[0034] A computer-readable storage medium, comprising a stored program, wherein when the program is run, the device where the computer-readable storage medium is located is controlled to execute the data kinship analysis method based on knowledge graph expansion as described above.

[0035] A processor is used to run a program, wherein when the program is running, the data blood relationship analysis method based on knowledge graph expansion as described above is executed.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] The present invention can achieve comprehensive management and optimization of the knowledge graph of the power grid by comprehensively applying technical solutions such as data lineage extraction, knowledge graph construction, reasoning engine expansion, and data lineage analysis. First, by collecting metadata related to the power grid, including data sources, data tables, fields, conversion logic, and business rules, we can build a knowledge graph that fully reflects the data structure and relationship of the power grid. This process not only involves the collection and organization of data, but also includes the understanding and representation of complex relationships between data. Then, by using the reasoning engine to expand the knowledge graph, new knowledge can be automatically derived, thereby enriching the content of the graph and enhancing its expression ability. This step is crucial because it can not only discover the implicit relationship between data, but also predict potential data problems and trends, and provide support for intelligent management and decision-making of the power grid. For example, through reasoning analysis, we can predict possible fault points in the power grid, perform maintenance and optimization in advance, and thus improve the stability and reliability of the power grid. In summary, the present invention can not only improve the construction quality of the knowledge graph of the power grid, but also enhance its practicality and effectiveness in power grid management. Through continuous expansion and optimization, the knowledge graph can better serve the stable operation of the power grid and the optimal allocation of resources, and provide strong data support for the intelligent management and decision-making of the power grid. In the field of power grid, this means fewer unexpected power outages, more efficient resource allocation, and better monitoring of the health of the power grid. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for the specific embodiments or the description of the prior art. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn according to the actual scale.

[0039] Figure 1 is a flow chart of the method of the present invention;

[0040] Figure 2 Constructing a flow chart for the knowledge graph of the present invention;

[0041] Figure 3 The figure is a flow chart of entity disambiguation of the present invention. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0044] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0045] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0046] The following is based on Figure 1-Figure 3 , the embodiments of the present invention are further described.

[0047] In one embodiment of the present invention, a method for analyzing data kinship based on knowledge graph expansion is provided, comprising the following steps:

[0048] S1: Collect metadata related to data lineage, including data sources, data tables, fields, conversion logic, business rules, and build a knowledge graph;

[0049] The specific process of this step is as follows:

[0050] 1. Identify areas and needs

[0051] In the domain understanding and demand analysis of the power sector, we must first identify key concepts such as power generation, transmission, distribution, and load management. Then, we must determine the entities (such as power plants, substations, transmission lines, etc.), attributes (such as power generation capacity, operating status), and relationships (such as power flow, connections between devices) of these concepts. Next, we must work with experts in the power sector to clarify the goals and scope of analyzing data lineage relationships, and determine the data sets and related business processes that need to be analyzed.

[0052] 2. Data collection and integration

[0053] Collect metadata related to data lineage, including data sources, data tables, fields, conversion logic, business rules, etc. These metadata related to data lineage include structured data, semi-structured data, and unstructured data.

[0054] Structured data lineage can track the change history of a database table; semi-structured data lineage may involve tracking the conversion path of a JSON file in the data processing flow; unstructured data lineage may track the editing history and source of a document in multiple versions.

[0055] 3. Extract knowledge from multi-source data

[0056] The metadata related to data lineage comes from many source systems and channels, including structured data from relational databases, semi-structured data collected through OGG or pushed by Kafka, and unstructured data such as docx documents. Knowledge extraction refers to the process of automatically extracting meaningful knowledge and information from unstructured, semi-structured or structured data. These data can be text, web pages, project documents, database tables, real-time streaming data, etc.

[0057] Knowledge extraction aims to transform unstructured text data into structured knowledge representations so that computers can understand and process them.

[0058] The data extracted from the knowledge graph is mainly divided into the following three types of data:

[0059] (1) Facing structured data

[0060] Power structured data is a form of data storage in a table-like form similar to that in a relational database, in which there are clear relationship names and corresponding relationships between items. This data model is widely used in the power field to describe the entities, attributes and the relationships between them in the power system. The process of processing power structured data includes data modeling, data import, data query and analysis, data cleaning and verification, data indexing and optimization, and data update and maintenance. First, when performing data modeling, we need to identify the entities in the power system, such as power plants, substations, transmission lines, etc., and their attributes, such as name, location, capacity, etc. At the same time, we also need to clarify the relationship between entities, such as connection relationship, power supply relationship, etc. By defining appropriate table structure, primary key, foreign key and constraint, the consistency and integrity of the data can be ensured. By extracting knowledge from power structured data, meaningful information and associations can be extracted from the data to support analysis, decision-making and application development in the power field. This may involve steps such as data understanding, entity extraction, attribute extraction, relationship extraction and knowledge representation to transform data into useful knowledge forms and provide deeper insights and applications for the power industry.

[0061] (2) Facing semi-structured data

[0062] Semi-structured data is a special form of structured data that is incompatible with traditional relational databases or other forms of data table structures, but uses tags or other markers to separate semantic elements and maintain the hierarchical structure of records and data fields. In infrastructure project documents, we can find rich semi-structured table data. The inventory of accepted equipment records the attribute data of various types of equipment used in the project, such as equipment type, material code, purchase price, etc. The test report records the parameter information and test data of the main equipment at the time of acceptance, such as the rated current, DC voltage, manufacturer, loop insulation test results, zero drift test results, etc. of the main transformer. These data have high accuracy and are in a wide range of applications, so they have become an important data source for building large-scale knowledge graphs and an important way to acquire knowledge.

[0063] From the perspective of project management, we can model the equipment used in the power grid infrastructure project as the entity node. At the same time, the descriptive data related to the equipment can also be modeled as the attributes of the entity node, such as the material name, material code, purchase price, etc. of the equipment. Through this modeling method, we can better manage and understand the equipment used in the project and its related information. This modeling method based on semi-structured data helps to improve the efficiency and accuracy of project management, and at the same time provides a valuable basis for further data analysis and decision-making.

[0064] Semi-structured data is a form between structured data and unstructured data. It does not strictly define the data model like the table structure in the traditional relational database, but contains tags or other markers to separate semantic elements to maintain the hierarchical structure of the data. This form of data is common in text documents, log files, web pages, etc. Due to the particularity of semi-structured data, traditional relational database management systems cannot process it directly. Therefore, when processing semi-structured data, the common method is to use specialized tools and technologies, such as XML (Extensible Markup Language) or JSON (JavaScript Object Notation). These tools and technologies can parse and extract information from semi-structured data and convert it into a structured form that can be analyzed and applied. In the field of engineering management, semi-structured data has important application value. The semi-structured table data in infrastructure engineering files contains key information such as equipment attributes, parameter information, and test data.

[0065] By extracting, organizing and analyzing these data, knowledge graph construction can be realized: semi-structured data is one of the important sources for building large-scale knowledge graphs. By modeling the entity nodes and attribute information in semi-structured data, a knowledge graph in the field of engineering management can be constructed to provide more comprehensive and accurate knowledge representation and query capabilities. Semi-structured data plays an important role in engineering management. They provide key information about equipment, attributes and test data, help optimize resources, support decision-making, and provide an important data foundation for the construction of knowledge graphs. Through appropriate tools and technologies, these data can be effectively extracted, processed and utilized to improve the efficiency and quality of engineering management.

[0066] (3) Unstructured data

[0067] In the unstructured natural language text of the power industry, relation extraction and attribute extraction remain important tasks of information extraction to identify semantic relations between entities and obtain attribute information of entities. The goal of relation extraction is to identify the relations between entities from the text. In the text of the power industry, there may be sentences or paragraphs describing the relations between entities such as equipment, components, suppliers, projects, etc. Through relation extraction, we can identify these relations, such as equipment A is provided by supplier B, equipment C is connected to equipment D, etc. These relations can help us establish connections between entities, thus forming a more complete knowledge graph or semantic network.

[0068] Attribute extraction is the process of extracting specific attribute information of entities from text. In the text of the power industry, entities can be equipment, components, technical specifications, etc. The goal of attribute extraction is to identify and extract the attributes of these entities, such as the model, manufacturer, rated power, etc. of the equipment. Through attribute extraction, we can obtain detailed features about the entity and further enrich the description and semantic information of the entity. The attribute extraction problem can be regarded as a special case of the relationship extraction problem, in which the attribute can be regarded as a relationship between the entity and the attribute value. For example, in a description "the model of equipment A is B", "equipment A" can be regarded as an entity, "model" as an attribute, and "B" as an attribute value. Therefore, attribute extraction can be performed under the framework of relationship extraction.

[0069] Relation extraction and attribute extraction have important application value in natural language texts in the power industry. They can help us extract useful structured information from a large amount of unstructured text and support knowledge management, intelligent analysis and decision-making in the power industry. These tasks can be solved by machine learning and natural language processing techniques, such as entity recognition, relationship classification and attribute extraction. Through relationship and attribute extraction, we can better understand the relationship between entities in the power industry text and obtain richer structured semantic information.

[0070] Specifically, structured data, semi-structured data and unstructured data are subjected to knowledge extraction respectively. Knowledge extraction includes entity extraction, relationship extraction and attribute extraction to generate preliminary knowledge representations, but these representations may still contain duplication, conflict and inconsistency.

[0071] 4. Knowledge fusion of multi-source data

[0072] Knowledge fusion refers to the integration and fusion of knowledge from different sources to form a more comprehensive, consistent and accurate knowledge representation. After extraction, the extracted information is integrated and normalized to resolve redundancy and conflict between data from different sources and ensure that all extracted information is represented under a unified semantic framework.

[0073] In knowledge fusion, entity recognition and disambiguation are important links. It involves identifying entity references from text and determining the specific entities to which these references correspond, mainly including entity disambiguation and coreference resolution.

[0074] 4.1. Entity disambiguation refers to solving the problem of entity reference ambiguity in text in natural language processing and information retrieval, that is, determining the specific entity that an entity reference should correspond to in a specific context. In the context of knowledge fusion, entity disambiguation is an important part of the knowledge fusion process, which is used to solve the problem of identifying and linking similar or identical entities in different data sources or knowledge graphs. When an entity reference appears in a text, there may be multiple candidate entities corresponding to it, which is caused by homonymous entities, synonyms, abbreviations, aliases, etc. The goal of entity disambiguation is to determine the correct meaning of the entity reference through contextual information and semantic reasoning technology, thereby eliminating the ambiguity of the entity and ensuring that it is correctly associated with the corresponding entity in the knowledge graph. The process of entity disambiguation usually includes the following steps: 1. Candidate entity generation: Based on the text form of the entity reference, a set of candidate entities are generated from the knowledge graph or other data sources, which may be related to the entity reference. 2. Context feature extraction: Analyze the context environment of the entity reference, including context vocabulary, syntactic structure, entity relationship, etc., and extract features that help distinguish candidate entities. 3. Feature matching and similarity calculation: Match the context features with the features of the candidate entities and calculate similarity to determine which candidate entity best matches the entity reference. 4. Disambiguation decision: Based on the results of matching and similarity calculation, select the best candidate entity as the disambiguation result of the entity reference. For example, the text "The equipment is a facility in the power grid that plays the role of voltage transformation, current transformation, and distribution of electric energy", then the entity candidate is equipment, and the entity in the knowledge graph is "Substation A: located in city X, belongs to the 500 kV level, and has a capacity of 100 MW". Entity disambiguation process: Based on the context information, we can infer that the "equipment" in the text refers to the facilities in the power grid for voltage transformation, current transformation, and distribution of electric energy. Based on this context, we can link the entity candidate "substation" to "substation A" in the knowledge graph, because they are both facilities in the power grid for voltage transformation and distribution of electric energy. The final link result: equipment->substation A.

[0075] 4.2 Co-reference resolution refers to solving the co-reference relationship problem in the text, that is, determining the same entity referred to by different expressions in the text. The goal of co-reference resolution is to determine the specific entities referred to by different expressions through context analysis and reasoning techniques, and associate them into the same co-reference chain. Co-reference resolution can help understand the complex semantic relationships in the text and improve the accuracy and semantic understanding ability of the natural language processing system.

[0076] In the process of knowledge fusion, coreference resolution is an important link. When there are problems in identifying and linking similar or identical entities in different data sources or knowledge graphs, coreference resolution technology can help determine which entity references are mutually referenced and represent the same entity. By associating entity references with coreference relationships to the same entity representation, the ambiguity of entities can be eliminated and the accuracy and consistency of knowledge fusion can be improved. Coreference resolution is to solve the problem that multiple words (referent terms) correspond to the same entity. For example, the text: Current overload on the transmission line will lead to increased losses, and timely measures need to be taken to prevent damage. In addition, the equipment in the substation also needs to be regularly inspected to ensure its normal operation. Coreference resolution process: In this example, there are multiple coreference relationships that need to be resolved. First, "current overload on the transmission line" and "lead to increased losses" refer to the same entity, that is, the overload condition. Secondly, the phrases "need to take timely measures to prevent damage" and "in addition" refer to the overload condition mentioned above. Finally, "the equipment of the substation" and "its normal operation" refer to the same entity, that is, the equipment of the substation. Final analysis result: current overload on transmission line -> overload condition; resulting in increased losses -> overload condition; timely measures need to be taken to prevent damage -> overload condition; in addition -> overload condition; equipment in substation -> equipment in substation; its normal operation -> equipment in substation.

[0077] Through entity disambiguation and coreference resolution, the extracted entities are aligned.

[0078] 5. Use the knowledge and specifications of domain experts to model the data and define entity categories, attribute definitions, and relationship types.

[0079] Knowledge modeling is the process of converting knowledge and information in the real world into a form that computers can understand and process. It involves the extraction, organization and representation of domain knowledge so that computer systems can reason, infer and apply it.

[0080] The following is the general knowledge modeling process:

[0081] Determine the domain and goal: Clarify the domain and goal to be modeled. For example, in the knowledge modeling of the power grid, the goal may be to describe the attributes and relationships of power equipment, substations, lines, etc.

[0082] Entity Modeling: Identify and define the entities that need to be modeled, which can be physical objects, concepts, events, etc. in the power grid. For example, generators, substations, loads, etc. Define unique identifiers and related attributes for each entity.

[0083] Relationship modeling: Define the relationships between entities, which can be connections, dependencies, hierarchies, etc. For example, the connection relationship between generators and substations, the dependency relationship between substations and lines, etc. Define unique identifiers and related attributes for each relationship.

[0084] Attribute modeling: define attributes for entities and relationships to describe their characteristics and properties, such as the power output attribute of a generator, the capacity attribute of a line, etc.

[0085] 6. Knowledge graph establishment and representation

[0086] Graph databases are database systems specifically designed to store and process graph data. They are widely used to represent and query knowledge graphs because knowledge graphs naturally have a graph structure and graph databases can effectively process the relationships and attributes between entities. Unlike relational databases, graph databases use a graph data model to represent data. Graphs are composed of nodes and edges, where nodes represent entities and edges represent relationships between entities. Each node and edge can have attributes that describe the characteristic information of the node and edge.

[0087] The following are some key concepts of graph databases when storing knowledge graphs:

[0088] 1. Node

[0089] Represents entities in the knowledge graph, such as people, places, concepts, etc. Each node can have attributes to describe the characteristics of the entity.

[0090] 2. Edge

[0091] Represents the relationship between entities in the knowledge graph, such as associations, attributes, etc. Each edge can have attributes to describe the characteristics of the relationship.

[0092] 3. Tags

[0093] Each node can be assigned one or more labels, which specify the type or category of the node.

[0094] 4. Attributes

[0095] Each node and edge can have attributes to store characteristic information related to the entity or relationship.

[0096] Using graph databases, you can flexibly query data in knowledge graphs and perform complex graph analysis and reasoning operations. For example, you can query the properties of specific nodes, find paths between nodes, discover patterns and clusters in graphs, and more.

[0097] Some common graph databases include Neo4j, Amazon Neptune, JanusGraph, etc. They provide a rich graph database query language and API, as well as efficient graph traversal and query performance, suitable for storing and processing large-scale knowledge graph data. Graph databases are a powerful tool for storing and querying knowledge graph data. They can represent entities and relationships in a graphical structure and provide a wealth of query and analysis functions, making the management and application of knowledge graphs more flexible and efficient.

[0098] 6.1. Define entity types and their attributes based on entity modeling results, define relationship types and their constraints based on relationship modeling results, and then integrate these modeling results to construct an ontology.

[0099] 6.2. Map the constructed ontology into the graph database and create corresponding nodes, edges and their attributes to effectively store and manage knowledge data.

[0100] 6.3. Use graph query languages ​​such as Cypher or SPARQL to import the structured, semi-structured and unstructured data after entity alignment into the ontology to obtain standard knowledge representation.

[0101] It should be noted that the above is only one way to obtain a knowledge graph. In actual operation, the knowledge graph can be directly downloaded or a knowledge graph can be constructed through other methods well known to technicians in this field for subsequent use.

[0102] S2: Utilize the structure and relationships in the power grid knowledge graph and apply reasoning rules and logic to derive new knowledge.

[0103] This step is mainly achieved through the reasoning engine, which can automatically identify implicit relationships and rules based on existing ontologies and data to generate additional knowledge. For example, if the knowledge graph contains the entity "transformer" and the relationship "connected to", the reasoning engine can infer all the transmission lines connected to a specific transformer, even if these connections are not directly mentioned in the original data.

[0104] An inference engine is a software component that uses formal logic to infer new facts and relationships in data. In the power grid knowledge graph, the inference engine can identify implicit relationships between entities, extend the graph content, and support decision making. A well-known inference engine is Apache Jena, which provides a framework for building applications that can process RDF (Resource Description Framework) graphs, perform inferences, and access RDF data stores. Apache Jena's features include RDF graph storage, SPARQL query engine, rule engine (for inference), and transformation tools. In the power grid knowledge graph, the inference engine can identify implicit relationships between entities, extend the graph content, and support decision making.

[0105] Furthermore, after the new knowledge is acquired in step S2, the confidence of the original knowledge and the inferred new knowledge is further acquired, and the confidence is evaluated to retain the knowledge with high confidence. This can be achieved by comparing the consistency of knowledge in different data sources and checking whether the relationships in the knowledge graph conform to the physical laws and business logic of the real world. For example, if there are significant differences in the parameters of a transformer collected from different data sources, the confidence of this knowledge may be low and further verification is required.

[0106] Let's define a function to evaluate the quality:

[0107]

[0108] Where C is the confidence level, which ranges from 0 to 1. The closer the value is to 1, the higher the confidence level is. Ps is the precision score, which is the number of correctly predicted positive examples divided by the number of all predicted positive examples (including true positive examples and false positive examples). It reflects the correctness of entities and relationships in the knowledge graph. S is the recall rate, which is the number of correctly predicted positive examples divided by the number of actual positive examples (including true positives and false negatives). It reflects the integrity of the knowledge graph. TSTS is the F1 score, which is the harmonic mean of precision and recall, and is used to measure the balance between precision and recall. It is a comprehensive indicator of the quality of the knowledge graph. In this formula, P S and R S It can be calculated using manually annotated sample data or by using external knowledge sources for comparative evaluation. S It can be calculated by the following formula:

[0109]

[0110] Evaluation criteria can be set based on business needs and historical data performance. For example, a threshold can be set, such as Cs >0.8, the quality of the knowledge graph is considered acceptable, otherwise the knowledge graph needs to be further cleaned and optimized.

[0111] Based on the result of this function, it can be determined whether the knowledge graph meets the evaluation criteria, thereby deciding whether to retain or remove inappropriate knowledge. For example, if the confidence of a knowledge graph is lower than the set threshold, the graph may need to be further reviewed and revised to improve its quality.

[0112] S3: Based on the expanded knowledge graph, perform data lineage extraction and data lineage analysis.

[0113] The data lineage extraction specifically uses the graph query language provided by the graph database, such as Cypher (used in Neo4j), to query and analyze the data lineage information in the graph, including:

[0114] Data flow: Identify the flow of data from the source table to the target table. Specifically, use a graph database such as Neo4j, which provides a powerful graph query language Cypher, which can be used to query and analyze data lineage information in the knowledge graph. Through the Cypher query language, trace the flow of data from the source entity to the target entity. In other words, after obtaining the entities and relationships in the knowledge graph, you can generate a data flow report based on the query language, including the starting point, end point, and various processing steps of the data.

[0115] The data lineage analysis is to compare the data of the source table and the target table to determine whether the data is lost or converted incorrectly;

[0116] Data loss: data in the source table does not exist in the target table;

[0117] Incorrect conversion: The source table stores attribute X of A, and the target table stores attribute Y of A, which changes the attribute value of A.

[0118] In addition, dependency analysis can also be performed: analyzing data dependencies to evaluate data integrity and reliability. Data integrity and reliability generally have data specifications when defining data. Data integrity can be checked from aspects such as whether the data is missing or whether the data is complete; data reliability can be checked from aspects such as whether the data meets the filling specifications and whether the data is valid; data dependency checks can be carried out from the primary and foreign key dependencies of the data.

[0119] By extracting and analyzing data lineage relationships, we can help adjust entities and relationships in real time, or update data mapping, etc., to ensure that new data can be accurately reflected in the knowledge graph.

[0120] Adjust entities and relationships: Adjust entities and relationships with problems based on the data quality check results. For example, if attribute X of entity A is empty, we need to supplement the value of attribute X of entity A according to the actual situation. There is a many-to-many relationship between entity A and entity B, but according to the actual situation, there should be a one-to-one relationship between the two. In this case, we analyze the cause, find out the wrong data, and adjust it.

[0121] Update data mapping: Data models are not static. Depending on business development needs, data models may need to be adjusted based on changes in business models. For example, previously there was a one-to-one relationship between entity A and entity B. Based on changes in the business model, entity B now needs to be adjusted to entity B and entity C. At this time, entity A and entity C are in a one-to-one relationship, and entity C and entity B are in a one-to-many relationship.

[0122] Those of ordinary skill in the art will appreciate that the units of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition of each example has been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0123] In the embodiments provided by the present invention, it should be understood that the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units can be combined into one unit, one unit can be split into multiple units, or some features can be ignored, etc.

[0124] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0125] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-0nlyMemory), random access memory (RAM, RandomAccessMemory), mobile hard disk, magnetic disk or optical disk, etc., which can store program code.

[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

Claims

1. A data kinship analysis method based on knowledge graph expansion, characterized in that: The following steps are involved: Collect metadata related to data lineage, including data sources, data tables, fields, conversion logic, business rules, and build a knowledge graph; Utilize the structure and relationship in the power grid knowledge graph, apply reasoning rules and logic to derive new knowledge, and use the reasoning engine to obtain the expanded knowledge graph; Based on the expanded knowledge graph, data lineage extraction and data lineage analysis are performed. The data lineage extraction specifically involves querying the data lineage information in the knowledge graph, including the extraction of data flow; the data lineage analysis specifically involves comparing the data in the source table and the target table to determine whether data is lost or converted incorrectly.

2. According to the method for analyzing data kinship based on knowledge graph expansion according to claim 1, it is characterized in that: The specific steps to build a knowledge graph are as follows: Collect metadata related to data lineage, including data sources, data tables, fields, conversion logic, business rules, and other information; Convert unstructured text data into structured knowledge representation; Integrate and fuse knowledge from different sources, including entity disambiguation and coreference resolution; Define entity types and their attributes based on entity modeling results, define relationship types and their constraints based on relationship modeling results, and then integrate these modeling results to build an ontology; Map the constructed ontology into the graph database and create corresponding nodes, edges and their attributes to effectively store and manage knowledge data; The structured, semi-structured and unstructured data after entity alignment are imported into the ontology using graph query language to obtain standard knowledge representation.

3. According to claim 2, a method for analyzing data kinship based on knowledge graph expansion is characterized in that: Entity disambiguation specifically includes the following steps: Candidate entity generation: Generate a set of candidate entities from the knowledge graph or other data sources based on the text form of the entity reference; Context feature extraction: extracting the context of entity reference, including contextual vocabulary, syntactic structure, and entity relationship; Feature matching and similarity calculation: Match the context features with the features of the candidate entities and calculate the similarity; Disambiguation decision: Based on the results of matching and similarity calculation, select the best candidate entity as the disambiguation result of the entity reference.

4. According to claim 2, a method for analyzing data kinship based on knowledge graph expansion is characterized in that: Coreference resolution is specifically about associating entity references with coreference relationships to the same entity representation.

5. According to the method for analyzing data kinship based on knowledge graph expansion according to claim 1, it is characterized in that: Before performing data lineage extraction and data lineage analysis, and after obtaining the expanded knowledge graph, the following steps are also included: Obtain the knowledge graph before and after the inference engine; The quality of the knowledge graph is calculated by the following function: Where C is the confidence level, which ranges from 0 to 1; Ps is the precision; R S is the recall rate; T S is the score, which is the harmonic mean of precision and recall, T S It can be calculated by the following formula: Set a confidence threshold, compare the calculated confidence with the confidence threshold, and determine whether data lineage extraction and data lineage analysis are necessary for the knowledge graph.

6. According to the method of data kinship analysis based on knowledge graph expansion according to claim 1, it is characterized in that: The data lineage extraction includes obtaining data flow information, and the specific operations are as follows: Use the graph query language Cypher provided by the graph database to track the flow of data from the source entity to the target entity, and generate a data flow report based on the query language, including the starting point, end point, and various processing steps of the data.

7. According to claim 1, a method for analyzing data kinship based on knowledge graph expansion is characterized in that: The data lineage analysis includes analyzing whether data is lost, and the criterion for determining whether data is lost is: data that exists in the source table but not in the target table.

8. The method for analyzing data kinship based on knowledge graph expansion according to claim 1, characterized in that: The data lineage analysis includes whether an erroneous conversion occurs. The criterion for determining an erroneous conversion is that the attributes stored in the source table are inconsistent with the attributes at the corresponding location in the target table.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the data kinship analysis method based on knowledge graph expansion as described in any one of claims 1 to 8.

10. A processor, characterized in that: The processor is used to run a program, wherein when the program is run, the data blood relationship analysis method based on knowledge graph expansion described in any one of claims 1 to 8 is executed.

Images