A metadata-based data governance knowledge graph construction method

Abstract

The present invention provides a method for constructing a data governance knowledge graph based on metadata, which includes defining the metamodel structure of each type of metadata and the relationship between metamodels, summarizing and abstracting all metamodel structures, and forming a unified metamodel structure after constructing corresponding relationships; dynamically adjusting the unified metamodel structure; constructing a dynamic ontology model, mapping the unified metamodel structure to the dynamic ontology model, and adjusting the dynamic ontology model according to the dynamic changes of the unified metamodel structure; processing the metadata in the metadata lake; mapping the structured data after data processing to the dynamic ontology model according to the corresponding mapping relationship between the unified metamodel structure and the dynamic ontology model, completing the construction of the initial knowledge graph; and detecting and preprocessing the real-time data stream to form the updated content of the knowledge graph. The present invention aims to achieve the effective integration and dynamic update of multi-source heterogeneous metadata knowledge in the data governance process.

Description

Technical Field

[0001] The present invention relates to the field of metadata management technology, and in particular to a method for constructing a data governance knowledge graph based on metadata. Background Art

[0002] With the rapid development of information technology, enterprise digital transformation has become an irreversible trend. In this process, businesses are continuously accumulating massive, diverse, and heterogeneous data resources, which contain rich business value and innovation potential. However, this explosive growth of data also brings unprecedented management challenges. How to effectively govern this data and tap into its inherent value has become a pressing issue facing enterprises.

[0003] Metadata is the cornerstone of data governance. Data governance begins with metadata management. When metadata is clearly defined and well-designed, data quality will inevitably improve. The most common definition of metadata is "data about data" or "data describing data." GB / T 18391.1-2009, Information Technology Metadata Registration System (MDR). Part 1: Framework, defines metadata as data that defines and describes other data. It describes the structure, content, context, and management rules of the data.

[0004] Metadata is generally divided into business metadata, technical metadata, and operational metadata. Business metadata describes the business meaning, business rules, and relationships of data. It includes business definitions, business terms, business rules (business engine rules, data quality detection rules), business models (conceptual models, logical models), data security, sensitivity levels, etc. Technical metadata describes relevant conceptual information in the technical field of the system. It includes data structure (name, length, type, constraints, relationships, etc.), data processing (ETL information, scheduling information, update frequency, etc.), data storage (type, location, file format, compression type, etc.), etc. Operational metadata describes the operational attributes of data and clarifies data management responsibilities. It includes data owners, data custodians, data access control information (organizational roles, access methods, access cycles, access scopes), data backup (archiving location, archiving date, archiving cycle), etc.

[0005] Data governance, at a micro level, refers to the management of individual data, encompassing the entire data lifecycle. This encompasses the holistic management of data's utility, availability, integrity, and security, with the core goal of ensuring data security and usability. Currently, this encompasses data integration management, data exchange management, data modeling management, metadata management, data standards management, data quality management, master data management, data asset management, data security management, and data lifecycle management. Traditional data governance approaches often rely on manual intervention, which is inefficient and error-prone, making them unable to meet the demands of modern enterprises for efficient and accurate data management. Therefore, exploring automated and intelligent data governance approaches, particularly metadata management solutions, is crucial.

[0006] As a powerful tool for knowledge representation and reasoning, knowledge graphs have demonstrated tremendous potential in data governance. Essentially, a knowledge graph is a knowledge base based on a semantic network. It stores entities and their relationships in a graph format, representing real-world entities. From a practical perspective, a knowledge graph can be simply understood as a multi-relational graph. Entities are the basic units in a knowledge graph, representing real-world objects or concepts. Relationships are relationships between entities, describing how entities are interconnected. Attributes are characteristics or properties of entities, providing detailed information about them. A graph structure refers to the way a knowledge graph stores data, with nodes representing entities and edges representing relationships. Semantics refers to the fact that the information in a knowledge graph has clear semantics, enabling machines to understand the data's meaning. Reasoning refers to the ability of a knowledge graph to perform complex reasoning and discover implicit relationships between entities. Data integration refers to the ability of a knowledge graph to integrate data from different sources and provide a unified view.

[0007] Knowledge fusion is based on multi-source heterogeneous data. With the support of ontology and rule bases, it obtains knowledge factors and their relationships hidden in data resources through knowledge extraction and transformation, fuses the descriptive information of the same entity or concept from multiple sources, and combines, infers, and creates new knowledge at the semantic level. This process needs to be dynamically adjusted in real time based on changes in data sources and user feedback.

[0008] However, existing methods for constructing knowledge graphs based on metadata still have many shortcomings. On the one hand, these methods often rely on preset features or rules to categorize metadata and extract relationships, which limits their flexibility and universality, making them difficult to adapt to the data characteristics of different enterprises and fields. On the other hand, some methods are overly dependent on data quality and fail to adequately handle noise and outliers in the data, resulting in limited accuracy of the constructed knowledge graphs. Furthermore, existing methods often lack the ability to dynamically adjust in real time when implementing knowledge fusion, making it difficult to update them in response to changes in data sources and user feedback.

[0009] In summary, a more flexible, accurate, and intelligent metadata-based knowledge graph construction method is urgently needed to address the data governance challenges of enterprise digital transformation. This method should be able to automatically identify and extract key information from metadata to construct a unified and comprehensive metadata knowledge graph. It should also be adaptable and scalable, allowing customization based on the data characteristics of different enterprises and fields. Furthermore, it should be capable of real-time dynamic updates to ensure the timeliness and accuracy of the knowledge graph. Summary of the Invention

[0010] In response to the problems existing in the prior art, the purpose of the present invention is to provide a metadata-based data governance knowledge graph construction method, aiming to solve the problem of how to effectively integrate and dynamically update the multi-source heterogeneous metadata knowledge generated in multiple links in the data governance process.

[0011] The present invention achieves the above-mentioned purpose through the following technical solutions:

[0012] A method for constructing a data governance knowledge graph based on metadata includes the following steps:

[0013] Establish multiple collection task areas, and divide collection tasks into metadata real-time collection tasks and business data real-time collection tasks;

[0014] Based on the collected metadata, the metamodel structure of each type of metadata and the relationship between metamodels are defined. All metamodel structures are summarized and abstracted, and after building corresponding relationships, a unified metamodel structure is formed to accommodate information of all metamodel types.

[0015] Real-time detection is performed through metadata collection tasks, and the unified meta-model structure is dynamically adjusted;

[0016] Construct a dynamic ontology model and map the unified metamodel structure to the dynamic ontology model, including mapping metadata types to classes, metadata to entities, metadata attribute sets to attributes, and metadata relationship sets to relationships, and adjust the dynamic ontology model according to the dynamic changes of the unified metamodel structure;

[0017] Preprocess, extract knowledge, and fuse knowledge from the metadata lake;

[0018] According to the corresponding mapping relationship between the unified metamodel structure and the dynamic ontology model, the structured data after knowledge fusion is mapped to the dynamic ontology model, and the converted data is stored in the graph database to complete the construction of the first generation knowledge graph;

[0019] After detecting and preprocessing the real-time data stream, the pre-trained Graph RAG model is connected to automatically extract entities, relationships and their attributes from the pre-processed data, and represent the processing in the form of a graph structure to form the updated content of the knowledge graph; the newly extracted entities and relationships are added to the existing first-generation knowledge graph, and the existing entity and relationship information is updated at the same time.

[0020] According to a metadata-based data governance knowledge graph construction method provided by the present invention, for structured database data sources, a change data capture (CDC) framework is used to monitor the database's binlog logs to capture change events in the database in real time, and the captured change events are parsed to obtain structured metadata or business data.

[0021] For log file data sources, use the Logstash log collection tool for real-time collection. By configuring Logstash's input, filtering, and output plug-ins, you can achieve real-time reading, processing, and forwarding of log files, and then collect the data in the log files to the specified storage location.

[0022] For text file data sources, file monitoring tools are used to monitor the updates of text files in real time. When a file changes, a collection task is triggered to read and parse the updated file content to obtain the metadata or business data in the text file.

[0023] According to a metadata-based data governance knowledge graph construction method provided by the present invention, the unified meta-model structure further defines the specific composition and hierarchical relationship of metadata types, specifically including:

[0024] Data modeling related entities, including database link entities, database instance entities, database entities, data table entities, field entities, view entities, function entities, and data modeling rule entities. Data modeling rule entities are used to specify the standards and specifications for database design, data table structure, field definition, view creation, and function writing;

[0025] Data integration related entities include task entities and scheduling entities. Task entities are used to describe specific jobs or operations in the data integration process, while scheduling entities are used to define the order, time, and frequency of task execution.

[0026] Data service-related entities include system entities, page entities, API entities, and dataset entities. System entities represent the software system that provides data services, page entities refer to the user interface within the system, API entities represent the interface provided by the system to the outside world, and dataset entities are the specific data sets of the data services.

[0027] Data management related entities include account entities, log entities, message entities and document entities. Account entities are used to manage user access rights, log entities are used to record system operation history, message entities are used for information transmission within the system or across systems, and document entities contain various types of documents and materials in the data governance process.

[0028] According to a metadata-based data governance knowledge graph construction method provided by the present invention, the defined unified metamodel structure is classified and labeled according to three classification methods: entity information, relationship information, and attribute information. Among them, entity information is used to regard each metadata item as a metadata entity, which includes data modeling-related entities, data integration-related entities, data service-related entities, and data management-related entities; relationship information is used to define the association, dependency, or hierarchical relationship between various entities in the metamodel; and attribute information defines the various attributes of the metamodel entities themselves.

[0029] The classified and labeled metamodel structures, together with their corresponding label information, are stored in the metamodel structure library, and a unified metamodel structure framework is abstracted based on the classification and labeling results. The framework contains at least metamodel type, metamodel entity information, relationship information, and attribute information.

[0030] According to a metadata-based data governance knowledge graph construction method provided by the present invention, a unified meta-model structure is dynamically adjusted, including:

[0031] For structured databases, metadata collection tasks are set up to detect log information in the database in real time, parse the DDL statement information related to data structure changes in the log, and match the type of DDL operation objects with the metamodel type pre-stored in the metamodel structure library;

[0032] If the type matching is successful, the attributes of the operation object in the DDL statement are further parsed and these attributes are matched one by one with the attributes of the corresponding metamodel in the metamodel structure library;

[0033] If the attribute matching is successful, it indicates that the data structure changes caused by the DDL statement have been reflected in the metamodel structure library, and no processing will be performed; if the attribute matching is unsuccessful, it indicates that the DDL statement introduces new attributes or changes existing attributes, then new attributes are added to the metamodel in the metamodel structure library, and this change information is synchronously updated to the unified metamodel structure.

[0034] According to the metadata-based data governance knowledge graph construction method provided by the present invention, natural language processing technology and machine learning algorithms are used to actively identify and classify newly generated data structures to determine the metamodel type to which they belong; the newly identified metamodels are automatically stored in the metamodel structure library and classified and labeled;

[0035] For newly identified metamodels, the conversion operation is automatically performed to convert them into a unified metamodel structure according to the metamodel type to which they belong and the pre-defined standardized conversion rules;

[0036] Among them, standardized transformation rules are used to describe how to transform different types of metamodels into a unified metadata model structure.

[0037] According to a metadata-based data governance knowledge graph construction method provided by the present invention, metadata in a metadata lake is preprocessed according to data standards and preset data quality rules, specifically including:

[0038] Data integrity verification: Perform integrity checks on metadata in the metadata lake according to data standards;

[0039] Data uniqueness verification: When metadata has a business-unique key, uniqueness verification is performed. Specific indicators include the unique key conflict rate.

[0040] Data accuracy verification: Verify the metadata generation trigger mechanism and attribute value filling according to data standards, with specific indicators including accuracy;

[0041] Data consistency verification: Checks the logical consistency between different attributes of business objects and the consistency of business process magnitude fluctuations, which is performed through cross-table value comparison and fluctuation consistency comparison;

[0042] Data timeliness verification: Verify the metadata generation time and arrival delay according to data standards;

[0043] Data standardization / normalization: Standardize or normalize metadata to eliminate the impact of dimensions or values ​​on calculation results, including Z-Score standardization and Max-Min normalization;

[0044] Data quality report generation: Based on the definition of data quality rules, data quality reports are generated regularly to record the quality status of metadata in terms of completeness, uniqueness, accuracy, consistency and timeliness;

[0045] Data quality issue handling: Based on the records in the data quality report, respond to, locate, analyze, and resolve metadata quality issues found to ensure that the metadata in the metadata lake continues to meet data standards and preset data quality rules.

[0046] According to a metadata-based data governance knowledge graph construction method provided by the present invention, the knowledge extraction process includes entity extraction, attribute extraction, and relationship extraction, specifically including:

[0047] Metamodel structure matching: Using the metamodel structure library, pre-processed metadata is matched with the metamodel structure; based on the classification labels in the metamodel structure, the entity and attribute information in the metadata is directly identified and located;

[0048] Entity extraction: Apply natural language processing technology and machine learning algorithms to preprocess and extract features from the text content in metadata, and perform entity extraction, attribute extraction, and relationship extraction;

[0049] Attribute extraction: construct an attribute list for each identified entity, apply attribute extraction algorithms to extract attribute information related to the entity from the metadata text, and assign attribute values ​​to these attributes;

[0050] Relationship extraction: Based on entity extraction and attribute extraction, further identify and extract the relationships between entities; apply relationship extraction algorithms to analyze the context and semantics in metadata text to determine the types of associations between entities; construct entity relationship graphs to intuitively display the complex relationship network between entities;

[0051] Knowledge integration and storage: Integrate the results of entity extraction, attribute extraction, and relationship extraction to form a structured knowledge representation, and store the structured knowledge in the knowledge base.

[0052] According to a metadata-based data governance knowledge graph construction method provided by the present invention, the knowledge extraction and fusion process specifically includes:

[0053] Select the entities to be aligned and choose their key attributes for each entity. The entity itself, its attributes and attribute values, and its relationship with other entities are then combined into three sets of features. Using vectorization technology, each of these three sets of features is converted into a vector representation. These vectors are then concatenated into a unified representation in a low-dimensional vector space.

[0054] In the constructed low-dimensional vector space, a semantic distance calculation method is used to measure the similarity between entities. A semantic distance threshold is set, and for entity pairs with a semantic distance less than the threshold, it is preliminarily determined that they may be aligned or equivalent entities.

[0055] For entity pairs that are initially judged to be potentially aligned, attribute conflicts and relationship conflicts are detected and processed. By comparing whether the attribute values ​​of the entities are consistent or compatible, and whether the relationships between the entities match each other, it is determined whether these entities can indeed be merged into a unified entity. Among them, for entities that are confirmed to be mergeable, a merge operation is performed and integrated into a new entity representation.

[0056] According to a metadata-based data governance knowledge graph construction method provided by the present invention, pre-processed new information is embedded into the constructed knowledge graph structure to form a graph structure fragment containing new entities, new relationships and their attributes. The embedding layer of the GraphRAG model is used to convert the new information and existing information into low-dimensional vector representations.

[0057] Based on the reasoning ability of the GraphRAG model, the logical relationship and consistency between the new information and the existing information in the knowledge graph are checked;

[0058] The GraphRAG model uses its attention mechanism to focus on entities and relationships directly related to the new information, as well as the connections and constraints between them. It evaluates the rationality and credibility of the new information by comparing the distance and similarity between the new information and existing information in vector space. Based on the results of logical relationships and consistency checks, the GraphRAG model automatically infers whether the new information is consistent with existing information in the knowledge graph and whether there are any conflicts or contradictions.

[0059] The GraphRAG model outputs verification decisions, including accepting new information, rejecting new information, or marking it for further verification. For accepted new information, it is formally integrated into the knowledge graph, and the structure and content of the graph are updated. For rejected or pending verification information, corresponding feedback is output.

[0060] Thus, the present invention proposes a method for constructing a metadata-based data governance knowledge graph. This method can form a unified metadata governance view by integrating the multi-source heterogeneous metadata knowledge generated during the data governance process. Therefore, the present invention brings the following significant beneficial effects:

[0061] 1. This invention builds a unified metadata governance view by integrating metadata from different sources and formats, enabling data governance personnel to quickly and accurately locate the required metadata and its corresponding data information, greatly improving the efficiency and convenience of data governance.

[0062] 2. Under a unified metadata governance view, data governance personnel can quickly find relevant metadata and data information without having to switch and search back and forth between multiple heterogeneous data sources, which greatly saves time costs and improves the response speed of data governance.

[0063] 3. The knowledge graph constructed by the present invention contains rich metadata information and the relationships between them, providing a solid data foundation for the automatic generation of data quality rules. By analyzing and mining the metadata relationships in the knowledge graph, data quality rules that meet data governance requirements can be automatically generated to ensure the quality and accuracy of the data.

[0064] 4. The metadata and relationships in the knowledge graph also provide strong support for the automatic generation of data standards. By classifying, summarizing, and organizing the metadata in the knowledge graph, this invention can automatically generate data standards that meet business needs and industry standards, thereby standardizing the use and management of data.

[0065] 5. Data security auditing is a crucial task in the data governance process. The knowledge graph constructed by this invention can clearly display the relationships between metadata, enabling data security auditors to quickly locate potential security risks and conduct targeted audits and inspections to ensure data security.

[0066] 6. By monitoring changes in data sources and automatically identifying the characteristics of newly introduced datasets using natural language processing techniques, the present invention can recommend appropriate classifications and labels, thereby reducing the need for manual intervention. This not only improves the automation level of data governance but also reduces the risk of human error.

[0067] 7. In response to the complex relationships involved in the data governance process, the present invention designs a unified knowledge representation model that can clearly express the relationship between data, business, and management, so that data governance personnel can better understand and analyze various problems and challenges in the data governance process, thereby improving the ability and level of data governance.

[0068] 8. Based on Graph RAG technology, the present invention realizes incremental knowledge graph construction, which can avoid the reconstruction of the entire knowledge graph, thereby maintaining the timeliness and accuracy of information, enabling data governance personnel to obtain the latest metadata and its relationship information in a timely manner, providing strong support for data governance work.

[0069] To sum up, the metadata-based data governance knowledge graph construction method proposed in the present invention brings many beneficial effects, can significantly improve the efficiency and accuracy of data governance, and provide strong support for the digital transformation and business development of enterprises.

[0070] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 It is a flowchart of an embodiment of a method for constructing a data governance knowledge graph based on metadata of the present invention.

[0072] Figure 2 This is a schematic diagram of an embodiment of a method for constructing a data governance knowledge graph based on metadata according to the present invention. DETAILED DESCRIPTION

[0073] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0074] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0075] See also Figure 1 and Figure 2 This embodiment provides a method for constructing a data governance knowledge graph based on metadata, which includes the following steps:

[0076] Step S1: formulate multiple collection task areas, and divide the collection tasks into metadata real-time collection tasks and business data real-time collection tasks;

[0077] Step S2: Based on the collected metadata, define the metamodel structure of each type of metadata and the relationship between metamodels, summarize and abstract all metamodel structures, and build corresponding relationships to form a unified metamodel structure, which is used to accommodate information of all metamodel types;

[0078] Step S3, performing real-time detection through metadata collection tasks and dynamically adjusting the unified metamodel structure;

[0079] Step S4: construct a dynamic ontology model, mapping the unified metamodel structure to the dynamic ontology model, including mapping metadata types to classes, metadata to entities, metadata attribute sets to attributes, and metadata relationship sets to relationships, and adjusting the dynamic ontology model according to dynamic changes in the unified metamodel structure;

[0080] Step S5: preprocessing, knowledge extraction, and knowledge fusion of the metadata in the metadata lake;

[0081] Step S6: According to the corresponding mapping relationship between the unified metamodel structure and the dynamic ontology model, the structured data after knowledge fusion is mapped to the dynamic ontology model, and the converted data is stored in the graph database to complete the construction of the initial generation knowledge graph;

[0082] Step S7: After detecting and preprocessing the real-time data stream, the pre-trained Graph RAG model is connected to automatically extract entities, relationships and their attributes from the pre-processed data, and represent the processing in the form of a graph structure to form the updated content of the knowledge graph; the newly extracted entities and relationships are added to the existing first-generation knowledge graph, and the existing entity and relationship information is updated at the same time.

[0083] In step S1, when data is collected in real time, two different types of real-time collection tasks are distinguished: metadata collection tasks and business data collection tasks. Data collected in real time by metadata collection tasks is stored in the metadata lake, while data collected by business data collection tasks is stored in the data lake.

[0084] The metadata real-time collection task and the business data real-time collection task are logically and / or physically separated to ensure the independent collection, processing and storage of metadata and business data, thereby improving the efficiency of data collection and the flexibility of data management.

[0085] In this embodiment, the data source types involved in the data governance process are mainly divided into structured databases and text files. For structured database data sources, the change data capture CDC framework is used to monitor the binlog logs of the database to capture change events in the database in real time, and the captured change events are parsed to obtain structured metadata or business data. For log file data sources, the Logstash log collection tool is used for real-time collection. By configuring the input, filtering and output plug-ins of Logstash, the real-time reading, processing and forwarding of log files are realized, and the data in the log files are collected to the specified storage location. For text file data sources, the file monitoring tool is used to monitor the update of the text file in real time. When the file changes, the collection task is triggered to read and parse the updated file content to obtain the metadata or business data in the text file.

[0086] Among them, the above-mentioned specific collection methods for different data source types can ensure the real-time and accuracy of data collection, while adapting to the characteristics and formats of different data sources, and improving the flexibility and versatility of data collection.

[0087] Specifically, for structured databases, Flink CDC is used to monitor changes in the database's binlog logs, trigger and capture change events, and parse the changes. Since the Flink CDC connector cannot directly support DDL statement parsing, a custom parser is built and integrated into the Flink job. After synchronizing the binlog logs to Kafka based on Flink CDC, Flink's Kafka Source is used to read the Kafka topic containing the DDL statements, then parse the DDL statements and pass the parsed results to the downstream. For text files, since they are uniformly stored in the file management system, the file system's operation logs can be used to identify which files have changed, and then all files involved in the changes can be extracted.

[0088] In step S2 above, the unified metamodel structure further defines the specific composition and hierarchical relationship of metadata types, including:

[0089] Data modeling related entities include database link entities, database instance entities, database entities, data table entities, field entities, view entities, function entities and data modeling rule entities. Among them, the data modeling rule entity is used to specify the standards and specifications for database design, data table structure, field definition, view creation and function writing.

[0090] Data integration related entities include task entities and scheduling entities. Task entities are used to describe specific jobs or operations in the data integration process, while scheduling entities are used to define the order, time, and frequency of task execution.

[0091] Data service-related entities include system entities, page entities, API entities, and dataset entities. System entities represent the software system that provides data services, page entities refer to the user interface within the system, API entities represent the interface provided by the system to the outside world, and dataset entities are the specific data sets of data services.

[0092] Data management related entities include account entities, log entities, message entities and document entities. Account entities are used to manage user access rights, log entities are used to record system operation history, message entities are used for information transmission within the system or across systems, and document entities contain various types of documents and materials in the data governance process.

[0093] In addition, the unified metamodel structure also defines the association relationships and hierarchical structures between the above-mentioned entities, as well as the relationships between the internal attributes of the entities and the attributes, thus forming a complete and standardized metadata model system, providing a solid foundation for data governance and knowledge graph construction.

[0094] Specifically, a unified metadata model was defined. Through preliminary research and identification of all metadata in data governance, metadata types were identified as falling into four categories: data modeling-related, data integration-related, data service-related entities, and data management-related entities. Data modeling-related entities include database links, database instances, databases, data tables, fields, views, and functions; data integration-related entities include tasks and schedules; data service-related entities include systems, pages, APIs, and datasets; and data management-related entities primarily include accounts, logs, messages, and documents. A metamodel is a model about models; it defines the specifications of a particular model, specifically the elements that make up the model and the relationships between them. The specific metamodel structure is shown in Table 1.

[0095] [Table 1]

[0096]

[0097] In this embodiment, the defined unified metamodel structure is classified and labeled according to three classification methods: entity information, relationship information, and attribute information. Among them, entity information is used to regard each metadata item as a metadata entity, which includes data modeling-related entities, data integration-related entities, data service-related entities, and data management-related entities. Relationship information is used to define the association, dependency, or hierarchical relationship between various entities of the metamodel. Attribute information defines the various attributes of the metamodel entity itself, such as name, description, type, status, etc.

[0098] The classified and labeled metamodel structures, together with their corresponding label information, are stored in the metamodel structure library for subsequent query, call, and management. Based on the classification and labeling results, a unified metamodel structure framework is abstracted. The framework contains at least metamodel type, metamodel entity information, relationship information, and attribute information, providing a standardized template and guide for constructing metadata models in specific fields, as shown in Table 2.

[0099] [Table 2]

[0100]

[0101] In the above step S3, the unified metamodel structure is dynamically adjusted, including:

[0102] For structured databases, metadata collection tasks are set up to detect log information in the database in real time, parse the DDL (Data Definition Language) statement information related to data structure changes in the log, and match the type of the DDL operation object (such as table, databese, etc.) with the metamodel type pre-stored in the metamodel structure library. If the type match is successful, the attributes of the operation object in the DDL statement are further parsed, and these attributes are matched one by one with the attributes of the corresponding metamodel in the metamodel structure library. If the attribute match is successful, it indicates that the data structure change caused by the DDL statement has been reflected in the metamodel structure library and no processing is performed; if the attribute match is not successful, it indicates that the DDL statement introduces new attributes or changes existing attributes, then new attributes are added to the metamodel in the metamodel structure library, and this change information is synchronously updated to the unified metamodel structure.

[0103] Through the above steps, the automatic recognition, dynamic update and unified management of the meta-model structure in the structured database are realized, which greatly improves the efficiency and accuracy of metadata management.

[0104] In this embodiment, if a new data structure is generated, the name, description, field information, etc. of the newly generated data structure are subjected to text parsing and semantic understanding. Natural language processing technology and machine learning algorithms (such as classifiers and clustering algorithms) are used to actively identify and classify the newly generated data structure to determine its metamodel type, such as a table model, view model, or function model. The newly identified metamodel is automatically stored in the metamodel structure library and classified and labeled. For the newly identified metamodel, a conversion operation is automatically performed to convert it into a unified metamodel structure based on its metamodel type and pre-defined standardized conversion rules. The standardized conversion rules are used to describe how to convert different types of metamodels into a unified metadata model structure.

[0105] It can be seen that this embodiment realizes the automatic identification, classification and standardized conversion of the metamodel of newly generated data structures in the database through the steps of active identification and classification, automatic storage and classification labeling, and standardized conversion and consistency maintenance, providing strong support for database metadata management, data governance and value mining of data assets.

[0106] In step S4, regarding the construction of the dynamic ontology model, the ontology is a formalized, standardized, and explicit description of a shared conceptual model, encompassing elements such as classes (concepts), relations, functions, axioms, and instances. Through a one-to-one mapping between the ontology model and the unified metamodel, the metamodel's "types" are mapped to the ontology model's "classes," the metamodel's "entities" are mapped to the ontology model's "entities," the metamodel's "attributes" are mapped to the ontology model's "attributes," and the metamodel's "relationships" are mapped to the ontology model's "relationships." When the unified metamodel structure is dynamically adjusted by detecting changes to the metamodel's structure, the ontology model will also dynamically adjust its model structure accordingly.

[0107] In step S5 above, the metadata in the metadata lake is preprocessed according to data standards and pre-set data quality rules. The data is converted into a unified format and unit to eliminate data inconsistencies. Missing values ​​are filled in. Duplicate records of duplicate values ​​are removed, and standard and rule information is supplemented. This specifically includes:

[0108] Data integrity verification: Based on data standards, perform integrity checks on the metadata in the metadata lake to ensure that the metadata is not lost or duplicated during upload, transmission, and storage. Specific indicators include but are not limited to transmission loss rate and transmission duplication rate.

[0109] Data uniqueness verification: When metadata has a business-unique key, uniqueness verification is performed to ensure the uniqueness of the metadata in business logic. Specific indicators include the unique key conflict rate.

[0110] Data accuracy verification: Verify the metadata generation trigger mechanism and attribute value filling according to data standards to ensure that the metadata conforms to business logic and accuracy requirements. Specific indicators include accuracy.

[0111] Data consistency verification: Checks the logical consistency between different attributes of business objects and the consistency of the magnitude fluctuations of business processes. This is done through cross-table and value comparisons, and fluctuation consistency comparisons, to ensure the consistency of metadata in logic and magnitude.

[0112] Data timeliness verification: Based on data standards, the metadata generation time and arrival delay are verified to ensure that the metadata meets the preset requirements in terms of timeliness. Specific indicators include generation delay, date drift, etc.

[0113] Data standardization / normalization: To meet the needs of machine learning models or data analysis, metadata is standardized or normalized to eliminate the impact of dimensions or values ​​on calculation results. Specifically, this includes Z-Score standardization and Max-Min normalization to ensure the validity and accuracy of metadata in model calculations.

[0114] Data quality report generation: Based on the definition of data quality rules, data quality reports are generated regularly to record the quality status of metadata in terms of completeness, uniqueness, accuracy, consistency and timeliness, providing decision support for metadata management.

[0115] Data quality issue handling: Based on the records in the data quality report, respond to, locate, analyze, and resolve metadata quality issues found to ensure that the metadata in the metadata lake continues to meet data standards and preset data quality rules.

[0116] Through the above preprocessing steps, we ensure that the metadata in the metadata lake complies with the preset data quality rules in terms of completeness, uniqueness, accuracy, consistency, timeliness and standardization / normalization, providing a solid guarantee for the effective management and efficient use of metadata.

[0117] In the above step S5, the knowledge extraction process includes entity extraction, attribute extraction and relationship extraction, specifically including:

[0118] Metamodel structure matching: Use the metamodel structure library to match the preprocessed metadata with the metamodel structure; based on the classification labels in the metamodel structure, directly identify and locate the entity and attribute information in the metadata, providing a basis for subsequent entity extraction and attribute extraction tasks.

[0119] Entity extraction: Apply natural language processing technology and machine learning algorithms to preprocess and extract features from the text content in the metadata, and perform entity extraction, attribute extraction, and relationship extraction. Through a well-trained named entity recognition model, entities with specific meanings, such as names of people, places, and organizations, are identified from the text and assigned corresponding entity labels.

[0120] Attribute extraction: Construct an attribute list for each identified entity, apply the attribute extraction algorithm to extract attribute information related to the entity from the metadata text, and assign attribute values ​​to these attributes.

[0121] Relationship extraction: Based on entity extraction and attribute extraction, further identify and extract the relationships between entities; apply relationship extraction algorithms to analyze the context and semantics in metadata text to determine the types of associations between entities, such as parent-child relationships, affiliation relationships, and cooperative relationships; and construct entity relationship graphs to intuitively display the complex relationship network between entities.

[0122] Knowledge integration and storage: Integrate the results of entity extraction, attribute extraction, and relationship extraction to form a structured knowledge representation, and store the structured knowledge in the knowledge base for subsequent knowledge query, analysis, and application.

[0123] Specifically, knowledge extraction includes three core tasks: entity extraction, attribute extraction, and relationship extraction. Entity extraction, also known as named entity recognition (NER), identifies entities with specific meanings from text; attribute extraction constructs a list of attributes for each identified entity and appends attribute values; and relationship extraction identifies and extracts relationships between entities, such as parent-child relationships and affiliation relationships. Through the aforementioned metamodel structure library, the metamodel structure and metadata are matched, and entity and attribute information can be directly found based on the classification labels in the metamodel structure. Take the metamodel structure in Table 3 as an example:

[0124] [Table 3]

[0125]

[0126] For the text in the metadata, the data is preprocessed and features are extracted using natural language processing technology and machine learning algorithms, and then entity extraction, attribute extraction and relationship extraction are performed respectively.

[0127] In the above step S5, the knowledge extraction and fusion process specifically includes:

[0128] Select the entities to be aligned and choose the key attributes for each entity. The entity itself, its attributes and attribute values, and its relationship with other entities are respectively composed of three sets of features. Using vectorization technology, the three sets of features are converted into vector representations respectively, and these vectors are fused into a unified representation in a low-dimensional vector space through splicing operations.

[0129] In the constructed low-dimensional vector space, semantic distance calculation methods, such as cosine similarity and Euclidean distance, are used to measure the similarity between entities. A semantic distance threshold is set, and for entity pairs with a semantic distance less than the threshold, it is preliminarily judged that they may be aligned or equivalent entities.

[0130] For entity pairs that are initially judged to be potentially aligned, attribute conflicts and relationship conflicts are detected and processed. By comparing whether the attribute values ​​of the entities are consistent or compatible, and whether the relationships between the entities match each other, it is determined whether these entities can indeed be merged into a unified entity. Among them, for entities that are confirmed to be mergeable, a merge operation is performed and integrated into a new entity representation.

[0131] Specifically, entity alignment is an important task to solve the same entities in the knowledge graph and is the key foundation for realizing the integration of knowledge graphs. This embodiment is completed through four steps: entity calculation, entity matching, entity alignment, and result evaluation optimization. Entity calculation includes selecting the key attributes of the entity, and then dividing the entity, attributes and attribute values, and relationships into three groups, respectively represented by vectors, and then splicing them to form a unified low-dimensional vector space. Entity matching includes comparing the semantic distance and threshold between entities in the vector space to determine whether the entities are aligned. Entities with a smaller semantic distance can be considered equivalent. Entity alignment includes merging entities with a smaller semantic distance into one entity by processing attribute conflicts and relationship conflicts.

[0132] Finally, the result evaluation and optimization are carried out: the evaluation indicators precision and recall rate are defined, and the entity calculation method and threshold are optimized and adjusted according to the evaluation results.

[0133] Through the above steps, this embodiment can achieve efficient and accurate entity alignment in the metadata lake, thereby promoting the integration of knowledge graphs and knowledge fusion, and providing strong support for knowledge management, intelligent analysis, and decision support in the metadata lake.

[0134] In the above step S6, the unified metadata model in data governance includes meta-models such as databases, tables, fields, tasks, scheduling, APIs, and the relationships between them. According to the one-to-one correspondence between the metadata model and the ontology model in the dynamic ontology model construction step, the ontology model also includes classes such as databases, tables, fields, tasks, scheduling, APIs, and the relationships between them. In this embodiment, it is only necessary to import the metadata of each type of meta-model into the entity of the corresponding class in the ontology model. Assuming that there are two tables "Sales Order" and "Product Information" and a data integration task "Sales Order Data Synchronization", the metadata of the "Sales Order" and "Product Information" tables are imported into the table entity of the ontology model, and the metadata of the "Sales Order Data Synchronization" task is imported into the task entity in the ontology model, and the relationship between the task entity and the table entity is constructed. Through the above steps, the final data is stored in the graph database, and the construction of the first generation knowledge graph is completed.

[0135] In step S7 above, the pre-trained and optimized GraphRAG (Relational Attention Graph) model is loaded. This model integrates graph neural networks and attention mechanisms, is good at processing graph-structured data, and has strong reasoning and verification capabilities. The preprocessed new information is embedded into the constructed knowledge graph structure, forming a graph structure fragment containing new entities, new relationships, and their attributes. The embedding layer of the GraphRAG model is used to convert the new and existing information into low-dimensional vector representations. Based on the reasoning ability of the GraphRAG model, the logical relationship and consistency between the new information and the existing information in the knowledge graph are checked. The attention mechanism of the GraphRAG model focuses on entities and relationships directly related to the new information, as well as the connections and constraints between them. The rationality and credibility of the new information are evaluated by comparing the distance and similarity between the new information and the existing information in the vector space. Based on the results of the logical relationship and consistency check, the GraphRAG model automatically infers whether the new information is consistent with the existing information in the knowledge graph and whether there are any conflicts or contradictions. The GraphRAG model outputs a verification decision, which may include accepting the new information, rejecting the new information, or marking it for further verification. For accepted new information, it is formally integrated into the knowledge graph, and the structure and content of the graph are updated. For rejected or pending verification information, corresponding feedback is output, and the system provides corresponding feedback or suggestions for further processing by the user or the system. The results of each inference and verification, as well as user feedback and correction information, are recorded as a data source for continuous model learning. The GraphRAG model is regularly retrained and optimized to improve the accuracy and efficiency of its inference and verification, ensuring the update and maintenance quality of the knowledge graph.

[0136] In summary, this embodiment proposes a method for constructing a data governance knowledge graph based on metadata. This method can form a unified metadata governance view by integrating multi-source heterogeneous metadata knowledge generated in the data governance process.

[0137] Furthermore, this embodiment constructs a unified metadata governance view by integrating metadata from different sources and formats, enabling data governance personnel to quickly and accurately locate the required metadata and its corresponding data information, greatly improving the efficiency and convenience of data governance.

[0138] Furthermore, under a unified metadata governance view, data governance personnel can quickly find relevant metadata and data information without having to switch and search back and forth between multiple heterogeneous data sources, which greatly saves time costs and improves the response speed of data governance.

[0139] Furthermore, the knowledge graph constructed in this embodiment contains rich metadata information and the relationships between them, providing a solid data foundation for the automatic generation of data quality rules. By analyzing and mining the metadata relationships in the knowledge graph, data quality rules that meet data governance requirements can be automatically generated to ensure the quality and accuracy of the data.

[0140] Furthermore, the metadata and their relationships in the knowledge graph also provide strong support for the automatic generation of data standards. By classifying, summarizing, and organizing the metadata in the knowledge graph, the method of this embodiment can automatically generate data standards that meet business needs and industry standards, thereby standardizing the use and management of data.

[0141] Furthermore, data security auditing is an important task in the data governance process. The knowledge graph constructed in this embodiment can clearly display the associations between metadata, allowing data security auditors to quickly locate potential security risks and conduct targeted audits and inspections to ensure data security.

[0142] Furthermore, by monitoring changes in data sources and automatically identifying the features of newly introduced datasets using natural language processing, the method in this embodiment can recommend appropriate classifications and labels, thereby reducing the need for manual intervention. This not only improves the automation level of data governance but also reduces the risk of human error.

[0143] Furthermore, in response to the complex relationships involved in the data governance process, this embodiment designs a unified knowledge representation model that can clearly express the relationship between data, business, and management, enabling data governance personnel to better understand and analyze various problems and challenges in the data governance process, thereby improving the ability and level of data governance.

[0144] Furthermore, based on Graph RAG technology, this embodiment realizes incremental knowledge graph construction, which can avoid the reconstruction of the entire knowledge graph, thereby maintaining the timeliness and accuracy of information, enabling data governance personnel to obtain the latest metadata and its relationship information in a timely manner, providing strong support for data governance work.

[0145] Therefore, the metadata-based data governance knowledge graph construction method proposed in this embodiment brings many beneficial effects, can significantly improve the efficiency and accuracy of data governance, and provide strong support for the digital transformation and business development of enterprises.

[0146] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0147] The above embodiments are only preferred embodiments of the present invention and cannot be used to limit the scope of protection of the present invention. Any non-substantial changes and replacements made by technicians in this field on the basis of the present invention fall within the scope of protection required by the present invention.

Claims

1. A method for constructing a data governance knowledge graph based on metadata, characterized in that: The following steps are involved: Establish multiple collection task areas, and divide collection tasks into metadata real-time collection tasks and business data real-time collection tasks; Based on the collected metadata, the metamodel structure of each type of metadata and the relationship between metamodels are defined. All metamodel structures are summarized and abstracted, and after building corresponding relationships, a unified metamodel structure is formed to accommodate information of all metamodel types. Real-time detection is performed through metadata collection tasks, and the unified meta-model structure is dynamically adjusted; Construct a dynamic ontology model and map the unified metamodel structure to the dynamic ontology model, including mapping metadata types to classes, metadata to entities, metadata attribute sets to attributes, and metadata relationship sets to relationships, and adjust the dynamic ontology model according to the dynamic changes of the unified metamodel structure; Preprocess, extract knowledge, and fuse knowledge from the metadata lake; According to the corresponding mapping relationship between the unified metamodel structure and the dynamic ontology model, the structured data after knowledge fusion is mapped to the dynamic ontology model, and the converted data is stored in the graph database to complete the construction of the first generation knowledge graph; After detecting and preprocessing the real-time data stream, the pre-trained Graph RAG model is connected to automatically extract entities, relationships and their attributes from the pre-processed data, and represent the processing in the form of a graph structure to form the updated content of the knowledge graph; the newly extracted entities and relationships are added to the existing first-generation knowledge graph, and the existing entity and relationship information is updated at the same time.

2. The method according to claim 1, wherein: For structured database data sources, the Change Data Capture (CDC) framework is used to monitor the database's binlog logs to capture database change events in real time. The captured change events are then parsed to obtain structured metadata or business data. For log file data sources, use the Logstash log collection tool for real-time collection. By configuring Logstash's input, filtering, and output plug-ins, you can achieve real-time reading, processing, and forwarding of log files, and then collect the data in the log files to the specified storage location. For text file data sources, file monitoring tools are used to monitor the updates of text files in real time. When a file changes, a collection task is triggered to read and parse the updated file content to obtain the metadata or business data in the text file.

3. The method according to claim 1, wherein: The unified metamodel structure further defines the specific composition and hierarchical relationships of metadata types, including: Data modeling related entities, including database link entities, database instance entities, database entities, data table entities, field entities, view entities, function entities, and data modeling rule entities. Data modeling rule entities are used to specify the standards and specifications for database design, data table structure, field definition, view creation, and function writing; Data integration related entities include task entities and scheduling entities. Task entities are used to describe specific jobs or operations in the data integration process, while scheduling entities are used to define the order, time, and frequency of task execution. Data service-related entities include system entities, page entities, API entities, and dataset entities. System entities represent the software system that provides data services, page entities refer to the user interface within the system, API entities represent the interface provided by the system to the outside world, and dataset entities are the specific data sets of the data services. Data management related entities include account entities, log entities, message entities and document entities. Account entities are used to manage user access rights, log entities are used to record system operation history, message entities are used for information transmission within the system or across systems, and document entities contain various types of documents and materials in the data governance process.

4. The method according to claim 3, wherein: The defined unified metamodel structure is classified and labeled according to three categories: entity information, relationship information, and attribute information. Entity information is used to treat each metadata item as a metadata entity, including data modeling-related entities, data integration-related entities, data service-related entities, and data management-related entities. Relationship information is used to define the associations, dependencies, or hierarchical relationships between various entities in the metamodel. Attribute information defines the various attributes of the metamodel entities themselves. The classified and labeled metamodel structures, together with their corresponding label information, are stored in the metamodel structure library, and a unified metamodel structure framework is abstracted based on the classification and labeling results. The framework contains at least metamodel type, metamodel entity information, relationship information, and attribute information.

5. The method according to claim 4, characterized in that Dynamically adjust the unified metamodel structure, including: For structured databases, metadata collection tasks are set up to detect log information in the database in real time, parse the DDL statement information related to data structure changes in the log, and match the type of DDL operation objects with the metamodel type pre-stored in the metamodel structure library; If the type matching is successful, the attributes of the operation object in the DDL statement are further parsed and these attributes are matched one by one with the attributes of the corresponding metamodel in the metamodel structure library; If the attribute matching is successful, it indicates that the data structure changes caused by the DDL statement have been reflected in the metamodel structure library, and no processing will be performed; if the attribute matching is unsuccessful, it indicates that the DDL statement introduces new attributes or changes existing attributes, then new attributes are added to the metamodel in the metamodel structure library, and this change information is synchronously updated to the unified metamodel structure.

6. The method according to claim 5, characterized in that: For newly generated data structures, natural language processing technology and machine learning algorithms are used to actively identify and classify them to determine the type of metamodel they belong to; the newly identified metamodels are automatically stored in the metamodel structure library and classified and labeled; For newly identified metamodels, the conversion operation is automatically performed to convert them into a unified metamodel structure according to the metamodel type to which they belong and the pre-defined standardized conversion rules; Among them, standardized transformation rules are used to describe how to transform different types of metamodels into a unified metadata model structure.

7. The method according to claim 1, characterized in that Preprocess the metadata in the metadata lake according to data standards and preset data quality rules, including: Data integrity verification: Perform integrity checks on metadata in the metadata lake according to data standards; Data uniqueness verification: When metadata has a business-unique key, uniqueness verification is performed. Specific indicators include the unique key conflict rate. Data accuracy verification: Verify the metadata generation trigger mechanism and attribute value filling according to data standards, with specific indicators including accuracy; Data consistency verification: Checks the logical consistency between different attributes of business objects and the consistency of business process magnitude fluctuations, which is performed through cross-table value comparison and fluctuation consistency comparison; Data timeliness verification: Verify the metadata generation time and arrival delay according to data standards; Data standardization / normalization: Standardize or normalize metadata to eliminate the impact of dimensions or values ​​on calculation results, including Z-Score standardization and Max-Min normalization; Data quality report generation: Based on the definition of data quality rules, data quality reports are generated regularly to record the quality status of metadata in terms of completeness, uniqueness, accuracy, consistency and timeliness; Data quality issue handling: Based on the records in the data quality report, respond to, locate, analyze, and resolve metadata quality issues found to ensure that the metadata in the metadata lake continues to meet data standards and preset data quality rules.

8. The method according to claim 4, characterized in that The knowledge extraction process includes entity extraction, attribute extraction, and relationship extraction, specifically including: Metamodel structure matching: Using the metamodel structure library, pre-processed metadata is matched with the metamodel structure; based on the classification labels in the metamodel structure, the entity and attribute information in the metadata is directly identified and located; Entity extraction: Apply natural language processing technology and machine learning algorithms to preprocess and extract features from the text content in metadata, and perform entity extraction, attribute extraction, and relationship extraction; Attribute extraction: construct an attribute list for each identified entity, apply attribute extraction algorithms to extract attribute information related to the entity from the metadata text, and assign attribute values ​​to these attributes; Relationship extraction: Based on entity extraction and attribute extraction, further identify and extract the relationships between entities; apply relationship extraction algorithms to analyze the context and semantics in metadata text to determine the types of associations between entities; construct entity relationship graphs to intuitively display the complex relationship network between entities; Knowledge integration and storage: Integrate the results of entity extraction, attribute extraction, and relationship extraction to form a structured knowledge representation, and store the structured knowledge in the knowledge base.

9. The method according to any one of claims 1 to 8, characterized in that The knowledge extraction and fusion process specifically includes: Select the entities to be aligned and choose their key attributes for each entity. The entity itself, its attributes and attribute values, and its relationship with other entities are then combined into three sets of features. Using vectorization technology, each of these three sets of features is converted into a vector representation. These vectors are then concatenated into a unified representation in a low-dimensional vector space. In the constructed low-dimensional vector space, a semantic distance calculation method is used to measure the similarity between entities. A semantic distance threshold is set, and for entity pairs with a semantic distance less than the threshold, it is preliminarily determined that they may be aligned or equivalent entities. For entity pairs that are initially judged to be potentially aligned, attribute conflicts and relationship conflicts are detected and processed. By comparing whether the attribute values ​​of the entities are consistent or compatible, and whether the relationships between the entities match each other, it is determined whether these entities can indeed be merged into a unified entity. Among them, for entities that are confirmed to be mergeable, a merge operation is performed and integrated into a new entity representation.

10. The method according to any one of claims 1 to 8, characterized in that: The pre-processed new information is embedded into the constructed knowledge graph structure to form a graph structure fragment containing new entities, new relationships and their attributes. The embedding layer of the GraphRAG model is used to convert the new information and existing information into low-dimensional vector representations. Based on the reasoning ability of the GraphRAG model, the logical relationship and consistency between the new information and the existing information in the knowledge graph are checked; The GraphRAG model uses its attention mechanism to focus on entities and relationships directly related to the new information, as well as the connections and constraints between them. It evaluates the rationality and credibility of the new information by comparing the distance and similarity between the new information and existing information in vector space. Based on the results of logical relationships and consistency checks, the GraphRAG model automatically infers whether the new information is consistent with existing information in the knowledge graph and whether there are any conflicts or contradictions. The GraphRAG model outputs verification decisions, including accepting new information, rejecting new information, or marking it for further verification. For accepted new information, it is formally integrated into the knowledge graph, and the structure and content of the graph are updated. For rejected or pending verification information, corresponding feedback is output.

Images