Uniform metadata standardization management system oriented to heterogeneous data sources

Abstract

The invention discloses a heterogeneous data source-oriented unified metadata standardization management system, which comprises a unified access layer used for receiving original metadata from various heterogeneous data sources, and the heterogeneous data sources at least comprise a relational database, a non-relational database and an application program interface, performing preliminary cleaning and format packaging on the original metadata, and outputting intermediate metadata; the semantic mapping and standardization layer is connected with the unified access layer and used for analyzing the intermediate metadata, mapping elements in the intermediate metadata to corresponding standard concepts in a global standard metadata model through semantic similarity calculation on the basis of the pre-constructed global standard metadata model, and storing the elements in the intermediate metadata in the unified access layer; and marking a standardized label on the intermediate metadata according to a mapping result to generate standardized metadata. Data standards can be unified, and management efficiency and quality can be intelligently improved.

Description

Technical Field

[0001] This invention relates to the field of data management, and more specifically to a unified metadata standardization management system for heterogeneous data sources. Background Technology

[0002] In the field of data management, data sources for enterprises and various organizations are increasingly exhibiting heterogeneous characteristics, covering various types such as relational databases, non-relational databases, and application programming interfaces. The communication protocols and data formats of different data sources differ significantly, resulting in scattered, disordered, inconsistently formatted, and semantically inconsistent metadata (including table names, field names, and descriptive text).

[0003] To maximize the value of data, these scattered and heterogeneous metadata need to be integrated into a unified and manageable set, namely, unified metadata for heterogeneous data sources. However, the unified metadata still faces problems such as semantic ambiguity, chaotic classification, and difficulty in retrieval. It is urgent to establish unified rules through standardized management to achieve semantic alignment, format standardization, and tag definition of metadata, so as to meet the needs of business adaptation and compliance control, and support efficient retrieval and full lifecycle management of metadata.

[0004] However, existing technologies lack a unified metadata standardization management solution that can adapt to multiple heterogeneous data sources, and cannot effectively solve problems such as accurate mapping, dynamic optimization, and comprehensive control in the process of metadata integration and standardization. Therefore, a unified metadata standardization management system for heterogeneous data sources is proposed. Summary of the Invention

[0005] The present invention solves the above-mentioned technical problems through the following technical solutions, the present invention comprising: A unified access layer is used to receive raw metadata from multiple heterogeneous data sources, including at least relational databases, non-relational databases, and application programming interfaces, and to perform preliminary cleaning and format encapsulation on the raw metadata, and output intermediate metadata. The semantic mapping and standardization layer, connected to the unified access layer, is used to parse the intermediate metadata. Based on the pre-built global standard metadata model, the elements in the intermediate metadata are mapped to the corresponding standard concepts in the global standard metadata model through semantic similarity calculation, and standardized tags are added to the intermediate metadata according to the mapping results to generate standardized metadata. The central metadata repository layer, connected to the semantic mapping and standardization layer, is used to store and manage the standardized metadata, the global standard metadata model, and the history of mapping relationships, and provides version control and lineage tracing functions for metadata. The management and service layer, connected to the central metadata repository layer, is used to provide metadata registration, metamodel management, standardized strategy configuration, metadata retrieval and visualization services.

[0006] Furthermore, the semantic mapping and standardization layer includes: The model registration module is used to maintain and update the global standard metadata model, which defines core business entities, attributes, and relationships between entities in the form of an ontology. The semantic feature extraction module has a built-in trainable feature extraction model for extracting semantic features from elements in the intermediate metadata. The elements include table names, field names, and descriptive text. The extracted features include text feature vectors generated by the feature extraction model and graph representations based on metadata context relationships. The mapping calculation module has a built-in trainable similarity calculation model, which compares the extracted semantic features with the semantic features of standard concepts in the global standard metadata model. By combining the calculation of semantic similarity and structural similarity, it matches the most relevant standard concept to each intermediate metadata element. The calculation process is as follows: First, for a given intermediate metadata element ei and a global standard concept cj, obtain their text feature vectors and calculate the cosine similarity between them as the text semantic similarity. ; Secondly, by combining the structural context of an element in its respective data source pattern with the structural context of a concept in the global model, the structural similarity is calculated using the aforementioned similarity calculation model. This similarity is measured by comparing the Jaccard index of their neighboring node sets; Finally, the overall similarity is calculated using a weighted fusion function. The calculation method is as follows: ; in, These are configurable weighting coefficients, set through the management and service layer, used to adjust the contribution of text semantic and structural information in matching; The tag generation module is used to assign a standard concept unique identifier as a standardized tag to the intermediate metadata element based on the best matching result output by the mapping calculation module.

[0007] Furthermore, the management and service layer includes an active learning feedback module, which collects user correction operations on the automatic matching results of the mapping calculation module, and adds the corrected positive and negative sample pairs to the training set to continuously optimize the feature extraction model in the semantic feature extraction module and the similarity calculation model in the mapping calculation module in an incremental update manner.

[0008] Furthermore, the standardized tags used by the tag generation module are multi-level semantic tags; wherein, the first-level tag is the core business entity identifier in the global standard metadata model, which is directly determined by the mapping result; the second-level tag is a business purpose classification tag attached by predefined rules based on the usage scenarios of the intermediate metadata elements in the business value chain; and the third-level tag is a data sensitivity level tag determined based on data content and compliance requirements.

[0009] Furthermore, the metadata retrieval service provided by the management and service layer supports accurate retrieval based on the combination of the multi-level semantic tags and the comprehensive similarity. Perform fuzzy searches and related searches based on lineage records in the central metadata repository.

[0010] Furthermore, the unified access layer is configured with dedicated connectors for different heterogeneous data sources. The connectors encapsulate the data source-specific communication protocols and metadata extraction logic, and perform invalid character filtering, basic format verification, and redundant information removal in the initial cleaning process.

[0011] Furthermore, the version control function of the central metadata repository layer records the status, change operation type, operation time and operator of each change to standardized metadata, and the lineage tracing function displays the upstream data source and downstream consumption application of specific standardized metadata in the form of a directed graph.

[0012] Furthermore, the pre-built global standard metadata model is formed in the following way: core data models from multiple authoritative business systems are extracted and integrated to form an initial concept set; the initial concept set is semantically aligned, conflict resolved, and relationship supplemented using a domain knowledge graph, and finally a unified standardized model described in ontology language that can be read and reasoned about by machines is constructed.

[0013] Compared with existing technologies, this invention has the following advantages: This unified metadata standardization management system for heterogeneous data sources is compatible with various heterogeneous data sources such as relational databases, non-relational databases, and application programming interfaces (APIs). It achieves unified access and preliminary cleaning and encapsulation of raw metadata through a dedicated connector. Relying on a global standard metadata model and similarity calculation combining semantics and structure, it completes metadata standardization mapping and generates multi-level semantic tags containing information on core business entities, business uses, and data sensitivity levels. It supports precise, fuzzy, and associative retrieval, provides metadata version control and lineage tracing functions, and can continuously optimize the model through proactive learning feedback. It solves the problems of single protection types and poor protection effects in existing security protection systems, realizes unified and standardized management of heterogeneous metadata, improves the standardization, retrieval efficiency, and security of metadata management, and facilitates users to manage metadata throughout its entire lifecycle, making the system more worthy of widespread adoption. Attached Figure Description

[0014] Figure 1 This is a system block diagram of the present invention. Detailed Implementation

[0015] The embodiments of the present invention are described in detail below. These embodiments are implemented based on the technical solution of the present invention, and provide detailed implementation methods and specific operation processes. However, the scope of protection of the present invention is not limited to the following embodiments.

[0016] like Figure 1 As shown, this embodiment provides a technical solution: a unified metadata standardization management system for heterogeneous data sources, including: A unified access layer is used to receive raw metadata from multiple heterogeneous data sources, including at least relational databases, non-relational databases, and application programming interfaces, and to perform preliminary cleaning and format encapsulation on the raw metadata, and output intermediate metadata. The semantic mapping and standardization layer, connected to the unified access layer, is used to parse the intermediate metadata. Based on the pre-built global standard metadata model, the elements in the intermediate metadata are mapped to the corresponding standard concepts in the global standard metadata model through semantic similarity calculation, and standardized tags are added to the intermediate metadata according to the mapping results to generate standardized metadata. The central metadata repository layer, connected to the semantic mapping and standardization layer, is used to store and manage the standardized metadata, the global standard metadata model, and the history of mapping relationships, and provides version control and lineage tracing functions for metadata. The management and service layer, connected to the central metadata repository layer, is used to provide metadata registration, metamodel management, standardized strategy configuration, metadata retrieval and visualization services.

[0017] The semantic mapping and normalization layer includes: The model registration module is used to maintain and update the global standard metadata model, which defines core business entities, attributes, and relationships between entities in the form of an ontology. The semantic feature extraction module has a built-in trainable feature extraction model for extracting semantic features from elements in the intermediate metadata. The elements include table names, field names, and descriptive text. The extracted features include text feature vectors generated by the feature extraction model and graph representations based on metadata context relationships. The mapping calculation module has a built-in trainable similarity calculation model, which compares the extracted semantic features with the semantic features of standard concepts in the global standard metadata model. By combining the calculation of semantic similarity and structural similarity, it matches the most relevant standard concept to each intermediate metadata element. The calculation process is as follows: First, for a given intermediate metadata element ei and a global standard concept cj, obtain their text feature vectors and calculate the cosine similarity between them as the text semantic similarity. ; Secondly, by combining the structural context of an element in its respective data source pattern with the structural context of a concept in the global model, the structural similarity is calculated using the aforementioned similarity calculation model. This similarity is measured by comparing the Jaccard index of their neighboring node sets; Finally, the overall similarity is calculated using a weighted fusion function. The calculation method is as follows: ; in, These are configurable weighting coefficients, set through the management and service layer, used to adjust the contribution of text semantic and structural information in matching; The tag generation module is used to assign a standard concept unique identifier as a standardized tag to the intermediate metadata element based on the best matching result output by the mapping calculation module. By subdividing modules, the global standard metadata model can be dynamically maintained, the semantic and structural features of metadata can be accurately extracted, and efficient and accurate mapping can be achieved by combining dual similarity calculations to generate standardized tags, ensuring the accuracy and flexibility of heterogeneous metadata standardization.

[0018] For example, if a company has a "user information table" in a relational database and a "user_profile set" in a non-relational database, the semantic feature extraction module extracts the text feature vectors related to "user identity" and the association graph representation between the tables / sets. The mapping calculation module first matches the standard concept of "user core information" in the global model through cosine similarity, and then confirms the rationality of the matching by comparing the structural context through Jaccard index. Finally, standardized labels are generated to unify and standardize user-related metadata from different data sources.

[0019] The management and service layer includes an active learning feedback module, which is used to collect user correction operations on the automatic matching results of the mapping calculation module, and add the corrected positive and negative sample pairs to the training set to continuously optimize the feature extraction model in the semantic feature extraction module and the similarity calculation model in the mapping calculation module in an incremental update manner. By collecting user corrections through an active learning feedback mechanism, the corrected positive and negative sample pairs are incrementally updated to the training set, continuously optimizing the semantic feature extraction model and similarity calculation model. This makes metadata mapping and matching more accurate, adapts to dynamic changes in business scenarios, and improves the long-term reliability and adaptability of the system.

[0020] The standardized tags used by the tag generation module are multi-level semantic tags; wherein, the first-level tag is the core business entity identifier in the global standard metadata model, which is directly determined by the mapping result; the second-level tag is the business purpose classification tag attached by predefined rules based on the usage scenarios of the intermediate metadata elements in the business value chain; and the third-level tag is the data sensitivity level tag determined based on data content and compliance requirements. Multi-level semantic tags take into account core business attributes, business uses, and compliance requirements, making standardized metadata classification more refined and information more complete. This facilitates accurate identification of the core value of metadata, meets the needs of business scenario adaptation and compliance control, and improves the practicality and security of metadata management.

[0021] In scenarios with heterogeneous data sources for enterprises, the metadata of the "Bank Card Number" field in the "User Information Table" of a relational database is labeled as follows: First-level label: "Core User Information" (core business entity, determined by mapping results); Second-level label: "Account Funds Association" (business purpose, corresponding to user payment and fund management scenarios); Third-level label: "Highly Sensitive" (compliance requirement, due to involvement of financial privacy data). These three levels of labels quickly clarify the core attributes, application scenarios, and protection level of this field.

[0022] The metadata retrieval service provided by the management and service layer supports precise retrieval based on the combination of multi-level semantic tags and the comprehensive similarity. Perform fuzzy searches and related searches based on lineage records in the central metadata repository layer; It supports multi-level semantic tag precise retrieval, comprehensive similarity fuzzy retrieval, and blood relationship association retrieval, covering different retrieval needs, making metadata retrieval more efficient and accurate, and at the same time, it can trace the upstream and downstream of data through association retrieval, improving the data traceability and association analysis capabilities.

[0023] In scenarios involving the "User Information Table" and "Bank Card Number" fields, precise retrieval can be achieved by combining three levels of tags: "User Core Information + Account Fund Association + High Sensitivity," which can quickly locate the metadata of this field. Fuzzy retrieval can be achieved by matching fields such as "Bank Card Number" and "Balance" with high comprehensive similarity through keywords related to "User Finance." Related retrieval can be achieved by tracing the upstream data source (user registration system) and downstream consumer applications (payment settlement platform, risk control and audit system) through this field.

[0024] The unified access layer is configured with dedicated connectors for different heterogeneous data sources. The connectors encapsulate the data source-specific communication protocols and metadata extraction logic, and perform invalid character filtering, basic format verification, and redundant information removal in the initial cleaning process. Dedicated connectors adapt to the communication protocols and extraction logic of different heterogeneous data sources, enabling rapid access to multi-source raw metadata. The initial cleaning process removes invalid and redundant information and verifies the format, ensuring the efficiency and data quality of subsequent metadata processing.

[0025] In scenarios with heterogeneous data sources for enterprises, the unified access layer connects to relational databases (such as MySQL) and application programming interfaces (such as user registration APIs) through dedicated connectors. The MySQL connector encapsulates the JDBC protocol to extract metadata from the "user information table," while the API connector extracts the metadata submitted by users during registration according to the REST protocol. During the initial cleaning process, garbled characters in the table are filtered, duplicate fields are removed, and date formats are validated. Standardized intermediate metadata is then output for subsequent processing.

[0026] The version control function of the central metadata repository layer records the status, change operation type, operation time and operator of each change to standardized metadata. The lineage tracing function displays the upstream data source and downstream consumption application of specific standardized metadata in the form of a directed graph. Version control fully records the details of changes to standardized metadata, making it easy to trace the change trajectory and troubleshoot problems; lineage tracing uses a directed graph to intuitively display the upstream and downstream relationships of data, helping with data traceability, impact analysis and compliance auditing, and improving the transparency and controllability of data management.

[0027] In scenarios involving the "User Information Table" and "Bank Card Number" fields, version control records the operator, time, and reason for changing the field label from "Medium Sensitive" to "High Sensitive." Lineage tracing is displayed through a directed graph, clearly showing the data flow path from the upstream "Bank Card Information Submission" module of the user registration system to the downstream "Funds Deduction" function of the payment and settlement platform and the "Compliance Verification" module of the risk control system.

[0028] The pre-built global standard metadata model is formed in the following way: core data models from multiple authoritative business systems are extracted and integrated to form an initial concept set; the initial concept set is semantically aligned, conflict resolved and relationship supplemented using a domain knowledge graph, and finally a unified standardized model described in ontology language that can be read and reasoned by machines is constructed. A global standard metadata model is constructed based on the core data model of the authoritative business system and the domain knowledge graph. This ensures the model's authority, completeness, and semantic consistency, and allows for machine reading and reasoning. It provides a reliable benchmark for the unified and standardized mapping of heterogeneous metadata and guarantees the accuracy of cross-data source metadata integration.

[0029] When constructing a global standard metadata model for enterprise user-related data scenarios, the core data models such as "user information" and "account funds" from authoritative systems like e-commerce transaction systems and user management systems are first extracted to form an initial concept set. Then, through domain knowledge graphs, the semantics of "user ID" and "member number" in different systems are aligned, conflicts in the definition of the "user contact information" field are resolved, and the relationship between "user core information" and "payment transactions" is supplemented. Finally, a unified model is formed, allowing the "user information table" in MySQL and the "registration data" in the API to achieve standardized mapping based on this model.

[0030] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0031] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0032] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A unified metadata standardization management system for heterogeneous data sources, characterized in that: include: A unified access layer is used to receive raw metadata from multiple heterogeneous data sources, including at least relational databases, non-relational databases, and application programming interfaces, and to perform preliminary cleaning and format encapsulation on the raw metadata, and output intermediate metadata. The semantic mapping and standardization layer, connected to the unified access layer, is used to parse the intermediate metadata. Based on the pre-built global standard metadata model, the elements in the intermediate metadata are mapped to the corresponding standard concepts in the global standard metadata model through semantic similarity calculation, and standardized tags are added to the intermediate metadata according to the mapping results to generate standardized metadata. The central metadata repository layer, connected to the semantic mapping and standardization layer, is used to store and manage the standardized metadata, the global standard metadata model, and the history of mapping relationships, and provides version control and lineage tracing functions for metadata. The management and service layer, connected to the central metadata repository layer, is used to provide metadata registration, metamodel management, standardized strategy configuration, metadata retrieval and visualization services.

2. The unified metadata standardization management system for heterogeneous data sources according to claim 1, characterized in that: The semantic mapping and normalization layer includes: The model registration module is used to maintain and update the global standard metadata model, which defines core business entities, attributes, and relationships between entities in the form of an ontology. The semantic feature extraction module has a built-in trainable feature extraction model for extracting semantic features from elements in the intermediate metadata. The elements include table names, field names, and descriptive text. The extracted features include text feature vectors generated by the feature extraction model and graph representations based on metadata context relationships. The mapping calculation module has a built-in trainable similarity calculation model, which compares the extracted semantic features with the semantic features of standard concepts in the global standard metadata model. By combining the calculation of semantic similarity and structural similarity, it matches the most relevant standard concept to each intermediate metadata element. The calculation process is as follows: First, for a given intermediate metadata element ei and a global standard concept cj, obtain their text feature vectors and calculate the cosine similarity between them as the text semantic similarity. ; Secondly, by combining the structural context of an element in its respective data source pattern with the structural context of a concept in the global model, the structural similarity is calculated using the aforementioned similarity calculation model. This similarity is measured by comparing the Jaccard index of their neighboring node sets; Finally, the overall similarity is calculated using a weighted fusion function. The calculation method is as follows: ； in, These are configurable weighting coefficients, set through the management and service layer, used to adjust the contribution of text semantic and structural information in matching; The tag generation module is used to assign a standard concept unique identifier as a standardized tag to the intermediate metadata element based on the best matching result output by the mapping calculation module.

3. The unified metadata standardization management system for heterogeneous data sources according to claim 2, characterized in that: The management and service layer includes an active learning feedback module, which collects user correction operations on the automatic matching results of the mapping calculation module, and adds the corrected positive and negative sample pairs to the training set to continuously optimize the feature extraction model in the semantic feature extraction module and the similarity calculation model in the mapping calculation module in an incremental update manner.

4. The unified metadata standardization management system for heterogeneous data sources according to claim 3, characterized in that: The standardized tags used by the tag generation module are multi-level semantic tags; wherein, the first-level tag is the core business entity identifier in the global standard metadata model, which is directly determined by the mapping result; the second-level tag is the business purpose classification tag attached by predefined rules based on the usage scenarios of the intermediate metadata elements in the business value chain; and the third-level tag is the data sensitivity level tag determined based on data content and compliance requirements.

5. The unified metadata standardization management system for heterogeneous data sources according to claim 4, characterized in that: The metadata retrieval service provided by the management and service layer supports precise retrieval based on the combination of multi-level semantic tags and the comprehensive similarity. Perform fuzzy searches and related searches based on lineage records in the central metadata repository.

6. The unified metadata standardization management system for heterogeneous data sources according to claim 5, characterized in that: The unified access layer is configured with dedicated connectors for different heterogeneous data sources. The connectors encapsulate the data source-specific communication protocols and metadata extraction logic, and perform invalid character filtering, basic format verification, and redundant information removal in the initial cleaning process.

7. The unified metadata standardization management system for heterogeneous data sources according to claim 6, characterized in that: The version control function of the central metadata repository layer records the status, change operation type, operation time and operator of each change to standardized metadata. The lineage tracing function displays the upstream data source and downstream consumption application of specific standardized metadata in the form of a directed graph.

8. The unified metadata standardization management system for heterogeneous data sources according to claim 7, characterized in that: The pre-built global standard metadata model is formed in the following way: core data models from multiple authoritative business systems are extracted and integrated to form an initial concept set; the initial concept set is semantically aligned, conflict resolved and relationship supplemented using a domain knowledge graph, and finally a unified standardized model described in ontology language that can be read and reasoned by machines is constructed.

Images