A metadata management method and platform applied to enterprise-level data governance

Abstract

The technical scheme of the present application provides a metadata management method applied to enterprise-level data governance.Another technical scheme of the present application provides a metadata management platform, which comprises a meta-model management module, a metadata query module, a metadata API module, a metadata analysis module and an operation log module.The technical scheme disclosed by the present application structures metadata definition into specific information, so that the metadata of each system can be structured, expressed and recorded.In addition, the present application defines the relationship between meta-models, and the description of the relationship can clearly link different systems, describe the relationship between systems, and help prevent the circular association, repeated association or wrong association between systems.

Description

Technical Field

[0001] This invention relates to a metadata management method and platform that supports custom meta-models, belonging to the technical field of data asset management methods and platforms. Background Technology

[0002] In the field of data asset management, enterprises use metadata to describe the data itself and the information during its use. They also use metadata relationships to perform lineage analysis, impact analysis, and correlation analysis to track and record key data information, quickly grasp the potential impact of data changes, effectively assess the risks that data changes may bring, and intuitively identify the importance of data.

[0003] Existing metadata management methods are primarily system-specific, further including metadata acquisition modules, metadata identification modules, metadata classification modules, metadata viewing modules, and metadata application modules. In actual enterprise data asset management, it's necessary to connect various different business systems and build a global data view. However, the data types and sources of data from each business system are diverse, making single-system metadata management tools clearly insufficient. Summary of the Invention

[0004] The purpose of this invention is to provide a technical solution that supports unified metadata management across multiple systems.

[0005] To achieve the above objectives, the technical solution of the present invention provides a metadata management method for enterprise-level data governance, characterized by comprising the following steps:

[0006] Step 1: Define enterprise-level metadata. By defining enterprise-level metadata, the metadata is structured into specific information, enabling the metadata of different systems to be represented and recorded in a structured manner. This includes the following steps:

[0007] Step 101: The first user sets the first meta-model of the first system. The complete meta-model is constructed by defining attribute fields and primary key member fields. Each meta-model defines a business primary key, which consists of multiple attribute fields. The first data structure is generated based on the attribute fields. At the same time, the field types are parsed to generate JSON format templates for multiple scenarios, which are used as the first set of data interfaces.

[0008] Step 102: The second user, different from the first user, sets up the second metamodel of the second system and sets the relationship between the first metamodel and the second metamodel by foreign key association. The second system is different from the first system. The foreign key of the second metamodel stores the model ID of the first metamodel. At the same time, the relationship definition between the first metamodel and the second metamodel is generated. The relationship definition retains at least the association field. The association field replaces the metamodel ID of the first metamodel with the JSON structure of the first metamodel through the metamodel relationship tree. The associated metamodels constitute the metamodel relationship tree.

[0009] Step 103: Generate a second data structure based on the second meta-model of the second system set by the second user, and generate a second set of data interfaces. The second user focuses on the business definition of the first meta-model and locates the data of the first meta-model through the business primary key of the first meta-model.

[0010] Step 2, Metadata Management, includes data collection from the metamodel, metadata association and replacement, metadata relationship maintenance, and metadata query;

[0011] Metamodel data acquisition: The acquisition party enters metadata in JSON format through the data addition, deletion and modification interface. The JSON defines the metamodel relationship tree. The leaf metamodel data in the model relationship tree is defined as the first target metamodel data, and the root node metamodel data is defined as the second target metamodel data. The acquisition party determines the first target primary key and the second target primary key through the business primary key, and replaces the first target metamodel JSON format associated in the second target metamodel data with the first target primary key. After all the first target metamodel data is replaced, the second target metamodel data is saved.

[0012] Metadata association and replacement: If the first target metamodel and the second target metamodel constitute a metamodel association tree, the following steps are included:

[0013] Starting from the first metamodel in the leaf node, load the first target metamodel data. Retrieve the metamodel ID of the first target metamodel data from storage using the business primary key. If retrieval fails, a metadata association error will be displayed. If retrieval is successful, replace the first target metamodel data in the second target metamodel data with the metamodel ID. Repeat this process until all first target metamodel data has been replaced.

[0014] Processing the currently entered root node metamodel: Compare the current metadata with the historical metadata business primary key to determine if it already exists, and then take further add, delete, or modify operations: When adding, automatically generate the primary key of the second target metamodel data; when deleting or modifying, verify the upstream and downstream association of metadata through the metamodel relationship definition to prevent incorrect data association or deletion of associated data that could lead to a break in the relationship chain.

[0015] Metadata relationship maintenance: Metadata relationships are independent of metadata fields and stored separately in the distributed graph database. Based on the relationships and directions defined in the metamodel, the distributed graph database constructs a graph of all metamodel data, with metamodel IDs as nodes and metamodel relationships as edges. When collecting the first metamodel data, the metamodel ID of the first metamodel data is written to the distributed graph database as the first target metadata node. When collecting the second target metamodel data, the metamodel ID of the second metamodel data is written to the distributed graph database as the second target metadata node, and the edge from the second metadata node to the first metadata node is also written.

[0016] Metadata query: This includes metadata relationship query and metadata information query, where:

[0017] The metadata relationship module constructs a graph structure from all metadata data through the metadata model relationship. It can be used to find specific metadata types or metadata relationship types, and can also be used to query the scope of influence of a metadata or a class of metadata, or to perform statistics on a class of metadata or a class of metadata relationships.

[0018] After storing the specific metadata information in the metadata information module, the meta-model relationship tree is constructed in real time. When querying metadata information, the complete upstream metadata information is assembled through the meta-model relationship tree.

[0019] Preferably, in step 101, the first data structure is generated using a relational database table creation method.

[0020] Preferably, in step 102, the relationship definition retains the metamodel, metamodel association fields, and metamodel association direction.

[0021] Preferably, the first set of data interfaces in step 101 and the second set of data interfaces in step 103 include a query interface, a full update interface, a selective update interface, and a delete interface. The query interface is used to query metadata information in various scenarios, while the full update interface, the selective update interface, and the delete interface are used by the metadata collection program to insert, update, and delete metadata information.

[0022] Preferably, the user can locate its upstream and downstream metamodels through an attribute field of a metamodel.

[0023] Preferably, in step 1, the structured enterprise-level metadata includes basic metadata information and metadata attributes. The basic metadata information includes metamodel ID, metamodel category, metamodel name, metamodel description, and metamodel owner. The metadata attributes include attribute English name, attribute Chinese name, field type, whether it is a business primary key, whether it is associated with a metamodel, and associated metamodel information.

[0024] Another technical solution of the present invention is to provide a metadata management platform for implementing the above-mentioned metadata management method applied to enterprise-level data governance.

[0025] Preferably, it includes a metamodel management module, a metadata query module, a metadata API module, a metadata analysis module, and an operation log module, wherein:

[0026] The metamodel management module supports users in managing metamodel categories and defining basic metamodel information, attribute information, and relationship information.

[0027] The metadata query module allows users to query and locate metadata based on its various attribute values, and to manually add, update, and delete metadata information.

[0028] The metadata API module allows users to view query interfaces, perform full updates on interfaces, select and update interfaces, delete interface information and calling methods, apply for interface calling permissions, and view interface calling logs.

[0029] The metadata analysis module allows users to query data lineage and metadata correlation based on metadata relationship information;

[0030] The operation log module allows platform administrators to audit user operations.

[0031] Compared with existing technical solutions, the present invention has the following advantages:

[0032] The technical solution disclosed in this invention defines metadata as structured information, thus enabling the metadata of each system to be structurally represented and recorded. Furthermore, this invention defines the relationships between meta-models; the description of these relationships clearly connects different systems, describing the relationships between systems and helping to prevent circular, duplicate, or erroneous associations between them. Attached Figure Description

[0033] Figure 1 A flowchart for defining metadata methods;

[0034] Figure 2 A diagram illustrating metadata collection and storage;

[0035] Figure 3 This is a schematic diagram of a metadata management platform. Detailed Implementation

[0036] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0037] One aspect of this invention discloses a method for defining meta-templates. This method is applied to the flexible definition of basic information, attributes, and relationships of metadata within different systems, supporting unified management of metadata across the entire enterprise system, such as... Figure 1 As shown, it includes the following steps:

[0038] S10: The first user sets the first meta-model of the first system on the metadata management platform;

[0039] S20: In the metadata management platform, the first metamodel of the first system, defined by the first user, is constructed by defining attribute fields and primary key member fields to form a complete metamodel. The metadata management platform uses a method similar to that of relational database table creation to generate the first data structure based on the attribute fields, and at the same time parses the field types to generate JSON format templates for multiple scenarios, which are used as the first set of data interfaces.

[0040] S30: The second user sets up the second metamodel of the second system on the metadata management platform, and sets the relationship between the first metamodel and the second metamodel by using foreign key association. The foreign key of the second metamodel will store the ID of the first metamodel, and at the same time generate the relationship definition between the first metamodel and the second metamodel. The relationship definition retains information such as the metamodel, the metamodel association field, and the metamodel association direction.

[0041] S40: The metadata management platform generates a second data structure and a second set of data interfaces based on the second metamodel of the second system defined by the second user. Unlike the interfaces of the first metamodel, the association fields of the second metamodel will replace the ID of the first metamodel with the JSON structure of the first metamodel through the metamodel relationship tree. The second user needs to focus on the business definition of the first metamodel and locate the data of the first metamodel through its business primary key. The data interface definition is the metamodel relationship subtree, and users can locate the upstream and downstream metamodels through the attribute fields of a metamodel.

[0042] S50: Enterprise-level metadata definition complete.

[0043] In the above solution, the metadata management platform structures metadata into specific information. Its main structured modules include basic metadata information and metadata attributes. Basic metadata information includes, but is not limited to, metamodel ID, metamodel category, metamodel name, metamodel description, and metamodel owner. Metadata attributes include, but are not limited to, attribute English name, attribute Chinese name, field type, whether it is a business primary key, whether it is associated with a metamodel, and associated metamodel information. This allows the metadata of each system to be structurally represented and recorded. Furthermore, the relationships between metamodels are defined, and the description of these relationships clearly connects different systems. This not only describes the relationships between systems but also helps prevent circular, duplicate, or erroneous associations between systems.

[0044] In the above scheme, the metadata management platform generates a set of data interfaces for each defined meta-model, including query interfaces, full update interfaces, selective update interfaces, and deletion interfaces. The query interface can be used to query metadata information in various scenarios. The full update interface, selective update interface, and deletion interface are used by the metadata collection program to insert, update, and delete metadata information.

[0045] The second aspect of this invention provides a unified metadata collection, processing, and storage method adapted to the aforementioned custom meta-model, ensuring the accuracy of metadata and the scalability of metadata management.

[0046] For data collection from metamodels, after defining the metamodel, an interface for adding, deleting, and modifying metamodel data is generated. The data collector will input metadata in JSON format through this interface. The JSON defines a metamodel relationship tree, where each metamodel defines a business primary key, which consists of multiple attribute fields. Related metamodels form the metamodel relationship tree. Leaf metamodel data is defined as the first target metadata, and root node metamodel data is defined as the second target metadata. The data collector determines the first and second target primary keys using the business primary key and replaces the primary key in the JSON format of the associated first target metamodel in the second target metadata. After all first target metamodels have been replaced, the second target metamodel data is saved.

[0047] Metadata association and replacement method: The first target metamodel and the second target metamodel constitute a metamodel association tree. First, starting with the first target metadata at the leaf node, load the first target metadata. Retrieve the first target metadata ID from storage using the business primary key. If retrieval fails, a metadata association error will be displayed. If successful, replace the first target metadata data in the second target metamodel data with the ID, until all first target metadata data has been replaced. Then, process the currently entered root node metamodel. First, compare the current metadata with the historical metadata business primary key to determine if it already exists, and then take further add, delete, or modify operations. When adding data, the second target metamodel data primary key will be automatically generated. When deleting or modifying data, the upstream and downstream associations of metadata will be verified through the metamodel relationship definition to prevent incorrect data association or deletion of associated data that could break the relationship chain.

[0048] Metadata relationship maintenance method: Metadata relationships are stored independently of metadata fields in a distributed graph database. Using the relationships and directions defined in the metamodel, the graph database constructs a graph with metadata IDs as nodes and metamodel relationships as edges. When collecting the first metamodel data, the first metamodel data ID is written to the graph database as the first target metadata node; when collecting the second target metamodel data, the second metamodel data ID is written to the graph database as the second target metadata node, and the edge from the second metadata node to the first metadata node is created.

[0049] The attribute information defined in the metamodel definition specifies the field type, whether it is non-nullable, and whether it is associated with a metamodel. These parameters will be validated during data entry to ensure the consistency of data from the same metamodel.

[0050] Metadata query methods include metadata relationship queries and metadata information queries:

[0051] a) The metadata relationship module constructs a graph structure from all metadata through meta-model relationships, which is used to find specific metadata types or metadata relationship types. It can also be used to query the scope of influence of a single metadata type or a class of metadata, or to perform statistics on a class of metadata or a class of metadata relationships.

[0052] b) Metadata information module, which stores specific metadata information. It constructs a metamodel relationship tree in real time, and when querying metadata information, it assembles complete upstream metadata information through the metamodel relationship tree.

[0053] The third aspect of this invention discloses a metadata management platform, including modules for metamodel management, metadata query, metadata API, metadata analysis, and operation logs. The metamodel management module supports users in managing metamodel categories and defining basic, attribute, and relational information. The metadata query module allows users to locate metadata based on its attribute values ​​and manually add, update, and delete metadata information. The metadata API module allows users to view query interfaces, full update interfaces, select update interfaces, delete interface information and calling methods, apply for interface calling permissions, and view interface call logs. The metadata analysis module allows users to query data lineage, metadata correlation, and other information based on metadata relational information. The operation log module allows platform administrators to audit user operations.

Claims

1. A metadata management method applied to enterprise-level data governance, characterized in that, Comprising the following steps: Step 1, defining enterprise-level metadata, structuring metadata definition into specific information by defining enterprise-level metadata, so that the metadata of different systems can be structured, recorded and expressed, comprising the following steps: Step 101, the first user sets the first meta model of the first system, and the complete meta model is composed of attribute fields and primary key member fields, each meta model defines a business primary key composed of multiple attribute fields, and generates a first data structure according to the attribute fields, and parses the field type to generate a multi-scene JSON format template for the first group of data interfaces; Step 102, unlike the first user, the second user sets the second meta model of the second system, and sets the relationship between the first meta model and the second meta model in the form of foreign key association, wherein the second system is different from the first system, the foreign key of the second meta model saves the model ID of the first meta model, and the relationship definition between the first meta model and the second meta model is generated, wherein at least the association field is reserved on the relationship definition, the association field replaces the meta model ID of the first meta model with the JSON structure of the first meta model through the meta model relationship tree, and the associated meta models constitute a meta model relationship tree; Step 103, according to the second meta model of the second system set by the second user, a second data structure is generated, and a second group of data interfaces is generated, the second user focuses on the business definition of the first meta model, and locates the first meta model data through the business primary key of the first meta model; Step 2, metadata management, including data collection of meta model, metadata association and replacement, metadata relationship maintenance and metadata query; Data collection of meta model: the collection party enters metadata in JSON format through add, delete and modify data interfaces, wherein the JSON defines a meta model relationship tree, the leaf meta model data in the model relationship tree is defined as first target meta model data, and the root node meta model data is defined as second target meta model data, then the collection party determines the first target primary key and the second target primary key through the business primary key, and replaces the first target meta model JSON format associated in the second target meta model data with the first target primary key, after all the first target meta model data is replaced, the second target meta model data is saved; Metadata association and replacement: the first target meta model and the second target meta model constitute a meta model association tree, comprising the following steps: Load the first target meta model data from the first meta model of the leaf node as the starting point, obtain the meta model ID of the first target meta model data from the storage through the business primary key: if the acquisition fails, it will prompt metadata association error; if the acquisition is successful, replace the first target meta model data in the second target meta model data with the meta model ID, until all the first target meta model data is replaced; Processing the current entered root node meta model: comparing the current metadata with the historical metadata business primary key, judging whether it already exists, and then taking further add, delete and modify operations: when adding, automatically generating a second target meta model data primary key; when deleting and modifying, checking the upstream and downstream association of the metadata through the meta model relationship definition to prevent data error association or the deletion of associated data from causing the relationship chain to break; Metadata relationship maintenance: the metadata relationship is independent of the metadata field and is stored separately in a distributed graph database. In the distributed graph database, the meta model ID is used as a point, the meta model relationship is used as an edge, and all meta model data is constructed into a graph. When collecting the first meta model data, the meta model ID of the first target meta data point is written into the distributed graph database. When collecting the second target meta model data, the meta model ID of the second meta data point is written into the distributed graph database, and the edge of the second meta data point points to the first meta data point. Metadata query: including metadata relationship query and metadata information query, wherein: The metadata relationship module uses the meta model relationship to construct all meta model data into a graph structure for finding specific metadata types or metadata relationship types, and can also be used to query the influence range of one or a class of metadata or to perform statistics on a class of metadata or a class of metadata relationship. After saving the specific metadata information in the metadata information module, a meta model relationship tree is constructed in real time, and when querying the metadata information, the upstream complete metadata information is assembled through the meta model relationship tree.

2. The metadata management method for enterprise data governance according to claim 1, wherein, In step 101, the first data structure is generated using the table building method of a relational database.

3. The metadata management method for enterprise data governance of claim 1, wherein, In step 102, the relationship definition retains the meta model, the meta model associated field, and the meta model associated direction.

4. The metadata management method for enterprise data governance of claim 1, wherein, The first group of data interfaces in step 101 and the second group of data interfaces in step 103 include query interfaces, full update interfaces, selection update interfaces, and deletion interfaces. The query interfaces are used to query metadata information in various scenarios, and the full update interfaces, the selection update interfaces, and the deletion interfaces are used for metadata collection programs to insert, update, and delete metadata information.

5. The metadata management method for enterprise data governance of claim 1, wherein, The user can locate the upstream and downstream meta models through the attribute fields of a meta model.

6. The metadata management method for enterprise data governance of claim 1, wherein, In step 1, the structured enterprise-level metadata includes metadata basic information and metadata attributes. The metadata basic information includes meta model ID, meta model classification, meta model name, meta model explanation, and meta model responsible person. The metadata attributes include attribute English name, attribute Chinese name, field type, whether it is a business primary key, whether it is associated with a meta model, and associated meta model information.

7. A metadata management platform for implementing the metadata management method for enterprise-level data governance according to claim 1.

8. A metadata management platform as claimed in claim 7, characterized in that, The metadata management platform includes a meta model management module, a metadata query module, a metadata API module, a metadata analysis module, and an operation log module, wherein: The meta model management module supports user management of meta model classification, definition of meta model basic information, attribute information, and relationship information; The metadata query module supports users to query and locate metadata according to attribute values of the metadata, and manually add, update and delete metadata information; The metadata API module supports users to view interface information and calling modes of a query interface, a full update interface, a selection update interface and a deletion interface, apply for calling permission of the interface, and view calling logs of the interface; The metadata analysis module supports users to query data blood relationship and metadata correlation based on metadata relationship information; The operation log module supports platform administrators to audit operation behaviors of users.

Images