A metadata management system and method for heterogeneous data sources

Abstract

The present invention discloses a metadata management system and method for heterogeneous data sources, belonging to the field of big data technology. In the system of the present invention, the user selects an adapter, configures the version and connection parameters of the collected tool or product, manages and maintains the data source configuration information; uses a workflow engine to manage metadata collection tasks, configures automated workflows, and supports parameter configuration of collection tasks; uses several collection programs to concurrently execute collection tasks, supports several collection methods and different types of data sources, and uses Excel file format for template collection to maintain and import metadata; organizes the collected metadata into a structure that conforms to the public warehousing model and stores it in a relational database; uses the TF-IDF algorithm for querying; constructs a data flow diagram, traces the data source forward, and traces the data flow backward; performs environmental consistency inspections, supports metadata lifecycle management, and provides change query and subscription functions.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a metadata management system and method for heterogeneous data sources. Background Art

[0002] With the advent of the big data era, the amount of data has exploded, making metadata collection even more important in order to manage and understand massive amounts of data. Metadata collection is a key link in the field of data management and data analysis, which involves collecting descriptive information about data from various data sources. Data comes from a variety of sources, including structured data, unstructured data, and semi-structured data, which requires metadata collection technology to be able to adapt to different types of data sources. In environments such as data warehouses and data lakes, data from different sources needs to be integrated. Metadata collection is the basis for achieving data integration and ensuring data consistency. Effective data governance relies on accurate metadata. Metadata collection helps to establish data catalogs, monitor data quality, and implement data security policies.

[0003] Many existing metadata management systems only support a single type or a few types of data sources, and cannot effectively cope with environments with multiple heterogeneous data sources. Traditional metadata collection methods mostly use a single program to serially execute collection tasks, which is inefficient and cannot meet the real-time or quasi-real-time requirements in large-scale data environments. Due to the lack of a unified metadata standard, metadata between different systems is difficult to communicate, resulting in serious data islands and the inability to achieve cross-system and cross-platform data integration and sharing. The metadata query and analysis functions of existing systems are limited and it is difficult to meet the diverse needs of users. Summary of the invention

[0004] The object of the present invention is to provide a metadata management system and method for heterogeneous data sources to solve the problems raised in the above background technology.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0006] A metadata management method for heterogeneous data sources, the method comprising the following steps:

[0007] S100, the user selects an adapter, configures the version and connection parameters of the collected tool or product, manages and maintains data source configuration information, and adds, modifies, deletes and authorizes data source configuration information;

[0008] S200. Use the workflow engine to manage metadata collection tasks, configure automated workflows, and implement scheduled tasks or event-driven collection strategies; support adding, modifying, and deleting collection tasks, and configure the time, status, and scheduling interval for automatic execution of tasks;

[0009] S300, using several collection programs to concurrently execute collection tasks, supporting several collection methods and different types of data sources, and using Excel file format for template collection to maintain and import metadata;

[0010] S400, organize the collected metadata into a structure that conforms to the public storage model and store it in a relational database; use the TF-IDF algorithm to perform fuzzy query, classification query and advanced query; build a data flow graph, use graph neural network to model and analyze the data flow, identify key nodes and data dependencies in the data processing flow, trace the data source forward, and trace the data flow backward; use the isolation forest algorithm to perform environmental consistency inspection, compare the metadata of different environments, and generate an environmental inconsistency report;

[0011] S500, supports metadata lifecycle management, provides version viewing and version publishing functions; provides change query and subscription functions, users can query and subscribe to metadata change information, the platform sends change details to the user's mailbox, and the user views the change details on the platform.

[0012] According to step S100, according to the source of metadata, the user selects a collection adapter, and the adapter types include: DB data dictionary collection adapter, template collection adapter, data record collection adapter, ETL tool collection adapter, big data collection adapter, modeling tool collection adapter, report tool collection adapter, log collection adapter and script collection adapter; wherein, the DB data dictionary collection adapter is used to collect the data dictionary of the database, the template collection adapter is used to collect data through Excel templates, the data record collection adapter is used to collect database data records, the ETL tool collection adapter supports metadata collection of ETL tools, the big data collection adapter is used to collect metadata of Hive, Hbase and HDFS, the modeling tool collection adapter supports metadata collection of Powerdesigner and ERWin, the report tool collection adapter supports metadata collection of Cognos and BI.Office, the log collection adapter supports metadata collection of Shell and Perl logs, and the script collection adapter supports metadata collection of DB data dictionary, DDL, SparkSql, Sqoop collection and stored procedures;

[0013] The DB data dictionary collection adapter supports the following common databases: Oracle, DB2, MSSQL Server, MySQL, Teradata, HyperSQL, Sybase ASE, SybaseIQ, Informix, DM, Oscar, KingBase, Netezza, PostgreSQL, HighGo, GBase; the data record collection adapter supports the collection of DB data records, such as Oracle, DB2, MSSQL Server, MySQL, Teradata, HyperSQL, Sybase ASE, Informix, Netezza; the ETL tool collection adapter supports ETL tools such as Datastage, PowerCenter, Kettle, DI, ODI, etc.

[0014] The user selects and configures the specific version information of the tool or product used so that the adapter can correctly identify and collect metadata of the corresponding version; based on the selected adapter and the collected tool, the user needs to provide necessary connection parameters, which include IP address, port, user name, password and database name for the database, server address, port, user name and password for the ETL tool, and corresponding connection parameters for other tools and interfaces according to the requirements of the specific tool.

[0015] According to step S200, the metadata collection task is managed using the workflow engine Apache Airflow. The workflow engine provides task scheduling and management functions, and can organize and control the execution process of the collection task. According to the needs of metadata collection, an automated workflow is designed. The workflow consists of several task nodes, each node represents a specific collection task, and the dependencies between tasks are defined in the workflow, and the execution strategy of the workflow is configured. The execution time and frequency of the scheduled task are set, and the setting of the frequency according to every minute, every hour, every day, every week, every month and one-time is supported.

[0016] Distributed deployment of multiple collectors solves the problem of cross-cloud metadata collection. For the same metadata collection source, it supports active and standby collectors to achieve highly reliable metadata collection. When adding a new collector, it will communicate with the collection end to monitor whether the heartbeat of the collection end is normal. Only when it is normal can it be added. After the addition is completed, it will maintain a heartbeat status check with the collection end and determine whether it is offline based on the heartbeat status.

[0017] According to step S300, the collection tasks are assigned to different collection programs to ensure balanced distribution of tasks. The workflow engine Apache Airflow is responsible for scheduling and monitoring concurrently executed collection tasks. It supports several collection methods, allows direct connection to the database, obtains metadata through the API interface for data sources and applications that support the API interface, and parses the generated log files to extract metadata. It supports different types of data sources, collects data dictionaries and data records of relational databases, and collects metadata of big data platforms, ETL tools, modeling tools, reporting tools, scripts, and stored procedures.

[0018] Template collection adopts Excel file format, and users can maintain and import metadata through predefined Excel templates; the metadata of each "class" in the Excel template corresponds to a Sheet worksheet, the combination relationship between metadata is reflected through the path, and the dependency relationship is stored in a special Sheet; users can customize the correspondence between the Sheet name and column in Excel and the metamodel as needed; add data validity verification in Excel to ensure the correctness of data format and type; provide template management function, support template creation, modification, deletion, import and export operations, technicians can automatically import metadata through the template function, and can also manually fill in the data in the template and import it into the system.

[0019] According to step S400, metadata from different sources are organized according to the common warehouse model to support unified management of various metadata, and metadata of database tables, columns, and indexes are mapped to corresponding classes and attributes in the common warehouse model, metadata of business indicators, calculation formulas, and code segments are mapped to the business layer of the common warehouse model, metadata of ETL processes, data conversion rules, and data flows are mapped to the process layer of the common warehouse model, and metadata of log files, scripts, and reports are mapped to corresponding positions in the common warehouse model;

[0020] The organized metadata is stored in a relational database, and the table structure of the relational database is designed according to the public storage model for storage.

[0021] According to step S400, when using the TF-IDF algorithm for fuzzy query, the query statement and metadata description text input by the user are preprocessed to ensure that the query statement and the metadata description text are matched in the same corpus space; the TF-IDF vector of the user query text is calculated to obtain a query vector, the TF-IDF vectors of all metadata record description texts are calculated to obtain a document vector, the similarity between the query vector and each document vector is calculated using cosine similarity, the metadata records are sorted according to the similarity scores, and several metadata records with the highest similarity are returned as fuzzy query results;

[0022] When using the TF-IDF algorithm for classification query, the description text of all metadata records is preprocessed, the feature vector of the metadata record is calculated using the TF-IDF algorithm, the decision tree algorithm is selected as the classification algorithm, and the classification model is trained using the labeled training data set. The model input is the TF-IDF vector and the output is the metadata category. The description text of the newly collected metadata record is converted into a TF-IDF vector, and the trained classification model is used to perform classification prediction on the newly collected metadata. The user selects a classification label to query, and the metadata records belonging to the selected category in the classification model prediction results are returned;

[0023] When using the TF-IDF algorithm for advanced queries, the user inputs a query statement containing several conditions, the query conditions are parsed and preprocessed, and the TF-IDF vectors of each condition are generated; a priority is set for each query condition and they are sorted according to their importance; the similarity with the metadata record is calculated for each condition, and the similarity scores of several conditions are weighted to obtain the comprehensive similarity of each metadata record; the metadata records are sorted according to the comprehensive similarity, and the metadata records that meet the advanced query conditions are returned, providing matching information.

[0024] According to step S400, data flow information is extracted from metadata to determine flow relationships between data entities, where the data entities include database tables and fields, ETL job nodes, business logic code segments, log files, scripts, reporting tools, unstructured data, and data dictionaries; a graph structure is constructed, where each data entity is a node and the relationship where data flows from one entity to another entity is an edge; an adjacency matrix is ​​used to represent the graph data structure to store node and edge information;

[0025] Preprocess the constructed graph structure, remove isolated nodes, ensure the connectivity of the graph, extract features from the nodes and edges of the graph, and generate initial feature vectors; select Graph Convolutional Network as the graph neural network model, initialize the model parameters, and build the model based on the graph structure and feature vectors; use part of the known data flow information as the training set and the rest as the test set, and optimize the model parameters through cross entropy loss to ensure that the model can capture the key nodes and dependencies in the data flow;

[0026] Using the trained graph neural network model, the key nodes in the graph are identified through node feature propagation and aggregation. The key nodes represent the core links in the data processing flow. Through the graph neural network model, the dependencies between nodes are analyzed, and a data dependency graph is constructed to display the dependencies between data entities. Starting from the target node, all predecessor nodes are recursively accessed, the access paths are recorded, and a data source path graph is generated to display the flow of data from the source to the target. Starting from the source node, all successor nodes are recursively accessed, the access paths are recorded, and a data flow path graph is generated to display all nodes and processing flows that the data passes through from the source. The graph visualization tool Graphviz is used to display the data flow graph, showing the key nodes, data dependencies, and data flow paths.

[0027] According to step S400, metadata is collected from different environments of development, testing, and operation, and features are extracted from the collected metadata to form feature vectors; metadata feature vectors in different environments are combined into a training data set, and the model is trained using an isolation forest algorithm, which isolates data points by constructing a number of random trees to identify potential anomalies;

[0028] The trained isolation forest model is used to detect new metadata features and identify outliers, which represent metadata that is different in different environments. The detected outliers are further analyzed to determine the specific content of the differences. An environmental inconsistency report is generated based on the results of the difference analysis. The report contains the following information: the identification of the difference metadata, the difference type, and a detailed description of the difference.

[0029] According to step S500, a detailed information display page for the version is provided, and the user can view the specific content and changes of each version; the user selects specific metadata to release the version and creates a new version. When releasing a new version, the user can add a version description and release notes, and the system records the version release information; the version rollback function is supported, and the user can restore the metadata to a specified historical version. When rolling back, the system prompts the user to confirm the operation and records the detailed information of the rollback operation; the version comparison function is provided, and the user can select two versions for comparison to view the differences between the versions;

[0030] The system automatically records metadata change information. Each change record includes the operation type, operation time, operator and change content. Users can query metadata change information within a specified time period on the platform, and support filtering by operation type, time period and operator; users can subscribe to change information of specific metadata, and users can set subscription conditions. Subscription conditions include subscribed metadata objects, change types and notification frequency; the system monitors metadata changes in real time according to the user's subscription conditions. When the subscribed metadata changes, the system automatically generates change details and sends them to the user via email. Users can view the change details on the platform. The change details include a list of change records and specific content.

[0031] A metadata management system for heterogeneous data sources, comprising:

[0032] Data source management module: including: version and connection parameter configuration unit and data source configuration information management unit; wherein, the version and connection parameter configuration unit is responsible for allowing users to select adapters and configure the version and connection parameters of the collected tools or products; the data source configuration information management unit is responsible for managing and maintaining data source configuration information, and adding, modifying, deleting and authorizing data source configuration information;

[0033] Collection task management module: including: workflow engine configuration unit and task management unit; the workflow engine configuration unit is responsible for using the workflow engine to manage metadata collection tasks, configure automated workflows, and implement scheduled tasks or event-driven collection strategies; the task management unit supports adding, modifying, and deleting collection tasks, and configures the time, status, and scheduling interval for automatic execution of tasks;

[0034] Concurrent collection module: including concurrent collection unit and template collection unit; the concurrent collection unit is responsible for using several collection programs to concurrently execute collection tasks, supporting several collection methods and different types of data sources; the template collection unit uses Excel file format for template collection, and maintains and imports metadata;

[0035] Data storage and analysis module: including: metadata storage unit, query unit, data flow modeling unit and environment consistency inspection unit; the metadata storage unit is responsible for organizing the collected metadata into a structure that conforms to the public storage model and storing it in a relational database; the query unit is responsible for using the TF-IDF algorithm for fuzzy query, classification query and advanced query; the data flow modeling unit is responsible for building a data flow graph, using graph neural network to model and analyze the data flow, identifying key nodes and data dependencies in the data processing process, tracing back the data source, and tracing back the data flow; the environment consistency inspection unit is responsible for using the isolation forest algorithm to perform environment consistency inspections, compare metadata of different environments, and generate environment inconsistency reports;

[0036] Metadata lifecycle management module: includes: version control unit and change query and subscription unit; among them, the version control unit supports metadata lifecycle management, provides version viewing and version publishing functions, and the change query and subscription unit provides change query and subscription functions. Users can query and subscribe to metadata change information. The platform sends change details to the user's mailbox, and the user views the change details on the platform.

[0037] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0038] The present invention supports various types of data sources and collection methods, including direct database connection, interface, log files, etc., and can adapt to complex and changeable data environments. By using multiple collection adapters, it covers common databases, ETL tools, modeling tools, reporting tools, etc.

[0039] The present invention utilizes a workflow engine to manage metadata collection tasks, supports automated workflow configuration, implements scheduled tasks or event-driven collection strategies, and adopts a concurrent execution method, which significantly improves metadata collection efficiency and meets real-time collection requirements in large-scale data environments.

[0040] The present invention uses the isolation forest algorithm to perform environmental consistency inspection on metadata of different environments, timely discovers and reports metadata inconsistency problems between environments, and ensures system stability and data quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0042] Figure 1 It is a schematic diagram of the steps of a metadata management method for heterogeneous data sources of the present invention;

[0043] Figure 2 It is a system structure diagram of a metadata management system for heterogeneous data sources according to the present invention. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] See also Figure 1 and Figure 2, the present invention provides a technical solution:

[0046] According to one embodiment of the present invention, Figure 1 A step diagram of a metadata management method for heterogeneous data sources is shown in the figure. A metadata management method for heterogeneous data sources includes the following steps:

[0047] S100, the user selects an adapter, configures the version and connection parameters of the collected tool or product, manages and maintains data source configuration information, and adds, modifies, deletes and authorizes data source configuration information;

[0048] S200. Use the workflow engine to manage metadata collection tasks, configure automated workflows, and implement scheduled tasks or event-driven collection strategies; support adding, modifying, and deleting collection tasks, and configure the time, status, and scheduling interval for automatic execution of tasks;

[0049] S300, using several collection programs to concurrently execute collection tasks, supporting several collection methods and different types of data sources, and using Excel file format for template collection to maintain and import metadata;

[0050] S400, organize the collected metadata into a structure that conforms to the public storage model and store it in a relational database; use the TF-IDF algorithm to perform fuzzy query, classification query and advanced query; build a data flow graph, use graph neural network to model and analyze the data flow, identify key nodes and data dependencies in the data processing flow, trace the data source forward, and trace the data flow backward; use the isolation forest algorithm to perform environmental consistency inspection, compare the metadata of different environments, and generate an environmental inconsistency report;

[0051] S500, supports metadata lifecycle management, provides version viewing and version publishing functions; provides change query and subscription functions, users can query and subscribe to metadata change information, the platform sends change details to the user's mailbox, and the user views the change details on the platform.

[0052] According to step S100, according to the source of metadata, the user selects a collection adapter, and the adapter types include: DB data dictionary collection adapter, template collection adapter, data record collection adapter, ETL tool collection adapter, big data collection adapter, modeling tool collection adapter, report tool collection adapter, log collection adapter and script collection adapter; wherein, the DB data dictionary collection adapter is used to collect the data dictionary of the database, the template collection adapter is used to collect data through Excel templates, the data record collection adapter is used to collect database data records, the ETL tool collection adapter supports metadata collection of ETL tools, the big data collection adapter is used to collect metadata of Hive, Hbase and HDFS, the modeling tool collection adapter supports metadata collection of Powerdesigner and ERWin, the report tool collection adapter supports metadata collection of Cognos and BI.Office, the log collection adapter supports metadata collection of Shell and Perl logs, and the script collection adapter supports metadata collection of DB data dictionary, DDL, SparkSql, Sqoop collection and stored procedures;

[0053] The DB data dictionary collection adapter supports the following common databases: Oracle, DB2, MSSQL Server, MySQL, Teradata, HyperSQL, Sybase ASE, SybaseIQ, Informix, DM, Oscar, KingBase, Netezza, PostgreSQL, HighGo, GBase; the data record collection adapter supports the collection of DB data records, such as Oracle, DB2, MSSQL Server, MySQL, Teradata, HyperSQL, Sybase ASE, Informix, Netezza; the ETL tool collection adapter supports ETL tools such as Datastage, PowerCenter, Kettle, DI, ODI, etc.

[0054] In this embodiment, metadata collection uses MetaCube DGS metadata management, MetaCube Extractor metadata collector, MetaCube Api metadata interface service engine, and MetaCube Warehouse metadata warehouse.

[0055] MetaCube DGS metadata management is a WEB-based metadata management function that can integrate data assets that are independent of each link of enterprise data flow, and display various metadata in a tree-like and horizontal manner. The hierarchical structure is clear, which is convenient for users to intuitively understand the basic metadata information and parent and child metadata information. It provides a graphical analysis function of metadata relationships, which helps users understand the relationship between information and processing and how they are used.

[0056] MetaCube Extractor metadata collector refers to a set of programs that use corresponding collection methods to obtain metadata from different metadata sources. MetaCube Extractor supports the collection of multiple metadata sources, including: relational databases, Hadoop, modeling tools, data integration tools, BI tools, and customer personalized metadata. MetaCubeExtractor stores the collected metadata in a unified way in the metadata warehouse to provide data for upper-level metadata applications.

[0057] The MetaCube Api metadata interface service engine uses the currently popular REST interface architecture and combines Spring's unified management capabilities to provide a unified HTTP+JSON format interface to the outside world.

[0058] MetaCube Warehouse is a metadata storage architecture designed to store different types of metadata.

[0059] The user selects and configures the specific version information of the tool or product used so that the adapter can correctly identify and collect metadata of the corresponding version; based on the selected adapter and the collected tool, the user needs to provide necessary connection parameters, which include IP address, port, user name, password and database name for the database, server address, port, user name and password for the ETL tool, and corresponding connection parameters for other tools and interfaces according to the requirements of the specific tool.

[0060] According to step S200, the metadata collection task is managed using the workflow engine Apache Airflow. The workflow engine provides task scheduling and management functions, and can organize and control the execution process of the collection task. According to the needs of metadata collection, an automated workflow is designed. The workflow consists of several task nodes, each node represents a specific collection task, and the dependencies between tasks are defined in the workflow, and the execution strategy of the workflow is configured. The execution time and frequency of the scheduled task are set, and the setting of the frequency according to every minute, every hour, every day, every week, every month and one-time is supported.

[0061] Distributed deployment of multiple collectors solves the problem of cross-cloud metadata collection. For the same metadata collection source, it supports active and standby collectors to achieve highly reliable metadata collection. When adding a new collector, it will communicate with the collection end to monitor whether the heartbeat of the collection end is normal. Only when it is normal can it be added. After the addition is completed, it will maintain a heartbeat status check with the collection end and determine whether it is offline based on the heartbeat status.

[0062] According to step S300, the collection tasks are assigned to different collection programs to ensure balanced distribution of tasks. The workflow engine Apache Airflow is responsible for scheduling and monitoring concurrently executed collection tasks. It supports several collection methods, allows direct connection to the database, obtains metadata through the API interface for data sources and applications that support the API interface, and parses the generated log files to extract metadata. It supports different types of data sources, collects data dictionaries and data records of relational databases, and collects metadata of big data platforms, ETL tools, modeling tools, reporting tools, scripts, and stored procedures.

[0063] Template collection adopts Excel file format, and users can maintain and import metadata through predefined Excel templates; the metadata of each "class" in the Excel template corresponds to a Sheet worksheet, the combination relationship between metadata is reflected through the path, and the dependency relationship is stored in a special Sheet; users can customize the correspondence between the Sheet name and column in Excel and the metamodel as needed; add data validity verification in Excel to ensure the correctness of data format and type; provide template management function, support template creation, modification, deletion, import and export operations, technicians can automatically import metadata through the template function, and can also manually fill in the data in the template and import it into the system.

[0064] According to step S400, metadata from different sources are organized according to the common warehouse model to support unified management of various metadata, and metadata of database tables, columns, and indexes are mapped to corresponding classes and attributes in the common warehouse model, metadata of business indicators, calculation formulas, and code segments are mapped to the business layer of the common warehouse model, metadata of ETL processes, data conversion rules, and data flows are mapped to the process layer of the common warehouse model, and metadata of log files, scripts, and reports are mapped to corresponding positions in the common warehouse model;

[0065] The organized metadata is stored in a relational database, and the table structure of the relational database is designed according to the public storage model for storage.

[0066] According to step S400, when using the TF-IDF algorithm for fuzzy query, the query statement and metadata description text input by the user are preprocessed to ensure that the query statement and the metadata description text are matched in the same corpus space; the TF-IDF vector of the user query text is calculated to obtain a query vector, the TF-IDF vectors of all metadata record description texts are calculated to obtain a document vector, the similarity between the query vector and each document vector is calculated using cosine similarity, the metadata records are sorted according to the similarity scores, and several metadata records with the highest similarity are returned as fuzzy query results;

[0067] When using the TF-IDF algorithm for classification query, the description text of all metadata records is preprocessed, the feature vector of the metadata record is calculated using the TF-IDF algorithm, the decision tree algorithm is selected as the classification algorithm, and the classification model is trained using the labeled training data set. The model input is the TF-IDF vector and the output is the metadata category. The description text of the newly collected metadata record is converted into a TF-IDF vector, and the trained classification model is used to perform classification prediction on the newly collected metadata. The user selects a classification label to query, and the metadata records belonging to the selected category in the classification model prediction results are returned;

[0068] When using the TF-IDF algorithm for advanced queries, the user inputs a query statement containing several conditions, the query conditions are parsed and preprocessed, and the TF-IDF vectors of each condition are generated; a priority is set for each query condition and they are sorted according to their importance; the similarity with the metadata record is calculated for each condition, and the similarity scores of several conditions are weighted to obtain the comprehensive similarity of each metadata record; the metadata records are sorted according to the comprehensive similarity, and the metadata records that meet the advanced query conditions are returned, providing matching information.

[0069] According to step S400, data flow information is extracted from metadata to determine flow relationships between data entities, where the data entities include database tables and fields, ETL job nodes, business logic code segments, log files, scripts, reporting tools, unstructured data, and data dictionaries; a graph structure is constructed, where each data entity is a node and the relationship where data flows from one entity to another entity is an edge; an adjacency matrix is ​​used to represent the graph data structure to store node and edge information;

[0070] Preprocess the constructed graph structure, remove isolated nodes, ensure the connectivity of the graph, extract features from the nodes and edges of the graph, and generate initial feature vectors; select Graph Convolutional Network as the graph neural network model, initialize the model parameters, and build the model based on the graph structure and feature vectors; use part of the known data flow information as the training set and the rest as the test set, and optimize the model parameters through cross entropy loss to ensure that the model can capture the key nodes and dependencies in the data flow;

[0071] Using the trained graph neural network model, the key nodes in the graph are identified through node feature propagation and aggregation. The key nodes represent the core links in the data processing flow. Through the graph neural network model, the dependencies between nodes are analyzed, and a data dependency graph is constructed to display the dependencies between data entities. Starting from the target node, all predecessor nodes are recursively accessed, the access paths are recorded, and a data source path graph is generated to display the flow of data from the source to the target. Starting from the source node, all successor nodes are recursively accessed, the access paths are recorded, and a data flow path graph is generated to display all nodes and processing flows that the data passes through from the source. The graph visualization tool Graphviz is used to display the data flow graph, showing the key nodes, data dependencies, and data flow paths.

[0072] According to step S400, metadata is collected from different environments of development, testing, and operation, and features are extracted from the collected metadata to form feature vectors; metadata feature vectors in different environments are combined into a training data set, and the model is trained using an isolation forest algorithm, which isolates data points by constructing a number of random trees to identify potential anomalies;

[0073] The trained isolation forest model is used to detect new metadata features and identify outliers, which represent metadata that is different in different environments. The detected outliers are further analyzed to determine the specific content of the differences. An environmental inconsistency report is generated based on the results of the difference analysis. The report contains the following information: the identification of the difference metadata, the difference type, and a detailed description of the difference.

[0074] According to step S500, a detailed information display page for the version is provided, and the user can view the specific content and changes of each version; the user selects specific metadata to release the version and creates a new version. When releasing a new version, the user can add a version description and release notes, and the system records the version release information; the version rollback function is supported, and the user can restore the metadata to a specified historical version. When rolling back, the system prompts the user to confirm the operation and records the detailed information of the rollback operation; the version comparison function is provided, and the user can select two versions for comparison to view the differences between the versions;

[0075] The system automatically records metadata change information. Each change record includes the operation type, operation time, operator and change content. Users can query metadata change information within a specified time period on the platform, and support filtering by operation type, time period and operator; users can subscribe to change information of specific metadata, and users can set subscription conditions. Subscription conditions include subscribed metadata objects, change types and notification frequency; the system monitors metadata changes in real time according to the user's subscription conditions. When the subscribed metadata changes, the system automatically generates change details and sends them to the user via email. Users can view the change details on the platform. The change details include a list of change records and specific content.

[0076] According to another embodiment of the present invention, Figure 2 As shown in the system structure diagram of a metadata management system for heterogeneous data sources, a metadata management system for heterogeneous data sources includes:

[0077] Data source management module: including: version and connection parameter configuration unit and data source configuration information management unit; wherein, the version and connection parameter configuration unit is responsible for allowing users to select adapters and configure the version and connection parameters of the collected tools or products; the data source configuration information management unit is responsible for managing and maintaining data source configuration information, and adding, modifying, deleting and authorizing data source configuration information;

[0078] Collection task management module: including: workflow engine configuration unit and task management unit; the workflow engine configuration unit is responsible for using the workflow engine to manage metadata collection tasks, configure automated workflows, and implement scheduled tasks or event-driven collection strategies; the task management unit supports adding, modifying, and deleting collection tasks, and configures the time, status, and scheduling interval for automatic execution of tasks;

[0079] Concurrent collection module: including concurrent collection unit and template collection unit; the concurrent collection unit is responsible for using several collection programs to concurrently execute collection tasks, supporting several collection methods and different types of data sources; the template collection unit uses Excel file format for template collection, and maintains and imports metadata;

[0080] Data storage and analysis module: including: metadata storage unit, query unit, data flow modeling unit and environment consistency inspection unit; the metadata storage unit is responsible for organizing the collected metadata into a structure that conforms to the public storage model and storing it in a relational database; the query unit is responsible for using the TF-IDF algorithm for fuzzy query, classification query and advanced query; the data flow modeling unit is responsible for building a data flow graph, using graph neural network to model and analyze the data flow, identifying key nodes and data dependencies in the data processing process, tracing back the data source, and tracing back the data flow; the environment consistency inspection unit is responsible for using the isolation forest algorithm to perform environment consistency inspections, compare metadata of different environments, and generate environment inconsistency reports;

[0081] Metadata lifecycle management module: includes: version control unit and change query and subscription unit; among them, the version control unit supports metadata lifecycle management, provides version viewing and version publishing functions, and the change query and subscription unit provides change query and subscription functions. Users can query and subscribe to metadata change information. The platform sends change details to the user's mailbox, and the user views the change details on the platform.

[0082] In this embodiment, the data source management module can be used to perform the following operations: query based on the data source name, query the currently created data source information; add a collection data source and fill in the connection information of the data source; modify the detailed information of the data source; delete a data source; assign roles to data sources, and data source authorization controls the permissions of roles to achieve collection control of relevant personnel; you can select an adapter to view the detailed information of the collection adapter and view the adapted information: adapter name, tool / product name and version, adapter working mode, and detailed parameter list.

[0083] The following operations can be performed using the collection task management module: query currently created tasks, add a collection task, modify a collection task, and delete a collection task; the collection task controls scheduling by setting its expression, and the collection trigger cycle and time can be set in a variety of ways. Select a task in the table that needs to be set or adjusted in time; after configuring the task, if the collection is not scheduled within a certain time period, it can be configured to suspend or run; immediately trigger a collection task to execute; add, delete, modify, and query collectors.

[0084] The concurrent collection module allows you to perform the following operations: manually collect metadata by selecting the local server file source, configuring the data source, storage strategy, and file source. The storage strategy supports the following three methods: Conservative full volume: overwrite the original collected data while retaining the dependency relationship between the data and the data outside the hanging node; Aggressive full volume: overwrite the original collected data and completely follow the latest collected data; Incremental: accumulate the newly added data on the basis of the original data;

[0085] Query the configuration name and display the list information; solve the problem of collecting metadata through the DB data record collection adapter, and map the records in the DB record with the metamodel through the DB data collection configuration; add and modify the configuration of the data record collection template, and delete the data record collection template; add, modify and delete the mapping between the Excel collection template and the metamodel, and download the Excel collection template; when performing metadata collection again, in order to locate the source table and the target table, it is necessary to configure the data source mapping, and map and match the data source connection name of the collection tool with the hanging point of the data dictionary of the metadata system.

[0086] The following operations can be performed using the data storage and analysis module: query which data currently needs to be reviewed. By default, the metadata to be reviewed for all collected data sources is queried. You can also select a specified data source to filter the query; when the collected data source is set to require storage review, it must first pass the review after the collection is completed. To view the specific metadata to be reviewed, you can double-click the batch to be reviewed, or check the "Details" button to enter the metadata review page; after the review is passed, it will be entered into the metadata database. If the review fails, the metadata will not be entered; for the collected data to be reviewed, all the changed metadata information, attribute information and dependencies are passed and entered into the metadata database; for the collected data to be reviewed, all the changed metadata information, attribute information and dependencies are rejected and will not be entered into the metadata database; confirm whether the "all passed" and "all rejected" operations are correct, confirm the review results, and save them to take effect; collection log; query the log information of automatic collection and manual collection; to view specific log information, you can click the "View Log" link of the collection, or double-click the line record to enter the log details interface; collection task execution log deletion; the log can be deleted according to the date of task execution and the status of the task; cancel the execution of the collection task; export the currently viewed collection log;

[0087] Manage metadata, add metamodel root node, which includes package name and code; query metamodel; add, delete and modify metamodel, which includes classes, class combination relationships and dependencies between classes; switch the metamodel code and name displayed on the metamodel tree, and refresh the metamodel tree.

[0088] The metadata lifecycle management module allows you to perform the following operations: Model change is to manage the model in the metadata platform. When the model is changed, you can query the change analysis and change details of the changed model; it provides a change query function, which can query the change information of metadata within a certain period of time, such as the total number of new additions, total number of modifications, total number of deletions, etc.; you can subscribe to the metadata you are interested in, and when these metadata are changed, the platform can send the subscribed change details to the mailbox, and you can also view the details of the changed metadata on the platform;

[0089] Tag management expands the standard collection management approach for business. By maintaining business metadata information such as business metadata classification and business terms in tag management, the association between business metadata and technical metadata is established to achieve unified management of business metadata and technical metadata. Version maintenance supports the lifecycle management of platform metadata. All of this is to ensure the quality of metadata and guarantee the authority and reliability of the subsequent use of the metadata platform. Combined with the version viewing function, all-round management and application of versions are achieved. Through version release, metadata of a certain type can be finalized, and the granularity of the finalization can be selected by yourself. The system provides metadata export capabilities, and all metadata that needs to be exported on the metadata tree can be exported as Excel files.

[0090] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0091] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A metadata management method for heterogeneous data sources, characterized in that: The method comprises the following steps: S100, the user selects an adapter, configures the version and connection parameters of the collected tool or product, manages and maintains data source configuration information, and adds, modifies, deletes and authorizes data source configuration information; S200. Use the workflow engine to manage metadata collection tasks, configure automated workflows, and implement scheduled tasks or event-driven collection strategies; support adding, modifying, and deleting collection tasks, and configure the time, status, and scheduling interval for automatic execution of tasks; S300, using several collection programs to concurrently execute collection tasks, supporting several collection methods and different types of data sources, and using Excel file format for template collection to maintain and import metadata; According to step S300, the collection tasks are assigned to different collection programs to ensure balanced distribution of tasks. The workflow engine Apache Airflow is responsible for scheduling and monitoring concurrently executed collection tasks. It supports several collection methods, allows direct connection to the database, obtains metadata through the API interface for data sources and applications that support the API interface, and parses the generated log files to extract metadata. It supports different types of data sources, collects data dictionaries and data records of relational databases, and collects metadata of big data platforms, ETL tools, modeling tools, reporting tools, scripts, and stored procedures. Template collection adopts Excel file format. Users can maintain and import metadata through predefined Excel templates. The metadata of each "class" in the Excel template corresponds to a Sheet worksheet. The combination relationship between metadata is reflected through the path, and the dependency relationship is stored in a special Sheet. Users can customize the correspondence between the Sheet name and column in Excel and the meta-model as needed. Data validity verification is added in Excel to ensure the correctness of data format and type. Template management function is provided to support template creation, modification, deletion, import and export operations. Technical personnel can automatically import metadata through the template function, and can also manually fill in the data in the template and import it into the system. S400, organize the collected metadata into a structure that conforms to the public storage model and store it in a relational database; use the TF-IDF algorithm to perform fuzzy query, classification query and advanced query; build a data flow graph, use graph neural network to model and analyze the data flow, identify key nodes and data dependencies in the data processing flow, trace the data source forward, and trace the data flow backward; use the isolation forest algorithm to perform environmental consistency inspection, compare the metadata of different environments, and generate an environmental inconsistency report; S500, supports metadata lifecycle management, provides version viewing and version publishing functions; provides change query and subscription functions, users can query and subscribe to metadata change information, the platform sends change details to the user's mailbox, and the user views the change details on the platform.

2. The metadata management method for heterogeneous data sources according to claim 1, characterized in that: According to step S100, according to the source of metadata, the user selects a collection adapter, and the adapter types include: DB data dictionary collection adapter, template collection adapter, data record collection adapter, ETL tool collection adapter, big data collection adapter, modeling tool collection adapter, report tool collection adapter, log collection adapter and script collection adapter; wherein, the DB data dictionary collection adapter is used to collect the data dictionary of the database, the template collection adapter is used to collect data through Excel templates, the data record collection adapter is used to collect database data records, the ETL tool collection adapter supports metadata collection of ETL tools, the big data collection adapter is used to collect metadata of Hive, Hbase and HDFS, the modeling tool collection adapter supports metadata collection of Powerdesigner and ERWin, the report tool collection adapter supports metadata collection of Cognos and BI.Office, the log collection adapter supports metadata collection of Shell and Perl logs, and the script collection adapter supports metadata collection of DB data dictionary, DDL, SparkSql, Sqoop collection and stored procedures; The user selects and configures the specific version information of the tool or product used so that the adapter can correctly identify and collect metadata of the corresponding version; based on the selected adapter and the collected tool, the user needs to provide necessary connection parameters, which include IP address, port, user name, password and database name for the database, server address, port, user name and password for the ETL tool, and corresponding connection parameters for other tools and interfaces according to the requirements of the specific tool.

3. The metadata management method for heterogeneous data sources according to claim 1, characterized in that: According to step S200, the metadata collection task is managed using the workflow engine Apache Airflow, which provides task scheduling and management functions and can organize and control the execution process of the collection task; Design an automated workflow based on the needs of metadata collection. The workflow consists of several task nodes, each of which represents a specific collection task. Define the dependencies between tasks in the workflow, and configure the workflow execution strategy. Set the execution time and frequency of scheduled tasks, and support settings based on minute, hour, day, week, month, and one-time frequencies.

4. The metadata management method for heterogeneous data sources according to claim 1, characterized in that: According to step S400, metadata from different sources are organized according to the common warehouse model to support unified management of various metadata, and metadata of database tables, columns, and indexes are mapped to corresponding classes and attributes in the common warehouse model, metadata of business indicators, calculation formulas, and code segments are mapped to the business layer of the common warehouse model, metadata of ETL processes, data conversion rules, and data flows are mapped to the process layer of the common warehouse model, and metadata of log files, scripts, and reports are mapped to corresponding positions in the common warehouse model; The organized metadata is stored in a relational database, and the table structure of the relational database is designed according to the public storage model for storage.

5. The metadata management method for heterogeneous data sources according to claim 1, characterized in that: According to step S400, when using the TF-IDF algorithm for fuzzy query, the query statement and metadata description text input by the user are preprocessed to ensure that the query statement and the metadata description text are matched in the same corpus space; Calculate the TF-IDF vector of the user query text to get the query vector, calculate the TF-IDF vector of all metadata record description texts to get the document vector, use cosine similarity to calculate the similarity between the query vector and each document vector, sort the metadata records according to the similarity scores, and return the metadata records with the highest similarity as the fuzzy query results; When using the TF-IDF algorithm for classification query, preprocess the description text of all metadata records, use the TF-IDF algorithm to calculate the feature vector of the metadata record, select the decision tree algorithm as the classification algorithm, use the labeled training data set to train the classification model, the model input is the TF-IDF vector, and the output is the metadata category; convert the description text of the newly collected metadata record into a TF-IDF vector, and use the trained classification model to perform classification prediction on the newly collected metadata; Users select a classification label to query, and the metadata records belonging to the selected category in the classification model prediction results are returned; When using the TF-IDF algorithm for advanced queries, the user enters a query statement containing several conditions, the query conditions are parsed and preprocessed, and the TF-IDF vectors of each condition are generated; a priority is set for each query condition and sorted according to importance; The similarity with the metadata record is calculated condition by condition, and the similarity scores of several conditions are weighted to obtain the comprehensive similarity of each metadata record; the metadata records are sorted according to the comprehensive similarity, and the metadata records that meet the advanced query conditions are returned to provide matching information.

6. The metadata management method for heterogeneous data sources according to claim 1, characterized in that: According to step S400, data flow information is extracted from metadata to determine flow relationships between data entities, where the data entities include database tables and fields, ETL job nodes, business logic code segments, log files, scripts, reporting tools, unstructured data, and data dictionaries; a graph structure is constructed, where each data entity is a node and the relationship between data flowing from one entity to another is an edge; Use adjacency matrix to represent graph data structure to store node and edge information; Preprocess the constructed graph structure, remove isolated nodes, ensure the connectivity of the graph, extract features from the nodes and edges of the graph, and generate initial feature vectors; select Graph Convolutional Network as the graph neural network model, initialize the model parameters, and build the model based on the graph structure and feature vectors; use part of the known data flow information as the training set and the rest as the test set, and optimize the model parameters through cross entropy loss to ensure that the model can capture the key nodes and dependencies in the data flow; Using the trained graph neural network model, the key nodes in the graph are identified through node feature propagation and aggregation. The key nodes represent the core links in the data processing flow. Through the graph neural network model, the dependency relationship between nodes is analyzed, and a data dependency graph is constructed to display the dependency relationship between data entities. Starting from the target node, recursively visit all predecessor nodes, record the access path, generate a data source path diagram, and show the flow of data from the source to the target; starting from the source node, recursively visit all successor nodes, record the access path, generate a data flow path diagram, and show all the nodes and processing flows that the data passes through from the source; use the graph visualization tool Graphviz to display the data flow diagram, showing key nodes, data dependencies, and data flow paths.

7. The metadata management method for heterogeneous data sources according to claim 1, characterized in that: According to step S400, metadata is collected from different environments of development, testing, and operation, and features are extracted from the collected metadata to form feature vectors; metadata feature vectors in different environments are combined into a training data set, and the model is trained using an isolation forest algorithm, which isolates data points by constructing a number of random trees to identify potential anomalies; The trained isolation forest model is used to detect new metadata features and identify outliers, which represent metadata that is different in different environments. The detected outliers are further analyzed to determine the specific content of the differences. An environmental inconsistency report is generated based on the results of the difference analysis. The report contains the following information: the identification of the difference metadata, the difference type, and a detailed description of the difference.

8. The metadata management method for heterogeneous data sources according to claim 1, characterized in that: According to step S500, a detailed information display page of the version is provided, and the user can view the specific content and changes of each version; the user selects specific metadata to release the version and creates a new version. When releasing a new version, the user can add a version description and release notes, and the system records the version release information; Supports version rollback function, users can restore metadata to a specified historical version. When rolling back, the system prompts the user to confirm the operation and records the detailed information of the rollback operation; Provides version comparison function, users can select two versions for comparison and view the differences between them; The system automatically records metadata change information. Each change record includes the operation type, operation time, operator and change content. Users can query metadata change information within a specified time period on the platform, and support filtering by operation type, time period and operator; Users can subscribe to change information of specific metadata. Users can set subscription conditions, which include the subscribed metadata object, change type and notification frequency. The system monitors metadata changes in real time based on the user's subscription conditions. When the subscribed metadata changes, the system automatically generates change details and sends them to the user via email. Users can view the change details on the platform, including the list and specific content of the change records.

9. A metadata management system for heterogeneous data sources, using a metadata management method for heterogeneous data sources according to any one of claims 1 to 8, characterized in that: include: Data source management module: including: version and connection parameter configuration unit and data source configuration information management unit; wherein, the version and connection parameter configuration unit is responsible for allowing users to select adapters and configure the version and connection parameters of the collected tools or products; the data source configuration information management unit is responsible for managing and maintaining data source configuration information, and adding, modifying, deleting and authorizing data source configuration information; Collection task management module: including: workflow engine configuration unit and task management unit; the workflow engine configuration unit is responsible for using the workflow engine to manage metadata collection tasks, configure automated workflows, and implement scheduled tasks or event-driven collection strategies; the task management unit supports adding, modifying, and deleting collection tasks, and configures the time, status, and scheduling interval for automatic execution of tasks; Concurrent collection module: including concurrent collection unit and template collection unit; the concurrent collection unit is responsible for using several collection programs to concurrently execute collection tasks, supporting several collection methods and different types of data sources; the template collection unit uses Excel file format for template collection, and maintains and imports metadata; Data storage and analysis module: including: metadata storage unit, query unit, data flow modeling unit and environment consistency inspection unit; the metadata storage unit is responsible for organizing the collected metadata into a structure that conforms to the public storage model and storing it in a relational database; the query unit is responsible for using the TF-IDF algorithm for fuzzy query, classification query and advanced query; the data flow modeling unit is responsible for building a data flow graph, using graph neural network to model and analyze the data flow, identifying key nodes and data dependencies in the data processing process, tracing back the data source, and tracing back the data flow; the environment consistency inspection unit is responsible for using the isolation forest algorithm to perform environment consistency inspections, compare metadata of different environments, and generate environment inconsistency reports; Metadata lifecycle management module: including: version control unit and change query and subscription unit; Among them, the version control unit supports metadata lifecycle management and provides version viewing and version publishing functions. The change query and subscription unit provides change query and subscription functions. Users can query and subscribe to metadata change information. The platform sends change details to the user's mailbox, and the user views the change details on the platform.

Images