Method for metadata collection and data bloodline analysis in lake-warehouse integrated system

Abstract

The application relates to the technical field of data processing, and discloses a metadata collection and data bloodline analysis method in a lake warehouse integrated system. The method realizes comprehensive monitoring and traceability analysis of a data processing process, optimizes a data management and analysis process, covers each link of a data processing chain, ensures that comprehensive data information is captured, establishes a transfer relationship between data based on collected metadata and a data transfer process, realizes data traceability by analyzing the source, conversion process and target source of data, and simultaneously, the data bloodline relationship is graphically displayed to a user by using a visualization technology, the user can intuitively trace data through a graphical interface, and the visibility and ease of use of data management are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field, and particularly relates to a method for metadata collection and data bloodline analysis in a lake-warehouse integrated system. BACKGROUND

[0002] In the current digital era, the importance of data is increasingly prominent, and data analysis and processing have become the core business of many enterprises and organizations. However, in the lake-warehouse integrated system, as the data volume grows and the data processing complexity increases, the origin of data, the transformation process, and the relationship between data become complex and difficult to trace. In order to ensure the credibility, integrity and consistency of data, a method is needed to comprehensively monitor the flow and processing of data, so as to optimize data management and analysis.

[0003] In the field of data management, there are some methods for metadata collection, but there are often limitations. One common method is manual annotation-based metadata collection, that is, adding annotations and labels to data processes by artificial means. This method is not only very tedious, but also prone to errors. Another method is to use automated tools to collect metadata, but these tools often only focus on common data sources and part of the process, and cannot fully track data links, such as data flow between the source layer, standard layer and application layer in the lake-warehouse system. In addition, when the data source changes, these methods cannot actively synchronize the changes to the metadata.

[0004] In the storage of metadata, most systems use relational databases, which brings efficiency problems to the generation and maintenance of data bloodline graphs.

[0005] In terms of data bloodline analysis, some systems try to establish relationships between data by tracking data sources and data destinations in data processes. However, these methods are usually limited to specific business scenarios and can only track the starting point and ending point of data, lacking the ability to trace the intermediate process of data conversion, making it difficult to apply to complex and diversified data processing scenarios. In addition, the common data bloodline analysis method lacks the ability to update the data bloodline graph when the data source node changes.

[0006] In addition, the commonly used method also provides data bloodline visualization capability, but usually lacks the ability to reverse locate the data conversion intermediate process through the data bloodline graph, and does not allow users to manually maintain the data bloodline graph, greatly affecting its application scenarios. SUMMARY

[0007] The purpose of the present application is to solve the problems in the prior art and provide a method for metadata collection and data bloodline analysis in a lake-warehouse integrated system, comprising the following steps:

[0008] S1: The data source management service centrally manages external data sources, lake-warehouse integrated system source layer, and data warehouse standard layer data sources. When a new data source is defined in the data source management service, a message for starting metadata collection is generated and sent to a message middleware, and the message includes detailed information of the data source;

[0009] S2: After the metadata management service listens to the metadata collection message, it starts to collect the metadata information from the external data source at a fixed time. The metadata management service stores the collected metadata information as a data node in a graph database;

[0010] S3: The data lineage management unit in the metadata management service performs data lineage analysis based on the collected metadata node information, and finally persists the nodes and relationship edges of the data lineage relationship graph in the graph database;

[0011] S4: Data lineage visualization is achieved through a data lineage visualization page, which visually displays the mapping relationship and flow of data.

[0012] Further, the metadata collection and data lineage analysis process tracks the mapping relationship and conversion path of data through an ETL system and the lake-warehouse integrated system.

[0013] Preferably, the ETL system is responsible for synchronizing the data from the defined data source after cleaning and processing to the lake-warehouse integrated system and target data source. The ETL system includes a data model relationship analysis unit responsible for analyzing the data flow and conversion relationship between data models in the ETL process of data extraction, conversion, and loading, and sending the data model conversion relationship in JSON message body to the message middleware for use by the metadata management service.

[0014] More preferably, the lake-warehouse integrated system includes the functions of data lake and data warehouse for structured data storage, and a data model relationship analysis unit responsible for analyzing the data conversion relationship of the source layer and the standard layer, and sending the data model conversion relationship in JSON message body to the message middleware for use by the metadata management service.

[0015] Further, in step S1, the centralized management of the data source management service includes adding a new data source, configuring connection information, specifying a data source type, and defining a collection frequency.

[0016] Further, in step S2, the metadata collection is responsible for the metadata collection unit of the metadata management service, which obtains the data source connection information centrally managed in the data source management service, establishes a connection at a fixed time, and receives the collected metadata information.

[0017] Preferably, the metadata information includes the data source connection information, table structure, field information and data type.

[0018] Further, in step S3, the data bloodline management unit captures the mapping relationship and conversion path message of the data model in the ETL system and the lake-warehouse integrated system in real time based on the metadata node information collected by the metadata collection unit, tracks the data flow direction by analyzing the message, identifies the data source and intermediate process, determines the data flow direction, and constructs the data source relationship graph.

[0019] When the parsed data node message is not collected in the metadata collection process, the metadata related node is re-added.

[0020] Further, in step S4, the data bloodline visualization page supports manual addition and manual editing of the data bloodline relationship graph, actively initiates a request for synchronizing the data model relationship to the ETL system and the lake-warehouse integrated system, and realizes manual updating of the data bloodline relationship.

[0021] Preferably, the data bloodline visualization page displays data bloodlines in three levels of nodes, including data sources, data tables and fields.

[0022] More preferably, each data source represents a large node, the large node includes a plurality of data table sub-nodes, each data table sub-node includes a plurality of field sub-nodes, the nodes are connected by lines to represent data conversion rules, metadata information is displayed by clicking the corresponding node, detailed information of the conversion rule is displayed by clicking the line, and a specified job is jumped to by a detail tag.

[0023] Further, the graph database is used to uniformly store the metadata information and the data relationship graph information, and the difficulty of bloodline relationship analysis is reduced.

[0024] Compared with the prior art, the present application has the following advantages:

[0025] (1) Full-link metadata collection, identifying and collecting metadata of data sources related to data processing process. Metadata collection covers every link of the data processing chain. In addition to automatically collecting data source metadata defined in the data source, it also includes ETL data, data warehouse data, standard layer data and other data in the data processing chain, ensuring the completeness of the metadata.

[0026] (2) Metadata storage, the collected metadata is uniformly persisted in a graph database, the metadata management data (source) library, data table, and field information are saved as points of the graph database, and a relationship edge is used to establish a correlation relationship, so as to facilitate subsequent data blood relationship query and maintenance;

[0027] (3) Data blood relationship analysis, based on the collected metadata and the data flow paths between various services, a blood relationship graph between data is established, through analysis of the blood relationship graph, the data blood relationship at the field level can be tracked, the source, intermediate process and final result of the data are determined, and data full-link traceability is realized;

[0028] (4) Support for automatically comparing collected metadata information, when the metadata information changes, the data blood relationship graph is updated synchronously, and at the same time, manual synchronization capability of the data blood relationship is provided, the user can manually apply to the ETL service and the lake-warehouse integrated system to synchronize the latest data blood relationship information, so as to realize the real-time and accuracy of the data blood relationship;

[0029] (5) Support for user manual maintenance of data blood relationship, in addition to automatic addition of data blood relationship of ETL system and data warehouse model, the user can manually add or edit the data blood relationship graph, and at the same time, the ability of providing the data blood relationship graph to the third party system through the message middleware is also provided;

[0030] (6) Data blood relationship visualization, using visualization technology, the data blood relationship graph is visually displayed to the user. The user can not only query the data blood relationship graph through various conditions, but also can locate the intermediate process of data flow through the data blood relationship graph, so as to improve the visibility and ease of use of data management. BRIEF DESCRIPTION OF DRAWINGS

[0031] The accompanying drawings are used to provide a further understanding of the present application, and constitute a part of the specification, together with embodiments of the present application, for explaining the present application, and do not constitute a limitation of the present application. In the drawings:

[0032] Figure 1 A method for metadata collection and data blood relationship analysis in a lake-warehouse integrated system is provided;

[0033] Figure 2 A basic format code diagram of a JSON message body sent to a message middleware is provided;

[0034] Figure 3 A metadata collection workflow diagram is provided;

[0035] Figure 4 A data blood relationship analysis workflow diagram is provided;

[0036] Figure 5 A data bloodline visualization page display diagram. DETAILED DESCRIPTION

[0037] To make the objects, technical solutions and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0038] It can be understood by those skilled in the art that, unless specifically stated otherwise, the singular forms "a", "an" and "the" as used herein are intended to include plural forms. It should be further understood that the use of the term "including" in the specification of the present application means that the features, integers, steps, operations, elements, and / or components described are present, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0039] In the field of data management, there are some methods for metadata collection, but there are often limitations. One common method is manual annotation-based metadata collection, that is, adding annotations and labels to the data process in an artificial manner. This method is not only very tedious, but also prone to errors. Another method is to use automated tools to collect metadata, but these tools often only focus on commonly used data sources and part of the process, and cannot comprehensively track data links, such as data flow between the source layer, the standard layer and the application layer in the lake warehouse system. In addition, when the data source changes, these methods cannot actively synchronize the changes to the metadata.

[0040] In the storage of metadata, most systems use relational databases, which brings efficiency problems to the generation and maintenance of subsequent data bloodline relationship graphs.

[0041] In terms of data bloodline analysis, some systems attempt to establish relationships between data by tracking data sources and data destinations in the data process. However, these methods are usually limited to specific business scenarios and can only track the starting point and ending point of the data, lacking the ability to trace the intermediate process of data conversion, making it difficult to apply to complex and diversified data processing scenarios. In addition, the common data bloodline analysis method lacks the ability to update the data bloodline relationship graph when the data source node changes.

[0042] In addition, while commonly used methods also provide data lineage visualization capabilities, they typically lack the ability to reverse-locate intermediate data transformation processes through data lineage diagrams, and do not allow users to manually maintain data lineage diagrams, which greatly affects their application scenarios.

[0043] Therefore, this embodiment proposes a method for metadata collection and data lineage analysis in an integrated lake warehouse system, as detailed below:

[0044] Example 1

[0045] like Figure 1 The diagram illustrates the overall flow of a method for metadata collection and data lineage analysis in a lake warehouse integrated system provided in this embodiment. It mainly includes metadata collection, data lineage analysis, and data lineage visualization, achieved through the following steps:

[0046] S1: External data sources, lake warehouse integrated system data source layer, and data warehouse standard layer data sources are centrally managed through the data source management service. When a new data source is defined in the data source management service, a message to enable the collection of metadata is generated and sent to the message middleware. The message includes detailed information about the data source.

[0047] S2: After the metadata management service listens to the metadata collection message, it starts to collect the metadata information from the external data source at regular intervals. The metadata management service stores the collected metadata information as data nodes in the graph database.

[0048] S3: The data lineage management unit in the metadata management service performs the data lineage analysis based on the collected metadata node information, and finally persists the nodes and relationship edges of the data lineage graph to the graph database.

[0049] S4: The data lineage visualization page is used to visualize the data lineage, intuitively showing the mapping relationship and flow of data.

[0050] In addition, the data flow is tracked through the ETL system and the integrated lake warehouse system to track the data mapping relationship and conversion path.

[0051] The ETL system is responsible for cleaning and processing data from defined data sources (such as database models, file data models, APIs, etc.) and synchronizing them to the lake warehouse integrated system and the target data source. The ETL system includes a data model relationship parsing unit, responsible for parsing the data flow during data extraction, transformation, and loading during the ETL process, as well as the transformation relationships between data models. This data model transformation relationship is then sent as a JSON message body to the message middleware (KAFKA) for use by the metadata management service. The basic format of the JSON message body is as follows:Figure 2 As shown.

[0052] Specifically, in the present embodiment, the ETL data model relationship analysis unit is used to record the starting data source, data conversion process, and target data source information in the data ETL process, and convert it into a message and send it to the message middleware.

[0053] At the same time, the lake-warehouse integrated system includes the functions of data lake and data warehouse for structured data storage, and includes a data model relationship analysis unit responsible for analyzing the data conversion relationship of the source layer and the standard layer, and sending the data model conversion relationship in JSON message body to the message middleware (KAFKA) for the metadata management service, wherein the JSON message body structure is consistent with the ETL system, also refer to Figure 2 .

[0054] In the present embodiment, the lake-warehouse data model relationship analysis unit is used to record the source and target source, field mapping relationship of the source layer data and the standard layer data in the lake-warehouse system, and convert it into a message and send it to the message middleware.

[0055] Further, in step S1, the centralized management work of the data source management service includes adding new data sources, configuring connection information, specifying data source types, and defining collection frequency. At the same time, the data source management service can centrally manage each data source in a unified interface, and users can view, configure and modify all data sources to achieve integrated management of data sources.

[0056] In the present embodiment, the data source management system allows users to centrally manage the relevant information of all data sources, such as database connection information, API configuration, file path, etc. This allows users to view, configure and modify all data sources in a unified interface, and provides data source information query function through API for other services.

[0057] Further, in step S2, as Figure 3 shown, it is a metadata collection workflow diagram of the present embodiment, and the metadata collection is responsible for the metadata collection unit of the metadata management service, which obtains the data source connection information centrally managed in the data source management service, establishes connection and receives the collected metadata information in a timely manner.

[0058] Specifically, in the embodiment, the metadata management service includes two subsystems of a metadata collection unit and a data lineage management unit. The metadata collection unit is responsible for collecting metadata information from defined data sources (such as database models, file data models, data warehouse data models, etc.) at a time. The data lineage management unit is responsible for real-time consumption of data model relationships pushed into a message middleware (KAFKA) by ETL services, lake-warehouse systems, third-party services, and data model relationships added by users on a data lineage visualization page. Finally, the relationships of the data model are converted into starting nodes, relationship edges, target nodes, etc. of the data lineage graph, and are persisted into a database.

[0059] In the embodiment, the metadata information includes the data source connection information, table structure, field information, and data type.

[0060] Further, in step S3, as shown in a data lineage analysis workflow diagram of the embodiment, Figure 4 The data lineage management unit captures mapping relationships and conversion path messages of data models in the ETL system and the lake-warehouse integrated system in real time based on the metadata node information collected by the metadata collection unit, tracks data flow direction by analyzing the messages, identifies data sources and intermediate processes, determines data flow direction, and constructs the data source relationship graph.

[0061] Specifically, in the embodiment, the metadata management service includes two subsystems of a metadata collection unit and a data lineage management unit. The metadata collection unit is responsible for collecting metadata information from defined data sources (such as database models, file data models, data warehouse data models, etc.) at a time. The data lineage management unit is responsible for real-time consumption of data model relationships pushed into a message middleware (KAFKA) by ETL services, lake-warehouse systems, third-party services, and data model relationships added by users on a data lineage visualization page. Finally, the relationships of the data model are converted into starting nodes, relationship edges, target nodes, etc. of the data lineage graph, and are persisted into a database.

[0062] When the parsed data node message is not collected in the metadata collection process, the metadata related node is re-added.

[0063] Further, in step S4, referring to Figure 5 A data lineage visualization page display diagram of the embodiment, the data lineage visualization page supports manual addition and manual editing of the data lineage relationship graph, initiates a request for synchronizing data model relationships to the ETL system and the lake-warehouse integrated system, and realizes manual update of the data lineage relationship.

[0064] In the embodiment, the data bloodline visualization page displays data bloodline in three levels of nodes, including data source, data table and field.

[0065] Specifically, each data source represents a large node, the large node includes a plurality of data table sub-nodes, each data table sub-node includes a plurality of field sub-nodes, a connection line is used to represent a data conversion rule between the nodes, metadata information is displayed by clicking the corresponding node, detailed information of the conversion rule is displayed by clicking the connection line, and a specified job is jumped to by a detail label.

[0066] Therefore, the entire data link is displayed graphically, including data source, transmission, transformation and destination, and the user can clearly view the flow path and conversion relationship of data in different processing stages.

[0067] Finally, the metadata information and the data relationship graph information are uniformly stored by using the graph database, and the difficulty of bloodline relationship analysis is reduced.

[0068] The working principle or working process of the data bloodline management system provided by the application is as follows:

[0069] (1) The data source management service centrally manages all data sources, provides data source query capability, and other services can obtain connection, metadata acquisition and other configuration information of the data source through the service.

[0070] (2) The metadata acquisition unit acquires metadata information from different data sources at a time, automatically detects metadata changes and updates them to the database.

[0071] (3) The ETL data model relationship analysis unit analyzes input and output data, field mapping relationship and conversion rule in the ETL process, and converts them into messages to provide the metadata management unit for subsequent processing.

[0072] (4) The data warehouse data model relationship analysis unit analyzes input and output data, field mapping relationship and conversion rule in the lake warehouse system, and converts them into messages to provide the metadata management unit for subsequent processing.

[0073] (5) The data bloodline management unit captures messages in real time, obtains the starting point and ending point of the full-link data flow, and generates a complete data bloodline relationship graph.

[0074] (6) The data bloodline visualization page provides data bloodline adding, deleting, modifying and inquiring functions, and the user can also initiate a request for synchronizing data bloodline through the page.

[0075] (7) The data blood relationship visualization page visually displays the relationship among data sources, data tables, fields and the data flow path, and can jump to a specific data conversion job through the blood relationship diagram.

[0076] The above only describes the preferred embodiments of the present application, and the protection scope of the present application is not limited to the above-described embodiments. Any technical solution falling within the concept of the present application shall fall within the protection scope of the present application. It should be noted that, for ordinary skilled persons in the art, some improvements and refinements without departing from the principles of the present application shall also be considered as falling within the protection scope of the present application.

[0077] The technical features of the above-described embodiments can be combined arbitrarily, and to make the description concise, all possible combinations of the technical features in the above-described embodiments are not described, however, as long as the combinations of the technical features do not exist contradictory, they shall be considered as falling within the scope of the present application.

Claims

1. A method for metadata collection and data lineage analysis in a lake-warehouse integrated system, characterized in that, Includes the following steps: S1: External data sources, lake warehouse integrated system data source layer, and data warehouse standard layer data sources are centrally managed through the data source management service. When a new data source is defined in the data source management service, a message to enable metadata collection is generated and sent to the message middleware. The message includes detailed information about the data source. S2: After the metadata management service listens to the metadata collection message, it starts to collect metadata information from external data sources on a regular basis. The metadata management service stores the collected metadata information as data nodes in the graph database. S3: The data lineage management unit in the metadata management service performs data lineage analysis based on the collected metadata node information, and finally persists the nodes and relationship edges of the data lineage graph to the graph database. S4: The data lineage visualization page provides a visual representation of the data lineage, intuitively showing the mapping relationships and flow of data; The metadata collection and data lineage analysis process uses an ETL system and the lake warehouse integrated system to track the mapping relationship and transformation path of the data. The ETL system is responsible for synchronizing the defined data source to the lake warehouse integrated system and the target data source after cleaning and processing. The ETL system includes a data model relationship parsing unit, which is responsible for parsing the data flow and the transformation relationship between data models during the data extraction, transformation and loading ETL process, and sending the transformation relationship to the message middleware in JSON message body for use by the metadata management service. The integrated lake and warehouse system includes the functions of a data lake and a data warehouse for storing structured data. It also includes a data model relationship parsing unit, which is responsible for parsing the data transformation relationship between the source layer and the standard layer, and sending the transformation relationship as a JSON message body to the message middleware for use by the metadata management service. In step S3, based on the collected metadata node information, the mapping relationship and conversion path messages of the data models in the ETL system and the lake warehouse integrated system are captured in real time. By parsing the mapping relationship and conversion path messages, the data flow is tracked, the data source and intermediate process are identified, the data flow direction is determined, and the data lineage diagram is constructed.

2. The method for metadata collection and data lineage analysis in the integrated lake-warehouse system according to claim 1, characterized in that, In step S1, the centralized management work of the data source management service includes adding new data sources, configuring connection information, specifying data source types, and defining collection frequency.

3. The method for metadata collection and data lineage analysis in the integrated lake-warehouse system according to claim 1, characterized in that, In step S2, the metadata collection is handled by the metadata collection unit of the metadata management service. The metadata collection unit obtains the data source connection information centrally managed in the data source management service, establishes connections periodically, and receives the collected metadata information.

4. The method for metadata collection and data lineage analysis in the integrated lake-warehouse system according to claim 3, characterized in that, The metadata information includes the data source connection information, table structure, field information, and data type.

5. The method for metadata collection and data lineage analysis in the integrated lake-warehouse system according to claim 1, characterized in that, If the parsed metadata node information is not collected during the metadata collection process, then the metadata-related nodes are added again.

6. The method for metadata collection and data lineage analysis in the integrated lake-warehouse system according to claim 1, characterized in that, In step S4, the data lineage visualization page supports manually adding and editing the data lineage diagram, and actively initiates a request to the ETL system and the lake warehouse integrated system to synchronize the data model relationship, thereby realizing the manual update of the data lineage relationship.

7. The method for metadata collection and data lineage analysis in the integrated lake-warehouse system according to claim 6, characterized in that, The data lineage visualization page displays data lineage using a three-level node structure, which includes the data source, data table, and fields.

8. The method for metadata collection and data lineage analysis in the integrated lake-warehouse system according to claim 7, characterized in that, Each data source represents a large node, which includes multiple data table sub-nodes. Each data table sub-node includes multiple field sub-nodes. The nodes are connected by lines to represent data transformation rules. Clicking on the corresponding node displays metadata information, and clicking on the connecting line displays detailed information about the data transformation rules. The details tab allows users to jump to a specific job.

9. The method for metadata collection and data lineage analysis in the integrated lake-warehouse system according to claim 1, characterized in that, The graph database is used to uniformly store the metadata information and the data lineage diagram, reducing the difficulty of lineage analysis.

Images