Data blood relationship display method and system

Abstract

The invention relates to the technical field of big data, and particularly provides a data consanguinity display method and system, and the method comprises the following steps: S1, data collection and metadata management; s2, data source tracing; s3, data destination tracking; and S4, carrying out full-link analysis and visualization. Compared with the prior art, the method can improve data transparency, enhance data management, improve data quality, improve data security and compliance, support data analysis and decision, and improve data operation and maintenance efficiency.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and specifically provides a data lineage display method and system. Background Art

[0002] With the rapid development of big data technology, enterprises and organizations are facing increasingly complex data processing processes and data governance challenges. In this context, data lineage tracking has become a key link in ensuring data quality, compliance, and security. However, existing data lineage tracking methods have some common problems.

[0003] 1. Existing data lineage tracking systems are often too complex to be understood and operated by ordinary users. Complex interfaces and cumbersome operation steps make it difficult for users to obtain the required data lineage information, thus affecting the efficiency and reliability of data governance.

[0004] 2. Existing data lineage tracing methods have limitations in displaying data relationships. Some methods can only display part of the data flow relationship, but cannot fully display the source and destination of the data, which brings difficulties to data tracing and security auditing.

[0005] 3. Existing technologies often fail to provide intuitive visual displays and are unable to present the complete lineage relationship of data in a graphical manner, which limits users’ intuitive understanding and analysis of data flow.

[0006] In the face of the above problems, the industry has an increasingly urgent need for a method that is concise, intuitive, and comprehensive in displaying data lineage relationships. Therefore, it is crucial to propose a new data lineage display method to solve the above problems. Summary of the invention

[0007] The present invention aims at the above-mentioned deficiencies of the prior art and provides a data lineage display method with strong practicality.

[0008] A further technical task of the present invention is to provide a data lineage display system that is rationally designed, safe and applicable.

[0009] The technical solution adopted by the present invention to solve its technical problem is:

[0010] A data lineage display method comprises the following steps:

[0011] S1. Data collection and metadata management;

[0012] S2, data source tracing;

[0013] S3, data destination tracking;

[0014] S4. Full-link analysis and visualization.

[0015] Furthermore, in step S1, data and metadata during processing are collected from various data sources, and the collected metadata is stored in a central metadata management system. The central metadata management system adopts a storage and retrieval mechanism to ensure that the metadata can be read, written and analyzed.

[0016] Furthermore, in step S2, relevant metadata is extracted from the central metadata management system, the metadata of each data item is parsed, the initial source of the data item is determined, and based on the parsing results, a source relationship diagram of the data item is constructed, wherein the source relationship diagram shows the starting source of each data item and the initial association relationship with other data items.

[0017] Furthermore, in step S3, the data flow in each processing stage is tracked, the input and output relationship of each processing link is recorded, the data flow path between different processing nodes is analyzed, and the final destination of the data is determined;

[0018] Based on the flow tracking results, a data destination relationship diagram is constructed, which shows the flow process of data from one node to the next node and its final destination.

[0019] Furthermore, in step S4, the source relationship diagram and the destination relationship diagram of the data are comprehensively analyzed to generate a full-link diagram of the data, which covers all flow nodes of the data and shows all paths of the data from the source to the destination.

[0020] A data lineage display system that first collects data and manages metadata, traces back from the source of the data, tracks the destination of the data, and finally performs full-link analysis and visualization.

[0021] Furthermore, metadata of data and its processing process are collected from various data sources, and the collected metadata is stored in a central metadata management system. The central metadata management system adopts storage and retrieval mechanisms to ensure that the metadata can be read, written and analyzed.

[0022] Furthermore, relevant metadata is extracted from the central metadata management system, the metadata of each data item is parsed, the initial source of the data item is determined, and based on the parsing results, a source relationship diagram of the data item is constructed, which shows the starting source of each data item and the initial association relationship with other data items.

[0023] Furthermore, when tracking the destination of data, we track the flow of data in each processing stage, record the input and output relationship of each processing link, analyze the flow path of data between different processing nodes, and determine the final destination of the data;

[0024] Based on the flow tracking results, a data destination relationship diagram is constructed, which shows the flow process of data from one node to the next node and its final destination.

[0025] Furthermore, when performing full-link analysis and visualization, the source relationship diagram and destination relationship diagram of the data are comprehensively analyzed to generate a full-link diagram of the data. The full-link diagram covers all flow nodes of the data and displays all paths of the data from the source to the destination.

[0026] Compared with the prior art, the data lineage display method and system of the present invention has the following outstanding beneficial effects:

[0027] (1) Improve data transparency. This method can clearly display the entire flow process of data from source to destination, including the source of data, transmission path, conversion steps and storage location. This transparency enables data users to clearly understand the entire life cycle of data, helps to trace the generation and change history of data, and ensures the reliability and accuracy of data.

[0028] (2) Enhanced data governance: By providing detailed data lineage information, this method significantly enhances data governance capabilities. It supports organizations in defining and executing data management policies to ensure the consistency and compliance of data throughout the entire processing process. Data lineage tracking can also help discover and correct data quality issues and improve the overall level of data management.

[0029] (3) Improve data quality. The data lineage display method enables data quality problems to be quickly identified and located. For example, if data in a certain link is abnormal, lineage analysis can quickly trace the source of the problem so that corrective measures can be taken in a timely manner. This capability helps maintain the high quality and integrity of data.

[0030] (4) Improve data security and compliance. This method records the changes in data at different processing stages and can trace the source and responsible person of each step. This detailed record is very helpful for data auditing and compliance checks to ensure that data processing complies with relevant laws, regulations and industry standards.

[0031] (5) Supporting data analysis and decision-making, this method enables users to easily understand and analyze the flow and changes of data through an intuitive visual interface. This not only improves the efficiency of data analysis, but also provides a reliable basis for data-driven decision-making and optimizes business processes and operational strategies.

[0032] (6) Improve data operation and maintenance efficiency. In a complex data environment, operation and maintenance personnel often need to quickly locate and solve data problems. The data lineage display method provides a detailed data flow path, allowing operation and maintenance personnel to quickly identify fault points and potential risks, thereby greatly shortening the time for problem investigation and resolution and improving operation and maintenance efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0034] Attached Figure 1 It is a flowchart of a method for displaying data lineage. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] A best embodiment is given below:

[0037] like Figure 1 As shown, a data lineage display method in this embodiment has the following steps:

[0038] S1. Data collection and metadata management;

[0039] Collect data and metadata from various data sources (such as databases, data warehouses, data lakes, etc.) during the data processing process. Metadata includes but is not limited to the creation time, creator, data source, processing flow node information, data flow records, etc. of the data item. The collected metadata is stored in a central metadata management system. The system uses an efficient storage and retrieval mechanism to ensure that metadata can be quickly read, written and analyzed.

[0040] S2, data source tracing;

[0041] Extract relevant metadata from the metadata management system and parse the metadata of each data item. Determine the initial source of the data item, including the creator, creation time, data source and other information of the data. Based on the parsing results, construct a source relationship diagram of the data items. The diagram shows the starting source of each data item and the initial association relationship with other data items.

[0042] S3, data destination tracking;

[0043] Track the flow of data at each processing stage and record the input and output relationship of each processing link. Analyze the flow path of data between different processing nodes to determine the final destination of the data. Based on the flow tracking results, build a data destination relationship diagram. The diagram shows the flow process of data from one node to the next node and its final destination.

[0044] S4, full-link analysis and visualization;

[0045] Comprehensively analyze the source relationship diagram and destination relationship diagram of the data to generate a full-link diagram of the data. Ensure that the full-link diagram covers all flow nodes of the data and can fully display all paths of the data from the source to the destination.

[0046] The full-link diagram is displayed using a graphical interface, allowing users to intuitively see all relevant nodes and their relationships during the entire data processing process. Interactive operations are supported, such as zooming, dragging, and clicking nodes to view detailed information, making it easier for users to deeply analyze specific data flows.

[0047] Based on the above method, a data lineage display system in this embodiment first performs data collection and metadata management, traces back from the data source, tracks the data destination, and finally performs full-link analysis and visualization.

[0048] Among them, data and metadata during its processing are collected from various data sources, and the collected metadata is stored in a central metadata management system. The central metadata management system adopts a storage and retrieval mechanism to ensure that the metadata can be read, written and analyzed.

[0049] Relevant metadata is extracted from the central metadata management system, the metadata of each data item is parsed, the initial source of the data item is determined, and based on the parsing results, a source relationship diagram of the data item is constructed, wherein the source relationship diagram shows the starting source of each data item and the initial association relationship with other data items.

[0050] When tracking the destination of data, track the flow of data in each processing stage, record the input and output relationship of each processing link, analyze the flow path of data between different processing nodes, and determine the final destination of the data;

[0051] Based on the flow tracking results, a data destination relationship diagram is constructed, which shows the flow process of data from one node to the next node and its final destination.

[0052] When performing full-link analysis and visualization, the source relationship diagram and destination relationship diagram of the data are comprehensively analyzed to generate a full-link diagram of the data. The full-link diagram covers all flow nodes of the data and shows all paths of the data from the source to the destination.

[0053] The above-mentioned specific implementations are only specific cases of the present invention. The patent protection scope of the present invention includes but is not limited to the above-mentioned specific implementations. Any technical solutions that conform to the above-mentioned specific implementations of the present invention and any appropriate changes or substitutions made by ordinary technicians in the relevant technical field shall fall within the patent protection scope of the present invention.

[0054] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A data lineage display method, characterized in that: The steps are as follows: S1. Data collection and metadata management; S2, data source tracing; S3, data destination tracking; S4. Full-link analysis and visualization.

2. A data lineage display method according to claim 1, characterized in that: In step S1, data and metadata during its processing are collected from various data sources, and the collected metadata is stored in a central metadata management system. The central metadata management system adopts a storage and retrieval mechanism to ensure that the metadata can be read, written and analyzed.

3. A data lineage display method according to claim 2, characterized in that: In step S2, relevant metadata is extracted from the central metadata management system, the metadata of each data item is parsed, the initial source of the data item is determined, and based on the parsing results, a source relationship diagram of the data item is constructed, in which the starting source of each data item and the initial association relationship with other data items are displayed.

4. A data lineage display method according to claim 3, characterized in that: In step S3, the data flow in each processing stage is tracked, the input and output relationship of each processing link is recorded, the data flow path between different processing nodes is analyzed, and the final destination of the data is determined; Based on the flow tracking results, a data destination relationship diagram is constructed, which shows the flow process of data from one node to the next node and its final destination.

5. A data lineage display method according to claim 4, characterized in that: In step S4, the source relationship graph and the destination relationship graph of the data are comprehensively analyzed to generate a full-link graph of the data, which covers all flow nodes of the data and shows all paths of the data from the source to the destination.

6. A data lineage display system, characterized in that: First, data collection and metadata management are carried out, tracing back from the data source, tracking the data destination, and finally performing full-link analysis and visualization.

7. A data lineage display system according to claim 6, characterized in that: Collect data from various data sources and metadata during their processing, and store the collected metadata in a central metadata management system. The central metadata management system uses storage and retrieval mechanisms to ensure that metadata can be read, written and analyzed.

8. A data lineage display system according to claim 7, characterized in that: Relevant metadata is extracted from the central metadata management system, the metadata of each data item is parsed, the initial source of the data item is determined, and based on the parsing results, a source relationship diagram of the data item is constructed, wherein the source relationship diagram shows the starting source of each data item and the initial association relationship with other data items.

9. A data lineage display system according to claim 8, characterized in that: When tracking the destination of data, track the flow of data in each processing stage, record the input and output relationship of each processing link, analyze the flow path of data between different processing nodes, and determine the final destination of the data; Based on the flow tracking results, a data destination relationship diagram is constructed, which shows the flow process of data from one node to the next node and its final destination.

10. A data lineage display system according to claim 9, characterized in that: When performing full-link analysis and visualization, the source relationship diagram and destination relationship diagram of the data are comprehensively analyzed to generate a full-link diagram of the data. The full-link diagram covers all flow nodes of the data and shows all paths of the data from the source to the destination.

Images