Data blood relationship processing system and method

Abstract

The invention discloses a data blood relationship processing system and method, and the system comprises a data collection module which is used for automatically collecting data related to a data blood relationship; the blood relationship analysis module is used for automatically analyzing blood relationship information in the data acquired by the data acquisition module through a blood relationship analysis program; the blood relationship storage module is used for storing the analyzed blood relationship in a memory; the system provides a data full-link view for management, sharing and utilization of data assets, the influence of data model change can be found in time, and potential risks of change can be effectively recognized.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and specifically to a data lineage processing system and method. Background Art

[0002] Data lineage refers to a relationship that naturally forms among data during its entire life cycle, from generation, processing, transformation, integration, circulation to final extinction. This relationship is similar to human blood relationship, so it is called data lineage. Data lineage records the link relationship of data generation, including the source of data, transformation process, dependency relationship, etc. Through data lineage analysis, the source and flow of data, as well as the relationship and dependency between data, can be understood, thus helping data administrators and data analysts manage and apply data more efficiently.

[0003] As a goal of metadata management, data lineage depends on the entire metadata management activity. The common practice is to parse the storage and circulation process from data warehouses, big data platforms, and ETLs, which involves the unified storage of different types of data assets, the parsing of lineage relationships in multi-stage governance activities, and the visualization of data lineage relationships. For example: Patent 1: CN115408399A discloses a lineage analysis method based on SQL (Structured Query Language) scripts. The method is to parse and filter the content in the SQL script file to screen the target SQL statements in the SQL script file; perform syntax parsing on the target SQL statements to generate an abstract syntax tree; traverse all nodes of the abstract syntax tree to screen the nodes containing the preset data table keywords in the abstract syntax tree as target nodes; perform field parsing on the target nodes to parse the table fields of the data tables contained in the target nodes and the field lineage relationship of the table fields; construct a directed acyclic graph with each table field as a node according to the field lineage relationship to obtain the lineage analysis result.

[0004] In addition, patents CN113704343A and CN108228747A also disclose visualization methods for data lineage relationships, and the methods mainly perform visualization according to the relationships formed by the nodes in the data governance process; Currently, for modern data asset management activities, blood relationship is generated in each asset management activity. Data lineage should include parsing, storing, and visualizing the lineage information of products in the entire data life cycle. The lineage analysis method disclosed in the above-mentioned patent CN115408399A only performs lineage parsing on SQL scripts. The visualization methods of data lineage disclosed in patents CN113704343A and CN108228747A both emphasize visualization at the operator level. Therefore, the current lineage based only on SQL scripts is not sufficient to meet the requirements of modern data management. Based on the current situation, there is an urgent need to improve the existing technology. Summary of the Invention

[0005] The purpose of the present invention is to provide a data lineage processing system and method to solve the problems raised in the above background technology.

[0006] On the one hand, the present invention provides the following technical solution: A data lineage processing system, including: A data collection module, used to automatically collect data related to data lineage; A lineage parsing module, which automatically parses the lineage information in the data collected by the data collection module through a lineage parsing program; A lineage storage module, which stores the parsed lineage relationship in a memory; A lineage visualization module, which displays the lineage relationship stored in the memory in the form of a visualization interface.

[0007] Preferably, the data collection module provides the collection of various types of data nodes, processing nodes, and transfer path data.

[0008] Preferably, the lineage parsing module further includes a static lineage parsing program and a dynamic lineage parsing program.

[0009] Preferably, the lineage storage module further includes a multi-type memory adaptation program, which can store the lineage in different types of memories.

[0010] Preferably, the data lineage visualization module further includes operations such as multi-level lineage view visualization, lineage editing, multi-condition combination search, lineage graph depth configuration, lineage graph compression and expansion, and information viewing.

[0011] On the other hand, the present invention also provides the following another technical solution: A data lineage processing method, the steps include: Step S100: Collect data related to data lineage, and the data related to data lineage includes: data nodes, transfer paths, and processing nodes; Step S200: Parse the lineage relationship, and automatically parse the table-level, field-level, and operator-level lineage described in the data related to the data lineage relationship through a lineage parsing program; Step S300: Store the lineage relationship, abstract the parsed data lineage relationship into a unified entity relationship model, and persistently store the parsed data lineage relationship in a memory using a storage program; Step S40=0: Visualize the lineage relationship, read the stored entities and the relationship data between entities, construct a directed acyclic graph for the flow between entities, and intuitively display the data lineage relationships at the table level, field level, and operator level through a graphical interface.

[0012] The present invention has the following beneficial effects: The technical solution of a data lineage relationship processing method provided by the present invention provides a new path for the full-cycle management of metadata management; it can automatically parse the data lineage relationship from a data warehouse, a big data platform, and ETL, uniformly track and record the data lineage relationship, and provide visualization at the table level, field level, and operator level, solving the problems of unified parsing, storage, and visualization of multi-level, multi-type, multi-source, and full-process traceable data lineage.

[0013] The technical solution of a data lineage relationship processing system provided by the present invention provides a full-link view of data for the management, sharing, and utilization of data assets, can timely discover the impact of data model changes, and effectively identify potential risks of changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a schematic flowchart of the data lineage relationship processing method of the present invention; Figure 2 It is a schematic diagram of the specific steps of data lineage parsing of the present invention; Figure 3 It is a schematic block diagram of the structure of the data lineage relationship processing system of the present invention; Figure 4 It is a schematic diagram of a case for parsing column-level, table-level, and operator-level lineage of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0015] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0016] Please refer to Figure 3, on the one hand, the present invention provides the following technical solution: a data lineage processing system, including: A data collection module, which is used to automatically collect data related to data lineage; the data collection module provides the collection of multiple types of data nodes, processing nodes, and transfer path data; A lineage parsing module, which automatically parses the lineage information in the data collected by the data collection module through a lineage parsing program; the lineage parsing module also includes a static lineage parsing program and a dynamic lineage parsing program; A lineage storage module, which stores the parsed lineage relationships in a memory; the lineage storage module also includes multiple types of memory adaptation programs, which can store lineages in different types of memories; A lineage visualization module, which visually displays the lineage relationships stored in the storage in the form of a visualization interface. The data lineage visualization module also includes operations such as multi-level lineage view visualization, lineage editing, lineage addition, multi-condition combination search, lineage graph depth configuration, lineage graph compression and expansion, and information viewing.

[0017] Among them, the multi-level lineage view visualization means that the table-level, field-level, and operator-level lineage views can be viewed through configuration. The lineage editing means that the lineage graph can be added, deleted data nodes, and transfer paths through visual editing. The multi-condition combination search means that the lineage information can be queried through the combination of the attributes and extended attributes of the entity. The lineage graph depth configuration means that the upstream and downstream node depths of the default rendering of the lineage graph can be configured. The lineage graph compression and expansion means that the relationship display and collapse of the field level of the lineage graph can be switched. The information viewing means that the detailed information of the entity can be viewed by clicking on the data node, processing node, and transfer path.

[0018] Please refer to Figure 1 , on the other hand, the present invention also provides the following another technical solution: a data lineage processing method, the steps include: Step S100: Collect data related to data lineage, and the data related to data lineage includes data nodes, transfer paths, and processing nodes.

[0019] Specifically, the data node and the processing node jointly describe the process of data flow, transformation, and storage in the system. Among them, the data node represents the existence form and storage location of the data itself, and is an entity in the data lineage. In this embodiment, the data node is used to mark the data of the data lineage data node; the data nodes adopted include multiple types of data. Exemplarily, it includes different types of data such as MySQL, Greenplum, and Oracel.

[0020] The processing node is used to identify the processing methods and rules during the data flow process, and is usually used in the flow path between data nodes. By parsing the data in the processing node, the flow path between two data nodes is obtained, forming a data lineage relationship.

[0021] In the embodiment of the present invention, it further includes a flow path, which obtains the flow path between two data nodes by parsing the data in the processing node, forming a data lineage relationship; As an alternative embodiment of the present invention, the data scope related to the data lineage is specifically: The data of the data node includes the metadata of at least one of the following types of assets: Data table: Used to describe the metadata of tables and views in the database. For example, it describes the table name, field information, and business information of the data table; Message queue topic: Used to describe the metadata of topics in the message queue. For example, it describes the name of the Topic, schema information, and business information; Storage container: Used to describe the metadata of the data stored in the object storage container. For example, it describes the name and path information of the file stored in the S3 storage; Indicator: Used to describe the metadata of data indicators. In the embodiment of the present invention, it usually refers to the indicator data defined by the business, such as indicator name, indicator type, and indicator calculation logic; Label: Used to describe the metadata of entity labels. In the embodiment of the present invention, it is usually the entity label data defined by the business, such as label name, label type, and label calculation logic; Data dashboard: Used to describe the metadata of data visualization dashboards. In the embodiment of the present invention, it usually refers to the dashboards for visual analysis of data defined by the business, such as dashboard name and associated indicators; Data model: Used to describe the metadata of data models. In the embodiment of the present invention, it usually refers to the defined logical model and physical model of the data table, such as model name and model definition information; Data service application programming interface (API): Used to describe the metadata of data service APIs. In the embodiment of the present invention, it usually refers to the configuration information of the API, such as API name and data processing logic; Data product: Used to describe the metadata of data products. In the embodiment of the present invention, it usually refers to the deliverables after data governance, such as data product name, associated indicators, associated labels, and associated data tables.

[0022] The data of the processing node includes the metadata of at least one of the following types of assets: View: It is used to record data such as the definitions of ordinary views and physical views in the database, for example: view name, definition information, database type; Stored procedure: It is used to record data such as the definitions of stored procedures in the database, for example: stored procedure name, definition information, database type; Database query log: It is used to record data such as query logs in the database, for example: Binlog logs of the MySQL database; Data processing operator: It is used to record data in the data extraction, transformation, and loading processes, for example: data transformation nodes, data extraction nodes, and other data processing processes involved in the data extraction, transformation, and loading processes; further, this data processing operator can process data from different data platforms, including databases such as MySQL, Oracle, and DM, as well as computing platforms such as Spark. The data input from the above different platforms can be processed by the data operator and then reversely output to the target storage node. Further, the data processing operator includes most operations on data, including data merging, joining, transformation, processing, filtering, etc.

[0023] Data processing logic data: It is used to record the databases and data processing logics used in data nodes such as APIs, metrics, tags, data products, data dashboards, and data models.

[0024] Step S200: Parse the lineage relationship, and automatically implement the parsing of table-level, field-level, and operator-level lineage described in the data related to the data lineage relationship through a lineage parsing program; The lineage parsing program includes: a static lineage parsing program and a dynamic lineage parsing program.

[0025] The static lineage parsing program directly parses the lineage relationship in the collected data of the processing nodes. It reads the data carried in the processing nodes, such as data inflow nodes (sources of input data), processing logics, data outflow nodes, etc., including data in formats such as JSON and log files. By the factory method of the parsing program, according to characteristics such as the type or format of the data of the processing nodes, it is allocated to the appropriate adapter to parse the corresponding lineage information.

[0026] Exemplarily, the adapter can be: Database adapter, which is used to parse the data of the processing nodes of the database. Exemplarily, such as an SQL adapter, which takes view, stored procedure, processing script, and query log SQL codes as input, generates an abstract syntax tree by analyzing the SQL codes, depth-first traverses all nodes of the abstract syntax tree, parses the data inflow nodes, data outflow nodes, and processing processes, and generates a data lineage relationship graph based on the parsed association relationships.

[0027] The data processing logic adapter parses the definition information in the data processing logic, and parses out the incoming data nodes, outgoing data nodes, and processing procedures.

[0028] The dynamic lineage parsing program refers to placing a hook program at the executor level of processing nodes such as databases. When the executor receives any SQL or processing script, the hook program is triggered to immediately capture the incoming data nodes, outgoing data nodes, and processing procedure information, and pass this information to the downstream processing logic to build or update the data lineage graph.

[0029] Reference Figure 2 , as an alternative embodiment of the present invention, this embodiment is used to describe the process and specific operations of lineage parsing. The steps include: Step S201: Read the data of the processing node, Step S202: Obtain the type data from the data of the processing node, and distribute the appropriate adapter according to the data type by the parsing program factory method; Step S203: The adapter parses the lineage carried by the data of the data processing node using the ANTLR4 grammar parser to generate an abstract syntax tree (AST); ANTLR4, that is, ANother Tool for Language Recognition 4, uses the ANTLR4 grammar parser to parse the lineage carried by the data processing node, and its specific process is not specifically elaborated in this embodiment.

[0030] Step S204: Traverse the generated abstract syntax tree, and parse out the incoming data nodes, outgoing data nodes, and processing procedures through the Parse-Tree Visitor pattern. The Parse-Tree Visitor pattern, that is, the parse tree visitor pattern, is a way to actively traverse the syntax analysis tree, and returns different values according to the type of the node or the processing result.

[0031] Step S300: Store the lineage relationship, abstract the parsed lineage relationship into a unified entity relationship model, and use the storage program to store the parsed lineage relationship in the memory; where The entity relationship model includes an entity table and multiple extension tables. The entity table stores all entity information related to data, and the entity includes all data nodes, processing nodes, and transfer paths related to lineage; the extension table stores all extension information of the entity. The extension table refers to storing the extension information of multiple entities that match the data nodes, processing nodes, and transfer paths.

[0032] As an alternative embodiment of the present invention, the lineages collected from different platforms defined in this embodiment are uniformly stored in the following structure: Lineage Table {Global Unique Identifier, Inflow Node Identifier, Outflow Node Identifier, Lineage Type, Processing Procedure, Creation Time, Update Time} For a more intuitive description, define the logical structure of a stored lineage table as follows: lineage_table{lineage_guid,from_entity_guid,to_entity_guid,lineage_type,process_source,created_time,update_time} Where: lineage_guid is the global unique identifier of the lineage relationship, representing the edge of the lineage; from_entity_guid represents the inflow node identifier of the lineage, indicating the starting point of the lineage relationship; to_entity_guid represents the outflow node identifier of the lineage, indicating the ending point of the lineage relationship; Details of the inflow and outflow nodes of the lineage are obtained through associated queries using the inflow node identifier from_entity_guid and the outflow node identifier to_entity_guid respectively; lineage_type represents the lineage type, such as TABLE (table-level lineage), VIEW (view lineage), COLUMN (field-level lineage), PIPELINE_TO_TABLE (operator-table lineage), TABLE_TO_PIPELINE (table-operator lineage); process_source represents the processing procedure corresponding to this lineage, commonly an SQL statement; created_time indicates the creation time of this lineage; update_time indicates the update time of this lineage.

[0033] As an alternative embodiment of the present invention, in this embodiment, taking the lineage information recorded in a data warehouse, big data platform or ETL script as an example, collect table-level lineage and field-level lineage. Set the following script: create table test_a as select id,name from test_b; The above SQL statement describes the table-level lineage with test_a as the outflow data table and test_b as the inflow data table, as well as the field-level lineage associated with the id and name fields in tables test_a and test_b; there will be three records of the specific stored lineage types, namely: the lineage type between test_a and test_b is table-level lineage, the lineage type between test_a_id and test_b_id is field-level lineage, and the lineage type between test_a_name and test_b_name is field-level lineage.

[0034] As an alternative embodiment of the present invention, in this embodiment, taking the lineage carried by the stored procedure as an example, the following Oracle stored procedure syntax is set: DECLARE z_id source_table.id%TYPE; z_name source_table.name%TYPE; CURSOR load_cursor IS SELECT id,name FROM source_table; BEGIN OPEN load_cursor; LOOP FETCH load_cursor INTO z_id,z_name; EXIT WHEN load_cursor%NOTFOUND; INSERT INTO target_table(id,name) VALUES (z_id,z_name); END LOOP; CLOSE load_cursor; COMMIT; END; The above Oracle stored procedure defines the cursor of load_cursor, which is used to select the id and name columns from the source table source_table, and then loop through the LOOP with this cursor to retrieve the data id and name from the source table source_table and insert these data into the id and name fields of the target table target_table; through lineage parsing, a table-level lineage between the source table source_table and the target table target_table, and two field-level lineages between the id and name columns will be generated.

[0035] As an alternative embodiment of the present invention, in this embodiment, taking the collection of blood relationship information carried by the view as an example, the following Greenplum view creation syntax is set: CREATE VIEW VIEW_1 AS SELECT VIEW_2.NAME FROM VIEW_2; The above SQL statement describes the view-level blood relationship with VIEW_2 as the outflow view and VIEW_1 as the inflow view. The specific storage type includes two records: the record with the blood relationship type of VIEW (view blood relationship) for VIEW_1 and VIEW_2, and the field-level blood relationship of the NAME field between the two views.

[0036] As an alternative embodiment of the present invention, in this embodiment, taking the data processing operator as an example, the following data processing process is set: Refer to Figure 4 , in this embodiment, a complete data link from the tables A (MySQL) and B (Greenplum) collected from different data platforms is shown. Through the data processing operator P, after data merging, creating a temporary table, and data filtering, it is transformed into the output table D (Oracle) of the specified platform. This link includes the entire process from tables A and B through the data processing operator P to the output, such as the merging of data tables A and B; the temporary table C created based on the fields a1 and b2 extracted from data tables A and B respectively; the data is filtered based on the defined logic and then passed to the output table D. It includes two field-level blood relationships of a1 and b2, two table-level blood relationships of the data table D formed after the merging of data tables A and B, and the table-operator blood relationship from data tables A and B to the operator P, and the operator-table blood relationship from the operator P to the output table D; in this embodiment, by reading the data processing operator P, the field-level, table-level, and operator-level blood relationships are simultaneously parsed, and the above three parsed blood relationships are stored in the blood relationship table, and then visually displayed in the blood relationship visualization module. The stored blood relationships include field-level, table-level, and operator-level blood relationships, and four types of data blood relationship records are generated in the blood relationship storage table lineage_table: two field-level blood relationships with the blood relationship type lineage_type of COLUMN, such as the field-level blood relationship a1 in table A and the field-level blood relationship b2 in table B, two table-level blood relationships with the blood relationship type lineage_type of TABLE, two table-operator blood relationships with the blood relationship type lineage_type of TABLE_TO_PIPELINE, and one operator-table blood relationship with the blood relationship type lineage_type of PIPELINE_TO_TABLE.

[0037] Step S400: Visualize the lineage relationship, read the stored entities and the relationship data between entities, construct a directed acyclic graph for the flow between entities, and visually display the lineage relationships at the table level, field level, and operator level through a graphical interface; The visualized lineage relationship includes: data node rendering, flow path rendering, and processing node rendering. The data node rendering is the node on the visualization graph, the flow path rendering is the edge between two nodes on the visualization graph, and the processing node rendering is located on the flow path, indicating the processing logic that occurs during the data flow.

[0038] As an optional embodiment of the present invention, in this embodiment, the lineage graph at a specified depth upstream and downstream of the entity can be obtained by depth traversal through the primary key guid of a certain lineage entity: In this embodiment, in the lineage display processing program, the concepts of lineage node Node and lineage edge Edge are introduced. The main output structure of the lineage display is: Lineage visualization structure (node list, edge list), where the definitions of nodes and edges are as follows: The node is the globally unique identifier of the data and the entity type; The edge refers to the globally unique identifier of the lineage edge, the unique identifier of the entity into which the data flows, the unique identifier of the entity out of which the data flows, and the field list, where the definition of the field is as follows: The field refers to the globally unique identifier of the lineage edge, the unique identifier of the column of the entity into which the data flows, and the unique identifier of the column of the entity out of which the data flows; In this embodiment, for a more intuitive description, a lineage visualization structure is defined as: Lineage visualization structure: Lineage_output(List <node>nodes, List <edge>edges); Node: Node(guid, entity_type); Edge: Edge(edge_guid, from_entity_guid, to_entity_guid, List <column>columns); Field: Column(edge_guid, from_entity_guid, to_entity_guid); In the lineage node Node, guid represents the unique primary key identifier for the input inflow or data outflow node, and entity_type represents the entity type (such as a table or a data operator, etc.) corresponding to this node; In the lineage edge Edge, edge_guid represents the unique primary key identifier of this lineage edge, from_entity_guid represents the unique primary key identifier of the lineage data inflow node, to_entity_guid represents the unique primary key identifier of the lineage data outflow node, and columns represents the detailed list of field lineage subordinate to the table-level lineage; In the detailed list of field lineage Column, edge_guid represents the unique primary key identifier of the column lineage edge, from_entity_guid represents the unique primary key identifier of the data inflow column node, and to_entity_guid represents the unique primary key identifier of the target column node.

[0039] In this embodiment, in the lineage output lineage_output of a certain entity guid, there are multiple upstream and downstream related entities Nodes. Each Node stores the specific information of the related entity, such as the primary key guid, entity type, name, description, and other information. edges represents the lineage edges. Each Edge contains the source and destination entity guids of the superior entity (such as a table), and at the same time contains the corresponding relationship for finding the subordinate entity (such as a column) based on the source and destination entity guids.

[0040] In this embodiment, in the lineage depth traversal program, taking the current node as the root node, perform a depth traversal upstream to the upstream nodes (the edges in lineage_table where to_entity_guid is the current node guid), and at the same time add the node information and edge information to the lineage_output list, recording the current traversal depth. At the same time, perform a depth traversal downstream to the downstream nodes (the edges in lineage_table where from_entity_guid is the current node), adding the node information and edge information, and recording the current depth. When exceeding the user-specified depth, terminate the depth traversal, and finally the lineage output result of the specified depth of the current node can be obtained.

[0041] In this embodiment, the blood relationship storage visualization module supports both manual single - item addition and modification of blood relationship functions. The user only needs to manually select two nodes for which the blood relationship needs to be created, such as two tables or two columns. Then, the background can generate the corresponding type of blood relationship information based on the guid of the currently selected entity and store it in the lineage_table. At the same time, the user is supported to select the edge_guid (blood relationship edge guid) in the lineage_output or the edge_guid (blood relationship edge guid) in a single column for deletion operations. During the modification operation, the user can re - select the guid of one of the current edge nodes for re - binding.

[0042] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or perform equivalent replacements for some of the technical features. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.< / column> < / edge> < / node>

Claims

1. A data lineage processing system, characterized in that, including: a data collection module, configured to automatically collect data related to data lineage; and the data collection module provides collection of data of multiple types of data nodes, processing nodes, and transfer paths; a lineage parsing module, configured to automatically parse lineage information in the data collected by the data collection module through a lineage parsing program; and the lineage parsing module further includes a static lineage parsing program and a dynamic lineage parsing program; a lineage storage module, configured to store the parsed lineage relationships in a memory; and the lineage storage module further includes a multi-type memory adaptation program for storing lineages in different types of memories; a lineage visualization module, configured to visually display the lineage relationships of data; and the data lineage visualization module is further configured for multi-level lineage view visualization, lineage editing, multi-condition combination search, lineage graph depth configuration, lineage graph compression and expansion, and information viewing; wherein, the multi-level lineage view visualization is to view table-level, field-level, and operator-level lineage views in a configured manner; the lineage editing is to add or delete data nodes and transfer paths to the lineage graph in a visual editing manner; the multi-condition combination search is used to query lineage information by combining the attributes and extended attributes of entities; the lineage graph depth configuration is used to configure the upstream and downstream node depths for the default rendering of the lineage graph; the lineage graph compression and expansion is used to switch between showing and collapsing the relationships at the field level of the lineage graph; the information viewing is used to view the detailed information of entities of data nodes, processing nodes, and transfer paths.

2. A method for processing data lineage based on the system described in claim 1, characterized in that the steps including: Step S100: Collect data related to data lineage, where the data related to data lineage includes data nodes, transfer paths, and processing nodes; Step S200: Parse lineage relationships, and perform lineage parsing on the table level, field level, and operator level described in the data related to the data lineage relationship through a lineage parsing program; Step S300: Store lineage relationships, abstract the parsed data lineage relationships into a unified entity relationship model, and use a storage program to store the parsed data lineage relationships in a memory; [[ID= 3. The data lineage processing method according to claim 2, wherein: ​ ​ ​ ​ 4. The data lineage processing method according to claim 2 or 3, characterized in that: ​ ​ ​ Storage Container: Metadata used to describe data stored in an object storage container; Metric: Metadata used to describe data metrics, including metric data defined by the business; Tag: Metadata used to describe entity tags, including entity tag data defined by the business; Data Dashboard: Metadata used to describe a data visualization dashboard, including a dashboard for visual analysis of data defined by the business; Data Model: Metadata used to describe a data model, including the defined logical and physical models of data tables; Data Product: Metadata used to describe a data product, including the deliverables after data governance; 5. The data lineage processing method according to claim 2 or 3, characterized in that: In step S100, the data of the processing node includes metadata of at least one of the following types of assets: View: Data used to record the definitions of ordinary views and physical views in a database; Stored Procedure: Data used to record the definitions of stored procedures in a database; Database Query Log: Data used to record query logs in a database; Data Processing Operator: Data used to record the data extraction, transformation, and loading processes; Data Processing Logic Data: Data used to record the databases and data processing logic used in metric, tag, data product, data dashboard, and data model data nodes; 6. The data lineage processing method according to claim 2, wherein: In step S200, the lineage parsing program includes: a static lineage parsing program and a dynamic lineage parsing program; where the static lineage parsing program directly parses the lineage relationships in the collected data of the processing node. It reads the data carried in the processing node, and the parsing program factory method distributes it to the appropriate adapter according to the data characteristics of the processing node to parse the corresponding lineage information. The characteristics of the data include data type and data format; the dynamic lineage parsing program places a hook program at the executor level of the database waiting for the processing node. When the executor receives any SQL or processing script, the hook program is triggered to immediately capture the incoming data node, outgoing data node, and processing process information, and pass this information to the downstream processing logic to build or update the data lineage graph.

7. The data lineage processing method according to claim 6, wherein: The adapter includes: SQL Adapter: Takes the SQL code in views, stored procedures, processing scripts, and query logs as input, generates an abstract syntax tree by analyzing the SQL code, and traverses all nodes of the abstract syntax tree in depth-first order to parse the incoming data node, outgoing data node, and processing process; Data Processing Logic Adapter: Parses the definition information in the data processing logic to parse the incoming data node, outgoing data node, and processing process.

8. The data lineage processing method according to claim 2, wherein: In step S200, the specific steps for parsing the lineage relationship include: Step S201: Read the data of the processing node; Step S202: Read the type data of the data of the processing node, and the parsing program factory method distributes it to the appropriate adapter according to the type data; Step S203: The adapter parses the lineage carried by the data of the data processing node using an ANTLR4 syntax parser to generate an abstract syntax tree; Step S204: Traverse the abstract syntax tree, and parse the incoming data node, outgoing data node, and processing process through the Parse-Tree Visitor parsing tree accessor pattern.

9. The data lineage processing method according to claim 2, wherein: In step S300, the entity relationship model includes an entity table and multiple extension tables; among them, the entity table is used to store all entity information related to data requirements, and the entity includes all data nodes, processing nodes, and transfer paths related to blood relationship; the extension table is used to store the extension information of all entities, and the extension information includes metadata related to the entities and the relationships between the entities.

10. The data lineage processing method according to claim 2, wherein: In step S400, the visualized blood relationship includes: data node rendering, transfer path rendering, and processing node rendering; among them, the data node rendering is the node on the visualized graph; the transfer path rendering is the edge between two nodes on the visualized graph; the processing node rendering is located on the transfer path and represents the processing logic that occurs during the data transfer process.

Images