Data blood relationship analysis method, device, equipment, medium and product
By establishing an initial lineage graph in the data lake and correcting the lineage relationships across the entire data chain, the problem of cross-platform data flow tracking was solved, and accurate identification and modeling of data lineage relationships were achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-30
- Publication Date
- 2026-03-27
AI Technical Summary
Existing technologies struggle to achieve cross-platform data flow tracking, resulting in insufficient accuracy in identifying data lineage.
By establishing an initial lineage graph in the data lake, and generating full-link data lineage relationships based on the data table structure and target data at the data source end, the initial lineage graph is corrected using time-dimensional dependency chains and field-level lineage relationships, thus realizing cross-database data lineage modeling.
It improves the accuracy of data lineage identification, can track the flow of data lineage throughout the data lake, and enables data lineage modeling across databases and learning platforms.
Smart Images
Figure CN121743545A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data management technology, and in particular to a data lineage analysis method, apparatus, equipment, medium and product. Background Technology
[0002] A data lake is a centralized data storage architecture used to store structured, semi-structured, and unstructured data of any size in its raw format, supporting various analytical processing methods. However, unmanaged data lakes can easily become "data swamps," suffering from problems such as raw data accumulation, missing metadata, and leakage of sensitive information. The lack of standardized processes leads to low data utilization, making it difficult for business users to discover usable data. Existing technologies for automatically obtaining data lineage typically rely on metadata analysis, most commonly obtaining table-level lineage. Traditional methods struggle to track cross-platform data flow, resulting in insufficient accuracy in identifying data lineage. Summary of the Invention
[0003] This application provides a data lineage analysis method, apparatus, device, medium, and product, which solves the problem that current data lineage acquisition methods are difficult to achieve cross-platform data flow tracking, resulting in insufficient accuracy in identifying data lineage relationships.
[0004] Firstly, to achieve the above objectives, embodiments of this application provide a data lineage analysis method, comprising:
[0005] Establish an initial lineage graph of data in the data lake;
[0006] Based on the structure of the data table at the data source end and the target data, establish the full-link data lineage relationship corresponding to the data table at the data source end; wherein, the target data includes test data and / or feature data;
[0007] Based on the full-link data lineage, the initial lineage map is corrected to obtain the target lineage map.
[0008] This includes establishing the initial lineage graph of data in the data lake, including:
[0009] Obtain the database scripts used for data processing in the data lake and the execution logs corresponding to the database scripts;
[0010] Based on the execution order and context information in the execution log, establish a time-dimensional dependency chain in chronological order;
[0011] Based on the time-dimensional dependency chain, the tree structure of the database script is traversed to obtain the information corresponding to each node in the tree structure, wherein the information corresponding to each node includes at least one of the following: data table, field, and condition.
[0012] The initial bloodline relationship map is established based on the information corresponding to each node.
[0013] The initial kinship graph is established based on the information corresponding to each node, including:
[0014] Based on the associated tables and join conditions corresponding to the merge nodes in the tree structure of the selection statement, generate a set of table-level data lineage relationships;
[0015] Based on the fields corresponding to the column expression nodes of the tree structure of the selection statement, the associated tables in the merge nodes, and / or the aggregate fields corresponding to the grouping nodes, a first field-level lineage relationship set is generated.
[0016] Based on the target table, the fields corresponding to the setting nodes, and the fields corresponding to the filtering condition nodes in the tree structure of the insert or update statements, generate a second-level lineage relationship set.
[0017] The initial bloodline graph is established based on the table-level bloodline relationship set, the first field-level bloodline relationship set, and the second field-level bloodline relationship set.
[0018] The target data includes test data. Based on the structure of the data table at the data source end and the target data, a full-link data lineage relationship is established corresponding to the data table, including:
[0019] Select N empty data tables from the data source and insert i rows of target data into the i-th data table of the N empty data tables to obtain N data tables; N is a positive integer and i is a positive integer less than or equal to N;
[0020] In the data lake test environment, the data in the N data tables are processed to determine the target data table of the data platform and the target data table of the data lake. The data platform is the platform through which the data table from the data source flows, and the fields in the target data table include data related to the target data.
[0021] The end-to-end data lineage is established based on the data table at the data source, the target data table at the data platform, and the target data table at the data lake.
[0022] The target data includes feature data. Based on the structure of the data table at the data source end and the target data, a full-link data lineage relationship is established corresponding to the data table at the data source end, including:
[0023] Feature data is generated for each field of the data table at the data source using a feature data generation algorithm;
[0024] For the data tables at the data source end, execute the database scripts used for data processing in the data lake to determine the databases, data tables, and fields in the data lake that include the feature data;
[0025] The end-to-end data lineage is established based on at least one of the database, data table, and fields including the feature data and the data table at the data source end.
[0026] Specifically, the process of correcting the initial kinship map based on the full-link data kinship to obtain the target kinship map includes at least one of the following:
[0027] If no database or table relationship exists in the initial lineage graph, the tables not present in the initial lineage graph are sorted according to the time-dimensional dependency chain. A time-dimensional dependency chain is established with the data table at the data source as the head, and the initial lineage graph is updated according to the time-dimensional dependency chain to obtain the target lineage graph. The time-dimensional dependency chain is established based on the execution order and context information in the execution log corresponding to the database script.
[0028] If the initial lineage graph contains lineage relationships between databases and tables in the full-link data lineage relationship, but does not contain field-level lineage relationships in the full-link data lineage relationship, the initial lineage graph is updated by establishing field-level lineage relationship dependencies to obtain the target lineage graph.
[0029] The method further includes:
[0030] The target lineage graph is updated using the monitored task change events.
[0031] Secondly, to achieve the above objectives, embodiments of this application provide a data lineage analysis device, comprising:
[0032] The graph building module is used to build the initial lineage graph of data in the data lake;
[0033] The lineage establishment module is used to establish the full-link data lineage relationship corresponding to the data table at the data source end based on the structure of the data table at the data source end and the target data; wherein, the target data includes test data and / or feature data;
[0034] The correction module is used to correct the initial kinship map based on the full-link data kinship relationship to obtain the target kinship map.
[0035] Thirdly, to achieve the above objectives, embodiments of this application provide a data lineage analysis device, including a transceiver, a processor, a memory, and a program stored in the memory and executable on the processor; the transceiver is used to send and receive data under the control of the processor, and the processor executes the program to implement the data lineage analysis method as described in the first aspect.
[0036] Fourthly, to achieve the above objectives, embodiments of this application provide a readable storage medium having a program or instructions stored thereon, which, when executed by a processor, implement the data lineage analysis method as described in the first aspect.
[0037] Fifthly, to achieve the above objectives, embodiments of this application provide a computer program product including computer instructions that, when executed by a processor, implement the data lineage analysis method as described in the first aspect.
[0038] The beneficial effects of the above technical solution in this application are as follows:
[0039] In this embodiment, firstly, an initial lineage graph of data in the data lake is established; secondly, based on the structure of the data tables at the data source end and the target data, a full-link data lineage graph corresponding to the data tables at the data source end is established; wherein, the target data includes test data and / or feature data; thus, the lineage graph of data in the entire data lake (across databases) is established; finally, based on the full-link data lineage graph, the initial lineage graph is corrected to obtain the target lineage graph. In this way, the lineage flow of data in the entire data lake can be tracked, and the modeling of data lineage graphs across databases and learning platforms can be realized, thereby improving the accuracy of data lineage identification. Attached Figure Description
[0040] Figure 1 This is one of the flowcharts illustrating the data lineage analysis method according to an embodiment of this application;
[0041] Figure 2 This is a schematic diagram of the SQL syntax parsing process according to an embodiment of this application;
[0042] Figure 3 This is a schematic diagram of the abstract syntax tree structure of the selection statement in an embodiment of this application;
[0043] Figure 4 This is a schematic diagram of the abstract syntax tree structure of the update statement in an embodiment of this application;
[0044] Figure 5 This is a second schematic flowchart of the data lineage analysis method according to an embodiment of this application;
[0045] Figure 6This is a schematic diagram of the data lineage analysis device according to an embodiment of this application;
[0046] Figure 7 This is a schematic diagram of the data lineage analysis device according to an embodiment of this application. Detailed Implementation
[0047] To make the technical problems, technical solutions and advantages of this application clearer, a detailed description will be provided below in conjunction with the accompanying drawings and specific embodiments.
[0048] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.
[0049] In the various embodiments of this application, it should be understood that the sequence number of each process described below does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0050] In addition, the terms "system" and "network" are often used interchangeably in this article.
[0051] In the embodiments provided in this application, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined based on A. However, it should also be understood that determining B based on A does not mean that B is determined solely based on A, but can also be determined based on A and / or other information.
[0052] Before describing the embodiments of this application, the relevant technical points will first be described by way of example:
[0053] Definition of Data Lineage: Data lineage refers to the natural relationships, similar to human kinship, that form between data during the processes of data generation, extraction, transformation, loading (ETL), processing, fusion, flow, and eventual disposal. Simply put, it's the upstream and downstream source-to-destination relationship between data: where the data comes from and where it goes. In practical data application, clarifying data lineage from a data perspective means understanding the database and tables storing the data, the corresponding fields, and their attributes; clarifying data lineage from a business perspective means understanding the business line to which the data belongs, the logic behind the data's generation, the logic behind its use, and the relationships between business lines.
[0054] In view of this, embodiments of this application provide a data lineage analysis method, such as... Figure 1 As shown, the method includes:
[0055] Step 101: Establish an initial lineage graph of data in the data lake; for example, this step can establish the initial lineage graph based on the scripts of the databases related to the data lake and their corresponding execution logs.
[0056] Step 102: Based on the structure of the data table at the data source end and the target data, establish the full-link data lineage relationship corresponding to the data table at the data source end; wherein, the target data includes test data and / or feature data.
[0057] For example, end-to-end data lineage refers to cross-platform data lineage, that is, the data lineage that exists throughout the entire process of data flowing from the source to the target across all platforms. Specifically, it refers to the mapping relationship between the source table, the data tables on the platforms through which the data flows, and the data lake tables.
[0058] Step 103: Based on the full-link data lineage, correct the initial lineage map to obtain the target lineage map.
[0059] For example, the correction method of step 103 above may be to add non-existent data lineage relationships to the initial lineage relationship map, or to update the data lineage relationships that have changed.
[0060] In this embodiment, firstly, an initial lineage graph of data in the data lake is established; secondly, based on the structure of the data tables at the data source end and the target data, a full-link data lineage graph corresponding to the data tables at the data source end is established; wherein, the target data includes test data and / or feature data; thus, the lineage graph of data in the entire data lake (across databases) is established; finally, based on the full-link data lineage graph, the initial lineage graph is corrected to obtain the target lineage graph. In this way, the lineage flow of data in the entire data lake can be tracked, and the modeling of data lineage graphs across databases and learning platforms can be realized, thereby improving the accuracy of data lineage identification.
[0061] As an optional implementation, step 101 involves establishing an initial lineage graph of the data in the data lake, including:
[0062] Step A1: Obtain the database script used for data processing in the data lake and the corresponding execution log of the database script; for example, the database script used for data processing is a Structured Query Language (SQL) database script, and correspondingly, the execution log is an SQL execution log. Of course, the database script in this embodiment is not limited to an SQL database script, and can also be other types of database scripts.
[0063] Step A2: Based on the execution order and context information in the execution log, establish a time-dimensional dependency chain TimeL in chronological order; for example, the structure of each chain node in the time-dimensional dependency chain TimeL is: data entity (data table name, field name) - execution time - subsequent node.
[0064] Step A3: Based on the time-dimensional dependency chain, traverse the tree structure of the database script to obtain the information corresponding to each node in the tree structure. The information corresponding to each node includes at least one of the following: data table, field, and condition. For example, taking the database script as an SQL script, the tree structure is, for example, an Abstract Syntax Tree (AST).
[0065] It should be noted here that, prior to step A3 above, the method further includes:
[0066] The database script is parsed to abstract its syntax structure, and the syntax structure is represented by a tree structure. The tree structure of the database script is the syntax structure represented by the tree structure.
[0067] For example, taking SQL scripts as the database script, the above steps are implemented as follows: The SQL scripts used for data processing in the data lake are parsed using the SQL syntax parser JSqlParser, the SQL syntax structure is abstracted, the SQL is represented in a structured tree format, and an Abstract Syntax Tree (AST) is generated. Among these, such as... Figure 2 As shown, the above implementation process specifically includes: First, lexical parsing of the read SQL script to obtain a lexical unit stream (token stream). For large SQL files, block reading and streaming parsing are used to avoid memory overflow during single loading. Second, syntax parsing of the token stream to obtain an Abstract Syntax Tree (AST). For example, the JSqlParser tool can be used to convert the SQL string into an AST. The token stream can include keywords such as select, update, insert from, and condition / filter (where).
[0068] Step A4: Establish the initial bloodline relationship map based on the information corresponding to each node.
[0069] For example, step A4 above can utilize graph database technology to establish the initial lineage graph based on the information corresponding to each node. Specifically, for example, the structured storage of nodes (data entities) and edges (processes) can be used to realize path retrieval and influence range calculation, represented as Vec(data entity(database name, table name, field name), data lineage), and stored as a lineage graph DL-Field-G. Examples of relation type dictionaries related to step A4 above are shown in Table 1 below:
[0070] Table 1
[0071] Relationship type Semantics BELONGS_TO (belonging relationship) Field → Table Attribution Relationship LINEAGE (blood relation) Field-level lineage dependency DERIVED_FROM (derived relation) Derived calculation relationships, such as MAX (maximum), MIN (minimum), / SUM (summation), / GROUP BY (aggregation), etc.
[0072] As a specific implementation, taking SQL script as the database script as an example, step A3 above involves traversing the tree structure of the database script based on the time-dimensional dependency chain to obtain the information corresponding to each node in the tree structure, including:
[0073] For example Figure 3 The AST of the SELECT statement is shown. Traversing the AST of the SELECT statement, we extract the related tables and join conditions from the nodes containing JOIN and UNION. JOIN is typically a node in the query tree, representing a join operation between multiple data sources. UNION is used to merge the result sets of two or more SELECT statements into a longer result set.
[0074] against Figure 4 The AST of the update statement is shown. Traverse the AST of the update statement and select the target table, field 1 of the target table under the SET node and field 2 under the WHERE node.
[0075] The method for selecting information corresponding to each node in the AST of an INSERT statement is similar to that of an UPDATE statement, and will not be described further here. Furthermore, the method for selecting information corresponding to each node in the AST of other statements can be configured based on the actual situation.
[0076] As a specific implementation, step A4 above, which establishes the initial kinship graph based on the information corresponding to each node, includes:
[0077] Step A41: Based on the associated tables and join conditions corresponding to the merge nodes in the tree structure of the selection statement, generate a table-level data lineage set. Here, the merge node refers to... Figure 3For example, in the JOIN / UNION nodes, this step is as follows: Based on the association tables and connection conditions extracted from the JOIN and UNION nodes, generate a table-level data lineage set DL-Set, initially establish a network of data lineage relationships, and record the lineage relationship types such as JOIN and UNION.
[0078] Step A42: Based on the fields corresponding to the column expression nodes of the tree structure of the selection statement, the associated tables in the merge nodes, and / or the aggregation fields corresponding to the grouping nodes, a first field-level lineage set is generated on the basis of the table-level data lineage set; wherein, the column expression node refers to... Figure 3 In the query (MAX / MIN, etc.) node, the grouping node refers to... Figure 3 The GroupBy node in the data hierarchy; for example, this step is as follows: Obtain the function call node fields (assuming they are field 1) such as MAX / MIN / SUM from the Abstract Syntax Tree (AST) of the SELECT statement, and establish derived calculation relationships with the fields of the Join table (if the fields cannot be obtained, establish table-level lineage relationships); for cases with GROUP BY nodes, it is also necessary to establish the aggregation dependency lineage relationship between field 1 and the aggregation field. A field-level lineage relationship set DL-Field is then generated based on the data lineage relationship set DL-Set, recording the lineage relationship types MAX / MIN / SUM / GROUP BY, etc. The field-level lineage relationship set DL-Field generated based on the data lineage relationship set DL-Set is the first field-level lineage relationship set.
[0079] Step A43: Based on the target table, the fields corresponding to the set nodes, and the fields corresponding to the filter condition nodes in the tree structure of the insert or update statement, generate a second field-level lineage relationship set; for example, this step is: select the target table of INSERT / UPDATE in the abstract syntax tree AST, and establish the field lineage relationship between field 1 of the target table under the SET key node and field 2 under the Where condition node.
[0080] Step A44: Associate the second field-level bloodline relationship set with the first field-level bloodline relationship set to obtain the initial bloodline relationship map.
[0081] As an optional implementation, the target data includes test data. Based on this, step 102 establishes the full-link data lineage relationship corresponding to the data table according to the structure of the data table at the data source end and the target data, including:
[0082] Step B1: Select N empty data tables from the data source and insert at least one target data record into each of the N empty data tables to obtain N data tables; N is a positive integer.
[0083] It should be noted that this implementation is performed in a data lake test environment. Specifically, to correct the initial lineage graph, a data lake test environment can be set up, with the database name, data structure, etc., fully replicated, and the data tables empty. Test data is then inserted into these empty tables in the test environment, thereby identifying the full-link data lineage of the data source data in the data lake based on the processing of the test data. Step B1 above is used to insert test data into the empty data tables.
[0084] Step B2: In the data lake test environment, the data in the N data tables are processed to determine the target data table of the data platform and the target data table of the data lake. The data platform is the platform through which the data table from the data source flows, and the fields in the target data table include data related to the target data.
[0085] For example, the processing of test data in step B2 above can be a routine processing in a data lake, such as data extraction, data cleaning, data transformation, data loading, data layering and modeling, real-time and batch processing, etc.
[0086] Step B3: Establish the end-to-end data lineage based on the data table of the data source, the target data table of the data platform, and the target data table of the data lake.
[0087] Among the above-mentioned optional implementation methods, by changing the data of the source system, the tables and fields of each platform affected in the entire data processing flow can be tracked, and the cross-platform data lineage can be obtained, that is, the mapping relationship between the source table, the data table of the platform through which the data flows, and the data lake data table, which solves the problem of low accuracy in identifying cross-database data lineage in the current data lake.
[0088] It should be noted that, in the above optional implementation methods, the number of target data inserted into each empty data table can be the same or different for different functions. The implementation process of the above optional implementation methods will be explained below using the COUNT function, SUM function and MAX / MIN function as examples.
[0089] For COUNT function aggregation: First, test data is sequentially inserted into the N empty data tables on the data source side of the data lake. In the i-th empty data table, i rows of data are inserted, where i is a positive integer less than or equal to N. For example, 1 row of test data is inserted into the first empty data table, 2 rows into the second, n rows into the nth, and so on, until all N empty data tables on the source side have been filled with test data. Numerical fields in each test row do not contain positive integers; for example, they can be initialized to 0. Second, the data in the N tables after insertion is processed. In the data lake testing environment, the field values of the target tables (1, 2, ..., positive integers) are identified to establish a mapping chain between the source tables and the data lake tables based on the identification results.
[0090] For SUM function aggregation: First, test data is sequentially inserted into N empty data tables at the data source end of the data lake. One test data record is inserted into each empty table, and all numeric fields in the test data are set to 1. Second, the N data tables after data insertion are processed, and a snapshot of the data, SumData1, is stored. Third, another test data record is inserted into each of the N data tables. The numeric field values of the inserted test data are determined by the algorithm: ID + 3-digit field number; for example, if the data table ID is 100000 and the field number is 1, then data 100000001 is inserted for that field, and data 100000002 is inserted for the field with the field number 2. Afterward, the data tables after inserting the test data are processed to locate which data tables on each platform contain this data in the entire data processing flow, thus establishing a mapping chain from the source table to the data tables on the platforms through which the data flows to the data lake tables.
[0091] For the MAX / MIN functions: First, number the N empty data tables in the data lake sequentially, for example, with a 6-digit ID format. For each empty data table, insert 5 test data records. The specific algorithm for inserting data into numeric fields is as follows: ID + 3-digit field number + last 1, ID + 3-digit field number + last 2, ID + 3-digit field number + last 3, ID + 3-digit field number + last 4, ID + 3-digit field number + last 5. For example: Suppose the data table ID is 100000, and the data inserted for field number 1 is: 1000000011, 1000000012, 1000000013, 1000000014, 1000000015; the data inserted for field number 2 is: 1000000021, 1000000022, 1000000023, 1000000024, 1000000025. Then, by locating the database name, table name, and field name of the target table's maximum and minimum values in the data lake test environment after data processing, a field-level lineage relationship between the source table fields and the target table fields is established.
[0092] Based on the above optional implementation methods, step 103, according to the full-link data lineage relationship, corrects the initial lineage relationship map to obtain the target lineage relationship map, including:
[0093] For COUNT function aggregation tracing, in the initial lineage graph, a lineage relationship DERIVED_FROM is established between the source table and the target table fields, specifically the COUNT relationship;
[0094] For SUM function aggregation tracing, in the initial lineage graph, a field-level derived computation relationship DERIVED_FROM is established between the source table and the target table, specifically the SUM relationship. For example, if the value of field1 in the target table Table1 is detected as (100000002), then a field data lineage relationship is established between (TableID100000, field1) and (Table1, field1) in the target table.
[0095] For MAX / MIN function tracing, in the initial lineage graph, a field-level derived calculation relationship DERIVED_FROM lineage relationship is established between the source table field and the target table field, specifically the MAX / MIN relationship.
[0096] As another optional implementation, the target data includes feature data. Based on this, step 102 establishes the full-link data lineage relationship corresponding to the data table at the data source end according to the structure of the data table at the data source end and the target data, including:
[0097] Step C1: Using a feature data generation algorithm, feature data is generated for each field of the data table at the data source end, resulting in a feature dataset TZ-Set. For example, the feature data is generated as follows: Feature data = Feature sub-data + Non-repeating randomly generated data. For instance, a 7-digit non-repeating random number can be used. Based on this, the feature sub-data is a specific character defined by the system, such as: Feature data TZData = 01 + Radom (7), TE-Set = {TZData1, TZData2, ..., TZDatan}.
[0098] Step C2: For the data table at the data source end, execute the database script for data processing in the data lake to determine the database, data table, and fields in the data lake that include the feature data.
[0099] It should be noted that the databases, tables, and fields that include these feature data are the same as the databases, tables, and fields that generate these feature data. In other words, the data tables that generate these feature data are related to the databases, tables, and fields that include these feature data.
[0100] Step C3: Establish the full-link data lineage relationship based on at least one of the database, data table, and fields including the feature data and the data table at the data source end.
[0101] In other words, the above-mentioned optional implementation methods are used to establish a field-level lineage tracing path from the source system to the target system. Specifically, by inserting feature data into the source system, the system tracks the tables containing the feature fields in each platform throughout the entire data processing flow to obtain the cross-platform data lineage, that is, the mapping relationship between the source table fields, the data table fields of the platforms through which the data flows, and the data lake table fields.
[0102] Based on the above-mentioned optional implementation methods, as an optional implementation method, step 103, based on the full-link data lineage relationship, corrects the initial lineage relationship map to obtain the target lineage relationship map, including at least one of the following:
[0103] If no database or table relationship exists in the initial lineage graph, the tables not present in the initial lineage graph are sorted according to the time-dimensional dependency chain. A time-dimensional dependency chain is established with the data table at the data source as the head, and the initial lineage graph is updated according to the time-dimensional dependency chain to obtain the target lineage graph. The time-dimensional dependency chain is established based on the execution order and context information in the execution log corresponding to the database script.
[0104] If the initial lineage graph contains lineage relationships between databases and tables in the full-link data lineage relationship, but does not contain field-level lineage relationships in the full-link data lineage relationship, the initial lineage graph is updated by establishing field-level lineage relationship dependencies to obtain the target lineage graph.
[0105] In short, after locating the database, table, and field stored in the dataset TZ-V within the data lake, if the database and table lineages have been established in the lineage graph DL-Field-G, but field-level lineages have not, then a field-level lineage dependency LINEAGE is added. If the database and table are not found in the lineage graph DL-Field-G, then the head table is used as the source table, and the time-based dependency chain TimeL is used to arrange the order of the tables, thus establishing a time-based lineage chain TDLLian.
[0106] Furthermore, as an optional implementation, the method also includes:
[0107] The target lineage graph is updated using the detected task change events. For example, this step can employ an incremental update mechanism to monitor task change events; when a data change is detected, only the affected lineage relationships are updated, rather than a full refresh.
[0108] For example, a distributed probe (Agent) can be used to collect schema changes and data operation logs of storage layers such as Hadoop / S3 in real time, parse the changes in SQL scripts, and update the data lineage graph incrementally based on the changes.
[0109] like Figure 5 As shown, the data lineage analysis method in this application generally includes two aspects: constructing initial lineage relationships and optimizing the initial lineage relationships. Specifically, for constructing the initial lineage relationships, enhanced analysis of the runtime logs can be performed to obtain time-dimensional dependency chains. Then, the database (e.g., SQL) syntax tree is parsed, and based on the parsing results, graph database technology is used to construct the initial lineage relationship graph. For optimizing the initial lineage relationships, the entire data lineage relationship of the data table can be determined through two methods: feature data tracking and data change analysis. The initial lineage relationships are then optimized based on the linked data lineage relationships.
[0110] In the embodiments described above, based on SQL execution syntax data analysis and combined with log analysis of SQL execution timing, a data lineage graph (DL-Field-G) and a temporal lineage chain (TDLLian) can be automatically generated. Furthermore, through unique feature data tracking and data change tracking methods, the data lineage graph is optimized. This addresses three key issues: first, it solves the problems of disordered accumulation of raw data, large data volume, and inaccurate data lineage identification in data lakes; second, it solves the problem of high maintenance costs due to reliance on manual annotation for data lineage tracking; and third, it enables data lineage identification across data lakes (and databases). Thus, low-cost, fast, and accurate data lineage identification is achieved.
[0111] Embodiments of this application also provide a data lineage analysis device, such as... Figure 6 The device includes:
[0112] The graph creation module 601 is used to create an initial lineage graph of data in the data lake.
[0113] The lineage establishment module 602 is used to establish the full-link data lineage relationship corresponding to the data table at the data source end based on the structure of the data table at the data source end and the target data; wherein, the target data includes test data and / or feature data;
[0114] The correction module 603 is used to correct the initial kinship map based on the full-link data kinship relationship to obtain the target kinship map.
[0115] The map building module 601 includes:
[0116] The acquisition submodule is used to acquire the database scripts used for data processing in the data lake and the execution logs corresponding to the database scripts;
[0117] The first submodule is used to establish a time-dimensional dependency chain according to the execution order and context information in the execution log;
[0118] The traversal submodule is used to traverse the tree structure of the database script according to the time dimension dependency chain, and obtain the information corresponding to each node in the tree structure, wherein the information corresponding to each node includes at least one of data table, field and condition;
[0119] The initial bloodline relationship map is established based on the information corresponding to each node.
[0120] The first establishment submodule includes:
[0121] The first generation unit is used to generate a table-level data lineage set based on the association table and join conditions corresponding to the merge node in the tree structure of the selection statement.
[0122] The second generation unit is used to generate a first field-level lineage relationship set based on the table-level data lineage relationship set, according to the fields corresponding to the column expression nodes of the tree structure of the selection statement, the associated tables in the merge nodes, and / or the aggregate fields corresponding to the grouping nodes.
[0123] The third generation unit is used to generate a second field-level lineage relationship set based on the target table, the fields corresponding to the setting nodes, and the fields corresponding to the filtering condition nodes in the tree structure of the insert or update statement.
[0124] A unit is established to associate the second field-level bloodline relationship set with the first field-level bloodline relationship set to obtain the initial bloodline relationship map.
[0125] The target data includes test data, and based on this, the blood relationship establishment module 602 includes:
[0126] The selection submodule is used to select N empty data tables from the data source and insert i target data records into the i-th data table of the N empty data tables to obtain N data tables; N is a positive integer, and i is a positive integer less than or equal to N;
[0127] The first determination submodule is used to process the data in the N data tables in the data lake test environment to determine the target data table of the data platform and the target data table of the data lake. The data platform is the platform through which the data table of the data source flows, and the fields in the target data table include data related to the target data.
[0128] The second submodule is used to establish the end-to-end data lineage relationship based on the data table of the data source, the target data table of the data platform, and the target data table of the data lake.
[0129] The target data includes feature data, and based on this, the blood relationship establishment module 602 includes:
[0130] The generation submodule is used to generate feature data for each field of the data table at the data source end using a feature data generation algorithm;
[0131] The second determining submodule is used to execute the database script for data processing in the data lake for the data table at the data source end, and to determine the database, data table and fields in the data lake that include the feature data;
[0132] The third submodule is used to establish the full-link data lineage relationship based on at least one of the database, data table and field including the feature data and the data table of the data source.
[0133] The correction module 603 includes at least one of the following:
[0134] The sorting submodule is used to sort the data tables that are not present in the initial lineage graph, based on the time-dimensional dependency chain, when no lineage relationship exists between the databases and data tables in the full-link data lineage graph. This establishes a time-dimensional lineage chain with the data tables at the data source end as the head, and updates the initial lineage graph according to the time-dimensional lineage chain to obtain the target lineage graph. The time-dimensional dependency chain is established based on the execution order and context information in the execution log corresponding to the database script.
[0135] The update submodule is used to update the initial lineage graph by establishing field-level lineage dependency when the lineage relationship between the database and data table in the full-link data lineage relationship exists in the initial lineage graph, but the field-level lineage relationship in the full-link data lineage relationship does not exist, in order to obtain the target lineage graph.
[0136] The device further includes:
[0137] The update module is used to update the target bloodline graph by utilizing the monitored task change events.
[0138] It should be noted that the data lineage analysis device provided in this application embodiment can implement all the method steps implemented in the above data lineage analysis method embodiment and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.
[0139] Embodiments of this application also provide a data lineage analysis device, such as... Figure 7 As shown, it includes a transceiver 710, a processor 700, a memory 720, and a program or instructions stored in the memory 720 and executable on the processor 700; when the processor 700 executes the program or instructions, it implements the above-mentioned data lineage analysis method.
[0140] The transceiver 710 is used to receive and send data under the control of the processor 700.
[0141] Among them, Figure 7In this context, the bus architecture can include any number of interconnected buses and bridges, specifically linking various circuits of one or more processors represented by processor 700 and memory represented by memory 720 together. The bus architecture can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 710 can be multiple elements, including a transmitter and a receiver, providing a unit for communicating with various other devices over a transmission medium.
[0142] The processor 700 is responsible for managing the bus architecture and general processing, while the memory 720 can store the data used by the processor 700 during operation.
[0143] An embodiment of this application provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the steps in the data lineage analysis described above and achieve the same technical effect. To avoid repetition, further details are omitted here.
[0144] The processor mentioned above is the processor in the data lineage analysis device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0145] In this embodiment, the module can be implemented in software so that it can be executed by various types of processors. For example, an identified executable code module may include one or more physical or logical blocks of computer instructions, which may be constructed as objects, procedures, or functions. Nevertheless, the executable code of the identified module does not need to be physically located together, but may include different instructions stored in different bits, which, when logically combined, constitute the module and achieve the module's intended purpose.
[0146] Through the above description of the implementation methods, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, disk, optical disk) and includes several instructions for executing the methods described in the various embodiments of this application.
[0147] Therefore, embodiments of this application also provide a computer program product, including computer instructions, which, when executed by a processor, implement the data lineage analysis method described above and achieve the same technical effect. To avoid repetition, further details are omitted here.
[0148] In this embodiment, the module can be implemented in software so that it can be executed by various types of processors. For example, an identified executable code module may include one or more physical or logical blocks of computer instructions, which may be constructed as objects, procedures, or functions. Nevertheless, the executable code of the identified module does not need to be physically located together, but may include different instructions stored in different bits, which, when logically combined, constitute the module and achieve the module's intended purpose.
[0149] In practice, an executable code module can be a single instruction or many instructions, and can even be distributed across multiple different code segments, different programs, and across multiple memory devices. Similarly, operational data can be identified within the module and can be implemented in any suitable form and organized within any suitable data structure. This operational data can be collected as a single dataset or distributed across different locations (including different storage devices), and can exist, at least in part, solely as electronic signals within the system or network.
[0150] When a module can be implemented using software, considering the current level of hardware technology, modules that can be implemented in software can be implemented using hardware circuits by those skilled in the art to achieve the corresponding functions, without considering cost. These hardware circuits include conventional very-large-scale integrated circuits (VLSI) or gate arrays, as well as existing semiconductors such as logic chips and transistors, or other discrete components. Modules can also be implemented using programmable hardware devices, such as field-programmable gate arrays, programmable array logic, and programmable logic devices.
[0151] The exemplary embodiments described above are illustrated with reference to the accompanying drawings. Many different forms and embodiments are feasible without departing from the spirit and teachings of this application, and therefore, this application should not be construed as limiting the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided to make this application complete and convey the scope of this application to those skilled in the art. In these drawings, component dimensions and relative dimensions may be exaggerated for clarity. The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. As used herein, unless explicitly stated otherwise, the singular forms “a,” “an,” and “the” are intended to include all such forms. It will be further understood that the terms “comprising” and / or “including”, when used in this specification, indicate the presence of the stated features, integers, steps, operations, components, and / or elements, but do not exclude the presence or addition of one or more other features, integers, steps, operations, components, elements, and / or groups thereof. Unless otherwise indicated, a range of values is stated to include the upper and lower limits of the range and any subranges therebetween.
[0152] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles described in this application, and these improvements and modifications should also be considered within the scope of protection of this application.
Claims
1. A data lineage analysis method, characterized in that, include: Establish an initial lineage graph of data in the data lake; Based on the structure of the data table at the data source end and the target data, establish the full-link data lineage relationship corresponding to the data table at the data source end; wherein, the target data includes test data and / or feature data; Based on the full-link data lineage, the initial lineage map is corrected to obtain the target lineage map.
2. The method according to claim 1, characterized in that, Establish an initial lineage graph of the data in the data lake, including: Obtain the database scripts used for data processing in the data lake and the execution logs corresponding to the database scripts; Based on the execution order and context information in the execution log, establish a time-dimensional dependency chain in chronological order; Based on the time-dimensional dependency chain, the tree structure of the database script is traversed to obtain the information corresponding to each node in the tree structure, wherein the information corresponding to each node includes at least one of the following: data table, field, and condition. The initial bloodline relationship map is established based on the information corresponding to each node.
3. The method according to claim 2, characterized in that, Based on the information corresponding to each node, the initial kinship map is established, including: Based on the associated tables and join conditions corresponding to the merge nodes in the tree structure of the selection statement, generate a set of table-level data lineage relationships; Based on the fields corresponding to the column expression nodes of the tree structure of the selection statement, the associated tables in the merge nodes, and / or the aggregation fields corresponding to the grouping nodes, a first field-level lineage set is generated on the basis of the table-level data lineage set; Based on the target table, the fields corresponding to the setting nodes, and the fields corresponding to the filtering condition nodes in the tree structure of the insert or update statements, generate a second-level lineage relationship set. The second field-level bloodline relationship set is associated with the first field-level bloodline relationship set to obtain the initial bloodline relationship map.
4. The method according to claim 1, characterized in that, The target data includes test data. Based on the structure of the data table at the data source end and the target data, a full-link data lineage relationship corresponding to the data table is established, including: Select N empty data tables from the data source and insert at least one target data record into each of the N empty data tables to obtain N data tables; N is a positive integer. In the data lake test environment, the data in the N data tables are processed to determine the target data table of the data platform and the target data table of the data lake. The data platform is the platform through which the data table from the data source flows, and the fields in the target data table include data related to the target data. The end-to-end data lineage is established based on the data table at the data source, the target data table at the data platform, and the target data table at the data lake.
5. The method according to claim 1, characterized in that, The target data includes feature data. Based on the structure of the data table at the data source end and the target data, a full-link data lineage relationship is established corresponding to the data table at the data source end, including: Feature data is generated for each field of the data table at the data source using a feature data generation algorithm; For the data tables at the data source end, execute the database scripts used for data processing in the data lake to determine the databases, data tables, and fields in the data lake that include the feature data; The end-to-end data lineage is established based on at least one of the database, data table, and fields including the feature data and the data table at the data source end.
6. The method according to claim 5, characterized in that, Based on the full-link data lineage, the initial lineage map is corrected to obtain the target lineage map, including at least one of the following: If no database or table relationship exists in the initial lineage graph, the tables not present in the initial lineage graph are sorted according to the time-dimensional dependency chain. A time-dimensional dependency chain is established with the data table at the data source as the head, and the initial lineage graph is updated according to the time-dimensional dependency chain to obtain the target lineage graph. The time-dimensional dependency chain is established based on the execution order and context information in the execution log corresponding to the database script. If the initial lineage graph contains lineage relationships between databases and tables in the full-link data lineage relationship, but does not contain field-level lineage relationships in the full-link data lineage relationship, the initial lineage graph is updated by establishing field-level lineage relationship dependencies to obtain the target lineage graph.
7. The method according to claim 1, characterized in that, The method further includes: The target lineage graph is updated using the monitored task change events.
8. A data lineage analysis device, characterized in that, include: The graph building module is used to build the initial lineage graph of data in the data lake; The lineage establishment module is used to establish the full-link data lineage relationship corresponding to the data table at the data source end based on the structure of the data table at the data source end and the target data; wherein, the target data includes test data and / or feature data; The correction module is used to correct the initial kinship map based on the full-link data kinship relationship to obtain the target kinship map.
9. A data lineage analysis device, comprising a transceiver, a processor, a memory, and a program stored in the memory and executable on the processor; characterized in that, The transceiver is used to send and receive data under the control of the processor, and the processor implements the data lineage analysis method as described in any one of claims 1 to 7 when executing the program.
10. A readable storage medium having a program or instructions stored thereon, characterized in that, When the program or instructions are executed by the processor, they implement the data lineage analysis method as described in any one of claims 1 to 7.