Data lineage analysis method, device, electronic device and computer-readable storage medium

Abstract

The present application relates to a data lineage parsing method, system, computer device and storage medium. The method comprises: receiving a data lineage parsing instruction, extracting the SQL to be parsed from the data lineage parsing instruction; parsing the SQL to be parsed to generate an abstract syntax tree; traversing the nodes of the abstract syntax tree in a depth-first post-order manner, identifying the node type of each node, performing a breadth traversal parsing on the node based on the node type, and generating a lineage relationship tree; encapsulating the lineage relationship tree into field-level lineage data, and storing the field-level lineage data. Implementing the embodiments of the present application has the effect of improving the efficiency of data lineage tracing.

Description

Technical Field

[0001] The present application relates to the field of big data processing technology, and in particular to a data lineage analysis method, device, electronic device and computer-readable storage medium. Background Art

[0002] With the development of computer technology, the world has officially entered the era of big data. Big data, also known as massive data, is a data set that is so large that it far exceeds the capabilities of traditional database software tools in terms of acquisition, storage, management, and analysis. It has four major characteristics: massive data scale, fast data flow, diverse data types, and low value density.

[0003] The value of big data lies in data analysis and data mining and intelligent decision-making based on the analysis. By analyzing the data, we can discover changes in the data and the correlation between the data, explore the patterns that have been ignored in the past, obtain insights and new value, and predict human behavior, so that we can make targeted business decisions.

[0004] In order to improve the accuracy of data analysis and troubleshoot data problems, clear data lineage is required. Data lineage can clearly show the source of data, data processing method, mapping method and data export. Currently, data lineage is mainly generated through Atlas. However, in the process of Atlas generating data lineage, all lineage relationships will be retained. When the amount of data is huge, a large number of temporary tables will be generated, occupying too much storage resources. In addition, temporary tables will cause the expansion of invalid lineage relationships, resulting in low efficiency in tracing valid lineage. Summary of the invention

[0005] The present invention provides a data lineage analysis method, device, electronic device and computer-readable storage medium for

[0006] In a first aspect, an embodiment of the present application provides a data lineage analysis method, comprising:

[0007] Receive a data lineage parsing instruction, and extract the SQL to be parsed from the data lineage parsing instruction;

[0008] Parse the SQL to be parsed to generate an abstract syntax tree;

[0009] Depth-first post-order traversal of the nodes of the abstract syntax tree, identifying the node type of each node, performing breadth-first traversal parsing on the node based on the node type, and generating a kinship tree;

[0010] The blood relationship tree is encapsulated into field-level blood relationship data, and the field-level blood relationship data is stored.

[0011] In one of the embodiments, identifying the node type of the node includes: determining in sequence whether the node type of the node is TOK_TABREF, TOK_INSERT, TOK_SUBQUERY, TOK_CREATETABLE, or TOK_QUERY.

[0012] In one of the embodiments, the width traversal parsing of the node based on the node type includes: if the node type is the TOK_INSERT, determining the array of fields to be inserted based on the SQL to be parsed; determining the table to be inserted corresponding to the array of fields to be inserted through the child node TOK_DESTINATION; if the table to be inserted is a temporary table, obtaining the temporary table type of the temporary table; if the temporary table is the physical source table, width traversal parsing the array of fields to be inserted, for each field to be inserted in the array of fields to be inserted, determining the source field array corresponding to the field to be inserted, traversing the source field array, adding each source field in the source field array to the columns collection of currentTable and the from collection corresponding to the field to be inserted, adding the field to be inserted to the currentColumns collection, and marking the currentColumns collection as a state to be allocated.

[0013] In one of the embodiments, performing a width traversal analysis on the node based on the node type also includes: if the temporary table is a temporary query table of a subquery, performing field source analysis on the array of fields to be inserted, and determining a source field array corresponding to each field to be inserted in the array of fields to be inserted; for each field to be inserted, traversing and adding the source field array to the from collection corresponding to the field to be inserted, adding the field to be inserted to the currentColumns collection, and marking the currentColumns collection as being in a state to be allocated.

[0014] In one of the embodiments, performing a width traversal analysis on the node based on the node type also includes: if the table to be inserted is a target table, obtaining the library name and table name of the target table; creating a result table targetTable, performing field source analysis on the field array to be inserted, and determining the source field array corresponding to each field to be inserted in the field array to be inserted; for each field to be inserted, traversing and adding the source field array corresponding to the field to be inserted to the from collection corresponding to the field to be inserted, and adding the field to be inserted to the columns collection of the result table targetTable.

[0015] In one of the embodiments, performing field source resolution on the array of fields to be inserted includes: for each field to be inserted in the array of fields to be inserted, obtaining the field alias and field type corresponding to the field to be inserted; if the field type is a preset first type, determining the field subtree node to be inserted corresponding to the field to be inserted based on the subquery name, and querying the field subtree node to be inserted based on a preset mapping relationship to obtain the source field array; if the field type is a preset second type, creating a constant name field based on the field to be inserted; if the field type is a preset third type, width traversing the field subtree to be inserted to determine whether the field subtree to be inserted has a table alias, and if so, querying based on the table alias and the field alias to obtain the source field array.

[0016] In one of the embodiments, the determining whether the field subtree to be inserted has a table alias further includes: if the field subtree to be inserted does not have a table alias, obtaining the field name of the field to be inserted, and searching for the source field array in the columns set of the upper-level subquery or the corresponding physical source table corresponding to the field to be inserted based on the field name; if the source field array exists in the column set, returning the source field array; if the source field array does not exist in the column set, obtaining metadata of the physical source table corresponding to the field to be inserted, extracting the physical table field information corresponding to the metadata, and determining the source field array based on the physical table field information.

[0017] In a second aspect, an embodiment of the present application provides a data lineage analysis device, characterized in that it includes:

[0018] A receiving unit receives a data lineage parsing instruction and extracts a SQL statement to be parsed from the data lineage parsing instruction;

[0019] A parsing unit, parsing the SQL to be parsed to generate an abstract syntax tree;

[0020] A lineage generation unit, which traverses the nodes of the abstract syntax tree in a depth-first post-order manner, identifies the node type of each node, performs a breadth-based traversal analysis on the node based on the node type, and generates a lineage relationship tree;

[0021] The encapsulation unit encapsulates the blood relationship tree into field-level blood relationship data and stores the field-level blood relationship data.

[0022] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, and the program comprises instructions for executing some or all of the steps described in the method described in the first aspect of the embodiment of the present application.

[0023] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, wherein the computer program is executed by a processor to implement part or all of the steps described in the method described in the first aspect of the embodiment of the present application.

[0024] It can be seen that in the embodiment of the present application, a data lineage parsing instruction is received, and the SQL to be parsed is extracted from the data lineage parsing instruction; the SQL to be parsed is parsed to generate an abstract syntax tree; the nodes of the abstract syntax tree are traversed in depth-first post-order, and for each of the nodes, the node type of the node is identified, and a width traversal parsing is performed on the node based on the node type to generate a lineage tree; the lineage tree is encapsulated as field-level lineage data, and the field-level lineage data is stored. In this way, the lineage relationship of the data can be inferred through a hybrid traversal method combining depth-first post-order traversal and width traversal, simplifying the collection and reasoning process of lineage relationships, reducing the generation of temporary tables, and reducing the generation and storage of invalid lineages, which is conducive to releasing storage resources and improving the efficiency of lineage tracing. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 It is a schematic diagram of the application environment of a data lineage analysis method provided in an embodiment of the present application.

[0027] Figure 2 It is a flow chart of a data lineage analysis method provided in an embodiment of the present application.

[0028] Figure 3 It is a flowchart of a TOK_INSERT node parsing method provided in an embodiment of the present application.

[0029] Figure 4 A schematic diagram of the structure of an electronic device 400 provided in an embodiment of the present application.

[0030] Figure 5 A schematic diagram of the structure of a data lineage analysis device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] The terms "first", "second", "third" and "fourth" etc. in the specification and claims of the present invention and the drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.

[0033] Reference to "embodiments" herein means that a particular feature, result, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0034] Below, some terms in this application are explained to facilitate understanding by those skilled in the art.

[0035] Electronic devices may include various handheld devices with wireless communication functions, vehicle-mounted devices, wearable devices (such as smart watches, smart bracelets, pedometers, etc.), computing devices or other processing devices connected to wireless modems, as well as various forms of user equipment (UE), mobile stations (MS), terminal devices, etc. For the convenience of description, the above-mentioned devices are collectively referred to as electronic devices.

[0036] The data lineage analysis method provided in this application can be applied to Figure 1 In the application environment shown.

[0037] like Figure 1As shown, the terminal 101 communicates with the electronic device 102 through the network, the electronic device 102 is connected to the database 103, the terminal 101 sends a data lineage parsing instruction to the electronic device 102 through the network, the electronic device 101 receives the data lineage parsing instruction, extracts the SQL to be parsed from the data lineage parsing instruction; parses the SQL to be parsed to generate an abstract syntax tree; traverses the nodes of the abstract syntax tree in depth-first post-order, identifies the node type of each node, performs breadth traversal parsing on the node based on the node type, and generates a lineage relationship tree; encapsulates the lineage relationship tree into field-level lineage data, the electronic device 102 constructs a mapping relationship between the field-level lineage data and the data stored in the database 103, and stores the field-level lineage data. When the electronic device 102 receives the lineage display instruction from the terminal 101, the field-level lineage data is extracted from the database 103 and displayed visually.

[0038] The terminal 101 and the electronic device 102 may be, but are not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices, and the database 103 may be implemented using an independent server or a server cluster consisting of multiple servers.

[0039] See also Figure 2 ,like Figure 2 As shown, a flow chart of a data lineage analysis method is provided, and the method is applied to Figure 1 The electronic device in the example is used to illustrate, and the following steps are included:

[0040] Step 201, receiving a data lineage parsing instruction, and extracting the SQL to be parsed from the data lineage parsing instruction;

[0041] Optionally, a data lineage analysis instruction is received, and the data lineage analysis instruction is obtained from an HTTP request sent to the terminal, or the terminal obtains the instruction by regularly querying a task information library. The method for obtaining the data lineage analysis instruction is not limited herein.

[0042] The data lineage analysis instructions may include: task sql, database connection address, user name, password, etc.

[0043] Optionally, extracting the SQL to be parsed from the data lineage parsing instruction includes: extracting task SQL from the data lineage parsing instruction, wherein the task SQL includes at least one SQL statement, obtaining a preset SQL filter, using the task SQL as input to the SQL filter, obtaining business SQL related to the data, cutting the business SQL according to semicolons to obtain an array of SQL to be parsed, the array of SQL to be parsed containing at least one SQL to be parsed, and performing SQL parsing operations on each SQL to be parsed.

[0044] Furthermore, after executing the SQL parsing operation for each SQL to be parsed, it also includes: parsing the SQL to be parsed, obtaining the result table targetTable corresponding to the SQL to be parsed, continuing to traverse the SQL array to be parsed based on the result table targetTable to perform SQL parsing, and obtaining the blood relationship corresponding to the SQL array to be parsed.

[0045] In this way, in the process of traversing and parsing the SQL array to be parsed, the SQL to be parsed can be parsed based on the result table targetTable generated after the parsing operation is completed, so that the blood relationship tree obtained after traversing and parsing the SQL array to be parsed has a contextual blood relationship effect, thereby improving the blood relationship tracing effect.

[0046] Step 202: parse the SQL to be parsed to generate an abstract syntax tree;

[0047] Optionally, start the hive tool and mobilize the HiveParser parser built into the hive tool to parse the SQL to be parsed into the abstract syntax tree.

[0048] Before mobilizing the HiveParser parser built into the hive tool to parse the SQL to be parsed into the abstract syntax tree, it also includes: obtaining the connection method corresponding to the SQL to be parsed, judging whether the connection method is a metadata connection, if so, the HiveParser parser enters the online parsing mode to generate the abstract syntax tree; if not, the HiveParser parser enters the offline parsing mode to generate the abstract syntax tree. Step 203, traverse the nodes of the abstract syntax tree in a depth-first post-order manner, identify the node type of each node, perform a breadth-based traversal analysis on the node based on the node type, and generate a kinship tree;

[0049] Optionally, the depth-first post-order traversal of the nodes of the abstract syntax tree includes: for the abstract syntax tree, first traversing the left subtree of the abstract syntax tree, then traversing the right subtree of the abstract syntax tree, and finally traversing the root node of the abstract syntax tree.

[0050] Optionally, identifying the node type of the node includes: determining in sequence whether the node type of the node is TOK_TABREF, TOK_INSERT, TOK_SUBQUERY, TOK_CREATETABLE, or TOK_QUERY.

[0051] Among them, the TOK_TABREF, the TOK_INSERT, the TOK_SUBQUERY, the TOK_CREATETABLE, and the TOK_QUERY correspond to five parsing processes respectively: parsing the physical source table, parsing the insert subtree, parsing the subquery subtree, parsing the table creation subtree, and parsing the query subtree. The node corresponds to and only corresponds to one node type. Optionally, a width traversal parsing is performed on the node based on the node type, including: first determining whether the node type of the node is TOK_TABREF; if the node type is TOK_TABREF, entering the physical source table parsing process, obtaining the table name and library name of the node; if the node does not have a library name, obtaining the initialized preset library name; determining whether the preset source table set tables contains the physical source table corresponding to the table name; if so, pointing currentTable to the physical source table; if not, creating a current table object based on the table name, adding the current table object to the preset source table set tables; determining whether the node has a node alias; if so, adding the tree node corresponding to the node alias and the currentTable mapping to aliasTables; if the node type is not TOK_TABREF, determining whether the node type is TOK_INSERT.

[0052] Optionally, if the node type is TOK_INSERT, the subtree parsing insertion process is entered, and a width traversal parsing is performed on the node based on the node type TOK_INSERT.

[0053] Among them, the performing width traversal and parsing of the node based on the node type includes: if the node type is the TOK_INSERT, determining the array of fields to be inserted based on the sql to be parsed; determining the table to be inserted corresponding to the array of fields to be inserted through the child node TOK_DESTINATION; if the table to be inserted is a temporary table, obtaining the temporary table type of the temporary table; if the temporary table is the physical source table, width traversing and parsing the array of fields to be inserted, for each field to be inserted in the array of fields to be inserted, determining the source field array corresponding to the field to be inserted, traversing the source field array, adding each source field in the source field array to the columns collection of currentTable and the from collection corresponding to the field to be inserted, adding the field to be inserted to the currentColumns collection, and marking the currentColumns collection as a state to be allocated.

[0054] The breadth traversal and parsing of the array to be inserted includes: generating a field subtree to be inserted based on the array to be inserted, and breadth traversing the field subtree.

[0055] Optionally, performing a width traversal analysis on the node based on the node type further includes: if the temporary table is a temporary query table of a subquery, performing field source analysis on the array of fields to be inserted, and determining a source field array corresponding to each field to be inserted in the array of fields to be inserted; for each field to be inserted, traversing and adding the source field array to the from collection corresponding to the field to be inserted, adding the field to be inserted to the currentColumns collection, and marking the currentColumns collection as being in a state to be allocated.

[0056] Optionally, performing a width traversal analysis on the node based on the node type further includes: if the table to be inserted is a target table, obtaining the library name and table name of the target table; creating a result table targetTable, performing field source analysis on the field array to be inserted, and determining the source field array corresponding to each field to be inserted in the field array to be inserted; for each field to be inserted, traversing and adding the source field array corresponding to the field to be inserted to the from collection corresponding to the field to be inserted, and adding the field to be inserted to the columns collection of the result table targetTable.

[0057] Optionally, performing field source parsing on the array of fields to be inserted includes: for each field to be inserted in the array of fields to be inserted, obtaining the field alias and field type corresponding to the field to be inserted; if the field type is a preset first type, determining the field subtree node to be inserted corresponding to the field to be inserted based on the subquery name, and querying the field subtree node to be inserted based on a preset mapping relationship to obtain the source field array; if the field type is a preset second type, creating a constant name field based on the field to be inserted; if the field type is a preset third type, width traversing the field subtree to be inserted to determine whether the field subtree to be inserted has a table alias, and if so, querying based on the table alias and the field alias to obtain the source field array.

[0058] Among them, the preset first type includes * fields, the preset second type includes constant fields, and the preset third type includes ordinary fields.

[0059] Among them, the judging whether there is a table alias in the field subtree to be inserted also includes: if there is no table alias in the field subtree to be inserted, obtaining the field name of the field to be inserted, and searching the source field array in the columns set of the upper-level subquery or the corresponding physical source table corresponding to the field to be inserted based on the field name; if the source field array exists in the column set, returning the source field array; if the source field array does not exist in the column set, obtaining the metadata of the physical source table corresponding to the field to be inserted, extracting the physical table field information corresponding to the metadata, and determining the source field array based on the physical table field information.

[0060] Optionally, if the node type is not TOK_INSERT, determine whether the node type is TOK_SUBQUERY; if the node type is TOK_SUBQUERY, enter the subquery subtree parsing process, obtain at least one first subquery node corresponding to the node, traverse the at least one first subquery node, and determine whether there is any first subquery node whose node type is not TOK_SUBQUERY in the at least one first subquery node; if so, obtain the first child node whose node type is not TOK_SUBQUERY, obtain the second subquery node of the first subquery node, and determine whether the node type of the second subquery node is TOK_UNIONALL; if the node type of the second subquery node is TOK_UNIONALL, add the tree node corresponding to the first subquery node and the mapping of the corresponding unioncColumns array to aliasColumns, and clear the currentColumns array; if the node type of the second subquery node is not TOK_UNIONALL, add the tree node corresponding to the first subquery node and the mapping of the corresponding currentColumns to aliasColumns, and clear the currentColumns array.

[0061] Optionally, if the node type is not TOK_SUBQUERY, determine whether the node type is TOK_CREATETABLE. If the node type is TOK_CREATETABLE, enter the process of parsing and creating the table subtree, obtain the library name and table name under the node TOK_CREATETABLE subtree, create a result table targetTable object based on the library name and the table name, traverse the currentColumns array, and add each field in the currentColumns array to the columns collection of the result table targetTable.

[0062] Optionally, if the node type is not TOK_CREATETABLE, determine whether the node type is TOK_QUERY; if the node type is TOK_QUERY, enter the query subtree process, obtain the child node corresponding to the node, determine whether the child node is TOK_UNIONALL; if the child node is TOK_UNIONALL, enter the union subtree parsing process, obtain the hidden alias of the Union corresponding to the child node, determine the tree node corresponding to the hidden alias, determine the Union field array corresponding to the Union through unionColumns, traverse the Union field array and currentColumns array at the same time, merge the Union field and the currentColumns field for the Union field and the currentColumns field at the same position to obtain a merged field object, the from collection field corresponding to the Union field and the from collection field corresponding to the currentColumns field point to the merged field object; after the traversal of the Union field array and the currentColumns array is completed, generate a merged field object array, traverse the merged field object array, store each merged field object in the unionColumns table, and clear the currentColumns array.

[0063] Optionally, if the node type is not TOK_QUERY, skip the node and continue to perform the depth-first post-order traversal parsing on the abstract syntax tree.

[0064] Optionally, generating a blood relationship tree includes: completing the depth-first post-order traversal parsing of the abstract syntax tree to obtain a targetTable, wherein a columns set in the targetTable includes fields of the target table corresponding to the SQL to be parsed, and generating a blood relationship tree corresponding to each field based on a from set corresponding to the field.

[0065] Step 204: encapsulate the blood relationship tree into field-level blood relationship data, and store the field-level blood relationship data.

[0066] Optionally, encapsulating the lineage tree as field-level lineage data includes: for each field, traversing the result table targetTable, the columns set and the lineage tree corresponding to the field to obtain the field-level lineage data.

[0067] Further, the field-level lineage data is encapsulated into HIVE_COLUMN_LINEAGE, and it is determined whether the field has constant field data, wherein the constant field data may include: a date field generated by a HIVE function, a constant field generated by a HIVE function, etc., which are not limited here. The constant field data is set as a passive field, and the HIVE_COLUMN_LINEAGE is stored in a preset database.

[0068] During the specific implementation process, the preset database is associated with the enterprise big data task scheduling system, and the HIVE_COLUMN_LINEAGE of the preset database is extracted through the enterprise big data task scheduling system to visualize the field-level lineage data.

[0069] It can be seen that in the embodiment of the present application, a data lineage parsing instruction is received, and the SQL to be parsed is extracted from the data lineage parsing instruction; the SQL to be parsed is parsed to generate an abstract syntax tree; the nodes of the abstract syntax tree are traversed in depth-first post-order, and for each node, the node type of the node is identified. When the node type is TOK_INSERT, a width traversal is performed on the node to determine the source field array, and a lineage tree is generated based on the source field array. The lineage relationship of the data can be inferred by a hybrid traversal method combining depth-first post-order traversal and width traversal to generate a lineage relationship tree; at the same time, the next SQL to be parsed is parsed based on the generated lineage relationship tree to realize lineage context association, and finally the lineage relationship tree is encapsulated as field-level lineage data, and the field-level lineage data is stored. The present application can infer the lineage relationship of data by a hybrid traversal method combining depth-first post-order traversal and width traversal, simplify the process of collecting and reasoning lineage relationships, and reduce the generation of temporary tables, reduce the generation and storage of invalid lineages, thereby facilitating the release of storage resources and improving the efficiency of lineage tracing.

[0070] See also Figure 3 , Figure 3 is a flowchart of a TOK_INSERT node parsing method provided in an embodiment of the present application. This method is applied to Figure 1 The electronic device shown is as shown in the figure:

[0071] Receive data lineage parsing instructions, extract the SQL to be parsed from the data lineage parsing instructions through SQL Parser, start HiveParser to parse the SQL to be parsed to generate an abstract syntax tree, traverse the abstract syntax tree in depth-first and post-order, and for each node of the abstract syntax tree, determine whether the node type is TOK_TABREF, TOK_INSERT, TOK_SUBQUERY, TOK_CREATETABLE, TOK_QUERY in turn. If the node type is TOK_INSERT, determine the field to be inserted corresponding to the SQL to be parsed, extract the table to be inserted corresponding to the field to be inserted, if the table to be inserted is a temporary table, execute the temporary table insertion process, if the table to be inserted is the target table, execute the target table insertion process.

[0072] The inserting process into the temporary table is executed, including: obtaining the temporary table type, and if the temporary table is a physical source table, performing a width traversal to parse the field subtree.

[0073] Furthermore, if the temporary table is a subquery, or the table to be inserted is the target table, the subtree of TOK_SELEXPR is parsed to determine the field type of the field to be inserted, the field subtree is traversed and parsed based on the field type width, the blood relationship is constructed, and it is determined whether the field to be inserted has a field alias. If so, alias inference is performed to determine the source field array of the field to be inserted. If not, alias-free inference is performed to determine the source field array of the field to be inserted.

[0074] In this way, the present application can infer the lineage relationship of data through a hybrid traversal method combining depth-first post-order traversal and width-first traversal, simplify the lineage collection and reasoning process, and construct data lineage based on the type of the field to be inserted, reduce the generation of temporary tables, and reduce the generation and storage of invalid lineage, which is conducive to releasing storage resources and improving the efficiency of lineage tracing.

[0075] See also Figure 4 , Figure 4 A structural diagram of an electronic device 400 provided in an embodiment of the present application, as shown in the figure, the server 400 includes an application processor 410, a memory 420, a communication interface 430 and one or more programs 421, wherein the one or more programs 421 are stored in the above-mentioned memory 420 and are configured to be executed by the above-mentioned application processor 410, and the one or more programs 421 include instructions for performing the following steps:

[0076] Receive a data lineage parsing instruction, and extract the SQL to be parsed from the data lineage parsing instruction;

[0077] Parse the SQL to be parsed to generate an abstract syntax tree;

[0078] Depth-first post-order traversal of the nodes of the abstract syntax tree, identifying the node type of each node, performing breadth-first traversal parsing on the node based on the node type, and generating a kinship tree;

[0079] The blood relationship tree is encapsulated into field-level blood relationship data, and the field-level blood relationship data is stored.

[0080] It can be seen that in the embodiment of the present application, a data lineage parsing instruction is received, and the SQL to be parsed is extracted from the data lineage parsing instruction; the SQL to be parsed is parsed to generate an abstract syntax tree; the nodes of the abstract syntax tree are traversed in depth-first post-order, and for each of the nodes, the node type of the node is identified, and a width traversal parsing is performed on the node based on the node type to generate a lineage tree; the lineage tree is encapsulated as field-level lineage data, and the field-level lineage data is stored. In this way, the lineage relationship of the data can be inferred through a hybrid traversal method combining depth-first post-order traversal and width traversal, simplifying the collection and reasoning process of lineage relationships, reducing the generation of temporary tables, and reducing the generation and storage of invalid lineages, which is conducive to releasing storage resources and improving the efficiency of lineage tracing.

[0081] In a possible example, the node type of the node is identified, and the instructions in the program are specifically used to perform the following operations: determine in sequence whether the node type of the node is TOK_TABREF, TOK_INSERT, TOK_SUBQUERY, TOK_CREATETABLE, or TOK_QUERY.

[0082] In a possible example, the node is subjected to a width traversal analysis based on the node type, and the instructions in the program are specifically used to perform the following operations: if the node type is the TOK_INSERT, an array of fields to be inserted is determined based on the SQL to be parsed; a table to be inserted corresponding to the array of fields to be inserted is determined through the child node TOK_DESTINATION; if the table to be inserted is a temporary table, the temporary table type of the temporary table is obtained; if the temporary table is the physical source table, the array of fields to be inserted is subjected to a width traversal analysis, and for each field to be inserted in the array of fields to be inserted, a source field array corresponding to the field to be inserted is determined, the source field array is traversed, each source field in the source field array is added to the columns collection of the currentTable and the from collection corresponding to the field to be inserted, the field to be inserted is added to the currentColumns collection, and the currentColumns collection is marked as being in a state to be allocated.

[0083] In a possible example, the width traversal analysis is performed on the node based on the node type, and the instructions in the program are specifically used to perform the following operations: if the temporary table is a temporary query table of a subquery, field source analysis is performed on the array of fields to be inserted, and a source field array corresponding to each field to be inserted in the array of fields to be inserted is determined; for each field to be inserted, the source field array is traversed and added to the from collection corresponding to the field to be inserted, the field to be inserted is added to the currentColumns collection, and the currentColumns collection is marked as being in a state to be allocated.

[0084] In a possible example, the width traversal parsing is performed on the node based on the node type, and the instructions in the program are specifically used to perform the following operations: if the table to be inserted is a target table, obtain the library name and table name of the target table; create a result table targetTable, perform field source parsing on the field array to be inserted, and determine the source field array corresponding to each field to be inserted in the field array to be inserted; for each field to be inserted, traverse and add the source field array corresponding to the field to be inserted to the from collection corresponding to the field to be inserted, and add the field to be inserted to the columns collection of the result table targetTable.

[0085] In a possible example, the field source parsing is performed on the array of fields to be inserted, and the instructions in the program are specifically used to perform the following operations: for each field to be inserted in the array of fields to be inserted, the field alias and field type corresponding to the field to be inserted are obtained; if the field type is a preset first type, the field subtree node to be inserted corresponding to the field to be inserted is determined based on the subquery name, and the field subtree node to be inserted is queried based on a preset mapping relationship to obtain the source field array; if the field type is a preset second type, a constant name field is newly created based on the field to be inserted; if the field type is a preset third type, the field subtree to be inserted is traversed in width to determine whether the field subtree to be inserted has a table alias, and if so, the source field array is obtained by querying based on the table alias and the field alias.

[0086] In a possible example, the determination of whether the field subtree to be inserted has a table alias, the instructions in the program are specifically used to perform the following operations: if the field subtree to be inserted does not have a table alias, obtain the field name of the field to be inserted, and search for the source field array in the columns set of the upper-level subquery or the corresponding physical source table corresponding to the field to be inserted based on the field name; if the source field array exists in the column set, return the source field array; if the source field array does not exist in the column set, obtain the metadata of the physical source table corresponding to the field to be inserted, extract the physical table field information corresponding to the metadata, and determine the source field array based on the physical table field information.

[0087] The above mainly introduces the scheme of the embodiment of the present application from the perspective of the execution process on the method side. It is understandable that, in order to realize the above functions, the electronic device includes a hardware structure and / or software module corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0088] The embodiment of the present application can divide the electronic device into functional units according to the above method example. For example, each functional unit can be divided according to each function, or two or more functions can be integrated into a control unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units. It should be noted that the division of units in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0089] See also Figure 5 , Figure 5 A schematic diagram of the structure of a data lineage analysis device provided in an embodiment of the present application includes a receiving unit 501, a parsing unit 502, a lineage generation unit 503 and an encapsulation unit 504, wherein:

[0090] A receiving unit 501 receives a data lineage parsing instruction and extracts a SQL statement to be parsed from the data lineage parsing instruction;

[0091] Parsing unit 502, parses the SQL to be parsed to generate an abstract syntax tree;

[0092] A lineage generation unit 503 traverses the nodes of the abstract syntax tree in a depth-first post-order manner, identifies the node type of each node, performs a breadth-based traversal analysis on the node based on the node type, and generates a lineage relationship tree;

[0093] The encapsulation unit 504 encapsulates the blood relationship tree into field-level blood relationship data and stores the field-level blood relationship data.

[0094] It can be seen that in the embodiment of the present application, a data lineage parsing instruction is received, and the SQL to be parsed is extracted from the data lineage parsing instruction; the SQL to be parsed is parsed to generate an abstract syntax tree; the nodes of the abstract syntax tree are traversed in depth-first post-order, and for each of the nodes, the node type of the node is identified, and a width traversal parsing is performed on the node based on the node type to generate a lineage tree; the lineage tree is encapsulated as field-level lineage data, and the field-level lineage data is stored. In this way, the lineage relationship of the data can be inferred through a hybrid traversal method combining depth-first post-order traversal and width traversal, simplifying the collection and reasoning process of lineage relationships, reducing the generation of temporary tables, and reducing the generation and storage of invalid lineages, which is conducive to releasing storage resources and improving the efficiency of lineage tracing.

[0095] In a possible example, in terms of identifying the node type of the node, the lineage generation unit 503 is specifically used to: determine in sequence whether the node type of the node is TOK_TABREF, TOK_INSERT, TOK_SUBQUERY, TOK_CREATETABLE, TOK_QUERY.

[0096] In a possible example, in terms of performing a width traversal analysis on the node based on the node type, the lineage generation unit 503 is specifically used to: if the node type is the TOK_INSERT, determine the array of fields to be inserted based on the SQL to be parsed; determine the table to be inserted corresponding to the array of fields to be inserted through the child node TOK_DESTINATION; if the table to be inserted is a temporary table, obtain the temporary table type of the temporary table; if the temporary table is the physical source table, parse the array of fields to be inserted through a width traversal, and for each field to be inserted in the array of fields to be inserted, determine the source field array corresponding to the field to be inserted, traverse the source field array, add each source field in the source field array to the columns collection of currentTable and the from collection corresponding to the field to be inserted, add the field to be inserted to the currentColumns collection, and mark the currentColumns collection as a state to be allocated.

[0097] In a possible example, in terms of performing a width traversal analysis on the node based on the node type, the lineage generation unit 503 is specifically used to: if the temporary table is a temporary query table of a subquery, perform field source analysis on the array of fields to be inserted, and determine the source field array corresponding to each field to be inserted in the array of fields to be inserted; for each field to be inserted, traverse and add the source field array to the from collection corresponding to the field to be inserted, add the field to be inserted to the currentColumns collection, and mark the currentColumns collection as being in a state to be allocated.

[0098] In a possible example, in terms of performing a width traversal analysis on the node based on the node type, the lineage generation unit 503 is specifically used to: if the table to be inserted is a target table, obtain the library name and table name of the target table; create a result table targetTable, perform field source analysis on the field array to be inserted, and determine the source field array corresponding to each field to be inserted in the field array to be inserted; for each field to be inserted, traverse and add the source field array corresponding to the field to be inserted to the from collection corresponding to the field to be inserted, and add the field to be inserted to the columns collection of the result table targetTable.

[0099] In a possible example, in terms of performing field source resolution on the array of fields to be inserted, the lineage generation unit 503 is specifically used to: for each field to be inserted in the array of fields to be inserted, obtain the field alias and field type corresponding to the field to be inserted; if the field type is a preset first type, determine the field subtree node to be inserted corresponding to the field to be inserted based on the subquery name, and query the field subtree node to be inserted based on a preset mapping relationship to obtain the source field array; if the field type is a preset second type, create a constant name field based on the field to be inserted; if the field type is a preset third type, traverse the field subtree to be inserted to determine whether the field subtree to be inserted has a table alias, and if so, query based on the table alias and the field alias to obtain the source field array.

[0100] In a possible example, in terms of determining whether the field subtree to be inserted has a table alias, the lineage generation unit 503 is specifically used to: if the field subtree to be inserted does not have a table alias, obtain the field name of the field to be inserted, and search for the source field array in the columns set of the upper-level subquery or the corresponding physical source table corresponding to the field to be inserted based on the field name; if the source field array exists in the column set, return the source field array; if the source field array does not exist in the column set, obtain the metadata of the physical source table corresponding to the field to be inserted, extract the physical table field information corresponding to the metadata, and determine the source field array based on the physical table field information.

[0101] An embodiment of the present application also provides a computer storage medium, wherein the computer storage medium stores a computer program for electronic data exchange, wherein the computer program enables a computer to execute part or all of the steps of any method described in the above method embodiments, and the above computer includes an electronic device.

[0102] The embodiment of the present application also provides a computer program product, the computer program product includes a non-transitory computer-readable storage medium storing a computer program, the computer program is operable to cause a computer to execute some or all of the steps of any method described in the method embodiment. The computer program product may be a software installation package, and the computer includes an electronic device.

[0103] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0104] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0105] In the several embodiments provided in the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the above-mentioned units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0106] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0107] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0108] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the above-mentioned methods in each embodiment of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0109] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable memory, which may include a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0110] The embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data lineage analysis method, characterized in that: include: Receive a data lineage parsing instruction, and extract the SQL to be parsed from the data lineage parsing instruction; Parse the SQL to be parsed to generate an abstract syntax tree; Depth-first post-order traversal of the nodes of the abstract syntax tree, identifying the node type of each node, performing breadth-first traversal parsing on the node based on the node type, and generating a kinship tree; Encapsulating the blood relationship tree into field-level blood relationship data, and storing the field-level blood relationship data; The performing breadth traversal parsing on the node based on the node type includes: If the node type is TOK_INSERT, determine the array of fields to be inserted based on the SQL to be parsed; Determine the table to be inserted corresponding to the array of fields to be inserted through the child node TOK_DESTINATION; If the table to be inserted is a temporary table, obtain the temporary table type of the temporary table; If the temporary table is a physical source table, parse the array of fields to be inserted through a width traversal, determine the source field array corresponding to each field to be inserted in the array of fields to be inserted, traverse the source field array, add each source field in the source field array to the columns set of currentTable and the from set corresponding to the field to be inserted, add the field to be inserted to the currentColumns set, and mark the currentColumns set as a pending allocation state; The performing breadth traversal parsing on the node based on the node type further includes: If the temporary table is a temporary query table of a subquery, performing field source parsing on the array of fields to be inserted to determine a source field array corresponding to each field to be inserted in the array of fields to be inserted; For each field to be inserted, traverse and add the source field array to the from set corresponding to the field to be inserted, add the field to be inserted to the currentColumns set, and mark the currentColumns set as a to-be-allocated state; The performing breadth traversal parsing on the node based on the node type further includes: If the table to be inserted is a target table, obtain the library name and table name of the target table; Create a result table targetTable, perform field source parsing on the array of fields to be inserted, and determine the source field array corresponding to each field to be inserted in the array of fields to be inserted; For each field to be inserted, traverse and add the source field array corresponding to the field to be inserted to the from set corresponding to the field to be inserted, and add the field to be inserted to the columns set of the result table targetTable.

2. The method according to claim 1, characterized in that: The identifying the node type of the node comprises: Determine in turn whether the node type of the node is TOK_TABREF, TOK_INSERT, TOK_SUBQUERY, TOK_CREATETABLE, or TOK_QUERY.

3. The method according to claim 1, characterized in that The performing field source parsing on the array of fields to be inserted includes: For each field to be inserted in the array of fields to be inserted, obtaining a field alias and a field type corresponding to the field to be inserted; If the field type is the preset first type, determine the field subtree node to be inserted corresponding to the field to be inserted based on the subquery name, and query the field subtree node to be inserted based on the preset mapping relationship to obtain the source field array; If the field type is the preset second type, a constant name field is newly created based on the field to be inserted; If the field type is the preset third type, the field subtree to be inserted is traversed in width to determine whether the field subtree to be inserted has a table alias; if so, the source field array is obtained by querying based on the table alias and the field alias.

4. The method according to claim 3, characterized in that The step of determining whether the field subtree to be inserted has a table alias also includes: If the to-be-inserted field subtree does not have a table alias, obtain the field name of the to-be-inserted field, and search for the source field array in the columns collection of the upper-level subquery or the corresponding physical source table corresponding to the to-be-inserted field based on the field name; If the source field array exists in the columns set, return the source field array; If the source field array does not exist in the columns collection, metadata of the physical source table corresponding to the field to be inserted is obtained, physical table field information corresponding to the metadata is extracted, and the source field array is determined based on the physical table field information.

5. A data lineage analysis device, characterized in that: include: A receiving unit receives a data lineage parsing instruction and extracts a SQL statement to be parsed from the data lineage parsing instruction; A parsing unit, parsing the SQL to be parsed to generate an abstract syntax tree; A lineage generation unit, which traverses the nodes of the abstract syntax tree in a depth-first post-order manner, identifies the node type of each node, performs a breadth-based traversal analysis on the node based on the node type, and generates a lineage relationship tree; An encapsulation unit, encapsulating the blood relationship tree into field-level blood relationship data, and storing the field-level blood relationship data; Wherein, in the aspect of performing breadth traversal analysis on the node based on the node type, the lineage generation unit is specifically used for: If the node type is TOK_INSERT, determine the array of fields to be inserted based on the SQL to be parsed; Determine the table to be inserted corresponding to the array of fields to be inserted through the child node TOK_DESTINATION; If the table to be inserted is a temporary table, obtain the temporary table type of the temporary table; If the temporary table is a physical source table, parse the array of fields to be inserted through a width traversal, determine the source field array corresponding to each field to be inserted in the array of fields to be inserted, traverse the source field array, add each source field in the source field array to the columns set of currentTable and the from set corresponding to the field to be inserted, add the field to be inserted to the currentColumns set, and mark the currentColumns set as a pending allocation state; Wherein, in the aspect of performing breadth traversal analysis on the node based on the node type, the lineage generation unit is further used for: If the temporary table is a temporary query table of a subquery, performing field source parsing on the array of fields to be inserted to determine a source field array corresponding to each field to be inserted in the array of fields to be inserted; For each field to be inserted, traverse and add the source field array to the from set corresponding to the field to be inserted, add the field to be inserted to the currentColumns set, and mark the currentColumns set as a to-be-allocated state; Wherein, the performing breadth traversal analysis on the node based on the node type, the lineage generation unit is further used for: If the table to be inserted is a target table, obtain the library name and table name of the target table; Create a result table targetTable, perform field source parsing on the array of fields to be inserted, and determine the source field array corresponding to each field to be inserted in the array of fields to be inserted; For each field to be inserted, traverse and add the source field array corresponding to the field to be inserted to the from set corresponding to the field to be inserted, and add the field to be inserted to the columns set of the result table targetTable.

6. An electronic device, characterized in that: The method comprises a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, and the programs include instructions for executing the steps in the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method according to any one of claims 1 to 4.

Images