Analytic method for data provenance, storage medium, and processor

Abstract

Embodiments of the present application provide a method for data lineage analysis, a processor and a storage medium. The method comprises: obtaining a structured query language script to be analyzed, the structured query language script comprising at least one structured query statement; parsing the structured query language script to generate an abstract syntax tree corresponding to the structured query script, wherein the abstract syntax tree comprises different types of identifiers and identifier lists; splitting the abstract syntax tree using a segmented search method to obtain a plurality of sub-statements of the abstract syntax tree; determining a lineage data target table and an identifier included in the abstract syntax tree according to the plurality of sub-statements; and in the case where the abstract syntax tree is split, parsing the abstract syntax tree according to the lineage data target table and the identifier to obtain the lineage data of the structured query language script.

Description

Technical Field

[0001] This application relates to the field of computer data processing technology, specifically to a method for parsing data lineage, a storage medium, and a processor. Background Technology

[0002] The data lineage requirement in the construction machinery industry primarily stems from the data warehouse operation and maintenance process. During data warehouse operation and maintenance, task recovery after a data warehouse task failure requires clearly defining the upstream and downstream dependencies of the current task. Logical changes to data warehouse fields also affect downstream tables, and downstream tables must be notified promptly after any changes. Therefore, discovering the reference chain and influence chain of data, i.e., the data lineage relationship, becomes a common challenge faced by data warehouses.

[0003] Currently, traditional methods primarily use a combination of offline and online parsing to obtain lineage data within SQL scripts. The pre-parsing process involved in this method results in two separate parsing operations, consuming significant time. Furthermore, current offline parsing methods cannot retrieve table or field metadata and cannot handle cases where the SQL script contains asterisks (*). Summary of the Invention

[0004] The purpose of this application is to provide a method, storage medium, and processor for resolving data lineage.

[0005] To achieve the above objectives, the first aspect of this application provides a method for parsing data lineage, comprising:

[0006] Obtain the structured query language script to be parsed. The structured query language script must include at least one structured query statement.

[0007] The structured query language script is parsed to generate an abstract syntax tree corresponding to the structured query script, wherein the abstract syntax tree includes different types of identifiers and a list of identifiers;

[0008] A segmented search approach is used to split the abstract syntax tree to obtain multiple sub-statements of the abstract syntax tree;

[0009] The lineage data target table and identifiers contained in the abstract syntax tree are determined based on multiple sub-statements;

[0010] Once the abstract syntax tree has been split, the abstract syntax tree is parsed based on the lineage data target table and identifiers to obtain the lineage data of the structured query language script.

[0011] In one embodiment, the identifier includes at least a first type keyword, a second type keyword, a first keyword, a second keyword, and a third keyword. The abstract syntax tree (AST) is split using a segmented search method to obtain multiple sub-statements of the AST, including: determining whether the identifier of the first level of the AST is a first type keyword; if the identifier of the first level is a first type keyword, splitting the AST into a first type clause and a second type clause, wherein the first type clause is a clause composed of a first byte and a second byte; identifying the portion of the first type clause after the first byte as a first identifier, determining the first lineage data target table of the first type clause based on the first identifier, and identifying the structured query statement within the subquery after the second byte in the first type clause as a first union query clause; and splitting the AST based on the case where the first union query clause includes the first keyword.

[0012] In one embodiment, the parsing method further includes: determining the identifier of the first level as a second type key if the identifier of the first level is not a first type key; determining the first identifier that appears after the second type clause as the second lineage data target table of the second type clause; and splitting the abstract syntax tree according to the case that the second type clause includes the first key.

[0013] In one embodiment, the first key is a join query key. The abstract syntax tree is split based on whether the first join query clause includes the first key: if the first join query clause includes a join query key, determine whether the first select key following the first join query clause precedes the first join query key; if the first select key following the first join query clause precedes the first join query key, combine the identifiers between the first select key and the first join query key into the first join query clause; if the first select key following the first join query clause follows the first join query key, determine the first identifier or subquery before the first join query key as the first join query clause, and determine the identifiers between each join query key and after the last join query key as other join query clauses; convert the first join query clause and other join query clauses into first select clauses starting with the select key, and determine the first lineage data target table as the third lineage data target table of the first select clause; determine whether the first identifier in the first select clause is a select key, and split the abstract syntax tree based on the first identifier.

[0014] In one embodiment, the parsing method further includes: if the first join query clause does not include a join query keyword, obtaining the first select keyword after the first join query clause; determining the first select keyword after the first join query clause and the list of identifiers following the first select keyword after the first join query clause as the second select clause; determining whether the first identifier in the second select clause is a select keyword, so as to split the abstract syntax tree according to the case of the first identifier.

[0015] In one embodiment, the first key is a join query key. The abstract syntax tree is split based on the inclusion of the first key in the second type clause, including: if the second type clause includes a join query key, determining whether the first select key following the second type clause precedes the first join query key; if the first select key following the second type clause precedes the first join query key, combining the identifiers between the first select key and the first join query key into a first join query clause; if the first select key following the second type clause follows the first join query key, determining the first identifier or subquery before the first join query key as the first join query clause, and determining the identifiers between each join query key and after the last join query key as other join query clauses; converting both the first join query clause and other join query clauses into third select clauses starting with a select key, and determining the second lineage data target table as the fourth lineage data target table of the second select clause; determining whether the first identifier in the third select clause is a select key, and splitting the abstract syntax tree based on the first identifier.

[0016] In one embodiment, the parsing method further includes: if the second type clause does not include a union query keyword, obtaining the first selection keyword after the second type clause; determining the first selection keyword after the second type clause and the list of identifiers following the first selection keyword after the second type clause as the fourth selection clause; determining whether the first identifier in the fourth selection clause is a selection keyword, so as to split the abstract syntax tree according to the case of the first identifier.

[0017] In one embodiment, splitting the abstract syntax tree based on the first identifier includes: if the first identifier is a selection keyword, determining the clause between the selection keyword and the second keyword as a field domain clause, and obtaining the second identifier contained in the field domain clause; determining whether the list of identifiers after the second keyword includes a third keyword; if the list of identifiers after the second keyword includes the third keyword, determining the clause between the second keyword and the third keyword as an equi-join clause; if the list of identifiers after the second keyword does not include the third keyword, determining the identifier after the second keyword as a table retrieval clause.

[0018] In one embodiment, the parsing method further includes: determining that the first level of the abstract syntax tree has been split if the first identifier is not a selection keyword; and determining the lineage data of the structured query language script based on the field domain clause.

[0019] In one embodiment, lineage data includes column-level lineage data. Determining the lineage data of a Structured Query Language script based on field domain clauses includes: determining whether an identifier in a field domain clause has an alias; if an identifier in a field domain clause has an alias, combining the current position and alias of the aliased identifier with any one of the first, second, third, and fourth lineage data target tables to form the right column of the column-level lineage; if an identifier in a field domain clause does not have an alias, combining the field name of the identifier in a field domain clause with any one of the first, second, third, and fourth lineage data target tables to form the right column of the column-level lineage; determining the left column of the column-level lineage based on the type of the first identifier of the identifier in the field domain clause; and combining the right column of the column-level lineage with the left column of the column-level lineage to obtain the column-level lineage data.

[0020] In one embodiment, after the first level of the abstract syntax tree is split, the lineage data of the structured query language script is determined based on the equi-join clause and the table retrieval clause.

[0021] In one embodiment, lineage data includes table-level lineage data. Determining the lineage data of a Structured Query Language script based on equi-join clauses and table retrieval clauses includes: obtaining the database name and table name for each identifier in the equi-join clause and table retrieval clause; combining the database name and table name into the left column of the table-level lineage; and combining the left column with any one of the first, second, third, and fourth lineage data target tables to obtain the table-level lineage data.

[0022] In one embodiment, the parsing method further includes: determining whether the first identifier of the identifier in the equi-join clause and the table retrieval clause is a subquery; if the first identifier of the identifier in the equi-join clause and the table retrieval clause is a subquery, parsing the second level of the abstract syntax tree; if the first identifier of the identifier in the equi-join clause and the table retrieval clause is not a subquery, determining that the abstract syntax tree has been split.

[0023] In one embodiment, parsing the second level of the abstract syntax tree includes: obtaining aliases for identifiers in a subquery; determining the aliases for identifiers in the subquery as the lineage data target table of the first level of the abstract syntax tree, and determining the structured query statement within the subquery as the second union query clause; and parsing the second level of the abstract syntax tree based on the first identifier in the second union query clause.

[0024] A second aspect of this application provides a processor configured to perform the above-described method for parsing data lineage.

[0025] A third aspect of this application provides a machine-readable storage medium storing instructions that, when executed by a processor, configure the processor to perform the aforementioned processor described in the first aspect of this application, configured to perform the aforementioned method for resolving data lineage.

[0026] The above technical solution transforms the chain-like search process of lineage relationships into a segmented search by splitting the abstract syntax tree. The complete chain-like lineage relationship is then obtained by calculating the segmented lineage relationships. By using only one offline parsing method, not only is parsing time saved, but the case of * in the SQL script can also be handled.

[0027] Other features and advantages of the embodiments of this application will be described in detail in the following detailed description section. Attached Figure Description

[0028] The accompanying drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the following detailed description to explain the embodiments of this application, but do not constitute a limitation on the embodiments of this application. In the drawings:

[0029] Figure 1 The schematic diagram illustrates a flowchart of a method for resolving data lineage according to an embodiment of this application;

[0030] Figure 2 A schematic diagram illustrating the parsing of an abstract syntax tree according to an embodiment of this application is shown.

[0031] Figure 3aA flowchart illustrating the parsing of an abstract syntax tree according to an embodiment of this application is shown schematically;

[0032] Figure 3b This schematically illustrates another flowchart of parsing an abstract syntax tree according to an embodiment of this application;

[0033] Figure 3c This schematically illustrates yet another flowchart of parsing an abstract syntax tree according to an embodiment of this application;

[0034] Figure 4 The diagram illustrates the internal structure of a computer device according to an embodiment of this application. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for illustration and explanation of the embodiments of this application and are not intended to limit the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0036] Figure 1 A schematic flowchart illustrating a method for resolving data lineage according to an embodiment of this application is shown. Figure 1 As shown in one embodiment of this application, a method for resolving data lineage is provided, comprising the following steps:

[0037] Step 101: Obtain the structured query language script to be parsed. The structured query language script shall include at least one structured query statement.

[0038] Step 102: Parse the structured query language script to generate an abstract syntax tree corresponding to the structured query script. The abstract syntax tree includes different types of identifiers and a list of identifiers.

[0039] Step 103: Use a segmented search method to split the abstract syntax tree to obtain multiple sub-statements of the abstract syntax tree.

[0040] Step 104: Determine the lineage data target table and identifiers contained in the abstract syntax tree based on multiple sub-statements.

[0041] Step 105: After the abstract syntax tree has been split, the abstract syntax tree is parsed according to the lineage data target table and identifiers to obtain the lineage data of the structured query language script.

[0042] Data lineage analysis in a data warehouse includes two methods: offline analysis and online analysis. Offline analysis uses regular expressions to parse Structured Query Language (SCL) statements based on SCL keywords and syntax, generating an abstract syntax tree (Abstract Syntax Tree), and then parses the Abstract Syntax Tree to obtain the lineage data. Offline analysis does not require actual execution of the SCL script; it only focuses on the content of the SCL script, offering the advantage of fast parsing speed. However, it does not incorporate data warehouse metadata during the parsing process, so it cannot handle cases where the SCL script contains asterisks (*, here not referring to the multiplication operator, but rather an abbreviation for all fields in SCL; all instances of * in the following text have this meaning). Online analysis, on the other hand, first obtains the Abstract Syntax Tree from the logical plan during the execution of the SCL script, and then parses the Abstract Syntax Tree to obtain the lineage data. While online analysis incorporates data warehouse metadata and can handle cases where the SCL script contains asterisks, it requires actual execution of the SCL statements, making it slower than offline analysis in obtaining lineage data.

[0043] Typically, data warehouses contain a large number of structured query language scripts, making online parsing alone extremely time-consuming. While combining offline and online parsing to efficiently retrieve the data warehouse lineage presents three main drawbacks:

[0044] 1) When determining which parsing method to use for a structured query language script, a pre-parsing process is first employed to assess whether the chosen method meets the requirements. Therefore, the same structured query language script may be parsed twice, during both pre-parsing and actual parsing, resulting in redundant calculations and wasted time.

[0045] 2) Although a structured query language writing standard has been established, there are still many cases in structured query language scripts that contain *. This inevitably requires the use of online parsing to obtain the lineage data of most structured query language scripts.

[0046] 3) The offline parsing method does not incorporate data warehouse metadata during the parsing process, and therefore cannot handle cases where the structured query language script contains *.

[0047] To address the three issues mentioned above, this technical solution proposes an offline data lineage parsing method. This method involves dividing a single Structured Query Language (SCL) statement into different segments, and then parsing these segments to obtain the field lineage. Furthermore, this solution utilizes the contextual information of the lineage and the metadata information of the data warehouse to handle cases where the SCL script contains asterisks (*).

[0048] Further, the processor obtains the Structured Query Language (SQL) script to be parsed, which includes at least one structured query statement. After obtaining the SQL script, it parses it into an Abstract Syntax Tree (AST) according to the translation rules of the SQL, namely the definition rules of identifiers and identifier lists. An AST is a description of the general structure of data, representing an application-layer entity's representation of the data structure. In computer science, an AST is a tree-like representation of the abstract syntactic structure of a language's source code. Each node in the tree represents a structure in the source code. The syntax is described as "abstract" because it does not represent every detail of the actual syntax. Structured Query Language (SQL) is a full-featured database language. Identifiers can refer to tokens, and identifier lists can refer to token lists. A token can be a string generated by the server. In this technical solution, a Token can refer to every string appearing in the SQL statement. Tokens can be categorized into keywords (e.g., create, insert), identifiers (e.g., col1, table1), function names (e.g., nvl), operators (e.g., +, -, *, \), constants (e.g., 123, 'nan'), equations (e.g., a=b), comments (e.g., --xxx script), spaces (e.g., ''), punctuation marks (e.g., ...), and parentheses (e.g., ().). For example, in the SQL statement: `select a,b,(d+e)from c`, `select`, `from`, `a`, `b`, `c`, `d`, `e`, and `+` can all be considered Tokens. Furthermore, different types of Tokens can be combined to form token lists of various types, such as keywords (e.g., create, insert), equations (e.g., a+b), functions (e.g., nvl(a,b)), entity identifiers (e.g., col1), sub-SQL statements (e.g., `select(selectcol1from b)from a)`), entity identifier lists (e.g., col1, col2), special functions (without parentheses, e.g., Case when), and comparison operations (e.g., col1>2).In this context, the relationships between formulas, functions, entity identifiers, sub-SQL statements, entity identifier lists, and comparison formulas are mutually inclusive. For example, in the SQL statement: `select nvl(a,if(a>b,c,d))+(c*d)+if(a>b,c,d)+e as col1,col2from table1`, all tokens between `select` and `from` are recognized as entity identifier lists. Within these entity identifier lists, `nvl(a,if(a>b,c,d))+(c*d)+if(a>b,c,d)+e as col1,col2from table1` will be recognized as entity identifiers. Within the entity identifier list, `nvl(a,if(a>b,c,d))+(c*d)+if(a>b,c,d)+e as col1,col2` will be recognized as entity identifiers. Within the entity identifier list, `nvl(a,if(a>b,c,d))+(c*d)+if(a>b,c,d)+e as col1,col2` will be recognized as entity identifiers. In col1, nvl(a,if(a>b,c,d))+(c*d)+if(a>b,c,d)+e is an expression. In the expression nvl(a,if(a>b,c,d))+(c*d)+if(a>b,c,d)+e, nvl(a,if(a>b,c,d)) is a function, (c*d) is a sub-SQL statement, and if(a>b,c,d) is a function. In the function if(a>b,c,d), a>b is a comparison expression.

[0049] Furthermore, the processor obtains the SQL script to be parsed, parses the SQL script, and generates a corresponding abstract syntax tree containing different types of tokens and token lists. For example... Figure 2 As shown, a schematic diagram of an abstract syntax tree is provided, which parses the SQL statement insert into testable select ws_sold_date_sk sold_date_sk,ws_ext_sales_price sales_price from web_sales into an abstract syntax tree.

[0050] Furthermore, after parsing the Structured Query Language (SQL) script into an Abstract Syntax Tree (ABST), the processor uses a segmented search approach to split the ABST into multiple sub-statements of different types. Specifically, the SQL script is split into SQL clauses, based on the requirement that each SQL clause contains a complete lineage. The segmented search approach can refer to splitting the SQL script according to the priority order: WITH AS clause > UNION keyword > FROM keyword > JOIN keyword. Based on the multiple sub-statements obtained from the splitting, the target tables and identifiers of the lineage data contained in the ABST are determined. This process continues until the last priority level is reached, indicating that the current level of the ABST has been completely split. Then, the ABST is parsed based on the target tables and identifiers to obtain the lineage data corresponding to the SQL script. Here, an identifier is a symbol used to identify an entity; in programming languages, identifiers are the names used for variables, constants, functions, and statement blocks.

[0051] The above technical solution transforms the chain-like search process of lineage relationships into a segmented search by splitting the abstract syntax tree. Then, the complete chain-like lineage relationship is obtained by calculating the segmented lineage relationships. By using only one offline parsing method, it not only saves parsing time but also can handle the case where there are * in the SQL script.

[0052] Figure 1 This is a flowchart illustrating a method for resolving data lineage in one embodiment. It should be understood that, although... Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0053] In one embodiment, such as Figure 3aThe diagram shows a flowchart for parsing an abstract syntax tree. The identifiers include at least a first-type keyword, a second-type keyword, a first keyword, a second keyword, and a third keyword. Specifically, in this technical solution, the first-type keyword can refer to the `with as` keyword, and the first-type clause can refer to the `with as` clause. The first byte and second byte of the first-type clause can refer to `with` and `as`, respectively. The second-type keyword can refer to DDL / DML keywords, and the second-type clause can refer to DDL / DML clauses. DML clauses refer to Data Manipulation Language clauses; for example, if a table in a database is like a box, then the data is like the chocolates inside the box, and Data Manipulation Language performs operations such as adding, deleting, and modifying data in the table. DDL clauses refer to Data Definition Language clauses; Data Definition Language is a statement that changes the data structure. The first keyword can refer to the `union` keyword, which can be understood as a keyword for joining queries. The second keyword can refer to the `from` keyword, which can be understood as a keyword for table retrieval. The third keyword can refer to the `join` keyword, which can be understood as a keyword for equi-joins.

[0054] Furthermore, the processor uses a segmented search method to split the first level of the abstract syntax tree (AST). It determines whether the identifier of the first level is a first-type key. If the identifier is a first-type key, the AST is split into first-type clauses and second-type clauses. A first-type clause is a clause composed of the first byte and the second byte. The portion of the first-type clause following the first byte is identified as the first identifier, and the first lineage data target table of the first-type clause is determined based on the first identifier. The structured query statement within the subquery following the second byte of the first-type clause is identified as the first union query clause. The AST is then split based on whether the first union query clause includes the first key. Specifically, Figure 3a A flowchart for parsing the abstract syntax tree is provided. For example... Figure 3a As shown, it determines whether the token of the first level of the abstract syntax tree is a `with as` keyword. If the token is a `with as` keyword, the abstract syntax tree is split into `with as` clauses and DDL / DML clauses. For `with as` clauses, the AST following `with` is recognized as an identifier or a list of identifiers. The content before `as` is used as the target table for this lineage data, and the SQL within the subquery following `as` is used as the `union` clause. After using the SQL within the subquery following `as` as the `union` clause, the AST is split according to the `union` keyword in the `union` clause.

[0055] In one embodiment, if the identifier of the first level is not a first-type keyword, the identifier of the first level is determined to be a second-type keyword. The first identifier appearing after the second-type clause is determined as the second lineage data target table of the second-type clause, and the abstract syntax tree is split according to the case where the second-type clause includes the first keyword. Specifically, it is determined whether the token of the first level of the abstract syntax tree is a "with as" keyword. If the token is not a "with as" keyword, the identifier of the first level is determined to be a DDL / DML keyword. If the identifier of the first level is determined to be a DDL / DML keyword, for the DDL or DML clause part, the first identifier appearing after the DDL or DML clause is taken as the target table of the current lineage data, and the AST is split according to the case of the "union" keyword in the DDL / DML clause.

[0056] In one embodiment, such as Figure 3aThe diagram shows a flowchart for parsing the Abstract Syntax Tree (AST). The first key is the union query key. The AST is split based on whether the first union query clause includes the first key, including: If the first union query clause includes the union query key, determine whether the first select key following the first union query clause precedes the first union query key. Specifically, the AST is split based on the union key in the union clause, including: If the union clause includes the union key, determine whether the first select key following the union clause precedes the first union key. If the first select key following the first union query clause precedes the first union query key, combine the tokens between the first select key and the first union query key into the first union query clause. Specifically, if the first select key precedes the first union key, combine all tokens between the first select key and the first union key into the first union clause. If the first select key following the first union query clause is after the first union query key, determine the first identifier or subquery before the first union query key as the first union query clause, and determine the tokens between each union query key and after the last union query key as other union query clauses. Specifically, when the first select keyword follows the first union keyword, the first identifier or subquery before the first union keyword is taken as the first union clause, and all tokens between each union keyword and after the last union keyword are taken as the remaining union clauses. The first union clause and all other union clauses are converted into first select clauses starting with the select keyword, and the first lineage data target table is determined as the third lineage data target table of the first select clause. It is determined whether the first identifier in the first select clause is the select keyword, and the abstract syntax tree is split based on the first identifier. Specifically, the first union clause and the remaining union clauses are converted into select clauses starting with select, and the content before as in the with as clause is also taken as the target table of that lineage. It is also determined whether the first token in the select clause is the select keyword, and the AST is further split based on the first token.

[0057] In one embodiment, if the first union query clause does not include a union query keyword, the first select keyword following the first union query clause is obtained. The first select keyword following the first union query clause and the list of tokens following it are used to determine the second select clause. It is then determined whether the first token in the second select clause is a select keyword, and the abstract syntax tree (AST) is split based on the first token. Specifically, splitting the AST based on the union keyword in the union clause includes: if the union clause does not include a union keyword, obtaining the first select keyword following the union, and combining the first select keyword and all subsequent tokens into a select clause. After combining the first select keyword and all subsequent tokens into a select clause, it is determined whether the first token in the select clause is a select keyword, and the AST is further split based on the first token.

[0058] In one embodiment, the first key is a union query key. Splitting the abstract syntax tree (AST) based on the inclusion of the first key in the second type clause includes: if the second type clause includes a union query key, determining whether the first select key following the second type clause precedes the first union query key. Specifically, splitting the AST based on the union key in the DDL / DML clause includes: if the DDL / DML clause includes a union key, determining whether the first select key following the DDL / DML clause precedes the first union key. If the first select key following the second type clause precedes the first union query key, combining the tokens between the first select key and the first union query key into the first union query clause. Specifically, if the first select key precedes the first union key, combining all tokens between the first select key and the first union key into the first union clause. If the first select key following the second type clause follows the first union query key, determining the first identifier or subquery before the first union query key as the first union query clause, and determining the identifiers between each union query key and after the last union query key as other union query clauses. Specifically, when the first select keyword follows the first union keyword, the first identifier or subquery before the first union keyword is taken as the first union clause, and all tokens between each union keyword and after the last union keyword are taken as the remaining union clauses. The first union clause and all other union clauses are converted into third select clauses starting with the select keyword, and the second-lineage data target table is determined as the fourth-lineage data target table of the second select clause. It is determined whether the first identifier in the third select clause is the select keyword, and the abstract syntax tree is split based on the first identifier. Specifically, the first union clause and the remaining union clauses are converted into select clauses starting with select, and the first identifier appearing after the DDL or DML clause is also taken as the target table of that lineage. It is also determined whether the first token in the select clause is the select keyword, and the AST is further split based on the first token.

[0059] In one embodiment, if the union query keyword is not included in the second type clause, the first select keyword after the second type clause is obtained. The first select keyword after the second type clause and the list of tokens following it are determined as the fourth select clause. It is determined whether the first token in the fourth select clause is a select keyword, and the abstract syntax tree is split based on the first token. Specifically, splitting the AST based on the union keyword in the DDL / DML clause includes: if the union keyword is not included in the DDL / DML clause, obtaining the first select keyword after the union, and combining the first select keyword and all subsequent tokens into a select clause. After combining the first select keyword and all subsequent tokens into a select clause, it is determined whether the first token in the select clause is a select keyword, and the AST is further split based on the first token.

[0060] In one embodiment, splitting the abstract syntax tree based on the first identifier in the selection clause includes: if the first identifier is a selection keyword, determining the clause between the selection keyword and the second keyword as a field domain clause, and obtaining the second identifier contained in the field domain clause. Then, determining whether the list of identifiers following the second keyword includes a third keyword; if the list includes the third keyword, determining the clause between the second and third keywords as an equi-join clause; if the list does not include the third keyword, determining the identifier following the second keyword as a table retrieval clause. Specifically, Figure 3b Another flowchart for parsing abstract syntax trees is provided. For example... Figure 3bAs shown, determining whether the first token in the select clause is the select keyword, and then further decomposing the AST based on the first token, includes: if the first token in the select clause is the select keyword, finding the from keyword after the select keyword, taking the clause between the select keyword and the from keyword as the select_from clause, and obtaining the identifier contained in the select_from clause. After obtaining the identifier contained in the select_from clause, determining whether the token list after the from keyword includes the join keyword. If the token list after the from keyword includes the join keyword, taking the clause between the from keyword and the join keyword as the join clause. If the token list after the from keyword does not include the join keyword, taking the identifier after the from keyword as the from clause.

[0061] Furthermore, if the first token is not the select keyword, it is determined that the first level of the abstract syntax tree has been fully split, and the lineage data of the structured query language script is determined based on the field domain clause. Specifically, if the first token in the select clause is not the select keyword, it is determined that the first level of the AST has been fully split, and the lineage data of the SQL script is determined based on the select_from clause.

[0062] In one embodiment, lineage data includes column-level lineage data. Determining the lineage data of the Structured Query Language script based on the field domain clause includes: determining whether an identifier in the field domain clause has an alias; if an identifier in the field domain clause has an alias, combining the current position and alias of the identifier with any one of the first, second, third, and fourth lineage data target tables to form the right column of the column-level lineage. If an identifier in the field domain clause does not have an alias, combining the field name of the identifier in the field domain clause with any one of the first, second, third, and fourth lineage data target tables to form the right column of the column-level lineage. Determining the left column of the column-level lineage based on the type of the first identifier in the field domain clause, and combining the right column of the column-level lineage with the left column of the column-level lineage to obtain the column-level lineage data. Specifically, as shown... Figure 3bAs shown, determining the lineage data of an SQL script based on the `SELECT_FROM` clause includes: checking if an identifier in the `SELECT_FROM` clause has an alias; if an identifier has an alias, combining the current position and alias of the aliased identifier with any target lineage data table to form the right column of the column-level lineage. For identifiers without aliases, combining the field names of the identifier in the `SELECT_FROM` clause with any target lineage data table to form the left column of the column-level lineage. The left and right columns are then combined to obtain the complete column-level lineage data.

[0063] In one embodiment, such as Figure 3b As shown, another flowchart for parsing the Abstract Syntax Tree (AST) is provided. If the list of identifiers following the second key does not include the third key, the identifiers following the second key are identified as table retrieval clauses, and the first level of the AST is determined to be fully split. At this point, the lineage data of the Structured Query Language (SCL) script is determined based on the equi-join clause and the table retrieval clause. Specifically, if the list of tokens following the `from` key does not include the `join` key, the identifiers following the `from` key are identified as `from` clauses, and the first level of the AST is determined to be fully split. Lineage data includes table-level lineage data. Determining the lineage data of the SCL script based on the equi-join clause and the table retrieval clause includes: obtaining the database name and table name for each identifier in the equi-join clause and the table retrieval clause; combining the database name and table name into the left column of the table-level lineage; and combining the left column with any one of the first, second, third, and fourth lineage data target tables to obtain the table-level lineage data. Specifically, determining the lineage data of the SQL script based on the join and from clauses includes: for each identifier in the join and from clauses, obtaining the database name and table name for each identifier, combining the database name and table name to form the left column of the table-level lineage data, and combining the left column with any target table of lineage data to obtain the complete table-level lineage data.

[0064] In one embodiment, such as Figure 3cThe diagram illustrates another flowchart for parsing the Abstract Syntax Tree (AST). It determines whether the first token of the identifiers in the equi-join and table retrieval clauses is a subquery. If it is, the second level of the AST is parsed. If it is not, the AST is considered fully split. Specifically, it determines whether the first token of the identifiers in the join and from clauses is a subquery. If it is, the second level of the AST is parsed; if it is not, all levels of the AST are considered fully split.

[0065] Furthermore, if the first token is a subquery, the second level of the AST is parsed. Parsing the second level of the AST includes: obtaining aliases for identifiers in the join and from clauses, identifying these aliases as the target table for the lineage data of the first level of the AST, and identifying the SQL within the subquery as a union clause. The second level of the AST is then parsed based on the first token in the union clause.

[0066] In one embodiment, when there is no alias for the identifier in the field field clause, the left column of the column lineage is determined based on the type of the first token of the identifier in the field field clause. That is, when there is no alias for the identifier in the select_from clause, determining the left column of the column lineage based on the type of the first token of the identifier in the select_from clause includes: when the first token is a subquery, splitting the subquery into operator functions, identifiers, subqueries, etc.; when the first token is an operator, splitting the operator into functions, identifiers, subqueries, etc.; when the first token is a function, obtaining all field identifiers within the function parameters, and combining the database name, table name, and field name of the identifiers as the left column; when the first token is an identifier or *, combining the database name, table name, and field name of the identifiers as the left column; when the first token is a comparison operation, splitting it into left and right parts (possibly operator functions, identifiers, or subqueries) based on the comparison operator; when the first token is a case, splitting the case into subqueries, operators, functions, identifiers, and comparison operations.

[0067] In one embodiment, if the following two special cases occur when processing rank-based lineage acquisition:

[0068] The first method: When an SQL statement contains a union clause, the target column information for the lineage of fields following the first union keyword is based on the target column of the first select clause. For example: SQL statement: insert into tableA select col1 as col1,col2 as col2 from tableB union select col3,col4 from tableC. Without special processing, the resulting lineage data is: tableB.col1->tableA.col1, tableB.col2->tableA.col2, tableC.col3->tableA.col3, tableC.col4->tableA.col4. Observing the right column of the lineage data, we can see that tableA has fields col1, col2, col3, and col4. However, according to SQL semantics, tableA only has fields col1 and col2. Therefore, when processing the lineage of select col3,col4 from tableC, we interpret the SQL statement as insert into tableA select col3 as col1,col4 as col2 from tableC. The resulting lineage data is as follows: tableB.col1->tableA.col1, tableB.col2->tableA.col2, tableC.col3->tableA.col1, tableC.col4->tableA.col1.

[0069] The second scenario: When an SQL statement uses * and the * from keyword involves multiple tables, by default, * will only retrieve the first table after * from, not all tables. For example, in `insert into tableA select * from tableB, tableC`, the * in the SQL statement is ambiguous and should be avoided. In practice, we interpret the SQL as `insert into tableA select tableB.* from tableB, tableC`, resulting in a lineage of `tableB.*->tableA.*`.

[0070] In one embodiment, the lineage data obtained through abstract syntax tree parsing can be divided into two parts: one part is column-level lineage data containing *, which still needs further processing; the other part is lineage data that does not contain * (no processing required). The second innovation of this solution is to use the context information of lineage within the script, as well as the metadata information of the data table, to handle the case where * is present in the SQL script (lineage data).

[0071] For column-level lineage data containing *, distinguish the tables within the lineage data. Based on whether the table is persisted in the SQL script and the scope of its use, tables are categorized into: entity tables, temporary tables, and virtual tables. Entity tables: Data is persisted in the database and can be used by all scripts; Temporary tables: Data is persisted in the database (or cached in memory) and can only be used within the current script; Virtual tables: Data is not persisted and typically refers to a SQL statement or an alias within an SQL statement. For example, in the SQL statement: `insert into tableB(column2)select tableA.column1 from(select *from tableC)tableA`, `tableA` is a virtual table, and `tableB` and `tableC` can be entity tables or temporary tables (the naming conventions for tables can be used to distinguish between entity tables and temporary tables; typically, `tmp.tableB` indicates a temporary table, and `adl.tableC` indicates a entity table).

[0072] Data with * in both the left and right columns can be categorized into the following 9 cases based on the above classification: The left column is a temporary table: Temporary tables are created and used according to the principle of "first come, first served".

[0073] Scenario 1: The left and right tables are both temporary tables (create table tmp.tableA select * from tmp.tableB). Regarding the field information of tmp.tableB, following the principle of creating and using temporary tables first, there will definitely be SQL similar to the following before the current SQL script: create table tmp.tableB select col1,col3 from tmp.tableC. This shows the lineage: tmp.tableC.col1 -> tmp.tableB.col1. Based on this lineage data, tmp.tableB has only one column, col1. Therefore, tmp.tableA also has only one column, col1. Thus, tmp.tableB.col1 -> tmp.tableA.col1, tmp.tableB.col3 -> tmp.tableA.col3.

[0074] Scenario 2: The left table is a temporary table and the right table is an entity table (insert into db.tableA select *from tmp.tableB). Similarly, db.tableA also has only one column, but the column name needs to be obtained from the metadata of db.tableA. Assuming it's col2, the lineage data would be tmp.tableB.col1 -> db.tableA.col2.

[0075] Case 3: The left table is a temporary table and the right table is a virtual table (with v_table as select * from tmp.tableB). Similar to Case 1, because the right table is a virtual table with no fixed table structure and fields, its fields are determined by the left table tmp.tableB, i.e., it only has the field col1. Therefore, the lineage data is tmp.tableB.col1 -> <default>.v_table.col1. When the left table is an entity table, the metadata information of the entity table (which includes at least the database name, table name, field name, field sorting, etc.) can be queried. Whether it is a relational database (such as MySQL or ORACLE) or a data warehouse (such as HIVE), there are tables whose metadata information can be queried. Assume that the metadata information of db.tableB is as shown in Table (1):

[0076] Database name Table name field name Field sorting db tableB col1 1 db tableB col2 2

[0077] Table (1)

[0078] Scenario 4: The left table is a physical table and the right table is a temporary table (create table tmp.tableA select *from db.table). Since tmp.tableA is newly created, its fields are determined by db.tableB, so the lineage data is db.tableB.col1->tmp.tableA.col1, db.tableB.col2->tmp.tableA.col2.

[0079] Case 5: The left table is an entity table and the right table is an entity table (insert into db.tableC select *from db.tableB). Since db.tableC is an entity table, its fields are defined in the metadata. Assume that the metadata of tableC is as follows (2):

[0080] Database name Table name field name Field sorting db tableC col5 1 db tableC col4 2

[0081] Table (2)

[0082] Therefore, based on the metadata, the SQL can be translated as: insert into db.tableC(col5,col6)selectcol1,col2 from db.tableB, so the lineage data can be obtained: db.tableB.col1->db.tableC.col5, db.tableB.col2->db.tableC.col4.

[0083] Case 6: The left table is a physical table, and the right table is a virtual table (with v_tableD as select * from db.tableB). Because v_tableD is a virtual table, its fields are determined by db.tableB, so the lineage data is...

[0084] db.tableB.col1-> <default>.v_tableD.col1,db.tableB.col2-> <default>.v_tableD.col2. The left table is a virtual table. A virtual table is an intermediate variable between an entity table or a temporary table. The virtual table ultimately originates from an entity table or a temporary table.

[0085] Case 7: The left table is a virtual table and the right table is a temporary table (create table tmp.tableA select a.*from(select col1 from db.tableB)a). The lineage relationship in the above SQL is... <default>.a.*->tmp.tableA.*onedb.tableB.col1-> <default>.a.col1, <default>The lineage information for .a.* can be obtained from existing lineage field information. For example... <default>If the .a file only contains the field col1, then the complete bloodline relationship is: <default>.a.col1->tmp.tableA.col1onedb.tableB.col1-> <default>.a.col1.

[0086] Case 8: The left table is a virtual table and the right table is a physical table (insert into db.tableA select a.*from(select col1 from db.tableB)a). In the above SQL, the lineage relationship is... <default>.a.*->db.tableA.*onedb.tableB.col1-> <default>.a.col1, <default>The lineage information for .a.* can be obtained from the existing lineage field information. The field information for db.tableA can be obtained from metadata; assuming there is a field col3, the complete lineage relationship would be... <default>.a.col1->db.tableA.col3onlydb.tableB.col1-> <default>.a.col1.

[0087] Case 9: The left table is a virtual table and the right table is also a virtual table (insert into db.tableA select b.*from(select a.*from(select col1 from db.tableB)a)b). In the above SQL, the lineage relationship is... <default>.b.*->db.tableA.*、db.tableB.col1-> <default>.a.col1、 <default> .a.*-> <default>.b.*. For <default> .a.*-> <default> .b.*， <default>The lineage information for .a.* can be obtained from the existing lineage field information. <default>.b.* is a virtual table, and its field content is determined by... <default>.a determines. Then the complete blood relationship is: <default>.b.col1->db.tableA.col1、db.tableB.col1-> <default>.a.col1、 <default>.a.col1-> <default>.b.col1.

[0088] This solution proposes an offline data lineage parsing method. By dividing the SQL script into different sub-statements, the data lineage is transformed from a chain-like structure to a segmented structure. Segmented parsing ensures that the lineage data will only contain cases where both the left and right columns are simultaneously *. Furthermore, this solution uses metadata and the context of the lineage data to handle the case where both columns are simultaneously *, in order to obtain complete lineage data. This offline parsing method does not require actual execution of SQL statements, significantly improving parsing efficiency compared to online data lineage parsing.

[0089] This application provides a storage medium storing a program that, when executed by a processor, implements the above-described method for resolving data lineage.

[0090] This application provides a processor for running a program, wherein the program executes the above-described method for parsing data lineage.

[0091] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor A01, a network interface A02, memory (not shown), and a database (not shown) connected via a system bus. The processor A01 provides computing and control capabilities. The memory includes internal memory A03 and a non-volatile storage medium A04. The non-volatile storage medium A04 stores an operating system B01, a computer program B02, and a database (not shown). The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 in the non-volatile storage medium A04. The database stores data used for data lineage parsing. The network interface A02 communicates with external terminals via a network connection. When the computer program B02 is executed by the processor A01, it implements a method for data lineage parsing.

[0092] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0093] This application provides an apparatus including a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements a method for resolving data lineage.

[0094] This application also provides a computer program product that, when executed on a data processing device, is adapted to perform an initialization method for resolving data lineage.

[0095] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0096] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0097] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0098] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0099] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0100] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0101] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0102] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0103] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.< / default> < / default> < / default> < / default> < / default> < / default> < / default> < / default> < / default> < / default> < / default> < / default> < / default> < / default> < / default> < / default> < / default> < / default> < / default> < / default> < / default> < / default> < / default> < / default> < / default> < / default> < / default>

Claims

1. A method for parsing data lineage, characterized in that, The parsing method includes: Obtain the structured query language script to be parsed, wherein the structured query language script includes at least one structured query statement; The structured query language script is parsed to generate an abstract syntax tree corresponding to the structured query script, wherein the abstract syntax tree includes different types of identifiers and a list of identifiers; The abstract syntax tree is split using a segmented search method to obtain multiple sub-statements of the abstract syntax tree; The lineage data target table and identifiers contained in the abstract syntax tree are determined based on the multiple sub-statements; Once the abstract syntax tree has been split, the abstract syntax tree is parsed according to the lineage data target table and the identifier to obtain the lineage data of the structured query language script; The identifier includes at least a first type keyword, a second type keyword, a first keyword, a second keyword, and a third keyword. The step of splitting the abstract syntax tree using a segmented search method to obtain multiple sub-statements of the abstract syntax tree includes: Determine whether the identifier of the first level of the abstract syntax tree is a keyword of the first type; When the identifier of the first level is the first type keyword, the abstract syntax tree is split into the first type clause and the second type clause, wherein the first type clause is a clause composed of the first byte and the second byte; The portion of the first type clause following the first byte is determined as the first identifier, and the first lineage data target table of the first type clause is determined based on the first identifier. The structured query statement within the subquery following the second byte in the first type clause is determined as the first union query clause. The abstract syntax tree is split based on the case where the first keyword is included in the first joint query clause.

2. The method for parsing data lineage according to claim 1, characterized in that, The parsing method also includes: If the identifier of the first level is not the first type of keyword, then the identifier of the first level is determined to be the second type of keyword; The first identifier that appears after the second type clause is identified as the second lineage data target table of the second type clause; The abstract syntax tree is split according to the case where the second type of clause includes the first keyword.

3. The method for analyzing data lineage according to claim 2, characterized in that, The first keyword is a join query keyword, and the step of splitting the abstract syntax tree based on the case where the first keyword is included in the first join query clause includes: If the first join query clause includes the join query keyword, determine whether the first selection keyword after the first join query clause is located before the first join query keyword; If the first select keyword after the first join query clause is located before the first join query keyword, the identifier between the first select keyword and the first join query keyword is combined into the first join query clause; If the first select key after the first join query clause is located after the first join query key, the first identifier or subquery before the first join query key is determined as the first join query clause, and the identifiers between each join query key and after the last join query key are determined as other join query clauses; The first and all other joint query clauses are converted into first selection clauses that begin with the selection keyword, and the first lineage data target table is determined as the third lineage data target table of the first selection clause; Determine whether the first identifier in the first selection clause is a selection keyword, so as to split the abstract syntax tree according to the condition of the first identifier.

4. The method for analyzing data lineage according to claim 3, characterized in that, The parsing method also includes: If the first join query clause does not include the join query keyword, obtain the first selection keyword after the first join query clause; The first selection keyword after the first joint query clause and the list of identifiers following the first selection keyword after the first joint query clause are determined as the second selection clause; Determine whether the first identifier in the second selection clause is a selection keyword, and then split the abstract syntax tree according to the condition of the first identifier.

5. The method for parsing data lineage according to claim 4, characterized in that, The first keyword is a union query keyword, and the step of splitting the abstract syntax tree based on the case where the second type clause includes the first keyword includes: If the second type clause includes the join query keyword, determine whether the first selection keyword after the second type clause is located before the first join query keyword; If the first select keyword following the second type clause is located before the first join query keyword, the identifier between the first select keyword and the first join query keyword is combined into the first join query clause. If the first select keyword after the second type clause is located after the first join query keyword, the first identifier or subquery before the first join query keyword is determined as the first join query clause, and the identifiers located between each join query keyword and after the last join query keyword are determined as other join query clauses; The first and all other joint query clauses are converted into third selection clauses that begin with the selection keyword, and the second lineage data target table is determined as the fourth lineage data target table of the second selection clause; Determine whether the first identifier in the third selection clause is a selection keyword, so as to split the abstract syntax tree according to the condition of the first identifier.

6. The method for parsing data lineage according to claim 5, characterized in that, The parsing method also includes: If the second type clause does not include a join keyword, obtain the first select keyword following the second type clause; The first selection keyword following the second type clause and the list of identifiers following the first selection keyword following the second type clause are determined as the fourth selection clause; Determine whether the first identifier in the fourth selection clause is a selection keyword, so as to split the abstract syntax tree according to the condition of the first identifier.

7. The method for resolving data lineage according to any one of claims 3, 4, 5, and 6, characterized in that, Splitting the abstract syntax tree based on the first identifier includes: If the first identifier is a selection keyword, the clause located between the selection keyword and the second keyword is determined as a field domain clause, and the second identifier contained in the field domain clause is obtained; Determine whether the list of identifiers following the second keyword includes the third keyword; If the list of identifiers following the second keyword includes a third keyword, the clause located between the second keyword and the third keyword is identified as an equi-join clause; If the list of identifiers following the second keyword does not include the third keyword, the identifier following the second keyword is determined as the table retrieval clause.

8. The method for parsing data lineage according to claim 7, characterized in that, The parsing method also includes: If the first identifier is not a selection keyword, it is determined that the first level of the abstract syntax tree has been split. The lineage data of the structured query language script is determined based on the field field clause.

9. The method for parsing data lineage according to claim 8, characterized in that, The lineage data includes column-level lineage data, and determining the lineage data of the structured query language script based on the field clause includes: Determine whether the identifier in the field clause has an alias; If an identifier in the field clause has an alias, the current position and alias of the identifier with the alias are combined with any one of the first lineage data target table, the second lineage data target table, the third lineage data target table, and the fourth lineage data target table to form the right column of the column-level lineage. If there is no alias for the identifier in the field field clause, the field name of the identifier in the field field clause is combined with any one of the first lineage data target table, the second lineage data target table, the third lineage data target table, and the fourth lineage data target table to form the right column of the column-level lineage; The left column of the column lineage is determined based on the type of the first identifier in the field clause; The right column of the column-level lineage is combined with the left column of the column-level lineage to obtain the column-level lineage data.

10. The method for parsing data lineage according to claim 7, characterized in that, After the first level of the abstract syntax tree is split, the lineage data of the structured query language script is determined based on the equi-join clause and the table retrieval clause.

11. The method for parsing data lineage according to claim 9, characterized in that, The lineage data includes table-level lineage data, and determining the lineage data of the structured query language script based on the equi-join clause and the table retrieval clause includes: Obtain the database name and table name for each identifier in the equi-join clause and the table retrieval clause; Combine the database name and the table name into the left column of the table lineage; The left column is combined with any one of the first bloodline data target table, the second bloodline data target table, the third bloodline data target table, and the fourth bloodline data target table to obtain the table-level bloodline data.

12. The method for parsing data lineage according to claim 11, characterized in that, The parsing method also includes: Determine whether the first identifier in the equi-join clause and the table retrieval clause is a subquery; If the first identifier in the equi-join clause and the table retrieval clause is a subquery, the second level of the abstract syntax tree is parsed. If the first identifier in the equi-join clause and the table retrieval clause is not a subquery, it is determined that the abstract syntax tree has been completely split.

13. The method for parsing data lineage according to claim 12, characterized in that, Parsing the second level of the abstract syntax tree includes: Obtain the alias of the identifier in the subquery; The alias of the identifier in the subquery is determined as the lineage data target table of the first level of the abstract syntax tree, and the structured query statement in the subquery is determined as the second union query clause; The abstract syntax tree is then parsed based on the first identifier in the second joint query clause to determine if there is a second level.

14. A processor, characterized in that, It is configured to perform the data lineage resolution method according to any one of claims 1 to 13.

15. A machine-readable storage medium storing instructions thereon, characterized in that, When executed by a processor, this instruction causes the processor to be configured to perform the data lineage parsing method according to any one of claims 1 to 13.

Images