Field-level data lineage extraction method, device, equipment and storage medium

Abstract

The present invention relates to the field of artificial intelligence technology, and discloses a field-level data lineage extraction method, device, equipment and storage medium. The method includes: obtaining a target SQL statement by preprocessing the initial SQL statement, and parsing the target SQL statement into an abstract syntax tree; traversing multiple levels of nodes of the abstract syntax tree based on the parsing rules and parsing strategies of the abstract syntax tree to obtain the output fields of the target SQL statement; recursively backtracking the abstract syntax tree to obtain the physical table fields corresponding to each output field, and determining the correspondence between the output field and the physical table field; generating field lineage information between the output field and the physical table field according to the correspondence between the output field and the physical table field. This solution solves the technical problem that the prior art cannot cover multiple database systems and can accurately determine the lineage information between data by analyzing and extracting the lineage data of the output field of the input SQL statement.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a field-level data lineage extraction method, device, equipment and storage medium. Background Art

[0002] Data security control is one of the important ways to protect data security. It adopts the principle of least authorization and reduces the risk of data leakage by accurately controlling field-level data access permissions. One of the most important links in data security control is to trace the fields of user query results to obtain security control information of the actual tables corresponding to these fields, such as whether there is permission to access, whether encryption / desensitization is performed, etc., so as to take corresponding intervention measures.

[0003] Since data security management needs to cover all database systems of the enterprise, including relational databases such as MySQL, Oracle, and big data systems such as hive, spark, presto, etc., how to cover multiple database systems and accurately trace field-level query results to determine their corresponding physical tables and fields, and determine the lineage information between data has become a technical problem that technical personnel in this field need to solve. Summary of the invention

[0004] The main purpose of the present invention is to solve the technical problem that the prior art cannot cover multiple database systems and can accurately determine the blood relationship information between data by analyzing and extracting the blood relationship data of the output field of the input SQL statement.

[0005] The first aspect of the present invention provides a field-level data lineage extraction method, comprising: obtaining an initial SQL statement, and preprocessing the initial SQL statement to obtain a target SQL statement in a preset format; performing lexical analysis on the target SQL statement to generate an abstract syntax tree; based on the parsing rules and parsing strategies of the abstract syntax tree, traversing nodes of multiple levels of the abstract syntax tree to obtain output fields in the target SQL statement; recursively backtracing the abstract syntax tree to obtain physical table fields corresponding to the output fields, and determining the corresponding relationship between the output fields and the physical table fields; and generating field lineage information between the output fields and the physical table fields based on the corresponding relationship between the output fields and the physical table fields.

[0006] Optionally, in a first implementation method of the first aspect of the present invention, the obtaining of an initial SQL statement and preprocessing of the initial SQL statement to obtain a target SQL statement in a preset format includes: deleting useless data in the initial SQL statement based on a preset first regular expression, wherein the useless data includes comments, spaces, line breaks and terminators; deleting the paging operation limit code in the initial SQL statement; eliminating the life cycle in the initial SQL statement based on a preset second regular expression; and performing corresponding type of grammatical processing on the initial SQL statement based on a preset strategy pattern to obtain a target SQL statement in a preset format.

[0007] Optionally, in a second implementation of the first aspect of the present invention, the lexical parsing of the target SQL statement to generate an abstract syntax tree includes: obtaining a preset key field set, filtering out words that are consistent with the key fields in the key field set from the target SQL statement to obtain standard key fields; using the standard key fields to segment the target SQL statement to obtain multiple SQL subfields; and converting the multiple SQL subfields into an abstract syntax tree.

[0008] Optionally, in a third implementation manner of the first aspect of the present invention, the use of the standard key field to split the target SQL statement to obtain multiple SQL sub-fields includes: using the standard key field as a split node, splitting the target SQL statement to the left and to the right based on the split node to obtain multiple SQL sub-fields; based on the standard key field as a positioning point, performing a secondary split on the multiple split statements to obtain multiple SQL sub-fields.

[0009] Optionally, in a fourth implementation manner of the first aspect of the present invention, the parsing rules and parsing strategies based on the abstract syntax tree, traversing the nodes of multiple levels of the abstract syntax tree, and obtaining the output fields in the target SQL statement include: based on the parsing rules and parsing strategies of the abstract syntax tree, traversing the nodes of multiple levels of the abstract syntax tree, and obtaining the field data involved from the nodes of multiple levels of the abstract syntax tree; determining the input node, output node and intermediate conversion node according to the out-degree and in-degree; traversing all the output nodes to determine whether the target SQL statement of the output node contains a field value, if so, using the field in the target SQL statement as the output field, if not, obtaining all the fields of the table in the table output as the output field.

[0010] Optionally, in a fifth implementation manner of the first aspect of the present invention, generating field lineage information between the output field and the physical table field according to the correspondence between the output field and the physical table field includes: determining a lineage parser corresponding to the output field according to the type of a preset basic node Node in the abstract data tree; performing a field query on the basic node Node based on the lineage parser to obtain a corresponding query field list; traversing the query field list and the physical table field, and searching by recursive tracing to obtain the field lineage information between the output field and the physical table field.

[0011] Optionally, in the sixth implementation method of the first aspect of the present invention, after generating the field lineage information between the output field and the physical table field according to the correspondence between the output field and the physical table field, it also includes: inputting the field lineage information into the target SQL statement for data restoration to obtain the target physical field information corresponding to the output field.

[0012] The second aspect of the present invention provides a field-level data lineage extraction device, comprising: a preprocessing module, used to obtain an initial SQL statement, and preprocess the initial SQL statement to obtain a target SQL statement in a preset format; a parsing module, used to perform lexical parsing on the target SQL statement to generate an abstract syntax tree; a traversal module, used to traverse the nodes of multiple levels of the abstract syntax tree based on the parsing rules and parsing strategies of the abstract syntax tree to obtain the output fields of the target SQL statement; a determination module, used to recursively backtrack the abstract syntax tree to obtain the physical table fields corresponding to the output fields, and determine the corresponding relationship between the output fields and the physical table fields; a generation module, used to generate field lineage information between the output fields and the physical table fields according to the corresponding relationship between the output fields and the physical table fields.

[0013] Optionally, in a first implementation method of the second aspect of the present invention, the preprocessing module is specifically used to: delete useless data in the initial SQL statement based on a preset first regular expression, wherein the useless data includes comments, spaces, line breaks and terminators; delete the paging operation limit code in the initial SQL statement; eliminate the life cycle in the initial SQL statement based on a preset second regular expression; and perform corresponding type of grammatical processing on the initial SQL statement based on a preset strategy pattern to obtain a target SQL statement in a preset format.

[0014] Optionally, in a second implementation of the second aspect of the present invention, the parsing module includes: a screening unit, used to obtain a preset key field set, and screen out words that are consistent with the key fields in the key field set from the target SQL statement to obtain standard key fields; a segmentation unit, used to segment the target SQL statement using the standard key fields to obtain multiple SQL subfields; and a conversion unit, used to convert the multiple SQL subfields into an abstract syntax tree.

[0015] Optionally, in a third implementation of the second aspect of the present invention, the segmentation unit is specifically used to: use the standard key field as a segmentation node, segment the target SQL statement to the left and to the right based on the segmentation node to obtain multiple SQL sub-fields; based on the standard key field as a positioning point, perform secondary segmentation on the multiple segmentation statements to obtain multiple SQL sub-fields.

[0016] Optionally, in a fourth implementation manner of the second aspect of the present invention, the traversal module is specifically used to: traverse the nodes of multiple levels of the abstract syntax tree based on the parsing rules and parsing strategies of the abstract syntax tree, and obtain the field data involved from the nodes of multiple levels of the abstract syntax tree; determine the input node, output node and intermediate conversion node according to the out-degree and in-degree; traverse all the output nodes to determine whether the target SQL statement of the output node contains a field value, and if so, use the fields in the target SQL statement as output fields, and if not, obtain all the fields of the table in the table output as output fields.

[0017] Optionally, in a fifth implementation of the second aspect of the present invention, the generation module is specifically used to: determine the lineage parser corresponding to the output field according to the type of the preset basic node Node in the abstract data tree; based on the lineage parser, perform a field query on the basic node Node to obtain a corresponding query field list; traverse the query field list and the physical table fields, and search by recursive tracing to obtain the field lineage information between the output field and the physical table field.

[0018] Optionally, in a sixth implementation of the second aspect of the present invention, the field-level data lineage extraction device further includes: a data restoration module, used to input the field lineage information into the target SQL statement for data restoration to obtain target physical field information corresponding to the output field.

[0019] A third aspect of the present invention provides a field-level data lineage extraction device, comprising: a memory and at least one processor, wherein instructions are stored in the memory, and the memory and the at least one processor are interconnected via a line;

[0020] The at least one processor calls the instructions in the memory to enable the field-level data lineage extraction device to execute each step of the above-mentioned field-level data lineage extraction method.

[0021] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the various steps of the above-mentioned field-level data lineage extraction method.

[0022] In the technical solution provided by the present invention, the target SQL statement is obtained by preprocessing the initial SQL statement, and the target SQL statement is parsed into an abstract syntax tree; based on the parsing rules and parsing strategies of the abstract syntax tree, multiple levels of nodes of the abstract syntax tree are traversed to obtain the output fields of the target SQL statement; the abstract syntax tree is recursively backtracked to obtain the physical table fields corresponding to each output field, and the corresponding relationship between the output field and the physical table field is determined; according to the corresponding relationship between the output field and the physical table field, the field lineage information between the output field and the physical table field is generated. This solution solves the technical problem that the prior art cannot cover multiple database systems and can accurately determine the lineage information between data by analyzing and extracting the lineage data of the output field of the input SQL statement. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A schematic diagram of a first embodiment of the field-level data lineage extraction method provided by the present invention;

[0024] Figure 2 A schematic diagram of a second embodiment of the field-level data lineage extraction method provided by the present invention;

[0025] Figure 3 A schematic diagram of a third embodiment of the field-level data lineage extraction method provided by the present invention;

[0026] Figure 4 A schematic diagram of a fourth embodiment of the field-level data lineage extraction method provided by the present invention;

[0027] Figure 5 A schematic diagram of a fifth embodiment of the field-level data lineage extraction method provided by the present invention;

[0028] Figure 6 A schematic diagram of a first embodiment of a field-level data lineage extraction device provided by the present invention;

[0029] Figure 7 A schematic diagram of a second embodiment of the field-level data lineage extraction device provided by the present invention;

[0030] Figure 8 A schematic diagram of an embodiment of the field-level data lineage extraction device provided by the present invention. DETAILED DESCRIPTION

[0031] The embodiment of the present invention provides a field-level data lineage extraction method, device, equipment and storage medium. In the technical solution of the present invention, the target SQL statement is first obtained by preprocessing the initial SQL statement, and the target SQL statement is parsed into an abstract syntax tree; based on the parsing rules and parsing strategies of the abstract syntax tree, the nodes of multiple levels of the abstract syntax tree are traversed to obtain the output fields of the target SQL statement; the abstract syntax tree is recursively backtracked to obtain the physical table fields corresponding to each output field, and the corresponding relationship between the output field and the physical table field is determined; according to the corresponding relationship between the output field and the physical table field, the field lineage information between the output field and the physical table field is generated. This solution solves the technical problem that the prior art cannot cover multiple database systems and can accurately determine the lineage information between data by analyzing and extracting the lineage data of the output field of the input SQL statement.

[0032] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 The first embodiment of the field-level data lineage extraction method in the embodiment of the present invention includes:

[0034] 101. Obtain an initial SQL statement, and pre-process the initial SQL statement to obtain a target SQL statement in a preset format;

[0035] In this embodiment, the preprocessing is a system-customized setting for pre-processing the acquired initial SQL statement to be processed, which may include but is not limited to any one or more of the following combinations: using a first regular expression to remove useless information in the initial SQL statement, the useless information including at least one of the following: comments, spaces, line breaks and terminators; removing the paging operation limit code in the initial SQL statement; using a second regular expression to remove the life cycle in the SQL statement; using a preset strategy pattern to perform corresponding types of syntax processing on different types of initial SQL statements.

[0036] The target SQL statement is a standard SQL statement that complies with standard SQL syntax, for example, an SQL statement that does not include special symbols such as comments, spaces, terminators, and line breaks.

[0037] 102. Perform lexical analysis on the target SQL statement and generate an abstract syntax tree;

[0038] In this embodiment, the source code of the SQL statement is obtained, the character stream of the source code is read in sequence, and the source code is parsed to obtain an abstract syntax tree.

[0039] In another specific embodiment, the source code corresponding to the initial SQL statement is obtained, the source code is segmented based on the end symbol of the source code to obtain multiple lines of code corresponding to the source code; redundant characters in the multiple lines of code are removed to obtain multiple lines of processed code; and a syntax analyzer is used to perform semantic analysis on the processed multiple lines of code to obtain an abstract syntax tree.

[0040] Specifically, the source code of the SQL statement written by the developer is obtained, the source code is split into multiple lines of code using a semicolon terminator as a split node to form a code structure corresponding to the source code, redundant line breaks, indicators, and comment statements in the multiple lines of code are removed to obtain the processed multiple lines of code, and SQL semantic parsing is performed line by line on the processed multiple lines of code to obtain an abstract syntax tree, redundant characters are removed to improve the efficiency of semantic parsing, and the influence of redundant characters on semantic parsing is reduced, thereby improving the accuracy of semantic parsing, so that the obtained abstract syntax tree has a higher matching degree with the initial SQL statement.

[0041] 103. Based on the parsing rules and parsing strategies of the abstract syntax tree, traverse the nodes of multiple levels of the abstract syntax tree to obtain the output fields of the target SQL statement;

[0042] In this embodiment, based on the parsing rules and parsing strategies of the abstract syntax tree, multiple levels of nodes of the abstract syntax tree are traversed to obtain the output fields of the target SQL statement. Among them, the parsing rules and parsing strategies for multiple levels of nodes of the abstract syntax tree are pre-defined. After the parsing rules and parsing strategies of the abstract syntax tree are obtained, the nodes of the abstract syntax tree are analyzed based on the parsing rules and parsing strategies of the abstract syntax tree, from high-level nodes to low-level nodes until all nodes of the abstract syntax tree are traversed.

[0043] In another specific embodiment, the abstract syntax tree includes unary nodes, binary nodes and leaf nodes from high to low levels, and corresponding parsing rules are set for table objects and field objects in the abstract syntax tree. For objects of the same type, corresponding parsing strategies are set for unary nodes, binary nodes and leaf nodes. Based on the pre-set parsing rules and parsing strategies of the abstract syntax tree, different types of objects in the abstract syntax tree are analyzed in unary nodes, binary nodes and leaf nodes to obtain nodes of multiple levels.

[0044] Specifically, each of the multiple types of data corresponding to the SQL statement corresponds to a parsing rule, and the parsing rule includes different parsing strategies for nodes at different levels, and the parsing strategy is compatible with the grammatical rules of multiple types of SQL statements. Among them, the parsing rules and parsing strategies are backward compatible with other SQL grammar rules such as Presto. For different grammar rules, parsing strategies compatible with multiple grammar rules are pre-set to meet the parsing requirements of different types of data under multiple grammar rules, so as to improve the compatibility when parsing the abstract syntax tree.

[0045] 104. Recursively backtrack the abstract syntax tree to obtain the physical table field corresponding to each output field, and determine the corresponding relationship between the output field and the physical table field;

[0046] In this embodiment, the abstract syntax tree is recursively traced back to obtain the physical table fields corresponding to each output field, and the corresponding relationship between the output field and the physical table field is determined. The abstract syntax tree is recursively traced back to process each job node in turn, and the input field set, the output field set, and the field mapping information inside each job node are determined respectively. For the kettle script, Figure 2 As shown, determine the input field set and the output field set.

[0047] Specifically, the recursive backtracking is also called the recursive backtracking algorithm. The recursive backtracking algorithm is actually a search attempt process similar to enumeration. It mainly searches for the solution to the problem during the search attempt. When it is found that the solution conditions are no longer met, it "backtracks" and returns to try other paths. The backtracking method is a selection search method that searches forward according to the selection conditions to achieve the goal. However, when exploring a certain step, it is found that the original choice is not optimal or cannot achieve the goal, so it goes back one step and chooses again. This technique of going back and forth when it fails is called the backtracking method, and the point of a certain state that meets the backtracking conditions is called the "backtracking point". Many complex and large-scale problems can use the backtracking method, which is known as the "universal problem-solving method".

[0048] 105. According to the correspondence between the output field and the physical table field, the field lineage information between the output field and the physical table field is generated.

[0049] In this embodiment, field lineage information between the output field and the physical table field is generated according to the correspondence between the output field and the physical table field.

[0050] Specifically, the abstract syntax tree is traversed, and the nodes in the abstract syntax tree that are consistent with the nodes of the preset field are marked as target nodes; the data value in the target node is extracted, and a comparison query is performed based on the data value and the data table in the database to obtain the job table corresponding to the data value; the type of the job table is determined according to the label corresponding to the job table, the relationship between different types of job tables is determined, and the relationship between the job tables is summarized to obtain the blood relationship analysis result. In detail, in the embodiment of the present invention, the preset field is table, and the value corresponding to the preset field in the target node is extracted, such as: table A, then the corresponding value is A, and then the data table named A in the preset database is queried to obtain the corresponding job table.

[0051] Specifically, the type of the job table is determined according to the type of the label corresponding to the job table. For example, if the label type corresponding to the job table is the starting data table label, the job table is determined as the starting data table; if the label type corresponding to the data table is the target data table label, the job table is determined as the target data table. The blood relationship between the starting data table and the target data table is that the starting data table is the upstream data table of the target data table. All the starting data tables and the target data tables are summarized to obtain the blood relationship analysis result.

[0052] In the embodiment of the present invention, the target SQL statement is obtained by preprocessing the initial SQL statement, and the target SQL statement is parsed into an abstract syntax tree; based on the parsing rules and parsing strategies of the abstract syntax tree, multiple levels of nodes of the abstract syntax tree are traversed to obtain the output fields of the target SQL statement; the abstract syntax tree is recursively backtracked to obtain the physical table fields corresponding to each output field, and the corresponding relationship between the output field and the physical table field is determined; according to the corresponding relationship between the output field and the physical table field, the field lineage information between the output field and the physical table field is generated. This solution solves the technical problem that the prior art cannot cover multiple database systems and can accurately determine the lineage information between data by analyzing and extracting the lineage data of the output field of the input SQL statement.

[0053] See also Figure 2 The second embodiment of the field-level data lineage extraction method in the embodiment of the present invention includes:

[0054] 201. Deleting useless data in the initial SQL statement based on a preset first regular expression;

[0055] In this embodiment, the first regular expression is a custom setting in the programming language Java, which is a logical formula for string and special character operations. The present application uses the first regular expression to remove useless information in the SQL statement to be processed, and the useless information includes but is not limited to any one or more of the following combinations: comments, spaces, line breaks, terminators or other special characters / symbols.

[0056] 202. Delete the paging operation limit code in the initial SQL statement;

[0057] In this embodiment, the code snippet following the paging operation (limit) in the SQL statement to be processed is removed to prevent the failure to obtain the blood relationship between the data in the SQL statement.

[0058] 203. Eliminate the life cycle in the initial SQL statement based on the preset second regular expression;

[0059] In this embodiment, the life cycle in the initial SQL statement is removed based on a preset second regular expression. Specifically, the second regular expression is a custom setting in the programming language Java, which is a logical formula for string and special character operations. The first regular expression is different from the second regular expression. This application uses the second regular expression to remove the life cycle that may exist in the SQL statement to be processed.

[0060] Specifically, the regular expression, also known as a regular expression (English: Regular Expression, often abbreviated as regex, regexp or RE in code), is a concept in computer science. Regular expressions are usually used to retrieve and replace texts that meet a certain pattern (rule).

[0061] Many programming languages ​​support string manipulation using regular expressions. For example, Perl has a powerful regular expression engine built in. The concept of regular expressions was first popularized by Unix tools such as sed and grep. Regular expressions are often abbreviated to "regex", with the singular being regexp, regex, and the plural being regexps, regexes, and regexen.

[0062] 204. Based on the preset strategy mode, perform syntax processing of the corresponding type on the initial SQL statement to obtain a target SQL statement in a preset format;

[0063] In this embodiment, based on the preset strategy mode, the initial SQL statement is subjected to the corresponding type of grammatical processing to obtain the target SQL statement in the preset format. Specifically, the preset strategy mode is a system-defined setting, and the hadler method can be customized to perform corresponding special grammatical processing on different types of SQL statements. The present application can use the third regular expression indicated by the preset strategy mode to perform corresponding special grammatical processing on the SQL statement to be processed.

[0064] For example, this embodiment uses the following third regular expression to process the SQL statement to be processed: (i)(create\\s+(temp|TEMPORARY)\\s+table\\s+).*, which means that temp|TEMPORARY table in the SQL statement matching the regular expression is replaced with create table, so that subsequent operations can normally parse the blood relationship between data.

[0065] 205. Perform lexical analysis on the target SQL statement and generate an abstract syntax tree;

[0066] 206. Based on the parsing rules and parsing strategies of the abstract syntax tree, traverse the nodes of multiple levels of the abstract syntax tree to obtain the output fields of the target SQL statement;

[0067] 207. Recursively backtrack the abstract syntax tree to obtain the physical table field corresponding to each output field, and determine the corresponding relationship between the output field and the physical table field;

[0068] 208. Generate field lineage information between the output field and the physical table field according to the corresponding relationship between the output field and the physical table field.

[0069] Steps 205-208 in this embodiment are similar to steps 102-105 in the first embodiment and are not described again here.

[0070] In the embodiment of the present invention, the target SQL statement is obtained by preprocessing the initial SQL statement, and the target SQL statement is parsed into an abstract syntax tree; based on the parsing rules and parsing strategies of the abstract syntax tree, multiple levels of nodes of the abstract syntax tree are traversed to obtain the output fields of the target SQL statement; the abstract syntax tree is recursively backtracked to obtain the physical table fields corresponding to each output field, and the corresponding relationship between the output field and the physical table field is determined; according to the corresponding relationship between the output field and the physical table field, the field lineage information between the output field and the physical table field is generated. This solution solves the technical problem that the prior art cannot cover multiple database systems and can accurately determine the lineage information between data by analyzing and extracting the lineage data of the output field of the input SQL statement.

[0071] See also Figure 3 The third embodiment of the field-level data lineage extraction method in the embodiment of the present invention includes:

[0072] 301. Obtain an initial SQL statement, and pre-process the initial SQL statement to obtain a target SQL statement in a preset format;

[0073] 302. Obtain a preset key field set, filter out words that are consistent with the key fields in the key field set from the target SQL statement, and obtain a standard key field;

[0074] In this embodiment, a preset key field set is obtained, and words consistent with the key fields in the key field set are screened out from the target SQL statement to obtain a standard key field. Specifically, the key field set includes a starting key field set and a target key field set, wherein the starting key field set includes 'LEFT JOIN', 'RIGHTJOIN', 'LEFT OUTER JOIN', 'RIGHT OUTER JOIN', etc., and the target key field includes "CREATE", "INSERT", "SELECT" 'INTO', 'OVERWRITE'.

[0075] 303. Using the standard key field as a split node, split the target SQL statement to the left and to the right based on the split node to obtain multiple SQL subfields;

[0076] In this embodiment, the target SQL statement is segmented to the left and to the right based on the segmentation node using the standard key field as the segmentation node to obtain multiple SQL sub-fields. In particular, the SQL statement is segmented to the left and to the right based on the segmentation node using the standard key field as the segmentation node to obtain multiple SQL sub-statements; or the SQL statement is segmented based on a preset random segmentation length to obtain multiple segmentation statements, and the multiple segmentation statements are segmented again using the standard key field as a positioning point to obtain multiple SQL sub-statements.

[0077] For example, the SQL statement is select col_a from A, and the standard key fields are "select" and "from", so the standard key fields "select" and "from" are used as segmentation nodes to obtain two SQL sub-statements, select col_a and from A.

[0078] 304. Based on the standard key field as the positioning point, multiple segmentation statements are segmented twice to obtain multiple SQL subfields;

[0079] In this embodiment, based on the standard key field as the positioning point, multiple segmentation statements are segmented twice to obtain multiple SQL subfields. Wherein, the segmentation is to segment words. Chinese word segmentation (also known as Chinese word segmentation, Chinese Word Segmentation) refers to the segmentation of a sequence of Chinese characters into individual words. Chinese word segmentation is the basis of text mining. For a piece of Chinese input, successful Chinese word segmentation can achieve the effect of automatic computer recognition of the meaning of the sentence. This method is also called the mechanical word segmentation method. It matches the Chinese character string to be analyzed with the entries in a "sufficiently large" machine dictionary according to a certain strategy. If a certain string is found in the dictionary, the match is successful (a word is recognized).

[0080] 305. Convert multiple SQL subfields into an abstract syntax tree;

[0081] In this embodiment, the SQL statement is segmented based on a preset random segmentation length to obtain multiple segmentation statements, and the multiple segmentation fields are segmented twice using the standard keywords as positioning points to obtain multiple SQL subfields. For example, the SQL field is select col_a from A, and the standard keywords are "select" and "from", so the standard keywords "select" and "from" are used as segmentation nodes for segmentation to obtain two SQL subfields, select col_a and fromA.

[0082] Specifically, after the SQL field is segmented by the standard keyword to obtain multiple SQL subfields, the method further includes: labeling the multiple SQL subfields. For example, the keyword corresponding to the SQL subfield selectcol_a is a keyword in the target keyword set, so the label corresponding to the SQL subfield select col_a is the target label.

[0083] Furthermore, the converting of the plurality of SQL subfields into an abstract syntax tree comprises: using a preset lexical analyzer to analyze the SQL substatement to obtain a plurality of tokens; and constructing a syntax tree for the plurality of tokens according to a preset syntax analysis method to obtain the abstract syntax tree. The preset syntax analysis method comprises a top-down analysis method and a bottom-up analysis method.

[0084] 306. Based on the parsing rules and parsing strategies of the abstract syntax tree, traverse the nodes of multiple levels of the abstract syntax tree to obtain the output fields of the target SQL statement;

[0085] 307. Recursively backtrack the abstract syntax tree to obtain the physical table field corresponding to each output field, and determine the corresponding relationship between the output field and the physical table field;

[0086] 308. Generate field lineage information between the output field and the physical table field according to the corresponding relationship between the output field and the physical table field.

[0087] Steps 301, 306-308 in this embodiment are similar to steps 101, 103-105 in the first embodiment, and are not described again here.

[0088] In the embodiment of the present invention, the target SQL statement is obtained by preprocessing the initial SQL statement, and the target SQL statement is parsed into an abstract syntax tree; based on the parsing rules and parsing strategies of the abstract syntax tree, multiple levels of nodes of the abstract syntax tree are traversed to obtain the output fields of the target SQL statement; the abstract syntax tree is recursively backtracked to obtain the physical table fields corresponding to each output field, and the corresponding relationship between the output field and the physical table field is determined; according to the corresponding relationship between the output field and the physical table field, the field lineage information between the output field and the physical table field is generated. This solution solves the technical problem that the prior art cannot cover multiple database systems and can accurately determine the lineage information between data by analyzing and extracting the lineage data of the output field of the input SQL statement.

[0089] See also Figure 4 The fourth embodiment of the field-level data lineage extraction method in the embodiment of the present invention includes:

[0090] 401. Obtain an initial SQL statement, and pre-process the initial SQL statement to obtain a target SQL statement in a preset format;

[0091] 402. Perform lexical analysis on the target SQL statement and generate an abstract syntax tree;

[0092] 403. Based on the parsing rules and parsing strategies of the abstract syntax tree, traverse the nodes of multiple levels of the abstract syntax tree, and obtain the field data involved from the nodes of multiple levels of the abstract syntax tree;

[0093] In this embodiment, based on the parsing rules and parsing strategies of the abstract syntax tree, the nodes of multiple levels of the abstract syntax tree are traversed, and the field data involved are obtained from the nodes of multiple levels of the abstract syntax tree. Specifically, each of the multiple types of data corresponding to the SQL statement corresponds to a parsing rule, and the parsing rules include different parsing strategies for nodes at different levels, and the parsing strategies are compatible with the grammatical rules of multiple types of SQL statements. Among them, the parsing rules and parsing strategies are backward compatible with other SQL grammar rules such as Presto. For different grammar rules, parsing strategies compatible with multiple grammar rules are pre-set to meet the parsing requirements of different types of data under multiple grammar rules, so as to improve the compatibility when parsing the abstract syntax tree.

[0094] 404. Determine the input node, output node, and intermediate conversion node according to the out-degree and the in-degree;

[0095] In this embodiment, input nodes, output nodes, and intermediate transformation nodes are determined according to the out-degree and in-degree. Specifically, according to the out-degree and in-degree of the operation node in the acyclic directed graph, the operation node is divided into input nodes, output nodes, and intermediate transformation nodes. For input nodes, it is an operation node with a degree greater than 0 and an in-degree of 0. For output nodes, it is an operation node with a degree of 0 and an in-degree greater than 0. For an operation node with both an out-degree and an in-degree greater than 0, it is an intermediate transformation node.

[0096] 405. Traverse all output nodes to determine whether the target SQL statement of the output node contains field values. If so, use the fields in the target SQL statement as output fields. If not, obtain all fields of the table in the table output as output fields.

[0097] In this embodiment, all output nodes are traversed to determine whether the target SQL statement of the output node contains field values. If so, the fields in the target SQL statement are used as output fields. If not, all the fields of the table in the table output are obtained as output fields. Specifically, the core of generating a data lineage graph is to map between fields. Therefore, the present invention processes the input nodes, output nodes, and intermediate conversion nodes respectively to determine the corresponding input fields, output fields, and the mapping relationship between input fields and output fields. For input nodes. This embodiment obtains the SQL statement of the corresponding node to determine whether the SQL statement contains field values. The field values ​​in the SQL statement corresponding to the input node are directly used as input fields. When the SQL statement does not contain field values ​​and uses a full match, this embodiment directly obtains all the fields of the table as input fields based on the table input in the SQL statement.

[0098] Specifically, this embodiment determines the corresponding operation table based on the SQL statement, dynamically connects to the corresponding database according to the database connection information, and obtains all fields of the current table from the database as input fields. The present invention traverses all input nodes and forms an input field set with the input fields obtained by processing each input node.

[0099] 406. Recursively backtrack the abstract syntax tree to obtain the physical table field corresponding to each output field, and determine the corresponding relationship between the output field and the physical table field;

[0100] 407. Generate field lineage information between the output field and the physical table field according to the corresponding relationship between the output field and the physical table field.

[0101] Steps 401-402, 406-407 in this embodiment are similar to steps 101-102, 104-105 in the first embodiment, and are not described again here.

[0102] In the embodiment of the present invention, the target SQL statement is obtained by preprocessing the initial SQL statement, and the target SQL statement is parsed into an abstract syntax tree; based on the parsing rules and parsing strategies of the abstract syntax tree, multiple levels of nodes of the abstract syntax tree are traversed to obtain the output fields of the target SQL statement; the abstract syntax tree is recursively backtracked to obtain the physical table fields corresponding to each output field, and the corresponding relationship between the output field and the physical table field is determined; according to the corresponding relationship between the output field and the physical table field, the field lineage information between the output field and the physical table field is generated. This solution solves the technical problem that the prior art cannot cover multiple database systems and can accurately determine the lineage information between data by analyzing and extracting the lineage data of the output field of the input SQL statement.

[0103] See also Figure 5The fifth embodiment of the field-level data lineage extraction method in the embodiment of the present invention includes:

[0104] 501. Obtain an initial SQL statement, and pre-process the initial SQL statement to obtain a target SQL statement in a preset format;

[0105] 502. Perform lexical analysis on the target SQL statement to generate an abstract syntax tree;

[0106] 503. Based on the parsing rules and parsing strategies of the abstract syntax tree, traverse the nodes of multiple levels of the abstract syntax tree to obtain the output fields of the target SQL statement;

[0107] 504. Recursively backtrack the abstract syntax tree to obtain the physical table field corresponding to each output field, and determine the corresponding relationship between the output field and the physical table field;

[0108] 505. Determine the bloodline parser corresponding to the output field according to the type of the preset basic node Node in the abstract data tree;

[0109] In this embodiment, the bloodline parser corresponding to the output field is determined according to the type of the preset basic node Node in the abstract data tree. Specifically, the corresponding bloodline parser can be selected according to the type of the basic node Node to perform bloodline relationship analysis. It can be understood that bloodline relationships usually exist in InsertNode and CreateNode. Therefore, if the type of the basic node Node is InsertNode, the bloodline parser corresponding to the InsertNode is selected; conversely, if the type of the basic node Node is CreateNode, the bloodline parser corresponding to the CreateNode is selected.

[0110] 506. Based on the lineage resolver, perform a field query on the basic node Node to obtain a corresponding query field list;

[0111] In this embodiment, based on the bloodline parser, a field query is performed on the basic node Node to obtain a corresponding query field list. The query field list includes the main table and the field information corresponding to the main table. The main table here can be an insert table or a create table. The field and list query includes but is not limited to any one or more of the following combinations: regular / ordinary query, subquery, union query and function field query. This application can select the corresponding query method according to the actual situation of the basic node Node to perform query processing to obtain the final query field list.

[0112] 507. Traverse the query field list and the physical table fields, and search by recursive tracing to obtain the field lineage information between the output field and the physical table field;

[0113] In this embodiment, the query field list and the physical table fields are traversed, and the field lineage information between the output field and the physical table field is found by recursive tracing. Specifically, the query field list can be traversed in a loop to find the main table and the field information of the main table included in the list. Then, the lineage relationship between the corresponding data is found from the main table and the field information of the main table in the query field list by recursive tracing. The lineage relationship includes but is not limited to at least one of the following: table-level lineage relationship, field-level lineage relationship, or other-level lineage relationship, etc.

[0114] 508. Input the field lineage information into the target SQL statement to restore the data, and obtain the target physical field information corresponding to the output field.

[0115] Steps 501-504 in this embodiment are similar to steps 101-105 in the first embodiment and will not be described in detail here.

[0116] In an embodiment of the present invention, the target SQL statement is obtained by preprocessing the initial SQL statement, and the target SQL statement is parsed into an abstract syntax tree; based on the parsing rules and parsing strategies of the abstract syntax tree, the nodes of multiple levels of the abstract syntax tree are traversed to obtain the output fields of the target SQL statement; the abstract syntax tree is recursively backtracked to obtain the physical table fields corresponding to each output field, and the corresponding relationship between the output field and the physical table field is determined; according to the corresponding relationship between the output field and the physical table field, the field lineage information between the output field and the physical table field is generated. This solution solves the technical problem that the prior art cannot cover multiple database systems and can accurately determine the lineage information between data by analyzing and extracting the lineage data of the output field of the input SQL statement.

[0117] The above describes the field level data lineage extraction method in the embodiment of the present invention. The following describes the field level data lineage extraction device in the embodiment of the present invention. Figure 6 The first embodiment of the field-level data lineage extraction device in the embodiment of the present invention includes:

[0118] The preprocessing module 601 is used to obtain an initial SQL statement and preprocess the initial SQL statement to obtain a target SQL statement in a preset format;

[0119] A parsing module 602 is used to perform lexical analysis on the target SQL statement to generate an abstract syntax tree;

[0120] A traversal module 603 is used to traverse the nodes of multiple levels of the abstract syntax tree based on the parsing rules and parsing strategies of the abstract syntax tree to obtain the output fields of the target SQL statement;

[0121] A determination module 604 is used to recursively backtrack the abstract syntax tree to obtain the physical table fields corresponding to the output fields, and determine the corresponding relationship between the output fields and the physical table fields;

[0122] The generating module 605 is used to generate field lineage information between the output field and the physical table field according to the corresponding relationship between the output field and the physical table field.

[0123] In the embodiment of the present invention, the target SQL statement is obtained by preprocessing the initial SQL statement, and the target SQL statement is parsed into an abstract syntax tree; based on the parsing rules and parsing strategies of the abstract syntax tree, multiple levels of nodes of the abstract syntax tree are traversed to obtain the output fields of the target SQL statement; the abstract syntax tree is recursively backtracked to obtain the physical table fields corresponding to each output field, and the corresponding relationship between the output field and the physical table field is determined; according to the corresponding relationship between the output field and the physical table field, the field lineage information between the output field and the physical table field is generated. This solution solves the technical problem that the prior art cannot cover multiple database systems and can accurately determine the lineage information between data by analyzing and extracting the lineage data of the output field of the input SQL statement.

[0124] See also Figure 7 The second embodiment of the field-level data lineage extraction device in the embodiment of the present invention, the field-level data lineage extraction device specifically comprises:

[0125] The preprocessing module 601 is used to obtain an initial SQL statement and preprocess the initial SQL statement to obtain a target SQL statement in a preset format;

[0126] A parsing module 602 is used to perform lexical analysis on the target SQL statement to generate an abstract syntax tree;

[0127] A traversal module 603 is used to traverse the nodes of multiple levels of the abstract syntax tree based on the parsing rules and parsing strategies of the abstract syntax tree to obtain the output fields of the target SQL statement;

[0128] A determination module 604 is used to recursively backtrack the abstract syntax tree to obtain the physical table fields corresponding to the output fields, and determine the corresponding relationship between the output fields and the physical table fields;

[0129] The generating module 605 is used to generate field lineage information between the output field and the physical table field according to the corresponding relationship between the output field and the physical table field.

[0130] In this embodiment, the preprocessing module 601 is specifically used for:

[0131] Deleting useless data in the initial SQL statement based on a preset first regular expression, wherein the useless data includes comments, spaces, line breaks, and terminators;

[0132] Delete the paging operation limit code in the initial SQL statement;

[0133] Eliminate the life cycle in the initial SQL statement based on a preset second regular expression;

[0134] Based on the preset strategy mode, the initial SQL statement is subjected to a corresponding type of grammatical processing to obtain a target SQL statement in a preset format.

[0135] In this embodiment, the parsing module 602 includes:

[0136] The screening unit 6021 is used to obtain a preset key field set, and screen out words consistent with the key fields in the key field set from the target SQL statement to obtain standard key fields;

[0137] A segmentation unit 6022 is used to segment the target SQL statement using the standard key field to obtain multiple SQL subfields;

[0138] The conversion unit 6023 is used to convert the multiple SQL subfields into an abstract syntax tree.

[0139] In this embodiment, the segmentation unit 6022 is specifically used for:

[0140] Using the standard key field as a split node, the target SQL statement is split to the left and to the right based on the split node to obtain multiple SQL subfields.

[0141] Based on the standard key field as a positioning point, the multiple segmentation statements are split twice to obtain multiple SQL subfields.

[0142] In this embodiment, the traversal module 603 is specifically used for:

[0143] Based on the parsing rules and parsing strategies of the abstract syntax tree, traverse the nodes of multiple levels of the abstract syntax tree, and obtain the field data involved from the nodes of multiple levels of the abstract syntax tree;

[0144] Determine the input node, output node and intermediate transformation node according to the out-degree and in-degree;

[0145] Traverse all the output nodes to determine whether the target SQL statement of the output node contains field values. If so, use the fields in the target SQL statement as output fields. If not, obtain all the fields of the table in the table output as output fields.

[0146] In this embodiment, the generating module 605 is specifically used for:

[0147] Determine the blood relationship parser corresponding to the output field according to the type of the preset basic node Node in the abstract data tree;

[0148] Based on the bloodline parser, perform a field query on the basic node Node to obtain a corresponding query field list;

[0149] The query field list and the physical table fields are traversed, and the field lineage information between the output field and the physical table field is obtained by searching in a recursive tracing manner.

[0150] In this embodiment, the field-level data lineage extraction device further includes:

[0151] The data restoration module 606 is used to input the field lineage information into the target SQL statement to perform data restoration and obtain the target physical field information corresponding to the output field.

[0152] In the embodiment of the present invention, the target SQL statement is obtained by preprocessing the initial SQL statement, and the target SQL statement is parsed into an abstract syntax tree; based on the parsing rules and parsing strategies of the abstract syntax tree, multiple levels of nodes of the abstract syntax tree are traversed to obtain the output fields of the target SQL statement; the abstract syntax tree is recursively backtracked to obtain the physical table fields corresponding to each output field, and the corresponding relationship between the output field and the physical table field is determined; according to the corresponding relationship between the output field and the physical table field, the field lineage information between the output field and the physical table field is generated. This solution solves the technical problem that the prior art cannot cover multiple database systems and can accurately determine the lineage information between data by analyzing and extracting the lineage data of the output field of the input SQL statement.

[0153] above Figure 6 and Figure 7 The field-level data lineage extraction device in the embodiment of the present invention is described in detail from the perspective of modular functional entities. The field-level data lineage extraction device in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0154] Figure 81 is a schematic diagram of the structure of a field-level data lineage extraction device provided by an embodiment of the present invention. The field-level data lineage extraction device 800 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 810 (for example, one or more processors) and a memory 820, and one or more storage media 830 (for example, one or more massive storage devices) storing application programs 833 or data 832. Among them, the memory 820 and the storage medium 830 may be temporary storage or permanent storage. The program stored in the storage medium 830 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the field-level data lineage extraction device 800. Furthermore, the processor 810 may be configured to communicate with the storage medium 830, and execute a series of instruction operations in the storage medium 830 on the field-level data lineage extraction device 800 to implement the steps of the field-level data lineage extraction method provided by the above-mentioned method embodiments.

[0155] The field-level data lineage extraction device 800 may also include one or more power supplies 840, one or more wired or wireless network interfaces 850, one or more input and output interfaces 860, and / or one or more operating systems 831, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will appreciate that Figure 8 The structure of the field-level data lineage extraction device shown does not constitute a limitation on the field-level data lineage extraction device provided in the present application, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0156] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the above-mentioned field-level data lineage extraction method.

[0157] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0158] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0159] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for extracting lineage from field-level data, characterized in that: The field-level data lineage extraction method comprises: Obtaining an initial SQL statement, and preprocessing the initial SQL statement to obtain a target SQL statement in a preset format; Performing lexical analysis on the target SQL statement to generate an abstract syntax tree; Based on the parsing rules and parsing strategies of the abstract syntax tree, traverse the nodes of multiple levels of the abstract syntax tree, and obtain the field data involved from the nodes of multiple levels of the abstract syntax tree; Determine the input node, output node and intermediate transformation node according to the out-degree and in-degree, wherein the output node is a job node with an out-degree of 0 and an in-degree greater than 0; Traversing all the output nodes, determining whether the target SQL statement of the output node contains field values, if so, taking the fields in the target SQL statement as output fields, if not, obtaining all the fields of the table in the table output as output fields; Recursively backtracking the abstract syntax tree to obtain physical table fields corresponding to each of the output fields, and determining a corresponding relationship between the output fields and the physical table fields; Generate field lineage information between the output field and the physical table field according to the corresponding relationship between the output field and the physical table field; The generating, according to the correspondence between the output field and the physical table field, field lineage information between the output field and the physical table field comprises: Determine the blood relationship parser corresponding to the output field according to the type of the preset basic node Node in the abstract data tree; Based on the bloodline parser, perform a field query on the basic node Node to obtain a corresponding query field list; Traversing the query field list and the physical table fields, and searching by recursive tracing, to obtain field lineage information between the output field and the physical table field; The step of determining the blood relationship parser corresponding to the output field according to the type of the preset basic node Node in the abstract data tree includes: If the type of the basic node Node is InsertNode, select the blood relationship parser corresponding to the InsertNode; If the type of the basic node Node is CreateNode, a bloodline parser corresponding to the CreateNode is selected.

2. The field-level data lineage extraction method according to claim 1, characterized in that: The obtaining of the initial SQL statement and preprocessing the initial SQL statement to obtain a target SQL statement in a preset format includes: Deleting useless data in the initial SQL statement based on a preset first regular expression, wherein the useless data includes comments, spaces, line breaks, and terminators; Delete the paging operation limit code in the initial SQL statement; Eliminate the life cycle in the initial SQL statement based on a preset second regular expression; Based on the preset strategy mode, the initial SQL statement is subjected to a corresponding type of grammatical processing to obtain a target SQL statement in a preset format.

3. The field-level data lineage extraction method according to claim 1, characterized in that: The lexical analysis of the target SQL statement to generate an abstract syntax tree includes: Obtain a preset key field set, filter out words consistent with key fields in the key field set from the target SQL statement, and obtain standard key fields; Using the standard key field to segment the target SQL statement to obtain multiple SQL subfields; The plurality of SQL subfields are converted into an abstract syntax tree.

4. The field-level data lineage extraction method according to claim 3 is characterized in that: The target SQL statement is segmented by using the standard key field to obtain multiple SQL subfields, including: Using the standard key field as a splitting node, splitting the target SQL statement to the left and to the right based on the splitting node to obtain multiple SQL subfields; Based on the standard key field as a positioning point, the multiple segmentation statements are split twice to obtain multiple SQL subfields.

5. The field-level data lineage extraction method according to any one of claims 1 to 4, characterized in that: After generating the field lineage information between the output field and the physical table field according to the correspondence between the output field and the physical table field, the method further includes: The field lineage information is input into the target SQL statement to perform data restoration, and the target physical field information corresponding to the output field is obtained.

6. A field-level data lineage extraction device, characterized in that: The field-level data lineage extraction device comprises: A preprocessing module is used to obtain an initial SQL statement and preprocess the initial SQL statement to obtain a target SQL statement in a preset format; A parsing module, used for performing lexical analysis on the target SQL statement and generating an abstract syntax tree; A traversal module, configured to traverse nodes of multiple levels of the abstract syntax tree based on the parsing rules and parsing strategies of the abstract syntax tree, and obtain the field data involved from the nodes of multiple levels of the abstract syntax tree; Determine the input node, output node and intermediate transformation node according to the out-degree and in-degree, wherein the output node is a job node with an out-degree of 0 and an in-degree greater than 0; Traversing all the output nodes, determining whether the target SQL statement of the output node contains field values, if so, taking the fields in the target SQL statement as output fields, if not, obtaining all the fields of the table in the table output as output fields; A determination module, configured to recursively backtrack the abstract syntax tree to obtain a physical table field corresponding to each of the output fields, and determine a corresponding relationship between the output field and the physical table field; A generating module, used for generating field lineage information between the output field and the physical table field according to the corresponding relationship between the output field and the physical table field; The generation module is specifically used to: determine the lineage parser corresponding to the output field according to the type of the preset basic node Node in the abstract data tree; perform field query on the basic node Node based on the lineage parser to obtain the corresponding query field list; traverse the query field list and the physical table field, and search by recursive tracing to obtain the field lineage information between the output field and the physical table field; The step of determining the blood relationship parser corresponding to the output field according to the type of the preset basic node Node in the abstract data tree includes: If the type of the basic node Node is InsertNode, select the blood relationship parser corresponding to the InsertNode; If the type of the basic node Node is CreateNode, a bloodline parser corresponding to the CreateNode is selected.

7. The field-level data lineage extraction device according to claim 6, characterized in that: The preprocessing module is specifically used to: delete useless data in the initial SQL statement based on a preset first regular expression, wherein the useless data includes comments, spaces, line breaks and terminators; delete the paging operation limit code in the initial SQL statement; eliminate the life cycle in the initial SQL statement based on a preset second regular expression; and perform corresponding type of grammatical processing on the initial SQL statement based on a preset strategy mode to obtain a target SQL statement in a preset format.

8. The field-level data lineage extraction device according to claim 6, characterized in that: The field-level data lineage extraction device further includes: a data restoration module, which is used to input the field lineage information into the target SQL statement to perform data restoration and obtain the target physical field information corresponding to the output field.

9. A field-level data lineage extraction device, characterized in that: The field-level data lineage extraction device comprises: a memory and at least one processor, the memory stores instructions, and the memory and the at least one processor are interconnected via a line; The at least one processor calls the instructions in the memory to enable the field-level data lineage extraction device to execute each step of the field-level data lineage extraction method as described in any one of claims 1-5.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the field-level data lineage extraction method as described in any one of claims 1 to 5 is implemented.

Images