Data analysis-based data blood relationship graph construction method and related device

Abstract

The application provides a data blood relationship graph construction method and device based on data analysis, an electronic device and a storage medium. The data blood relationship graph construction method based on data analysis comprises the following steps: collecting metadata and SQL statements in a relational database to obtain a metadata set and an SQL statement data set; dividing the SQL statement data set to obtain a query statement data set and a non-query statement data set; performing lexical analysis and syntax analysis on the query statement data set to construct an abstract syntax tree; generating a field blood path based on the metadata set, the non-query statement data set and the abstract syntax tree; and constructing a data blood relationship graph based on the field blood path. The application uses the abstract syntax tree to parse the SQL statement, and generates the field blood path in combination with the non-query statement set and the metadata set. Finally, the field blood path can be clearly displayed in the data blood relationship graph, so that the accuracy and reliability of the obtained data blood relationship graph are improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, electronic device and storage medium for constructing a data lineage graph based on data analysis. Background Technology

[0002] With the development of big data, the tables and fields in databases are experiencing explosive growth. Massive and complex data naturally form data lineages through operations such as flow, mapping, and aggregation. Therefore, data analysis can help to organize, store, and visualize these lineages, which is crucial for data tracing, assessing data quality, and identifying data value.

[0003] However, due to the flexible and varied syntax of database SQL statements, deep nesting structures, and non-standard SQL script writing, existing technologies face significant challenges in extracting field-level lineage from SQL scripts, resulting in inaccurate extracted field-level lineage relationships. Summary of the Invention

[0004] In view of the above, it is necessary to propose a data lineage graph construction method and related equipment based on data analysis to solve the technical problem of how to improve the accuracy of extracted field-level lineage relationships. The related equipment includes a data lineage graph construction device based on data analysis, electronic equipment, and storage media.

[0005] This application provides a method for constructing a data lineage graph based on data analysis, the method comprising:

[0006] Metadata and SQL statements are collected from relational databases to obtain metadata datasets and SQL statement datasets;

[0007] The SQL statement dataset is divided into query statement dataset and non-query statement dataset;

[0008] Lexical and syntactic analysis are performed on the query statement dataset to construct an abstract syntax tree;

[0009] Generate field lineage paths based on the aforementioned meta-dataset, non-query statement dataset, and abstract syntax tree;

[0010] A data lineage graph is constructed based on the lineage path described in the field.

[0011] In some embodiments, the process of collecting metadata and SQL statements from a relational database to obtain a metadata dataset and an SQL statement dataset includes:

[0012] Threads are started to connect to a preset relational database according to a preset cycle;

[0013] Based on the thread, metadata and SQL statements in the relational database are collected to obtain a metadata dataset and an SQL statement dataset.

[0014] In some embodiments, dividing the SQL statement dataset into query statement datasets and non-query statement datasets includes:

[0015] The comment lines in the SQL statement dataset are identified using regular expressions, and the comment lines are replaced to obtain the first statement relation dataset;

[0016] The first statement relation dataset is filtered using regular expressions and preset keywords to obtain the second statement relation dataset;

[0017] Based on the preset keywords, the second statement relation dataset is divided into a query statement dataset and a non-query statement dataset.

[0018] In some embodiments, performing lexical and syntactic analysis on the query statement dataset to construct an abstract syntax tree includes:

[0019] Lexical analysis is performed on the SQL statements in the query statement dataset to obtain the position of each word in the SQL statement;

[0020] The meaning of each word in the SQL statement and the structure of the SQL statement are obtained through syntactic analysis.

[0021] An abstract syntax tree is constructed based on the position, meaning, and structure of each word in the SQL statement. The abstract syntax tree includes a root node and leaf nodes.

[0022] In some embodiments, generating field lineage paths based on the meta-dataset, the non-query statement dataset, and the abstract syntax tree includes:

[0023] Based on the metadata, obtain the selected fields and the table information of the SQL statement represented by the leaf node to obtain the clear leaf node;

[0024] Traverse the clear leaf nodes and root node in the abstract syntax tree to obtain the lineage base field;

[0025] Based on the non-query statement dataset, the blood relationship basic fields are matched to obtain the blood relationship positioning fields;

[0026] Multiple bloodline paths are constructed based on the bloodline location field.

[0027] In some embodiments, traversing the clear leaf nodes and root node in the abstract syntax tree to obtain the lineage base field includes:

[0028] The basic kinship field corresponding to the target leaf node is obtained by retaining the fields that exist in both the target leaf node and its parent node. The target leaf node is any one of the clear leaf nodes.

[0029] By traversing the clear leaf nodes and root node in the abstract syntax tree sequentially from the clear leaf node to the root node, the lineage relationship basic field corresponding to each clear leaf node is obtained.

[0030] In some embodiments, the data lineage graph includes a field-level lineage graph and a table-level lineage graph, and constructing the data lineage graph based on the field lineage path includes:

[0031] Construct a field-level lineage graph using each field in the lineage path as a vertex and the corresponding SQL statement as a directed edge;

[0032] A table-level lineage graph is constructed using the tables containing each field in the lineage path as vertices and the SQL statements corresponding to each field as directed edges.

[0033] This application embodiment also provides a data kinship graph construction device based on data analysis, the device comprising:

[0034] The acquisition unit is used to acquire metadata and SQL statements from relational databases to obtain metadata datasets and SQL statement datasets;

[0035] A partitioning unit is used to partition the SQL statement dataset to obtain a query statement dataset and a non-query statement dataset;

[0036] The analysis unit is used to perform lexical and syntactic analysis on the query statement dataset to construct an abstract syntax tree;

[0037] The generation unit is used to generate field lineage paths based on the meta-dataset, the non-query statement dataset, and the abstract syntax tree;

[0038] A construction unit is used to construct a data lineage graph based on the lineage path of the field.

[0039] This application embodiment also provides an electronic device, the electronic device comprising:

[0040] Memory, storing at least one instruction;

[0041] The processor executes the instructions stored in the memory to implement the data lineage graph construction method based on data analysis.

[0042] This application also provides a computer-readable storage medium storing at least one instruction, which is executed by a processor in an electronic device to implement the data lineage graph construction method based on data analysis.

[0043] This application parses SQL statements by constructing an abstract syntax tree and generates field lineage paths by combining non-query statement sets and meta datasets. Ultimately, the field lineage paths can be clearly represented in the data lineage graph, thereby improving the accuracy and reliability of the obtained data lineage graph. Attached Figure Description

[0044] Figure 1 This is a flowchart of a preferred embodiment of the data lineage graph construction method based on data analysis involved in this application.

[0045] Figure 2 This is a functional block diagram of a preferred embodiment of the data lineage diagram construction device based on data analysis involved in this application.

[0046] Figure 3 This is a schematic diagram of the structure of an electronic device according to a preferred embodiment of the data lineage graph construction method based on data analysis involved in this application. Detailed Implementation

[0047] To better understand the purpose, features, and advantages of this application, a detailed description of the application is provided below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of this application can be combined with each other. Numerous specific details are set forth in the following description to provide a thorough understanding of this application; the described embodiments are only a part of the embodiments of this application, and not all of them.

[0048] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0050] This application provides a data lineage graph construction method based on data analysis, which can be applied to one or more electronic devices. An electronic device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0051] Electronic devices can be any electronic product that allows human-computer interaction with a customer, such as personal computers, tablets, smartphones, personal digital assistants (PDAs), game consoles, interactive network television (IPTV), smart wearable devices, etc.

[0052] Electronic devices may also include network devices and / or client devices. The network devices include, but are not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0053] The networks in which electronic devices are located include, but are not limited to, the Internet, wide area networks, metropolitan area networks, local area networks, and virtual private networks (VPNs).

[0054] like Figure 1 The diagram shown is a flowchart of a preferred embodiment of the data lineage graph construction method based on data analysis according to this application. The order of steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements.

[0055] S10: Collect metadata and SQL statements from the relational database to obtain the metadata dataset and SQL statement dataset.

[0056] In an optional embodiment, the acquisition of metadata and SQL statements from the relational database to obtain the metadata dataset and SQL statement dataset includes:

[0057] Threads are started to connect to a preset relational database according to a preset cycle;

[0058] Based on the thread, metadata and SQL statements in the relational database are collected to obtain a metadata dataset and an SQL statement dataset.

[0059] In this optional embodiment, since this solution mainly extracts data lineage from SQL statements, it is necessary to use a relational database that includes SQL statements. The relational database can be MySQL, PostgreSQL, Oracle, openGauss, etc. This solution does not restrict the specific relational database.

[0060] In this optional embodiment, a JAVA service can start a thread according to a preset period, and the thread can periodically connect to the relational database from which data needs to be collected. This allows access to system tables or system views in the relational database that store metadata and SQL statements, thereby collecting the metadata and SQL statements stored in the relational database. In this solution, the collected metadata is used as a metadata dataset, and the collected SQL statements are used as an SQL statement dataset. The metadata describes the attribute information of the data, supporting functions such as indicating storage location, historical data, resource lookup, and file records.

[0061] In this way, metadata and SQL statements stored in the same relational database can be retrieved periodically by a thread, providing data support for subsequent retrieval of the lineage relationship between data.

[0062] S11, the SQL statement dataset is divided into query statement dataset and non-query statement dataset.

[0063] In an optional embodiment, the step of partitioning the SQL statement dataset to obtain a query statement dataset and a non-query statement dataset includes:

[0064] The comment lines in the SQL statement dataset are identified using regular expressions, and the comment lines are replaced to obtain the first statement relation dataset;

[0065] The first statement relation dataset is filtered using regular expressions and preset keywords to obtain the second statement relation dataset;

[0066] Based on the preset keywords, the second statement relation dataset is divided into a query statement dataset and a non-query statement dataset.

[0067] In this optional embodiment, comment lines of each SQL statement in the SQL statement dataset can be identified using regular expressions. A comment line is a line that begins with a specified comment symbol and ends with a newline character, such as the regular expression: "--.*\r\n". The regular expression describes a string matching pattern and can be used to check whether a string contains a substring or symbol that meets certain conditions. It is typically used in the form of a script to match substrings or symbols that meet certain conditions.

[0068] In this optional embodiment, all characters in the identified comment lines can be replaced with empty characters to avoid errors caused by confusion between the characters in the comment lines and the SQL statements. In this solution, the SQL statement dataset after the empty character replacement is used as the first statement relation dataset.

[0069] In this optional embodiment, since not all SQL statements will generate a lineage relationship, the first statement relationship dataset can be filtered to remove SQL statements that will not generate a lineage relationship, and the remaining SQL statements can be used as the second statement relationship dataset.

[0070] In this optional embodiment, since SQL statements belonging to the Data Definition Language (DDL) or Data Manipulation Language (DML) categories and containing the "SELECT" keyword generally have a lineage, such as INSERT…SELECT…, CREATE…SELECT…, etc., regular expressions can be used to determine whether each SQL statement in the first statement relationship dataset satisfies the DDL or DML type, and simultaneously determine whether each SQL statement contains the "SELECT" keyword. SQL statements that satisfy the DDL or DML type and contain the "SELECT" keyword are retained, thereby obtaining the second statement relationship dataset. The preset keyword can be the "SELECT" keyword.

[0071] In this optional embodiment, for the SQL statements in the obtained second statement relation dataset, the corresponding SQL statements can be divided into SQL query statements and SQL non-query statements from the position where the keyword "SELECT" first appears in each SQL statement. Specifically, the part before the keyword "SELECT" is considered a non-query statement, such as INSERT…, CREATE…, etc., while the rest including "SELECT" is considered a query statement, such as SELECT…. In this scheme, all SQL non-query statements are treated as a non-query statement dataset, and all SQL query statements are treated as a query statement dataset.

[0072] Thus, by replacing the comment lines in the SQL statement dataset, the errors caused by the comment lines can be effectively reduced. At the same time, by filtering out SQL statements that are not related and dividing the remaining SQL statements into SQL query statements and SQL non-query statements, it is easier for subsequent processes to construct an accurate abstract syntax tree based on this.

[0073] S12, perform lexical and syntactic analysis on the query statement dataset to construct an abstract syntax tree.

[0074] In an optional embodiment, the step of performing lexical and syntactic analysis on the query statement dataset to construct an abstract syntax tree includes:

[0075] Lexical analysis is performed on the SQL statements in the query statement dataset to obtain the position of each word in the SQL statement;

[0076] The meaning of each word in the SQL statement and the structure of the SQL statement are obtained through syntactic analysis.

[0077] An abstract syntax tree is constructed based on the position, meaning, and structure of each word in the SQL statement. The abstract syntax tree includes a root node and leaf nodes.

[0078] In this optional embodiment, the SQL statements in the query statement dataset can be parsed using the SQL engine in the relational database. The parsing process of the SQL statements includes lexical analysis, statement analysis, and semantic analysis. Lexical analysis breaks down the SQL statement into a sequence of words and identifies keywords, identifiers, constants, etc.; syntax analysis analyzes the meaning of each word and the SQL statement in which that word appears; and semantic analysis examines context-related properties based on grammatical correctness.

[0079] In this optional embodiment, each word in the SQL statement in the query statement dataset can be analyzed through SQL statement lexical analysis, and the position of each word in the SQL statement can be recorded simultaneously. At the same time, the meaning of each word in the SQL statement and the structure of the SQL statement are obtained based on syntax analysis. The structure is used to store the syntax analysis results of the corresponding SQL statement. The structure can be viewed as a multi-way tree, where each leaf node represents a syntax structure in the SELECT query statement. Since SQL statements generally have multi-level nested structures, the structure can clearly express the nesting relationships between the nested structures represented by each level of nodes.

[0080] In this optional embodiment, an abstract syntax tree can be constructed based on the position, meaning, and structure of each word in the SQL statement. The abstract syntax tree includes a root node and leaf nodes.

[0081] In this optional embodiment, each leaf node represents the field selected in the SQL query statement. After constructing the abstract syntax tree, each node can be parsed according to the SQL parser in semantic analysis, and the attributes of each node can be recorded. The attributes of the node include the name of the current node, the names of nodes at the same level, and the table in which the current node is located.

[0082] In this way, by performing lexical and syntactic analysis on the SQL statements in the query statement dataset, an abstract syntax tree can be constructed to clearly express the multi-level nested relationships in the SQL statements, which facilitates the subsequent process to obtain the lineage relationship between the fields in the SQL statements.

[0083] S13, generate field lineage paths based on the meta-dataset, non-query statement dataset, and abstract syntax tree.

[0084] In an optional embodiment, generating field lineage paths based on the meta-dataset, the non-query statement dataset, and the abstract syntax tree includes:

[0085] Based on the metadata, obtain the selected fields and the table information of the fields in the SQL query statement represented by the leaf node to obtain a clear leaf node;

[0086] Traverse the clear leaf nodes and root node in the abstract syntax tree to obtain the lineage base field;

[0087] Based on the non-query statement dataset, the blood relationship basic fields are matched to obtain the blood relationship positioning fields;

[0088] Multiple bloodline paths are constructed based on the bloodline location field.

[0089] In this optional embodiment, it can be determined whether the selected field information of the query statement represented by all leaf nodes of the abstract syntax tree is clear. The specific process is as follows:

[0090] If the selected part of the current SQL query statement is an asterisk (*), then it is necessary to retrieve all field information of the table containing the asterisk based on the metadata dataset and replace the asterisk in the statement. If the current SQL query statement contains join, union, or other related query terms, making it unclear which table the selected field belongs to, it is also necessary to retrieve the table information of the field by querying the metadata dataset.

[0091] In this optional embodiment, a leaf node with clear and explicit field information and the information of the table described in the field is taken as a clear leaf node, and the fields that exist in both the target leaf node and the parent node of the target leaf node are retained to obtain the bloodline relationship basic field corresponding to the target leaf node, wherein the target leaf node is any one of the clear leaf nodes.

[0092] In this optional embodiment, the lineage base field corresponding to each clear leaf node can be obtained by sequentially traversing the clear leaf nodes and the root node in the abstract syntax tree from the clear leaf node to the root node.

[0093] In this optional embodiment, if the SQL query statement contains UNION / JOIN or other related queries, it is necessary to traverse all clear leaf nodes in order from left to right, and then traverse the clear leaf nodes and root node in the abstract syntax tree in order from clear leaf nodes to root node to obtain the lineage relationship basic field corresponding to each clear leaf node.

[0094] In this optional embodiment, the table and field information in the non-query statement dataset can be traversed to match the bloodline basic field selected at the same position as the root node of the abstract syntax tree to obtain the bloodline positioning field.

[0095] In this way, accurate field lineage paths can be generated by combining non-query statement sets and metadata datasets, thereby improving the accuracy and reliability of the data lineage graphs obtained in subsequent processes.

[0096] S14, Construct a data lineage diagram based on the lineage path of the field.

[0097] In an optional embodiment, constructing a data lineage graph based on the field lineage path includes:

[0098] Construct a field-level lineage graph using each field in the lineage path as a vertex and the corresponding SQL statement as a directed edge;

[0099] A table-level lineage graph is constructed using the tables containing each field in the lineage path as vertices and the SQL statements corresponding to each field as directed edges.

[0100] In this optional embodiment, a field-level lineage graph and a table-level lineage graph can be constructed by taking each field in the field lineage path as a vertex and the table containing each field in the field lineage path as a vertex, and the SQL statement corresponding to each field as a directed edge. The graphs can clearly represent the lineage of data in the SQL statement, which helps to quickly perform source analysis, data quality assessment and data value assessment of tables and fields.

[0101] In this way, a lineage diagram can be constructed at both the table and field levels, providing a clear and intuitive representation of the lineage relationships of data in SQL statements.

[0102] Please see Figure 2 , Figure 2 This is a functional block diagram of a preferred embodiment of the data lineage graph construction device based on data analysis of this application. The data lineage graph construction device 11 based on data analysis includes an acquisition unit 110, a partitioning unit 111, an analysis unit 112, a generation unit 113, and a construction unit 114. The module / unit referred to in this application refers to a series of computer-readable instruction segments that can be executed by the processor 13 and perform a fixed function, and are stored in the memory 12. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0103] In an optional embodiment, the acquisition unit 110 is used to acquire metadata and SQL statements from a relational database to obtain a metadata dataset and an SQL statement dataset.

[0104] In an optional embodiment, the acquisition of metadata and SQL statements from the relational database to obtain the metadata dataset and SQL statement dataset includes:

[0105] Threads are started to connect to a preset relational database according to a preset cycle;

[0106] Based on the thread, metadata and SQL statements in the relational database are collected to obtain a metadata dataset and an SQL statement dataset.

[0107] In this optional embodiment, since this solution mainly extracts data lineage from SQL statements, it is necessary to use a relational database that includes SQL statements. The relational database can be MySQL, PostgreSQL, Oracle, openGauss, etc. This solution does not restrict the specific relational database.

[0108] In this optional embodiment, a JAVA service can start a thread according to a preset period, and the thread can periodically connect to the relational database from which data needs to be collected. This allows access to system tables or system views in the relational database that store metadata and SQL statements, thereby collecting the metadata and SQL statements stored in the relational database. In this solution, the collected metadata is used as a metadata dataset, and the collected SQL statements are used as an SQL statement dataset. The metadata describes the attribute information of the data, supporting functions such as indicating storage location, historical data, resource lookup, and file records.

[0109] In an optional embodiment, the partitioning unit 111 is used to partition the SQL statement dataset to obtain a query statement dataset and a non-query statement dataset.

[0110] In an optional embodiment, the step of partitioning the SQL statement dataset to obtain a query statement dataset and a non-query statement dataset includes:

[0111] The comment lines in the SQL statement dataset are identified using regular expressions, and the comment lines are replaced to obtain the first statement relation dataset;

[0112] The first statement relation dataset is filtered using regular expressions and preset keywords to obtain the second statement relation dataset;

[0113] Based on the preset keywords, the second statement relation dataset is divided into a query statement dataset and a non-query statement dataset.

[0114] In this optional embodiment, comment lines of each SQL statement in the SQL statement dataset can be identified using regular expressions. A comment line is a line that begins with a specified comment symbol and ends with a newline character, such as the regular expression: "--.*\r\n". The regular expression describes a string matching pattern and can be used to check whether a string contains a substring or symbol that meets certain conditions. It is typically used in the form of a script to match substrings or symbols that meet certain conditions.

[0115] In this optional embodiment, all characters in the identified comment lines can be replaced with empty characters to avoid errors caused by confusion between the characters in the comment lines and the SQL statements. In this solution, the SQL statement dataset after the empty character replacement is used as the first statement relation dataset.

[0116] In this optional embodiment, since not all SQL statements will generate a lineage relationship, the first statement relationship dataset can be filtered to remove SQL statements that will not generate a lineage relationship, and the remaining SQL statements can be used as the second statement relationship dataset.

[0117] In this optional embodiment, since SQL statements belonging to the Data Definition Language (DDL) or Data Manipulation Language (DML) categories and containing the "SELECT" keyword generally have a lineage, such as INSERT…SELECT…, CREATE…SELECT…, etc., regular expressions can be used to determine whether each SQL statement in the first statement relationship dataset satisfies the DDL or DML type, and simultaneously determine whether each SQL statement contains the "SELECT" keyword. SQL statements that satisfy the DDL or DML type and contain the "SELECT" keyword are retained, thereby obtaining the second statement relationship dataset. The preset keyword can be the "SELECT" keyword.

[0118] In this optional embodiment, for the SQL statements in the obtained second statement relation dataset, the corresponding SQL statements can be divided into SQL query statements and SQL non-query statements from the position where the keyword "SELECT" first appears in each SQL statement. Specifically, the part before the keyword "SELECT" is considered a non-query statement, such as INSERT…, CREATE…, etc., while the rest including "SELECT" is considered a query statement, such as SELECT…. In this scheme, all SQL non-query statements are treated as a non-query statement dataset, and all SQL query statements are treated as a query statement dataset.

[0119] In an optional embodiment, the analysis unit 112 is used to perform lexical and syntactic analysis on the query statement dataset to construct an abstract syntax tree.

[0120] In an optional embodiment, the step of performing lexical and syntactic analysis on the query statement dataset to construct an abstract syntax tree includes:

[0121] Lexical analysis is performed on the SQL statements in the query statement dataset to obtain the position of each word in the SQL statement;

[0122] The meaning of each word in the SQL statement and the structure of the SQL statement are obtained through syntactic analysis.

[0123] An abstract syntax tree is constructed based on the position, meaning, and structure of each word in the SQL statement. The abstract syntax tree includes a root node and leaf nodes.

[0124] In this optional embodiment, the SQL statements in the query statement dataset can be parsed using the SQL engine in the relational database. The parsing process of the SQL statements includes lexical analysis, statement analysis, and semantic analysis. Lexical analysis breaks down the SQL statement into a sequence of words and identifies keywords, identifiers, constants, etc.; syntax analysis analyzes the meaning of each word and the SQL statement in which that word appears; and semantic analysis examines context-related properties based on grammatical correctness.

[0125] In this optional embodiment, each word in the SQL statement in the query statement dataset can be analyzed through SQL statement lexical analysis, and the position of each word in the SQL statement can be recorded simultaneously. At the same time, the meaning of each word in the SQL statement and the structure of the SQL statement are obtained based on syntax analysis. The structure is used to store the syntax analysis results of the corresponding SQL statement. The structure can be viewed as a multi-way tree, where each leaf node represents a syntax structure in the SELECT query statement. Since SQL statements generally have multi-level nested structures, the structure can clearly express the nesting relationships between the nested structures represented by each level of nodes.

[0126] In this optional embodiment, an abstract syntax tree can be constructed based on the position, meaning, and structure of each word in the SQL statement. The abstract syntax tree includes a root node and leaf nodes.

[0127] In this optional embodiment, each leaf node represents the field selected in the SQL query statement. After constructing the abstract syntax tree, each node can be parsed according to the SQL parser in semantic analysis, and the attributes of each node can be recorded. The attributes of the node include the name of the current node, the names of nodes at the same level, and the table in which the current node is located.

[0128] In an optional embodiment, the generation unit 113 is used to generate field lineage paths based on the meta dataset, the non-query statement dataset, and the abstract syntax tree.

[0129] In an optional embodiment, generating field lineage paths based on the meta-dataset, the non-query statement dataset, and the abstract syntax tree includes:

[0130] Based on the metadata, obtain the selected fields and the table information of the SQL statement represented by the leaf node to obtain the clear leaf node;

[0131] Traverse the clear leaf nodes and root node in the abstract syntax tree to obtain the lineage base field;

[0132] Based on the non-query statement dataset, the blood relationship basic fields are matched to obtain the blood relationship positioning fields;

[0133] Multiple bloodline paths are constructed based on the bloodline location field.

[0134] In this optional embodiment, it can be determined whether the selected field information of the query statement represented by all leaf nodes of the abstract syntax tree is clear. The specific process is as follows:

[0135] If the selected part of the current SQL query statement is an asterisk (*), then it is necessary to retrieve all field information of the table containing the asterisk based on the metadata dataset and replace the asterisk in the statement. If the current SQL query statement contains join, union, or other related query terms, making it unclear which table the selected field belongs to, it is also necessary to retrieve the table information of the field by querying the metadata dataset.

[0136] In this optional embodiment, a leaf node with clear and explicit field information and the information of the table described in the field is taken as a clear leaf node, and the fields that exist in both the target leaf node and the parent node of the target leaf node are retained to obtain the bloodline relationship basic field corresponding to the target leaf node, wherein the target leaf node is any one of the clear leaf nodes.

[0137] In this optional embodiment, the lineage base field corresponding to each clear leaf node can be obtained by sequentially traversing the clear leaf nodes and the root node in the abstract syntax tree from the clear leaf node to the root node.

[0138] In this optional embodiment, if the SQL query statement contains UNION / JOIN or other related queries, it is necessary to traverse all clear leaf nodes in order from left to right, and then traverse the clear leaf nodes and root node in the abstract syntax tree in order from clear leaf nodes to root node to obtain the lineage relationship basic field corresponding to each clear leaf node.

[0139] In this optional embodiment, the table and field information in the non-query statement dataset can be traversed to match the bloodline basic field selected at the same position as the root node of the abstract syntax tree to obtain the bloodline positioning field.

[0140] In an optional embodiment, the construction unit 114 is used to construct a data lineage graph based on the field lineage path.

[0141] In an optional embodiment, constructing a data lineage graph based on the field lineage path includes:

[0142] Construct a field-level lineage graph using each field in the lineage path as a vertex and the corresponding SQL statement as a directed edge;

[0143] A table-level lineage graph is constructed using the tables containing each field in the lineage path as vertices and the SQL statements corresponding to each field as directed edges.

[0144] In this optional embodiment, a field-level lineage graph and a table-level lineage graph can be constructed by taking each field in the field lineage path as a vertex and the table containing each field in the field lineage path as a vertex, and the SQL statement corresponding to each field as a directed edge. The graphs can clearly represent the lineage of data in the SQL statement, which helps to quickly perform source analysis, data quality assessment and data value assessment of tables and fields.

[0145] As can be seen from the above technical solutions, this application can parse SQL statements by constructing an abstract syntax tree, and generate field lineage paths by combining non-query statement sets and meta datasets. Finally, the field lineage paths can be clearly represented in the data lineage graph, thereby improving the accuracy and reliability of the obtained data lineage graph.

[0146] Please see Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 1 includes a memory 12 and a processor 13. The memory 12 is used to store computer-readable instructions, and the processor 13 executes the computer-readable instructions stored in the memory to implement the data lineage graph construction method based on data analysis described in any of the above embodiments.

[0147] In an alternative embodiment, the electronic device 1 further includes a bus and a computer program stored in the memory 12 and executable on the processor 13, such as a data lineage graph construction program based on data analysis.

[0148] Figure 3 Only electronic device 1 with memory 12 and processor 13 is shown. It will be understood by those skilled in the art that... Figure 3 The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0149] Combination Figure 1 The memory 12 in the electronic device 1 stores a plurality of computer-readable instructions to implement a data lineage graph construction method based on data analysis, and the processor 13 can execute the plurality of instructions to achieve:

[0150] Metadata and SQL statements are collected from relational databases to obtain metadata datasets and SQL statement datasets;

[0151] The SQL statement dataset is divided into query statement dataset and non-query statement dataset;

[0152] Lexical and syntactic analysis are performed on the query statement dataset to construct an abstract syntax tree;

[0153] Generate field lineage paths based on the aforementioned meta-dataset, non-query statement dataset, and abstract syntax tree;

[0154] A data lineage graph is constructed based on the lineage path described in the field.

[0155] Specifically, the processor 13's implementation method for the above instructions can be found in [reference needed]. Figure 1 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0156] Those skilled in the art will understand that the schematic diagram is merely an example of electronic device 1 and does not constitute a limitation on electronic device 1. Electronic device 1 can be a bus-type structure or a star-type structure. Electronic device 1 may also include more or fewer other hardware or software than shown in the diagram, or different component arrangements. For example, electronic device 1 may also include input / output devices, network access devices, etc.

[0157] It should be noted that electronic device 1 is only an example. Other existing or future electronic products that are suitable for this application should also be included within the scope of protection of this application and are incorporated herein by reference.

[0158] The memory 12 includes at least one type of readable storage medium, which can be non-volatile or volatile. The readable storage medium includes flash memory, portable hard drives, multimedia cards, card-type memory (e.g., SD or DX memory), magnetic storage, magnetic disks, optical disks, etc. In some embodiments, the memory 12 can be an internal storage unit of the electronic device 1, such as a portable hard drive of the electronic device 1. In other embodiments, the memory 12 can also be an external storage device of the electronic device 1, such as a plug-in portable hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 1. The memory 12 can be used not only to store application software and various types of data installed on the electronic device 1, such as the code of a data lineage diagram construction program based on data analysis, but also to temporarily store data that has been output or will be output.

[0159] In some embodiments, the processor 13 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 13 is the control unit of the electronic device 1, connecting various components of the electronic device 1 via various interfaces and lines. It executes programs or modules stored in the memory 12 (e.g., executing a data lineage graph construction program based on data analysis) and calls data stored in the memory 12 to perform various functions of the electronic device 1 and process data.

[0160] The processor 13 executes the operating system of the electronic device 1 and various installed applications. The processor 13 executes these applications to implement the steps in the various embodiments of the data lineage graph construction method based on data analysis described above, for example... Figure 1 The steps are shown.

[0161] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to complete this application. The one or more modules / units may be a series of computer-readable instruction segments capable of performing a specific function, which describe the execution process of the computer program in the electronic device 1. For example, the computer program may be divided into a data acquisition unit 110, a partitioning unit 111, an analysis unit 112, a generation unit 113, and a construction unit 114.

[0162] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium. This software functional module, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, computer equipment, or network device, etc.) or processor to execute portions of the data lineage graph construction method based on data analysis described in the various embodiments of this application.

[0163] If the modules / units integrated in electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware devices. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above.

[0164] The computer program includes computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory, and other memory.

[0165] Furthermore, the computer-readable storage medium may primarily include a stored program area and a stored data area, wherein the stored program area may store the operating system, an application program required for at least one function, etc.; and the stored data area may store data created based on the use of blockchain nodes, etc.

[0166] The blockchain referred to in this application is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0167] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, in... Figure 3 The symbol is represented by only one arrow, but this does not indicate that there is only one bus or one type of bus. The bus is configured to enable communication between the memory 12 and at least one processor 13, etc.

[0168] This application also provides a computer-readable storage medium (not shown), which stores computer-readable instructions. These computer-readable instructions are executed by a processor in an electronic device to implement the data lineage graph construction method based on data analysis described in any of the above embodiments.

[0169] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0170] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0171] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0172] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices described in the specification may also be implemented by a single unit or device through software or hardware. Terms such as "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0173] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application.

Claims

1. A method for constructing a data lineage graph based on data analysis, characterized in that, The method includes: Metadata and SQL statements are collected from relational databases to obtain metadata datasets and SQL statement datasets; The SQL statement dataset is divided into query statement dataset and non-query statement dataset; Lexical and syntactic analysis are performed on the query statement dataset to construct an abstract syntax tree, which includes a root node and leaf nodes, and the leaf nodes represent the selected fields in the SQL query statement; Generating field lineage paths based on the metadata dataset, non-query statement dataset, and abstract syntax tree includes: obtaining clear leaf nodes by acquiring information about the selected fields and their respective tables in the SQL statements represented by the leaf nodes from the metadata dataset; traversing the clear leaf nodes and root nodes in the abstract syntax tree, and obtaining basic lineage fields based on fields in both the target leaf node and its parent node, where the target leaf node is any one of the clear leaf nodes; matching the basic lineage fields with the non-query statement dataset to obtain lineage location fields; and constructing multiple field lineage paths based on the lineage location fields. A data lineage graph is constructed based on the lineage path described in the field.

2. The data kinship graph construction method based on data analysis as described in claim 1, characterized in that, The metadata and SQL statements collected from the relational database to obtain the metadata dataset and SQL statement dataset include: Threads are started to connect to a preset relational database according to a preset cycle; Based on the thread, metadata and SQL statements in the relational database are collected to obtain a metadata dataset and an SQL statement dataset.

3. The data kinship graph construction method based on data analysis as described in claim 1, characterized in that, The step of dividing the SQL statement dataset into query statement datasets and non-query statement datasets includes: The comment lines in the SQL statement dataset are identified using regular expressions, and the comment lines are replaced to obtain the first statement relation dataset; The first statement relation dataset is filtered using regular expressions and preset keywords to obtain the second statement relation dataset; Based on the preset keywords, the second statement relation dataset is divided into a query statement dataset and a non-query statement dataset.

4. The data kinship graph construction method based on data analysis as described in claim 1, characterized in that, The step of performing lexical and syntactic analysis on the query statement dataset to construct an abstract syntax tree includes: Lexical analysis is performed on the SQL statements in the query statement dataset to obtain the position of each word in the SQL statement; The meaning of each word in the SQL statement and the structure of the SQL statement are obtained through syntactic analysis. An abstract syntax tree is constructed based on the position, meaning, and structure of each word in the SQL statement. The abstract syntax tree includes a root node and leaf nodes.

5. The data kinship graph construction method based on data analysis as described in claim 1, characterized in that, The process of traversing the clear leaf nodes and root node in the abstract syntax tree to obtain the lineage basic fields includes: The basic kinship field corresponding to the target leaf node is obtained by retaining the fields that exist in both the target leaf node and its parent node. The target leaf node is any one of the clear leaf nodes. By traversing the clear leaf nodes and root node in the abstract syntax tree sequentially from the clear leaf node to the root node, the lineage relationship basic field corresponding to each clear leaf node is obtained.

6. The data kinship graph construction method based on data analysis as described in claim 1, characterized in that, The data lineage diagram includes a field-level lineage diagram and a table-level lineage diagram. The construction of the data lineage diagram based on the field lineage paths includes: Construct a field-level lineage graph using each field in the lineage path as a vertex and the corresponding SQL statement as a directed edge; A table-level lineage graph is constructed using the tables containing each field in the lineage path as vertices and the SQL statements corresponding to each field as directed edges.

7. A data lineage diagram construction device based on data analysis, characterized in that, The apparatus is used to implement the data kinship graph construction method based on data analysis as described in any one of claims 1 to 6, the apparatus comprising: The acquisition unit is used to acquire metadata and SQL statements from relational databases to obtain metadata datasets and SQL statement datasets; A partitioning unit is used to partition the SQL statement dataset to obtain a query statement dataset and a non-query statement dataset; The analysis unit is used to perform lexical and syntactic analysis on the query statement dataset to construct an abstract syntax tree; The generation unit is used to generate field lineage paths based on the meta-dataset, the non-query statement dataset, and the abstract syntax tree; A construction unit is used to construct a data lineage graph based on the lineage path of the field.

8. An electronic device, characterized in that, The electronic device includes: Memory, which stores computer-readable instructions; and The processor executes computer-readable instructions stored in the memory to implement the data lineage graph construction method based on data analysis as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the data lineage graph construction method based on data analysis as described in any one of claims 1 to 6.

Images