Data blood relationship generation method and device, electronic equipment and storage medium

Abstract

The invention provides a data blood relationship generation method and device, electronic equipment and a storage medium. The method comprises the steps of obtaining a structured query statement; wherein the structured query statement comprises a plurality of data storage positions, and the structured query statement represents query logic for performing data query through the data storage positions; analyzing the structured query statement through a preset syntax analyzer to obtain an abstract syntax tree corresponding to the structured query statement; wherein the abstract syntax tree represents a structured query statement; according to an abstract syntax tree corresponding to the structured query statement, based on a preset syntax tree analysis rule, obtaining data blood relationship information of query logic represented by the structured query statement; wherein the data blood relationship information represents the dependency relationship between the data flow direction and the data in the query logic. According to the method, the accuracy of data blood relationship generation is improved, and the efficiency of data blood relationship generation is improved.

Description

Technical Field

[0001] The present application relates to the field of big data, and in particular to a method, device, electronic device and storage medium for generating data blood relationship. Background Art

[0002] As the demand for data management and analysis continues to increase, the generation and management of data lineage relationships have become particularly important. Data lineage relationships refer to a naturally formed relationship between data in the process of data generation, processing, fusion, circulation and final extinction, similar to human lineage relationships, which is used to describe the flow and dependency of data in the system. Data lineage relationships help with data governance, impact analysis and compliance checks.

[0003] At present, existing technologies mostly rely on manual coding and maintenance, which is costly, error-prone, and difficult to fully and accurately generate data lineage relationships. If lineage relationship data can be generated accurately and automatically, the cost of data governance can be reduced and effective use of data resources can be achieved. Summary of the invention

[0004] The present disclosure provides a data lineage relationship generation method, device, electronic device and storage medium to solve the problem of low efficiency in data lineage relationship generation and analysis.

[0005] According to a first aspect of the present disclosure, the present application provides a method for generating a data kinship relationship, comprising:

[0006] Obtaining a structured query statement; wherein the structured query statement includes multiple data storage locations, and the structured query statement represents the query logic for performing data query through the data storage locations;

[0007] The structured query statement is parsed by a preset syntax parser to obtain an abstract syntax tree corresponding to the structured query statement; wherein the abstract syntax tree represents the structured query statement;

[0008] According to the abstract syntax tree corresponding to the structured query statement, based on the preset syntax tree parsing rules, the data lineage relationship information of the query logic represented by the structured query statement is obtained; wherein the data lineage relationship information represents the data flow in the query logic and the dependency relationship between the data.

[0009] According to a second aspect of the present disclosure, the present application provides a data lineage generation device, comprising:

[0010] An acquisition unit, configured to acquire a structured query statement; wherein the structured query statement includes a plurality of data storage locations, and the structured query statement represents a query logic for performing a data query through the data storage locations;

[0011] A parsing unit, configured to parse the structured query statement through a preset syntax parser to obtain an abstract syntax tree corresponding to the structured query statement; wherein the abstract syntax tree represents the structured query statement;

[0012] A relationship generation unit is used to obtain data lineage relationship information of the query logic represented by the structured query statement based on the abstract syntax tree corresponding to the structured query statement and based on preset syntax tree parsing rules; wherein the data lineage relationship information represents the data flow in the query logic and the dependency relationship between the data.

[0013] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0014] at least one processor, and a memory in communication with the processor;

[0015] The memory stores computer-executable instructions;

[0016] The processor executes the computer-executable instructions stored in the memory to implement the method described in any one of the first aspects.

[0017] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement any one of the methods described in the first aspect.

[0018] According to a fifth aspect of the present disclosure, a computer program product is provided, the computer program product comprising: a computer program, which implements the method described in the first aspect when executed by a processor.

[0019] The present application provides a data lineage relationship generation method, device, equipment, medium and program product. After obtaining a structured query statement, an abstract syntax tree is generated by parsing the structured query data, and the data lineage relationship corresponding to the structured query statement is obtained by parsing the abstract syntax tree. The disadvantage of high maintenance cost of manual coding in the prior art is solved, the cost of data governance is reduced, the accuracy and efficiency of data lineage relationship generation are improved, and accurate and automatic generation of data lineage relationships is achieved.

[0020] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0022] Figure 1 A flowchart of a method for generating data blood relationship provided by an embodiment of the present disclosure;

[0023] Figure 2 A schematic diagram of an abstract syntax tree provided for an embodiment of the present disclosure;

[0024] Figure 3 A flowchart of a method for generating data blood relationship provided by an embodiment of the present disclosure;

[0025] Figure 4 A flowchart of a method for generating data blood relationship provided by an embodiment of the present disclosure;

[0026] Figure 5 A flowchart of a method for generating data blood relationship provided by an embodiment of the present disclosure;

[0027] Figure 6 A schematic diagram of a data lineage relationship diagram provided in an embodiment of the present disclosure;

[0028] Figure 7 A structural block diagram of a data kinship generation device provided in an embodiment of the present disclosure;

[0029] Figure 8 A structural block diagram of a data kinship generation device provided in an embodiment of the present disclosure;

[0030] Fig. 9 A structural block diagram of a data kinship generation device provided in an embodiment of the present disclosure;

[0031] Fig.10 A structural block diagram of an electronic device provided in an embodiment of the present disclosure;

[0032] Fig.11 It is a block diagram of an electronic device according to an exemplary embodiment.

[0033] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0034] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0035] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0036] With the increasing demand for data management and analysis, the generation and management of data lineage relationships have become particularly important. Data lineage relationships refer to a naturally formed relationship between data in the process of data generation, processing, processing, fusion, circulation and final extinction, similar to human lineage relationships, which is used to describe the flow and dependency of data in the system. Data lineage relationships play an important role in security big data. Based on the visualization of data lineage relationship information, it is possible to intuitively discover the data production chain and whether there are any abnormalities in each link.

[0037] Existing data lineage generation tools have poor performance when processing large-scale data sets and high-frequency queries, and cannot be effectively expanded to be compatible with a variety of different database types, making it difficult to meet the needs of enterprise-level applications, especially when facing a large number of concurrent queries. Performance bottlenecks are prone to occur. Existing data lineage generation tools also have deficiencies in user interaction and data lineage visualization. Existing tools usually lack intuitive interfaces and cannot effectively display data flows and transformation relationships.

[0038] The specific application scenarios of this application include data governance and data management platforms, which can be used to automatically generate and manage data lineage relationships and enhance data transparency and traceability; enterprise-level data warehouses and data lakes, which can be used to automatically manage complex structured data queries and data dependencies and improve data management efficiency; data analysis and business intelligence tools, which can be used to improve the accuracy and efficiency of data analysis and enhance the credibility and transparency of data analysis reports.

[0039] The data kinship generation method provided in this application is intended to solve the above technical problems of the prior art.

[0040] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0041] Figure 1 It is a flowchart of a method for generating data lineage relationship according to an embodiment of the present disclosure. The method can be executed by a device for generating data lineage relationship.

[0042] like Figure 1 As shown, the method comprises the following steps:

[0043] S101, obtaining a structured query statement; wherein the structured query statement includes multiple data storage locations, and the structured query statement represents the query logic for performing data query through the data storage locations.

[0044] For example, a SQL (Structured Query Language) statement may indicate that a user wants to query required information from a database, for example, it may represent the query logic for querying data through a data storage location. Query logic refers to the logic for obtaining data from a database. An SQL statement includes multiple data storage locations, and a data storage location may represent a database, a data table in a database, or a column in a data table. The system will obtain the SQL statement entered by the user in real time.

[0045] Depending on the database type, users can choose different SQL statements. Taking the SQL language database as an example, according to the query keywords, select SELECT, FROM, WHERE, GROUP BY, HAVING, ORDER BY digital query language keywords, as well as joins, subqueries, common table expressions and nested query types to perform query operations.

[0046] S102: parsing the structured query statement through a preset syntax parser to obtain an abstract syntax tree corresponding to the structured query statement; wherein the abstract syntax tree represents the structured query statement.

[0047] Exemplarily, after obtaining the SQL statement, it is determined whether the syntax of the SQL statement is correct according to the preset SQL syntax checking rules. For example, the preset SQL syntax checking rules may be to check the length of the SQL statement and the order of characters, and if yes, parse the SQL statement; if not, do not parse the SQL statement and remind the user that there is an error.

[0048] After obtaining the SQL statement that conforms to the correct grammatical rules, the SQL statement is parsed by a preset grammar parser. The preset grammar parser can be a parser constructed based on ANTLR4, which is a language recognition tool written in JAVA language, and can automatically construct a lexical analyzer and a grammar analyzer of a custom language by inputting a grammar description file. Using the parser constructed by ANTLR4, the data storage location involved in the SQL statement and the nested relationship between the storage locations can be obtained. In this embodiment, the working process of the parser constructed by ANTLR4 is not specifically limited.

[0049] The abstract syntax tree is a tree representation of the abstract syntax structure of SQL statements. Each node on the tree represents a structure in the source code. The abstract syntax tree can include multiple nodes. A node represents a storage location, and the connection relationship between nodes represents a nested relationship. The abstract syntax tree is constructed based on the data storage location and the nested relationship between the storage locations. The nested brackets in the abstract syntax tree are implicit in the tree structure and are not presented in the form of nodes. The nodes corresponding to the parts of the nested relationship that contain logical relationships on the periphery are set as the parent nodes of the inner nodes, and the nodes corresponding to the parts of the nested relationship that are on the inner layer are set as the child nodes of the corresponding parent nodes.

[0050] Figure 2 A diagram of an abstract syntax tree. Figure 2 The abstract syntax tree shown can be represented as an SQL statement "select username,ismale from userInfo where age>20and level>5and 1=1". Figure 2The root node "Select statement" in the second-level node represents the entire SQL statement above, and the "Select" in the second-level node represents the "select" in the above SQL statement; the "Fields" in the second-level node represents the column in the above SQL statement, and the child node "Username" of this node represents the "username" in the above SQL statement, and the child node "Ismale" of this node represents the "ismale" in the above SQL statement, indicating the column name; the "From" in the second-level node represents the "from" in the above SQL statement; the "Tables" in the second-level node represents the table in the above SQL statement, and the child node "Userinfo" of this node represents the "userinfo" in the above SQL statement, indicating the table name; the "Where" in the second-level node represents the "where" in the above SQL statement; the "Condition" in the second-level node represents the query restriction condition in the above SQL statement, and the child node "And" of this node represents the "and" in the above SQL statement, indicating the keyword "and", the "Gt" in the fourth-level node is the abbreviation of "Greater Than", indicating the comparison of two values, and the "Eq" in the fourth-level node is "equal The abbreviation of "to" indicates the use of an equal sign. The "Age" in the fifth-level node indicates the "age" in the above SQL statement, and the "Level" in the fifth-level node indicates the "level" in the above SQL statement.

[0051] The preset grammar parser can be a parser based on ANTLR 4. The benefit of the ANTLR 4 parser is that it can be used to parse complex SQL statements, support various SQL statement dialects and complex query structures using nested and common table expressions.

[0052] S103. According to the abstract syntax tree corresponding to the structured query statement, based on the preset syntax tree parsing rules, obtain the data lineage relationship information of the query logic represented by the structured query statement; wherein the data lineage relationship information represents the data flow in the query logic and the dependency relationship between the data.

[0053] Exemplarily, after obtaining the abstract syntax tree, the data kinship between the various data storage locations in the SQL statement can be determined according to the preset syntax tree parsing rules. For example, the preset syntax tree parsing rules can be the data kinship information corresponding to the child nodes of the parent node, which flows the data in the data storage location corresponding to the parent node according to the upper and lower hierarchical relationship of the nodes in the syntax tree. Perform semantic detection on the structured data. If the code statement of the structured data meets the preset semantic check rules, the data kinship information queried by the corresponding SQL statement is obtained according to the structured data. Dependency is a mutual constraint relationship between data. For example, if two instantiated data sets A and B have the same value for two elements in data set A, there must be another corresponding data set B with the same value, then B is said to be dependent on A. Perform semantic detection on the structured data. If the code statement of the structured data meets the preset semantic check rules, the data kinship information queried by the corresponding SQL statement is obtained according to the structured data.

[0054] In this embodiment, after obtaining a structured query statement, an abstract syntax tree is generated by parsing the structured query data, and the data lineage relationship corresponding to the structured query statement is obtained by parsing the abstract syntax tree. This solves the disadvantage of high maintenance cost of manual coding in the prior art, reduces the cost of data management, improves the accuracy and efficiency of data lineage relationship generation, and realizes accurate and automatic generation of data lineage relationships.

[0055] Figure 3 A flowchart of a method for generating data bloodline relationships provided in an embodiment of the present disclosure.

[0056] In this embodiment, a SQL statement is parsed by a preset syntax parser to obtain an abstract syntax tree corresponding to the SQL statement, including: a SQL statement is lexically parsed by a preset syntax parser to obtain lexical units in the SQL statement; wherein the lexical units represent data storage locations in the SQL statement; according to preset syntax rules, the lexical units are parsed to obtain an abstract syntax tree corresponding to the SQL statement; wherein the preset syntax rules are used to connect lexical units into an abstract syntax tree.

[0057] like Figure 3 As shown, the method comprises the following steps:

[0058] S301, obtaining a structured query statement; wherein the structured query statement includes multiple data storage locations, and the structured query statement represents the query logic for performing data query through the data storage locations.

[0059] Exemplarily, this step may refer to the above-mentioned step S101 and will not be described in detail.

[0060] S302: Perform lexical analysis on the structured query statement through a preset syntax analyzer to obtain lexical units in the structured query statement; wherein the lexical units represent constituent elements in the structured query statement.

[0061] Exemplarily, the SQL_Lexer class recognizes and lexically parses SQL statements, reads characters from left to right into the source program, recognizes words and symbols, and divides SQL statements into multiple lexical units. The SQL_Lexer class defines an ANTLR4 grammar for a Lexer (lexical analyzer) for parsing SQL query statements. A token (lexical unit) consists of a lexical unit name and an optional attribute value. The lexical unit name is an abstract symbol used to indicate the type of a lexical unit, such as a keyword, identifier, etc.; the attribute value usually indicates the transmission of additional information, such as the content, type, and occurrence position of the lexical unit.

[0062] In this embodiment, the SQL_Lexer class includes the following lexical rules: SELECT, used to match the data query language keyword "SELECT"; FROM, used to match the data query language keyword "FROM"; WHERE, used to match the data query language keyword "WHERE"; AND, used to match the data query language keyword "AND"; OR, used to match the data query language keyword "OR"; identifier rule IDENTIFIER, used to match a string starting with a letter or an underscore followed by any number of letters, numbers or underscores; number rule NUMBER, used to match a string consisting of one or more numbers; whitespace rule WHITESPACE, used to match one or more spaces, tabs, carriage returns or line feeds, and skip these characters.

[0063] S303: Perform grammatical analysis on the lexical units according to preset grammatical rules to obtain an abstract syntax tree corresponding to the structured query statement; wherein the preset grammatical rules are used to connect the lexical units into an abstract syntax tree.

[0064] Exemplarily, based on lexical analysis, the SQL_Parser class is used to determine the nested relationship between lexical units in the SQL statement. The SQL_Parser class defines the syntax part of a SQL parser based on ANTLR4, which relies on the previously defined SQL_Lexer lexical analyzer to identify lexical units. The SQL_Parser class includes the following rules: the options block specifies that the lexical analyzer is SQL_Lexer; the query rule defines the overall structure of the query statement; the selectClause rule indicates the beginning of the SELECT statement; the fromClause rule defines the FROM clause, which specifies which table to select data from; the whereClause rule defines the WHERE clause; the columns rule defines a list of column names; the table rule is an identifier, representing a table name; the condition rule defines the basic form of the condition, that is, an identifier or column name is equal to a certain value; the value rule defines the type of the value, which can be an identifier or a number.

[0065] In this embodiment, according to preset grammatical rules, grammatical parsing processing is performed on lexical units to obtain an abstract syntax tree corresponding to the SQL statement, including: determining the nesting relationship between lexical units in the SQL statement according to the preset grammatical rules; wherein the nesting relationship represents the connection relationship between nodes in the abstract syntax tree; and obtaining the abstract syntax tree corresponding to the SQL statement according to the nesting relationship between each lexical unit and each lexical unit.

[0066] Specifically, after obtaining multiple lexical units, the nesting relationship between lexical units in the SQL statement is determined according to the preset grammatical rules. For example, the preset grammatical rules can be used to determine the type and method of nesting. After determining the nesting relationship between lexical units, the lexical units are connected according to the nesting relationship between the lexical units. For example, according to the determined nesting relationship, the lexical units of the representation logic of the nested outer layer are used as parent nodes, and the lexical units of the nested inner layer are used as child nodes of the parent node. The abstract syntax tree corresponding to the SQL statement is obtained. In this embodiment, the preset grammatical rules are not specifically limited.

[0067] The beneficial effect of this setting is that it can avoid node omission, improve the efficiency of syntax tree construction, improve the accuracy of abstract syntax tree, and make the process more standardized.

[0068] S304. According to the abstract syntax tree corresponding to the structured query statement, based on the preset syntax tree parsing rules, obtain the data lineage relationship information of the query logic represented by the structured query statement; wherein the data lineage relationship information represents the data flow in the query logic and the dependency relationship between the data.

[0069] Exemplarily, this step may refer to the above-mentioned step S103 and will not be described in detail.

[0070] In the embodiment of the present application, the SQL statement obtained is identified and lexical parsed using the SQL_Lexer class, and the characters are read into the source program from left to right in turn, and words and symbols are identified therefrom, and the SQL statement is divided into multiple lexical units. After obtaining the lexical unit, on the basis of lexical parsing, the SQL_Parser class is used to determine the nested relationship between the lexical units in the SQL statement, and the lexical units are combined into various grammatical phrases according to the grammar and nested relationship of the current programming language, and the generated grammatical phrases are generated into a corresponding abstract syntax tree. The generation efficiency of the abstract syntax tree is improved, and the waste of manpower and time is saved. Both lexical parsing and syntactic parsing use the ANTLR4 parser, which can be used to generate lexical analyzers and grammatical analyzers for a variety of languages ​​including SQL statements. The benefit of using the ANTLR4 parser is that it has high performance, that is, it can quickly and accurately parse the input content in a complex language environment; it is easy to expand and customize, and ANTLR4 provides a rich application programming interface.

[0071] Figure 4 A flowchart of a method for generating data bloodline relationships provided in an embodiment of the present disclosure.

[0072] In this embodiment, according to the abstract syntax tree corresponding to the SQL statement, based on the preset syntax tree parsing rules, the data lineage relationship of the query logic represented by the SQL statement is obtained, including: traversing the abstract syntax tree, and converting the abstract syntax tree into statement data in a preset format based on the preset syntax tree parsing rules; according to the structured data, the data lineage relationship information of the query logic represented by the SQL statement is obtained.

[0073] like Figure 4 As shown, the method comprises the following steps:

[0074] S401, obtaining a structured query statement; wherein the structured query statement includes multiple data storage locations, and the structured query statement represents the query logic for performing data query through the data storage locations.

[0075] Exemplarily, this step may refer to the above-mentioned step S101 and will not be described in detail.

[0076] S402, parsing the SQL statement through a preset syntax parser to obtain an abstract syntax tree corresponding to the SQL statement; wherein the abstract syntax tree represents the SQL statement;

[0077] Exemplarily, this step may refer to the above-mentioned step S102 and will not be described in detail.

[0078] S403: traverse the abstract syntax tree, and convert the abstract syntax tree into statement data in a preset format based on preset syntax tree parsing rules.

[0079] Exemplarily, after obtaining the abstract syntax tree corresponding to the SQL statement, a depth-first search traversal operation is performed on the abstract syntax tree. Depth-first search is a method for traversing a tree, which traverses the nodes of the tree along the depth of the tree and searches the branches of the tree as deeply as possible.

[0080] Specifically, the depth-first search method includes: first checking whether each node belongs to a specific statement type, including SelectClauseContext, FromClauseContext, and WhereClauseContext, which correspond to the available column name statements, table name statements, and statements that do not directly conform to the preset format. If it belongs to a specific statement type, the information in the specific statement is obtained. Then, the search method is recursively called for each child node until the bottom layer of the tree is reached. When there are no recursive child nodes, the previous layer of nodes is traced back to continue processing its unvisited child nodes.

[0081] In this embodiment, an abstract syntax tree is traversed, and based on preset syntax tree parsing rules, the abstract syntax tree is converted into statement data in a preset format, including: traversing the abstract syntax tree to determine the node currently traversed; wherein the node in the abstract syntax tree represents the storage location of statement data in the SQL statement; according to the preset syntax tree parsing rules, the format of the currently traversed node is converted to obtain structured data corresponding to the currently traversed node; according to the structured data corresponding to each node in the abstract syntax tree and the connection relationship between each node in the abstract syntax tree, the code statement corresponding to the abstract syntax tree represents the abstract syntax tree.

[0082] Specifically, the result of traversing the abstract syntax tree corresponding to the SQL statement needs to be formatted. During the depth-first search traversal, the statement type of the node at the current traversal position is checked to determine the position information of the node at the current traversal position. The information that meets the structured data requirements is formatted, and the format of the data structure can be preset to obtain the structured data of the preset format corresponding to the node currently traversed. According to the structured data corresponding to each node and the connection relationship between each node in the abstract syntax tree, the query logic order between the structured data corresponding to each node is determined, and the nodes are spliced ​​according to the query logic order to obtain the code statement of the structured data containing the data lineage relationship.

[0083] The beneficial effect of such a setting is that the data lineage relationship information in the abstract syntax tree can be more accurately identified and extracted, which facilitates the generation of the data lineage relationship graph and improves the efficiency of generating data lineage relationships.

[0084] In this embodiment, after the target node is formatted according to the preset syntax tree parsing rules to obtain the structured data corresponding to the target node, it also includes: if it is determined that the code statement data meets the preset semantic checking rules, then the data lineage relationship information of the query logic represented by the structured query statement is obtained according to the structured data.

[0085] Specifically, semantic checking is performed on the code statements of the structured data. The contents of the semantic checking include: checking whether it is a valid statement in the SQL statement; checking whether the information containing the relationship between tables corresponds in sequence; checking whether it contains intermediate values, and eliminating unnecessary intermediate calculations, for example, replacing 0.6+0.4 with 1.0. If the structured data meets all the rules of the semantic check, the structured data is qualified structured data.

[0086] The beneficial effect of such a setting is that it makes the parsing of the abstract syntax tree more accurate, reduces the possibility of errors and omissions, optimizes unnecessary intermediate calculations, improves the efficiency of parsing, and reduces the load on the device.

[0087] S404: Obtain data lineage relationship information of the query logic represented by the SQL statement based on the structured data.

[0088] Exemplarily, the obtained structured data is processed according to keywords to obtain the node location information and data lineage relationship contained in the structured data. The keywords may be keywords such as data name, data time information, etc.

[0089] In the embodiment of the present application, the abstract syntax tree corresponding to the SQL statement is parsed, the abstract syntax tree is converted into structured data containing data lineage information, and then the data lineage relationship is obtained according to the structured data. The benefit is that all data lineage information can be retained without distortion, and lineage relationship information will not be lost, thereby improving the accuracy and efficiency of lineage relationship generation.

[0090] Figure 5 A flowchart of a method for generating data bloodline relationships provided in an embodiment of the present disclosure.

[0091] In this embodiment, the method further includes generating graphic information in a preset format, which is a blood relationship diagram, based on the blood relationship information; and displaying the blood relationship diagram on a visualization interface; wherein the blood relationship diagram represents the data blood relationship information.

[0092] like Figure 5As shown, the method comprises the following steps:

[0093] S501, obtaining an SQL statement; wherein the SQL statement includes multiple data storage locations, and the SQL statement represents the query logic for performing data query through the data storage locations.

[0094] Exemplarily, this step may refer to the above-mentioned step S101 and will not be described in detail.

[0095] S502: parse the SQL statement through a preset syntax parser to obtain an abstract syntax tree corresponding to the SQL statement; wherein the abstract syntax tree represents the SQL statement.

[0096] Exemplarily, this step may refer to the above-mentioned step S102 and will not be described in detail.

[0097] S503. According to the abstract syntax tree corresponding to the SQL statement and based on preset syntax tree parsing rules, obtain data lineage information of the query logic represented by the SQL statement; wherein the data lineage information represents the data flow in the query logic.

[0098] Exemplarily, this step may refer to the above-mentioned step S103 and will not be described in detail.

[0099] S504: Generate graphic information in a preset format according to the data lineage relationship information, which is a data lineage relationship diagram.

[0100] Exemplarily, based on the data lineage information, a data lineage diagram is generated according to a preset graphic format and classification. The preset graphic format may be a Sankey diagram, which is used to visualize the flow of data lineage relationships, and it uses directed arrows to connect different nodes to display the path and magnitude of the flow. The data lineage diagram includes the data flow direction and dependency relationship of the queried data.

[0101] S505: Display the blood relationship diagram on the visualization interface.

[0102] For example, a blood relationship diagram and corresponding information are displayed on a visualization interface that can interact with the user. The blood relationship diagram includes the relationship between tables such as JOIN and subquery, and the relationship between columns such as SELECT and UPDATE. The visualization interface supports different view modes, such as hierarchy diagram, relationship diagram, time series diagram, etc., which facilitates users to analyze blood relationships from different angles.

[0103] Figure 6 is a schematic diagram of the data lineage relationship diagram. Figure 6As shown, each table contains an ID (identifier) ​​and a value. The arrow lines in the data lineage diagram indicate the flow of data in the table. Data Table 1, Data Table 2, and Data Table 3 represent the upstream nodes of Data Table 4, and Data Table 4 merges the IDs and values ​​in Data Table 1, Data Table 2, and Data Table 3.

[0104] In this embodiment, the method also includes responding to a user's query instruction, obtaining a keyword identifier from the query instruction; wherein the keyword identifier represents a data storage location to be queried; querying a blood relationship graph corresponding to the keyword identifier, and displaying the blood relationship graph corresponding to the keyword identifier on a visualization interface.

[0105] Specifically, the user can issue a data viewing instruction in the query part of the visualization interface. The data viewing instruction contains the data name information keyword specified by the query personnel and the specified view mode keyword of the data lineage diagram such as the hierarchical diagram, the relationship diagram, and the time series diagram. According to different keywords, the corresponding data lineage diagram is obtained from the data storage location, and the visualization interface displays the corresponding graphic information. Among them, the data storage location is used to represent the storage location of the tables and columns contained in the data lineage diagram. The stored procedure is to define the class for data storage using the SQL language, obtain all the nodes in the data lineage diagram, insert the information of each node into the table corresponding to the node, and stop the statement after the insertion operation is completed to release the database resources.

[0106] In this embodiment, using a precompiled SQL statement, the SQL statement can be precompiled and optimized, which helps prevent SQL statement injection attacks and improves the storage performance of the method, ensuring that database resources can be closed correctly and avoiding resource leakage.

[0107] In the embodiment of the present application, the data lineage relationship information obtained by converting the abstract syntax tree is used to generate a data lineage relationship diagram, which includes information such as data flow dependency. The data lineage relationship diagram uses the form of a Sankey diagram, which has good interactivity and can clearly display data lineage relationship information in multiple levels and styles.

[0108] Figure 7 A structural block diagram of a data blood relationship generation device provided in an embodiment of the present disclosure.

[0109] For the sake of convenience, only the parts related to the embodiments of the present disclosure are shown. Figure 7 The data blood relationship generating device 700 includes: an acquiring unit 701, a parsing unit 702 and a relationship generating unit 703.

[0110] An acquisition unit 701 is used to acquire an SQL statement; wherein the SQL statement includes multiple data storage locations, and the SQL statement represents a query logic for performing data query through the data storage locations;

[0111] A parsing unit 702 is used to parse the SQL statement through a preset syntax parser to obtain an abstract syntax tree corresponding to the SQL statement; wherein the abstract syntax tree represents the SQL statement;

[0112] The relationship generation unit 703 is used to obtain the data lineage relationship information of the query logic represented by the SQL statement according to the abstract syntax tree corresponding to the SQL statement and based on the preset syntax tree parsing rules; wherein the data lineage relationship information represents the data flow in the query logic and the dependency relationship between the data.

[0113] Figure 8 A structural block diagram of a data blood relationship generation device provided in an embodiment of the present disclosure.

[0114] exist Figure 7 Based on the embodiment shown, Figure 8 As shown, the parsing unit 702 includes a lexical parsing module 7021 and a grammatical parsing module 7022 .

[0115] The lexical analysis module 7021 is used to perform lexical analysis on the SQL statement to obtain the lexical units in the SQL statement;

[0116] The syntax analysis module 7022 performs syntax analysis on the lexical unit to obtain an abstract syntax tree corresponding to the SQL statement;

[0117] In one example, the syntax parsing module 7022 is specifically used for:

[0118] According to the preset grammar rules, determine the nesting relationship between the lexical units in the SQL statement; wherein the nesting relationship represents the connection relationship between the nodes in the abstract syntax tree;

[0119] According to the nested relationships between the lexical units, an abstract syntax tree corresponding to the SQL statement is obtained.

[0120] In one example, the acquisition unit 701 is specifically configured to:

[0121] Obtain an SQL statement; wherein the SQL statement includes multiple data storage locations, and the SQL statement represents the query logic for performing data query through the data storage locations.

[0122] In one example, the parsing unit 702 is specifically configured to:

[0123] The SQL statement is parsed by a preset syntax parser to obtain an abstract syntax tree corresponding to the SQL statement; wherein the abstract syntax tree represents the SQL statement.

[0124] Fig. 9 A structural block diagram of a data blood relationship generation device provided in an embodiment of the present disclosure.

[0125] exist Figure 7 Based on the embodiment shown, Fig. 9 As shown, the relationship generation unit 703 includes a syntax tree traversal module 7031 and a blood relationship extraction module 7032.

[0126] The syntax tree traversal module 7031 is used to traverse the abstract syntax tree and convert the abstract syntax tree into statement data in a preset format based on preset syntax tree parsing rules;

[0127] The blood relationship extraction module 7032 is used to obtain the data blood relationship information of the query logic represented by the SQL statement based on the structured data.

[0128] In one example, the syntax tree traversal module 7031 is specifically used for:

[0129] Traversing the abstract syntax tree to determine the currently traversed node; wherein the node in the abstract syntax tree represents the storage location of statement data in the SQL statement;

[0130] According to the preset syntax tree parsing rules, the format of the currently traversed node is converted to obtain the structured data corresponding to the currently traversed node;

[0131] According to the structured data corresponding to each node in the abstract syntax tree and the connection relationship between each node in the abstract syntax tree, a code statement corresponding to the abstract syntax tree is obtained; wherein the code statement corresponding to the abstract syntax tree represents the abstract syntax tree.

[0132] In one example, the syntax tree traversal module 7031 is specifically used for:

[0133] After converting the format of the target node according to the preset syntax tree parsing rules and obtaining the structured data corresponding to the target node, it is determined whether the code statement meets the preset semantic checking rules. If so, the data lineage relationship information of the query logic represented by the SQL statement is obtained based on the structured data.

[0134] In one example, it also includes:

[0135] A graphic generation unit, used to generate graphic information of a preset format according to the blood relationship information, which is a blood relationship diagram;

[0136] The display unit is used to display the blood relationship diagram on a visual interface; wherein the blood relationship diagram represents the data blood relationship information.

[0137] In one example, it also includes:

[0138] An identifier acquisition unit, used to respond to a user's query instruction and acquire a keyword identifier from the query instruction; wherein the keyword identifier represents a data storage location to be queried;

[0139] The query unit is used to query the blood relationship diagram corresponding to the keyword identifier and display the blood relationship diagram corresponding to the keyword identifier on the visualization interface.

[0140] Fig.10 A structural block diagram of an electronic device provided in an embodiment of the present disclosure, the electronic device may be a terminal device or a server, such as Fig.10 As shown, the electronic device 1000 includes: at least one processor 1002; and a memory 1001 communicatively connected to the at least one processor 1002; wherein the memory stores instructions executable by the at least one processor 1002, and the instructions are executed by the at least one processor 1002 so that the at least one processor 1002 can execute the data blood relationship generation method disclosed in the present invention.

[0141] The electronic device 1000 further includes a receiver 1003 and a transmitter 1004. The receiver 1003 is used to receive instructions and data sent by other devices, and the transmitter 1004 is used to send instructions and data to external devices.

[0142] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0143] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0144] Fig.11 It is a block diagram of an electronic device according to an exemplary embodiment, which device may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0145] Device 1100 may include one or more of the following components: a processing component 1102 , a memory 1104 , a power component 1106 , a multimedia component 1108 , an audio component 1111 , an input / output (I / O) interface 1112 , a sensor component 1114 , and a communication component 1116 .

[0146] The processing component 1102 generally controls the overall operation of the device 1100, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 1102 may include one or more processors 1120 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 1102 may include one or more modules to facilitate the interaction between the processing component 1102 and other components. For example, the processing component 1102 may include a multimedia module to facilitate the interaction between the multimedia component 1108 and the processing component 1102.

[0147] The memory 1104 is configured to store various types of data to support operations on the device 700. Examples of such data include instructions for any application or method operating on the device 1100, contact data, phone book data, messages, pictures, videos, etc. The memory 1104 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0148] The power supply component 1106 provides power to the various components of the device 1100. The power supply component 1106 can include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 1100.

[0149] The multimedia component 1108 includes a screen that provides an output interface between the device 1100 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 1108 includes a front camera and / or a rear camera. When the device 1100 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0150] The audio component 1110 is configured to output and / or input audio signals. For example, the audio component 1110 includes a microphone (MIC), and when the device 1100 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 1104 or sent via the communication component 1116. In some embodiments, the audio component 1110 also includes a speaker for outputting audio signals.

[0151] I / O interface 1112 provides an interface between processing component 1102 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0152] The sensor assembly 1114 includes one or more sensors for providing various aspects of status assessment for the device 1100. For example, the sensor assembly 1114 can detect the open / closed state of the device 1100, the relative positioning of components, such as the display and keypad of the device 1100, and the sensor assembly 1114 can also detect the position change of the device 1100 or a component of the device 1100, the presence or absence of user contact with the device 1100, the orientation or acceleration / deceleration of the device 1100, and the temperature change of the device 1100. The sensor assembly 1114 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 1114 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 1114 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0153] The communication component 1116 is configured to facilitate wired or wireless communication between the device 1100 and other devices. The device 1100 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 1116 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1116 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0154] In an exemplary embodiment, device 1100 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0155] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1104 including instructions, and the instructions can be executed by the processor 1120 of the device 1100 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0156] A non-temporary computer-readable storage medium, when the instructions in the storage medium are executed by the processor of a terminal device, enables the terminal device to execute the above-mentioned data blood relationship generation method.

[0157] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0158] It should be further noted that, although the various steps in the flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0159] It should be understood that the above-mentioned device embodiments are only illustrative, and the device of the present application can also be implemented in other ways. For example, the division of units / modules in the above-mentioned embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0160] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present application may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.

[0161] If the integrated unit / module is implemented in the form of hardware, the hardware may be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. If not specifically stated, the processor may be any appropriate hardware processor, such as a CPU, a GPU, an FPGA, a DSP, an ASIC, etc. If not specifically stated, the storage unit may be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory RRAM (Resistive Random Access Memory), a dynamic random access memory DRAM (Dynamic Random Access Memory), a static random access memory SRAM (Static Random-Access Memory), an enhanced dynamic random access memory EDRAM (Enhanced Dynamic Random Access Memory), a high-bandwidth memory HBM (High-Bandwidth Memory), a hybrid memory cube HMC (Hybrid Memory Cube), etc.

[0162] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0163] In the above embodiments, the description of each embodiment has its own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0164] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0165] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for generating data kinship relationships, characterized in that: include: Obtaining a structured query statement; wherein the structured query statement includes a plurality of data storage locations, and the structured query statement represents a query logic for performing a data query through the data storage locations; The structured query statement is parsed by a preset syntax parser to obtain an abstract syntax tree corresponding to the structured query statement; wherein the abstract syntax tree represents the structured query statement; According to the abstract syntax tree corresponding to the structured query statement, based on preset syntax tree parsing rules, data lineage relationship information of the query logic represented by the structured query statement is obtained; wherein the data lineage relationship information represents the data flow in the query logic and the dependency relationship between the data.

2. The method according to claim 1, characterized in that: The structured query statement is parsed by a preset syntax parser to obtain an abstract syntax tree corresponding to the structured query statement, including: The structured query statement is subjected to lexical analysis by the preset grammar parser to obtain lexical units in the structured query statement; wherein the lexical units represent data storage locations in the structured query statement; According to preset grammatical rules, the lexical units are subjected to grammatical parsing to obtain an abstract syntax tree corresponding to the structured query statement; wherein the preset grammatical rules are used to connect the lexical units into an abstract syntax tree.

3. The method according to claim 2, characterized in that According to the preset grammatical rules, the lexical unit is parsed to obtain an abstract syntax tree corresponding to the structured query statement, including: According to the preset grammar rules, determining the nesting relationship between the lexical units in the structured query statement; wherein the nesting relationship represents the connection relationship between the nodes in the abstract syntax tree; According to the nested relationships between the lexical units and the lexical units, an abstract syntax tree corresponding to the structured query statement is obtained.

4. The method according to claim 1, characterized in that: According to the abstract syntax tree corresponding to the structured query statement, based on the preset syntax tree parsing rules, the data lineage relationship of the query logic represented by the structured query statement is obtained, including: Traversing the abstract syntax tree, and converting the abstract syntax tree into structured data in a preset format based on preset syntax tree parsing rules; According to the structured data, data lineage relationship information of the query logic represented by the structured query statement is obtained.

5. The method according to claim 4, characterized in that Traversing the abstract syntax tree and converting the abstract syntax tree into statement data in a preset format based on a preset syntax tree parsing rule, including: Traversing the abstract syntax tree to determine the currently traversed node; wherein the node in the abstract syntax tree represents the storage location of statement data in the structured query statement; According to the preset syntax tree parsing rules, the format of the currently traversed node is converted to obtain structured data corresponding to the currently traversed node; According to the structured data corresponding to each node in the abstract syntax tree and the connection relationship between each node in the abstract syntax tree, a code statement corresponding to the abstract syntax tree is obtained; wherein the code statement corresponding to the abstract syntax tree represents the abstract syntax tree.

6. The method according to claim 5, characterized in that After converting the format of the target node according to the preset syntax tree parsing rule to obtain structured data corresponding to the target node, the method further includes: If it is determined that the code statement data satisfies the preset semantic checking rule, data lineage relationship information of the query logic represented by the structured query statement is obtained according to the structured data.

7. The method according to claim 1, characterized in that Also includes: Generate graphic information in a preset format according to the blood relationship information, which is a blood relationship diagram; The blood relationship diagram is displayed on a visualization interface; wherein the blood relationship diagram represents data blood relationship information.

8. The method according to claim 7, characterized in that Also includes: In response to a query instruction of a user, obtaining a keyword identifier from the query instruction; wherein the keyword identifier represents a data storage location to be queried; The blood relationship diagram corresponding to the keyword identifier is queried, and the blood relationship diagram corresponding to the keyword identifier is displayed on a visualization interface.

9. A data kinship generation device, comprising: An acquisition unit, configured to acquire a structured query statement; wherein the structured query statement includes a plurality of data storage locations, and the structured query statement represents a query logic for performing a data query through the data storage locations; A parsing unit, configured to parse the structured query statement through a preset syntax parser to obtain an abstract syntax tree corresponding to the structured query statement; wherein the abstract syntax tree represents the structured query statement; A relationship generation unit is used to obtain data lineage relationship information of the query logic represented by the structured query statement based on the abstract syntax tree corresponding to the structured query statement and based on preset syntax tree parsing rules; wherein the data lineage relationship information represents the data flow in the query logic and the dependency relationship between the data.

10. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 8 when executed by a processor.

12. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 8 when being executed by a processor.

Images