Column operator bloodline construction method, server, computer readable storage medium

Abstract

The application provides a column operator blood relationship construction method, which is applied to the technical field of computers and comprises the following steps: S1) generating a parse tree ParseTree by parsing SQL through Antlr; S2) designing an abstract syntax tree AST, which is used for constructing blood relationship links from input columns to output columns and intermediate processing logic; S3) recursively traversing the parse tree ParseTree obtained in S1) to construct the abstract syntax tree AST designed in S2); S4) traversing the abstract syntax tree AST constructed in S3) and extracting a column operator blood relationship model; and S5) traversing the column operator blood relationship model and constructing point-edge relationships, and storing the column operator blood relationship into a graph database. The application provides the most fine-grained SQL parsing capability, helps users understand the operator-level blood relationship of data assets, includes column direct sources, used functions, processing caliber and indirect influences, and solves the problem that users cannot effectively enable business due to the lack of understanding of data assets.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to a column operator blood relationship construction method, a server and a computer readable storage medium. BACKGROUND

[0002] More and more enterprises need to carry out digital transformation, and through data analysis to empower business development, which has become an inevitable path for future enterprise development. However, in order to adapt to the development of enterprises, the data processing mode and processing logic in the enterprise are becoming more and more complex, and over time, it often leads to that users cannot quickly and conveniently understand the processing logic of historical data, and cannot quickly respond to increasingly agile business requirements. In order to not affect the agile implementation of data analysis, users need to write comments for processing scripts or write a complete document to backup the thinking logic of processing data at that time, so as to try to avoid errors in analysis results due to the lack of understanding of data when data analysts analyze data. However, this depends on the self-consciousness of users, and the knowledge landing in the enterprise cannot form large-scale at present.

[0003] Through the blood relationship of data, the upstream and downstream blood relationships of data assets can be inquired, and the existing technology can only explore the upstream and downstream paths of data assets, such as what is the upstream and downstream of the library, table and column. The current solution is basically as follows: parsing the processing logic script of data assets, forming table-level or column-level data blood relationship, and allowing users to understand the source and application of data in a timely manner through visualization, and providing a basis for subsequent data analysis; The defects of the above method are as follows, using the following code:

[0004] INSERT INTO T1(C1)

[0005] SELECT MAX(T2.C1+T3.C1)as C1 FROM T2,T3 WHERE T2.C2>1;

[0006] Among them, the table blood relationship can only show the processing relationship between tables, for example, the T1 table is derived from the upstream T2 and T3 tables, the column-level blood relationship can show that T1.C1 is derived from T2.C1 and T3.C1, but cannot know that T1.C1 is obtained by adding T2.C1 and T3.C1 through the MAX function, and also cannot know that the data range of T1.C1 is affected by the screening of T2.C2.

[0007] Therefore, the processing logic of different data assets may be consistent in terms of source used, such as the same table or column, but there are a large number of conversion rules for the real business logic of the data assets, and users cannot intuitively understand how the conversion, calculation and movement of data at each time affect the data through the existing way, which may lead to incomplete understanding of the users and still need to consult the developers offline, and cannot meet the existing agile business requirements, that is, only know where the column comes from, but do not know how it comes.

[0008] On the other hand, the direct data source of the data asset and the indirectly processed data source represent different data meanings and business meanings, and through the existing way, the user may develop a large-scale expansion of the range of upstream sources and downstream applications, and cannot intuitively perceive the direct source of the data, which may lead to abnormal data analysis, and now the processing logic needs to be recorded by manual query path to query the source of the data;

[0009] Furthermore, the label diffusion based on column-level blood relationship is also inaccurate, because the meaning carried by the data may change through layer-by-layer processing, because the program cannot know how the column is derived by processing, and the diffusion accuracy may be greatly reduced through layer-by-layer downward label diffusion. SUMMARY

[0010] The present application provides the most fine-grained SQL parsing capability to help users understand the operator-level blood relationship of the data asset, including the direct source of the column, the used function, the processing range and the indirect influence, and solves the problem that the user cannot effectively enable the business due to the misunderstanding of the data asset.

[0011] In order to solve the above-mentioned main technical problems, the following technical solutions are adopted:

[0012] The column operator blood relationship construction method comprises the following steps,

[0013] S1) generating a parse tree ParseTree by parsing SQL through Antlr;

[0014] S2) designing an abstract syntax tree AST for constructing the blood relationship link of the input column to the output column and the intermediate processing logic;

[0015] S3) recursively traversing the parse tree ParseTree obtained in S1) to construct the abstract syntax tree AST designed in S2);

[0016] S4) traversing the abstract syntax tree AST constructed in S3) and extracting the column operator blood relationship model;

[0017] S5) traversing the column operator blood relationship model and constructing point-edge relationship, and storing the column operator blood relationship into a graph database.

[0018] Preferably, S2) comprises the following steps:

[0019] S21) dividing a complete SQL into at least one paragraph based on relational algebra, each paragraph being a trunk, designing a tree structure Scope for abstracting the corresponding trunk of the SQL, and establishing a hierarchical relationship between the trunks and the trunks;

[0020] S22) dividing the Scope into two types of input type Scope and output type Scope;

[0021] Among them, the input type Scope is divided into ProjectScope, JoinScope, UnionScope and ScanScope, which are used to abstract the input part of the SQL statement;

[0022] The output type Scope is divided into CreateAsScope and InsertScope, which are used to abstract the output part of the SQL statement;

[0023] Among them: each Scope contains the type of the Scope, the alias of the Scope and the fields exposed by the Scope, the parent-child relationship between different Scopes is recorded by indexing, the outer trunk set of the Scope is set as parentScopeList, that is, the parent Scope set, the inner trunk set of the Scope is set as childrenScopeList, that is, the child Scope set, thereby establishing the parent-child relationship between different Scopes;

[0024] S23) defining the six types of Scopes divided in S22) for corresponding to the abstract syntax tree AST;

[0025] S24) defining the exposed fields of each Scope as the holding field set of the Scope, and each holding field set of the Scope can be referenced by the parent Scope in the parentScopeList;

[0026] S25) attaching a source attribute to each holding field in the holding field set

[0027] ExpressionOrigin to record the source information of the holding field, which is used to build the blood lineage link between the input column and the output column;

[0028] S26) establishing an expression for recording the processing logic of the column and the source of the column;

[0029] S27) Each Scope fills its holding field information with the holding field information of its children Scope in childrenScopeList, and passes the holding field information of the current Scope to its parent Scope, so as to build the processing link between the output column and the source column, and further extract the column operator blood relationship information.

[0030] Preferably, S3) comprises the following steps:

[0031] S31) recursively traversing the parse tree ParseTree generated by S1) from the root node;

[0032] S32) interpreting the root node as CreateAsScope or InsertScope according to the type of SQL, and then traversing the child node;

[0033] S33) If the node type is a query node select xx from, it is interpreted as ProjectScope, and the expressions after SELECT, WHERE, GROUP BY, HAVING and ORDER BY are built. The child node after FROM is parsed as the corresponding Scope, and is added to the childScopeList of the current ProjectScope. The holding field information of the Scope in the childScopeList fills the source information of the expression of the current ProjectScope, and then the expression information is used to fill the holding field of the current ProjectScope;

[0034] S34) If the child node after FROM is a physical table node, a ScanScope is created and set as the current Scope, and the holding field of the current Scope is filled according to the metadata information;

[0035] If the child node after FROM is a subquery node, jump to S33) for recursive processing;

[0036] If the child node after FROM contains more than one JOIN node, a JoinScope is created and set as the current Scope, the expressions after all ON conditions are built, and each JoinItem on both sides of the JOIN is jumped to S33) for recursive processing. The holding field of the final JoinScope is built in the form of the holding field of each child Scope in the childScopeList, and the source information of all ON expressions is filled;

[0037] If the child node after FROM contains more than one UNION node, create a UnionScope set as the current Scope, and jump to S33) to recursively process each side of the UNION in the UnionItem; the holding field of the final UnionScope is constructed by stacking the holding field of each child Scope in the childScopeList;

[0038] S35) Traverse the parse tree ParseTree by S32)-S34) to construct the abstract syntax tree AST designed in S2).

[0039] Preferably, S4) includes the following steps:

[0040] S41) Extract information from the Scope to construct a column operator blood relationship model, and divide the column operator blood relationship model into an operator Operator, a physical column Column, and a virtual column VirtualColumn;

[0041] Divide the operator Operator into six types, including:

[0042] SELECT operator: the content of each projection item, i.e., the content after SELECT and before AS, which is extracted from the ProjectScope;

[0043] WHERE operator: the content after WHERE, which is extracted from the ProjectScope;

[0044] GROUP operator: the content after GROUP, which is extracted from the ProjectScope;

[0045] HAVING operator: the content after HAVING, which is extracted from the ProjectScope;

[0046] JOIN operator: the content after the ON condition, which is extracted from the JoinScope;

[0047] UNION operator: used to aggregate multiple SELECT statements, which is extracted from the UnionScope;

[0048] Among them, the SELECT and UNION operators belong to direct blood relationship, which is used to trace the processing logic and processing link of the column, and the WHERE, GROUP, HAVING, and JOIN operators belong to indirect blood relationship, which is used for column impact analysis;

[0049] The physical column Column includes a real input physical column and an output physical column, which belongs to metadata information, and the input physical column is extracted from the ScanScope, and the output physical column is extracted from the CreateAsScope or InsertScope.

[0050] VirtualColumn: alias part in SELECT item AS alias in subquery, extracted from ProjectScope.

[0051] The second aspect of the application also proposes a server comprising a memory, a processor and a computer program stored on the memory and executable on the processor, said processor implementing the steps of the method when executing said program.

[0052] The third aspect of the application also proposes a computer-readable storage medium having stored thereon a computer program, the program implementing the steps of the method when executed by a processor.

[0053] The beneficial effects of the present application are: compared with the prior art, the column operator blood relationship construction method 1) analyzes the detailed logic of all direct and indirect dependency relationships between data assets in complex enterprise data environment by deeply analyzing the processing logic script of data assets, translates the real data processing code into user-friendly expression ability, avoids self-help data analysis due to data misunderstanding or data source application inflation in data analysis process, and affects the response of agile business requirements;

[0054] 2) Column operator blood relationship is a blood relationship of finer granularity, which can analyze column direct source, used function, processing caliber and indirect influence through column operator blood relationship. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained according to these drawings without creative labor for those skilled in the art.

[0056] Figure 1 The system flowchart of the present application is shown in the figure.

[0057] Figure 2 The system diagram of the six scopes of the present application is shown in the figure.

[0058] Figure 3 The relationship diagram of the six scopes of the present application is shown in the figure.

[0059] Figure 4 The step 3) recursive traversal flowchart of the present application is shown in the figure.

[0060] Figure 5 The scope construction and blood relationship filling relationship diagram of the present application is shown in the figure.

[0061] Figure 6 Flow chart for extracting operators in Scope of the present application;

[0062] Figure 7 Schematic diagram of extracted operators in Scope of the present application and Scope structure;

[0063] Figure 8 Schematic diagram of point edge storage in a graph database. DETAILED DESCRIPTION

[0064] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0065] Embodiment 1

[0066] Please refer to Figure 1 , column operator blood relationship construction method, comprising the following steps:

[0067] S1) generating a parse tree ParseTree by parsing SQL through Antlr; the parse tree ParseTree generated after parsing SQL through Antlr is only an abstract display of information of each node of the original SQL, and cannot directly extract column operator blood relationship information therefrom. Therefore, a kind of abstract syntax tree AST needs to be re-designed according to the basic syntax of relational algebra and SQL, which decomposes the SQL into several main stems and expressions, and records the blood relationship between the main stems and the main stems, and the blood relationship between the expressions and the expressions, to achieve the purpose of extracting column operator blood relationship.

[0068] Extracting column operator blood relationship requires that SQL has an input part and an output part, i.e. a query part and an output part, so the SQL that can extract column operator blood relationship in the present patent is of two types: INSERT INTO TABLE SELECT… and CREATE TABLE AS SELECT… The input part is a DQL type statement, and the output part is INSERT or CREATE.

[0069] S2) designing an abstract syntax tree AST for constructing blood relationship links from input columns to output columns and intermediate processing logic;

[0070] The structure of the above abstract syntax tree AST is as follows:

[0071] 1) Based on relational algebra, a complete SQL is divided into several paragraphs, each paragraph is a main body, this patent defines a new tree structure Scope to abstract the corresponding main body of SQL, and can reflect the hierarchical relationship between the main body and the main body. Each Scope contains the type of the Scope, the alias of the Scope and the field exposed by the Scope, the parent-child relationship between different Scopes is recorded by indexing, the outer main body set of the Scope is set as parentScopeList, that is, the parent Scope set; The inner main body set of the Scope is set as childrenScopeList, that is, the child Scope set, thereby constructing the parent-child relationship between different Scopes;

[0072] 2) The above-mentioned SQL supporting the extraction of column operator blood relationship includes input part and output part. Therefore, Scope is divided into two types: input type Scope and output type Scope; The input type Scope is divided into ProjectScope, JoinScope, UnionScope and ScanScope, which mainly abstracts the input part in the SQL statement; The output type Scope is divided into CreateAsScope and InsertScope, which mainly abstracts two main output parts in the SQL statement;

[0073] 3) Please refer to Figure 2The six scopes declared in 2) are defined as follows. If the SQL is CREATE TBALE table_name AS, it is interpreted as a CreateAsScope. Since CREATE TBALE table_name AS is at the beginning of the SQL as the output part, it corresponds to the root node in the tree structure Scope. The part after AS is interpreted as another type of Scope, and the parent-child relationship is recorded by index. If the SQL is INSERT INTO table_name(col1, col2…) SELECT, the part before SELECT is interpreted as an InsertScope. Similarly, since INSERT INTO table_name(col1, col2…) is also at the beginning of the SQL as the output part, it corresponds to the root node in the tree structure Scope. The part after AS is interpreted as another type of Scope, and the parent-child relationship is recorded by index. A ProjectScope is used to abstract the structure of a SELECT statement, and includes the projection part, the WHERE part, the GROUP BY part, the HAVING part, and the ORDER BY part of a SELECT statement. If the SELECT statement ends with UNION SELECT…, the first SELECT statement part to the last SELECT statement part are abstracted as a UnionScope. Each SELECT statement part is a separate ProjectScope, and is recorded in the childScopeList collection of the UnionScope by index. In the ProjectScope, the content after FROM and before WHERE is interpreted as a new Scope, and is recorded in the children Scope collection of the current layer Scope, i.e., the inner trunk. If the FROM is a physical table name table_name, the physical table name table_name is interpreted as a ScanScope, and the parent-child relationship is recorded by index. In the tree structure Scope designed in the present application, the leaf node is a ScanScope. If the FROM is followed by a JOIN part, the part after FROM and before WHERE is interpreted as a JoinScope. The alias of the JoinScope is empty. The content on both sides of each JOIN is parsed as a corresponding Scope, and the ON condition content of each JOIN is recorded.If the FROM is followed by a subquery AS alias, the subquery is also interpreted as ProjectScope, and the alias is recorded as alias, and the parent-child relationship is recorded through the index. If the FROM is followed by multiple subqueries UNIONed together, i.e. (subQuery UNION subQuery…) AS alias, this part is interpreted as UnionScope. The same applies to other scopes.

[0074] Please refer to Figure 3 In the design of the Scope abstract syntax tree AST, the parent-child relationship of different Scopes is as follows:

[0075] CreateAsScope or InsertScope, their parentScopeList must be empty, and childrenScopeList must not be empty and be one of ProjectScope or UnionScope.

[0076] ProjectScope's parentScopeList is not empty and is one of CreateAsScope, InsertScope, JoinScope, ProjectScope, UnionScope, and childrenScopeList is not empty and is one of JoinScope, ProjectScope, UnionScope, ScanScope.

[0077] JoinScope's parentScopeList is not empty and must be ProjectScope, and childrenScopeList is not empty and has at least two or more, being any combination of ProjectScope, UnionScope, ScanScope.

[0078] UnionScope's parentScopeList is not empty and is one of CreateAsScope, InsertScope, JoinScope, ProjectScope, UnionScope, and childrenScopeList is not empty and has at least two or more, being any combination of ProjectScope, JionScope.

[0079] ScanScope's parentScopeList is not empty and is one of ProjectScope, JoinScope, and childrenScopeList must be empty.

[0080] 4) Define the exposed field of each Scope as the holdFields of the Scope, the holdFields of each Scope can be referenced by its parentScopeList; Where the holdFields of ScanScope is the metadata information, that is, the column name set under the physical table is the holdFields of the ScanScope. For example, the t1 table has a, b, and c columns, and {“t1”:[“a”,“b”,“c”]} is the holdFields of the ScanScope. The holdFields of the ProjectScope is the projection part in the SELECT statement, for example, SELECT a, b, c as c1 FROM, the alias is alias, if there is none, it is an empty string, and the holdFields of the ProjectScope is {“alias”:[“a”,“b”,“c1”]}. The holdFields of the UnionScope is the superposition of the holdFields of each child Scope, for example, (SELECT d, e, f) UNION (SELECT g, h, i), and the holdFields of the UnionScope is {“alias”:[“d”,“e”,“f”]}. The holdFields of the JoinScope is the summary of the holdFields of each child Scope, for example, t1 JOIN (SELECT d, e, f), where t1 has a, b, and c columns, and the holdFields of the JoinScope is {“”:[a,b,c,d,e,f]}, and the alias of the JoinScope must be empty, so the key of the holdFields set is empty.

[0081] The holdFields of the CreateAsScope is the holdFields of the childrenScope, for example, CRAETETABLE table_name AS SELECT a, b, c, and the holdFields of the Scope is {“table_name”:[“a”,“b”,“c”]}. The holdFields of the InsertScope is the metadata column information set of the target table, for example, INSERT INTO table_name(a, b, c) SELECT d, e, f, and the holdFields of the Scope is {“table_name”:[“a”,“b”,“c”]}.

[0082] 5) The above design of Scope has a parent-child relationship, that is, a source relationship, so that each Scope's holding field also has a source relationship. Here, a source attribute ExpressionOrigin is added to the holding field in the holding field set, that is, which holding field in which Scope in the childrenScopeList is the source of the holding field. The holding field can be traced back from the root Scope, that is, the CreateAsScope or InsertScope, to the holding field of the physical layer ScanScope, thereby constructing the bloodline link between the input column and the output column.

[0083] 6) The processing logic of the column cannot be directly observed only by the holding field. The patent defines Expression to record the expression form of the column. The Expression can be a projection item, for example, SELECT MAX(a+b) AS col1, c AS col2, 1 AS col3, wherein MAX(a+b) is an Expression, and c and 1 are also an Expression. Similarly, the content after WHERE is also an Expression, for example, WHERE c1>c2 AND c3=c4 OR c5+c6–MAX(c7), wherein c1>c2 AND c3=c4 OR c5+c6–MAX(c7) is an Expression as a whole.

[0084] Each subpart after GROUP BY and ORDER BY is also an Expression. The Expression is designed as a tree structure in the storage structure, each node has the type of the current Expression, the reference of each sub-Expression, and the source attribute ExpressionOrigin for storing the source information of the Expression, which is the same attribute as the source attribute ExpressionOrigin of the holding field. Through the Expression, it can be clearly known what the processing logic of the column is, and through the source information of the Expression, it can be known what the source of the column is. Finally, through the holding field layer by layer transmission, the processing link between the output column and the source column can be constructed, and then the column operator bloodline can be extracted.

[0085] S3) recursively traverse the parse tree ParseTree obtained in S1) to build the abstract syntax tree AST designed in S2); the process of building Scope and Expression is as follows: building the column operator lineage between the output column and the source column can be converted into building the column dependency relationship between the current Scope and the inner Scope in the childrenScopeList, further, each holding field can only come from his "subquery", that is, from the childrenScopeList, therefore, "column operator lineage tracing and building" can be converted into "the filling problem of the original field held by the subquery". The specific process is as follows:

[0086] Please refer to Figure 4 and 5 ,

[0087] S31) recursively traverse the parse tree ParseTree generated in S1) from the root node;

[0088] S32) according to the type of SQL, interpret the root node as CreateAsScope or InsertScope, and then traverse the child node;

[0089] S33) if the node type is a query node select xx from, interpret it as ProjectScope, and build the expression after SELECT, WHERE, GROUP BY, HAVING and ORDER BY; continue to interpret the child node after FROM as the corresponding Scope, and add it to the childScopeList of the current ProjectScope, wait for the holding field information of the Scope in the childScopeList to fill in the source information of the expression of the current ProjectScope, and then use the expression information to fill in the holding field of the current ProjectScope;

[0090] S34) if the child node after FROM is a physical table node, create ScanScope and set it as the current Scope, and fill in the holding field of the current Scope according to the metadata information;

[0091] if the child node after FROM is a subquery node, jump to S33) for recursive processing;

[0092] If the child node after FROM contains more than one JOIN node, create a JoinScope set as the current Scope, build all expressions after ON conditions, and jump each JoinItem on both sides of JOIN to S33) for recursive processing; the holding field of the final JoinScope is built in the manner of widening the holding field of each child Scope in childScopeList, and fill in the source information of all ON expressions;

[0093] If the child node after FROM contains more than one UNION node, create a UnionScope set as the current Scope, and jump each UnionItem on both sides of UNION to S33) for recursive processing; the holding field of the final UnionScope is built in the manner of superimposing the holding field of each child Scope in childScopeList;

[0094] Please refer to Figure 6 , Figure 7 , S4) traverses the column lineage link model to extract column operators; after obtaining the above abstract syntax tree AST, it is necessary to traverse the root Scope to extract the column operator lineage model, which contains three major contents: Operator, physical column Column, and virtual column VirtualColumn.

[0095] In this patent, the Operator in the column operator lineage is named Operator, which has six types:

[0096] Define SELECT operator: the content of each projection item, that is, the content before AS after SELECT, which is extracted from ProjectScope; WHERE operator: the content after WHERE, which is extracted from ProjectScope; GROUP operator: the content after GROUP, which is extracted from ProjectScope; HAVING operator: the content after HAVING, which is extracted from ProjectScope; JOIN operator: the content after ON conditions, which is extracted from JoinScope; UNION operator: used to aggregate multiple SELECT statements, which is extracted from UnionScope.

[0097] Among them, SELECT and UNION operators belong to direct lineage, which are used to trace the processing logic and processing link of columns, and WHERE, GROUP, HAVING, and JOIN operators belong to indirect lineage, which are used for column impact analysis.

[0098] The physical column Column is the real input physical column and output physical column, and belongs to metadata information.

[0099] Please refer to Figure 7 , for example: INSERT INTO t2(a,b)SELECT tt1.a AS a,tt1.b AS b FROM(SELECT a AS a,MAX(b+c)AS b FROM t1 WHERE d>1)AS tt1.

[0100] The column operator blood relationship model is that from the input physical column, after each layer of subquery, a group of operators and virtual columns are output, wherein the operator is output to the virtual column, and the operator of the outermost query is directly output to the output physical column itself and is not output to the virtual column, and the link of the column operator blood relationship model is as follows:

[0101] Input physical column-->(operator-->virtual column)-->(…)--->operator-->output physical column.

[0102] That is, Column-->(Operator-->VirtualColumn)-->(…)--->Operator-->Column.

[0103] Please refer to Figure 8 , S5) traversing the operator blood relationship construction point edge relationship, and storing the column operator blood relationship into the graph database.

[0104] In the graph database, the corresponding physical column, operator and virtual column three point types are created, the column operator blood relationship model obtained in S4) is traversed, the corresponding point edge relationship is constructed, and is stored into the graph database:

[0105] INSERT INTO t2(a,b)SELECT tt1.a AS a,tt1.b AS b FROM(SELECT a AS a,MAX(b+c)AS b FROM t1 WHERE d>1)AS tt1.

[0106] Figure 8 The schematic diagram of the column operator blood relationship of the SQL stored in the graph database.

[0107] Embodiment 2

[0108] The application provides a server, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in embodiment 1 when executing the program.

[0109] Example 3

[0110] The present application provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, carries out the steps of the method described in Example 1.

[0111] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, device, or computer program product. Therefore, the embodiments of the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0112] The above-described embodiments are merely preferred embodiments of the present application and are not intended to limit the present application. Although the present application has been described in detail with reference to the foregoing embodiments, the technical solutions recorded in the foregoing embodiments can still be modified or some technical features can be replaced by equivalent features by those skilled in the art, and any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for constructing a column operator family, characterized in that, Comprising the following steps, S1) generating a parse tree ParseTree by parsing SQL through Antlr; S2) designing an abstract syntax tree AST for constructing bloodline links of input columns to output columns and processing logic in the middle; S3) recursively traversing the parse tree ParseTree obtained in S1) to construct the abstract syntax tree AST designed in S2); S4) traversing the abstract syntax tree AST constructed in S3) and extracting a column operator bloodline model; S5) traversing the column operator bloodline model and constructing point-edge relationships, and storing the column operator bloodline into a graph database; S2) comprising the following steps: S21) based on relational algebra, dividing a complete SQL into at least one paragraph, each paragraph being a main trunk, designing a tree structure Scope for abstracting the corresponding main trunk of the SQL, and establishing a hierarchical relationship between the main trunks; S22) Scope is divided into two categories: input type Scope and output type Scope; Among them: the input type Scope is divided into ProjectScope, JoinScope, UnionScope and ScanScope, which are used to abstract the input part in the SQL statement; The output type Scope is divided into CreateAsScope and InsertScope, which are used to abstract the output part in the SQL statement; Among them: each Scope contains the type of the Scope, the alias of the Scope, and the fields exposed by the Scope, the parent-child relationship between different Scopes is recorded through indexing, the outer main trunk set of the Scope is set as parentScopeList, that is, the parent Scope set; the inner main trunk set of the Scope is set as childrenScopeList, that is, the child Scope set, thereby constructing the parent-child relationship between different Scopes; S23) defining the six kinds of Scope divided in S22) for corresponding to the abstract syntax tree AST; S24) defining the exposed fields of each Scope as the holding field set of the Scope, and the holding field set of each Scope can be referenced by the parent Scope in the parentScopeList; S25) attaching a source attribute ExpressionOrigin to each holding field in the holding field set to record the source information of the holding field, which is used to construct the bloodline link between the input column and the output column; S26) establishing an expression for recording the processing logic of the column and the source of the column; S27) each Scope uses the holding field information of the child Scope in its childrenScopeList to fill the holding field information of the current Scope, and then passes the holding field information of the current Scope to the parent Scope, thereby constructing the processing link between the output column and the source column, and further extracting the column operator bloodline information.

2. The column operator lineage construction method of claim 1, wherein, S3) comprising the following steps: S31) recursively traversing the parse tree ParseTree generated in S1) from the root node; S32) According to the type of SQL, the root node is interpreted as CreateAsScope or InsertScope, and then the child nodes are traversed; S33) If the node type is a query node select xx from, it is interpreted as ProjectScope, and the expressions after SELECT, WHERE, GROUP BY, HAVING, and ORDER BY are constructed. The child nodes after FROM are parsed as corresponding Scope, and added to the childScopeList of the current ProjectScope. The holding field information of the Scope in the childScopeList is used to fill in the source information of the expressions of the current ProjectScope, and then the holding field of the current ProjectScope is filled with the expression information; S34) If the child node after FROM is a physical table node, a ScanScope is created and set as the current Scope, and the holding field of the current Scope is filled according to the metadata information; If the child node after FROM is a subquery node, go to S33) for recursive processing; If the child node after FROM contains more than one JOIN node, a JoinScope is created and set as the current Scope, the expressions after all ON conditions are constructed, and each JoinItem on both sides of the JOIN is recursively processed by jumping to S33); finally, the holding field of the JoinScope is constructed in a wide manner with the holding field of each subScope in the childScopeList, and the source information of all ON expressions is filled; If the child node after FROM contains more than one UNION node, a UnionScope is created and set as the current Scope, and each UnionItem on both sides of the UNION is recursively processed by jumping to S33); finally, the holding field of the UnionScope is constructed in a stacked manner with the holding field of each subScope in the childScopeList; S35) The column operator bloodline model is constructed by traversing and parsing the tree ParseTree in S32)-S34).

3. The column operator lineage construction method of claim 1, wherein, S4) includes the following steps: S41) Extract information from Scope to construct column operator bloodline model, divide column operator bloodline model into Operator, physical column Column and virtual column VirtualColumn; The Operator is divided into 6 types, including: SELECT operator: the content of each projection item, i.e. the content after SELECT and before AS, extracted from ProjectScope; WHERE operator: the content after WHERE, extracted from ProjectScope; GROUP operator: the content after GROUP BY, extracted from ProjectScope; HAVING operator: the content after HAVING is extracted from ProjectScope; JOIN operator: the content after ON condition is extracted from JoinScope; UNION operator: used to aggregate multiple SELECT statements, extracted from UnionScope; Among them, SELECT and UNION operators belong to direct blood relationship, used to trace the processing logic and processing link of columns, WHERE, GROUP, HAVING and JOIN operators belong to indirect blood relationship, used to analyze the influence range of columns; Physical column Column includes real input physical column and output physical column, which belongs to metadata information, the input physical column is extracted from ScanScope, and the output physical column is extracted from CreateAsScope or InsertScope; Virtual column VirtualColumn: alias part in SELECT item AS alias in subquery, extracted from ProjectScope.

4. A server, characterized by A computer program product, comprising a memory and a processor, wherein the processor executes the program to implement the steps of the method of any one of claims 1-3.

5. A computer readable storage medium, characterized in that, A computer program product, comprising a memory and a processor, wherein the processor executes the program to implement the steps of the method of any one of claims 1-3.

Images