A data blood relationship tracing method based on data entities

Abstract

The present application relates to a data blood relationship tracing method based on data entities, comprising: obtaining mapping associated data by establishing mapping with data entities, collecting data entity metadata blood relationship data records, and generating tracing data; performing blood relationship data definition, setting table level and field level blood relationship granularity; performing mapping data analysis, and analyzing field level and table level blood relationship of data entities; storing data of data entity mapping relationship and inter-table relationship according to the structure of data entity field level and table level blood relationship, obtaining data blood relationship level and mapping relationship data set corresponding to data entity ID; using recursive loop and breadth-first algorithm to traverse blood relationship data set, and performing blood relationship data analysis, organization and visual display. The method of the present application can show the context relationship of data objects of a data governance platform, and quickly analyze the influence between data objects.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a data blood relationship tracing method based on data entities. BACKGROUND

[0002] The mainstream methods of data tracing include labeling method, reverse query method, data tracking method, bidirectional pointer tracking method, and tracking method using graph theory and special query language. The labeling method is simple and convenient for data tracing, but it requires additional storage space to store the labeling information, which is not suitable for fine-grained data, especially for data tracing in large data sets. Although the reverse query method requires less storage space than the labeling method, it is more complex to implement and has certain limitations in application scenarios. The above methods are difficult to effectively meet the data blood tracing at the table and field level of data entities. The data blood relationship tracing adopts the data reverse query method combined with data relationship mapping and recursive breadth-first traversal algorithm to realize the tracing collection and query display of data blood relationship. It can meet the data blood tracing at the table and field level of data entities.

[0003] The principle of data blood relationship tracing is shown in Figure 1

[0004] The data blood relationship tracing is realized by establishing the mapping relationship with the data entities in the data life cycle, and then collecting the data relationship and evolution of the data entity metadata at different stages of data collection, storage, processing, transmission, exchange and archiving, generating data tracing links, and storing the data mapping relationship data, table relationship data and other data according to the blood relationship data structure.

[0005] When displaying the blood relationship of the selected data entity by query, the corresponding data mapping can be compared and associated according to the data entity ID. The blood relationship tracing algorithm is used to traverse the data and form a data set. According to the data set, the data blood relationship visualization effect of different levels of nodes is displayed by a specific graph drawing algorithm. SUMMARY

[0006] The purpose of the present application is to provide a data blood relationship tracing method based on data entities to realize the table-level and field-level blood relationship tracing.

[0007] To achieve the above purpose, the data blood relationship tracing method based on data entities comprises:

[0008] Step 1: Establish a mapping with the data entity, obtain the mapping associated data, and collect the data entity metadata blood relationship data record according to the mapping relationship, perform data labeling, and generate tracing data;

[0009] ​Step 2, according to the data entity traceability concern requirement, blood relationship data definition is carried out, and table level, field level blood relationship coarse and fine granularity setting is carried out;

[0010] Step 3, mapping data analysis is carried out by using blood relationship analysis algorithm, and the field level blood relationship and the table level blood relationship of the data entity are parsed out;

[0011] Step 4, according to the structure of the data entity field level blood relationship and the table level blood relationship, the data of the data entity mapping relationship and the inter-table relationship is stored, and the storage database adopts a relational database or a graph database;

[0012] Step 5, blood relationship traceability is carried out by querying the data entity, and the data blood relationship level corresponding to the data entity ID and the mapping relationship data set are obtained;

[0013] Step 6, the blood relationship data set is traversed by using recursive loop and breadth first algorithm, blood relationship data analysis, organization and visual display are carried out.

[0014] Further, the step 2 comprises establishing an ANTLR grammar file.

[0015] Further, the step 3 comprises generating a lexical syntax analysis class.

[0016] Further, the step 3 comprises tree parsing.

[0017] Further, the tree parsing comprises field table relationship parsing.

[0018] Further, the step of the tree parsing comprises: 1) parsing INSERT; 2) parsing SELECT; 3) processing the case containing asterisk, if the insertion field is not specified in the insertion statement, the field alias of the first layer query statement is used as the insertion field, if the field has no alias, the field name is specified as the insertion field; if the insertion field is empty and the query field is asterisk, this type of query parsing is not supported temporarily; processing the query containing asterisk, if the first layer query field is asterisk or contains asterisk, the asterisk is replaced with the complement of the insertion field list and the query field list; if the last layer query field contains asterisk, the asterisk is replaced with the complement of the parent query field and the query field of this layer; if the intermediate layer query contains asterisk, the asterisk is replaced with the complement of the child query field list and the query field list of the intermediate layer; the case not supported, the asterisk in the union is not supported, the asterisk in the parent-child query is not supported; 4) association relationship parsing.

[0019] The method has the following advantages:

[0020] Supporting the generation of data object blood relationship through lexical analysis technology and visual data processing, clearly showing the context relationship of data object of the data management platform, and quickly understanding the influence analysis between data objects on the data flow link through blood relationship. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 A schematic diagram of data blood relationship traceability principle is shown.

[0022] Figure 2 The blood relationship traceability implementation process is shown. DETAILED DESCRIPTION

[0023] The technical solutions of the present application will be described in detail below in conjunction with specific embodiments, but those skilled in the art should understand that the embodiments described below are only used to illustrate the present application, and should not be regarded as limiting the scope of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0024] Technical terms related to the present application:

[0025] Data mapping association is used to establish association mapping with data entity objects, and realize the contrast of data blood entity ID.

[0026] Data definition is used to mark the granularity of blood relationship record of data association mapping, such as table level, field level or table record level, etc.

[0027] Blood relationship analysis is used to analyze and process the blood relationship of data records such as entity metadata mapping relationship and table relationship according to data definition.

[0028] Blood relationship storage is used to store data after data blood relationship analysis processing. The storage content includes data entity metadata, blood mapping relationship, table relationship and other data.

[0029] Blood relationship query is used to query the selected data entity blood relationship, which can be selected through a visual interface or imported by SQL.

[0030] Blood relationship traversal analysis is used to analyze and analyze blood relationship according to specified model or method.

[0031] Blood relationship display is used to display the visual effect of blood relationship according to the blood relationship level and distribution of data entity.

[0032] Traceability application is established on the basis of data traceability and traceability access. The traceability application direction includes data quality, audit tracking, data reprocessing and data analysis scenes.

[0033] Data lineage, which reflects the origin of data, can help us track the source of data and the data processing process, and can also present the data source, conversion process, storage, etc. in a visual graph. Data lineage starts from a certain entity, traces back to its processing process, and ends at the data source interface of the data system. For different types of entities, the conversion process involved may be of different types, such as: for the underlying warehouse entity, the ETL processing process is involved. For warehouse summary tables, both ETL processing and warehouse summary processing may be involved. For indicators, in addition to the above processing, the indicator generation process is also involved. The data source interface entity is provided by the source system and serves as the data input of the data system. Other data entities have undergone one or more different types of processing. Lineage analysis provides users with the ability to understand different processing processes, inputs and outputs of each processing process, etc.

[0034] Through data lineage analysis, table-level and field-level lineage tracing can be achieved, and data maps can also be formed based on data lineage.

[0035] The implementation process of lineage tracing is shown in Figure 2 .

[0036] Data lineage tracing can realize data lineage tracing of data collection, processing, storage, application, etc. in the data lifecycle stage based on data entity metadata. The specific design and implementation process is as follows:

[0037] Step 1: Through mapping with data entities, mapping associated data is obtained, and data entity metadata lineage data records are collected based on the mapping relationship, data annotation is performed, and traceable data is generated.

[0038] Step 2: According to the data entity tracing demand, define the data lineage, and set the granularity of table-level and field-level data lineage.

[0039] Step 3: Use the lineage analysis algorithm to analyze the mapping data, and parse the field-level and table-level lineage of the data entity.

[0040] Step 4: According to the structure of the field-level and table-level lineage of the data entity, store the data entity mapping relationship, table relationship, etc. The storage database can use a relational database or a graph database.

[0041] Step 5: Query the data entity for lineage, and obtain the data lineage level and mapping relationship data set corresponding to the data entity ID.

[0042] Step 6, using recursive loop and breadth first algorithm to traverse the blood relationship dataset, blood relationship data analysis, organization and visualization. Embodiments

[0044] 1. Create ANTLR grammar file

[0045] Create tokens

[0046] INSERT_SELECT_STATEMENT token represents the entire INSERTSELECT statement, INSERT_COL_LIST represents the inserted column. Example grammar file:

[0047]

[0048]

[0049] 2. Generate lexical grammar class

[0050] ANTLRWORKS tool generates SqlLexer.java, SqlParser.java according to the lexical rules, tokens, and grammar in the grammar file.

[0051] SqlLexer.java analyzes the input stream according to the SQL lexical rules, and generates m rules for each lexical rule Method name for splitting input stream.

[0052] SqlParser.java. Construct a grammar analyzer according to the lexical analysis class, and apply the grammar rules to the token stream, and generate a tree for each grammar in the grammar file.

[0053] 3. Tree parsing

[0054] 1) Specific parsing process of tree

[0055] Root node

[0056] The root node name is the token INSERT_STATEMENT

[0057] First layer of child nodes

[0058] The first layer of child nodes is mainly divided into two categories, one is related to INSERT, and the other is the entire SELECT statement (including nested subqueries)

[0059] INSERT node

[0060] The child nodes of the INSERT node are TABLE_NAME, TABLE_ALIAS,

[0061] INSERT_COL_LIST, where the symbol TABLE_NAME refers to the name of the table to be inserted, TABLE_ALIAS refers to the table alias, and INSERT_COL_LIST refers to the inserted field. The table name, table alias, and table field name can be obtained by traversing each tree respectively.

[0062] INSERT_SELECT_STATEMENT node

[0063] The root node name of this node is INSERT_SELECT_STATEMENT, and its child nodes are SELECT_COL_EXPRS, SELECT_TAB_EXPRS, SELECT_WITH_EXPRS,

[0064] SELECT_UNION, SELECT_JOIN, SELECT_DBLINK nodes

[0065] SELECT_COL_EXPRS node

[0066] The root node name is SELECT_COL_EXPRS, and its main child node is COLUMN_EXPR, which is a column syntax symbol. The main child node of COLUMN_EXPR is ATOM_EXPR and SELECT_COL_ALIAS. ATOM_EXPR is a column name node, and its main child nodes are FUNCTION_EXPR and sql_identifier.

[0067] SELECT_TAB_EXPRS node

[0068] This node is a query table node, and the root node name is SELECT_TAB_EXPRS. Its main child nodes include SELECT_TABLE, which is a query table node, SELECT_TAB_ALIAS, which is a table alias node, SELECT_UNION, which is a union or unionall node, SELECT_JOIN, which is a join node, and SELECT_STATEMENT node. Traverse each node in turn to obtain the table name and table alias. Note that the table name of the subquery is null when traversed.

[0069] SELECT_TAB_ALIAS node

[0070] This node is a table alias node, and the table alias can be obtained by taking the child node.

[0071] SELECT_UNION node

[0072] The root node is named SELECY_UNION, and its main child nodes are SELECT_COL_EXPRS, SELECT_COL_ALIAS, SELECT_TAB_EXPRS, SELECT_TAB_ALIAS, and SELECT_WITH_EXPRS. The information of the union node can be obtained by sequentially traversing each child node.

[0073] SELECT_JOIN node

[0074] The root node is named SELECT_JOIN, and its main child nodes are SELECT_TABLE, SELECT_ALIAS, and ON_EXPRS. The information of the join node can be obtained by sequentially traversing each child node.

[0075] SELECT_DBLINK node

[0076] The root node is named SELECT_DBLINK, and its child nodes are AT_SIGN and SQL_IDENTIFIER.

[0077] 2) Field table relationship parsing process:

[0078] Due to the complex structure of the tree and the continuous nesting of downward recursion, it is difficult to parse the relationship between the target table, target field, and source table, source field by relying on single logical judgment. Therefore, auxiliary classes are used for parsing.

[0079] · Auxiliary class introduction

[0080] QueryEntity class records the information of each query statement parsing result. If there is a subquery, the entire subquery is replaced with a table name.

[0081] Table 1 QueryEntity class attributes

[0082]

[0083]

[0084] TableEntity class records the information of the query table.

[0085] Table 2 TableEntity class attributes

[0086]

[0087] FieldEntity class records the information of the field.

[0088] Table 3 FieldEntity class attributes

[0089]

[0090] UnionQueryEntity class, records information of union query.

[0091] Table 4 UnionQueryEntity class attributes

[0092]

[0093] RelationEntity class, parses result entity class.

[0094] Table 5 RelationEntity class attributes

[0095]

[0096] QueryComparator class

[0097] implements Comparator interface, used in List <queryentity>Sorts the specified list according to the order in the specified layer object. Collections. sort(List <queryentity>list, newQueryComparator()).

[0098] 3) Tree parsing step summary:

[0099] (1) Parsing INSERT

[0100] Separate the insert table name, alias, insert field name from the first layer of the main tree node. If there is no insert field name, set the field name to empty.

[0101] (2) Parsing SELECT

[0102] Parse SELECT by layers, construct a QueryEntity class object for each layer, and construct a FieldEntity class object for each query column, a TableEntity object for each query table, and a UnionQueryEntity class object if there is SELECT_UNION. Establish the relationship between the objects.

[0103] (3) Handle cases containing asterisks

[0104] · No insert field specified

[0105] If no insert field is specified in the insert statement, use the field alias of the first layer query statement as the insert field. If the field has no alias, specify the field name as the insert field. If the insert field is empty and the query field is an asterisk, this type of SQL query parsing is not supported.

[0106] · Handle queries containing asterisks

[0107] If the first layer query field is an asterisk or contains an asterisk, replace the asterisk with the complement of the insert field list and the query field list. If the last layer query field contains an asterisk, replace the asterisk with the complement of the parent query field and this layer query field.

[0108] If the intermediate layer query contains an asterisk, replace the asterisk with the complement of the child query field list and this layer query field list.

[0109] · Unsupported cases

[0110] Do not support union containing asterisks, do not support parent-child queries containing asterisks.

[0111] (4) Association relationship parsing

[0112] · Detailed algorithm

[0113] Firstly, the table entity list in the query entity is traversed. If the table name of the field is equal to the table alias, if the table type is join, a RelationEntity class object is constructed and put into the result list. If the table type is normal, if the table name is not empty, a RelationEntity class object is constructed and put into the result list. If the table name is empty and it is a subquery, jump to the next layer QueryEntity to traverse, and repeat the above operation steps.

[0114] For each layer QueryEntity entity, if the List <unionqueryentity>If the list is not empty, find the insert field corresponding to the input field index, table, construct the RelationEntity class object, and put it into the result list.

[0115] • Parsing result

[0116] The parsing result is put into the RelationEntity list, and one insert field can come from multiple tables and multiple fields.

[0117] Although the present application has been described in detail with general description and specific embodiments above, some modifications or improvements can be made on the basis of the present application, which is obvious to those skilled in the art. Therefore, these modifications or improvements made on the basis of not deviating from the spirit of the present application, all belong to the scope of the present application claimed.< / unionqueryentity> < / queryentity> < / queryentity>

Claims

1.A data lineage tracing method based on data entities, comprising: Step 1, obtaining mapping associated data by establishing a mapping with data entities, and collecting data entity metadata lineage data records according to the mapping relationship, performing data labeling, and generating tracing data; Step 2, defining the lineage data according to the data entity tracing concern requirements, and setting the table-level and field-level lineage relationship granularity; Step 3, using a lineage analysis algorithm to analyze the mapping data, and parsing the field-level and table-level lineage relationship of the data entity; including generating a lexical syntax analysis class and tree parsing, the tree parsing includes field table relationship parsing; the steps of the tree parsing include: 1) parsing INSERT; 2) parsing SELECT; 3) processing the case containing an asterisk; 4) association relationship parsing; the parsing SELECT includes: hierarchical parsing SELECT, constructing a QueryEntity class object at each level to record the information of each query statement parsing result, constructing a FieldEntity class object for each query column to record the information of the field, constructing a TableEntity class object for each query table to record the information of the query table, and constructing a UnionQueryEntity class object for SELECT_UNION to record the information of the union query, and establishing the relationship between the objects; The association relationship analysis includes: firstly traversing the table entity list in the query entity, if the table name of the field is equal to the table alias, if the table type is join, then constructing a RelationEntity class object and putting it into the result list, if the table type is normal, if the table name is not empty, then constructing a RelationEntity class object and putting it into the result list, if the table name is empty and it is a subquery, then jumping to the next layer QueryEntity to traverse, for each layer QueryEntity entity, if the List <unionqueryentity>if the list is not empty, find the inserted field and table corresponding to the input field index, construct a RelationEntity class object, and put it into the result list, and put the parsing result into the RelationEntity list;< / unionqueryentity> Step 4, storing the data entity mapping relationship and the data of the relationship between tables according to the structure of the data entity field-level and table-level lineage relationship, and using a relational database or a graph database for storage; Step 5, querying the data entity to trace the lineage relationship, and obtaining the data lineage relationship level and the mapping relationship data set corresponding to the data entity ID; Step 6, using recursive loop and breadth-first algorithm to traverse the lineage relationship data set, and performing lineage relationship data analysis, organization and visualization display. 2.The data entity-based data provenance tracing method of claim 1, wherein, The step 2 includes establishing an ANTLR grammar file. 3.The data entity-based data provenance tracing method of claim 1, wherein, The processing of the case containing an asterisk in step 3 includes, if the inserted field is not specified in the insert statement, using the field alias of the first layer query statement as the inserted field, and if the field has no alias, using the field name as the inserted field; if the inserted field is empty and the query field is an asterisk, the asterisk type query parsing is temporarily not supported; processing the query containing an asterisk, if the first layer query field is an asterisk or contains an asterisk, replacing the asterisk with the complement of the inserted field list and the query field list; if the last layer query field contains an asterisk, replacing the asterisk with the complement of the parent query field and this layer query field; if the intermediate layer query contains an asterisk, replacing the asterisk with the complement of the child query field list and the query field list of the intermediate layer; the unsupported cases include not supporting the asterisk in union, and not supporting the asterisk in parent-child query.

Images