Data lineage tracking method, apparatus, and electronic device

Abstract

The application relates to a data bloodline tracking method and device and electronic equipment, and belongs to the technical field of data security. The method comprises the following steps: acquiring metadata of a target database, and performing marking processing on the metadata according to a hierarchical dimension of the metadata to obtain marked data, wherein the hierarchical dimension comprises a library, a table and a field; each operation statement sent to the target database is analyzed and processed, a data flow log of the target database is constructed and generated based on an analysis result and the marked data, the data flow log carries record data representing data flow information in the target database; and data bloodline tracking is performed on the target data based on the data flow log. The technical scheme can effectively realize field-level data bloodline tracking, and effectively adapt to different database types based on specific analysis configurations.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data security, and particularly relates to a data blood relationship tracking method and device and electronic equipment. BACKGROUND

[0002] Blood relationship tracking (or blood relationship analysis) is a technical means for realizing comprehensive tracking of a data processing process, so as to find all related metadata objects of a certain data object as a starting point and the relationship between the metadata objects. In the current environment, the data of each enterprise is in the process of interaction within the enterprise and with other enterprises, and the database generates new data, and the data is also transferred, fused, cleaned and the like. Based on such application practice, the data blood relationship is a process chain from the generation of a table to the formation of a new table after the table is subjected to a series of actions, and a relationship graph composed of associated data directly or indirectly related to the table. When data analysis is performed, the traceability analysis of data sources and data causes cannot be implemented without blood relationship analysis of tables and tables, table fields and table fields.

[0003] The current blood relationship tracking implementation mainly includes a scheduler reverse derivation and a blood relationship tracking interface provided by a computing engine system. The scheduler reverse derivation has high feasibility and low cost, but cannot accurately track data at the field level. The blood relationship tracking interface provided by the big data computing engine (such as HIVE) can track data at the field level, but is too strong in customization and is not suitable for implementation of other database types.

[0004] Therefore, in the common database type application scenario, how to provide a data blood relationship tracking method for data at the field level has become a technical problem to be solved.

[0005] The above content is only used to assist in understanding the technical solutions of the application and does not represent the acknowledgement of the above content as prior art. SUMMARY

[0006] In order to at least overcome the problems in the related art, the application provides a data blood relationship tracking method, device and electronic equipment, and solves the technical problem of how to realize data blood relationship tracking of data at the field level in the common database type application scenario.

[0007] In order to achieve the above purpose, the application adopts the following technical solutions:

[0008] In a first aspect,

[0009] The application provides a data blood relationship tracking method, which comprises:

[0010] Obtaining metadata of a target database, and performing marking processing on the metadata according to a hierarchical dimension of the metadata to obtain marked data, wherein the hierarchical dimension comprises a library, a table and a field;

[0011] Performing parsing processing on each operation statement sent to the target database, and constructing a data flow log of the target database based on a parsing result and the marked data, wherein the data flow log carries record data representing data flow information in the target database;

[0012] Performing data bloodline tracking on target data based on the data flow log.

[0013] Optionally, the obtaining metadata of a target database, and performing marking processing on the metadata according to a hierarchical dimension of the metadata to obtain marked data comprises:

[0014] Loading a driver of the target database, and obtaining the metadata of the target database through a JDBC interface;

[0015] Performing marking processing on the metadata according to a hierarchical dimension of the metadata, and storing the obtained marked data;

[0016] The categories of the metadata comprise catalog, schema, table and column.

[0017] Optionally, the operation statement is an sql statement; and the performing parsing processing on each operation statement sent to the target database, and constructing a data flow log of the target database based on a parsing result and the marked data comprises:

[0018] Calling a general Sql parser to parse each sql statement to generate an abstract syntax tree corresponding to the statement;

[0019] Determining an execution action corresponding to the statement according to the abstract syntax tree, and performing data operation analysis on the execution action based on the marked data to obtain a data operation corresponding to the statement;

[0020] Collecting and classifying the data operations corresponding to each sql statement, and constructing the data flow log according to a time sequence of the statements based on a collection and classification result.

[0021] Optionally, the general Sql parser comprises a Druid parser and an anltr4 parser.

[0022] Optionally, the record data comprises metadata change record data; and the performing data bloodline tracking on target data based on the data flow log specifically comprises:

[0023] The metadata change record information carried by the metadata change record data is sorted in chronological order to obtain a data processing link of each hierarchical dimension data, and a data processing link set for data bloodline tracking is constructed according to the data processing link;

[0024] According to the data processing link set, the target data is queried and matched to obtain a dependency relationship of metadata and related other metadata of the target data, and a data bloodline relationship graph of the target data is obtained.

[0025] Optionally, the data bloodline tracking of the target data based on the data flow log further includes:

[0026] According to the data processing link set, the target data is queried and matched to obtain iteration information of the target data from a starting state to a current state, and a traceability map of the target data is obtained.

[0027] The second aspect,

[0028] The application provides a data bloodline tracking device, which comprises:

[0029] A marking processing module is configured to obtain metadata of a target database, and mark the metadata according to a hierarchical dimension of the metadata to obtain marked data, wherein the hierarchical dimension comprises a library, a table and a field.

[0030] An analysis and construction module is configured to analyze each database operation statement sent to the target database, and construct a data flow log of the target database based on an analysis result and the marked data, wherein the data flow log carries record data representing data flow information in the target database.

[0031] A tracking implementation module is configured to track data bloodline of target data based on the data flow log.

[0032] The third aspect,

[0033] The application provides an electronic device, which comprises:

[0034] A memory having an executable program stored thereon;

[0035] A processor configured to execute the executable program in the memory to implement the steps of the above method.

[0036] The application has at least the following beneficial effects by adopting the above technical solutions:

[0037] The data bloodline tracking method of the application comprises the following steps: obtaining metadata of a target database, and performing marking processing on the metadata according to a hierarchical dimension, to obtain marked data, wherein the hierarchical dimension comprises a library, a table and a field; performing parsing processing on each operation statement sent to the target database, constructing a data flow log of the target database based on the parsing result and the marked data, the data flow log carrying record data representing data flow information in the target database; and performing data bloodline tracking on the target data based on the data flow log. The technical solution of the application adopts specific configurations, and in the implementation process, the metadata of the target database can be marked processed at the minimum hierarchical dimension of the field, and the data flow log representing the data flow direction of the target database is constructed in combination with the parsing processing of each operation statement, and then the data bloodline tracking is performed on the target data based on the constructed data flow log. This kind of way can effectively realize the field-level data bloodline tracking, and in the implementation, specific parsing configurations can be made based on the type of the target database, so as to realize effective adaptation to different database types.

[0038] Other advantages, objects, and features of the application will be understood in view of the following detailed description, and will be apparent to those skilled in the art from the teachings of the following detailed description, or can be learned from the practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0039] The accompanying drawings are used to provide a further understanding of the technical solutions of the application or the prior art, and constitute a part of the specification. The drawings expressing the embodiments of the application are used together with the embodiments of the application to explain the technical solutions of the application, but do not constitute a limitation on the technical solutions of the application.

[0040] Figure 1 A flowchart of a data bloodline tracking method provided by an embodiment of the application is shown in the figure;

[0041] Figure 2 An implementation configuration schematic diagram of a data bloodline tracking method provided by another embodiment of the application is shown in the figure;

[0042] Figure 3 A structure schematic diagram of a data bloodline tracking device provided by an embodiment of the application is shown in the figure;

[0043] Figure 4 A structure schematic diagram of an electronic device provided by an embodiment of the application is shown in the figure. DETAILED DESCRIPTION

[0044] In order to make the purposes, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described in detail below. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work belong to the scope of protection of the present application.

[0045] As described in the background, the current bloodline tracking implementation mainly includes scheduler reverse derivation, bloodline tracking interface provided by the computing engine system, etc. The implementation of scheduler reverse derivation is feasible and has low cost, but cannot track data to the field level. The bloodline tracking interface provided by the big data computing engine (such as HIVE) can track data to the field level, but is too customized and is not suitable for implementation of other database types. In addition, for non-database data, data feature recognition or manual recognition is used to judge the blood relationship of the data. This method has high business requirements, and its implementation needs to be dynamically adjusted based on specific business changes, which has too strong limitations.

[0046] In view of this, the present application provides a data bloodline tracking method to solve the technical problem of how to implement data field-level data bloodline tracking in common database type application scenarios.

[0047] As shown in Figure 1 In an embodiment, the data bloodline tracking method provided by the present application comprises:

[0048] Step S110, obtaining the metadata of the target database, and marking the metadata according to the hierarchical dimensions of the metadata to obtain marked data, wherein the hierarchical dimensions include library, table and field;

[0049] The target database here refers to the business database in the actual application scenario. For example, a student management system uses a MySQL database in the background.

[0050] The marking processing here refers to a standardized marking and sorting of the obtained metadata information. For example, the a field of a t1 table can be marked as “ / 10.10.10.10:3306 / mysql / mysql01 / t1 / a”, a library mysql can be marked as “ / 10.10.10.10:3306 / mysql / ”, a schema (intermediate level between library and table) mysql01 can be marked as “ / 10.10.10.10:3306 / mysql / mysql01”, a table t1 can be marked as “ / 10.10.10.10:3306 / mysql / mysql01 / t1”, etc.

[0051] In this step, the marking processing facilitates subsequent blood relationship search of metadata at different levels. For example, to search the blood relationship of the t1 table, only the data with the prefix of / 10.10.10.10:3306 / mysql / mysql01 / t1 needs to be searched, which is much more efficient than overall marking. Specifically, the minimum level dimension of the marking here is a field, so field-level data blood relationship tracking can be implemented subsequently.

[0052] After step S110, as shown in FIG. 1, step S120 is performed, in which each operation statement (for example, the operation statement is an sql statement in this embodiment) sent to the target database is parsed, and a data flow log of the target database is constructed based on the parsing result and the marking data, the data flow log carrying record data representing data flow information in the target database. Figure 2

[0053] It should be noted that step S110 is a static process in actual implementation, that is, it is performed once for the business database in a specific scenario, while step S120 is a dynamic process, which needs to parse each operation statement sent to the target database, and further construct the data flow log based on the parsing result (the parsing result in a certain time period) and the marking data.

[0054] Specifically, in this embodiment, each operation statement sent to the target database is parsed, and a data flow log of the target database is constructed based on the parsing result and the marking data, including:

[0055] A general Sql parser is called to parse each sql statement, and an abstract syntax tree (AST, Abstract Syntax Tree) corresponding to the statement is generated. For example, the general Sql parser here can be a Druid parser, an anltr4 parser or other types of parsers.

[0056] The execution action corresponding to the statement is determined according to the abstract syntax tree (for example, the categories of the execution action include: drop, del, add, update, etc.), and the data operation corresponding to the corresponding statement is obtained by performing data operation analysis on the execution action based on the marking data obtained in step S110.

[0057] It should be noted that the data operation in this application refers to a combination of a data object and an execution action. For example, the categories of the data operation include: adding a field column, deleting a field column, creating a table, deleting a table, etc. For example: the statement ALTERTABLE user ADD account INT NULL COMMENT 'account' will be parsed into an operation of adding a field. ​

[0058] After processing the statements in a certain time period, the data operations corresponding to each operation statement are collected and classified, and based on the collection and classification results, a data flow log is constructed according to the time sequence of the statements. The constructed data flow log contains metadata change data, marked change data, etc.

[0059] After obtaining the data flow log, step S130 of data bloodline tracking of target data based on the data flow log can be performed.

[0060] Specifically, in this embodiment, the metadata change record information carried by the metadata change data in the data flow log is sorted in chronological order to obtain the data processing link of each hierarchical dimension data, and a data processing link set for data bloodline tracking is constructed according to the obtained data processing link.

[0061] For example, in actual database operations, the column1 field of table A is inserted into table B as column1, the column1 field of table B is then deleted, the column1 field of table C is inserted into table B as column1, and the data of column1 is also inserted. There are three steps in total, and the metadata change information of each step is recorded in the data flow log. The change information sorted in chronological order is the data processing link.

[0062] After obtaining the data processing link set, the target data is queried and matched according to the data processing link set to obtain the metadata of the target data and the dependency relationship of other related metadata, and a data bloodline relationship graph of the target data is obtained. The data bloodline relationship graph shows the association relationship between the metadata of the target database and other metadata, and is a common requirement for actual data bloodline tracking applications.

[0063] Continuing the example from the previous text, in actuality, the column1 field of the final table B and the column1 field of table C are the same, and the data sources are the same, so a simple relationship network based on the field can be obtained, B-column1-C, indicating that the field is equivalent in B and C, which is equivalent to the same field, that is, the parent node of column1 is the relationship between B and C; in this way, obviously, such a bloodline relationship graph can reflect the relationship between fields and tables, fields and fields, tables and tables, and databases and databases.

[0064] The technical scheme of the present application adopts specific configurations, and in the implementation process, the metadata of the target database is marked and processed in the minimum level dimension of the field, and the data flow log representing the data flow direction of the target database is constructed by combining the parsing processing of each operation statement, and then the data blood relationship of the target data is tracked based on the constructed data flow log. This method can effectively realize field-level data blood relationship tracking, and specific parsing configurations can be made based on the type of the target database in the implementation, thereby realizing effective adaptation to different database types.

[0065] To facilitate understanding of the technical scheme of the present application, the technical scheme of the present application is introduced and described below with another embodiment.

[0066] As shown in Figure 2 , it is a schematic illustration of the implementation configuration of the data blood relationship tracking method in this embodiment.

[0067] In this embodiment, as shown in Figure 2 , the application scenario business system includes an application front end, an application server, and a business database (the business database can use mysql, oracle, sqlserver, hive, db2, h2, etc.). Users implement specific business operation processes by accessing the application front end deployed on the application server. During the business process, the business database implements data support management for related business data.

[0068] To implement the technical scheme of the present application, an analysis server and an analysis database are added in the existing business system architecture.

[0069] Based on the specific configuration of the analysis server, the metadata of the target database (the business database in this embodiment) is first obtained and marked to obtain marked data, and then stored in the analysis database. Figure 2

[0070] Specifically, in the implementation of this embodiment, the driver of the target database is loaded, the metadata of the target database is obtained through the JDBC interface, the metadata is marked according to the metadata level dimension, and the obtained marked data is stored. The categories of metadata include: catalog, schema, table, column, etc.

[0071] After that, as shown in Figure 2 , based on the specific configuration of the analysis server, the database operation statements sent to the business database are filtered and monitored, and each operation statement sent to the target database is parsed. The data flow log of the target database is constructed and generated based on the parsing result and the marked data, and the data flow log is stored in the analysis database. The parsing and construction process has been introduced in the foregoing, and will not be described here. ​

[0072] After obtaining the data flow log, the target data can be subjected to data bloodline tracking based on the data flow log. Specifically, in this embodiment, on the basis of the foregoing embodiment, the implementation of data bloodline tracking further includes:

[0073] According to the data processing link set, the target data (specifically input by a user) is subjected to query matching, iteration information of the target data from a starting state to a current state is obtained based on the change record of the data marking, and a traceability map of the target data is obtained. It is easy to understand that, based on actual tracking business requirements, the traceability map obtained can also be stored in a database (into an analysis database).

[0074] It should be noted that the traceability map herein focuses on showing the change process of the metadata itself, and the data bloodline relationship map mentioned in the foregoing embodiment focuses on the association relationship between the metadata and other metadata.

[0075] In this embodiment, the metadata information is obtained by using a JDBC standard and general interface, and a commonly used database is adapted, instead of being limited to a specified database type, which is more suitable for popularization in actual applications. In addition, the metadata is marked in a hierarchical manner, facilitating the display of the relationship map of data bloodlines at different levels, and can be classified according to different levels. The data flow is realized by using Sql analysis, the data flow direction is mapped, and the processing process is converted into intuitive relational data presentation. In the tracking implementation, the data bloodline relationship map can be generated according to the change of the metadata and the change record of the metadata, the link formed according to the change record of the metadata and the change record of the metadata marking are aggregated to generate a traceability map of data bloodlines, and the change of the target data in any process is effectively found.

[0076] Figure 3 A structural schematic diagram of a data bloodline tracking device provided by an embodiment of the present application is shown in FIG. 3. Figure 3 As shown in FIG. 3, the data bloodline tracking device 300 includes:

[0077] A marking processing module 301 is configured to obtain metadata of a target database, and mark the metadata according to a hierarchical dimension of the metadata to obtain marked data, wherein the dimension includes a library, a table, and a field.

[0078] An analysis and construction module 302 is configured to analyze and process each database operation statement sent to the target database, and construct and generate a data flow log of the target database based on the analysis result and the marked data. The data flow log carries record data representing data flow direction information in the target database.

[0079] A tracking implementation module 303 is configured to track data bloodlines of target data based on the data flow log.

[0080] As to the data bloodline tracking device 300 in the above-mentioned related embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments related to the method, and thus will not be described in detail here.

[0081] Figure 4 A structural schematic diagram of an electronic device provided for an embodiment of the present application is shown in FIG. 4. As shown in FIG. 4, the electronic device 400 includes: Figure 4

[0082] a memory 401 having an executable program stored thereon;

[0083] a processor 402 configured to execute the executable program in the memory 401 to implement the steps of the above-mentioned method.

[0084] As to the electronic device 400 in the above-mentioned embodiments, the specific manner in which the processor 402 of the electronic device 400 executes the program in the memory 401 has been described in detail in the embodiments related to the method, and thus will not be described in detail here.

[0085] The above merely provides the preferred embodiments of the present application, but the protection scope of the present application is not limited thereto, and any changes or substitutions easily conceived by those skilled in the art within the technical scope disclosed by the present application should be encompassed within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.​

Claims

1. A data lineage tracking method, characterized in that, The method comprises the following steps: acquiring metadata of a target database, and performing marking processing on the metadata according to a hierarchical dimension of the metadata, to obtain marked data, wherein the hierarchical dimension comprises a library, a table and a field; performing parsing processing on each operation statement sent to the target database, and constructing and generating a data flow log of the target database based on a parsing result and the marked data, wherein the data flow log carries record data representing data flow information in the target database; wherein the operation statement is a sql statement; the parsing processing on each operation statement sent to the target database, and the constructing and generating of the data flow log of the target database based on the parsing result and the marked data, comprise the following steps: calling a general Sql parser to parse each sql statement, to generate an abstract syntax tree corresponding to the statement; determining an execution action corresponding to the statement according to the abstract syntax tree, and performing data operation analysis on the execution action based on the marked data, to obtain a data operation corresponding to the corresponding statement; collecting and classifying the data operations corresponding to each sql statement, and constructing the data flow log according to the time sequence of the statements based on the collection and classification result; wherein the record data comprises metadata change record data; performing data bloodline tracking on target data based on the data flow log, specifically comprising the following steps: sorting metadata change record information carried by the metadata change record data according to time sequence, to obtain a data processing link of each hierarchical dimension data, and constructing a data processing link set for data bloodline tracking according to the data processing link; querying and matching the target data according to the data processing link set, to obtain a dependency relationship between metadata of the target data and other related metadata, and obtaining a data bloodline relationship graph of the target data.

2. The method of claim 1, wherein, The method comprises the following steps: loading a driver of the target database, and acquiring the metadata of the target database through a JDBC interface; performing marking processing on the metadata according to the hierarchical dimension of the metadata, and storing the obtained marked data; wherein the categories of the metadata comprise catalog, schema, table and column.

3. The data lineage tracking method of claim 1, wherein, The general Sql parser comprises a Druid parser and an anltr4 parser.

4. The data lineage tracking method of claim 1, wherein, The data bloodline tracking on the target data based on the data flow log further comprises the following steps: querying and matching the target data according to the data processing link set, to obtain iteration information of the target data from a starting state to a current state, and obtaining a traceability map of the target data.

5. A data lineage tracking apparatus, comprising: The method comprises the following steps: a marking processing module, configured to acquire metadata of a target database, and perform marking processing on the metadata according to a hierarchical dimension of the metadata, to obtain marked data, wherein the hierarchical dimension comprises a library, a table and a field; The parsing and constructing module is configured to parse each database operation statement sent to the target database, and construct a data flow log of the target database based on a parsing result and the marking data, the data flow log carrying record data representing data flow information in the target database; the operation statement is a sql statement; the parsing and constructing module is specifically configured to call a general sql parser to parse each sql statement to generate an abstract syntax tree corresponding to the statement; determine an execution action corresponding to the statement according to the abstract syntax tree, and perform data operation parsing on the execution action based on the marking data to obtain a data operation corresponding to the statement; collect and classify the data operations corresponding to each sql statement, and construct the data flow log according to a time sequence of the statements based on a collection and classification result; The tracking implementation module is configured to perform data bloodline tracking on target data based on the data flow log; the record data includes metadata change record data; the tracking implementation module is specifically configured to sort metadata change record information carried by the metadata change record data according to time to obtain a data processing link of each hierarchical dimension data, and construct a data processing link set for data bloodline tracking according to the data processing link; According to the data processing link set, the target data is queried and matched to obtain a dependency relationship between metadata of the target data and related other metadata, and a data bloodline relationship graph of the target data is obtained.

6. An electronic device, comprising: Comprise: A memory having an executable program stored thereon; A processor configured to execute the executable program in the memory to implement the steps of the method of any one of claims 1-4.

Images