Method and device for realizing data consanguinity

Abstract

The invention relates to the field of big data, and particularly provides a method and device for realizing data consanguinity, and the method comprises the following steps: S1, building a batch flow task; s2, acquiring data consanguinity information of an uplink service system; s3, acquiring integration task details from the data lake to the standard layer; and S4, acquiring data consanguinity information of the downlink service system. Compared with the prior art, the method has the advantage that the data generation and extinction process can be better managed.

Description

Technical Field

[0001] The present invention relates to the field of big data, and specifically provides a method and device for realizing data lineage. Background Art

[0002] In today's era of information explosion, enterprises are faced with the challenges of fast data growth and complex and diverse data sources. Data lineage can trace the source, operation and flow of data, helping enterprises better grasp the value and risks of data.

[0003] How to build a comprehensive and accurate full-link data lineage view, identify the upstream and downstream users of the data, and ensure communication and information synchronization are issues that need to be urgently addressed in this technical field. Summary of the invention

[0004] The present invention aims at the above-mentioned deficiencies of the prior art and provides a method for realizing data lineage with strong practicability.

[0005] A further technical task of the present invention is to provide a device that is rationally designed, safe and applicable to realize data lineage.

[0006] The technical solution adopted by the present invention to solve its technical problem is:

[0007] A method for realizing data lineage has the following steps:

[0008] S1. Establish batch flow tasks;

[0009] S2. Obtaining data lineage information of the upstream business system;

[0010] S3, get the details of the integration task from the data lake to the standard layer;

[0011] S4. Obtain data lineage information of the downstream business system.

[0012] Furthermore, in step S1, the upstream system is the data flow from the business source to the data lake, the central system is the data flow from the data lake to the standard layer, and the downstream system is the data flow from the standard layer to the subject layer, indicator layer, and the data flow for providing API services externally;

[0013] The flow of data is based on the execution of batch flow tasks, which requires recording task information and field mapping;

[0014] Before creating an integration task, save the business source library ID, business source library name, task name, task status, task type, source table name, target table name, field mapping, and task details information to the T_DRM_DI_TASK system table.

[0015] Furthermore, the properties of the source table and source field and the corresponding properties of the target table and target field are constructed. The task ID corresponds to the WID in the T_DRM_DI_TASK table and is saved in the T_DRM_DI_TASKDETAIL system table. The field correspondence is saved in the T_DRM_DI_TASKDETAILMAPPING system table.

[0016] After the field mapping is completed, the execution script of the batch flow task is built and sent to the batch flow platform to complete the data flow. If the task execution fails, the built task information will be deleted.

[0017] Further, in step S2, when all data of the upstream business system is obtained, the task ID, source library name, source table name, source table field attributes, target library name, target table name and target table attributes are obtained from the table T_DRM_DI_TASKDETAILMAPPING;

[0018] Obtain wid, source table name, target table name, and task execution status from table T_DRM_DI_TASK; perform joint query and screening on the data queried from the two tables, including matching the corresponding source table ID and target table ID in the two tables, and screening the task status;

[0019] Then group by source library name, task attribute, target library name, and task ID, and calculate some count aggregation results;

[0020] The database name of the data lake is defined as 0, so when querying the data flow of the upstream business system, just filter out the target database name as 0.

[0021] Furthermore, in step S3, a joint query is performed based on the table name according to the T_DRM_DI_TASK task table and the T_DRM_DI_TASKDETAILMAPPING field mapping table, and the result is filtered out according to the filter conditions of the source library name being 0 and the destination library name being 1, and the integration tasks from the data lake to the standard layer are displayed in a list, and then the detailed information of each task is viewed.

[0022] Further, in step S4, when all data of the downstream business system is obtained, the task ID, source library name, source table name, source table field attributes, target library name, target table name and target table attributes are obtained from the table T_DRM_DI_TASKDETAILMAPPING;

[0023] Get wid, source table name, target table name, and task execution status from table T_DRM_DI_TASK;

[0024] Perform joint query and screening on the data queried from the two tables, including matching the corresponding source table ID and target table ID in the two tables and the screening task status.

[0025] Furthermore, we group the data by source database name, task attribute, target database name, and task ID, and calculate some count aggregation results. We define the database name of the standard layer as 1. When querying the data flow of the downstream business system, we can filter out the source database name as 1.

[0026] Then, the query data of the upstream system is associated with the query through the table name to obtain the entire blood relationship from the business source to the subject library.

[0027] A device for implementing data lineage, comprising: at least one memory and at least one processor;

[0028] The at least one memory is used to store a machine-readable program;

[0029] The at least one processor is used to call the machine-readable program to execute a method for implementing data lineage.

[0030] Compared with the prior art, the method and device for realizing data lineage of the present invention have the following outstanding beneficial effects:

[0031] According to the batch flow task, the present invention obtains the lineage relationship through multi-table joint association query according to the task ID and the source library table ID when the task is running. The data lineage is displayed in real time following the establishment of the batch flow task. The process of data generation and disappearance is better managed. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0033] Attached Figure 1 It is a flowchart of a method for implementing data lineage. DETAILED DESCRIPTION

[0034] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] A best embodiment is given below:

[0036] like Figure 1 As shown, a method for realizing data lineage in this embodiment has the following steps:

[0037] S1. Establish batch flow tasks;

[0038] Uplink system: data flow from business sources to the data lake;

[0039] Central system: data flow from the data lake to the standard layer;

[0040] Downstream system: data flow from the standard layer to the subject layer, indicator layer, and external API services.

[0041] The flow of data is based on the execution of batch flow tasks, which requires recording task information and field mapping to support the display of data lineage.

[0042] When saving task information, before creating an integration task, save the business source library ID, business source library name, task name, task status, task type, source table name, target table name, field mapping, and task details information to the T_DRM_DI_TASK system table.

[0043] When accessing task field mapping, construct the properties of the source table and source field and the corresponding target table and target field properties. The task ID corresponds to the WID in the T_DRM_DI_TASK table and is saved in the T_DRM_DI_TASKDETAIL system table. The field correspondence is saved in the T_DRM_DI_TASKDETAILMAPPING system table.

[0044] After the field mapping is completed, the execution script of the batch flow task is built and sent to the batch flow platform to complete the data flow. If the task execution fails, the built task information will be deleted.

[0045] S2. Obtaining data lineage information of the upstream business system;

[0046] When obtaining all data from the upstream business system, obtain the task ID, source library name, source table name, source table field attributes, target library name, target table name, and target table attributes from the table T_DRM_DI_TASKDETAILMAPPING;

[0047] Get wid, source table name, target table name, and task execution status from table T_DRM_DI_TASK;

[0048] Perform joint query and screening on the data queried from the two tables, including matching the corresponding source table ID and target table ID in the two tables and the screening task status;

[0049] Then group by source library name, task attribute, target library name, and task ID, and calculate some count aggregation results.

[0050] We define the database name of the data lake as 0, so when querying the data flow of the upstream business system, you can filter out the target database name as 0.

[0051] S3, get the details of the integration task from the data lake to the standard layer;

[0052] Perform a joint query based on the table name according to the T_DRM_DI_TASK task table and the T_DRM_DI_TASKDETAILMAPPING field mapping table. The results obtained filter out the integration tasks from the data lake to the standard layer according to the filter conditions of the source library name being 0 and the destination library name being 1. You can display them in a list and then view the detailed information of each task.

[0053] S4. Obtain data lineage information of the downlink business system;

[0054] Get all the data of the downstream business system: get the task ID, source library name, source table name, source table field attributes, target library name, target table name and target table attributes from the table T_DRM_DI_TASKDETAILMAPPING; get wid, source table name, target table name and task execution status from the table T_DRM_DI_TASK; perform joint query and filtering on the data queried from the two tables, including matching the corresponding source table ID and target table ID in the two tables and filtering the task status; then group by source library name, task attributes, target library name and task ID, and calculate some count aggregation results.

[0055] We define the library name of the standard layer as 1, so when querying the data flow of the downstream business system, you can filter out the source library name as 1. Then, you can associate the query data of the upstream system with the table name to obtain the entire blood relationship from the business source to the subject library.

[0056] Based on the above method, a device for implementing data lineage in this embodiment includes: at least one memory and at least one processor;

[0057] The at least one memory is used to store a machine-readable program;

[0058] The at least one processor is used to call the machine-readable program to execute a method for implementing data lineage.

[0059] The above-mentioned specific implementations are only specific cases of the present invention. The patent protection scope of the present invention includes but is not limited to the above-mentioned specific implementations. Any technical solutions that comply with the claims of the present invention and any appropriate changes or substitutions made by ordinary technicians in the relevant technical field shall fall within the patent protection scope of the present invention.

[0060] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for realizing data lineage, characterized in that: The steps are as follows: S1. Establish batch flow tasks; S2. Obtaining data lineage information of the upstream business system; S3, get the details of the integration task from the data lake to the standard layer; S4. Obtain data lineage information of the downstream business system.

2. A method for realizing data lineage according to claim 1, characterized in that: In step S1, the upstream system is the data flow from the business source to the data lake, the central system is the data flow from the data lake to the standard layer, and the downstream system is the data flow from the standard layer to the subject layer, indicator layer, and the data flow for providing API services to the outside world; The flow of data is based on the execution of batch flow tasks, which requires recording task information and field mapping; Before creating an integration task, save the business source library ID, business source library name, task name, task status, task type, source table name, target table name, field mapping, and task details information to the T_DRM_DI_TASK system table.

3. A method for realizing data lineage according to claim 2, characterized in that: Construct the properties of the source table and source field and the corresponding properties of the target table and target field. The task ID corresponds to the WID in the T_DRM_DI_TASK table and is saved in the T_DRM_DI_TASKDETAIL system table. The field correspondence is saved in the T_DRM_DI_TASKDETAILMAPPING system table. After the field mapping is completed, the execution script of the batch flow task is built and sent to the batch flow platform to complete the data flow. If the task execution fails, the built task information will be deleted.

4. A method for realizing data lineage according to claim 3, characterized in that: In step S2, when all data of the upstream business system is obtained, the task ID, source library name, source table name, source table field attributes, target library name, target table name and target table attributes are obtained from the table T_DRM_DI_TASKDETAILMAPPING; Obtain wid, source table name, target table name, and task execution status from table T_DRM_DI_TASK; perform joint query and screening on the data queried from the two tables, including matching the corresponding source table ID and target table ID in the two tables, and screening the task status; Then group by source library name, task attribute, target library name, and task ID, and calculate some count aggregation results; The database name of the data lake is defined as 0, so when querying the data flow of the upstream business system, just filter out the target database name as 0.

5. A method for realizing data lineage according to claim 4, characterized in that: In step S3, a joint query is performed based on the table name according to the T_DRM_DI_TASK task table and the T_DRM_DI_TASKDETAILMAPPING field mapping table. The result is filtered out according to the filter conditions of the source library name being 0 and the destination library name being 1, and the integration tasks from the data lake to the standard layer are displayed in a list, and then the detailed information of each task is viewed.

6. A method for realizing data lineage according to claim 5, characterized in that: In step S4, when all data of the downstream business system is obtained, the task ID, source library name, source table name, source table field attributes, target library name, target table name and target table attributes are obtained from the table T_DRM_DI_TASKDETAILMAPPING; Get wid, source table name, target table name, and task execution status from table T_DRM_DI_TASK; Perform joint query and screening on the data queried from the two tables, including matching the corresponding source table ID and target table ID in the two tables and the screening task status.

7. A method for realizing data lineage according to claim 6, characterized in that: Group by source database name, task attribute, target database name, and task ID, and calculate some count aggregation results. Define the database name of the standard layer as 1. When querying the data flow of the downstream business system, filter out the source database name as 1. Then, the query data of the upstream system is associated with the query through the table name to obtain the entire blood relationship from the business source to the subject library.

8. A device for realizing data lineage, characterized in that: include: at least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is configured to call the machine-readable program to execute the method according to any one of claims 1 to 7.

Images