Data Task Analysis Method, System and Medium Based on Data Lineage

Abstract

The present invention provides a data task analysis method, system and medium based on data lineage, including: when initializing the execution of a data task, the task analysis module analyzes the data lineage relationship in the data task, establishes a data task lineage mapping relationship and saves it in the data task lineage mapping library; the mapping relationship display module queries the data task, displays the associated databases, tables, and fields in the data task graph, and marks the mapping situation between the task and the lineage; when the metadata information changes and the data task is executed, the task analysis module compares the data task lineage mapping relationship of the current task and marks the changes in the task lineage for the changed content; then synchronizes the changed data task information and marks the task lineage changes; the mapping relationship display module displays the data task information and highlights the affected jobs in the same data task flow graph. The present invention can solve the problem that manual parsing is required to analyze the impact when the data lineage of a data task changes.

Description

Technical Field

[0001] The present invention relates to the technical field of computer data processing, and specifically, to a data task analysis method, system, and medium based on data lineage. Background Art

[0002] Processes such as data generation, transfer, and processing cause a link relationship to be generated between data. This relationship describes the dependency relationship of data, which we call the data lineage.

[0003] The lineage provides a data source tracing function. Relying on various syntax tree parsers in the Internet, we can implement the lineage parsing function for various databases by parsing the syntax tree of database operation statements. Applying this function to the data governance platform, through the generation and transfer of each data table carried out by the governance platform, we can generate lineage for it.

[0004] The data governance process covers data analysis from the front-end transaction processing system, the back-end business database to the terminal, forming a closed-loop negative feedback system from the source to the terminal and then back to the source. The purpose of data governance is to supervise the acquisition, processing, and use of data.

[0005] The invention patent with the publication number CN114116856A discloses a field-level lineage analysis method based on the entire data governance link. By extracting the field-level lineage relationship for data uploading to the cloud, cloud computing, and data downloading from the cloud, a full-linkage lineage topology graph is obtained, which is convenient for users to understand the overall processing link. This method starts from the innermost layer of the parsed syntax tree, reversely simulates the SQL execution process, gradually calculates the truly corresponding fields, can extract the lineage relationship between fields and functions, and at the same time has higher flexibility and can handle complex business scenarios. This patent can obtain the full-linkage data lineage relationship, but it cannot associate with the actual production data tasks, cannot perceive the data tasks affected according to the data lineage relationship, and still requires manual parsing of the impact on the data tasks when the data lineage changes. Summary of the Invention

[0006] Aiming at the defects in the prior art, the present invention provides a data task analysis method, system, and medium based on data lineage.

[0007] According to the data task analysis method, system, and medium based on data lineage provided by the present invention, the solutions are as follows:

[0008] In the first aspect, a data task analysis method based on data lineage is provided, and the method includes:

[0009] Step S1: When initializing the execution of a data task, the task analysis module analyzes the data lineage relationship in the data task, and simultaneously establishes a data task lineage mapping relationship and saves it in the data task lineage mapping library;

[0010] Step S2: The mapping relationship display module queries the data task from the data task lineage mapping library, and displays all associated databases, tables, and fields in the data task graph through the data task lineage mapping relationship, and marks the mapping situation between the task and the lineage according to the data task lineage mapping relationship;

[0011] Step S3: When the metadata information changes and the data task is executed, the task analysis module compares the data task lineage mapping relationship of the current task in the data task lineage mapping library, and marks the task lineage change for the changed content; then synchronously changes the relevant data task information according to the data lineage relationship and the data task lineage mapping relationship, and marks the task lineage change;

[0012] Step S4: The mapping relationship display module classifies and displays the data task information according to the task lineage change mark, and highlights the affected jobs in the same data task flow graph.

[0013] Preferably, the step S1 includes:

[0014] Input: Data task information Ti;

[0015] Output: Lineage mapping relationship Ri of data task Ti, task lineage change mark of data task Ti;

[0016] Step S1.1: The task analysis module reads all data operation sets Tq included in the task through the data task information Ti;

[0017] Step S1.2: For each data operation in Tq, obtain the data lineage relationship, mark the source end of all data operations as Ts, and the target end of the data operation as Td, to obtain the set of libraries Tsd, tables Tst, and fields Tsc of the data operation source Ts = {Tsd, Tst, Tsc} and the set of libraries Tdd, tables Tdt, and fields Tdc of the data target end Ts = {Tdd, Tdt, Tdc};

[0018] Step S1.3: Obtain the data task number Td through the data task information Ti, associate Ts and Td to obtain the temporary mapping relationship Rt = {Td, Ts, Td}; expand the defined identification constants S = {D, T, C} for the database D, table T, and field C, and establish the lineage mapping relationship Ri of the data task Ti = {S, Ti, Td, Ts} in combination with the data task number Td;

[0019] Step S1.4: The task analysis module saves the blood relationship mapping Ri to the data task blood relationship mapping library.

[0020] Preferably, the step S3 includes:

[0021] Input: Data task information Ti;

[0022] Output:

[0023] The blood relationship mapping Ri of the data task Ti;

[0024] The task blood relationship change mark set Rc of the data task Ti;

[0025] Step S3.1: The task analysis module obtains the blood relationship mapping Rn = {S, Ti, Tnd, Tns} for establishing the data task Ti through the data task information Ti, and obtains the existing data task blood relationship mapping Ri = {S, Ti, Tid, Tis} from the data task blood relationship mapping library;

[0026] Step S3.2: Traverse each source end and destination end in Rn, and add them to the task blood relationship change mark set Rc according to the judgment;

[0027] Step S3.3: Save the obtained task blood relationship change mark set Rc to the data task blood relationship mapping library.

[0028] Preferably, the step S3.2 specifically includes:

[0029] Step S3.2.1: Traverse each source end and destination end in Rn. If Tnd = Tid, then judge whether Tns exists in Ri. If not, define the destination end change CS and create a new task blood relationship change mark item Ri = {S, Ti, Tnd, Tns, CS}, and add it to the task blood relationship change mark set Rc. If it exists, continue;

[0030] Step S3.2.2: If Tns = Tis, then judge whether Tnd exists in Ri. If not, define the source end change CD and create a new task blood relationship change mark item Ri = {S, Ti, Tnd, Tns, CD}, and add it to the task blood relationship change mark set Rc. If it exists, continue;

[0031] Step S3.2.3: If Tnd!= Tid and Tns!= Tis, then define the new data task blood relationship mapping CN, create a new task blood relationship change mark item Ri = {S, Ti, Tnd, Tns, CN}, and add it to the task blood relationship change mark set Rc.

[0032] In a second aspect, a data task analysis system based on data blood relationship is provided. The system includes:

[0033] Module M1: When initializing and executing a data task, the task analysis module analyzes the data lineage relationship in the data task, and simultaneously establishes a data task lineage mapping relationship and saves it in the data task lineage mapping library;

[0034] Module M2: The mapping relationship display module queries the data task from the data task lineage mapping library, and displays all associated databases, tables, and fields in the data task graph through the data task lineage mapping relationship, and marks the mapping situation between the task and the lineage according to the data task lineage mapping relationship;

[0035] Module M3: When the metadata information changes and a data task is executed, the task analysis module compares the data task lineage mapping relationship of the current task in the data task lineage mapping library, and marks the changes in the task lineage for the changed content; then synchronizes and changes the relevant data task information according to the data lineage relationship and the data task lineage mapping relationship, and marks the task lineage change;

[0036] Module M4: The mapping relationship display module classifies and displays the data task information according to the task lineage change mark, and highlights the affected jobs in the same data task flow graph.

[0037] Preferably, the module M1 includes:

[0038] Input: Data task information Ti;

[0039] Output: Lineage mapping relationship Ri of data task Ti, task lineage change mark of data task Ti;

[0040] Module M1.1: The task analysis module reads all data operation sets Tq included in the task through the data task information Ti;

[0041] Module M1.2: For each data operation in Tq, obtain the data lineage relationship, mark the source end of all data operations as Ts, and mark the target end of the data operation as Td, to obtain the set Ts = {Tsd, Tst, Tsc} of the library Tsd, table Tst, and field Tsc of the data operation source and the set Ts = {Tdd, Tdt, Tdc} of the library Tdd, table Tdt, and field Tdc of the data target end;

[0042] Module M1.3: Obtain the data task number Td through the data task information Ti, associate Ts and Td to obtain a temporary mapping relationship Rt = {Td, Ts, Td}; expand the defined identification constants S = {D, T, C} for the database D, table T, and field C and establish the lineage mapping relationship Ri = {S, Ti, Td, Ts} of the data task Ti in combination with the data task number Td;

[0043] Module M1.4: The task analysis module saves the blood relationship mapping Ri to the data task blood relationship mapping library.

[0044] Preferably, the module M3 includes:

[0045] Input: Data task information Ti;

[0046] Output:

[0047] The blood relationship mapping Ri of the data task Ti;

[0048] The set Rc of task blood relationship change marks for the data task Ti;

[0049] Module M3.1: The task analysis module obtains the blood relationship mapping Rn = {S, Ti, Tnd, Tns} of the data task Ti through the data task information Ti, and obtains the existing blood relationship mapping Ri = {S, Ti, Tid, Tis} from the data task blood relationship mapping library;

[0050] Module M3.2: Traverse each source end and destination end in Rn, and add them to the set Rc of task blood relationship change marks according to the judgment;

[0051] Module M3.3: Save the obtained set Rc of task blood relationship change marks to the data task blood relationship mapping library.

[0052] Preferably, the module M3.2 specifically includes:

[0053] Module M3.2.1: Traverse each source end and destination end in Rn. If Tnd = Tid, then judge whether Tns exists in Ri. If not, define the destination end change CS and create a new task blood relationship change mark item Ri = {S, Ti, Tnd, Tns, CS}, and add it to the set Rc of task blood relationship change marks. If it exists, continue;

[0054] Module M3.2.2: If Tns = Tis, then judge whether Tnd exists in Ri. If not, define the source end change CD and create a new task blood relationship change mark item Ri = {S, Ti, Tnd, Tns, CD}, and add it to the set Rc of task blood relationship change marks. If it exists, continue;

[0055] Module M3.2.3: If Tnd!= Tid and Tns!= Tis, then define the new data task blood relationship mapping CN, create a new task blood relationship change mark item Ri = {S, Ti, Tnd, Tns, CN}, and add it to the set Rc of task blood relationship change marks.

[0056] In a third aspect, a computer-readable storage medium storing a computer program is provided, wherein the computer program, when executed by a processor, implements the steps in the method.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] 1. By analyzing the mapping relationship between data tasks and data lineage in the decoupled task analysis module, the present invention minimizes the intrusion of data task analysis into data task execution, and obtains the mapping between data lineage and data tasks;

[0059] 2. By adopting the method of graph computing self-check, the present invention can dynamically perceive the affected data tasks when the data lineage changes, thus solving the problem that data tasks need to be manually parsed for impacts when the data lineage changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] By reading the following detailed description of non-limiting embodiments with reference to the accompanying drawings, other features, objects, and advantages of the present invention will become more apparent:

[0061] Figure 1 is a schematic diagram of the whole of the present invention;

[0062] Figure 2 is a specific flowchart of step S1 of the present invention;

[0063] Figure 3 is a specific flowchart of step S3 of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0064] The present invention will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but do not limit the present invention in any form. It should be noted that those of ordinary skill in the art can make several changes and improvements without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0065] An embodiment of the present invention provides a data task analysis method based on data lineage. Referring to Figure 1 and Figure 2 as shown, the method specifically includes:

[0066] Step S1: When initializing the execution of a data task, the task analysis module analyzes the data lineage relationship in the data task, and at the same time establishes a data task lineage mapping relationship and saves it in the data task lineage mapping library.

[0067] Specifically, referring to Figure 2 as shown, step S1 includes:

[0068] Input: Data task information Ti;

[0069] Output: Lineage mapping relationship Ri of data task Ti, task lineage change mark of data task Ti;

[0070] Step S1.1: The task analysis module reads all data operation sets Tq included in the task through the data task information Ti;

[0071] Step S1.2: For each data operation in Tq, obtain the data lineage relationship, mark the source end Ts of all data operations, and the target end Td of the data operation, to obtain the library Tsd, table Tst, and field Tsc sets Ts = {Tsd, Tst, Tsc} of the data operation source and the library Tdd, table Tdt, and field Tdc sets Ts = {Tdd, Tdt, Tdc} of the data target end;

[0072] Step S1.3: Obtain the data task number Td through the data task information Ti, associate Ts and Td to obtain the temporary mapping relationship Rt = {Td, Ts, Td}; expand the defined identification constants S = {D, T, C} for the database D, table T, and field C and combine them with the data task number Td to establish the lineage mapping relationship Ri = {S, Ti, Td, Ts} of the data task Ti.

[0073] Step S1.4: The task analysis module saves the lineage mapping relationship Ri to the data task lineage mapping library.

[0074] Step S2: The mapping relationship display module queries the data task from the data task lineage mapping library, and can display all associated databases, tables, and fields in the data task diagram through the data task lineage mapping relationship, and mark the mapping situation between the task and the lineage according to the data task lineage mapping relationship.

[0075] Step S3: When the metadata information changes and the data task is executed, the task analysis module compares the data task lineage mapping relationship of the current task in the data task lineage mapping library, and marks the changed content with the task lineage change mark; then synchronously changes the relevant data task information according to the data lineage relationship and the data task lineage mapping relationship, and marks the task lineage change.

[0076] Specifically, referring to Figure 3 as shown, Step S3 includes:

[0077] Input: Data task information Ti;

[0078] Output:

[0079] Lineage mapping relationship Ri of data task Ti;

[0080] Task lineage change mark set Rc of data task Ti;

[0081] Step S3.1: The task analysis module obtains the lineage mapping relationship Rn = {S, Ti, Tnd, Tns} for establishing the data task Ti through the data task information Ti, and obtains the existing data task lineage mapping relationship Ri = {S, Ti, Tid, Tis} from the data task lineage mapping library;

[0082] Step S3.2: Traverse each source end and destination end in Rn, and add them to the task lineage change mark set Rc according to the judgment;

[0083] This step S3.2 specifically includes:

[0084] Step S3.2.1: Traverse each source end and destination end in Rn. If Tnd = Tid, then judge whether Tns exists in Ri. If not, define the destination end change CS and create a new task lineage change mark item Ri = {S, Ti, Tnd, Tns, CS}, and add it to the task lineage change mark set Rc. If it exists, continue.

[0085] Step S3.2.2: If Tns = Tis, then judge whether Tnd exists in Ri. If not, define the source end change CD and create a new task lineage change mark item Ri = {S, Ti, Tnd, Tns, CD}, and add it to the task lineage change mark set Rc. If it exists, continue.

[0086] Step S3.2.3: If Tnd!= Tid and Tns!= Tis, then define the new data task lineage mapping relationship CN, create a new task lineage change mark item Ri = {S, Ti, Tnd, Tns, CN}, and add it to the task lineage change mark set Rc.

[0087] Step S3.3: Save the obtained task lineage change mark set Rc to the data task lineage mapping library.

[0088] Step S4: The mapping relationship display module classifies and displays the data task information according to the task lineage change marks, and highlights the affected jobs in the same data task flow diagram.

[0089] The embodiment of the present invention provides a data task analysis method, system and medium based on data lineage, which establishes the relationship between data tasks and data lineage, and the mapping relationship between data lineage and data tasks can be queried bidirectionally. The data lineage relationship between data tasks is displayed hierarchically in the data task, and the data lineage relationship at the library, table, and field levels between jobs is displayed, so as to further clarify the data task lineage dependence through the dependence between tasks. When the data lineage relationship changes, the affected tasks can be clarified according to the data task lineage dependence relationship, and all data tasks that need to be changed can be clarified.

[0090] Those skilled in the art know that, in addition to implementing the system and its various devices, modules, and units provided by the present invention in the form of pure computer-readable program code, the method steps can be logically programmed to enable the system and its various devices, modules, and units provided by the present invention to be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers, etc. to achieve the same functions. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a kind of hardware component, and the devices, modules, and units included therein for implementing various functions can also be regarded as the structures within the hardware component; the devices, modules, and units for implementing various functions can also be regarded as either software modules for implementing the method or structures within the hardware component.

[0091] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Without conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A data task analysis method based on data lineage, characterized in that, Including: Step S1: When initializing the execution of a data task, the task analysis module analyzes the data lineage relationship in the data task, and at the same time establishes a data task lineage mapping relationship and saves it in the data task lineage mapping library; Step S2: The mapping relationship display module queries the data task from the data task lineage mapping library, and displays all associated databases, tables, and fields in the data task graph through the data task lineage mapping relationship, and marks the mapping situation between the task and the lineage according to the data task lineage mapping relationship; Step S3: When the metadata information changes and the data task is executed, the task analysis module compares the data task lineage mapping relationship of the current task in the data task lineage mapping library, and marks the task lineage change for the changed content; then synchronizes and changes the relevant data task information according to the data lineage relationship and the data task lineage mapping relationship, and marks the task lineage change; Step S4: The mapping relationship display module classifies and displays the data task information according to the task lineage change mark, and highlights the affected jobs in the same data task flow graph.

2. The data task analysis method based on data lineage according to claim 1, wherein The said step S1 includes: Input: Data task information Ti; Output: The lineage mapping relationship Ri of data task Ti, the task lineage change mark of data task Ti; Step S1.1: The task analysis module reads all data operation sets Tq included in the task through the data task information Ti; Step S1.2: For each data operation in Tq, obtain the data lineage relationship, mark the source end of all data operations as Ts, and mark the target end of the data operation as Td, to obtain the library Tsd, table Tst, and field Tsc sets Ts = {Tsd, Tst, Tsc} of the data operation source and the library Tdd, table Tdt, and field Tdc sets Ts = {Tdd, Tdt, Tdc} of the data target end; Step S1.3: Obtain the data task number Td through the data task information Ti, associate Ts and Td to get the temporary mapping relationship Rt = {Td, Ts, Td}; expand the defined identification constants S = {D, T, C} for the database D, table T, and field C, and combine them with the data task number Td to establish the lineage mapping relationship Ri = {S, Ti, Td, Ts} of data task Ti; Step S1.4: The task analysis module saves the lineage mapping relationship Ri into the data task lineage mapping library.

3. The data task analysis method based on data lineage according to claim 1, wherein The said step S3 includes: Input: Data task information Ti; Output: The lineage mapping relationship Ri of data task Ti; The set Rc of task lineage change marks of data task Ti; Step S3.1: The task analysis module obtains the lineage mapping relationship Rn = {S, Ti, Tnd, Tns} of data task Ti through the data task information Ti, and obtains the existing data task lineage mapping relationship Ri = {S, Ti, Tid, Tis} from the data task lineage mapping library; Step S3.2: Traverse each source end and target end in Rn, and add them to the task lineage change mark set Rc according to the judgment; Step S3.3: Save the obtained task lineage change mark set Rc into the data task lineage mapping library.

4. The data task analysis method based on data lineage according to claim 3, wherein The specific steps of step S3.2 are as follows: Step S3.2.1: Traverse each source end and destination end in Rn. If Tnd = Tid, then determine whether Tns exists in Ri. If it does not exist, define the destination end change CS and create a new task blood relationship change marking item Ri = {S, Ti, Tnd, Tns, CS}, and add it to the task blood relationship change marking set Rc. If it exists, continue; Step S3.2.2: If Tns = Tis, then determine whether Tnd exists in Ri. If it does not exist, define the source end change CD and create a new task blood relationship change marking item Ri = {S, Ti, Tnd, Tns, CD}, and add it to the task blood relationship change marking set Rc. If it exists, continue; Step S3.2.3: If Tnd!= Tid and Tns!= Tis, then define the new data task blood relationship mapping CN, create a new task blood relationship change marking item Ri = {S, Ti, Tnd, Tns, CN}, and add it to the task blood relationship change marking set Rc.

5. A data task analysis system based on data lineage, characterized in that, It includes: Module M1: When initializing and executing a data task, the task analysis module analyzes the data blood relationship in the data task, and at the same time establishes a data task blood relationship mapping and saves it in the data task blood relationship mapping library; Module M2: The mapping relationship display module queries the data task from the data task blood relationship mapping library, displays all associated databases, tables, and fields in the data task diagram through the data task blood relationship mapping, and marks the mapping situation between the task and the blood relationship according to the data task blood relationship mapping; Module M3: When the metadata information changes and the data task is executed, the task analysis module compares the data task blood relationship mapping of the current task in the data task blood relationship mapping library, and marks the changed content with a task blood relationship change; then synchronously changes the relevant data task information according to the data blood relationship and the data task blood relationship mapping, and marks the task blood relationship change; Module M4: The mapping relationship display module classifies and displays the data task information according to the task blood relationship change marking, and highlights the affected jobs in the same data task flow diagram.

6. The data task analysis system based on data lineage according to claim 5, wherein The said module M1 includes: Input: Data task information Ti; Output: The blood relationship mapping Ri of data task Ti, the task blood relationship change marking of data task Ti; Module M1.1: The task analysis module reads all data operation sets Tq included in the task through the data task information Ti; Module M1.2: For each data operation in Tq, obtain the data blood relationship, mark the source end of all data operations as Ts, and the target end of the data operation as Td, to obtain the library Tsd, table Tst, and field Tsc sets Ts = {Tsd, Tst, Tsc} of the data operation source and the library Tdd, table Tdt, and field Tdc sets Ts = {Tdd, Tdt, Tdc} of the data target end; Module M1.3: Obtain the data task number Td through the data task information Ti, associate Ts and Td to obtain the temporary mapping relationship Rt = {Td, Ts, Td}; expand the defined identification constants S = {D, T, C} for the database D, table T, and field C, and then establish the lineage mapping relationship Ri = {S, Ti, Td, Ts} of the data task Ti in combination with the data task number Td. Module M1.4: The task analysis module saves the lineage mapping relationship Ri to the data task lineage mapping library.

7. The data task analysis system based on data lineage according to claim 5, characterized in that, The said module M3 includes: Input: Data task information Ti; Output: The lineage mapping relationship Ri of the data task Ti; The task lineage change mark set Rc of the data task Ti; Module M3.1: The task analysis module obtains and establishes the lineage mapping relationship Rn = {S, Ti, Tnd, Tns} of the data task Ti through the data task information Ti, and obtains the existing data task lineage mapping relationship Ri = {S, Ti, Tid, Tis} from the data task lineage mapping library. Module M3.2: Traverse each source end and destination end in Rn, and add them to the task lineage change mark set Rc according to the judgment. Module M3.3: Save the obtained task lineage change mark set Rc to the data task lineage mapping library.

8. The data task analysis system based on data lineage according to claim 7, wherein The said module M3.2 specifically includes: Module M3.2.1: Traverse each source end and destination end in Rn. If Tnd = Tid, then judge whether Tns exists in Ri. If it does not exist, define the destination end change CS and create a new task lineage change mark item Ri = {S, Ti, Tnd, Tns, CS}, and add it to the task lineage change mark set Rc. If it exists, continue. Module M3.2.2: If Tns = Tis, then judge whether Tnd exists in Ri. If it does not exist, define the source end change CD and create a new task lineage change mark item Ri = {S, Ti, Tnd, Tns, CD}, and add it to the task lineage change mark set Rc. If it exists, continue. Module M3.2.3: If Tnd!= Tid and Tns!= Tis, then define the new data task lineage mapping relationship CN, create a new task lineage change mark item Ri = {S, Ti, Tnd, Tns, CN}, and add it to the task lineage change mark set Rc.

9. A computer-readable storage medium storing a computer program, characterized in that, When the said computer program is executed by a processor, it realizes the steps of the method described in any one of claims 1 to 4.

Images