Data lineage tracing method, device, electronic device, medium and product

Abstract

The present invention provides a data lineage tracing method, device, electronic device, medium and product, wherein the data lineage tracing method comprises: respectively determining the flow relationship between metadata objects of each data processing component in the target data processing link, the first mapping relationship between the field in the first data object and the output field of the data loading component, the second mapping relationship between the field in the first data object and the input field of the data extraction component, and the third mapping relationship between the field in the second data object and the input field of the data extraction component; then according to the second mapping relationship and the third mapping relationship, the association relationship between the field in the first data object and the field in the second data object is obtained. The present invention can accurately, quickly and with high coverage make the most downstream data of the link correspond to the most upstream data of the link under a complex ETL process, and realize the data source and process can be explained.

Description

Technical Field

[0001] The present invention relates to the technical field of electronic digital data processing, and in particular to a data lineage tracing method, device, electronic equipment, medium and product. Background Art

[0002] Lineage tracing (i.e. lineage analysis) is a technical means to comprehensively track the data processing process, so as to find all the related metadata objects starting from a certain data object and the relationship between these metadata objects. The relationship between metadata objects specifically refers to the input and output relationship of the data flow representing these metadata objects, which can be used to locate program operation problems, difference analysis, indicator fluctuation analysis, data physical examination and evaluation, etc.

[0003] At present, the implementation methods of lineage tracking mainly include scheduler reverse deduction, lineage tracking interface provided by computing engine system, parsing process source code of computing engine system and general SQL parser tool.

[0004] Among them, although the scheduler reverse deduction method is very feasible and low-cost, its disadvantages are also obvious: it cannot support field-level lineage, and field relationships are very important for metadata, and are even one of the important bases for subsequent expansion of data quality and data security. Therefore, this method is not suitable for business impact analysis.

[0005] The lineage tracking interface provided by the computing engine system (such as HIVE) supports field-level lineage tracking and is low-cost. However, since the computing engine and lineage tracking are bound together, adding the lineage tracking function will inevitably have an impact on the computing engine, making the entire system poorly coupled and unable to be customized according to needs.

[0006] Although the source code of the parsing process of the computing engine system can be customized according to needs, if the computing engine system does not provide a parser, the application scope will be small due to the different rules of different computing engines.

[0007] The general SQL parser tool approach has a low maturity due to the large workload of the SQL parsing process and the need for a high level of proficiency in grammatical rules. Manual intervention is required for supplementary corrections during the lineage tracing process (such as Apache Atlas), and the accuracy of the tracing results is low.

[0008] In summary, existing bloodline tracing methods have problems such as low accuracy, poor applicability, and low coverage. Summary of the invention

[0009] The present invention provides a data lineage tracing method, device, electronic device, medium and product to solve the above-mentioned defects.

[0010] The present invention provides a data lineage tracing method, comprising: determining a flow relationship between metadata objects of various data processing components in a target data processing link; wherein the data processing components include at least a data extraction component and a data loading component; determining a first mapping relationship between a field in a first data object obtained by a data loading process executed by the data loading component and an output field of the metadata object of the data loading component; determining a data lineage tracing order according to the reverse order of the flow relationship, determining a mapping relationship between an input field and an output field of each data processing component in turn according to the first mapping relationship and according to the data lineage tracing order, and obtaining a second mapping relationship between a field in the first data object and an input field of the metadata object of the data extraction component; determining a third mapping relationship between a field in a second data object as a data source and an input field of the metadata object of the data extraction component; and obtaining an association relationship between a field in the first data object and a field in the second data object according to the second mapping relationship and the third mapping relationship.

[0011] According to a data lineage tracing method provided by the present invention, the data processing component also includes a data conversion component, and the data conversion component includes a first data conversion component, a second data conversion component and a third data conversion component; wherein the input field and the output field of the metadata object of the first data conversion component have a one-to-one correspondence; the metadata object of the second data conversion component includes an output field obtained by modifying the input field; the metadata object of the third data conversion component includes an output field obtained by adding the input field; accordingly, the mapping relationship between the input field and the output field of each data processing component is determined in sequence according to the first mapping relationship and according to the data lineage tracing order, and the second mapping relationship between the field in the first data object and the input field of the metadata object of the data extraction component is obtained, including: taking the first mapping relationship as the current mapping relationship and taking the data loading component as the current data processing component; determining the type of the current data processing component; in the case where the current data processing component is the data loading component or the first data conversion component In the following, the current mapping relationship is not adjusted, and then according to the data lineage tracing order, the next data processing component is used as the current data processing component, and the step of determining the type of the current data processing component is re-executed; when the current data processing component is the second data conversion component, the current mapping relationship is modified to the mapping relationship between the field in the first data object and the input field of the metadata object of the current data processing component, and then according to the data lineage tracing order, the next data processing component is used as the current data processing component, and the step of determining the type of the current data processing component is re-executed; when the current data processing component is the third data conversion component, the current mapping relationship is modified to the mapping relationship between the field in the first data object and the source input field of the metadata object of the current data processing component, and then according to the data lineage tracing order, the next data processing component is used as the current data processing component, and the step of determining the type of the current data processing component is re-executed; when the current data processing component is the data extraction component, the current mapping relationship is used as the second mapping relationship.

[0012] According to a data lineage tracing method provided by the present invention, the current mapping relationship is modified to be a mapping relationship between a field in a first data object and an input field of a metadata object of a current data processing component, comprising: obtaining a field having a mapping relationship with a field in the first data object from the current mapping relationship; determining the input field corresponding to the field having a mapping relationship with a field in the first data object according to the conversion logic of the current data processing component; and modifying the value of the field having a mapping relationship with the field in the first data object to be the value of the corresponding input field, to obtain a modified current mapping relationship.

[0013] According to a data lineage tracing method provided by the present invention, the current mapping relationship is modified to be a mapping relationship between a field in a first data object and a source input field of a metadata object of a current data processing component, comprising: obtaining a field having a mapping relationship with a field in the first data object from the current mapping relationship; determining, according to the conversion logic of the current data processing component, a source input field corresponding to a field having a mapping relationship with a field in the first data object, wherein the source input field is an input field of the metadata object of the current data processing component; and modifying the value of the field having a mapping relationship with a field in the first data object to be the value of the corresponding source input field, to obtain a modified current mapping relationship.

[0014] According to a data lineage tracing method provided by the present invention, the method further includes: storing the association relationship between the field in the first data object and the field in the second data object in a database.

[0015] The present invention also provides a data lineage tracking device, comprising: a flow relationship determination module, used to determine the flow relationship between the metadata objects of each data processing component in the target data processing link; wherein the data processing component includes at least a data extraction component and a data loading component; a first mapping determination module, used to determine the first mapping relationship between the field in the first data object obtained by the data loading process executed by the data loading component and the output field of the metadata object of the data loading component; a second mapping acquisition module, used to determine the data lineage tracking order according to the reverse order of the flow relationship, and determine the mapping relationship between the input field and the output field of each data processing component in turn according to the first mapping relationship and according to the data lineage tracking order, and obtain the second mapping relationship between the field in the first data object and the input field of the metadata object of the data extraction component; a third mapping determination module, used to determine the third mapping relationship between the field in the second data object as the data source and the input field of the metadata object of the data extraction component; an association acquisition module, used to obtain the association relationship between the field in the first data object and the field in the second data object according to the second mapping relationship and the third mapping relationship.

[0016] A data lineage tracing device provided according to the present invention includes: the data processing component also includes a data conversion component, the data conversion component includes a first data conversion component, a second data conversion component and a third data conversion component; wherein the input field and the output field of the metadata object of the first data conversion component have a one-to-one correspondence; the metadata object of the second data conversion component includes an output field obtained by modifying the input field; the metadata object of the third data conversion component includes an output field obtained by adding the input field; accordingly, the second mapping acquisition module is specifically used to: use the first mapping relationship as the current mapping relationship, and use the data loading component as the current data processing component; determine the type of the current data processing component; when the current data processing component is the data loading component or the first data conversion component, do not adjust the current mapping relationship, and then use the next data processing component as the current one according to the data lineage tracing order After determining the type of the current data processing component, re-execute the step of determining the type of the current data processing component; when the current data processing component is the second data conversion component, modify the current mapping relationship to the mapping relationship between the field in the first data object and the input field of the metadata object of the current data processing component, and then according to the data lineage tracing order, take the next data processing component as the current data processing component, and re-execute the step of determining the type of the current data processing component; when the current data processing component is the third data conversion component, modify the current mapping relationship to the mapping relationship between the field in the first data object and the source input field of the metadata object of the current data processing component, and then according to the data lineage tracing order, take the next data processing component as the current data processing component, and re-execute the step of determining the type of the current data processing component; when the current data processing component is the data extraction component, take the current mapping relationship as the second mapping relationship.

[0017] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-mentioned data lineage tracing methods are implemented.

[0018] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any of the above-mentioned data lineage tracing methods are implemented.

[0019] The present invention also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned data lineage tracing methods when executed by a processor.

[0020] The present invention provides a data lineage tracing method, device, electronic device, medium and product, wherein the data lineage tracing method determines the mapping relationship between the input field and the output field of each data processing component according to the reverse order of the flow relationship, and obtains the second mapping relationship between the field in the first data object and the input field of the metadata object of the data extraction component based on the first mapping relationship between the field in the first data object obtained by the data loading process performed by the data loading component and the output field of the metadata object of the data loading component, and then obtains the association relationship between the field in the first data object and the field in the second data object based on the third mapping relationship between the field in the second data object as the data source and the input field of the metadata object of the data extraction component, so that in a complex ETL process, the first data object at the downstream end of the link corresponds to the second data object at the upstream end of the link accurately, quickly and with high coverage, so as to realize the explainability of data source and process, and effectively avoid the problems of redundant links, redundant fields or inaccurate tracking faced in forward lineage tracing. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 It is a flowchart of a data lineage tracing method provided by an embodiment of the present invention;

[0023] Figure 2 is a schematic diagram of flow direction relationship provided by an embodiment of the present invention;

[0024] Figure 3 A schematic diagram of the structure of a data lineage tracing device provided by an embodiment of the present invention;

[0025] Figure 4 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] Figure 1 is a flow chart of a data lineage tracing method provided by an embodiment of the present invention; Figure 1 As shown, the data lineage tracing method may include the following steps:

[0028] Step 101, determining the flow relationship between metadata objects of various data processing components in the target data processing link; wherein the data processing components at least include a data extraction component and a data loading component.

[0029] The target data processing link refers to the data processing link in the ETL (full name Extract-Transform-Load) process. The ETL process is used to describe the process of extracting, transforming, and loading data from the source to the destination. Specifically, data from distributed and heterogeneous data sources (such as relational data, flat data files, etc.) is extracted to a temporary intermediate layer, then cleaned, transformed, integrated, and finally loaded into a data warehouse or data mart, becoming the basis for online analytical processing and data mining.

[0030] The data processing component corresponds to the ETL process, including data extraction component, data conversion component and data loading component. Among them, the data extraction component is at the upstream of the target data processing link and is responsible for providing data; the data conversion component has a large amount of data operation logic and is responsible for converting and cleaning the data to make the data available; the data loading component is at the downstream of the target data processing link and is responsible for the final landing of the data and storing the data cleaned by the data conversion component.

[0031] A metadata object refers to an intermediate process object generated each time the current data is processed in the entire ETL process. The object can explain how the data is converted from the previous step to this step. The metadata object stores the mapping relationship between the upstream data and the current data.

[0032] In this step, the JSON data involved in the entire target data processing link is first parsed, and then the flow relationship between the metadata objects of each data processing component is determined based on the parsed JSON data, that is, the dependency relationship between the metadata objects corresponding to the data extraction component, data conversion component and data loading component is determined.

[0033] Step 102: determine a first mapping relationship between a field in a first data object obtained by a data loading process performed by the data loading component and an output field of a metadata object of the data loading component.

[0034] In this embodiment, the fields in the first data object refer to the fields in the data domain table after actual implementation. The first mapping relationship between the fields in the first data object and the output fields of the metadata object of the data loading component can be represented by the ID corresponding to the fields in the data domain table and the output fields of the metadata object of the data loading component. It can also be directly represented by the correspondence between the fields in the data domain table and the output fields of the metadata object of the data loading component. The present invention is not limited to this.

[0035] Table 1 Schematic table of the first mapping relationship

[0036]

[0037] As shown in Table 1, the first mapping relationship can represent the corresponding relationship between the landed data and the output data of the metadata object of the data loading component.

[0038] Step 103: determine the data lineage tracing order according to the reverse order of the flow relationship, determine the mapping relationship between the input field and the output field of each data processing component in turn according to the first mapping relationship and the data lineage tracing order, and obtain the second mapping relationship between the field in the first data object and the input field of the metadata object of the data extraction component.

[0039] The reverse order of the flow relationship refers to the order of metadata object generation deduced from the dependency relationship between metadata objects. Figure 2 Give an example.

[0040] Figure 2 is a schematic diagram of flow direction relationship provided by an embodiment of the present invention; Figure 2 The flow relationship between the metadata objects of each data processing component is that the output of metadata object 1 and metadata object 2 in the data extraction component becomes the input of metadata object 4 in the data conversion component, and the output of metadata object 3 in the data extraction component and metadata object 4 in the data conversion component becomes the input of metadata object 5 in the data conversion component. The reverse order of the above flow relationship corresponds to metadata object 5->metadata object 3 and metadata object 4, where metadata object 4->metadata object 1 and metadata object 2.

[0041] In this step, the correspondence between the input field and the output field of each data processing component is determined in reverse order (including the correspondence between the input field and the output field of the metadata object of the data loading component, the correspondence between the input field and the output field of the metadata object of the data conversion component, and the correspondence between the input field and the output field of the metadata object of the data extraction component), and then the correspondence between the field in the first data object and the input field of the metadata object of the data extraction component is determined according to the correspondence between the field in the first data object and the output field of the metadata object of the data loading component, and it is used as the second mapping relationship.

[0042] Step 104: determine a third mapping relationship between a field in a second data object serving as a data source and an input field of the metadata object of the data extraction component.

[0043] In this embodiment, the field in the second data object refers to the physical field of the data source. The third mapping relationship between the physical field of the data source and the input field of the metadata object of the data extraction component can be represented by the correspondence between the ID corresponding to the physical field of the data source and the input field of the metadata object of the data extraction component, or it can be directly represented by the correspondence between the physical field of the data source and the input field of the metadata object of the data extraction component. The present invention is not limited to this.

[0044] Table 2 Schematic diagram of the third mapping relationship

[0045]

[0046]

[0047] The third mapping relationship shown in Table 2 can represent the corresponding relationship between the source data and the input data of the metadata object of the data extraction component.

[0048] In this step, the correspondence between different data sources and metadata objects of data extraction components is determined, that is, from which data source the input data of the metadata object of each data extraction component comes, and the correspondence is used as the third mapping relationship.

[0049] Step 105: Obtain an association relationship between a field in the first data object and a field in the second data object according to the second mapping relationship and the third mapping relationship.

[0050] Specifically, it is known from step 103 and step 104 that the correspondence between the fields in the first data object and the input fields of the metadata object of the data extraction component, and the correspondence between the fields in the second data object and the input fields of the metadata object of the data extraction component, the association relationship between the fields in the first data object and the fields in the second data object can be obtained, that is, the lineage tracing between the landed data and the data source is completed.

[0051] The data lineage tracing method provided by the embodiment of the present invention determines the mapping relationship between the input field and the output field of each data processing component according to the reverse order of the flow relationship, and obtains the second mapping relationship between the field in the first data object and the input field of the metadata object of the data extraction component based on the first mapping relationship between the field in the first data object obtained by the data loading process executed by the data loading component and the output field of the metadata object of the data loading component. Then, based on the third mapping relationship between the field in the second data object as the data source and the input field of the metadata object of the data extraction component, the association relationship between the field in the first data object and the field in the second data object is obtained. In this way, in a complex ETL process, the first data object at the downstream end of the link is accurately, quickly and with high coverage to correspond to the second data object at the upstream end of the link, so as to realize explainable data source and process, and effectively avoid the problems of redundant links, redundant fields or inaccurate tracking faced in forward lineage tracing.

[0052] Furthermore, the data processing component also includes a data conversion component, and the data conversion component includes a first data conversion component, a second data conversion component and a third data conversion component.

[0053] Among them, the input field and output field of the metadata object of the first data conversion component have a one-to-one correspondence; the metadata object of the second data conversion component includes the output field modified based on the input field; the metadata object of the third data conversion component includes the output field added based on the input field.

[0054] The first data conversion component, the second data conversion component and the third data conversion component are described below with reference to examples.

[0055] In the first data conversion component, as shown in Table 3, the output fields of the metadata object corresponding to the upstream component A are id, name, and age, respectively. The input fields of the metadata object corresponding to component B, which is the output end of component A, are the output fields of component A, that is, the input fields of component B are also id, name, and age. It can be seen that the characteristics of the first data conversion component type (conversion component with one-to-one correspondence between ETL process fields, component A and component B are both first data conversion components) are that the input field id corresponds to the output field id, the input field name corresponds to the output field name, and the input field age corresponds to the output field age. It cannot change the input or output field name, and cannot modify the input field source (that is, the input field id of component B is derived from the id of component A, and cannot be modified to the id of component C or component D), but the number of fields can be deleted. For example, if the input fields of the metadata object of component B only select id and name, then the corresponding output fields of component A are also only id and name.

[0056] Table 3 Schematic diagram of input and output fields of the first data conversion component

[0057]

[0058] In the second data conversion component, as shown in Table 4, the input fields of the metadata object of component B only select id and name, so its two output fields are id and name. If the output of component B is connected to component C (component C is the second data conversion component), the input fields of the metadata object of component C are id and name. At this time, id and name can be selected as the output fields of component C, that is, the output fields of the metadata object corresponding to component C are id and name. In addition, for component C of the second data conversion component type, the user can manually change the names of id and name, for example, modify them to id111, name222, then the output fields of the metadata object of component C are id111 and name222, which will be applied to subsequent processes (for example, component C is connected to component D, and the input fields of component D are id111, name222). It can be seen from this that the feature of the second data conversion component is that the output metadata object field can change the name based on the input metadata object field, and the number of metadata object fields can be deleted, but the upstream source of the metadata object field cannot be changed, and new metadata object fields cannot be added.

[0059] Table 4 Schematic diagram of input and output fields of the second data conversion component

[0060]

[0061] In the third data conversion component, as shown in Table 5, component D (component D is the third data conversion component) is the downstream component of component C (i.e., the second data conversion component mentioned above). The output fields of the metadata object corresponding to component C are id111, name222. Then the input fields of the metadata object of component D are also id111, name222. At this time, a series of configuration operations will be performed on the input fields in component D, such as changing the field name222 into two fields, name222 and name333. Here, the upstream data sources (lineage) of name222 and name333 are the same, both of which are the output field name222 of component C. At this time, the output fields of component D are id111, name222, and name333, i.e., new fields are added on the basis of the original metadata object input fields. From this, it can be seen that the characteristic of the third data conversion component is that the output metadata object field can add new metadata object fields based on the input metadata object field, and can also delete the number of metadata object fields, but it cannot change the upstream source of the original metadata object field, nor can it change the field name of the original metadata object field.

[0062] Table 5 Schematic diagram of input and output fields of the third data conversion component

[0063]

[0064] Correspondingly, the mapping relationship between the input field and the output field of each data processing component is determined in sequence according to the first mapping relationship and according to the data lineage tracing order, and the second mapping relationship between the field in the first data object and the input field of the metadata object of the data extraction component is obtained, including:

[0065] Using the first mapping relationship as the current mapping relationship and using the data loading component as the current data processing component;

[0066] Determine the type of current data processing component;

[0067] In the case where the current data processing component is a data loading component or a first data conversion component, the current mapping relationship is not adjusted, and then according to the data lineage tracing order, after the next data processing component is used as the current data processing component, the step of determining the type of the current data processing component is re-executed;

[0068] In the case where the current data processing component is the second data conversion component, the current mapping relationship is modified to be a mapping relationship between a field in the first data object and an input field of the metadata object of the current data processing component, and then according to the data lineage tracing order, after the next data processing component is taken as the current data processing component, the step of determining the type of the current data processing component is re-executed;

[0069] In the case where the current data processing component is the third data conversion component, the current mapping relationship is modified to be a mapping relationship between a field in the first data object and a source input field of the metadata object of the current data processing component, and then according to the data lineage tracing order, after the next data processing component is taken as the current data processing component, the step of determining the type of the current data processing component is re-executed;

[0070] When the current data processing component is a data extraction component, the current mapping relationship is used as the second mapping relationship.

[0071] Specifically, starting from the most downstream component of the target data processing link (i.e., the ETL process) - the data loading component, each data processing component on the target data processing link is judged, and the first mapping relationship is processed according to the judgment result. Taking the entire target data processing link as an example, the processing process of the first mapping relationship is described in detail:

[0072] First, the data loading component is used as the current data processing component, and the first mapping relationship is used as the current mapping relationship.

[0073] Then, the type of the current data processing component is determined. In this case, the type of the current data processing component is a data loading component. In this case, the first mapping relationship is not adjusted and remains as it is.

[0074] Next, according to the data lineage tracing order (i.e. the reverse order of the flow relationship, the flow relationship of the ETL process is extraction-transformation-loading, and the reverse order is loading-transformation-extraction), the data conversion component is used as the current data processing component, and the type of the current data processing component is judged.

[0075] If it is the first data conversion component, the first mapping relationship is not adjusted and remains the same.

[0076] If it is the second data conversion component, the original first mapping relationship is modified, specifically, the first mapping relationship is modified to a mapping relationship between a field in the first data object and an input field of the metadata object of the second data conversion component.

[0077] If it is the third data conversion component, the original first mapping relationship is modified, specifically, the first mapping relationship is modified to be a mapping relationship between a field in the first data object and a source input field of the metadata object of the third data conversion component.

[0078] After the data conversion component is processed, the data extraction component is used as the current data processing component according to the data lineage tracing order. At this time, the first mapping relationship adjusted by the data conversion component needs to be used as the second mapping relationship. So far, under the entire ETL process, the adjustment of the first mapping relationship has been completed, and the second mapping relationship representing the relationship between the field in the first data object and the input field of the metadata object of the data extraction component is obtained.

[0079] It should be noted that in different ETL processes, the data conversion component may be the first data conversion component, the second data conversion component, the third data conversion component or a combination of multiple components, and the original first mapping relationship may undergo multiple adjustments in the meantime, which is not limited in this embodiment.

[0080] The data lineage tracing method provided by the embodiment of the present invention first determines the type of each data processing component according to the data lineage tracing order, and then determines how to adjust the first mapping relationship according to the determination result, thereby being able to accurately trace the source of different types of data processing components, making the data source and process explainable, and effectively avoiding the problems of redundant links, redundant fields or inaccurate tracking encountered in forward lineage tracing.

[0081] Furthermore, the modifying of the current mapping relationship into a mapping relationship between a field in the first data object and an input field of the metadata object of the current data processing component includes:

[0082] Acquire a field having a mapping relationship with a field in the first data object from the current mapping relationship;

[0083] Determine, according to the conversion logic of the current data processing component, an input field corresponding to a field having a mapping relationship with a field in the first data object;

[0084] The value of the field having a mapping relationship with the field in the first data object is modified to the value of the corresponding input field to obtain a modified current mapping relationship.

[0085] Specifically, if the data conversion component is the second data conversion component, the fields in the original first mapping relationship that have a mapping relationship with the fields in the first data object are modified to the corresponding input field values, thereby obtaining the modified first mapping relationship. Taking the first mapping relationship shown in Table 1 above as an example, the field IDs in the current first data object are 1, 2, and 3 respectively; the output fields of the metadata objects of the corresponding data loading components are id, name, and age (that is, the input fields of the metadata objects of the data loading components or the output fields of the data conversion components). The input fields of the metadata objects of the data conversion components (that is, the input fields or output fields of the metadata objects of the data extraction components) are id1, name1, and age1. At this time, the values ​​of the fields in the first data object that have a mapping relationship are modified to the values ​​of the corresponding input fields, as shown in Table 6.

[0086] Table 6 Modified first mapping relationship table 1

[0087]

[0088] The data lineage tracing method provided by the embodiment of the present invention modifies the first mapping relationship according to the conversion logic of the second data conversion component to obtain a modified first mapping relationship, so that the final landed data can accurately correspond to the input field of the metadata object of the data extraction component, thereby enabling accurate lineage tracing.

[0089] Furthermore, the modifying of the current mapping relationship into a mapping relationship between a field in the first data object and a source input field of the metadata object of the current data processing component includes:

[0090] Acquire a field having a mapping relationship with a field in the first data object from the current mapping relationship;

[0091] According to the conversion logic of the current data processing component, determine the source input field corresponding to the field having a mapping relationship with the field in the first data object, wherein the source input field is the input field of the metadata object of the current data processing component;

[0092] The value of the field having a mapping relationship with the field in the first data object is modified to the value of the corresponding source input field (ie, the input field of the third data conversion component) to obtain a modified current mapping relationship.

[0093] Specifically, if the data conversion component is the third data conversion component, the fields in the original first mapping relationship that have a mapping relationship with the fields in the first data object are modified to the corresponding source input fields to obtain the modified first mapping relationship. Taking the first mapping relationship shown in Table 1 above as an example, the field IDs in the current first data object are 1, 2, and 3 respectively; the output fields of the metadata object of the corresponding data loading component are id, name, and age (that is, the input fields of the metadata object of the data loading component or the output fields of the data conversion component). The input fields of the metadata object of the data conversion component (that is, the input fields or output fields of the metadata object of the data extraction component) are id and info, where name and age are both derived from info. At this time, the modified current mapping relationship is shown in Table 7.

[0094] Table 7 Modified first mapping relationship Table 2

[0095]

[0096] In combination with Table 2 and Table 7, if the data conversion component is the third data conversion component, then in the mapping process of the input and output fields corresponding to the data loading component, the third data conversion component and the data extraction component, the modified first mapping relationship Table 2 shown in Table 7 is the second mapping relationship, and the association relationship between the fields in the first data object and the fields in the second data object is as shown in Table 8.

[0097] Table 8 Relationship between data domain table and data source

[0098]

[0099] The data lineage tracing method provided by the embodiment of the present invention modifies the first mapping relationship according to the conversion logic of the third data conversion component to obtain a modified first mapping relationship, so that the final landed data can accurately correspond to the input field of the metadata object of the data extraction component, thereby enabling accurate lineage tracing.

[0100] Furthermore, the method further comprises:

[0101] The association relationship between the field in the first data object and the field in the second data object is stored in a database.

[0102] In this implementation, the association relationship (i.e., the lineage tracing result) between the fields in the data domain (i.e., the fields in the first data object) and the fields in the data source (i.e., the fields in the second data object) is stored, thereby facilitating subsequent data analysis based on the lineage tracing results, such as program operation problem location, difference analysis, indicator fluctuation analysis, data physical examination and evaluation, etc.

[0103] The data lineage tracing method provided by the embodiment of the present invention stores accurate lineage tracing results, thereby facilitating the calling of the lineage tracing results when performing other data analysis, thereby improving efficiency and shortening time.

[0104] The data lineage tracing device provided by the present invention is described below. The data lineage tracing device described below and the data lineage tracing method described above can be referenced to each other.

[0105] Figure 3 A schematic diagram of the structure of a data lineage tracing device provided in an embodiment of the present invention is shown in FIG. Figure 3 As shown, a data lineage tracking device includes:

[0106] The flow direction relationship determination module 301 is used to determine the flow direction relationship between metadata objects of each data processing component in the target data processing link.

[0107] Wherein, the data processing component at least includes a data extraction component and a data loading component.

[0108] The target data processing link refers to the data processing link in the ETL (full name Extract-Transform-Load) process. The ETL process is used to describe the process of extracting, transforming, and loading data from the source to the destination. Specifically, data from distributed and heterogeneous data sources (such as relational data, flat data files, etc.) is extracted to a temporary intermediate layer, then cleaned, transformed, integrated, and finally loaded into a data warehouse or data mart, becoming the basis for online analytical processing and data mining.

[0109] The data processing component corresponds to the ETL process, including data extraction component, data conversion component and data loading component. Among them, the data extraction component is at the upstream of the target data processing link and is responsible for providing data; the data conversion component has a large amount of data operation logic and is responsible for converting and cleaning the data to make the data available; the data loading component is at the downstream of the target data processing link and is responsible for the final landing of the data and storing the data cleaned by the data conversion component.

[0110] A metadata object refers to an intermediate process object generated each time the current data is processed in the entire ETL process. The object can explain how the data is converted from the previous step to this step. The metadata object stores the mapping relationship between the upstream data and the current data.

[0111] In the flow relationship determination module 301, the json data involved in the entire target data processing link is first parsed, and then the flow relationship between the metadata objects of each data processing component is determined based on the parsed json data, that is, the dependency relationship between the metadata objects corresponding to the data extraction component, the data conversion component and the data loading component is determined.

[0112] The first mapping determination module 302 is used to determine a first mapping relationship between a field in a first data object obtained by the data loading process performed by the data loading component and an output field of a metadata object of the data loading component.

[0113] In this embodiment, the fields in the first data object refer to the fields in the data domain table after actual implementation. The first mapping relationship between the fields in the first data object and the output fields of the metadata object of the data loading component can be represented by the ID corresponding to the fields in the data domain table and the output fields of the metadata object of the data loading component. It can also be directly represented by the correspondence between the fields in the data domain table and the output fields of the metadata object of the data loading component. The present invention is not limited to this.

[0114] The second mapping acquisition module 303 is used to determine the data lineage tracing order according to the reverse order of the flow relationship, and determine the mapping relationship between the input field and the output field of each data processing component in turn according to the first mapping relationship and the data lineage tracing order, so as to obtain the second mapping relationship between the field in the first data object and the input field of the metadata object of the data extraction component.

[0115] The reverse order of the flow relationship refers to the order of generating metadata objects deduced from the dependency relationship between metadata objects.

[0116] exist Figure 2 The flow relationship between the metadata objects of each data processing component is that the output of metadata object 1 and metadata object 2 in the data extraction component becomes the input of metadata object 4 in the data conversion component, and the output of metadata object 3 in the data extraction component and metadata object 4 in the data conversion component becomes the input of metadata object 5 in the data conversion component. The reverse order of the above flow relationship corresponds to metadata object 5->metadata object 3 and metadata object 4, where metadata object 4->metadata object 1 and metadata object 2.

[0117] In the second mapping acquisition module 303, the correspondence between the input field and the output field of each data processing component is determined in reverse order (including the correspondence between the input field and the output field of the metadata object of the data loading component, the correspondence between the input field and the output field of the metadata object of the data conversion component, and the correspondence between the input field and the output field of the metadata object of the data extraction component), and then the correspondence between the field in the first data object and the input field of the metadata object of the data extraction component is determined according to the correspondence between the field in the first data object and the output field of the metadata object of the data loading component, and it is used as the second mapping relationship.

[0118] The third mapping determination module 304 is used to determine a third mapping relationship between a field in the second data object as a data source and an input field of the metadata object of the data extraction component.

[0119] In this embodiment, the field in the second data object refers to the physical field of the data source. The third mapping relationship between the physical field of the data source and the input field of the metadata object of the data extraction component can be represented by the correspondence between the ID corresponding to the physical field of the data source and the input field of the metadata object of the data extraction component, or it can be directly represented by the correspondence between the physical field of the data source and the input field of the metadata object of the data extraction component. The present invention is not limited to this.

[0120] In the third mapping determination module 304, the correspondence between different data sources and metadata objects of the data extraction component is determined, that is, which data source the input data of the metadata object of each data extraction component comes from, and the correspondence is used as the third mapping relationship.

[0121] The association acquisition module 305 is used to obtain the association relationship between the field in the first data object and the field in the second data object according to the second mapping relationship and the third mapping relationship.

[0122] Specifically, the second mapping acquisition module 303 and the third mapping determination module 304 know the correspondence between the fields in the first data object and the input fields of the metadata object of the data extraction component, and the correspondence between the fields in the second data object and the input fields of the metadata object of the data extraction component. The association relationship between the fields in the first data object and the fields in the second data object can be obtained, that is, the lineage tracing between the landed data and the data source is completed.

[0123] The data lineage tracing device provided by the embodiment of the present invention determines the mapping relationship between the input field and the output field of each data processing component according to the reverse order of the flow relationship, and obtains the second mapping relationship between the field in the first data object and the input field of the metadata object of the data extraction component based on the first mapping relationship between the field in the first data object obtained in the data loading process executed by the data loading component and the output field of the metadata object of the data loading component. Then, based on the third mapping relationship between the field in the second data object as the data source and the input field of the metadata object of the data extraction component, the association relationship between the field in the first data object and the field in the second data object is obtained. In this way, in a complex ETL process, the first data object at the downstream end of the link is accurately, quickly and with high coverage to correspond to the second data object at the upstream end of the link, so as to realize the explainability of data source and process, and effectively avoid the problems of redundant links, redundant fields or inaccurate tracking faced in forward lineage tracing.

[0124] Furthermore, the data processing component further includes a data conversion component, and the data conversion component includes a first data conversion component, a second data conversion component and a third data conversion component. The first data conversion component, the second data conversion component and the third data conversion component are described below with reference to examples.

[0125] In the first data conversion component, as shown in Table 3, assuming that the output fields of the metadata object corresponding to the upstream component A are id, name, and age, respectively, then the input fields of the metadata object corresponding to component B, which is the output end of component A, are the output fields of component A, that is, the input fields of component B are also id, name, and age. It can be seen that the characteristics of the first data conversion component type (conversion component with one-to-one correspondence between ETL process fields, component A and component B are both first data conversion components) are that the input field id corresponds to the output field id, the input field name corresponds to the output field name, and the input field age corresponds to the output field age. It cannot change the input or output field name, and cannot modify the input field source (that is, the input field id of component B is derived from the id of component A, and cannot be modified to the id of component C or component D), but the number of fields can be deleted. For example, if the input fields of the metadata object of component B only select id and name, then the corresponding output fields of component A are also only id and name.

[0126] In the second data conversion component, as shown in Table 4, the input fields of the metadata object of component B only select id and name, so its two output fields are id and name. If the output of component B is connected to component C (component C is the second data conversion component), the input fields of the metadata object of component C are id and name. At this time, id and name can be selected as the output fields of component C, that is, the output fields of the metadata object corresponding to component C are id and name. In addition, for component C of the second data conversion component type, the user can manually change the names of id and name, for example, modify them to id111, name222, then the output fields of the metadata object of component C are id111 and name222, which will be applied to subsequent processes (for example, component C is connected to component D, and the input fields of component D are id111, name222). It can be seen from this that the feature of the second data conversion component is that the output metadata object field can change the name based on the input metadata object field, and the number of metadata object fields can be deleted, but the upstream source of the metadata object field cannot be changed, and new metadata object fields cannot be added.

[0127] In the third data conversion component, as shown in Table 5, component D (component D is the third data conversion component) is the downstream component of component C (i.e., the second data conversion component mentioned above). The output fields of the metadata object corresponding to component C are id111, name222. Then the input fields of the metadata object of component D are also id111, name222. At this time, a series of configuration operations will be performed on the input fields in component D, such as changing the field name222 into two fields, name222 and name333. Here, the upstream data sources (lineage) of name222 and name333 are the same, both of which are the output field name222 of component C. At this time, the output fields of component D are id111, name222, and name333, i.e., new fields are added on the basis of the original metadata object input fields. From this, it can be seen that the characteristic of the third data conversion component is that the output metadata object field can add new metadata object fields based on the input metadata object field, and can also delete the number of metadata object fields, but it cannot change the upstream source of the original metadata object field, nor can it change the field name of the original metadata object field.

[0128] Among them, the input field and output field of the metadata object of the first data conversion component have a one-to-one correspondence; the metadata object of the second data conversion component includes the output field modified based on the input field; the metadata object of the third data conversion component includes the output field added based on the input field.

[0129] Accordingly, the second mapping acquisition module is specifically used for:

[0130] Using the first mapping relationship as the current mapping relationship and using the data loading component as the current data processing component;

[0131] Determine the type of current data processing component;

[0132] In the case where the current data processing component is a data loading component or a first data conversion component, the current mapping relationship is not adjusted, and then according to the data lineage tracing order, after the next data processing component is used as the current data processing component, the step of determining the type of the current data processing component is re-executed;

[0133] In the case where the current data processing component is the second data conversion component, the current mapping relationship is modified to be a mapping relationship between a field in the first data object and an input field of the metadata object of the current data processing component, and then according to the data lineage tracing order, after the next data processing component is taken as the current data processing component, the step of determining the type of the current data processing component is re-executed;

[0134] In the case where the current data processing component is the third data conversion component, the current mapping relationship is modified to be a mapping relationship between a field in the first data object and a source input field of the metadata object of the current data processing component, and then according to the data lineage tracing order, after the next data processing component is taken as the current data processing component, the step of determining the type of the current data processing component is re-executed;

[0135] When the current data processing component is a data extraction component, the current mapping relationship is used as the second mapping relationship.

[0136] Specifically, starting from the most downstream component of the target data processing link (i.e., the ETL process) - the data loading component, each data processing component on the target data processing link is judged, and the first mapping relationship is processed according to the judgment result. Taking the entire target data processing link as an example, the processing process of the first mapping relationship is described in detail:

[0137] First, the data loading component is used as the current data processing component, and the first mapping relationship is used as the current mapping relationship.

[0138] Then, the type of the current data processing component is determined. In this case, the type of the current data processing component is a data loading component. In this case, the first mapping relationship is not adjusted and remains as it is.

[0139] Next, according to the data lineage tracing order (i.e. the reverse order of the flow relationship, the flow relationship of the ETL process is extraction-transformation-loading, and the reverse order is loading-transformation-extraction), the data conversion component is used as the current data processing component, and the type of the current data processing component is judged.

[0140] If it is the first data conversion component, the first mapping relationship is not adjusted and remains the same.

[0141] If it is the second data conversion component, the original first mapping relationship is modified, specifically, the first mapping relationship is modified to a mapping relationship between a field in the first data object and an input field of the metadata object of the second data conversion component.

[0142] If it is the third data conversion component, the original first mapping relationship is modified, specifically, the first mapping relationship is modified to be a mapping relationship between a field in the first data object and a source input field of the metadata object of the third data conversion component.

[0143] After the data conversion component is processed, the data extraction component is used as the current data processing component according to the data lineage tracing order. At this time, the first mapping relationship adjusted by the data conversion component needs to be used as the second mapping relationship. So far, under the entire ETL process, the adjustment of the first mapping relationship has been completed, and the second mapping relationship representing the relationship between the field in the first data object and the input field of the metadata object of the data extraction component is obtained.

[0144] The data lineage tracing device provided by the embodiment of the present invention first determines the type of each data processing component according to the data lineage tracing order, and then determines how to adjust the first mapping relationship according to the determination result, thereby being able to accurately trace the source of different types of data processing components, making the data source and process explainable, and effectively avoiding the problems of redundant links, redundant fields or inaccurate tracking encountered in forward lineage tracing.

[0145] Figure 4 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention, such as Figure 4As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the data lineage tracing method, including: determining the flow relationship between the metadata objects of each data processing component in the target data processing link; wherein the data processing component at least includes a data extraction component and a data loading component;

[0146] Determine a first mapping relationship between a field in a first data object obtained by a data loading process performed by the data loading component and an output field of a metadata object of the data loading component;

[0147] Determine a data lineage tracing order according to the reverse order of the flow relationship, determine the mapping relationship between the input field and the output field of each data processing component according to the first mapping relationship and the data lineage tracing order, and obtain a second mapping relationship between the field in the first data object and the input field of the metadata object of the data extraction component;

[0148] Determine a third mapping relationship between a field in a second data object as a data source and an input field of a metadata object of the data extraction component;

[0149] According to the second mapping relationship and the third mapping relationship, an association relationship between the field in the first data object and the field in the second data object is obtained.

[0150] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0151] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the data lineage tracing method provided by the above methods, including: determining the flow relationship between metadata objects of each data processing component in the target data processing link; wherein the data processing component at least includes a data extraction component and a data loading component;

[0152] Determine a first mapping relationship between a field in a first data object obtained by a data loading process performed by the data loading component and an output field of a metadata object of the data loading component;

[0153] Determine a data lineage tracing order according to the reverse order of the flow relationship, determine the mapping relationship between the input field and the output field of each data processing component according to the first mapping relationship and the data lineage tracing order, and obtain a second mapping relationship between the field in the first data object and the input field of the metadata object of the data extraction component;

[0154] Determine a third mapping relationship between a field in a second data object as a data source and an input field of a metadata object of the data extraction component;

[0155] According to the second mapping relationship and the third mapping relationship, an association relationship between the field in the first data object and the field in the second data object is obtained.

[0156] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the data lineage tracing method provided by the above methods, including: determining the flow relationship between metadata objects of each data processing component in the target data processing link; wherein the data processing component at least includes a data extraction component and a data loading component;

[0157] Determine a first mapping relationship between a field in a first data object obtained by a data loading process performed by the data loading component and an output field of a metadata object of the data loading component;

[0158] Determine a data lineage tracing order according to the reverse order of the flow relationship, determine the mapping relationship between the input field and the output field of each data processing component according to the first mapping relationship and the data lineage tracing order, and obtain a second mapping relationship between the field in the first data object and the input field of the metadata object of the data extraction component;

[0159] Determine a third mapping relationship between a field in a second data object as a data source and an input field of a metadata object of the data extraction component;

[0160] According to the second mapping relationship and the third mapping relationship, an association relationship between the field in the first data object and the field in the second data object is obtained.

[0161] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0162] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0163] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data lineage tracing method, characterized in that: include: Determine the flow relationship between metadata objects of each data processing component in the target data processing link; wherein the data processing component includes at least a data extraction component and a data loading component; Determine a first mapping relationship between a field in a first data object obtained by a data loading process performed by the data loading component and an output field of a metadata object of the data loading component; Determine a data lineage tracing order according to the reverse order of the flow relationship, determine the mapping relationship between the input field and the output field of each data processing component according to the first mapping relationship and the data lineage tracing order, and obtain a second mapping relationship between the field in the first data object and the input field of the metadata object of the data extraction component; Determine a third mapping relationship between a field in a second data object as a data source and an input field of a metadata object of the data extraction component; Obtaining an association relationship between a field in the first data object and a field in the second data object according to the second mapping relationship and the third mapping relationship; The data processing component further includes a data conversion component, and the data conversion component includes a first data conversion component, a second data conversion component and a third data conversion component; The metadata object of the first data conversion component has a one-to-one correspondence between the input field and the output field; the metadata object of the second data conversion component includes an output field modified based on the input field; the metadata object of the third data conversion component includes an output field added based on the input field; Correspondingly, the mapping relationship between the input field and the output field of each data processing component is determined in sequence according to the first mapping relationship and according to the data lineage tracing order, and the second mapping relationship between the field in the first data object and the input field of the metadata object of the data extraction component is obtained, including: Using the first mapping relationship as the current mapping relationship and using the data loading component as the current data processing component; Determine the type of current data processing component; In the case where the current data processing component is a data loading component or a first data conversion component, the current mapping relationship is not adjusted, and then according to the data lineage tracing order, after the next data processing component is used as the current data processing component, the step of determining the type of the current data processing component is re-executed; In the case where the current data processing component is the second data conversion component, the current mapping relationship is modified to be a mapping relationship between a field in the first data object and an input field of the metadata object of the current data processing component, and then according to the data lineage tracing order, after the next data processing component is taken as the current data processing component, the step of determining the type of the current data processing component is re-executed; In the case where the current data processing component is the third data conversion component, the current mapping relationship is modified to be a mapping relationship between a field in the first data object and a source input field of the metadata object of the current data processing component, and then according to the data lineage tracing order, after the next data processing component is taken as the current data processing component, the step of determining the type of the current data processing component is re-executed; When the current data processing component is a data extraction component, the current mapping relationship is used as the second mapping relationship.

2. The data lineage tracing method according to claim 1, characterized in that: The step of modifying the current mapping relationship to a mapping relationship between a field in the first data object and an input field of the metadata object of the current data processing component includes: Acquire a field having a mapping relationship with a field in the first data object from the current mapping relationship; Determine, according to the conversion logic of the current data processing component, an input field corresponding to a field having a mapping relationship with a field in the first data object; The value of the field having a mapping relationship with the field in the first data object is modified to the value of the corresponding input field to obtain a modified current mapping relationship.

3. The data lineage tracing method according to claim 1, characterized in that: The step of modifying the current mapping relationship to a mapping relationship between a field in the first data object and a source input field of the metadata object of the current data processing component includes: Acquire a field having a mapping relationship with a field in the first data object from the current mapping relationship; According to the conversion logic of the current data processing component, determine the source input field corresponding to the field having a mapping relationship with the field in the first data object, wherein the source input field is the input field of the metadata object of the current data processing component; The value of the field having a mapping relationship with the field in the first data object is modified to the value of the corresponding source input field to obtain a modified current mapping relationship.

4. The data lineage tracing method according to any one of claims 1 to 3, characterized in that: The method also includes: The association relationship between the field in the first data object and the field in the second data object is stored in a database.

5. A data lineage tracing device, characterized in that: The invention is characterized by comprising: A flow direction relationship determination module, used to determine the flow direction relationship between metadata objects of various data processing components in the target data processing link; wherein the data processing components at least include a data extraction component and a data loading component; A first mapping determination module, used to determine a first mapping relationship between a field in a first data object obtained by a data loading process performed by the data loading component and an output field of a metadata object of the data loading component; A second mapping acquisition module is used to determine the data lineage tracing order according to the reverse order of the flow relationship, determine the mapping relationship between the input field and the output field of each data processing component according to the first mapping relationship and the data lineage tracing order, and obtain the second mapping relationship between the field in the first data object and the input field of the metadata object of the data extraction component; A third mapping determination module, used to determine a third mapping relationship between a field in a second data object as a data source and an input field of a metadata object of the data extraction component; an association acquisition module, configured to obtain an association relationship between a field in the first data object and a field in the second data object according to the second mapping relationship and the third mapping relationship; The data processing component further includes a data conversion component, and the data conversion component includes a first data conversion component, a second data conversion component and a third data conversion component; The metadata object of the first data conversion component has a one-to-one correspondence between the input field and the output field; the metadata object of the second data conversion component includes an output field modified based on the input field; the metadata object of the third data conversion component includes an output field added based on the input field; Accordingly, the second mapping acquisition module is specifically used for: Using the first mapping relationship as the current mapping relationship and using the data loading component as the current data processing component; Determine the type of current data processing component; In the case where the current data processing component is a data loading component or a first data conversion component, the current mapping relationship is not adjusted, and then according to the data lineage tracing order, after the next data processing component is used as the current data processing component, the step of determining the type of the current data processing component is re-executed; In the case where the current data processing component is the second data conversion component, the current mapping relationship is modified to be a mapping relationship between a field in the first data object and an input field of the metadata object of the current data processing component, and then according to the data lineage tracing order, after the next data processing component is taken as the current data processing component, the step of determining the type of the current data processing component is re-executed; In the case where the current data processing component is the third data conversion component, the current mapping relationship is modified to be a mapping relationship between a field in the first data object and a source input field of the metadata object of the current data processing component, and then according to the data lineage tracing order, after the next data processing component is taken as the current data processing component, the step of determining the type of the current data processing component is re-executed; When the current data processing component is a data extraction component, the current mapping relationship is used as the second mapping relationship.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the data lineage tracing method according to any one of claims 1 to 4 are implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data lineage tracing method according to any one of claims 1 to 4 are implemented.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the data lineage tracing method according to any one of claims 1 to 4 are implemented.

Images