Data blood relationship analysis method and device, computer device, and readable storage medium

Abstract

The application relates to the field of data processing, and discloses a data bloodline analysis method and device, computer equipment and a readable storage medium. The method comprises the following steps: obtaining a target change script and constructing a script list; analyzing each script in the script list to obtain a single-script bloodline relationship of each script; obtaining all objects to be calculated for bloodline relationship; and determining a cross-script bloodline relationship of each object by using the single-script bloodline relationship. The method provided by the application directly determines the cross-script bloodline relationship by using the single-script bloodline relationship, does not need to write an SQL script, improves processing efficiency, and provides data services for front-end reporting.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data lineage analysis method, apparatus, computer equipment, and readable storage medium. Background Technology

[0002] The data lineage requirements of the construction machinery industry primarily stem from the data warehouse operation and maintenance process. During data warehouse operation and maintenance, issues such as unreasonable data warehouse layering leading to reverse references, excessively long data execution chains, and excessively long script execution times negatively impact the user experience when querying front-end reports. Discovering data reference chains and influence chains has become a common challenge for data warehouses.

[0003] Currently, data lineage analysis within a single script is primarily achieved by parsing the logical plan during the execution of SQL (Structured Query Language), and cross-script lineage is queried by writing self-referential SQL scripts. However, when the query hierarchy is too large, the self-referential SQL becomes extremely complex, and the query overhead also increases accordingly. Summary of the Invention

[0004] In view of this, the purpose of the embodiments of this application is to provide a data lineage resolution method, apparatus, computer device and readable storage medium, which can solve the problem of low efficiency in existing methods of querying cross-script lineages by writing self-related SQL scripts.

[0005] To solve the above-mentioned technical problems, this application is implemented as follows:

[0006] In a first aspect, embodiments of this application provide a data lineage analysis method, including:

[0007] Obtain the target change script and build a script list;

[0008] Each script in the script list is parsed to obtain the single script lineage of each script;

[0009] Retrieve all objects whose blood relations are to be calculated;

[0010] The cross-script lineage of each object is determined using the single-script lineage.

[0011] According to a specific embodiment disclosed in this application, the step of obtaining the target change script and constructing the script list includes:

[0012] Obtain the target change script;

[0013] The target change script is standardized to obtain a standardized script;

[0014] Based on the standardized script, a script list is constructed.

[0015] According to a specific embodiment disclosed in this application, parsing each script in the script list to obtain the single script lineage of each script includes:

[0016] The SQL statements of each script in the script list are processed to generate an abstract syntax tree for each script;

[0017] Parse the abstract syntax tree of each script to obtain the pre-parsed result of each script;

[0018] If the pre-parsing result meets the preset rules, the script pre-parsing is determined to be successful, and the pre-parsing result is taken as the single script lineage of the script;

[0019] If the pre-parsing result does not conform to the preset rules, it is determined that the script pre-parsing is unsuccessful. The SQL statements of each script are run to generate a logical plan, and the logical plan is parsed to obtain the single script lineage of the script.

[0020] According to a specific embodiment disclosed in this application, determining the cross-script lineage of each object using the single-script lineage includes:

[0021] For each of the objects, a first loop step is performed until the bloodline relationship of all objects is obtained, wherein the first loop step includes:

[0022] Determine the maximum recursion level and the current lineage level of the object, wherein the initial value of the current lineage level is one;

[0023] Determine whether the object meets the first preset condition, wherein the first preset condition includes the existence of a first target object that is the same as the object in the single script lineage relationship, the current lineage level being less than or equal to the maximum recursion level, and not meeting the loop termination condition;

[0024] If the object meets the first preset condition, the first target object is stored as the blood relationship of the current blood relationship level of the object in the blood relationship data set, the current blood relationship level is incremented by one, the next object is determined according to the blood relationship of the first target object, and the step of judging whether the object meets the preset condition is executed.

[0025] If the object does not meet any of the first preset conditions, the bloodline relationships in the bloodline data set are normalized to obtain the cross-script bloodline relationship of the object.

[0026] According to a specific embodiment disclosed in this application, determining the cross-script lineage of each object using the single-script lineage includes:

[0027] The bloodline calculation order for each object is determined based on the single-script bloodline relationship;

[0028] For each of the objects, a second loop step is executed based on the bloodline calculation order until the bloodline relationship of all objects is obtained, wherein the second loop step includes:

[0029] Determine the current lineage level of the object, wherein the initial value of the current lineage level is one;

[0030] Determine whether the object meets the second preset condition, wherein the second preset condition includes the existence of a second target object identical to the object in the single script lineage relationship and failure to meet the termination condition;

[0031] If the object meets the second preset condition, the next object is determined according to the blood relationship of the target object, and it is determined whether there is a second target object that is the same as the next object in the blood relationship data set;

[0032] If a second target object that is identical to the next object exists in the bloodline data set, the bloodline relationship of the second target object and the bloodline relationship of the second target object plus one are taken as the bloodline relationship of the object and stored in the bloodline data set.

[0033] If there is no target object identical to the next object in the bloodline data set, the second target object is stored in the bloodline data set as the bloodline relationship of the current bloodline level of the object;

[0034] If the object does not meet any of the second preset conditions, the bloodline relationships in the bloodline data set are normalized to obtain the cross-script bloodline relationship of the object.

[0035] According to a specific embodiment disclosed in this application, the method further includes:

[0036] Clean up the cross-script lineage of all described objects.

[0037] According to a specific embodiment disclosed in this application, the method further includes:

[0038] Use visualization tools to show the cross-script lineage of the objects.

[0039] Secondly, embodiments of this application provide a data lineage analysis device, comprising:

[0040] The build module is used to obtain the target change script and build a list of scripts;

[0041] The parsing module is used to parse each script in the script list to obtain the single script lineage of each script;

[0042] The acquisition module is used to acquire all objects whose blood relations are to be calculated;

[0043] A determination module is used to determine the cross-script lineage of each object using the single-script lineage.

[0044] Thirdly, embodiments of this application provide a computer device including a processor and a memory, wherein the memory stores a program or instructions, and when the program or instructions are executed by the processor, they implement the steps of the method described in the first aspect.

[0045] Fourthly, embodiments of this application provide a computer-readable storage medium storing a program or instructions that, when executed by a processor, implement the steps of the method described in the first aspect.

[0046] The data lineage analysis method, apparatus, computer equipment, and readable storage medium provided in the above embodiments of this application directly determine cross-script lineage relationships using single-script lineage relationships, eliminating the need to write SQL scripts, thus improving processing efficiency and providing data services for front-end reports.

[0047] Other features and advantages of this application will be described in detail in the following detailed description section. Attached Figure Description

[0048] To more clearly illustrate the technical solutions of this application, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of this application and should not be considered as a limitation on the scope of protection of this application. In the various drawings, similar components are numbered similarly.

[0049] Figure 1 A flowchart of a data lineage analysis method provided in an embodiment of this application is shown;

[0050] Figure 2 A schematic diagram of a data lineage analysis device provided in an embodiment of this application is shown. Detailed Implementation

[0051] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0052] The components of the embodiments of this application described and illustrated in the accompanying drawings can be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of this application provided in the drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0053] In the following, the terms “comprising,” “having,” and their cognates, which may be used in various embodiments of this application, are intended only to indicate a particular feature, number, step, operation, element, component, or combination thereof, and should not be construed as excluding, firstly, the presence of one or more other features, numbers, steps, operations, elements, components, or combinations thereof, or adding the possibility of one or more features, numbers, steps, operations, elements, components, or combinations thereof.

[0054] Furthermore, the terms "first," "second," and "third" are used only to distinguish descriptions and should not be interpreted as indicating or implying relative importance.

[0055] Unless otherwise specified, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which the various embodiments of this application pertain. Terms (such as those defined in commonly used dictionaries) shall be interpreted as having the same meaning as in their contextual meaning in the relevant technical field and shall not be construed as having an idealized or overly formal meaning, unless clearly defined in the various embodiments of this application.

[0056] Please see Figure 1 , Figure 1 A flowchart of the data lineage analysis method provided in the embodiments of this application is shown below. Figure 1 As shown, the method includes the following steps:

[0057] Step 110: Obtain the target change script and build a script list.

[0058] Specifically, data lineage refers to the origin or influence of tables or fields in a data warehouse. This application uses data lineage as an example for illustration. During the operation of a data warehouse, script changes may occur, leading to changes in data lineage. Therefore, it is necessary to periodically obtain the target change scripts. If this is the first time data lineage analysis is performed, all scripts need to be obtained; that is, the target change scripts include all scripts. If this is not the first time data lineage analysis is performed, only change scripts within a preset period need to be obtained. For example, if the preset period is one day, then when data lineage analysis is performed at 6 AM today, change scripts from 6 AM the previous day to 6 AM today are obtained as the target change scripts. It can be understood that the script list includes all target change scripts.

[0059] In one optional implementation, step 110 includes:

[0060] Obtain the target change script;

[0061] The target change script is standardized to obtain a standardized script;

[0062] Based on the standardized script, a script list is constructed.

[0063] Specifically, when the target modification script uses a non-standard framework, such as Spark, the script contains server configuration parameters, script execution configurations, and time parameters, which cannot meet the parsing requirements. Therefore, the target modification script needs to be standardized first. The standardization process includes comparing the target modification script with scripts using standard frameworks and replacing redundant parameters in the target modification script. For example, the time parameter is replaced with a fixed value.

[0064] Initialization can be performed before step 110 to obtain intermediate and result tables for the data lineage resolution process. The intermediate and result tables are designed as follows:

[0065]

[0066] The tables in numbers 1 to 7 contain fields such as source system (database name of the source table), source table name, source field, target system (database name of the target table), target table, target field, script name (script name of the currently parsed lineage relationship), script path (path of the parsed script), hierarchy (hierarchy of the current lineage relationship in the lineage relationship tree), and target column of the lineage relationship (identifying which table or field the current lineage relationship belongs to). The tables in numbers 1 to 7 have the same format for easy management. The table in number 8 contains the script name, script path, whether it is parsed correctly (1 for yes, 0 for no), and pre-parsing time.

[0067] Step 120: Parse each script in the script list to obtain the single script lineage of each script.

[0068] Specifically, each script is parsed to obtain its single-script lineage, so that cross-script lineage can be obtained later using the single-script lineage.

[0069] In one optional implementation, step 120 includes:

[0070] The SQL statements of each script in the script list are processed to generate an abstract syntax tree for each script;

[0071] Parse the abstract syntax tree of each script to obtain the pre-parsed result of each script;

[0072] If the pre-parsing result meets the preset rules, the script pre-parsing is determined to be successful, and the pre-parsing result is taken as the single script lineage of the script;

[0073] If the pre-parsing result does not conform to the preset rules, it is determined that the script pre-parsing is unsuccessful. The SQL statements of each script are run to generate a logical plan, and the logical plan is parsed to obtain the single script lineage of the script.

[0074] Specifically, this application improves the efficiency of data lineage resolution by performing pre-parsing. The pre-parsing adopts offline parsing, that is, it processes the SQL statements of each script in the script list, generates an abstract syntax tree for each script, parses the abstract syntax tree of each script, and obtains the pre-parsing result of each script.

[0075] Offline parsing is based on SQL scripts and does not require actual execution of the statements in the SQL scripts, so it has advantages in parsing speed and efficiency; however, since it cannot rely on the metadata of the data warehouse, offline parsing has high requirements for the writing standards of SQL scripts. For example, offline parsing cannot parse statements in SQL scripts containing *.

[0076] If the pre-parsing result conforms to the preset rules, it indicates that the script pre-parsing was successful, and the pre-parsing result can be used as the single script lineage. If the pre-parsing result does not conform to the preset rules, it indicates that the script pre-parsing was unsuccessful, and online parsing is required to obtain the single script lineage. This involves running the SQL statements of each script, generating a logical plan, and parsing the logical plan to obtain the single script lineage. The criterion for whether script pre-parsing is successful can be whether it can be stored in tables numbered 1 to 7. For example, if the field length of the data result parsed offline is 8, while the preset rule requires a field length of 9, it does not conform to the preset rule and cannot be stored in the result table, indicating that the parsing failed. The result of successful pre-parsing can be stored in table number 8, and the single script lineage can be stored in tables numbered 5 to 7.

[0077] Step 130: Obtain all objects whose blood relations are to be calculated.

[0078] Specifically, the object can be a table or a field. It can be obtained by deduplicating the target system, target table, and target field in the table with serial number 6.

[0079] Step 140: Determine the cross-script lineage of each object using the single-script lineage.

[0080] Specifically, based on the single-script lineage of a script, the tables (fields) referenced by the tables (fields) within that script can be obtained. Since single-script lineages are unrelated, the ultimate source or influence of the tables (fields) cannot be determined. For example, through single-script lineages, it is known that object A originates from object B, object D originates from object C, object B originates from object C, and object Z originates from object Y. However, this single-script lineage does not reveal whether object A and object C are related. In reality, within the lineages of object A originating from object B and object B originating from object C, two lineage chains can be formed: "A->B->C" and "B->C". "A->B" is at the first level of the lineage chain "A->B->C"; "B->C" is at the second level of the lineage chain "A->B->C" and at the first level of the lineage chain "B->C". Both "A->B" and "B->C" are lineages of object A, and "B->C" is also a lineage of object B. Therefore, the cross-script lineage of each object can be determined using single-script lineage. The following will specifically describe two methods provided in this application for determining the cross-script lineage of each object using single-script lineage.

[0081] In one optional implementation, step 140 includes:

[0082] For each of the objects, a first loop step is performed until the bloodline relationship of all objects is obtained, wherein the first loop step includes:

[0083] Determine the maximum recursion level and the current lineage level of the object, wherein the initial value of the current lineage level is one;

[0084] Determine whether the object meets the first preset condition, wherein the first preset condition includes the existence of a first target object that is the same as the object in the single script lineage relationship, the current lineage level being less than or equal to the maximum recursion level, and not meeting the loop termination condition;

[0085] If the object meets the first preset condition, the first target object is stored as the blood relationship of the current blood relationship level of the object in the blood relationship data set, the current blood relationship level is incremented by one, the next object is determined according to the blood relationship of the first target object, and the step of judging whether the object meets the preset condition is executed.

[0086] If the object does not meet any of the first preset conditions, the bloodline relationships in the bloodline data set are normalized to obtain the cross-script bloodline relationship of the object.

[0087] Specifically, first, determine the maximum recursion level and the current lineage level of the object. The maximum recursion level is to prevent infinite loops. For example, if object A originates from object B, object B originates from object C, and object C originates from object A, without limiting the maximum recursion level, an infinite loop will occur. The initial value of the object's current lineage level is one.

[0088] When an object meets the first preset condition, namely, there is a first target object with the same name in the single script lineage, the current lineage level is less than or equal to the maximum recursive level, and the loop termination condition is not met, it means that the object's lineage relationship has been found. Then, the first target object is stored as the lineage relationship of the object's current lineage level in the lineage data set, the current lineage level is incremented by one, and the object in the lineage relationship is used as the next object for recursive search.

[0089] If an object does not meet any of the following conditions: there is no first target object identical to the object in the single script lineage, the current lineage level is greater than the maximum recursion level, or the loop termination condition is met, the search can end. The lineage relationships in the lineage data set are then regularized, for example, by adding information such as the calculation time and script path of the lineage relationship data, thereby obtaining the cross-script lineage relationship of the object.

[0090] In one optional implementation, step 140 includes:

[0091] The bloodline calculation order for each object is determined based on the single-script bloodline relationship;

[0092] For each of the objects, a second loop step is executed based on the bloodline calculation order until the bloodline relationship of all objects is obtained, wherein the second loop step includes:

[0093] Determine the current lineage level of the object, wherein the initial value of the current lineage level is one;

[0094] Determine whether the object meets the second preset condition, wherein the second preset condition includes the existence of a second target object identical to the object in the single script lineage relationship and failure to meet the termination condition;

[0095] If the object meets the second preset condition, the next object is determined according to the blood relationship of the target object, and it is determined whether there is a second target object that is the same as the next object in the blood relationship data set;

[0096] If a second target object that is identical to the next object exists in the bloodline data set, the bloodline relationship of the second target object and the bloodline relationship of the second target object plus one are taken as the bloodline relationship of the object and stored in the bloodline data set.

[0097] If there is no target object identical to the next object in the bloodline data set, the second target object is stored in the bloodline data set as the bloodline relationship of the current bloodline level of the object;

[0098] If the object does not meet any of the second preset conditions, the bloodline relationships in the bloodline data set are normalized to obtain the cross-script bloodline relationship of the object.

[0099] In the first method of looping, the objects in the script list are unordered, resulting in a large number of redundant calculations when recursively searching for lineage data. For example, in a single script lineage set, object A originates from object B, object B originates from object C, object C originates from object D, and object D originates from object E. When calculating all lineage relationships of objects A, B, C, and D in the order of A, B, C, and D, the relationship that object B originates from object C is calculated once when calculating all lineage relationships of object A, and then again when calculating all lineage relationships of object B, for a total of two calculations. Similarly, the relationship that object C originates from object D is calculated once when calculating all lineage relationships of object A, once when calculating all lineage relationships of object B, and then once again when calculating all lineage relationships of object C, for a total of three calculations.

[0100] Therefore, this application also provides a second method to reduce redundant calculations and avoid recursive operations, thereby greatly improving computational efficiency.

[0101] Specifically, the difference from the first method is that the order of calculating the lineage of each object is first determined based on the lineage of the single script.

[0102] If the lineage calculation order is determined for the first time, first obtain the cross-script lineage relationships of all tables according to the steps of the first method, and then calculate the calculation order for each table. Specifically, this includes: first finding all lineage relationships related to the object to construct a lineage relationship tree, calculating the object's level in each tree, and taking the highest level number as the object's calculation order. The calculation order can be stored in an intermediate table.

[0103] If the lineage calculation order is not determined for the first time, meaning the intermediate table already contains the lineage calculation order, then the lineage calculation order is updated based on the single-script lineage relationships in the script list. Specifically, this involves: first obtaining the calculation order of all source tables in the single-script lineage relationship, taking the largest value and adding 1 to it as the calculation order of the scripts in the script list. Alternatively, the calculation order can be stored in the intermediate table.

[0104] Since the order of kinship calculation is determined beforehand, recursion in the first method can be avoided, thus reducing redundant calculations. The second target object is directly searched in the kinship dataset. If the second target object exists, the second target object and its kinship hierarchy relationship (plus one) are stored as the object's kinship relationship in the kinship dataset; if the second target object does not exist, it is directly stored as the object's current kinship hierarchy relationship in the kinship dataset.

[0105] Understandably, the lineage data set is empty during the first execution of the second loop step. Subsequent executions of the second loop step may then store the lineage relationships of the objects calculated previously.

[0106] The cross-script lineage relationships obtained through the two methods described above both have a lineage hierarchy, which allows for a clear understanding of the lineage relationships of the data, i.e., the reference chain or the influence chain.

[0107] In one optional implementation, the method further includes:

[0108] Clean up the cross-script lineage of all described objects.

[0109] Specifically, after obtaining the cross-script lineage of all objects, data cleaning can be performed to remove potential errors and ensure data consistency.

[0110] In one optional implementation, the method further includes:

[0111] Use visualization tools to show the cross-script lineage of the objects.

[0112] Specifically, the obtained cross-script lineage relationships are integrated into a lineage tree for display, allowing users to understand data lineage more intuitively. The cross-script lineage relationships of some objects can be displayed based on user actions, or the cross-script lineage relationships of all objects can be displayed.

[0113] The technical solution of this application will be illustrated below with examples.

[0114] The tables and fields in a data warehouse include:

[0115] The customer information table i_ip_cust_info includes the following fields: customer ID cust_id, customer name cust_name, and customer's province cust_prov.

[0116] The customer information table s_cust_info includes the following fields: customer ID cust_id, customer name cust_name, and customer's province cust_prov.

[0117] The product information table i_ip_goods_info includes the following fields: product ID (goods_id), product name (goods_name), and product price (goods_price).

[0118] The order details table i_sal_order_info includes the following fields: order number (order_id), order item number (order_item_id), order time (order_time), order customer number (order_cust_id), order customer name (order_cust_name), order product number (order_goods_id), order product name (order_goods_name), quantity, price, and amount (amt).

[0119] The wide table a_xs_order_wtbl for orders includes the following fields: province (prov) and sales amount (sale_amt).

[0120] The order wide table 1, a_xs_order_wtbl_1, includes the fields: area and sales amount.

[0121] The data warehouse already contains scripts and their contents, including:

[0122] Customer Information 1i_ip_cust_info.sql (Retrieves customer ID, customer name, and customer's province from Customer Information 2, and inserts them into Customer Information 1 table);

[0123] Customer Information 2s_cust_info.sql (Retrieves customer ID, customer name, and customer's province from the o_cust_info table and inserts them into Customer Information 2 table);

[0124] Order details i_sal_order_info.sql (Retrieves the product number from the product information table, and inserts it into the order details table according to the correspondence between orders and products);

[0125] The order wide table a_xs_order_wtbl.sql (retrieves customer ID and corresponding amount from the order details table, retrieves the province of the customer corresponding to the customer ID from the customer information table 1, calculates the sales amount for each province, and inserts it into the order wide table);

[0126] The product information table `i_ip_goods_info.sql` retrieves the product ID, product name, and product price from the `s_goods_info` table and inserts them into the product information table.

[0127] The new script and its contents include: Order Wide Table 1a_xs_order_wtbl_1.sql (retrieves the province and corresponding sales amount from the Order Wide Table, calculates the sales amount of each region by accumulating the correspondence between the province and the region, and inserts it into the Order Wide Table 1).

[0128] It is understandable that the target change script is the same as the new script. Tables without special annotations come from the business system and therefore do not belong to the data warehouse.

[0129] The target change script is parsed to obtain the single script lineage, as shown in the table below:

[0130] Results Table A

[0131]

[0132]

[0133]

[0134] After combining the three columns of target system, target table, and target field in result table A and removing duplicates, we can obtain all the objects whose lineage needs to be calculated.

[0135] The following example, using the calculated lineage relationship between field a.a_xs_order_wtbl.prov and field i.i_ip_cust_info.cust_prov, illustrates the differences between the two methods for obtaining cross-script lineage relationships.

[0136] The first method:

[0137] The first step is to find the lineage of the field a.a_xs_order_wtbl.prov.

[0138] First, search for data in result table A where the target system is 'a', the target table is 'a_xs_order_wtbl', and the target field is 'prov'. The data found is as follows:

[0139]

[0140] Since the current lineage level is 1 and the lineage field is a.a_xs_order_wtbl.prov, the lineage relationship set contains the following data:

[0141] i i_ip_cust_info cust_prov a a_xs_order_wtbl prov a_xs_order_wtbl 1 a.a_xs_order_wtbl.prov

[0142] Then, in result table A, search for data with target system i, target table i_ip_cust_info, and target field cust_prov. The data is found as follows:

[0143] S s_cust_info cust_prov i i_ip_cust_info cust_prov i_ip_cust_info

[0144] The current lineage level is 1+1=2, the lineage field is a.a_xs_order_wtbl.prov, and the lineage relationship set contains the following data:

[0145] i i_ip_cust_info cust_prov a a_xs_order_wtbl prov a_xs_order_wtbl 1 a.a_xs_order_wtbl.prov S s_cust_info cust_prov i i_ip_cust_info cust_prov a_xs_order_wtbl 2 a.a_xs_order_wtbl.prov

[0146] Next, in result table A, search for data where the target system is S, the target table is s_cust_info, and the target field is cust_prov.

[0147] o o_cust_info cust_prov S s_cust_info cust_prov s_cust_info

[0148] The current lineage level is 2+1=3, the lineage field is a.a_xs_order_wtbl.prov, and the lineage relationship set contains the following data:

[0149] i i_ip_cust_info cust_prov a a_xs_order_wtbl prov a_xs_order_wtbl 1 a.a_xs_order_wtbl.prov S s_cust_info cust_prov i i_ip_cust_info cust_prov a_xs_order_wtbl 2 a.a_xs_order_wtbl.prov o o_cust_info cust_prov S s_cust_info cust_prov a_xs_order_wtbl 3 a.a_xs_order_wtbl.prov

[0150] Finally, search for data in result table A with target system 'o', target table 'o_cust_info', and target field 'cust_prov'. If not found, terminate the recursive search. Save the above lineage data to result table B.

[0151] Results Table B

[0152]

[0153]

[0154] The second step is to find the lineage of the field i.i_ip_cust_info.cust_prov.

[0155] First, in result table A, search for data with target system i, target table i_ip_cust_info, and target field cust_prov. The data is found as follows:

[0156] S s_cust_info cust_prov i i_ip_cust_info cust_prov i_ip_cust_info

[0157] The current lineage level is 1, the lineage field is i.i_ip_cust_info.cust_prov, and the lineage relationship set contains the following data:

[0158] S s_cust_info cust_prov i i_ip_cust_info cust_prov i_ip_cust_info 1 i.i_ip_cust_info.cust_prov

[0159] Next, search for data in result table A where the target system is S, the target table is s_cust_info, and the target field is cust_prov. The data found is as follows:

[0160] o o_cust_info cust_prov S s_cust_info cust_prov s_cust_info

[0161] The current lineage level is 1+1=2, the lineage field is a.a_xs_order_wtbl.prov, and the lineage relationship set contains the following data:

[0162] Results Table C

[0163] S s_cust_info cust_prov i i_ip_cust_info cust_prov i_ip_cust_info 1 i.i_ip_cust_info.cust_prov o o_cust_info cust_prov S s_cust_info cust_prov i_ip_cust_info 2 i.i_ip_cust_info.cust_prov

[0164] Finally, search for data in result table A with target system 'o', target table 'o_cust_info', and target field 'cust_prov'. If no data is found, terminate the recursive search.

[0165] Combining result table B with result table C yields the final blood relation result table D:

[0166] Results Table D

[0167]

[0168]

[0169] The second method: First, the lineage calculation order of field a.a_xs_order_wtbl.prov and field i.i_ip_cust_info.cust_prov is determined. If this is the initial calculation of cross-script lineage, then according to Table D, the calculation order of field i_ip_cust_info is 2. i_ip_cust_info has two lineage paths: "o.o_cust_info.cust_prov->s.s_cust_info.cust_prov->i_ip_cust_info.cust_prov" and "o.o_cust_info.cust_prov->s.s_cust_info.cust_prov->i.i_ip_cust_info.cust_prov->a.a_xs_order_wtbl.prov". The level of field i_ip_cust_info.cust_prov is 2 in both cases, so the maximum value of 2 is taken. Therefore, the lineage calculation order of field i.i_ip_cust_info.cust_prov is 2.

[0170] If this is not the first calculation of cross-script lineage, assume that the calculation order field s.s_cust_info is 1, the field i.i_ip_cust_info is 2, and the field a.a_xs_order_wtbl is 3.

[0171] For the target change script order wide table 1, the lineage relationship "a_xs_order_wtbl->a.a_xs_order_wtbl_1" is obtained from the order wide table 1. Then the calculation order of a.a_xs_order_wtbl_1 should be 1 more than a.a_xs_order_wtbl, which is 4.

[0172] Then, calculate the cross-script lineage according to the above calculation order. Since 4 is greater than 2, calculate the lineage of field i.i_ip_cust_info.cust_prov first, which will give you the result table C.

[0173] Next, calculate the lineage of field a.a_xs_order_wtbl.prov, and search for data in result table A where the target system is 'a', the target table is 'a_xs_order_wtbl', and the target field is 'prov'. The data found is as follows:

[0174] i i_ip_cust_info cust_prov a a_xs_order_wtbl prov a_xs_order_wtbl

[0175] The current lineage level is 1, and the lineage field is a.a_xs_order_wtbl.prov. Then, search for the lineage of field i.i_ip_cust_info.cust_prov in the lineage database to obtain result table C. Increment the level in result table C by one, modify the lineage field information, and combine it with result table D to obtain the cross-script lineage relationship of field a.a_xs_order_wtbl.prov.

[0176] If no data is found with target system 'a', target table 'a_xs_order_wtbl', and target field 'prov', then directly add the current lineage level and object to obtain result table E as the cross-script lineage relationship of field a.a_xs_order_wtbl.prov.

[0177] Results Table E

[0178] i i_ip_cust_info cust_prov a a_xs_order_wtbl prov a_xs_order_wtbl 1 a.a_xs_order_wtbl.prov

[0179] The data lineage parsing method provided in this application embodiment directly uses single-script lineage relationships to determine cross-script lineage relationships, eliminating the need to write SQL scripts, thus improving processing efficiency and providing data services for front-end reports.

[0180] For a method embodiment corresponding to the above, please refer to [link / reference]. Figure 2 , Figure 2 This is a schematic diagram of the data lineage analysis device provided in the embodiments of this application, as shown below. Figure 2 As shown, the data lineage analysis device 1000 includes:

[0181] Module 1001 is used to obtain the target change script and build a script list;

[0182] The parsing module 1002 is used to parse each script in the script list to obtain the single script lineage of each script;

[0183] Module 1003 is used to retrieve all objects whose blood relations are to be calculated.

[0184] The determination module 1004 is used to determine the cross-script lineage of each object using the single script lineage.

[0185] The data lineage analysis device provided in this application embodiment can realize each process of the data lineage analysis method in the method embodiment and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0186] Optionally, this application embodiment also provides a computer device, including a processor and a memory, wherein the memory stores a program or instructions, which, when executed by the processor, implement the various processes of the above-described data lineage analysis method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0187] Optionally, embodiments of this application also provide a computer-readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described data lineage analysis method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0188] The processor is the processor in the computer device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0189] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that, in alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0190] In addition, the functional modules or units in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0191] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a smartphone, personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0192] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A data lineage analysis method, characterized in that, The method comprises the following steps: acquiring target change scripts and constructing a script list; parsing each script in the script list to obtain a single-script blood relationship of each script; acquiring all objects to be calculated for blood relationship; determining a cross-script blood relationship of each object by using the single-script blood relationship; the step of determining a cross-script blood relationship of each object by using the single-script blood relationship comprises: for each object, performing a first loop step until the blood relationship of all objects is obtained, wherein the first loop step comprises: determining a maximum recursion level and a current blood level of the object, wherein the initial value of the current blood level is one; judging whether the object meets a first preset condition, wherein the first preset condition comprises that there is a first target object same as the object in the single-script blood relationship, the current blood level is less than or equal to the maximum recursion level, and a loop termination condition is not met; in the case that the object meets the first preset condition, storing the first target object as the blood relationship of the current blood level of the object to a blood data set, increasing the current blood level by one, determining a next object according to the blood relationship of the first target object, and performing the step of judging whether the object meets the preset condition; in the case that the object does not meet any first preset condition, regularizing the blood relationship in the blood data set to obtain the cross-script blood relationship of the object. 2.The data pedigree analysis method of claim 1, wherein, The step of acquiring target change scripts and constructing a script list comprises: acquiring target change scripts; standardizing the target change scripts to obtain standardized scripts; constructing a script list based on the standardized scripts. 3.The data provenance analysis method of claim 1, wherein, The step of parsing each script in the script list to obtain a single-script blood relationship of each script comprises: processing SQL statements of each script in the script list to generate an abstract syntax tree of each script; parsing the abstract syntax tree of each script to obtain a pre-parsing result of each script; in the case that the pre-parsing result meets a preset rule, determining that the script pre-parsing is successful, and taking the pre-parsing result as the single-script blood relationship of the script; in the case that the pre-parsing result does not meet the preset rule, determining that the script pre-parsing is not successful, running the SQL statements of each script to generate a logical plan, and parsing the logical plan to obtain the single-script blood relationship of the script. 4.The data provenance analysis method of claim 1, wherein, The step of determining a cross-script blood relationship of each object by using the single-script blood relationship comprises: determining a blood calculation order of each object according to the single-script blood relationship; for each object, performing a second loop step based on the blood calculation order until the blood relationship of all objects is obtained, wherein the second loop step comprises: determining a current blood level of the object, wherein the initial value of the current blood level is one; determining whether the object meets a second preset condition, wherein the second preset condition comprises that a second target object identical to the object exists in the single-script blood relationship and does not meet a termination condition; in a case where the object meets the second preset condition, determining a next object according to the blood relationship of the target object, and determining whether a second target object identical to the next object exists in the blood data set; in a case where the blood data set contains a second target object identical to the next object, storing the second target object and a blood relationship of the second target object after the blood relationship of the second target object is incremented by one as the blood relationship of the object to the blood data set; in a case where the blood data set does not contain a target object identical to the next object, storing the second target object as the blood relationship of the current blood level of the object to the blood data set; in a case where the object does not meet any of the second preset conditions, regularizing the blood relationship in the blood data set to obtain the cross-script blood relationship of the object. 5.The data provenance analysis method of claim 1, wherein, The method further comprises: cleaning the cross-script blood relationship of all the objects. 6.The data kinship resolving method of claim 1 or 5, wherein, The method further comprises: displaying the cross-script blood relationship of the object by using a visualization tool.

7. A data lineage resolution apparatus, comprising: The method comprises: a construction module configured to obtain a target change script and construct a script list; a parsing module configured to parse each script in the script list to obtain a single-script blood relationship of each script; an obtaining module configured to obtain all objects whose blood relationship is to be calculated; a determination module configured to determine a cross-script blood relationship of each object by using the single-script blood relationship; the determination module is further configured to: for each object, perform a first loop step until the blood relationship of all the objects is obtained, wherein the first loop step comprises: determining a maximum recursive level and a current blood level of the object, wherein the initial value of the current blood level is one; determining whether the object meets a first preset condition, wherein the first preset condition comprises that a first target object identical to the object exists in the single-script blood relationship, the current blood level is less than or equal to the maximum recursive level, and a loop termination condition is not met; in a case where the object meets the first preset condition, storing the first target object as the blood relationship of the current blood level of the object to a blood data set, incrementing the current blood level by one, determining a next object according to the blood relationship of the first target object, and performing the step of determining whether the object meets a preset condition; in a case where the object does not meet any of the first preset conditions, regularizing the blood relationship in the blood data set to obtain the cross-script blood relationship of the object.

8. A computer device, comprising: The system comprises a processor and a memory, wherein the memory stores a program or instructions, and the program or instructions are executed by the processor to implement the steps of the data blood relationship analysis method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores programs or instructions, which are executed by the processor to implement the steps of the data kinship analysis method according to any one of claims 1-6.

Images